Considering that the audio sample's time is always increased, the decoded sample's time decoder returns should always be equal or larger than its demuxed time.
When the decoded sample's time is smaller than the time of first demuxed sample, that time would probably cause a problem so we should throw an error for that.
Differential Revision: https://phabricator.services.mozilla.com/D28167
--HG--
extra : moz-landing-system : lando
Considering that the audio sample's time is always increased, the decoded sample's time decoder returns should always be equal or larger than its demuxed time.
When the decoded sample's time is smaller than the time of first demuxed sample, that time would probably cause a problem so we should throw an error for that.
Differential Revision: https://phabricator.services.mozilla.com/D28167
--HG--
extra : moz-landing-system : lando
If the RDD process is enabled and fails to start properly for some reason,
make sure we cannot accidentally fall back to decoding AV1 on the content
process.
Differential Revision: https://phabricator.services.mozilla.com/D28204
--HG--
extra : moz-landing-system : lando
It allows for more readable code, not having to store multiple times different storage type across multiple objects.
Now each class does one task and only deal with a single texture data type.
Differential Revision: https://phabricator.services.mozilla.com/D26473
--HG--
extra : moz-landing-system : lando
YUVColorSpace is inseparable from the bit depth as the matrix coefficients to be calculated need the bit depth information.
So let's put the two types together. gfx namespace also makes more sense as that's where we find IntRect, IntSize and other.
The extent of the changes highlight how much similar data structures are duplicated across the code, to the point it's scary.
Differential Revision: https://phabricator.services.mozilla.com/D25347
--HG--
extra : moz-landing-system : lando
All compositors support 10/12 bits images now.
Additionally, add BT2020 support to AOM decoder.
Differential Revision: https://phabricator.services.mozilla.com/D25344
--HG--
extra : moz-landing-system : lando
mJavaDecoder is invalid after the decoder is shut down and should never
be used.
Differential Revision: https://phabricator.services.mozilla.com/D26438
--HG--
extra : moz-landing-system : lando
HandleOutput() runs on Android binder thread pool and could be preempted
by RemoteDateDecoder task queue. That means ProcessOutput() could be scheduled
after ProcessShutdown() or ProcessFlush(). When that happens, aBuffer is no
long valid and should never be processed, and aSample can be
recycled immediately.
Also assert preconditions of buffers received from Java callbacks.
Differential Revision: https://phabricator.services.mozilla.com/D26189
--HG--
extra : moz-landing-system : lando
To prevent new buffer object from being created per frame, either
Sample.CREATOR has to keep track of all buffers from every remote codec,
or the client must memorize seen buffers and avoid asking for them again
and again. The former saves client code from modifications but complicates
the implementation of Sample, a data structure class, while the latter
requires changes to client code but avoid overcomplicating Sample.CREATOR
implementation.
The 2nd approach is taken:
- move SampleBuffer out of Sample, and update clients accordingly
- add a new IPC method for clients to get the buffers only when needed
Differential Revision: https://phabricator.services.mozilla.com/D24590
--HG--
extra : moz-landing-system : lando
While not required in the two examples provided, should those streams change resolution and continue to use the same type of bytstreams we would miss the changes as the keyframe never contains the new SPS/PPS NALs.
So we add an option to handle this case, so we can separate the cases where this could be needed without regressing bug 1469257
Differential Revision: https://phabricator.services.mozilla.com/D24854
--HG--
extra : moz-landing-system : lando
We limited the search for a SPS/PPS change to keyframe only in bug 1469257.
However this causes issues if the first frame containing a SPS/PPS isn't a keyframe.
We also need to error on content with no SPS/PPS as to inform the caller that something is amiss. Such content was invalid to start with.
Differential Revision: https://phabricator.services.mozilla.com/D24853
--HG--
extra : moz-landing-system : lando
While not required in the two examples provided, should those streams change resolution and continue to use the same type of bytstreams we would miss the changes as the keyframe never contains the new SPS/PPS NALs.
So we add an option to handle this case, so we can separate the cases where this could be needed without regressing bug 1469257
Differential Revision: https://phabricator.services.mozilla.com/D24854
--HG--
extra : moz-landing-system : lando
We limited the search for a SPS/PPS change to keyframe only in bug 1469257.
However this causes issues if the first frame containing a SPS/PPS isn't a keyframe.
We also need to error on content with no SPS/PPS as to inform the caller that something is amiss. Such content was invalid to start with.
Differential Revision: https://phabricator.services.mozilla.com/D24853
--HG--
extra : moz-landing-system : lando
Function arguments may not evaluate in order, as such, this ensures width is read
before height from VP9 streams.
Differential Revision: https://phabricator.services.mozilla.com/D24372
--HG--
extra : moz-landing-system : lando
Despite its name and apparent use LocalAllocPolicy could only handle a single call to Alloc() at a time which was okay due to how it was used by the MediaFormatReader.
However, we do want to handle multiple calls to Alloc().
These changes allow to perform simultaneous requests and have them be processed one at a time serially.
Differential Revision: https://phabricator.services.mozilla.com/D23651
--HG--
extra : moz-landing-system : lando
Chrome has had it enabled for years, we had disabled it originally due to crashes seen on Windows 7.
Differential Revision: https://phabricator.services.mozilla.com/D23656
--HG--
extra : moz-landing-system : lando
clang-cl only acts on five MSVC warning flags: 7219c7e9af/clang/include/clang/Driver/CLCompatOptions.td (L188-L197)
With MSVC now unsupported, most -wdNNNN have no effect and can be removed.
This patch converts the five supported warnings to their clang spellings, as preparation for a subsequent patch that will remove all remaining `[/-]w[edo][0-9]{4}`.
Differential Revision: https://phabricator.services.mozilla.com/D22582
--HG--
extra : moz-landing-system : lando
The rate changes after decoding the first sample to what was indicated in the container. The code to handle that case was incorrectly removed in bug 1530234
Differential Revision: https://phabricator.services.mozilla.com/D21618
--HG--
extra : moz-landing-system : lando
The WMF audio decoder recalculated the timestamp of each audio sample according to the number of frames decoded so far.
This is incompatible with the trimming mechanism that rely on the timestamps of the audio to be matching what is found in the container.
All the other audio decoders do it that way already.
Depends on D20969
Differential Revision: https://phabricator.services.mozilla.com/D21305
--HG--
extra : moz-landing-system : lando
Counting CPUs accesses the filesystem (sysfs or procfs), which we'd like
to disallow when sandboxed if possible, and fails silently if access
is denied. Because the CPU count rarely changes, this patch handles
that problem for the RDD process by caching a copy before starting
sandboxing.
Tested with a local patch to have the sandbox file broker client crash
if accessing the sysfs node for the CPU count, to verify that it's not
accessed.
Depends on D14524
Differential Revision: https://phabricator.services.mozilla.com/D20895
--HG--
extra : moz-landing-system : lando
We didn't set the duration on the created IMF sample before sending it to the decoder.
For audio this isn't a problem as the duration is calculated from the number of frames returned.
For video however, the duration appears to have been calculated by WMF as the delta of pts. However, for the last frame, the value returned was set to a value not matching our input.
So we set the duration on the IMFSample so it doesn't have to make it up.
Setting the duration on the IMFSample isn't sufficient with Windows 7, which still continues to calculate it based on previous samples durations.
So we store the last sample duration and set it when draining.
Differential Revision: https://phabricator.services.mozilla.com/D20653
--HG--
extra : moz-landing-system : lando
A simplistic decoder wrapper that will take decoded frames and truncate them should the originally compressed frame contained trimming information.
Differential Revision: https://phabricator.services.mozilla.com/D20173
--HG--
extra : moz-landing-system : lando
It can be determined from the size of the buffer and the number of audio frames. Additionally, it ensures that the duration of the frame is always exactly what the AudioData contains.
Differential Revision: https://phabricator.services.mozilla.com/D20170
--HG--
extra : moz-landing-system : lando
This will allow to remove mFrames member and calculate from the size of the content, which will dynamically change depending on a cropping filter.
Differential Revision: https://phabricator.services.mozilla.com/D20165
--HG--
extra : moz-landing-system : lando
I suspect these may have been cargo-culted from elsewhere.
We only need -Wno-sign-compare, which we can add regardless of compiler because clang-cl understands it natively in addition to the -wdNNNN form.
Differential Revision: https://phabricator.services.mozilla.com/D20616
--HG--
extra : moz-landing-system : lando
- Use a single remote decoder IPDL spec and make a remote decoding
base class.
Renames PRemoteVideoDecoder.ipdl to PRemoteDecoder.ipdl
Renames RemoteVideoDecoder{Child|Parent}.{cpp|h} to
RemoteDecoder{Child|Parent}.{cpp|h}
- Move remote video decoding to new subclasses.
Creates RemoteVideoDecoder.{cpp|h} that contains both the parent
and child sides of the RemoteVideoDecoder{Child|Parent} classes.
- Create new remote audio decoder
Creates RemoteAudioDecoder.{cpp|h} that contains both the parent
and child sides of the RemoteAudioDecoder{Child|Parent} classes.
- Connect all the plumbing to use the new remote audio decoder to
decode Vorbis in RDD including a new pref to control whether
Vorbis is decoding in the AgnosticDecoderModule or the
RemoteDecoderModule/RDD (media.rdd-vorbis.enabled).
Depends on D18640
Differential Revision: https://phabricator.services.mozilla.com/D18641
--HG--
rename : dom/media/ipc/PRemoteVideoDecoder.ipdl => dom/media/ipc/PRemoteDecoder.ipdl
rename : dom/media/ipc/RemoteVideoDecoderChild.cpp => dom/media/ipc/RemoteDecoderChild.cpp
rename : dom/media/ipc/RemoteVideoDecoderChild.h => dom/media/ipc/RemoteDecoderChild.h
rename : dom/media/ipc/RemoteVideoDecoderParent.cpp => dom/media/ipc/RemoteDecoderParent.cpp
rename : dom/media/ipc/RemoteVideoDecoderParent.h => dom/media/ipc/RemoteDecoderParent.h
extra : moz-landing-system : lando
Consequently, this removes:
- MOZ_LIBPRIO, which is now always enabled.
- non_msvc_compiler, which is now always true.
- The cl.py wrapper, since it's not used anymore.
- CL_INCLUDES_PREFIX, which was only used for the cl.py wrapper.
- NONASCII, which was only there to ensure CL_INCLUDES_PREFIX still
worked in non-ASCII cases.
This however keeps a large part of detecting and configuring for MSVC,
because we still do need it for at least headers, libraries, and midl.
Depends on D19614
Differential Revision: https://phabricator.services.mozilla.com/D19615
--HG--
extra : moz-landing-system : lando
VideoInfo.mImageRect is typically only used for VP8/VP9 content, however the Windows decoder use a common code between H264 and VP8/VP9 decoding and use this data to determine the image size. So we end up with conflicting image size and image rect (which contains cropping data).
Differential Revision: https://phabricator.services.mozilla.com/D18976
--HG--
extra : moz-landing-system : lando
There's a strong cycle of references between SamplesWaitingForKey and
CDMProxy that does not get cleared unless keys actually come in. This
causes a leak. One way to avoid that leak is to clear the proxy
reference when the things holding the SamplesWaitingForKey is shut
down.
Differential Revision: https://phabricator.services.mozilla.com/D17960
--HG--
extra : moz-landing-system : lando
Processing MediaCodec output in Android binder threads while flushing
in task queue could cause race condition and leftover frames. Dispatch
the processing to task queue ensures all frames prior to flushing will
be cleared (by mDecodedData.Clear()) or ignored (by mInputInfos.Clear()).
Also consolidate all flushing operations in one task to avoid frame
insertion between emptying mDecodedData and mInputInfos.
Differential Revision: https://phabricator.services.mozilla.com/D15228
--HG--
extra : moz-landing-system : lando
Usually the threshold is reset internally in MediaDataDecoder subclasses
that support the hint in their Flush() implementations so the value
will start fresh after seeking completed. But sometimes when there are
consecutive seek requests, MediaFormatReader::DecoderData::Flush() could
return early because DecoderData::mFlushed stays true when there is no
sample demuxed yet, and the threshold will not be cleared. Also, in
MediaFormatReader::SetVideoDecodeThreshold() we decide not to set the
hint when the seek target is close to EOS by checking the existence of
the next keyframe, and that could fail when there are gaps between MSE
buffered ranges. To make sure the hint is never out of date, we should
clear it rather than leaving it untouched.
Differential Revision: https://phabricator.services.mozilla.com/D15227
--HG--
extra : moz-landing-system : lando
Explicitly store the crypto scheme being used on our crypto structs to let us
differentiate between cenc and cbcs data. In doing so remove mMode and replace
mValid with IsEncrypted() for the following reasons:
- Different modes within the existing schemes are not currently utilized by the
spec: the scheme implies mode. Having a mode and a scheme could lead to confusion
between the two. We can return mMode if ever needed by the spec --
possibly if the isProtected flag which we were tracking with mMode, is
ever changed to be more than a bool in the spec.
- mValid was typically used to check if these structs contained valid crypto
data or not. With only one scheme this was often shorthand for 'IsEncrypted',
but with multiple schemes what is considered valid data for one may not be for
another. Do away with this and just explicitly have an 'IsEncrypted'.
Differential Revision: https://phabricator.services.mozilla.com/D15874
--HG--
extra : moz-landing-system : lando
Explicitly store the crypto scheme being used on our crypto structs to let us
differentiate between cenc and cbcs data. In doing so remove mMode and replace
mValid with IsEncrypted() for the following reasons:
- Different modes within the existing schemes are not currently utilized by the
spec of implementation. Having a mode and a scheme could lead to confusion
between the two. We can return mMode if ever needed by the spec.
- mValid was typically used to check if these structs contained valid crypto
data or not. With only one scheme this was often shorthand for 'IsEncrypted',
but with multiple schemes what is considered valid data for one may not be for
another. Do away with this and just explicitly have an 'IsEncrypted'.
Differential Revision: https://phabricator.services.mozilla.com/D15874
--HG--
extra : moz-landing-system : lando
Processing MediaCodec output in Android binder threads while flushing
in task queue could cause race condition and leftover frames. Dispatch
the processing to task queue ensures all frames prior to flushing will
be cleared (by mDecodedData.Clear()) or ignored (by mInputInfos.Clear()).
Also consolidate all flushing operations in one task to avoid frame
insertion between emptying mDecodedData and mInputInfos.
Differential Revision: https://phabricator.services.mozilla.com/D15228
--HG--
extra : moz-landing-system : lando
Usually the threshold is reset internally in MediaDataDecoder subclasses
that support the hint in their Flush() implementations so the value
will start fresh after seeking completed. But sometimes when there are
consecutive seek requests, MediaFormatReader::DecoderData::Flush() could
return early because DecoderData::mFlushed stays true when there is no
sample demuxed yet, and the threshold will not be cleared. Also, in
MediaFormatReader::SetVideoDecodeThreshold() we decide not to set the
hint when the seek target is close to EOS by checking the existence of
the next keyframe, and that could fail when there are gaps between MSE
buffered ranges. To make sure the hint is never out of date, we should
clear it rather than leaving it untouched.
Differential Revision: https://phabricator.services.mozilla.com/D15227
--HG--
extra : moz-landing-system : lando
In libavcodec 58 and later, the old avcodec_decode_video2 is broken and only return the first visible frame found after a VP9 super-frame.
This resulted in some YouTube videos for about 10% of the frames to never be returned.
Only the new API properly behaves so we upgrade our code to use it.
Differential Revision: https://phabricator.services.mozilla.com/D14682
--HG--
extra : moz-landing-system : lando
Latest dav1d version supports to store the timing information in undecoded frame and restore it later from decoded picture. This can provide more accurate timing especially during drain. In additionto that, colorspace information is set according to the size of the image. Finally this patch addresses some style comments.
Differential Revision: https://phabricator.services.mozilla.com/D13428
--HG--
extra : moz-landing-system : lando
Latest dav1d version supports to store the timing information in undecoded frame and restore it later from decoded picture. This can provide more accurate timing especially during drain. In additionto that, colorspace information is set according to the size of the image. Finally this patch addresses some style comments.
Differential Revision: https://phabricator.services.mozilla.com/D13428
--HG--
extra : moz-landing-system : lando
Child processes cannot access textures allocated in the parent process,
which is needed by the compositor to render video elements efficiently.
Unfortunately, Android doesn't expose Sufrace buffers (sharable across
processes) in the SDK/NDK as other platforms, so we need to generate
extra texture/surface in the child process and update texture images
through the surface, which is passed to the parent process for the remote
texture to copy its contents into.
Differential Revision: https://phabricator.services.mozilla.com/D11939
--HG--
rename : mobile/android/geckoview/src/main/aidl/org/mozilla/gecko/gfx/ISurfaceAllocator.aidl => mobile/android/geckoview/src/main/aidl/org/mozilla/gecko/gfx/SyncConfig.aidl
extra : moz-landing-system : lando
Due to changes in where things end up in the unified builds.
Depends on D8482
Differential Revision: https://phabricator.services.mozilla.com/D8483
--HG--
extra : moz-landing-system : lando
We also set the TrackInfoSharedPtr to each decoded sample so that on decoder that can be recycled (android) the frames are displayed with the right size.
Depends on D11588
Differential Revision: https://phabricator.services.mozilla.com/D11589
--HG--
extra : moz-landing-system : lando
And use it to determine if a frame is a keyframe, its size and display size.
Differential Revision: https://phabricator.services.mozilla.com/D11588
--HG--
extra : moz-landing-system : lando
- also makes RemoteMediaDecoder generic so it can work with the remote decoders on the RDD process
- changes VideoDecoderChild to subclass IRemoteDecoderChild
Differential Revision: https://phabricator.services.mozilla.com/D8482
--HG--
rename : dom/media/ipc/RemoteVideoDecoder.cpp => dom/media/ipc/GpuDecoderModule.cpp
rename : dom/media/ipc/RemoteVideoDecoder.h => dom/media/ipc/GpuDecoderModule.h
extra : moz-landing-system : lando
Due to changes in where things end up in the unified builds.
Depends on D8482
Differential Revision: https://phabricator.services.mozilla.com/D8483
--HG--
extra : moz-landing-system : lando
- also makes RemoteMediaDecoder generic so it can work with the remote decoders on the RDD process
- changes VideoDecoderChild to subclass IRemoteDecoderChild
Differential Revision: https://phabricator.services.mozilla.com/D8482
--HG--
rename : dom/media/ipc/RemoteVideoDecoder.cpp => dom/media/ipc/GpuDecoderModule.cpp
rename : dom/media/ipc/RemoteVideoDecoder.h => dom/media/ipc/GpuDecoderModule.h
extra : moz-landing-system : lando
ConvertSampleToAnnexB takes the sample's out of band extradata to prepend it to the data. It was theorically possible that the first sample would contain the SPS/PPS from the previous format.
Depends on D11559
Differential Revision: https://phabricator.services.mozilla.com/D11560
--HG--
extra : moz-landing-system : lando
keyframe was tracked in the CodecChangeMonitor in order to determine when to prepend the SPS/PPS to a sample for decoder using AnnexB format.
The assumption was that it was needed only once per the lifetime of the decoder (and indeed this is true with the Windows and Android decoder). However this causes issue with the Widevine H264 decoder and it will error on some content if the first sample passed following a flush doesn't contain a SPS/PPS.
Interestingly, this issue isn't observed with Netflix content or other Widevine sample videos. Only Amazon PrimeVideo content so far.
We instead use the global MediaChangeMonitor keyframe status that knows when the decoder has been drained or flushed.
Differential Revision: https://phabricator.services.mozilla.com/D11556
--HG--
extra : moz-landing-system : lando
It only used to work as the H264Converter used to check that the conversion needed was != kNeedAVCC (the default being kNone)
Differential Revision: https://phabricator.services.mozilla.com/D11526
--HG--
extra : moz-landing-system : lando
Due to changes in where things end up in the unified builds.
Depends on D8482
Differential Revision: https://phabricator.services.mozilla.com/D8483
--HG--
extra : moz-landing-system : lando
- also makes RemoteMediaDecoder generic so it can work with the remote decoders on the RDD process
- changes VideoDecoderChild to subclass IRemoteDecoderChild
Differential Revision: https://phabricator.services.mozilla.com/D8482
--HG--
rename : dom/media/ipc/RemoteVideoDecoder.cpp => dom/media/ipc/GpuDecoderModule.cpp
rename : dom/media/ipc/RemoteVideoDecoder.h => dom/media/ipc/GpuDecoderModule.h
extra : moz-landing-system : lando
This ensures that the test always work whenever the sample contains an extradata that is different to the info specified in the VideoInfo.
Depends on D10908
Differential Revision: https://phabricator.services.mozilla.com/D10909
--HG--
extra : moz-landing-system : lando
For AVC1 content (where the SPS/PPS is provided out of band), we would use VideoInfo has found in the container's metadata which may not always be correct.
Likely the reason on why we had bug 1498788 (decoded sample size didn't match the video config's dimensions)
We also always set the MediaRawData::mTrackInfo member for all samples, not just the first one as it's the right thing to do.
Depends on D10907
Differential Revision: https://phabricator.services.mozilla.com/D10908
--HG--
extra : moz-landing-system : lando
So that we can later expand and use it for other codecs.
Differential Revision: https://phabricator.services.mozilla.com/D10907
--HG--
rename : dom/media/platforms/wrappers/H264Converter.cpp => dom/media/platforms/wrappers/MediaChangeMonitor.cpp
rename : dom/media/platforms/wrappers/H264Converter.h => dom/media/platforms/wrappers/MediaChangeMonitor.h
extra : moz-landing-system : lando
For channel mapping 0 (which is always mono or sterero), the mapping is identical to the SMPTE one, so can copy the mapping as-is.
Differential Revision: https://phabricator.services.mozilla.com/D11000
--HG--
extra : moz-landing-system : lando
OmxDataDecoder, OmxPromiseLayer and PureOmxPlatformLayer consist
circular reference by RefPtr, and no one sever the reference. As a
result their refcount never decrease to 0.
This commit sever it at PureOmxPlatformLayer::Shutdown() which is
called by OmxDataDecoder.
Differential Revision: https://phabricator.services.mozilla.com/D10028
--HG--
extra : moz-landing-system : lando
Why we would need this is unclear however. the destination texture should always be smaller than what comes out of the decoder.
Differential Revision: https://phabricator.services.mozilla.com/D8672
--HG--
extra : moz-landing-system : lando
The Windows' hardware decoder always return an image whose dimensions are multiple of 16 pixels. As such, the image coming out of the decoder is typically bigger than the wanted image.
The D3D11 documentation states that " If you try and copy outside the destination resource or specify a source box that is larger than the source resource, the behavior of CopySubresourceRegion is undefined."
We've always copied from a bigger texture into a smaller one without specifying clipping. It seems to have always worked but falls into the undefined behaviour category.
So to be extra safe, we clip the source so that it matches the dimension of the destination texture.
Depends on D8129
Differential Revision: https://phabricator.services.mozilla.com/D8203
--HG--
extra : moz-landing-system : lando
P2 changed the way the H264Converter would be calling the decoder. The assumption in the EMEDecryptor was pretty incorrect to start with.
Depends on D7865
Differential Revision: https://phabricator.services.mozilla.com/D7882
--HG--
extra : moz-landing-system : lando
When used with the LowLatency option, we guarantee that the stream will contain no B-frame, as such we can reduce the re-ordering queue to zero. The apple VT decoder already returns frames in decode order making this change trivial.
Depends on D7861
Differential Revision: https://phabricator.services.mozilla.com/D7862
--HG--
extra : moz-landing-system : lando
The H264Converter can be used on a thread that isn't a nsThread or a TaskQueue, so having the H264Converter dispatching tasks isn't going to work
So instead we run all the code on the decoder's taskqueue using promise chaining.
All internal methods are made to assert that they are running on the task queue accordingly
Depends on D7860
Differential Revision: https://phabricator.services.mozilla.com/D7861
--HG--
extra : moz-landing-system : lando
As we do not have an IMF nor D3D11 NV12 image, we always require a full copy of the data that will deinterleave the chroma channels.
Depends on D7316
Differential Revision: https://phabricator.services.mozilla.com/D7318
This allows more easily the creation of the MFT required to convert to a RGBA32 image when doing a readback.
Depends on D7295
Differential Revision: https://phabricator.services.mozilla.com/D7296
