When we create a video decoder with ffmpeg, we check if the underlying
decoder implementation is supplied by OpenH264. If so, and the
media.ffmpeg.allow-openh264 is set to false (the default), we refuse to
create the video decoder. This is a problem because the decoder module
for ffmpeg declared support for H264, so we did not try falling back to
our own OpenH264 plugin exposed via GMP.
This patch duplicates the logic check for ffmpeg + OpenH264 in
FFmpegDecoderModule::Supports to ensure we fallback properly.
Differential Revision: https://phabricator.services.mozilla.com/D223843
When running hevc test on the try server, the MFT on the try server has
MFVideoFormat_NV12 and MFVideoFormat_IYUV on its list. If we choose last
available type, it would be MFVideoFormat_IYUV, causing 10-bit hevc
failed to play on the try server.
As we know what format is usually used for video and audio, we should
use the preferred type first if we can't find the compatible type. If
none of them are found, then the last available type should be used as
a final fallback option.
Differential Revision: https://phabricator.services.mozilla.com/D223652
By using the new marcos, it would show error via logging or error for
the failure function, instead of a general ENSURE_TRUE check.
Differential Revision: https://phabricator.services.mozilla.com/D223444
`kVTCompressionPropertyKey_AllowFrameReordering` is set to
`kCFBooleanFalse` in `Init()`. No need to set it to `SetRealtime` again.
Differential Revision: https://phabricator.services.mozilla.com/D223366
It doesn't error out, but create a broken decoder, that only has 2 channels, and
then I don't know what's going to happen, but it isn't going to be correct.
Differential Revision: https://phabricator.services.mozilla.com/D221652
This patch enhances `DecoderAgent` by enabling it to reconfigure the
underlying decoder when processing *AAC/ADTS* data via
`MediaChangeMonitor`.
In cases where the `description` field is absent in the
`AudioDecoderConfig` for `mp4a.*` (*AAC*) codec, the data stream is
assumed to be in *ADTS* format. However, some websites provide raw AAC
data under these circumstances, leading to decoding failure since the
underlying decoder requires AAC-specific configuration.
This patch resolves the issue by allowing the underling decoder to be
reconfigured - recreated and reinitialized - through
`MediaChangeMonitor` whenever an AAV format change is detected. This
process mirros the handling of video format changes, such as when H264
data format changes between AnnexB and AVCC. With this change,
`DecoderAgent` simply cues the `PDMFactory` to create a
`MediaChangeMonitor` that wraps the underlying decoder, similar to the
approach used for video decoders, ensuring correct handling of both AAC
and ADTS formats without altering other components.
Depends on D220505
Differential Revision: https://phabricator.services.mozilla.com/D220506
This patch implements a `CodecChangeMonitor` specifically for *AAC*
format, which will later be integrated into the `MediaChangeMonitor`.
The `CheckForChange` function within `CodecChangeMonitor` is designed to
detect changes in the data format and signal `MediaChangeMonitor` to
re-create and re-initialize the underlying decoder accordingly.
When the data stream format changes from *ADTS* to *AAC*, the
`CodecChangeMonitor` will add an AAC-specific configuration that is
missing in *ADTS* for proper decoder configuration. Conversely, when the
format switches from *AAC* to *ADTS*, it will reconfigure the decoder
without the AAC-specific configuration.
Depends on D219147
Differential Revision: https://phabricator.services.mozilla.com/D220505
On Android, SurfaceTextures provide a transform that should be applied
to texture coordinates when sampling from the texture. Usually this is
simply a y-flip, but sometimes it includes a scale and slight
translation, eg when the video frame is contained within a larger
texture. Previously we ignored this transform but performed a y-flip,
meaning we rendered correctly most of the time, but not all of the
time.
Our first attempt to fix this was in bug 1731980. When rendering as a
compositor surface with RenderCompositorOGLSWGL, we supplied the
transform to CompositorOGL's shaders, which correctly fixed the bug
for this rendering path.
However, the attempted fix for hardware webrender in fact made things
worse. As UV coordinates are supplied to webrender unnormalized, then
the shaders normalize them by dividing by the actual texture size,
this effectively handled the scale component of the transform. (Though
not quite scaling by the correct amount, and ignoring the translation
component, sometimes resulting in a pixel-wide green seam being
visible at the video's edges.) When we additionally applied the
transformation to the coordinates, it resulted in the scale being
applied twice, and the video being rendered too far zoomed
in.
To make matters worse, when we received subsequent bug reports of
incorrect rendering on various devices we mistakenly assumed that the
devices must be buggy, rather than our code being incorrect. We
therefore reverted to ignoring the transform on these devices, thereby
breaking the software webrender path again.
Additionally, on devices without GL_OES_EGL_image_external_essl3
support, we must sample from the SurfaceTexture using an ESSL1
shader. This means we do not have access to the correct texture size,
meaning we cannot correctly normalize the UV coordinates. This results
in the video being rendered too far zoomed out. And in the
non-compositor-surface software webrender path, we were accidentally
downscaling the texture when reading back into a CPU buffer, resulting
in the video being rendered at the correct zoom, but being very
blurry.
This patch aims to handle the transform correctly, in all rendering
paths, hopefully once and for all.
For hardware webrender, we now supply the texture coordinates to
webrender already normalized, using the functionality added in the
previous patch. This avoids the shaders scaling the coordinates again,
or using an incorrect texture size to do so.
For RenderCompositorOGLSWGL, we continue to apply the transform using
CompositorOGL's shaders.
In the non-compositor-surface software webrender path, we make
GLReadPixelsHelper apply the transform when reading from the
SurfaceTexture in to the CPU buffer. Again using functionality added
earlier in this patch series. This avoids downscaling the image. We
can then provide the default untransformed and unnormalized UVs to
webrender. As a result we can now remove the virtual function
RenderTextureHost::GetUvCoords(), added in bug 1731980, as it no
longer serves any purpose: we no longer want to share the
implementation between RenderAndroidSurfaceTextureHost::Lock and
RenderTextureHostSWGL::LockSWGL.
Finally, we remove all transform overrides on the devices we
mistakenly assumed were buggy.
Differential Revision: https://phabricator.services.mozilla.com/D220582
On Android, SurfaceTextures provide a transform that should be applied
to texture coordinates when sampling from the texture. Usually this is
simply a y-flip, but sometimes it includes a scale and slight
translation, eg when the video frame is contained within a larger
texture. Previously we ignored this transform but performed a y-flip,
meaning we rendered correctly most of the time, but not all of the
time.
Our first attempt to fix this was in bug 1731980. When rendering as a
compositor surface with RenderCompositorOGLSWGL, we supplied the
transform to CompositorOGL's shaders, which correctly fixed the bug
for this rendering path.
However, the attempted fix for hardware webrender in fact made things
worse. As UV coordinates are supplied to webrender unnormalized, then
the shaders normalize them by dividing by the actual texture size,
this effectively handled the scale component of the transform. (Though
not quite scaling by the correct amount, and ignoring the translation
component, sometimes resulting in a pixel-wide green seam being
visible at the video's edges.) When we additionally applied the
transformation to the coordinates, it resulted in the scale being
applied twice, and the video being rendered too far zoomed
in.
To make matters worse, when we received subsequent bug reports of
incorrect rendering on various devices we mistakenly assumed that the
devices must be buggy, rather than our code being incorrect. We
therefore reverted to ignoring the transform on these devices, thereby
breaking the software webrender path again.
Additionally, on devices without GL_OES_EGL_image_external_essl3
support, we must sample from the SurfaceTexture using an ESSL1
shader. This means we do not have access to the correct texture size,
meaning we cannot correctly normalize the UV coordinates. This results
in the video being rendered too far zoomed out. And in the
non-compositor-surface software webrender path, we were accidentally
downscaling the texture when reading back into a CPU buffer, resulting
in the video being rendered at the correct zoom, but being very
blurry.
This patch aims to handle the transform correctly, in all rendering
paths, hopefully once and for all.
For hardware webrender, we now supply the texture coordinates to
webrender already normalized, using the functionality added in the
previous patch. This avoids the shaders scaling the coordinates again,
or using an incorrect texture size to do so.
For RenderCompositorOGLSWGL, we continue to apply the transform using
CompositorOGL's shaders.
In the non-compositor-surface software webrender path, we make
GLReadPixelsHelper apply the transform when reading from the
SurfaceTexture in to the CPU buffer. Again using functionality added
earlier in this patch series. This avoids downscaling the image. We
can then provide the default untransformed and unnormalized UVs to
webrender. As a result we can now remove the virtual function
RenderTextureHost::GetUvCoords(), added in bug 1731980, as it no
longer serves any purpose: we no longer want to share the
implementation between RenderAndroidSurfaceTextureHost::Lock and
RenderTextureHostSWGL::LockSWGL.
Finally, we remove all transform overrides on the devices we
mistakenly assumed were buggy.
Differential Revision: https://phabricator.services.mozilla.com/D220582
Previously, `AudioTrimmer` and `MediaChangeMonitor` relied on the same
signal for their creation since `AudioTrimmer` was used exclusively for
audio and `MediaChangeMonitor` for video. However, changes introduced in
the patches for Bug 1891082 will enable MediaChangeMonitor to work with
audio as well. To allow for the possibility of wrapping an
`AudioTrimmer` within a `MediaChangeMonitor`, these components now
require separate signals for their creation.
This patch replaces the `NoWrapper` class, which was previosuly used to
signal the creation of both `AudioTrimmer` and `MediaChangeMonitor`,
with a new `Wrapper` class that includes two bit flags:
`Wrapper::AudioTrimmer` and `Wrapper::MediaChangeMonitor`. This allows
the creation of `AudioTrimmer` and `MediaChangeMonitor` to be controlled
independently, or together, by using `Wrapper::AudioTrimmer`,
`Wrapper::MediaChangeMonitor`, or `Wrapper::AudioTrimmer |
Wrapper::MediaChangeMonitor` respectively.
Depends on D220965
Differential Revision: https://phabricator.services.mozilla.com/D220753
This patch introduces a test that demonstrates a crash occurring when
`DecoderAgent::Shutdown` is called without an active `MediaDataDecoder`.
The issue arises when `PDMFactory::CreateDecoder` in
`DecoderAgent::Configure`, rejects decoder creation, leaving
`DecoderAgent::mDecoder` as `null`. Subsequently, when
`DecoderAgent::Shutdown` is invoked, the assertion that checks for the
existence of `mDecoder` fails, leading to a crash.
To replicate this scenario, the test includes a pref that forces the use
of a `NullDecoderModule`, and ensures that this module rejects decoder
creation.
Differential Revision: https://phabricator.services.mozilla.com/D219709
With AMD GPUs, crash happened when ID3D11Query was used for video rendering to remote canvas. Then the ID3D11Query usage needs to be limited only for video overlay with AMD GPUs. To do it, video rendering optimization to remove canvas is disabled when the ID3D11Query usage is detected with AMD GPUs.
GpuProcessQueryId.mOnlyForOverlay is added to deliver the information.
Differential Revision: https://phabricator.services.mozilla.com/D218390
