On Android, SurfaceTextures provide a transform that should be applied
to texture coordinates when sampling from the texture. Usually this is
simply a y-flip, but sometimes it includes a scale and slight
translation, eg when the video frame is contained within a larger
texture. Previously we ignored this transform but performed a y-flip,
meaning we rendered correctly most of the time, but not all of the
time.
Our first attempt to fix this was in bug 1731980. When rendering as a
compositor surface with RenderCompositorOGLSWGL, we supplied the
transform to CompositorOGL's shaders, which correctly fixed the bug
for this rendering path.
However, the attempted fix for hardware webrender in fact made things
worse. As UV coordinates are supplied to webrender unnormalized, then
the shaders normalize them by dividing by the actual texture size,
this effectively handled the scale component of the transform. (Though
not quite scaling by the correct amount, and ignoring the translation
component, sometimes resulting in a pixel-wide green seam being
visible at the video's edges.) When we additionally applied the
transformation to the coordinates, it resulted in the scale being
applied twice, and the video being rendered too far zoomed
in.
To make matters worse, when we received subsequent bug reports of
incorrect rendering on various devices we mistakenly assumed that the
devices must be buggy, rather than our code being incorrect. We
therefore reverted to ignoring the transform on these devices, thereby
breaking the software webrender path again.
Additionally, on devices without GL_OES_EGL_image_external_essl3
support, we must sample from the SurfaceTexture using an ESSL1
shader. This means we do not have access to the correct texture size,
meaning we cannot correctly normalize the UV coordinates. This results
in the video being rendered too far zoomed out. And in the
non-compositor-surface software webrender path, we were accidentally
downscaling the texture when reading back into a CPU buffer, resulting
in the video being rendered at the correct zoom, but being very
blurry.
This patch aims to handle the transform correctly, in all rendering
paths, hopefully once and for all.
For hardware webrender, we now supply the texture coordinates to
webrender already normalized, using the functionality added in the
previous patch. This avoids the shaders scaling the coordinates again,
or using an incorrect texture size to do so.
For RenderCompositorOGLSWGL, we continue to apply the transform using
CompositorOGL's shaders.
In the non-compositor-surface software webrender path, we make
GLReadPixelsHelper apply the transform when reading from the
SurfaceTexture in to the CPU buffer. Again using functionality added
earlier in this patch series. This avoids downscaling the image. We
can then provide the default untransformed and unnormalized UVs to
webrender. As a result we can now remove the virtual function
RenderTextureHost::GetUvCoords(), added in bug 1731980, as it no
longer serves any purpose: we no longer want to share the
implementation between RenderAndroidSurfaceTextureHost::Lock and
RenderTextureHostSWGL::LockSWGL.
Finally, we remove all transform overrides on the devices we
mistakenly assumed were buggy.
Differential Revision: https://phabricator.services.mozilla.com/D220582
Some external images must be sampled from by providing normalized UV
coordinates to webrender, but currently webrender only supports
unnormalized UVs.
This patch adds a flag to webrender's external image API that
specifies whether the UV coordinates supplied when the texture is
locked are normalized or unnormalized. This flag is plumbed through
webrender to the required locations. We then add support for taking
normalized UVs as inputs to the brush_image and cs_scale shaders. The
only other shader that can be used with external textures is the
composite shader, which already supports normalized UVs.
This does not change any behaviour, that will happen in the next patch
in this series.
Differential Revision: https://phabricator.services.mozilla.com/D220581
On Android, SurfaceTextures provide a transform that should be applied
to texture coordinates when sampling from the texture. Usually this is
simply a y-flip, but sometimes it includes a scale and slight
translation, eg when the video frame is contained within a larger
texture. Previously we ignored this transform but performed a y-flip,
meaning we rendered correctly most of the time, but not all of the
time.
Our first attempt to fix this was in bug 1731980. When rendering as a
compositor surface with RenderCompositorOGLSWGL, we supplied the
transform to CompositorOGL's shaders, which correctly fixed the bug
for this rendering path.
However, the attempted fix for hardware webrender in fact made things
worse. As UV coordinates are supplied to webrender unnormalized, then
the shaders normalize them by dividing by the actual texture size,
this effectively handled the scale component of the transform. (Though
not quite scaling by the correct amount, and ignoring the translation
component, sometimes resulting in a pixel-wide green seam being
visible at the video's edges.) When we additionally applied the
transformation to the coordinates, it resulted in the scale being
applied twice, and the video being rendered too far zoomed
in.
To make matters worse, when we received subsequent bug reports of
incorrect rendering on various devices we mistakenly assumed that the
devices must be buggy, rather than our code being incorrect. We
therefore reverted to ignoring the transform on these devices, thereby
breaking the software webrender path again.
Additionally, on devices without GL_OES_EGL_image_external_essl3
support, we must sample from the SurfaceTexture using an ESSL1
shader. This means we do not have access to the correct texture size,
meaning we cannot correctly normalize the UV coordinates. This results
in the video being rendered too far zoomed out. And in the
non-compositor-surface software webrender path, we were accidentally
downscaling the texture when reading back into a CPU buffer, resulting
in the video being rendered at the correct zoom, but being very
blurry.
This patch aims to handle the transform correctly, in all rendering
paths, hopefully once and for all.
For hardware webrender, we now supply the texture coordinates to
webrender already normalized, using the functionality added in the
previous patch. This avoids the shaders scaling the coordinates again,
or using an incorrect texture size to do so.
For RenderCompositorOGLSWGL, we continue to apply the transform using
CompositorOGL's shaders.
In the non-compositor-surface software webrender path, we make
GLReadPixelsHelper apply the transform when reading from the
SurfaceTexture in to the CPU buffer. Again using functionality added
earlier in this patch series. This avoids downscaling the image. We
can then provide the default untransformed and unnormalized UVs to
webrender. As a result we can now remove the virtual function
RenderTextureHost::GetUvCoords(), added in bug 1731980, as it no
longer serves any purpose: we no longer want to share the
implementation between RenderAndroidSurfaceTextureHost::Lock and
RenderTextureHostSWGL::LockSWGL.
Finally, we remove all transform overrides on the devices we
mistakenly assumed were buggy.
Differential Revision: https://phabricator.services.mozilla.com/D220582
Some external images must be sampled from by providing normalized UV
coordinates to webrender, but currently webrender only supports
unnormalized UVs.
This patch adds a flag to webrender's external image API that
specifies whether the UV coordinates supplied when the texture is
locked are normalized or unnormalized. This flag is plumbed through
webrender to the required locations. We then add support for taking
normalized UVs as inputs to the brush_image and cs_scale shaders. The
only other shader that can be used with external textures is the
composite shader, which already supports normalized UVs.
This does not change any behaviour, that will happen in the next patch
in this series.
Differential Revision: https://phabricator.services.mozilla.com/D220581
Though I personally think this fix is subtle, I have no other good idea to
solve the issue. Once after we have a separate API for dynamic toolbar, we
could handle this bug case in a clearer way.
This change doesn't have a dedicated test since there have been two JUnit tests
hitting this bug conditions.
(If we had been running tests on browsers which support both dynamic toolbar
and pull-to-refresh, this bug might have been noticed earlier?)
Differential Revision: https://phabricator.services.mozilla.com/D221105
What goes on here is that there's a couple of unfortunate style change
sequences which end up making us not do the EffectsInfo dance correctly.
Twitter uses (maybe didn't use to, which would explain the regression) a
visibility: hidden, out-of-flow iframe for a bit (which we correctly
throttle). But then they switch to an in-flow, visible, zero-height
iframe. That we should _not_ throttle. However, we end up not getting to
the display list code at all, because nsBlockFrame decides that we don't
need to descend into an empty line[1].
It seems less error prone to re-use the IntersectionObserver timing and
computation to determine whether the iframe is visible. That completely
matches in-process iframes, too.
Removing the empty frame border and putting them on an empty line in
dom/base/test/test_bug1639328.html is enough to reproduce the issue
without this patch.
[1]: https://searchfox.org/mozilla-central/rev/fe2743c6c5c708061c7f6504b26958fcc815bb4a/layout/generic/nsBlockFrame.cpp#7569-7579
Differential Revision: https://phabricator.services.mozilla.com/D207479
What goes on here is that there's a couple of unfortunate style change
sequences which end up making us not do the EffectsInfo dance correctly.
Twitter uses (maybe didn't use to, which would explain the regression) a
visibility: hidden, out-of-flow iframe for a bit (which we correctly
throttle). But then they switch to an in-flow, visible, zero-height
iframe. That we should _not_ throttle. However, we end up not getting to
the display list code at all, because nsBlockFrame decides that we don't
need to descend into an empty line[1].
It seems less error prone to re-use the IntersectionObserver timing and
computation to determine whether the iframe is visible. That completely
matches in-process iframes, too.
Removing the empty frame border and putting them on an empty line in
dom/base/test/test_bug1639328.html is enough to reproduce the issue
without this patch.
[1]: https://searchfox.org/mozilla-central/rev/fe2743c6c5c708061c7f6504b26958fcc815bb4a/layout/generic/nsBlockFrame.cpp#7569-7579
Differential Revision: https://phabricator.services.mozilla.com/D207479
Besides porting helper_fission_inactivescroller_under_oopif, this adds
helper_fission_utils.js::hitTestOOPIF() which will replace
helper_fission_utils.js::fissionHitTest(). The latter will be removed
with the completion of Bug 1841896.
Differential Revision: https://phabricator.services.mozilla.com/D217165
For long touch scrolls that change direction, using the start point of
the touch scroll as the origin to calculate the angle of the scroll is
problematic for determine the axis that should be locked. Maintain a
list of recent touch gesture points and calculate the gesture angle from
the most recent touch gesture points.
Differential Revision: https://phabricator.services.mozilla.com/D219228
Add a minimum size parameter to the RecentEventsBuffer, for the scenario
in which we would prefer data from stale events to that of the fallback.
Differential Revision: https://phabricator.services.mozilla.com/D220141
During fast-fling, any event isn't delivered to contents, but fast-fling
causes scrolling, thus we need to tell the state to GeckoView via
the APZHandledResult.
The JUnit test case causes an assertion (bug 1852854) without this change.
Differential Revision: https://phabricator.services.mozilla.com/D218898
With NVIDIA GPU, original ID3D11Texture2D created by compositor device and ID3D11Texture2D opened from shared handle compositor device seemed to work differently. The original ID3D11Texture2D seemed to work well with STR of Bug 1911237 with NVIDIA GPU.
Then if ID3D11Texture2D is created by compositor device, it seems better to use GpuProcessTextureId instead of shared handle.
Differential Revision: https://phabricator.services.mozilla.com/D218903
When ID3D11VideoProcessor is used, the color conversion becomes simpler than using DXVA2Manager. And the color conversion becomes same to video overlay. ID3D11VideoProcessor is wrapped by VideoProcessorD3D11. VideoProcessorD3D11 is easier to use than ID3D11VideoProcessor.
And by keeping VideoProcessorD3D11 alive, the flickering of STR of Bug 1911937 was reduced.
Differential Revision: https://phabricator.services.mozilla.com/D218713
There are many other uses of OtherPid which could be switched over to
OtherChildID, but these were a couple of obvious low-hanging fruit use-cases
which will work better when using OtherChildID.
Differential Revision: https://phabricator.services.mozilla.com/D217120
This requires quite a bit of piping to get the ChildID passed everywhere where
we currently pass the pid in IPC. This is done by adding a new struct type
(EndpointProcInfo), which is passed around instead of OtherPid in these places,
and contains the full pid.
In most cases, it was a fairly painless change to move over, however in some
cases, more complex changes were required, as the pid was being stored
previously in something like an Atomic<...>, and needed to be switched to using
a mutex-protected value.
In the future, it may be possible to remove OtherPid from IPDL actors once
everything is migrated to ChildID, but we're still a long way off from that, so
for now we unfortunately need to pass both around.
Differential Revision: https://phabricator.services.mozilla.com/D217118
willReadFrequently already implies this; however that can't
be enabled on Windows. We can use our own attribute to control
the value as it plumbs into the depths of Canvas2D and WebGL
Differential Revision: https://phabricator.services.mozilla.com/D212178
So that even if the chrome window gets resized by the software keyboard, the
contents can be laid out as if there's no software keyboard for overlays-content
and resizes-visual modes.
Also use the exapanded size (we call it "layout display size") for all call sites of
MobileViewportManager::DisplaySize().
Differential Revision: https://phabricator.services.mozilla.com/D204167
YUV422P10 is needed to properly tag decoded YUV422P10 video. NV16 is
needed to describe the macOS 10-bit YUV422 formats.
Differential Revision: https://phabricator.services.mozilla.com/D217334
This is just a renaming patch, motivated by a need for clarity before
addding more YUV formats. Comments and log messages are updated as well,
and one function in MacIOSurface is renamed.
There is one small change in TextureHost.cpp to satisfy clang-tidy.
Differential Revision: https://phabricator.services.mozilla.com/D217883
