gecko-dev/gfx/webrender_bindings/RenderTextureHostSWGL.cpp
Jamie Nicol 0c43a18e35 Bug 1913568 - Handle SurfaceTexture transforms in webrender, for reals this time. r=gfx-reviewers,media-playback-reviewers,padenot,nical
On Android, SurfaceTextures provide a transform that should be applied
to texture coordinates when sampling from the texture. Usually this is
simply a y-flip, but sometimes it includes a scale and slight
translation, eg when the video frame is contained within a larger
texture. Previously we ignored this transform but performed a y-flip,
meaning we rendered correctly most of the time, but not all of the
time.

Our first attempt to fix this was in bug 1731980. When rendering as a
compositor surface with RenderCompositorOGLSWGL, we supplied the
transform to CompositorOGL's shaders, which correctly fixed the bug
for this rendering path.

However, the attempted fix for hardware webrender in fact made things
worse. As UV coordinates are supplied to webrender unnormalized, then
the shaders normalize them by dividing by the actual texture size,
this effectively handled the scale component of the transform. (Though
not quite scaling by the correct amount, and ignoring the translation
component, sometimes resulting in a pixel-wide green seam being
visible at the video's edges.) When we additionally applied the
transformation to the coordinates, it resulted in the scale being
applied twice, and the video being rendered too far zoomed
in.

To make matters worse, when we received subsequent bug reports of
incorrect rendering on various devices we mistakenly assumed that the
devices must be buggy, rather than our code being incorrect. We
therefore reverted to ignoring the transform on these devices, thereby
breaking the software webrender path again.

Additionally, on devices without GL_OES_EGL_image_external_essl3
support, we must sample from the SurfaceTexture using an ESSL1
shader. This means we do not have access to the correct texture size,
meaning we cannot correctly normalize the UV coordinates. This results
in the video being rendered too far zoomed out. And in the
non-compositor-surface software webrender path, we were accidentally
downscaling the texture when reading back into a CPU buffer, resulting
in the video being rendered at the correct zoom, but being very
blurry.

This patch aims to handle the transform correctly, in all rendering
paths, hopefully once and for all.

For hardware webrender, we now supply the texture coordinates to
webrender already normalized, using the functionality added in the
previous patch. This avoids the shaders scaling the coordinates again,
or using an incorrect texture size to do so.

For RenderCompositorOGLSWGL, we continue to apply the transform using
CompositorOGL's shaders.

In the non-compositor-surface software webrender path, we make
GLReadPixelsHelper apply the transform when reading from the
SurfaceTexture in to the CPU buffer. Again using functionality added
earlier in this patch series. This avoids downscaling the image. We
can then provide the default untransformed and unnormalized UVs to
webrender. As a result we can now remove the virtual function
RenderTextureHost::GetUvCoords(), added in bug 1731980, as it no
longer serves any purpose: we no longer want to share the
implementation between RenderAndroidSurfaceTextureHost::Lock and
RenderTextureHostSWGL::LockSWGL.

Finally, we remove all transform overrides on the devices we
mistakenly assumed were buggy.

Differential Revision: https://phabricator.services.mozilla.com/D220582
2024-09-09 14:06:26 +00:00

227 lines
7.1 KiB
C++

/* -*- Mode: C++; tab-width: 8; indent-tabs-mode: nil; c-basic-offset: 2 -*- */
/* vim: set ts=8 sts=2 et sw=2 tw=80: */
/* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/. */
#include "RenderTextureHostSWGL.h"
#include "mozilla/gfx/Logging.h"
#include "mozilla/layers/TextureHost.h"
#include "RenderThread.h"
namespace mozilla {
namespace wr {
bool RenderTextureHostSWGL::UpdatePlanes(RenderCompositor* aCompositor) {
wr_swgl_make_current(mContext);
size_t planeCount = GetPlaneCount();
bool texInit = false;
if (mPlanes.size() < planeCount) {
mPlanes.reserve(planeCount);
while (mPlanes.size() < planeCount) {
mPlanes.push_back(PlaneInfo(wr_swgl_gen_texture(mContext)));
}
texInit = true;
}
gfx::SurfaceFormat format = GetFormat();
gfx::ColorDepth colorDepth = GetColorDepth();
for (size_t i = 0; i < planeCount; i++) {
PlaneInfo& plane = mPlanes[i];
if (!MapPlane(aCompositor, i, plane)) {
if (i > 0) {
UnmapPlanes();
}
return false;
}
GLenum internalFormat = 0;
switch (format) {
case gfx::SurfaceFormat::B8G8R8A8:
case gfx::SurfaceFormat::B8G8R8X8:
MOZ_ASSERT(colorDepth == gfx::ColorDepth::COLOR_8);
internalFormat = LOCAL_GL_RGBA8;
break;
case gfx::SurfaceFormat::YUV420:
switch (colorDepth) {
case gfx::ColorDepth::COLOR_8:
internalFormat = LOCAL_GL_R8;
break;
case gfx::ColorDepth::COLOR_10:
case gfx::ColorDepth::COLOR_12:
case gfx::ColorDepth::COLOR_16:
internalFormat = LOCAL_GL_R16;
break;
}
break;
case gfx::SurfaceFormat::NV12:
switch (colorDepth) {
case gfx::ColorDepth::COLOR_8:
internalFormat = i > 0 ? LOCAL_GL_RG8 : LOCAL_GL_R8;
break;
case gfx::ColorDepth::COLOR_10:
case gfx::ColorDepth::COLOR_12:
case gfx::ColorDepth::COLOR_16:
internalFormat = i > 0 ? LOCAL_GL_RG16 : LOCAL_GL_R16;
break;
}
break;
case gfx::SurfaceFormat::P010:
MOZ_ASSERT(colorDepth == gfx::ColorDepth::COLOR_10);
internalFormat = i > 0 ? LOCAL_GL_RG16 : LOCAL_GL_R16;
break;
case gfx::SurfaceFormat::YUY2:
MOZ_ASSERT(colorDepth == gfx::ColorDepth::COLOR_8);
internalFormat = LOCAL_GL_RGB_RAW_422_APPLE;
break;
default:
MOZ_RELEASE_ASSERT(false, "Unhandled external image format");
break;
}
wr_swgl_set_texture_buffer(mContext, plane.mTexture, internalFormat,
plane.mSize.width, plane.mSize.height,
plane.mStride, plane.mData, 0, 0);
}
if (texInit) {
// Initialize the mip filters to linear by default.
for (const auto& plane : mPlanes) {
wr_swgl_set_texture_parameter(mContext, plane.mTexture,
LOCAL_GL_TEXTURE_MIN_FILTER,
LOCAL_GL_LINEAR);
wr_swgl_set_texture_parameter(mContext, plane.mTexture,
LOCAL_GL_TEXTURE_MAG_FILTER,
LOCAL_GL_LINEAR);
}
}
return true;
}
bool RenderTextureHostSWGL::SetContext(void* aContext) {
if (mContext != aContext) {
CleanupPlanes();
mContext = aContext;
wr_swgl_reference_context(mContext);
}
return mContext != nullptr;
}
wr::WrExternalImage RenderTextureHostSWGL::LockSWGL(
uint8_t aChannelIndex, void* aContext, RenderCompositor* aCompositor) {
if (!SetContext(aContext)) {
return InvalidToWrExternalImage();
}
if (!mLocked) {
if (!UpdatePlanes(aCompositor)) {
return InvalidToWrExternalImage();
}
mLocked = true;
}
if (aChannelIndex >= mPlanes.size()) {
return InvalidToWrExternalImage();
}
const PlaneInfo& plane = mPlanes[aChannelIndex];
// Prefer native textures, unless our backend forbids it.
layers::TextureHost::NativeTexturePolicy policy =
layers::TextureHost::BackendNativeTexturePolicy(
layers::WebRenderBackend::SOFTWARE, plane.mSize);
return policy == layers::TextureHost::NativeTexturePolicy::FORBID
? RawDataToWrExternalImage((uint8_t*)plane.mData,
plane.mStride * plane.mSize.height)
: NativeTextureToWrExternalImage(
plane.mTexture, 0.0, 0.0,
static_cast<float>(plane.mSize.width),
static_cast<float>(plane.mSize.height));
}
void RenderTextureHostSWGL::UnlockSWGL() {
if (mLocked) {
mLocked = false;
UnmapPlanes();
}
}
void RenderTextureHostSWGL::CleanupPlanes() {
if (!mContext) {
return;
}
if (!mPlanes.empty()) {
wr_swgl_make_current(mContext);
for (const auto& plane : mPlanes) {
wr_swgl_delete_texture(mContext, plane.mTexture);
}
mPlanes.clear();
}
wr_swgl_destroy_context(mContext);
mContext = nullptr;
}
RenderTextureHostSWGL::~RenderTextureHostSWGL() { CleanupPlanes(); }
bool RenderTextureHostSWGL::LockSWGLCompositeSurface(
void* aContext, wr::SWGLCompositeSurfaceInfo* aInfo) {
if (!SetContext(aContext)) {
return false;
}
if (!mLocked) {
if (!UpdatePlanes(nullptr)) {
return false;
}
mLocked = true;
}
MOZ_ASSERT(mPlanes.size() <= 3);
for (size_t i = 0; i < mPlanes.size(); i++) {
aInfo->textures[i] = mPlanes[i].mTexture;
}
switch (GetFormat()) {
case gfx::SurfaceFormat::YUV420:
case gfx::SurfaceFormat::NV12:
case gfx::SurfaceFormat::P010:
case gfx::SurfaceFormat::YUY2: {
aInfo->yuv_planes = mPlanes.size();
auto colorSpace = GetYUVColorSpace();
aInfo->color_space = ToWrYuvRangedColorSpace(colorSpace);
auto colorDepth = GetColorDepth();
aInfo->color_depth = ToWrColorDepth(colorDepth);
break;
}
case gfx::SurfaceFormat::B8G8R8A8:
case gfx::SurfaceFormat::B8G8R8X8:
break;
default:
gfxCriticalNote << "Unhandled external image format: " << GetFormat();
MOZ_RELEASE_ASSERT(false, "Unhandled external image format");
break;
}
aInfo->size.width = mPlanes[0].mSize.width;
aInfo->size.height = mPlanes[0].mSize.height;
return true;
}
bool wr_swgl_lock_composite_surface(void* aContext, wr::ExternalImageId aId,
wr::SWGLCompositeSurfaceInfo* aInfo) {
RenderTextureHost* texture = RenderThread::Get()->GetRenderTexture(aId);
if (!texture) {
return false;
}
RenderTextureHostSWGL* swglTex = texture->AsRenderTextureHostSWGL();
if (!swglTex) {
return false;
}
return swglTex->LockSWGLCompositeSurface(aContext, aInfo);
}
void wr_swgl_unlock_composite_surface(void* aContext, wr::ExternalImageId aId) {
RenderTextureHost* texture = RenderThread::Get()->GetRenderTexture(aId);
if (!texture) {
return;
}
RenderTextureHostSWGL* swglTex = texture->AsRenderTextureHostSWGL();
if (!swglTex) {
return;
}
swglTex->UnlockSWGL();
}
} // namespace wr
} // namespace mozilla