Bug 1594446 - Clamp scale at 32k to avoid excessively large visible regions r=mattwoodrow

The size of the visible region, for either a painted layer or a webrender blob
image, is calculated from the building rects of the contained display items, in
local-space. This should be restricted to the display port, to prevent the
visible regions growing too large leading to excessive memory usage.

For items within large scale transforms, the local-space visible region should
be very small. However, as we do not allow fractional sizes, the size of the
visible region will be rounded up to at least 1. This means that when we convert
the region back to screen-space, we are multiplying the extremely large scale by
at least one, rather than by a much smaller fraction. This can result in
incredibly large visible regions, and was causing OOM crashes.

To avoid this, we clamp the maximum chosen scale for these layers/blob images to
32k. Layers affected by this problem should have a visible region with
dimensions of 1 or 2, so this limits the resulting screen-space size for
those to an acceptable value. Layers with visible regions sized greater than
that should not have scales anywhere near this large, so will not be affected.

Differential Revision: https://phabricator.services.mozilla.com/D55691

--HG--
extra : moz-landing-system : lando

layout/painting/FrameLayerBuilder.cpp

@@ -6133,6 +6133,14 @@ Size FrameLayerBuilder::ChooseScale(nsIFrame* aContainerFrame,
     scale = Size(1.0, 1.0);
   }
 
+  // Prevent the scale from getting too large, to avoid excessive memory
+  // allocation. Usually memory allocation is limited by the visible region,
+  // which should be restricted to the display port. But at very large scales
+  // the visible region itself can become excessive due to rounding errors.
+  // Clamping the scale here prevents that.
+  scale =
+      Size(std::min(scale.width, 32768.0f), std::min(scale.height, 32768.0f));
+
   return scale;
 }