gfx_thumbnail: preview multi-gigabyte videos from a fixed footprint

A 7.28 GB 4K H.264 MP4 did not preview in the file browser while a
1.3 GB one did.  Nothing about the file is exotic - its structural
twin (4K59.94 High profile, AAC, trailing moov) and a 4.3 GB co64
variant both decode fine handed to the decoders directly.  The
failure is the loading architecture, twice over:

1. The browser preview is still-first: request_file pushes
   task_push_image_load, which nbio-reads the ENTIRE file into
   memory, and the windowed animation only ever adopts the stream
   that still decode leaves behind.  A 7.3 GB allocation fails, so
   neither still nor animation happens.
2. The windowed path would not have saved it either: growing the
   available prefix toward a trailing moov went through
   data_transfer_window_extend, an eager sequential read - the open
   committed and read the whole file through the window.  1.3 GB
   survives that transient commit, which is exactly why that file
   worked; 7.28 GB does not.

Both are fixed:

- Video containers (MP4/WEBM) in the file browser now open through
  the windowed animation path directly; the first decoded frame is
  the still.  gfx_thumbnail_animate accepts PENDING with an
  installed stream as the bootstrap state and the first frame upload
  flips it to AVAILABLE.  When the windowed open declines
  (admission, no reservation support, malformed), the file falls
  through to the historical whole-file still task, so nothing that
  worked before works less.
- rmp4_open_memory_avail reports the exact byte range that unblocks
  a stalled progressive open - a box header past the wall, or the
  whole moov body - through rmp4_video_stream_open_avail and
  image_transfer_anim_stream_new_avail.  rwebm and the audio side
  keep prefix semantics (their metadata is front-loaded).
- data_transfer_window_ensure() commits and reads an arbitrary
  island without paging the bytes between the frontier and it;
  data_transfer_window_rebase() then restarts the sequential window
  at the media floor, so the existing extend/advance feeder streams
  the samples exactly as it does for a front-moov file.  Islands sit
  above the frontier, so the decommit sweeps never touch them, and
  rmp4 materialises all tables at open, so nothing dereferences moov
  bytes afterwards.
- The anim-open loop jumps via the reported range with a progress
  guard, falling back to the sequential growth when no range is
  named.

Measured on the 4.3 GB co64 trailing-moov fixture: the open performs
three jumps totalling 3,382 bytes (two box headers plus the moov)
where it previously read 4.3 GB; resident memory after open is
9.8 MB against the head-plus-window budget, first-frame pixels are
byte-identical to the small-file decode, and playback streams over
the sliding window.  A two-hour recording now previews from the same
few-MiB footprint as a ten-second clip, still image included.

Validation: the still/chunked thumbnail suite passes across every
fixture (H.264/H.265 8/10-bit, WPP, scaling lists, 4K, the >4 GB
co64 layout, and the original user file); the 51-stream decoder
battery stays byte-exact vs HM; the MP4 glue lifecycle test passes;
ASan/LSan clean over the windowed open and playback of both the
small and the 4.3 GB fixtures; every touched libretro-common TU is
c89 -pedantic clean.

The audio-preview open in task_audio_mixer still grows its window
sequentially; wiring it to the need-range contract is a follow-up.

gfx/gfx_thumbnail.c

@@ -992,21 +992,64 @@ static void gfx_thumbnail_anim_open(gfx_thumbnail_t *thumbnail,
        * window until the header/index is covered. */
       {
          int need_more = 0;
+         size_t need_lo = 0, need_hi = 0;
          size_t avail  = GFX_THUMB_ANIM_WINDOW_KEEP;
+         int    jumps  = 0;
          if (avail > blen)
             avail = blen;
          for (;;)
          {
             stream = image_transfer_anim_stream_new_avail(
-                  (void*)base, blen, avail, type, &need_more);
+                  (void*)base, blen, avail, type, &need_more,
+                  &need_lo, &need_hi);
             if (stream || !need_more || avail >= blen)
                break;
+            if (need_hi > need_lo && need_hi > avail && need_hi <= blen)
+            {
+               /* The demuxer named the exact bytes that unblock it -
+                * a box header past the wall, or a trailing moov body.
+                * Committing just that island skips the mdat between
+                * the frontier and the metadata, which is what makes a
+                * multi-gigabyte tail-moov recording open from a
+                * few-MiB footprint instead of paging the whole file
+                * through the window.  The jump cap only guards
+                * against a demuxer bug looping without progress;
+                * a well-formed file needs one or two. */
+               if (++jumps > 64 ||
+                   !data_transfer_window_ensure(dt, need_lo, need_hi))
+                  break;
+               avail = need_hi;
+               continue;
+            }
+            /* No range named (WEBM, or a stall at unread front
+             * bytes): the historical sequential growth. */
             avail += GFX_THUMB_ANIM_WINDOW_KEEP;
             if (avail > blen)
                avail = blen;
             if (!data_transfer_window_extend(dt, avail))
                break;
          }
+         if (stream && jumps)
+         {
+            /* The open jumped over the media to reach a trailing
+             * moov; the sequential read frontier is still at the
+             * head.  Restart it at the media floor and prime one
+             * window so the first frames decode from resident bytes,
+             * after which the animate feeder streams as usual. */
+            size_t fl = image_transfer_anim_stream_media_floor(stream,
+                  type);
+            size_t hi = fl + GFX_THUMB_ANIM_WINDOW_KEEP
+                  + GFX_THUMB_ANIM_WINDOW_AHEAD;
+            if (hi > blen)
+               hi = blen;
+            data_transfer_window_rebase(dt, fl);
+            if (!data_transfer_window_extend(dt, hi))
+            {
+               image_transfer_anim_stream_free(stream, type);
+               stream = NULL;
+            }
+         }
+
          /* Types without a progressive open (animated WEBP) return
           * NULL with need_more clear: fall back to the whole buffer,
           * which the window has to make resident. */
@@ -1062,6 +1105,14 @@ static void gfx_thumbnail_anim_upload(gfx_thumbnail_t *thumbnail,
       thumbnail->texture = new_texture;
       thumbnail->width   = width;
       thumbnail->height  = height;
+      /* Anim-first bootstrap: the first frame of a video opened
+       * without a still decode makes the thumbnail drawable.
+       * Release-store pairs with the acquire-load in the draw path,
+       * as with the still upload. */
+      if (GFX_THUMB_STATUS_LOAD(&thumbnail->status) ==
+            GFX_THUMBNAIL_STATUS_PENDING)
+         GFX_THUMB_STATUS_STORE(&thumbnail->status,
+               GFX_THUMBNAIL_STATUS_AVAILABLE);
    }
 }
 
@@ -1124,10 +1175,19 @@ void gfx_thumbnail_animate(gfx_thumbnail_t *thumbnail)
 
    if (   !thumbnail
        || !(thumbnail->flags & GFX_THUMB_FLAG_ANIM_ACTIVE)
-       || !thumbnail->anim
-       || (GFX_THUMB_STATUS_LOAD(&thumbnail->status) !=
-             GFX_THUMBNAIL_STATUS_AVAILABLE))
+       || !thumbnail->anim)
       return;
+   {
+      /* AVAILABLE is the normal running state.  PENDING with a stream
+       * installed is the anim-first bootstrap for video files opened
+       * over a window without a preceding still decode: the first
+       * uploaded frame flips the status in anim_upload. */
+      enum gfx_thumbnail_status st = (enum gfx_thumbnail_status)
+            GFX_THUMB_STATUS_LOAD(&thumbnail->status);
+      if (   st != GFX_THUMBNAIL_STATUS_AVAILABLE
+          && st != GFX_THUMBNAIL_STATUS_PENDING)
+         return;
+   }
 
    now  = cpu_features_get_time_usec();
    type = (enum image_type_enum)thumbnail->anim_type;
@@ -1895,6 +1955,32 @@ void gfx_thumbnail_request_file(
        || !path_is_valid(file_path))
       return;
 
+   /* Video containers open over a sliding window before anything
+    * else: the still task reads the whole file into memory, which a
+    * multi-gigabyte recording fails outright (the allocation) or
+    * turns into a minute of disk reads for one frame - while the
+    * windowed animation path holds a fixed few-MiB footprint at any
+    * length, and its first decoded frame doubles as the still.  Only
+    * when the windowed open declines (admission, no reservation
+    * support, malformed container) does the file fall through to the
+    * historical whole-file still decode, which keeps small files on
+    * exotic platforms working exactly as before. */
+   {
+      enum image_type_enum ptype = image_texture_get_type(file_path);
+      if (   (ptype == IMAGE_TYPE_WEBM)
+          || (ptype == IMAGE_TYPE_MP4))
+      {
+         gfx_thumbnail_anim_open(thumbnail, file_path);
+         if (thumbnail->anim)
+         {
+            thumbnail->list_id = p_gfx_thumb->list_id;
+            GFX_THUMB_STATUS_STORE(&thumbnail->status,
+                  GFX_THUMBNAIL_STATUS_PENDING);
+            return;
+         }
+      }
+   }
+
    /* Load thumbnail */
    if (!(thumbnail_tag = (gfx_thumbnail_tag_t*)malloc(sizeof(gfx_thumbnail_tag_t))))
       return;

libretro-common/formats/data_transfer.c

@@ -596,6 +596,46 @@ bool data_transfer_window_extend(data_transfer_t *dt, size_t hi)
    return true;
 }
 
+bool data_transfer_window_ensure(data_transfer_t *dt, size_t lo,
+      size_t hi)
+{
+   if (!dt || !dt->window || dt->failed)
+      return false;
+   if (hi > dt->len)
+      hi = dt->len;
+   if (lo >= hi)
+      return true;
+   if (!dt->map_len)
+      return true;               /* fallback: whole file resident */
+   if (!data_transfer_wcommit(dt, lo, hi))
+   {
+      data_transfer_wfail(dt);
+      return false;
+   }
+   if (!data_transfer_read_at(dt, lo, dt->map + lo, hi - lo))
+   {
+      data_transfer_wfail(dt);
+      return false;
+   }
+   return true;
+}
+
+void data_transfer_window_rebase(data_transfer_t *dt, size_t pos)
+{
+   size_t p;
+   if (!dt || !dt->window || !dt->map_len || dt->failed)
+      return;
+   if (pos > dt->len)
+      pos = dt->len;
+   p = (pos / dt->page) * dt->page;
+   if (p <= dt->whi)
+      return;                    /* frontier already covers it */
+   dt->wlo    = p;
+   dt->whi    = p;
+   dt->wfreed = p;
+   return;
+}
+
 void data_transfer_window_advance(data_transfer_t *dt, size_t lo)
 {
    if (!dt || !dt->window || !dt->map_len || dt->failed)

libretro-common/formats/image_transfer.c

@@ -1083,10 +1083,15 @@ void *image_transfer_anim_stream_new(void *buf, size_t len,
 }
 
 void *image_transfer_anim_stream_new_avail(void *buf, size_t len,
-      size_t avail, enum image_type_enum type, int *need_more)
+      size_t avail, enum image_type_enum type, int *need_more,
+      size_t *need_lo, size_t *need_hi)
 {
    if (need_more)
       *need_more = 0;
+   if (need_lo)
+      *need_lo = 0;
+   if (need_hi)
+      *need_hi = 0;
    switch (type)
    {
       case IMAGE_TYPE_PNG:
@@ -1106,7 +1111,7 @@ void *image_transfer_anim_stream_new_avail(void *buf, size_t len,
       case IMAGE_TYPE_MP4:
 #ifdef HAVE_RMP4
          return rmp4_video_stream_open_avail((const uint8_t*)buf, len,
-               avail, need_more);
+               avail, need_more, need_lo, need_hi);
 #else
          break;
 #endif

libretro-common/formats/mp4/rmp4.c

@@ -1018,15 +1018,20 @@ static void rmp4_parse_moof(rmp4_t *m, const uint8_t *body, uint64_t size,
 /* ===== public API ===== */
 
 rmp4_t *rmp4_open_memory_avail(const uint8_t *data, size_t size,
-      size_t avail, int *need_more)
+      size_t avail, int *need_more, size_t *need_lo, size_t *need_hi)
 {
    rmp4_t  *m;
    uint64_t pos = 0, next;
    rmp4_box b;
    int      seen_known = 0, have_moov = 0, hit_wall = 0;
+   uint64_t want_lo = 0, want_hi = 0;
 
    if (need_more)
       *need_more = 0;
+   if (need_lo)
+      *need_lo = 0;
+   if (need_hi)
+      *need_hi = 0;
    if (avail > size)
       avail = size;
    if (!data || size < 16)
@@ -1051,7 +1056,13 @@ rmp4_t *rmp4_open_memory_avail(const uint8_t *data, size_t size,
        * header; at worst this costs one extra retry near the wall,
        * never a read past it). */
       if (avail < size && pos + 16 > avail)
-      { hit_wall = 1; break; }
+      {
+         /* the walk needs exactly this header; reporting the range
+          * lets a windowed caller commit it without reading the
+          * skipped body bytes in between */
+         want_lo = pos; want_hi = pos + 16;
+         hit_wall = 1; break;
+      }
       if (!rmp4_box_at(data, (uint64_t)size, pos, &b, &next))
          break;
       switch (b.type)
@@ -1067,7 +1078,11 @@ rmp4_t *rmp4_open_memory_avail(const uint8_t *data, size_t size,
          case RMP4_FOURCC('m','o','o','v'):
             seen_known = 1;
             if ((uint64_t)(b.body - data) + b.size > avail)
-            { hit_wall = 1; break; }   /* moov body still arriving */
+            {
+               want_lo = (uint64_t)(b.body - data) - 16;
+               want_hi = (uint64_t)(b.body - data) + b.size;
+               hit_wall = 1; break;    /* moov body still arriving */
+            }
             if (rmp4_parse_moov(m, b.body, b.size))
                have_moov = 1;
             break;
@@ -1085,7 +1100,16 @@ rmp4_t *rmp4_open_memory_avail(const uint8_t *data, size_t size,
    {
       rmp4_close(m);
       if ((hit_wall || avail < size) && need_more)
+      {
          *need_more = 1;
+         if (hit_wall && want_hi > want_lo && want_hi <= (uint64_t)size)
+         {
+            if (need_lo)
+               *need_lo = (size_t)want_lo;
+            if (need_hi)
+               *need_hi = (size_t)want_hi;
+         }
+      }
       return NULL;
    }
    if (!seen_known)
@@ -1124,7 +1148,7 @@ rmp4_t *rmp4_open_memory_avail(const uint8_t *data, size_t size,
 
 rmp4_t *rmp4_open_memory(const uint8_t *data, size_t size)
 {
-   return rmp4_open_memory_avail(data, size, size, NULL);
+   return rmp4_open_memory_avail(data, size, size, NULL, NULL, NULL);
 }
 
 const int64_t *rmp4_track_pts(const rmp4_t *m, int track, uint32_t *count)

libretro-common/formats/mp4/rmp4_audio.c

@@ -459,8 +459,10 @@ static int rmp4_audio_decode_any(const void *buf, size_t len,
     * below returns a non-1 value at the first sample past the wall,
     * which every codec loop already treats as "stop and keep what was
     * decoded". */
+   /* Prefix semantics suffice here: the audio preview grows its
+    * window by probing, so the precise need-range stays unused. */
    if (!(m = rmp4_open_memory_avail((const uint8_t*)buf, len, avail,
-         need_more)))
+         need_more, NULL, NULL)))
       return 0;
 
    for (i = 0; i < rmp4_num_tracks(m); i++)

libretro-common/formats/mp4/rmp4_video.c

@@ -529,7 +529,8 @@ static void rmp4_video_stream_close_decoder(rmp4_video_stream_t *s)
  * one-shot behaviour. */
 
 static rmp4_video_stream_t *rmp4_video_stream_open_begin(
-      const uint8_t *buf, size_t len, size_t avail, int *need_more)
+      const uint8_t *buf, size_t len, size_t avail, int *need_more,
+      size_t *need_lo, size_t *need_hi)
 {
    rmp4_video_stream_t *s;
    const rmp4_track *trk = NULL;
@@ -543,7 +544,8 @@ static rmp4_video_stream_t *rmp4_video_stream_open_begin(
    if (!(s = (rmp4_video_stream_t*)calloc(1, sizeof(*s))))
       return NULL;
 
-   if (!(s->demux = rmp4_open_memory_avail(buf, len, avail, need_more)))
+   if (!(s->demux = rmp4_open_memory_avail(buf, len, avail, need_more,
+         need_lo, need_hi)))
       goto fail;
 
    /* Pick the first video track whose codec we can decode. */
@@ -657,7 +659,8 @@ rmp4_video_stream_t *rmp4_video_stream_open(const uint8_t *buf,
 {
    rmp4_video_stream_t *s;
 
-   if (!(s = rmp4_video_stream_open_begin(buf, len, len, NULL)))
+   if (!(s = rmp4_video_stream_open_begin(buf, len, len, NULL,
+         NULL, NULL)))
       return NULL;
    rmp4_video_stream_scan_step(s, 0);
    if (rmp4_video_stream_open_finish(s) != 0)
@@ -669,13 +672,15 @@ rmp4_video_stream_t *rmp4_video_stream_open(const uint8_t *buf,
 }
 
 rmp4_video_stream_t *rmp4_video_stream_open_avail(const uint8_t *buf,
-      size_t len, size_t avail, int *need_more)
+      size_t len, size_t avail, int *need_more,
+      size_t *need_lo, size_t *need_hi)
 {
    rmp4_video_stream_t *s;
 
    if (need_more)
       *need_more = 0;
-   if (!(s = rmp4_video_stream_open_begin(buf, len, avail, need_more)))
+   if (!(s = rmp4_video_stream_open_begin(buf, len, avail, need_more,
+         need_lo, need_hi)))
       return NULL;
    /* The pre-scan reads the moov sample tables (no media bytes), so
     * once the open itself succeeded it always completes. */
@@ -1309,7 +1314,7 @@ int rmp4_video_process_image(rmp4_video_t *mp4, void **buf,
             mp4->stream = NULL;
          }
          if (!(mp4->stream = rmp4_video_stream_open_begin(
-               mp4->buf, mp4->len, avail, &need_more)))
+               mp4->buf, mp4->len, avail, &need_more, NULL, NULL)))
             /* The moov is still arriving (or, for a trailing moov or a
              * fragmented movie, most of the file is): wait for more
              * bytes rather than failing. */

libretro-common/include/formats/data_transfer.h

@@ -174,6 +174,27 @@ bool data_transfer_window_is_reserved(data_transfer_t *dt);
  * would silently take the wrong branch. */
 bool data_transfer_reserve_supported(void);
 bool data_transfer_window_extend(data_transfer_t *dt, size_t hi);
+/* Make an arbitrary byte range resident and read, without touching
+ * the sequential window: pages between the frontier and the range
+ * stay uncommitted.  This is what lets a trailing MP4 moov become
+ * readable without paging the multi-gigabyte mdat before it through
+ * memory.  Idempotent; safe to overlap the head, the window, or a
+ * previous island.  The island stays resident until the handle is
+ * freed - the decommit sweeps only run below the sequential
+ * frontier, which an island by construction sits above.  Callers
+ * must only dereference bytes inside the head, the window, or a
+ * range this call has covered. */
+bool data_transfer_window_ensure(data_transfer_t *dt, size_t lo,
+      size_t hi);
+/* Restart the sequential window at pos: after a need-range open
+ * jumped over the mdat to a trailing moov, the read frontier still
+ * sits at the head, and the first extend() toward the media would
+ * read the skipped gigabytes after all.  Rebasing moves the frontier
+ * to the media floor so extend()/advance() stream the samples from
+ * there, exactly as they do for a front-moov file from the head.
+ * No-op when pos does not lie past the frontier, so front-moov files
+ * are unaffected. */
+void data_transfer_window_rebase(data_transfer_t *dt, size_t pos);
 void data_transfer_window_advance(data_transfer_t *dt, size_t lo);
 void data_transfer_window_rewind(data_transfer_t *dt);
 /* Raise the permanently-resident head.  For codecs whose loop

libretro-common/include/formats/image.h

@@ -268,8 +268,13 @@ void *image_transfer_anim_stream_new(void *buf, size_t len,
  * resident and a larger prefix should be retried.  Returns NULL for
  * types without a partial open (animated WEBP), so the caller keeps
  * the whole-buffer path for those. */
+/* need_lo/need_hi (optional): a precise byte range that unblocks a
+ * stalled progressive open, when the container can name one (MP4's
+ * box headers and moov body).  0/0 otherwise; the caller then grows
+ * the available prefix instead. */
 void *image_transfer_anim_stream_new_avail(void *buf, size_t len,
-      size_t avail, enum image_type_enum type, int *need_more);
+      size_t avail, enum image_type_enum type, int *need_more,
+      size_t *need_lo, size_t *need_hi);
 
 void image_transfer_anim_stream_free(void *stream,
       enum image_type_enum type);

libretro-common/include/formats/rmp4.h

@@ -143,8 +143,13 @@ int rmp4_read_packet(rmp4_t *mp4, rmp4_packet *pkt);
  * non-NULL) is set when the failure was running out of available
  * bytes rather than malformed data - retry with a larger avail.
  * rmp4_open_memory is this with avail == size. */
+/* need_lo/need_hi (optional): when the open stalls on the available
+ * prefix, the exact byte range that would let it progress - a box
+ * header past the wall, or the whole moov body.  A windowed caller
+ * can commit just that range instead of reading everything up to it;
+ * both stay 0 when the stall has no such range. */
 rmp4_t *rmp4_open_memory_avail(const uint8_t *data, size_t size,
-      size_t avail, int *need_more);
+      size_t avail, int *need_more, size_t *need_lo, size_t *need_hi);
 
 /* Raise the number of valid bytes (monotonic; clamped to the file
  * size). */

libretro-common/include/formats/rmp4_video.h

@@ -93,8 +93,10 @@ rmp4_video_stream_t *rmp4_video_stream_open(const uint8_t *buf,
 /* Open against a partially-read buffer: 'avail' leading bytes are
  * valid (raise later with the stream set_avail).  On NULL, *need_more
  * distinguishes "feed more bytes and retry" from malformed data. */
+/* need_lo/need_hi: see rmp4_open_memory_avail. */
 rmp4_video_stream_t *rmp4_video_stream_open_avail(const uint8_t *buf,
-      size_t len, size_t avail, int *need_more);
+      size_t len, size_t avail, int *need_more,
+      size_t *need_lo, size_t *need_hi);
 
 void rmp4_video_stream_close(rmp4_video_stream_t *stream);