Commit Graph
12 Commits
Author SHA1 Message Date
LibretroAdmin f140e9592a Soft-patching: apply a streamed IPS patch to the loaded ROM where it lies
A streamed IPS patch built its target as the source arrived: a second
buffer the size of the ROM, taken at open, every byte of the source
copied into it, then the records laid over it, and the source freed.
The target of an IPS patch is the source with the records on it, and
the source is the content buffer the load already owns.

patch_stream_finish_into() finishes a stream whose source the caller
owns (malloc'd, given to patch_stream_set_source()): an IPS patch puts
its records on that buffer in place, growing it with realloc() only
when the target is longer, and any other format hands back its target
in the buffer's place. On failure the buffer is still the unpatched
source, as before. Fed in place, IPS copies nothing while the source
arrives; a chunk fed from elsewhere copies what came before through to
the target and carries on as before. The target is no longer taken at
open but by the first copy-through, so the in-place path never takes
one. The content load finishes its patch with finish_into().

A 16 MiB ROM with a 2000-record IPS patch, first touch: 6.7 ms ->
0.05 ms to apply, and 16 MiB less heap during the load.

The patch_stream test finishes every in-place and fallback comparison
with finish_into() on a buffer of its own, and checks a 4 MiB IPS
source is patched where it lies without being moved or copied. Leaving
the grown tail unzeroed, skipping the records in place or in the
catch-up, copying nothing in the catch-up, or copying through while fed
in place each fail it.
2026-11-03 00:25:08 +01:00
LibretroAdmin 69d936201c Soft-patching: read the source in place while streaming BPS, UPS, xdelta
The content load streams a soft patch alongside the read: each span the
transfer fills is fed to the applier as it arrives. BPS and xdelta may
copy from anywhere in the source, so both kept their own growing copy of
every byte fed (BPS in src_buf, xdelta in own_src), and UPS kept the
part it had not consumed yet in its carry. The transfer buffer already
holds the whole source in one allocation that never moves, and stays
alive, detached as the content, until the patch is finished - so that
copy is a second ROM in memory, written once and read back.

patch_stream_set_source() (and vcdiff_stream_set_source() under it)
tells the stream where the source it is about to be fed lies. Chunks
fed from there are read in place: no copy, no realloc as it grows, no
memmove of the UPS carry. A chunk fed from anywhere else falls back to
the old copy, starting with what came before it, so a caller that does
not keep the source whole still gets the right answer. IPS keeps no
source and is unchanged. content_file_pump_source() sets the transfer
buffer as the source.

A 32 MiB BPS hack patch: 32 MiB less heap during the load, and
143 -> 137 ms to apply.

The patch_stream test feeds each comparison as a copy, in place, or in
place then from another buffer (freed at once, so ASan catches a stream
still reading it), and checks with glibc's heap counters that a 4 MiB
BPS source and the xdelta source are no longer copied. Dropping the
bytes the fallback copies, restarting the UPS carry at 0, reading the
wrong byte in place, or ignoring the setter each fail it.
2026-11-03 00:25:03 +01:00
LibretroAdmin bd7f70065e samples/tasks/patch_stream: tiny feeds for small sources only
Every trial fed its source a byte, two and three at a time, sources of
up to 8 KiB among them: some two hundred million feed calls under ASan,
most of the test's time, reaching no record boundary the seven-byte
feed does not. Those three feed sizes now go over sources of up to
1025 bytes, past the 1024-byte edges; larger ones keep the 7, 64, 1024
and whole-buffer feeds.

Measured on two CPUs: 103 s before, 54 s after; 238510 comparisons of
the 280000, all matching.
2026-10-24 09:21:21 +02:00
LibretroAdmin 0466e44485 samples/tasks/patch_stream: build at -O1, and one scratch buffer
The test ran under ASan + UBSan at -O0, the slowest step of
samples-tasks. -O1, as the other samples build, keeps every check and
runs faster; and the 2 MiB buffer each of the 20000 trials built its
patches in is allocated once, as each comparison reads its own
exact-size copy of the patch, where an overread is caught.

Measured on two CPUs: 138 s before, 103 s after, the same report.
2026-10-24 09:21:09 +02:00
LibretroAdmin 21635dd171 samples: build with -g1 instead of -g
The sample and test builds used full debug info. -g1 keeps what
sanitizer reports and backtraces need - function names, files and line
numbers - and drops the variable and type information nobody reads in
CI. Objects get about 30% smaller (rh265.o under ASan+UBSan: 5.1MB to
3.7MB), so the ccache caches hold more of each job's build and are
quicker to restore and save.
2026-10-13 11:50:11 +02:00
libretroadmin afbad7a104 samples: shared sources build into each sample's own obj/
Samples compiled shared tree sources in place, so whichever sample ran
next linked those objects with the flags they were built with, as
unity_identity did with mic_handshake's ASan objects. Every sample that
builds a source from outside its directory now includes
sample_objdir.mk and maps those objects under its own obj/; its own
objects stay where they were. The rule for the mapped objects has a
slash in its target, so make 3.81 does not strip the directory.

tools/sample_objdir_check.py dry-runs every sample Makefile and fails
on an object written outside the sample; the Samples-Objdir-Check
workflow runs it.
2026-10-10 19:43:34 +02:00
LibretroAdminandlibretroadmin e152c55d72 encodings/crc32: fix a data race in table init, enable slicing-by-8 on every target, add a PCLMULQDQ fold (#19278)
* encodings/crc32: make slicing-by-8 const, universal and thread-safe

The slicing-by-8 tables were built at first call into a mutable .bss
array behind an unsynchronised "ready" flag. Two threads entering
encoding_crc32() concurrently both run the generator; the flag is
stored with a plain write after the table stores, so nothing orders
the publication against the payload on a weakly ordered target. A
reader can observe ready == 1 and then load a half-written entry,
which yields a silently wrong checksum rather than a crash.

ThreadSanitizer over eight threads released from a barrier reports
nine distinct races on the old code, including reads of the 8 KiB
table concurrent with its construction.

Precompute the tables instead and emit them const from a generated
header. They land in .rodata, cost nothing to construct, are shared
between processes and cannot race.

The fast path was also gated on x86 alone. Every other target -- all
ARM builds that do not pass +crc, PPC, MIPS, RISC-V -- fell through to
the byte-at-a-time loop. Only the assumption of little-endian word
loads actually tied the loop to x86, so byte-swap under MSB_FIRST and
let everyone use it.

While here: load through memcpy rather than casting uint8_t* to
uint32_t*, drop the const cast in the ARM path, and decrement in the
body of the byte tails rather than in the loop test, which wrapped len
to SIZE_MAX on the final iteration. That last one is defined behaviour
but trips -fsanitize=unsigned-integer-overflow.

Measured on Xeon @2.8GHz, 256 KiB buffer, gcc 13 -O2:

  byte-at-a-time (what non-x86 got)   0.39 GB/s
  slicing-by-8, .bss + ready flag     1.94 GB/s
  slicing-by-8, const .rodata         2.10 GB/s

So ~5.4x for every non-x86 target and ~8% on x86 from dropping the
per-call flag test and letting the tables be constant.

Verified bit-exact against a bitwise reference for lengths 0-1200 at
all 16 alignments, streaming splits, non-zero seeds and 1 MiB blocks,
on x86-64, i386, aarch64 with and without +crc, and big-endian
powerpc under qemu. Clean under ASan, LSan, UBSan, MSan, TSan and
valgrind memcheck.

* encodings/crc32: add PCLMULQDQ folding path on x86

Slicing-by-8 issues one table load per input byte and accumulates all
eight lookups into a single register, so each 8-byte iteration is a
serial chain of eight dependent XORs behind an L1 load. Measured 1.47
cycles/byte, against a load-port floor of roughly 0.5. It is
latency-bound on that chain, not bound by cache misses: cachegrind
over a 256 KiB working set shows the 8 KiB table staying resident,
with D1 misses accounting for compulsory stream fill only. There are
no register spills in the loop on x86-64 or i386.

Carry-less multiply folding removes the problem rather than tuning it.
Four independent 128-bit accumulators consume 64 bytes per iteration
with no loop-carried scalar dependency, and data references drop from
1.25 per byte to 0.064 -- a factor of 19.6.

Dispatch is at runtime on the CPUID PCLMULQDQ bit, so generic x86
builds pick it up without raising the baseline ISA; pre-Westmere and
pre-Bulldozer parts keep the table path. Buffers under 64 bytes go to
the table path too, since below four accumulators the reduction tail
costs more than it saves.

The detection result is cached in a single retro_atomic_int_t holding
the whole tri-state rather than a plain flag guarding a separate
payload, so concurrent probers merely store the same value and there
is no unsynchronised write to order against. Same acquire/release
idiom as cpu_features_get().

The folding constants are all K(n) = bit_reflect_32(x^n mod P(x)) << 1
and are derived rather than copied; the derivation is in the comment
above them and reproduces the published values.

Xeon @2.8GHz, gcc 13 -O2, GB/s:

                        64B    4K    32K   256K    4M    64M
  before               2.41  1.91  1.93   1.94  2.01  1.99
  after                7.26 19.69 20.80  20.82 19.73  7.43
  zlib 1.3             0.51  3.11  3.22   3.31  3.33  3.10

10.8x over the previous code and 6.5x over zlib in cache; the 64 MiB
column is DRAM-bandwidth bound on this box, not compute bound.

Callers that benefit most are the per-frame core-memory hash in
netplay_frontend.c, content scanning through interface_stream.c, the
zip/7z/zstd archive CRCs and CHD hunk verification in rchd.c.

Verified bit-exact against a bitwise reference and against zlib
crc32() for lengths 0-1200 at all 16 alignments, streaming splits,
non-zero seeds, 1 MiB blocks and a 5 MiB single shot. Clean under
ASan, LSan, UBSan, MSan, TSan and valgrind memcheck, including a
guard-page harness that hashes 901 buffers ending flush against an
unmapped page to prove the 16-byte SIMD loads never overread.

* samples/encodings: add a crc32 regression test

encoding_crc32() now selects between three implementations -- a
PCLMULQDQ fold chosen at runtime on x86, the ARMv8 CRC32 instructions
chosen at compile time, and a portable slicing-by-8 table -- and
nearly every way that can go wrong is silent.  A wrong checksum is
still a plausible 32-bit number.  Nothing pinned the value from
outside the implementation, so add something that does.

Seven sections.  Values are checked against a bit-by-bit reference
built straight from the polynomial: no tables, no word loads, and
endian-agnostic, which the slicing-by-8 loop is not.  Twelve frozen
constants (the published CRC-32/ISO-HDLC check value plus zlib
outputs) sit alongside it so that a broken reference cannot cancel out
a broken implementation.

The sweep hashes lengths 0-1200 at all 16 alignments with each buffer
allocated at exactly the size read, so the last valid byte abuts an
ASan redzone and a SIMD load that runs one iteration too far is an
error rather than a silent success.  The guard-page section repeats
that check without a sanitizer, hashing 901 buffers that end flush
against a PROT_NONE page, which is what gives the qemu lanes overread
coverage.  Cross-path agreement hashes the same megabyte one-shot and
in chains of 1 to 4096 bytes, forcing the fold path and the table path
to agree with each other rather than only with the reference.

Verified by injecting eleven regressions and confirming each is
caught: fold-tail overread, dispatch threshold, three separate
constant bit-flips, both complements, a transposed slice index, a
dropped seed on the SIMD path, the atomic feature cache reverted to a
plain int, and the MSB_FIRST byteswap removed.  That last one is
caught only on big-endian -- on x86 the swap compiles away and every
other check still passes -- which is why powerpc is in the cross-arch
matrix.

Two things about the concurrency section are load-bearing and are
commented as such in the source.  It runs first, because any section
ahead of it warms the feature-detection cache on the main thread and
leaves the workers reading an already published value.  And its
workers make exactly one dispatching call each: ThreadSanitizer keeps
only a few shadow entries per address, so a one-write/many-read lazy
init gets crowded out of that history by further traffic.  Measured on
this test, eight threads making one call each report the race in 30
runs out of 30; the same eight threads making forty calls each report
it in none, and raising the thread count makes it worse rather than
better.  The sustained-load checks therefore live in a separate
section that runs afterwards.

Registered in the samples workflow's executed-target list, with a
Clang/ThreadSanitizer step and a cross-arch job covering aarch64 with
and without +crc, armv7, and big-endian powerpc under qemu-user.
Stock aarch64 gcc does not define __ARM_FEATURE_CRC32, so those two
aarch64 entries really do exercise different code: the +crc object
contains three CRC32 instructions, the default object none.

Runs in 0.3s natively, 0.4s under ASan.  Clean under ASan, LSan,
UBSan, MSan, TSan and valgrind memcheck, with GCC and Clang, at
-m32 and -m64, and with HAVE_THREADS=0.

* samples/encodings: run the crc32 TSan step in the job that checks out

The step was inserted immediately before the first step of
retro-atomic-cross rather than after the last step of samples, so it
ran as that job's opening step -- before its checkout -- and failed
with

  An error occurred trying to start process '/usr/bin/bash' with
  working directory '.../libretro-common/samples/encodings/crc32'.
  No such file or directory

The directory was never missing; nothing had been cloned yet. Move it
to the end of the samples job, which has already checked out and
installed clang by that point.

* encodings/crc32: gate the PCLMULQDQ path on compiler support too

The path was selected on architecture alone. Being on x86 says the CPU
might have PCLMULQDQ -- which is why the runtime CPUID check exists --
but it says nothing about whether the toolchain can compile
_mm_clmulepi64_si128 or honour __attribute__((target("pclmul"))). A
toolchain defining __i386__ without wmmintrin.h would fail to build.

Test __has_attribute(target) and __has_include(<wmmintrin.h>) rather
than a compiler version, with a !__clang__ guarded GCC >= 4.4 fallback
for compilers too old to have those. The negation matters: Clang
reports __GNUC__ 4, __GNUC_MINOR__ 2 permanently, so a bare ">= 4.4"
predicate rejects it, and Clang is the toolchain on every Apple
platform, the Android NDK since r18, Emscripten and PS4-ORBIS. Getting
that backwards would silently drop the fold on all of them while still
compiling and passing every test.

Verified the fold is still emitted, not just that the file builds:
pclmulqdq instruction counts of 33 (gcc), 41 (clang), 30 (gcc -m32),
47 (clang -m32) and 30 (gcc -m32 -march=i686); zero on aarch64. The
i686 baseline case matters because the target attribute has to pull
SSE2 into that one function without raising it for the translation
unit.

* features/cpu: drop the VFS dependency, add RETRO_SIMD_PCLMUL, use it for crc32

Three things that only make sense together.

features_cpu.c read /proc/cpuinfo and the sysfs CPU lists through
filestream_*, which pulls streams/file_stream.c, vfs/vfs_implementation.c
and compat/compat_strl.c behind it. Nothing about a kernel pseudo-file
wants to be interceptable by a user-supplied VFS, so use stdio. Three
call sites: the ARM feature-name scan, the sysfs cpulist reader, and the
Linux model-name lookup. cpulist_read_from() now reads into a fixed
buffer rather than sizing from the file, because sysfs reports a page
for files that contain "0-7\n". After this the translation unit's only
undefined symbols are libc ones -- verified on x86-64, aarch64, armv7
and powerpc.

That matters because encoding_crc32() wants to ask whether the CPU has
PCLMULQDQ, and a leaf utility should not acquire a transitive
dependency on file I/O to find out. With the VFS gone it costs
features_cpu.c and compat_strl.c, both leaves.

There was no bit to ask for. RETRO_SIMD_AES is CPUID.1:ECX[25];
PCLMULQDQ is ECX[1]. They shipped together on most parts but
hypervisors mask them independently and some early Westmere SKUs had
AES fused off, so one cannot stand in for the other. Add
RETRO_SIMD_PCLMUL at bit 24 and probe it properly.

Darwin needs its own path. x86_cpuid() is compiled under
"CPU_X86 && !__MACH__", so Intel macOS never reaches the CPUID branch
and relies on sysctl -- and Apple publishes no hw.optional key for
PCLMULQDQ. Read machdep.cpu.features instead, which is the leaf-1
feature-name list, matching on whole space-separated tokens so a
future "PCLMULQDQ2" cannot false-positive a plain strstr().

encoding_crc32() then drops its private CPUID and its own atomic cache
in favour of cpu_features_get(), which already probes once behind an
acquire/release pair. Note the probe needs no OSXSAVE/XCR0 dance:
PCLMULQDQ is an XMM instruction, so CR4.OSFXSR is the only OS
requirement and that is universal. Only AVX-width state needs xgetbv.

crc32_test passes under gcc, clang, -m32, ASan+UBSan, TSan and MSan
natively, and on aarch64, aarch64+crc, armv7 and big-endian powerpc
under qemu-user -- the cross lanes now linking features_cpu.c with no
architecture conditional, because there is nothing left to
conditionalise.

* features/cpu: stop treating CPUID leaf numbers and results as signed

UndefinedBehaviorSanitizer's implicit-conversion check fires twice in
cpu_features_probe() and cpu_features_get_model_name():

  features_cpu.c:940:14: implicit conversion from type 'unsigned int'
  of value 2147483648 to type 'int' changed the value to -2147483648

x86_cpuid() took its leaf number as int, so every extended-leaf call
site -- 0x80000000, 0x80000001, 0x80000002+i -- passed a value with
the top bit set through a signed parameter. Widen it to uint32_t.

That in turn exposed the reverse conversion: flags[] is int32_t but
holds raw register contents, and leaf 0x80000000 legitimately returns
a maximum-leaf value above INT32_MAX, so assigning flags[0] to the
unsigned max_flag converted implicitly. Spell those two out with a
cast rather than changing the array type, which would break source
compatibility for anything outside the tree calling x86_cpuid().

Neither was a live defect -- the values round-trip on every ABI we
build for -- but they made the file impossible to run under
-fsanitize=implicit-conversion, which matters now that the PCLMULQDQ
probe lives here.

Predates the crc32 work; found while sanitizing that series.

Clean afterwards under gcc ASan+LeakSan+UBSan, clang MSan, clang
TSan, clang UBSan+integer+implicit-conversion+unsigned-integer-overflow,
and valgrind memcheck, exercising cpu_features_get(),
cpu_features_get_core_amount() and cpu_features_get_model_name() from
eight threads. Feature detection unchanged: same bitmask, PCLMULQDQ
still reported.

* samples, libretro-db: link features_cpu.c wherever encoding_crc32.c is linked

encoding_crc32() now asks cpu_features_get() whether the CPU has
PCLMULQDQ, so encoding_crc32.o carries an undefined reference to it.
Every source list that already linked encoding_crc32.c needs
features_cpu.c beside it, and features_cpu.c uses strlcpy, so those
lists need compat_strl.c too. Seven sample builds and libretro-db
itself were failing to link:

  libretro-db/samples/libretrodb
  libretro-db/samples/rmsgpack
  libretro-common/samples/streams/rzip
  libretro-common/samples/formats/png
  libretro-common/samples/formats/json
  libretro-common/samples/file/archive_file
  samples/tasks/patch_stream

Done per source list rather than per file. png/Makefile has four
independent lists and only two of them already carried compat_strl.c;
adding a duplicate object to a link line is a multiple-definition
error, so each list is checked on its own.

Verified by building every Makefile under a samples/ directory
repo-wide with SANITIZER=address,undefined -- 50 of 51 clean, the
lone failure being deps/SPIRV-Cross/samples/cpp, which wants glm
headers and is untouched by this series.

* features/cpu: elect a single writer for the cached feature mask

cpu_features_get() publishes the mask with a release store and reads it
with an acquire load, which orders publisher against consumer. What it
does not order is publisher against publisher: if several threads reach
the probe before any has published, they all write cpu_features_cache
with plain stores. The comment there argued that was fine because the
probe is idempotent and they store the same bits, but same value or
not, concurrent unordered writes to one object are a data race:

  WARNING: ThreadSanitizer: data race
    Write of size 8 ... in cpu_features_get
    Previous write of size 8 ... in cpu_features_get
    Location is global 'cpu_features_get.cpu_features_cache'

Elect exactly one writer with an atomic increment. Threads that lose
return the value they computed themselves, which is identical, so
nobody spins and nobody writes memory another thread is writing.
retro_atomic_fetch_add_int() returns the previous value in all seven
backends.

Latent until something called cpu_features_get() from several threads
at once during startup; crc32_test's first-touch storm is the first
thing in the tree that does. Reproduced deliberately by widening the
probe with a 30ms sleep and calling from eight barrier-released
threads: one race before this change, none after. The window is hard
to hit on a single-core machine, which is why local runs passed while
the CI runner failed.

* encodings/crc32: dispatch the ARMv8 CRC32 path at runtime

The ARM path was gated on __ARM_FEATURE_CRC32 alone, which the compiler
only defines when the target baseline already includes the instructions.
In tree that is arm64 macOS, where Clang derives it from the default
-mcpu, and Makefile.libnx, which passes +crc explicitly. Everything else
on ARM ran slicing-by-8: Android arm64, Linux ARM, Raspberry Pi, iOS and
tvOS, on hardware that has had the instructions since ARMv8.1.

Compile the path in whenever the toolchain can build it into a single
function without raising the ISA for the translation unit, and choose
between it and the table at runtime, the way the x86 side chooses on
CPUID. Measured on an Apple M1, 256 KiB buffer: 1.44 GB/s for
slicing-by-8 against 3.38 for the instructions, and the target-attribute
build is indistinguishable from the baseline build at 3.38 vs 3.38, so
nothing is lost by not raising the baseline.

Raising it instead is not an option on Apple: iOS deployment targets in
pkg/apple reach back to 6.0 and Clang reports CRC32 as absent through
A10, so an unconditional +crc would be an illegal instruction on those
devices. Runtime detection is the only route that covers them.

Detection goes through a new RETRO_SIMD_CRC32 bit rather than a second
private probe: hw.optional.armv8_crc32 on Darwin with the FEAT_CRC32
spelling as a fallback, and the existing check_arm_cpu_feature() reader
on Linux, where aarch64 lists it as "crc32". Both keys confirmed present
and answering on an M1.

aarch64 with no baseline flags now emits the same three crc32
instructions as a +crc build and takes them at runtime. Verified on
aarch64 with and without +crc, armv7, big-endian powerpc and x86-64,
which correctly reports the bit absent and is otherwise untouched. Every
sample in the tree still builds.

* encodings/crc32: pick the ARM intrinsics header by compiler, not by _M_ARM64

CLANGARM64 on windows-11-arm fails to build:

  encoding_crc32.c:286:10: fatal error: 'arm64_neon.h' file not found

arm64_neon.h is an MSVC header. Selecting it on _M_ARM64 alone breaks
any Clang that defines that macro for Windows-on-ARM compatibility --
MSYS2's CLANGARM64 does, and it ships arm_acle.h and no arm64_neon.h.
Clang targeting aarch64-pc-windows-msvc defines it too, while the
mingw32 triple does not, so the macro says nothing useful about which
header exists. Key on _MSC_VER && !__clang__ instead.

The bad include predates this series, but nothing reached it until the
runtime ARM dispatch started compiling the block on targets whose
baseline lacks __ARM_FEATURE_CRC32, which is exactly this one.

Reproduced by forcing -D_M_ARM64 on an aarch64 Clang: the same fatal
error before, clean after. Rechecked across gcc and clang on x86-64,
aarch64 gcc with and without +crc, aarch64 clang plain and with
-D_M_ARM64, armv7 and big-endian powerpc.

---------

Co-authored-by: libretroadmin <>
2026-07-28 15:48:11 +02:00
libretroadmin 151b954951 samples: compare refusals, not just output, in the patch_stream suite
Two real bugs went through this suite untouched - the UPS and the BPS
streaming appliers both accepted patches built for a different revision
of the content, where the whole-buffer appliers refuse. Neither was
found here. Both were found by applying a patch to a real ROM through a
real build.

The reason is structural, not a missing case. The oracle was a copy of
the whole-buffer logic with the checksum validation deliberately left
out, on the reasoning that byte-identical output was what the streaming
appliers owed. So the suite compared bytes, and on a patch whose
checksums are correct - which is everything the generator makes - both
sides agreed on the bytes while disagreeing on whether the patch should
have been applied at all. No number of additional generated patches
would have closed that, because the disagreement is invisible to a
byte comparison.

Put the validation back in the oracle and compare verdicts. Then add
the cases that need it: a valid patch applied to a source one byte
different from the one it was built against, which is the ordinary way
patching goes wrong and precisely what the formats carry a source
checksum to catch. Four hundred of them, UPS and BPS, where the only
correct answer is that both appliers refuse.

Checked against the bugs it exists for: with the checksum validation
removed from the appliers again, this section reports "oracle refused,
streamed accepted" and the suite fails. With it in place everything
passes, including the 300000 output comparisons, clean under address
and undefined-behaviour sanitizers.
2026-07-23 10:38:59 +02:00
libretroadmin ba610e33b6 tasks: validate the UPS checksums when streaming
build showed the streaming applier accepting it and handing the core
silently corrupt content, where the whole-buffer applier refuses the
same patch and the content runs unpatched.

UPS carries a checksum for the patch, for the source and for the
target, and ups_apply_patch checks all three - refusing on any
mismatch, with an allowance for a patch applied in reverse, where the
two content checksums swap. The streaming applier checked none of them.
It decoded the diff stream, produced a target, and returned it. For a
patch that is intact the result is identical, which is why nothing
noticed; for a damaged one the caller's careful fallback - keep the
unpatched buffer when a patch is rejected - never fires, and the
corrupt target is adopted instead.

Track the same three checksums and refuse exactly what the whole-buffer
applier refuses. The source checksum accumulates as chunks are fed, the
target's is taken once the target is complete, and the patch's is over
the patch minus its own trailing four bytes.

The differential suite could not have caught this. It compares output
bytes for patches its generator builds with correct checksums, and its
oracle deliberately omits the validation verdicts - so both sides
agreed on the bytes while disagreeing on whether the patch should have
been applied at all. That is the gap: a contract stated as "whatever
the whole-buffer applier does, the streaming one does too" was only
ever enforced for the output, never for the refusal.

BPS was tested the same way and behaves correctly, but for a weaker
reason than its checksums: damage to its instruction stream trips the
structural bounds first. Worth knowing that its correct behaviour here
is not evidence its checksums are being verified either.

Verified: a valid .ups still applies to the real ROM with the expected
CRC, a damaged one is now refused by both appliers and the content runs
with the source ROM's own CRC and its internal checksum intact. The
differential suite is unchanged at 300000 comparisons.
2026-07-23 10:11:39 +02:00
libretroadmin dd19019453 tasks: record what a damaged patch actually does, per format
Running the streaming appliers against a real ROM through a real build
turned up a claim of mine that was wrong. The commit that introduced
streaming asserted that a deliberately corrupt patch would leave the
content unpatched, and offered a truncated .ips as the check. It does
not, and the check was the one format where the claim does not hold.

BPS and UPS carry checksums, and xdelta is structured enough that
damage almost always violates it, so for those three a damaged patch is
refused: finish() fails, the caller keeps the unpatched buffer, and the
content runs unpatched. That was confirmed end to end - a bit-flipped
and a truncated .xdelta each had the streamed attempt refuse, the
whole-buffer pass refuse after it, and the content load with the
source ROM's own checksum.

IPS has no checksum, no length and no framing beyond a terminator, so a
truncated one is a shorter valid one as far as the format is concerned.
Both appliers apply its intact leading records and produce a partially
patched target. That is not a regression and not a decision taken in
the streaming appliers: the whole-buffer applier has always done it,
and the two were confirmed to produce byte-identical output for exactly
this input. It is a property of the format.

Write both down - in the header, where a caller decides what to do
about a failed patch, and in the test, which deliberately does not
assert refusal because the contract that holds for every format is the
weaker one: whatever the whole-buffer applier does with a patch, the
streaming one does too.

No functional change.
2026-07-23 09:53:26 +02:00
libretroadmin 5163bccf9f samples: cover the xdelta streaming path
Adding .xdelta to the streaming appliers left this suite testing three
of the four formats. The gap is thin - the format is a shell delegating
to the VCDIFF decoder - but thin wiring is exactly what gets connected
wrongly and never noticed, since a format that silently fell back to
the whole-buffer pass would still produce correct content.

Cover it the way its shape allows. The decoder itself is verified
against the reference implementation in samples/encodings/vcdiff, so
what is checked here is the delegation: feeding through this module's
own API, in chunks, must reach that decoder and return what decoding
the whole buffer at once returns. Five chunk sizes from one byte up,
and a feed that stops short, which must be refused rather than answered
with a target built from a partial source.

The vector is a golden patch from the reference encoder rather than a
generated one, because there is no xdelta encoder in the tree to
generate with - the same reason the vcdiff sample carries golden
vectors. It exercises an edited span, a shifted region and a run, so
the patch carries some instruction variety rather than one flat copy.

The whole suite still reports 300000 comparisons for the other three
formats with zero mismatches, and is clean under address,undefined.
2026-07-23 09:24:27 +02:00
libretroadmin 50bc68358d tasks: move the patch_stream test into samples
samples/ is where the tree's standalone regression tests live, each with
its own Makefile - samples/tasks/bps_patch is the directly analogous
case, covering the whole-buffer .bps applier this streaming one has to
reproduce. Move the differential harness there rather than leaving it in
a one-off tasks/test directory, and give it the same Makefile shape,
including the SANITIZER= hook.

The oracle's formatting is cleaned up in the process: it arrived as
compressed prototype code and built with -Wmisleading-indentation
warnings. It is the reference the streaming appliers are checked
against, so it should read as such.

No change to what is tested: 300000 comparisons (280000 IPS/UPS, 20000
BPS), zero mismatches, clean under address,undefined via the Makefile's
sanitizer path.
2026-07-23 07:43:00 +02:00