mirror of
https://github.com/libretro/RetroArch.git
synced 2026-11-22 01:33:23 -10:00
codex/android-native-streaming
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
f140e9592a |
Soft-patching: apply a streamed IPS patch to the loaded ROM where it lies
A streamed IPS patch built its target as the source arrived: a second buffer the size of the ROM, taken at open, every byte of the source copied into it, then the records laid over it, and the source freed. The target of an IPS patch is the source with the records on it, and the source is the content buffer the load already owns. patch_stream_finish_into() finishes a stream whose source the caller owns (malloc'd, given to patch_stream_set_source()): an IPS patch puts its records on that buffer in place, growing it with realloc() only when the target is longer, and any other format hands back its target in the buffer's place. On failure the buffer is still the unpatched source, as before. Fed in place, IPS copies nothing while the source arrives; a chunk fed from elsewhere copies what came before through to the target and carries on as before. The target is no longer taken at open but by the first copy-through, so the in-place path never takes one. The content load finishes its patch with finish_into(). A 16 MiB ROM with a 2000-record IPS patch, first touch: 6.7 ms -> 0.05 ms to apply, and 16 MiB less heap during the load. The patch_stream test finishes every in-place and fallback comparison with finish_into() on a buffer of its own, and checks a 4 MiB IPS source is patched where it lies without being moved or copied. Leaving the grown tail unzeroed, skipping the records in place or in the catch-up, copying nothing in the catch-up, or copying through while fed in place each fail it. |
||
|
|
69d936201c |
Soft-patching: read the source in place while streaming BPS, UPS, xdelta
The content load streams a soft patch alongside the read: each span the transfer fills is fed to the applier as it arrives. BPS and xdelta may copy from anywhere in the source, so both kept their own growing copy of every byte fed (BPS in src_buf, xdelta in own_src), and UPS kept the part it had not consumed yet in its carry. The transfer buffer already holds the whole source in one allocation that never moves, and stays alive, detached as the content, until the patch is finished - so that copy is a second ROM in memory, written once and read back. patch_stream_set_source() (and vcdiff_stream_set_source() under it) tells the stream where the source it is about to be fed lies. Chunks fed from there are read in place: no copy, no realloc as it grows, no memmove of the UPS carry. A chunk fed from anywhere else falls back to the old copy, starting with what came before it, so a caller that does not keep the source whole still gets the right answer. IPS keeps no source and is unchanged. content_file_pump_source() sets the transfer buffer as the source. A 32 MiB BPS hack patch: 32 MiB less heap during the load, and 143 -> 137 ms to apply. The patch_stream test feeds each comparison as a copy, in place, or in place then from another buffer (freed at once, so ASan catches a stream still reading it), and checks with glibc's heap counters that a 4 MiB BPS source and the xdelta source are no longer copied. Dropping the bytes the fallback copies, restarting the UPS carry at 0, reading the wrong byte in place, or ignoring the setter each fail it. |
||
|
|
bd7f70065e |
samples/tasks/patch_stream: tiny feeds for small sources only
Every trial fed its source a byte, two and three at a time, sources of up to 8 KiB among them: some two hundred million feed calls under ASan, most of the test's time, reaching no record boundary the seven-byte feed does not. Those three feed sizes now go over sources of up to 1025 bytes, past the 1024-byte edges; larger ones keep the 7, 64, 1024 and whole-buffer feeds. Measured on two CPUs: 103 s before, 54 s after; 238510 comparisons of the 280000, all matching. |
||
|
|
0466e44485 |
samples/tasks/patch_stream: build at -O1, and one scratch buffer
The test ran under ASan + UBSan at -O0, the slowest step of samples-tasks. -O1, as the other samples build, keeps every check and runs faster; and the 2 MiB buffer each of the 20000 trials built its patches in is allocated once, as each comparison reads its own exact-size copy of the patch, where an overread is caught. Measured on two CPUs: 138 s before, 103 s after, the same report. |
||
|
|
21635dd171 |
samples: build with -g1 instead of -g
The sample and test builds used full debug info. -g1 keeps what sanitizer reports and backtraces need - function names, files and line numbers - and drops the variable and type information nobody reads in CI. Objects get about 30% smaller (rh265.o under ASan+UBSan: 5.1MB to 3.7MB), so the ccache caches hold more of each job's build and are quicker to restore and save. |
||
|
|
afbad7a104 |
samples: shared sources build into each sample's own obj/
Samples compiled shared tree sources in place, so whichever sample ran next linked those objects with the flags they were built with, as unity_identity did with mic_handshake's ASan objects. Every sample that builds a source from outside its directory now includes sample_objdir.mk and maps those objects under its own obj/; its own objects stay where they were. The rule for the mapped objects has a slash in its target, so make 3.81 does not strip the directory. tools/sample_objdir_check.py dry-runs every sample Makefile and fails on an object written outside the sample; the Samples-Objdir-Check workflow runs it. |
||
|
|
e152c55d72 |
encodings/crc32: fix a data race in table init, enable slicing-by-8 on every target, add a PCLMULQDQ fold (#19278)
* encodings/crc32: make slicing-by-8 const, universal and thread-safe
The slicing-by-8 tables were built at first call into a mutable .bss
array behind an unsynchronised "ready" flag. Two threads entering
encoding_crc32() concurrently both run the generator; the flag is
stored with a plain write after the table stores, so nothing orders
the publication against the payload on a weakly ordered target. A
reader can observe ready == 1 and then load a half-written entry,
which yields a silently wrong checksum rather than a crash.
ThreadSanitizer over eight threads released from a barrier reports
nine distinct races on the old code, including reads of the 8 KiB
table concurrent with its construction.
Precompute the tables instead and emit them const from a generated
header. They land in .rodata, cost nothing to construct, are shared
between processes and cannot race.
The fast path was also gated on x86 alone. Every other target -- all
ARM builds that do not pass +crc, PPC, MIPS, RISC-V -- fell through to
the byte-at-a-time loop. Only the assumption of little-endian word
loads actually tied the loop to x86, so byte-swap under MSB_FIRST and
let everyone use it.
While here: load through memcpy rather than casting uint8_t* to
uint32_t*, drop the const cast in the ARM path, and decrement in the
body of the byte tails rather than in the loop test, which wrapped len
to SIZE_MAX on the final iteration. That last one is defined behaviour
but trips -fsanitize=unsigned-integer-overflow.
Measured on Xeon @2.8GHz, 256 KiB buffer, gcc 13 -O2:
byte-at-a-time (what non-x86 got) 0.39 GB/s
slicing-by-8, .bss + ready flag 1.94 GB/s
slicing-by-8, const .rodata 2.10 GB/s
So ~5.4x for every non-x86 target and ~8% on x86 from dropping the
per-call flag test and letting the tables be constant.
Verified bit-exact against a bitwise reference for lengths 0-1200 at
all 16 alignments, streaming splits, non-zero seeds and 1 MiB blocks,
on x86-64, i386, aarch64 with and without +crc, and big-endian
powerpc under qemu. Clean under ASan, LSan, UBSan, MSan, TSan and
valgrind memcheck.
* encodings/crc32: add PCLMULQDQ folding path on x86
Slicing-by-8 issues one table load per input byte and accumulates all
eight lookups into a single register, so each 8-byte iteration is a
serial chain of eight dependent XORs behind an L1 load. Measured 1.47
cycles/byte, against a load-port floor of roughly 0.5. It is
latency-bound on that chain, not bound by cache misses: cachegrind
over a 256 KiB working set shows the 8 KiB table staying resident,
with D1 misses accounting for compulsory stream fill only. There are
no register spills in the loop on x86-64 or i386.
Carry-less multiply folding removes the problem rather than tuning it.
Four independent 128-bit accumulators consume 64 bytes per iteration
with no loop-carried scalar dependency, and data references drop from
1.25 per byte to 0.064 -- a factor of 19.6.
Dispatch is at runtime on the CPUID PCLMULQDQ bit, so generic x86
builds pick it up without raising the baseline ISA; pre-Westmere and
pre-Bulldozer parts keep the table path. Buffers under 64 bytes go to
the table path too, since below four accumulators the reduction tail
costs more than it saves.
The detection result is cached in a single retro_atomic_int_t holding
the whole tri-state rather than a plain flag guarding a separate
payload, so concurrent probers merely store the same value and there
is no unsynchronised write to order against. Same acquire/release
idiom as cpu_features_get().
The folding constants are all K(n) = bit_reflect_32(x^n mod P(x)) << 1
and are derived rather than copied; the derivation is in the comment
above them and reproduces the published values.
Xeon @2.8GHz, gcc 13 -O2, GB/s:
64B 4K 32K 256K 4M 64M
before 2.41 1.91 1.93 1.94 2.01 1.99
after 7.26 19.69 20.80 20.82 19.73 7.43
zlib 1.3 0.51 3.11 3.22 3.31 3.33 3.10
10.8x over the previous code and 6.5x over zlib in cache; the 64 MiB
column is DRAM-bandwidth bound on this box, not compute bound.
Callers that benefit most are the per-frame core-memory hash in
netplay_frontend.c, content scanning through interface_stream.c, the
zip/7z/zstd archive CRCs and CHD hunk verification in rchd.c.
Verified bit-exact against a bitwise reference and against zlib
crc32() for lengths 0-1200 at all 16 alignments, streaming splits,
non-zero seeds, 1 MiB blocks and a 5 MiB single shot. Clean under
ASan, LSan, UBSan, MSan, TSan and valgrind memcheck, including a
guard-page harness that hashes 901 buffers ending flush against an
unmapped page to prove the 16-byte SIMD loads never overread.
* samples/encodings: add a crc32 regression test
encoding_crc32() now selects between three implementations -- a
PCLMULQDQ fold chosen at runtime on x86, the ARMv8 CRC32 instructions
chosen at compile time, and a portable slicing-by-8 table -- and
nearly every way that can go wrong is silent. A wrong checksum is
still a plausible 32-bit number. Nothing pinned the value from
outside the implementation, so add something that does.
Seven sections. Values are checked against a bit-by-bit reference
built straight from the polynomial: no tables, no word loads, and
endian-agnostic, which the slicing-by-8 loop is not. Twelve frozen
constants (the published CRC-32/ISO-HDLC check value plus zlib
outputs) sit alongside it so that a broken reference cannot cancel out
a broken implementation.
The sweep hashes lengths 0-1200 at all 16 alignments with each buffer
allocated at exactly the size read, so the last valid byte abuts an
ASan redzone and a SIMD load that runs one iteration too far is an
error rather than a silent success. The guard-page section repeats
that check without a sanitizer, hashing 901 buffers that end flush
against a PROT_NONE page, which is what gives the qemu lanes overread
coverage. Cross-path agreement hashes the same megabyte one-shot and
in chains of 1 to 4096 bytes, forcing the fold path and the table path
to agree with each other rather than only with the reference.
Verified by injecting eleven regressions and confirming each is
caught: fold-tail overread, dispatch threshold, three separate
constant bit-flips, both complements, a transposed slice index, a
dropped seed on the SIMD path, the atomic feature cache reverted to a
plain int, and the MSB_FIRST byteswap removed. That last one is
caught only on big-endian -- on x86 the swap compiles away and every
other check still passes -- which is why powerpc is in the cross-arch
matrix.
Two things about the concurrency section are load-bearing and are
commented as such in the source. It runs first, because any section
ahead of it warms the feature-detection cache on the main thread and
leaves the workers reading an already published value. And its
workers make exactly one dispatching call each: ThreadSanitizer keeps
only a few shadow entries per address, so a one-write/many-read lazy
init gets crowded out of that history by further traffic. Measured on
this test, eight threads making one call each report the race in 30
runs out of 30; the same eight threads making forty calls each report
it in none, and raising the thread count makes it worse rather than
better. The sustained-load checks therefore live in a separate
section that runs afterwards.
Registered in the samples workflow's executed-target list, with a
Clang/ThreadSanitizer step and a cross-arch job covering aarch64 with
and without +crc, armv7, and big-endian powerpc under qemu-user.
Stock aarch64 gcc does not define __ARM_FEATURE_CRC32, so those two
aarch64 entries really do exercise different code: the +crc object
contains three CRC32 instructions, the default object none.
Runs in 0.3s natively, 0.4s under ASan. Clean under ASan, LSan,
UBSan, MSan, TSan and valgrind memcheck, with GCC and Clang, at
-m32 and -m64, and with HAVE_THREADS=0.
* samples/encodings: run the crc32 TSan step in the job that checks out
The step was inserted immediately before the first step of
retro-atomic-cross rather than after the last step of samples, so it
ran as that job's opening step -- before its checkout -- and failed
with
An error occurred trying to start process '/usr/bin/bash' with
working directory '.../libretro-common/samples/encodings/crc32'.
No such file or directory
The directory was never missing; nothing had been cloned yet. Move it
to the end of the samples job, which has already checked out and
installed clang by that point.
* encodings/crc32: gate the PCLMULQDQ path on compiler support too
The path was selected on architecture alone. Being on x86 says the CPU
might have PCLMULQDQ -- which is why the runtime CPUID check exists --
but it says nothing about whether the toolchain can compile
_mm_clmulepi64_si128 or honour __attribute__((target("pclmul"))). A
toolchain defining __i386__ without wmmintrin.h would fail to build.
Test __has_attribute(target) and __has_include(<wmmintrin.h>) rather
than a compiler version, with a !__clang__ guarded GCC >= 4.4 fallback
for compilers too old to have those. The negation matters: Clang
reports __GNUC__ 4, __GNUC_MINOR__ 2 permanently, so a bare ">= 4.4"
predicate rejects it, and Clang is the toolchain on every Apple
platform, the Android NDK since r18, Emscripten and PS4-ORBIS. Getting
that backwards would silently drop the fold on all of them while still
compiling and passing every test.
Verified the fold is still emitted, not just that the file builds:
pclmulqdq instruction counts of 33 (gcc), 41 (clang), 30 (gcc -m32),
47 (clang -m32) and 30 (gcc -m32 -march=i686); zero on aarch64. The
i686 baseline case matters because the target attribute has to pull
SSE2 into that one function without raising it for the translation
unit.
* features/cpu: drop the VFS dependency, add RETRO_SIMD_PCLMUL, use it for crc32
Three things that only make sense together.
features_cpu.c read /proc/cpuinfo and the sysfs CPU lists through
filestream_*, which pulls streams/file_stream.c, vfs/vfs_implementation.c
and compat/compat_strl.c behind it. Nothing about a kernel pseudo-file
wants to be interceptable by a user-supplied VFS, so use stdio. Three
call sites: the ARM feature-name scan, the sysfs cpulist reader, and the
Linux model-name lookup. cpulist_read_from() now reads into a fixed
buffer rather than sizing from the file, because sysfs reports a page
for files that contain "0-7\n". After this the translation unit's only
undefined symbols are libc ones -- verified on x86-64, aarch64, armv7
and powerpc.
That matters because encoding_crc32() wants to ask whether the CPU has
PCLMULQDQ, and a leaf utility should not acquire a transitive
dependency on file I/O to find out. With the VFS gone it costs
features_cpu.c and compat_strl.c, both leaves.
There was no bit to ask for. RETRO_SIMD_AES is CPUID.1:ECX[25];
PCLMULQDQ is ECX[1]. They shipped together on most parts but
hypervisors mask them independently and some early Westmere SKUs had
AES fused off, so one cannot stand in for the other. Add
RETRO_SIMD_PCLMUL at bit 24 and probe it properly.
Darwin needs its own path. x86_cpuid() is compiled under
"CPU_X86 && !__MACH__", so Intel macOS never reaches the CPUID branch
and relies on sysctl -- and Apple publishes no hw.optional key for
PCLMULQDQ. Read machdep.cpu.features instead, which is the leaf-1
feature-name list, matching on whole space-separated tokens so a
future "PCLMULQDQ2" cannot false-positive a plain strstr().
encoding_crc32() then drops its private CPUID and its own atomic cache
in favour of cpu_features_get(), which already probes once behind an
acquire/release pair. Note the probe needs no OSXSAVE/XCR0 dance:
PCLMULQDQ is an XMM instruction, so CR4.OSFXSR is the only OS
requirement and that is universal. Only AVX-width state needs xgetbv.
crc32_test passes under gcc, clang, -m32, ASan+UBSan, TSan and MSan
natively, and on aarch64, aarch64+crc, armv7 and big-endian powerpc
under qemu-user -- the cross lanes now linking features_cpu.c with no
architecture conditional, because there is nothing left to
conditionalise.
* features/cpu: stop treating CPUID leaf numbers and results as signed
UndefinedBehaviorSanitizer's implicit-conversion check fires twice in
cpu_features_probe() and cpu_features_get_model_name():
features_cpu.c:940:14: implicit conversion from type 'unsigned int'
of value 2147483648 to type 'int' changed the value to -2147483648
x86_cpuid() took its leaf number as int, so every extended-leaf call
site -- 0x80000000, 0x80000001, 0x80000002+i -- passed a value with
the top bit set through a signed parameter. Widen it to uint32_t.
That in turn exposed the reverse conversion: flags[] is int32_t but
holds raw register contents, and leaf 0x80000000 legitimately returns
a maximum-leaf value above INT32_MAX, so assigning flags[0] to the
unsigned max_flag converted implicitly. Spell those two out with a
cast rather than changing the array type, which would break source
compatibility for anything outside the tree calling x86_cpuid().
Neither was a live defect -- the values round-trip on every ABI we
build for -- but they made the file impossible to run under
-fsanitize=implicit-conversion, which matters now that the PCLMULQDQ
probe lives here.
Predates the crc32 work; found while sanitizing that series.
Clean afterwards under gcc ASan+LeakSan+UBSan, clang MSan, clang
TSan, clang UBSan+integer+implicit-conversion+unsigned-integer-overflow,
and valgrind memcheck, exercising cpu_features_get(),
cpu_features_get_core_amount() and cpu_features_get_model_name() from
eight threads. Feature detection unchanged: same bitmask, PCLMULQDQ
still reported.
* samples, libretro-db: link features_cpu.c wherever encoding_crc32.c is linked
encoding_crc32() now asks cpu_features_get() whether the CPU has
PCLMULQDQ, so encoding_crc32.o carries an undefined reference to it.
Every source list that already linked encoding_crc32.c needs
features_cpu.c beside it, and features_cpu.c uses strlcpy, so those
lists need compat_strl.c too. Seven sample builds and libretro-db
itself were failing to link:
libretro-db/samples/libretrodb
libretro-db/samples/rmsgpack
libretro-common/samples/streams/rzip
libretro-common/samples/formats/png
libretro-common/samples/formats/json
libretro-common/samples/file/archive_file
samples/tasks/patch_stream
Done per source list rather than per file. png/Makefile has four
independent lists and only two of them already carried compat_strl.c;
adding a duplicate object to a link line is a multiple-definition
error, so each list is checked on its own.
Verified by building every Makefile under a samples/ directory
repo-wide with SANITIZER=address,undefined -- 50 of 51 clean, the
lone failure being deps/SPIRV-Cross/samples/cpp, which wants glm
headers and is untouched by this series.
* features/cpu: elect a single writer for the cached feature mask
cpu_features_get() publishes the mask with a release store and reads it
with an acquire load, which orders publisher against consumer. What it
does not order is publisher against publisher: if several threads reach
the probe before any has published, they all write cpu_features_cache
with plain stores. The comment there argued that was fine because the
probe is idempotent and they store the same bits, but same value or
not, concurrent unordered writes to one object are a data race:
WARNING: ThreadSanitizer: data race
Write of size 8 ... in cpu_features_get
Previous write of size 8 ... in cpu_features_get
Location is global 'cpu_features_get.cpu_features_cache'
Elect exactly one writer with an atomic increment. Threads that lose
return the value they computed themselves, which is identical, so
nobody spins and nobody writes memory another thread is writing.
retro_atomic_fetch_add_int() returns the previous value in all seven
backends.
Latent until something called cpu_features_get() from several threads
at once during startup; crc32_test's first-touch storm is the first
thing in the tree that does. Reproduced deliberately by widening the
probe with a 30ms sleep and calling from eight barrier-released
threads: one race before this change, none after. The window is hard
to hit on a single-core machine, which is why local runs passed while
the CI runner failed.
* encodings/crc32: dispatch the ARMv8 CRC32 path at runtime
The ARM path was gated on __ARM_FEATURE_CRC32 alone, which the compiler
only defines when the target baseline already includes the instructions.
In tree that is arm64 macOS, where Clang derives it from the default
-mcpu, and Makefile.libnx, which passes +crc explicitly. Everything else
on ARM ran slicing-by-8: Android arm64, Linux ARM, Raspberry Pi, iOS and
tvOS, on hardware that has had the instructions since ARMv8.1.
Compile the path in whenever the toolchain can build it into a single
function without raising the ISA for the translation unit, and choose
between it and the table at runtime, the way the x86 side chooses on
CPUID. Measured on an Apple M1, 256 KiB buffer: 1.44 GB/s for
slicing-by-8 against 3.38 for the instructions, and the target-attribute
build is indistinguishable from the baseline build at 3.38 vs 3.38, so
nothing is lost by not raising the baseline.
Raising it instead is not an option on Apple: iOS deployment targets in
pkg/apple reach back to 6.0 and Clang reports CRC32 as absent through
A10, so an unconditional +crc would be an illegal instruction on those
devices. Runtime detection is the only route that covers them.
Detection goes through a new RETRO_SIMD_CRC32 bit rather than a second
private probe: hw.optional.armv8_crc32 on Darwin with the FEAT_CRC32
spelling as a fallback, and the existing check_arm_cpu_feature() reader
on Linux, where aarch64 lists it as "crc32". Both keys confirmed present
and answering on an M1.
aarch64 with no baseline flags now emits the same three crc32
instructions as a +crc build and takes them at runtime. Verified on
aarch64 with and without +crc, armv7, big-endian powerpc and x86-64,
which correctly reports the bit absent and is otherwise untouched. Every
sample in the tree still builds.
* encodings/crc32: pick the ARM intrinsics header by compiler, not by _M_ARM64
CLANGARM64 on windows-11-arm fails to build:
encoding_crc32.c:286:10: fatal error: 'arm64_neon.h' file not found
arm64_neon.h is an MSVC header. Selecting it on _M_ARM64 alone breaks
any Clang that defines that macro for Windows-on-ARM compatibility --
MSYS2's CLANGARM64 does, and it ships arm_acle.h and no arm64_neon.h.
Clang targeting aarch64-pc-windows-msvc defines it too, while the
mingw32 triple does not, so the macro says nothing useful about which
header exists. Key on _MSC_VER && !__clang__ instead.
The bad include predates this series, but nothing reached it until the
runtime ARM dispatch started compiling the block on targets whose
baseline lacks __ARM_FEATURE_CRC32, which is exactly this one.
Reproduced by forcing -D_M_ARM64 on an aarch64 Clang: the same fatal
error before, clean after. Rechecked across gcc and clang on x86-64,
aarch64 gcc with and without +crc, aarch64 clang plain and with
-D_M_ARM64, armv7 and big-endian powerpc.
---------
Co-authored-by: libretroadmin <>
|
||
|
|
151b954951 |
samples: compare refusals, not just output, in the patch_stream suite
Two real bugs went through this suite untouched - the UPS and the BPS streaming appliers both accepted patches built for a different revision of the content, where the whole-buffer appliers refuse. Neither was found here. Both were found by applying a patch to a real ROM through a real build. The reason is structural, not a missing case. The oracle was a copy of the whole-buffer logic with the checksum validation deliberately left out, on the reasoning that byte-identical output was what the streaming appliers owed. So the suite compared bytes, and on a patch whose checksums are correct - which is everything the generator makes - both sides agreed on the bytes while disagreeing on whether the patch should have been applied at all. No number of additional generated patches would have closed that, because the disagreement is invisible to a byte comparison. Put the validation back in the oracle and compare verdicts. Then add the cases that need it: a valid patch applied to a source one byte different from the one it was built against, which is the ordinary way patching goes wrong and precisely what the formats carry a source checksum to catch. Four hundred of them, UPS and BPS, where the only correct answer is that both appliers refuse. Checked against the bugs it exists for: with the checksum validation removed from the appliers again, this section reports "oracle refused, streamed accepted" and the suite fails. With it in place everything passes, including the 300000 output comparisons, clean under address and undefined-behaviour sanitizers. |
||
|
|
ba610e33b6 |
tasks: validate the UPS checksums when streaming
build showed the streaming applier accepting it and handing the core silently corrupt content, where the whole-buffer applier refuses the same patch and the content runs unpatched. UPS carries a checksum for the patch, for the source and for the target, and ups_apply_patch checks all three - refusing on any mismatch, with an allowance for a patch applied in reverse, where the two content checksums swap. The streaming applier checked none of them. It decoded the diff stream, produced a target, and returned it. For a patch that is intact the result is identical, which is why nothing noticed; for a damaged one the caller's careful fallback - keep the unpatched buffer when a patch is rejected - never fires, and the corrupt target is adopted instead. Track the same three checksums and refuse exactly what the whole-buffer applier refuses. The source checksum accumulates as chunks are fed, the target's is taken once the target is complete, and the patch's is over the patch minus its own trailing four bytes. The differential suite could not have caught this. It compares output bytes for patches its generator builds with correct checksums, and its oracle deliberately omits the validation verdicts - so both sides agreed on the bytes while disagreeing on whether the patch should have been applied at all. That is the gap: a contract stated as "whatever the whole-buffer applier does, the streaming one does too" was only ever enforced for the output, never for the refusal. BPS was tested the same way and behaves correctly, but for a weaker reason than its checksums: damage to its instruction stream trips the structural bounds first. Worth knowing that its correct behaviour here is not evidence its checksums are being verified either. Verified: a valid .ups still applies to the real ROM with the expected CRC, a damaged one is now refused by both appliers and the content runs with the source ROM's own CRC and its internal checksum intact. The differential suite is unchanged at 300000 comparisons. |
||
|
|
dd19019453 |
tasks: record what a damaged patch actually does, per format
Running the streaming appliers against a real ROM through a real build turned up a claim of mine that was wrong. The commit that introduced streaming asserted that a deliberately corrupt patch would leave the content unpatched, and offered a truncated .ips as the check. It does not, and the check was the one format where the claim does not hold. BPS and UPS carry checksums, and xdelta is structured enough that damage almost always violates it, so for those three a damaged patch is refused: finish() fails, the caller keeps the unpatched buffer, and the content runs unpatched. That was confirmed end to end - a bit-flipped and a truncated .xdelta each had the streamed attempt refuse, the whole-buffer pass refuse after it, and the content load with the source ROM's own checksum. IPS has no checksum, no length and no framing beyond a terminator, so a truncated one is a shorter valid one as far as the format is concerned. Both appliers apply its intact leading records and produce a partially patched target. That is not a regression and not a decision taken in the streaming appliers: the whole-buffer applier has always done it, and the two were confirmed to produce byte-identical output for exactly this input. It is a property of the format. Write both down - in the header, where a caller decides what to do about a failed patch, and in the test, which deliberately does not assert refusal because the contract that holds for every format is the weaker one: whatever the whole-buffer applier does with a patch, the streaming one does too. No functional change. |
||
|
|
5163bccf9f |
samples: cover the xdelta streaming path
Adding .xdelta to the streaming appliers left this suite testing three of the four formats. The gap is thin - the format is a shell delegating to the VCDIFF decoder - but thin wiring is exactly what gets connected wrongly and never noticed, since a format that silently fell back to the whole-buffer pass would still produce correct content. Cover it the way its shape allows. The decoder itself is verified against the reference implementation in samples/encodings/vcdiff, so what is checked here is the delegation: feeding through this module's own API, in chunks, must reach that decoder and return what decoding the whole buffer at once returns. Five chunk sizes from one byte up, and a feed that stops short, which must be refused rather than answered with a target built from a partial source. The vector is a golden patch from the reference encoder rather than a generated one, because there is no xdelta encoder in the tree to generate with - the same reason the vcdiff sample carries golden vectors. It exercises an edited span, a shifted region and a run, so the patch carries some instruction variety rather than one flat copy. The whole suite still reports 300000 comparisons for the other three formats with zero mismatches, and is clean under address,undefined. |
||
|
|
50bc68358d |
tasks: move the patch_stream test into samples
samples/ is where the tree's standalone regression tests live, each with its own Makefile - samples/tasks/bps_patch is the directly analogous case, covering the whole-buffer .bps applier this streaming one has to reproduce. Move the differential harness there rather than leaving it in a one-off tasks/test directory, and give it the same Makefile shape, including the SANITIZER= hook. The oracle's formatting is cleaned up in the process: it arrived as compressed prototype code and built with -Wmisleading-indentation warnings. It is the reference the streaming appliers are checked against, so it should read as such. No change to what is tested: 300000 comparisons (280000 IPS/UPS, 20000 BPS), zero mismatches, clean under address,undefined via the Makefile's sanitizer path. |