An archive whose table sits at its end (a PBG4 .dat) reads its first
bytes and its last before the title. --rest-paths 'PATH HEAD tail' puts
the file's first HEAD MiB and its last, partial chunk in the bundle, and
the kernel holds both as fetched (tmpfs.rs seed_head), told apart from a
plain head by the copy's length.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
eject-site.mjs --rest-paths takes "PATH HEAD" lines (HEAD in MiB): the
file goes to the rest bundle whole, and its first HEAD MiB stay in the
bundle at its first-read place (site-bundle.mjs placeHeads), named in
lazy.json as the entry's sixth field. The page adds the lazy file over
the bundled head (gex.mjs), and the kernel takes a bundled copy that is
shorter than the lazy file and a whole number of chunks long as the
file's first chunks, already fetched (tmpfs.rs seed_head): what the boot
reads of a game's music, read at process start, comes from the bundle
and the rest bundle is not asked for it.
Tests: tests/eject/site-rest-heads.mjs; gex-kernel
a_bundled_head_seeds_a_lazy_file.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
Every Worker built an export wrapper, a name and a property for each
function a native wasm32 library exports, and the loader then put most of
them in the function table by name (`X`). KDE's boot set exports about
150,000 functions and imports one in eight by name: in plasmashell's
Worker those wrappers, names and export dictionaries were about 28 of its
40 MiB of JS heap.
The page now rewrites each such module before compiling it
(`packExports`, its own module key): the function exports not called by
name and the constant globals go, and the `X` slots are filled by added
element segments. An import of a dropped export comes from its slot or
value, which the binding list's `E` line now carries. A module of another
shape (fpcast thunks, more than one table, a lowered 64-bit one) is
compiled as it was. GEX_LOG=nodlpack keeps the old way, GEX_LOG=dlpack
says what was packed.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
The eject site now keeps the lazy files in rest.bin (`rest.bin#OFFSET`,
read by Range through rest.bin.json's frames), not one store file each:
the split check looks there, and a lazy read is a store file's request or
a Range of rest.bin.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
The wine32 bundle's files go in the order the recorded run first read
them, so its download decodes in the order the start wants them; an
identical file goes with its first-read twin. --rest-paths FILE moves
read files to the rest bundle (a game's music, read by offset). The rest
options take values in the argument filter.
tests/eject/site-pack-order.mjs checks the order.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
Comments named the test harness gex is developed with (its gate and bench
runner) and one still said a site's kind is w3; they say what they mean
without it.
No key inputs. wine32-dist.sh is in TOOLS_KEY and build-mesa-wasm.sh in
Mesa's stamp: the dist and Mesa rebuild.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
- WINE32_KUSER (gex_abi), GEX_WINE32_KUSER (the provider header), the
`:wine32kuser` unit tag; the eject kind "wine32" and the recording format
"gex-wine32-session" (the old "w3" and "gex-w3-session" still read).
- The page mounts 32-bit Wine's dist at /wine32 (tools/wine-wasm/run.sh,
page-run.mjs, units.sh; a recording's /w3 paths still map) and reports
"== wine32 page:".
- Comments no longer point at docs that are not in the repository; the
eject ones point at crates/gex-eject/README.md.
Changes unit keys (gex-abi, gex-cgen's header and tag).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
Changes unit keys; rebuild inputs: the gexcc install.
Comments only, plus code the removed NT layer left behind: the session
manifest's `nt` record (NtSession, NtImage), the native shim's unused
profile::note_pe, and gex-kernel's gex-gl dependency. "The NT layer's
split" is now "the 2 GiB split"; W3 and W64 in prose are 32-bit and
64-bit Wine. Keyed inputs (cgen providers, Wine dist, guest/gl, web/mesa,
the Xwayland crates, the mono provider) are left for a separate commit.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
Changes unit keys; rebuild inputs: the Mesa key, the Wine dist keys.
32-bit Wine (Wine's own Unix side as wasm32) and 64-bit native Wine run
Windows programs; the NT layer, gex's own Windows personality, goes.
Deleted: crates/gex-kernel/src/nt, the native and browser shims' nt.rs,
gexc's NT run path (nt_run.rs, --wine-prefix, cgen-prefix), gex-eject's
PE projects (pe.rs) and gex-rt, whose only job was running them,
tests/nt and tests/providers, and the tools/nt-gen-* generators with
their lists. tests/nt's programs move to the dev repo, which the Wine
legs already run them from.
Kept, because 32-bit Wine uses them: the trampoline unit and its
providers, aot_only, web/nt.html and nt-page.js (the Wine page),
tools/nt-stage-wine.sh (the prefix the wine32 dist stages from, now
without its unit build), and in the browser shim NATIVE_LEFT, the
in-place serve of an AOT-only run and the process end a provider asks
for, now in lib.rs. gex_provider_alloc takes Linux pages, and the
nt_qpc import returns 0: it binds only to a PE ntdll. --no-cgen is
accepted and does nothing. web/test/browser-harness.mjs is the old
tests/nt harness, for the Wine page's runner.
pewasm is gone too: the PE loader comparison test and its label
helper, and every comment that measured gex against it.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
No key inputs (unit keys); rebuild inputs: the Wine dist keys.
wine32/prefix.sh ran tests/nt's prebuilt childcon.exe for wineboot -u. Its
source moves beside prefix.sh as wine32/launch.c (a command is now required),
and prefix.sh builds it with clang's i686-w64-mingw32 target
(WINE32_MINGW_CC), so the prefix build needs nothing from tests/nt. The
launcher's source is part of the tools key.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
No key inputs (unit keys); rebuild inputs: the Wine dist keys, the gexcc install.
Paths, env names (W3_*, W64_*, GEX_W3_* become WINE32_*, WINE64_*,
GEX_WINE32_*), page switches and exports, identifiers and comments.
Unit-keyed and recovery-keyed names stay (W3_KUSER, GEX_W3_KUSER), so
unit keys are unchanged; the Wine dist keys change with the scripts
they hash.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
A dlopen'd module with thread-local data was refused. Now each thread
that instantiates it gets a block of its own: allocated from per-thread
chunks and set up with the module's __wasm_init_tls, natively and in
each browser Worker. A fork's child keeps the forking thread's blocks
(the link map carries which modules the program started with, whose
blocks libc's registry handles). A thread's chunks are unmapped when it
exits.
A thread already running when another one dlopens caught up with the
function table only when it called through a slot it had not filled,
and a slot holding the resolve stand-in traps on a type mismatch for any
signature but the stand-in's. Every dlopen now bumps a generation, and
a thread that has not seen the latest catches up at its next system
call's return, so a pointer handed over through the kernel (a futex, a
pipe) is in its table before it can call it.
That leaves a pointer handed over lock-free, with no system call on the
receiving side before the call. A trap there cannot be retried, since a
wasm32 program's frames are gone by then. gex now names the cause:
"called a function from a dlopen this thread had not caught up with".
GEX_DL_CHECKED_CALLS=1 (the page's dlchecked=1), off by default and
costing nothing when off, rewrites every wasm32 module of the program
so that an indirect call checks its index against the thread's
caught-up slot count and catches up on a miss
(gex_dylink::checked_calls).
tests/gexcc/dltls.sh: a startup library's and a dlopen'd library's
thread-locals in main, in a thread started before the dlopen and two
after, and in a fork's child. tests/gexcc/dlspin.sh: the lock-free
handoff, a known gap unchecked and passing checked.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
Tests, benches and experiment runners leave gex for gexdev, which runs
them against a gex checkout named by GEX_ROOT: tests/ (but tests/nt and
tests/providers, which go with the NT layer, tests/eject's soju pages, and
the diff-fuzz workspace members), bench/, tools/explorer,
tools/unity-dotnet/page, tools/gexcc/bench-codegen and
tools/wine-wasm/spike-{a,b,c,w3}.
What gex itself builds from there stays, in new homes: the GL probes and
gears that web/build-assets.sh and guest/gl/demo.sh build move to
guest/gl/probes, and the freestanding headers native-libs compiles
against move to tools/gexcc/item3/inc. mono/build.sh's em-smoke step runs
gexdev's smoke test through GEXDEV. NOTICE drops bench-codegen's entry.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_40f26e8988a74ea098d8bc80ff68287c
A writer thread bursts writes to one object while readers loop reading it:
a window's extra bytes (two readers, then eight at once, whose copies of
the object overlap), a queue's posted messages (GetQueueStatus and an
empty PeekMessage) and two attached threads' key state. A read begun
after a write was published must see it or a later one, never a mixed
copy. argv[1] sets the writes a case, argv[2] a mask of the cases. It
builds for x86-64 and for i686 (a 32-bit window value carries the write's
number and a check of it).
The page and runs without the door (every read a server request) take
2000 writes a case, 20000 otherwise. The SP page runs the queue and input
cases only: there a process whose threads read and wrote another thread's
process that kept a window for a few seconds never exits after its last
line (base Wine too; open, noted in nt-x64.sh). A watchdog fails a case that stops making progress, and ends
before the program does.
tests/w3/seqrace.sh runs it on W3: natively with and without the door,
and on the page.
GEX_SESSION_COPY_DELAY_US (debug, native): every 8th session copy of a
thread holds the bytes it read that long before writing them, as a copy
preempted midway would, so the race of copies into one process-wide
mirror shows in a short run.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
The single-thread build's threads are fibers on one thread: none runs while
a fork copies the memory, so there is nothing to hold. And one parked at a
safepoint is not out of guest code, so gex_fork_hold waited for it, with a
wait its unshared memory traps on ("Atomics.wait cannot be called in this
context"): a fork beside a spinning thread killed the forking thread, and a
W64 page under load lost processes that way.
tests/fork-snapshot runs its last case in the single-thread build too.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
What clang needs to start, and what it runs out of on a function, depend on
its build: the retry case steps up from 150 MiB to the first limit under
which the build succeeds, and runs when clang ran out there on big()'s or
flat()'s unit, saying which limit; otherwise it is skipped, and says so.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
The instruction limit is a cheap first cut, but clang's memory follows a
function's shape more than its size. Each clang now runs under RLIMIT_AS,
8 GiB by default (`gexc cgen-image --clang-mem-mb N` or
GEX_CGEN_CLANG_MEM_MB, 0 for none). When clang fails on memory under it,
the unit's largest function becomes a stub that leaves for tier-1 (its
trap-free callers stop being trap-free), one build-log line says which, and
the units build again; only the changed units recompile. The in-process
build drops the function the same way.
tests/cgen-max-insns gains a build under 300 MiB, where clang runs out on
both large functions' units and the retry leaves them to tier-1; the run
agrees with the host. A 42,593-instruction function that needs about 12 GB
is caught at 8 GiB and left to tier-1.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
clang's memory grows faster than a function's size, and unevenly: a C++
library's 40,000-instruction function took clang 67 GB. A function of more
guest instructions than the limit (default 30,000; `gexc cgen-image
--max-insns N` or GEX_CGEN_MAX_INSNS, 0 for none) now stays out of its unit
as a stub that leaves for tier-1, as one clang gave a shadow stack does,
with one build-log line naming it and its size. A stub is not trap-free,
so neither is it, nor any trap-free caller of it. In the in-process build
such a function is not taken at all. A limit other than the default is in
the unit key.
tests/cgen-max-insns: a function of calls and branches and a straight-line
trap-free one with its trap-free caller, each about 3,000 instructions; under
a limit of 1,000 both stay on tier-1, with no limit the unit takes them, and
the runs agree with the host.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
The page kept a lazy file's chunks under its URL and chunk number only. A
list that names files by path (not an eject site's hashed URLs) can change
a file under the same URL, and the next visit then read the kept chunks of
the old file mixed with the new: a restaged native wasm module failed to
load with "unexpected end-of-file".
The key now carries the list's size and mtime for the file
(lazy-cache.mjs chunkKey), and the prune drops a path's chunks of any other
version. Chunks kept under the old key are fetched once more.
tests/lazy-cache/changed-file.mjs runs the owner Worker over a cache kept
on disk: a file rewritten under its URL reads as its new bytes, the same
version reads from the cache alone, and the old unversioned key is shown
to give the stale chunks.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
The demo page's gears starts its program without app.js, so the SP leg
never saw what app.js hands the system Worker (its Mesa source, a
function: every SP run with GL from app.js failed in postMessage).
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
(cherry picked from commit f30acfd84057ebb5702c80cdcb6643b12123c9d2)
A pipeline's SIGPIPE ends a child on every `yes | head`, and the session
ends its processes with SIGTERM: neither is said unless asked for. Said by
default: SIGILL, SIGTRAP, SIGABRT, SIGBUS, SIGFPE, SIGSEGV, SIGSYS and
SIGKILL (gex never ends a process with SIGKILL itself), and any gex
limitation delivered as a signal.
tests/child-signal.sh checks both modes, with a SIGSEGV child and a child
another process kills, on a64 and x86.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
A forked child that a signal ended (a SIGPIPE, a SIGKILL, an abort no
crash handler catches) went with nothing on gex's stderr, and a gex
limitation delivered as a signal in a child was as quiet: only the top
process's exit was reported. Now one line names the pid, the signal and
the pc, as a shell would. SIGHUP, SIGINT and SIGTERM, which a session
ends its processes with, stay quiet.
tests/child-signal.sh: SIGPIPE, SIGABRT and SIGKILL children are named;
a SIGTERM child is not.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
A thread running guest code that makes no call stopped for a fork's copy
only when its safepoint countdown ran out on its own. gex_fork_hold now runs
each other thread's counter out, as a signal for it does, so its next
back-edge stops it. A thread that still isn't out of guest code after two
seconds (lost in a host call) is said in one line: that child's memory may
be torn.
tests/fork-snapshot runs three cases: one writer that never makes a
syscall; four such writers; and four writers, all but one making calls,
beside a thread that creates and joins threads all along, whose children
each start a thread of their own. Each case prints its mean ms per fork.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
The host copies a forking process's memory a region at a time
(#processImage) while its other threads run on, so a child's memory held
pieces from different moments: one thread's writes to a low page from
before the copy reached it, to a high page from after. glibc's child walks
its lists of thread stacks after fork, and found nodes as no thread ever
left them (a pointer into a library's code), one fork in six on a KDE page.
Now the other threads are held out of guest code while the memory is
copied, as Linux's fork copies it at once: gex_fork_hold, before the copy,
tells each to hold and waits until it is out of guest code (in a call, a
dispatch miss or a safepoint, or stopped for its Worker's run loop: it
parks at its return to guest code) or parked; gex_fork_release lets them
go. A thread that never gets there is given up on after two seconds, as an
execve's are, and a held one goes on by itself after ten.
tests/fork-snapshot: threads write B then A, A in .bss and B 64 MiB above,
while the main thread forks 20 times; every child must see A as B or B - 1.
Before, 10 forks in 10 had A thousands behind.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
SSE under MXCSR is now asked for by the program that needs it:
gex_ir::set_mxcsr_exact (the page's ?mxcsr=1, gex_set_mxcsr_exact;
GEX_MXCSR=1 natively), before anything is translated. Off, the lifter makes
no Expr::Mx and tier-1 and the function JIT import no `mx`: every tier's
output is what it was before SSE honoured MXCSR (tier-1's module and the
function tier's C byte for byte), and the keys are as they were. On, the
unit key names it (`:mxcsr`), as do tier-1's knobs and the page's
translation cache, so units with it and without it never mix.
It is the rounding control only: FTZ and DAZ are honoured in no tier, as
before, and the exact import is given the rounding control alone. A
program that runs with FTZ or DAZ set (much audio code) keeps the fast
path.
The function tier tests the rounding control once for a run of up to 32
operations in a block and emits the run twice: the operations' own C,
taken when it is to nearest, and their exact calls, out of line in
runtime/mx.h (one call an operation, however many lanes). A block that
writes MXCSR or makes a host call tests each operation. A test and an
inline slow path per operation took clang over 65 minutes and past 15 GB
on one function of 13,600 SSE operations; the runs take
it 41 s and 287 MB, against 13 s and 194 MB without MXCSR. Tier-1 and the
function JIT test each operation's rounding control and call the import.
Tests: gex-lift-x86 sse_is_under_mxcsr_only_when_asked (no Mx, no MXCSR
read without the flag; one per operation with it); gex-cgen
sse_operations_under_mxcsr_test_once_a_block (without it, no MXCSR in the
C; with it, one test a run, one out-of-line call an operation, at most
twice the C, and clang takes it); tests/x86-programs fp-modes under
GEX_MXCSR=1, its four rounding modes against the hardware on tier-1, the
function JIT, the function tier's C and the page.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
The persist worker snapshots about a second after a save (k_fs_saves).
When the probe's exit reaches the page later than that (a loaded host),
that snapshot stores the probe's file and the page's last one has nothing
left to store: "stored 0 file(s)" for a visit that wrote one. The page now
reports what every snapshot of the visit stored (app.js state
persistedWrote), the worker's own included. With the last snapshot held
back 3 s, the old check fails visits 1 and 2 as the loaded gate did; this
one passes.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
tests/wine64's unixctx case: a thread with known xmm6-xmm15 inside a Unix
call (ws2_32's gethostname), then in guest code, suspended 2000 times each
and read with CONTEXT_FLOATING_POINT; each phase ends with a
SetThreadContext changing xmm15, which the thread must see. tests/w3/
suspctx.sh: the same in guest code on W3, xmm0-xmm7, MXCSR and
CONTEXT_EXTENDED_REGISTERS, natively and in the page's runtime.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
A fork staged its child (`FORK`, its descriptors, cwd, program and signal
state) in statics every Worker of the process shares, claimed by the thread
index that made it. Two threads forking at once on their own Workers
replaced each other's words before the first Worker drained them, and the
first child was never started (its parent waited for good). A vfork child
running in its parent's Worker numbers its thread 0, so its fork could be
claimed by the process's first thread's Worker, which copied the memory
while the parent ran on: the grandchild started at the instruction after
its fork on the parent's later context (a SIGSEGV in its first block).
Each Worker now stages its own fork, which only it can claim.
tests/fork-staged: detached starts (a vfork child forking and exiting)
from two threads at once; every grandchild reports, its registers whole.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_c8c04c00da9a4525ab21b0b9f146c598
The call-out loops read the stack pointer at x86's place in the context
(`ctx::x(4)`) on every guest. On aarch64 that is x4, so a watched return
never matched the call-out waiting for it: the return ran on as a bypass,
the call-out never ended, and each later one opened inside it until the
depth limit. Their unwound check compared x4 too, and a call-out whose x4
held a large value left as unwound while its callee still ran. The shim
now says where the stack pointer is (`gex_ctx_sp`).
tests/callout-entry-return: a call-out returning to an entry of its unit
ends at its return each time.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_8b3490cccb4848b495dde1c5cdcacba0
A third argument opens the window fullscreen (SDL_WINDOW_FULLSCREEN, as
Half-Life does by default) or fullscreen-desktop; the scene then draws at
the drawable size, read every frame, into a texture big enough for any
size up to 2048. The centre pixel is read back again at frame 120, once a
fullscreen window has its size.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_8b3490cccb4848b495dde1c5cdcacba0
`__gexMemories` held each threaded process's memory weakly, and a page
leg reading a game's counter through it relied on something else on the
page's main thread keeping that memory alive: the fork record the page
now lets go of once the child's Worker has it. The memory was collected
while the game ran, and the leg read nothing.
The registry is now a test opt-in, `?memories=1` (or GEX_MEMORIES), read
once at load: with it a process's memory is held until the process ends;
without it nothing is registered or kept.
tests/spawn-release: a fork child with a thread counts while the page
reads its counter after collections (GEX_TEST_MEMORIES in run-node); the
counter climbs, the memory is let go at the child's exit, and nothing is
registered without the switch.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_8b3490cccb4848b495dde1c5cdcacba0
