Hard-learned lessons

Drawn from six co-op projects (STALKER, Fallout New Vegas, Dungeon Keeper 1, Ocarina of Time, Animal Crossing, Deus Ex). Each one wasted time or produced a false result at least once.

"No co-op exists" is usually wrong. Searches on the default branch of a repo missed a complete multiplayer fork that lived on another branch, and a "no working Deus Ex co-op" claim was wrong because an older mod existed. Before telling the owner nothing exists, search forks, branches, release pages, forums and archive.org, state what you searched, and be ready to correct yourself in the open. Finding the existing work early changed the plan from "build" to "revive and extend" every time.
Prove a fix before you claim it. A "corpse cap fixes the frame rate" claim shipped before any test, and an A/B run with one variable changed showed it did nothing. Every fix needs a before and after run that changes only that one thing, and the claim in the notes should quote the run.
Check the measuring instrument. A 60 FPS engine cap hid real drops across several runs; a desync log printed turn 0 for every mismatch; a -level flag silently fell back to single player so the run "passed" while testing nothing. At the start of each run assert the mode you think you are in, and compare frame time, not capped FPS.
A test that checks once is a flaky test. One false FAIL came from checking ten seconds after a trigger when the flag arrived a few seconds later. Poll with a timeout for everything, and leave generous slack for late answers under load (client probes arrived late when many processes ran).
Exit code zero is not a pass. A pipeline through tail hid failing builds, and a test runner printed "is not recognized as an internal or external command" while exiting cleanly. A suite counts only if its output proves tests ran; a missing executable is a FAIL, not a skip.
Put the test bot inside the game. Simulated mouse and keyboard are slow, fragile and take over the owner's PC. A bot in the game's own scripting or code (writing packets, calling script functions, a JSON-RPC bridge plugin) is deterministic, invisible, and can assert against real state. "Needs a human playtest" is a gap to close, not a step in the plan.
Test co-op with co-op. Single client against a simulated partner finds protocol bugs but not the real ones. Two real clients, comparing shared state (and a lockstep checksum where the engine has one), found the camera desync while possessing, the wrong campaign progress file, and the stats screen hang. Most real bugs were in transitions (map, briefing, level, stats, loss, retry, disconnect, late join), not in plain in-level play.
Mutation twins make tests trustworthy. Ship a deliberately broken variant with every feature and require its test to fail on it. Add self-checks too: two dumps of one save must diff equal, and a known change must show up in the diff exactly. A test that cannot fail proves nothing.
Never rebuild what you cannot rebuild; add a package next to it. The original Deus Ex co-op package could not be recompiled (the compiler asserts on a class without script text, and the native DLL has no source), so fixes went into a separate package of subclasses and mutator modules selected by config. The pattern avoids binary compatibility traps and keeps the original intact for comparison.
Single-instance checks come in layers. Renaming the named mutex was necessary but not sufficient: the second game copy exited silently because of a window-class lookup in the start-up code. Search for every instance check (mutexes, events, window lookups, shared document folders, Steam checks) and fix them inside your own process, not by closing handles in other processes.
Never touch the owner's install, config or saves. Copy to a test folder, use instance-local INIs and saves, back up before editing, and never set attributes on the owner's files. One experiment set an owner INI read-only and had to be repaired. Also never use reserved save names and never use an old save that is known to crash. MW2 (2009): a test launch without the Steam app id variables made Steam start the owner's real install, which rewrote its config and left a lock file behind; both were put back from a backup taken before the first run. Back up first, then guard every launch.
Never take desktop screenshots. A screen recorder captures whatever else is on the screen, including a browser or private apps. Use the game's own screenshot command, in-engine capture or logs, tell the owner if anything else was captured, and delete the frames. MW2 (2009): with no console or in-game screenshot reachable, copying the screen rectangle of the game window captured another app lying on top of it; even a topmost capture of the game window shows overlays (Discord) drawn into the game. Delete such shots, keep the rest local, and prefer logs. PrintWindow with PW_RENDERFULLCONTENT on the game's own window (Shadow of War, D3D11) returned the game's frame without the windows on top of it, but it would still include overlays drawn into the game, so the same rules apply. BLACK SOULS II (RGSS3): early bot screenshots copied the window's screen rectangle and caught Discord and another app on top; switching the runner to PrintWindow (flag 2) captured the game window cleanly.
Do not make the owner's PC unusable. Full-screen launches by test agents were the loudest complaint. Always windowed, minimized without activation, "always active" set so the game does not pause, and a game-slot lease so agents do not launch more than one at a time. Memory pressure from many launches caused the host to kill a suite mid-run; run fewer at a time.
Clean up after tests or you ship your test files. A test archive left in an add-on made the server checksum differ, and other players saw "different versions". Remove test files, delete screenshots, and compare the installed patch byte for byte with the shipped one before handing anything over.
Delete the log before each run. A boot test of many levels showed the next level failing with the previous level's error because the log still held it. Other logs only flush on clean exit or rarely; for data you must have, write your own file and flush it.
Find a hook that actually fires. In STALKER the obvious per-tick hook (the actor binder update) never ran on a dedicated server; an effect attached to an always-online restrictor did. In another game, client-side binders did not fire while server ones did. Probe each hook with a trivial log line on the exact setup (dedicated, listen, client) before building on it.
Textbook algorithms differ from the game's variant. The standard LZHUF frequency limit corrupted decompressed output after about 97 KB because the game used a different maximum. When a decoder works for small files and breaks later, suspect a constant, and round-trip test every archive tool (pack, extract, diff).
Do not trust claims in chat or by agents. A moderator's "new version" was the old files under an unchanged number; a forum "re-upload" was a repost; an agent said the host could not use computers when a test showed it could. Verify by hash, date, version string, or a test, and label what is inference.
Treat tokens and third-party data as radioactive. A Discord token was pasted into the chat once; that means rotate it immediately. Account tokens in third-party tools also break the platform's terms, so prefer admin exports, bot accounts or manual reading. Never copy handles, IDs, invite links or personal paths into public docs, and do not argue with a safety classifier that blocks bulk collection: hand the step to the owner or find an in-process alternative.
Never overclaim. Bot-versus-bot runs, a mock Steam, and fixes reasoned from code with no two-player test are all useful, but none is a human or two-account proof. Set the QA level honestly, write only what was actually tested, and keep the open items in your notes or GitHub issues so the next agent picks them up.
Several copies of a single-instance game for automated tests. Besides the named mutex, check WinMain for a FindWindow(own window class) early exit. Neutralize both inside your own process via IAT hooks, and launch the copies from a long-lived process so they are not killed along with a short-lived launcher's job object. Example: Fallout: New Vegas (fnv-coop).
A copied install is not enough when settings live in Documents. The Witcher 3 saves its settings from the main menu, so test flags like -width 960 -height 540 ended up in the owner's real settings file and its backup. Give every test instance its own user folder: hook SHGetKnownFolderPath(FOLDERID_Documents) and SHGetFolderPath(CSIDL_PERSONAL) in your own process, seed the sandbox with copies of the settings and the one save the test loads, and make the test plugin refuse to start without the sandbox.
Faking focus has side effects. To keep an alt-tabbed test instance running, The Witcher 3 needed GetForegroundWindow faked and WM_ACTIVATE/WM_ACTIVATEAPP/WM_KILLFOCUS filtered. A game that believes it is focused also clips the cursor, warps the mouse, steals focus and reads the keyboard, so in test mode neutralize ClipCursor, SetCursorPos, SetForegroundWindow, GetAsyncKeyState and raw-input INPUTSINK too, and mute the instance's own audio session.
Two modern DX12 instances can take down the GPU driver. Two Witcher 3 Remastered instances with ray tracing ended in a device-removed crash and a driver reset the owner could notice. Apply the game's own lowest preset to test instances (expand it from the game's options XML rather than guessing keys), turn ray tracing off, cap the frame rate, and check free VRAM first: desktop apps can hold most of the card.
First-run screens are not yours to answer. Account-link and telemetry consent screens appear on a fresh launch. Close them in test mode without signing in or recording a choice, so the owner still sees them and decides. System dialogs (for example a one-time "skill points reset" notice after loading an old save) do pause the game, so log and answer those.
Record test video from inside the game, never from the screen. Agents judge camera and animation work much better from video than from screenshots, but a screen-region capture of a test window on the owner's second monitor also recorded the owner's chat window lying on top of it. Capture frames in your own process instead: hook IDirect3DDevice9::Present (vtable slot 17), resolve the multisampled back buffer with StretchRect, copy it with GetRenderTargetData and write raw frames that ffmpeg turns into MP4 and contact sheets. Example: Clive Barker's Jericho (native/capture.c in jericho-coop).
A pawn pinned by per-frame writes can die, and a spawned puppet can end the mission. In Far Cry 2 the peer's avatar was created from the player's own archetype and moved by writing its physics body every tick at a fixed height. The pinned body took damage and both copies showed "Game over", although every entity check still said valid. Make a puppet invincible right after the spawn (the engine's own event, here CFCXCountersComponent_SetInvincible) and keep the writes per frame (about 140 per second) with smoothing; four writes per second looked like teleports and let gravity act between them.
A scan of a live game's memory must not be able to fault. Walking every committed region with direct reads crashed one Far Cry 2 copy when another thread released a page between the region query and the read. Copy in chunks with ReadProcessMemory (a failed read skips the chunk), and install a vectored exception handler that logs access violations, so a crash leaves a record. Many access violations at start-up are the OS or the game handling its own exceptions; log module and offset before you worry.
Find objects by class, not by a pointer chain. A breadth-first chain from the entity to the physics body (found in one session) broke in the next one, because it ran through lists whose order changes. The body was found in every session by its class (a static vtable) plus the nearest position to the entity. Also dry-run any loop that writes to memory: a helper named Dir hit the PowerShell alias, the test measured nothing and left its test value in 244 places because the restore step never ran. Print the first measurement and the first restore before looping.
Do not copy a huge mod archive for every test run. Each Shadow of COOP test packed a small test archive into a copy of the 4.2 GB mod add-on: 30 s to 2 min per run, and the copies filled the disk until runs failed to start. Loading the test files from elsewhere did not work: xrMPE mounts the add-on last, so a patches folder or loose files lose to it, and it reads only the first -addon=. What works: keep one permanent copy per test lane, and before each run cut it back to the end of the original archive data and append the new test archive. The build went from minutes to 3 to 4 seconds and a level boot check to about 70 s. Also run offline checks first (seconds): files shipped by two patches, broken escapes in script string paths ("misc\coop.ltx" in Lua reads \c as c), and config references to missing sections. Example: Shadow of COOP (tools/stalker_xrmpe, qa_build.py --lane).
Bound a function by .pdata, not by int3 padding. A script that disassembled each Crimson Desert message handler "until the first int3" ran straight into the next handler, because MSVC packed them without padding, and borrowed its calls. Two stubbed dev requests were reported as working, and the live test that followed failed for a reason the static pass had hidden. On x64 PE files read the exception directory (RUNTIME_FUNCTION begin/end) for exact bounds, and sanity-check that the next function's start is where your scan stopped. Example: Crimson Desert (games/crimson-desert/tools/re/classify2.py).
Leftover dev and cheat messages may be stubs in the retail build. Games that keep an MMO-style client/server split often still register requests like CompleteQuestByCheatReq or ForcePlayStageDevReq, but in Crimson Desert their handlers only parse the payload and return. Compare each candidate's calls after parsing with a request you know works (here KillCheatReq, used by cdmp) before designing on it, and include a stubbed request as the mutation run that must fail. A failure path that returns success (0) means "no answer" is not proof the request was rejected.
A Windows-only C++ mod can be built and smoke-tested on Linux. cdmp (MSVC, SEH, DX12 hooks) built unchanged apart from one clang strictness fix with clang-cl 19, lld-link and the Windows SDK/CRT from xwin in a container; the MS STL refuses Clang older than 19. Its unit tests ran under the Wine shipped in Proton, inside the Steam Linux Runtime (SteamLinuxRuntime_4/run -- <proton>/files/bin/wine test.exe; outside it the loader is missing). The game ran from a hardlinked test copy with its own prefix through proton waitforexitandrun, WINEDLLOVERRIDES="winmm=n,b" for the ASI loader, and HOME set to the one snap Steam uses. Do not wrap a game that has its own windowed setting in a Wine virtual desktop: the two sizes clashed and left a small window on a black screen. Set windowed mode in the test prefix's settings file instead. Example: Crimson Desert (games/crimson-desert/tools/).
Diff the whole world between host and client, at every checkpoint. Per-feature checks ("the client saw the spawn", "the hit landed") all passed in MDK2 HD while the two worlds quietly disagreed. Each game now dumps its world on command (players, puppets, synced enemies with hp, actors, items, doors), a script diffs host against client, and a sweep does that at all 96 checkpoints of the debug menu's list after a little walking and shooting. It found six desyncs in its first runs, among them items made on one side only and enemies with 0 hp on the client. Dump a few seconds after a load: the client finishes loading later, and early dumps show transient differences. Before fixing, confirm which side is wrong with a single-player run. Example: MDK2 HD (tools/sweep.sh, tools/statecheck.py in mdk2-coop).
Deleting is often deferred: a name lookup in the same frame finds the dying object. In MDK2 HD the engine's delete only marks a gob, which stays in the gob table and keeps its Lua global until the scene's cleanup. A level script deleted two enemies and made new ones with the same names; the client deleted its copies for the despawns and then bound the new ids to those same dying gobs, so both enemies vanished. Send despawns before spawns, and keep your own "deleted this frame" set that every lookup and liveness check honours. Look out too for sub-objects that carry their owner's name (spawned soldiers had two info-less parts named like them), which make names look duplicated. Example: MDK2 HD (gob_kill / gob_doomed in mdk2-coop bridge/game.c).
Wake the world by the same rule on every machine. The MDK2 HD room manager wakes the rooms around the camera; co-op added the rooms around each remote player's body. So the host woke rooms by its camera and the client woke them by the host's body. One step put the host's body, but not its camera, inside the next room, whose OnCreate then made seven items on the client only. Each machine now also wakes the rooms at its own player. The same rule applies to any activation, streaming or level-of-detail test that keys off the local camera. Example: MDK2 HD (bridge/rooms.c in mdk2-coop).
Give each parallel line of work its own worktree, game copies, ports and kill rule. Two people (or agents) worked on the same MDK2 HD mod in one working tree; one's half-finished header change broke the other's build, and a test kill step of taskkill /IM game.exe would have killed the other's eight test clients. What works: one git worktree and branch per line of work, separate game copies and UDP ports, a kill step that matches processes by exe path inside its own copies, and a heads-up to the others before each test run. Shared per-user folders (settings, saves) still collide, so keep test state out of them or accept it. Example: MDK2 HD (mdk_kill in mdk2-coop tools/mdk.sh).
A projectile's constructor may not launch it: capture the finished launch, and test flight on the peer. Some engines create a grenade or sniper round inactive (attached to a hand, no position or direction) and launch it later from an animation key, with aim spread and an arc applied then. A create-only hook sends an inert shot; hooking the constructor and the launcher can announce one shot twice. Hook the final launch step and send its transform, velocity, fuse and the translated owner and target; while replaying, suppress the constructor's own auto-launch and any shooter scripts. A replay log saying "ok" proved nothing in MDK2 HD: bouncing grenades take their velocity from a separate rigid body, so only the sender's grenade flew. Have the real peer retain the replayed object and sample its motion, assert exactly one launch per peer and no damage from the visual copy, and include a client-to-host-to-client case. Example: MDK2 HD (bridge/shots.c, tools/qa_shots.sh in mdk2-coop; launch investigation).
Put the pre-launch screen in the proxy DLL, inside the game's first window call. A launcher for host/join, connection and startup-only settings does not need a separate exe or a Steam launch option: the proxy DLL is loaded before WinMain, so hooking the exe's import of CreateWindowExA and running a modal window the first time it is called shows the launcher on the game's own thread, after the loader lock is released and before the game reads its config. Anything it writes (the mod's ini, the game's video line) applies to that start. Draw it with the game's own art read from the player's archive so nothing copyrighted ships, keep a plain fallback, skip it when the platform launched the game to join a session (Steam's +connect_lobby), and give it a hidden, command-file-driven mode so QA can screenshot and drive it without a desktop. Example: MDK2 HD (bridge/launcher.c, tools/qa_launcher.sh in mdk2-coop).
Merging parallel feature branches: read every pair for what they share, not just the conflicts. Six MDK2 HD branches, each tested alone, merged with only small textual conflicts, yet three things broke without a conflict. Two features defined the same global Lua function, and the last one loaded won without an error. A guard one branch had renamed was copied in its old form by another. A settings screen wrote keys another branch had replaced, resetting values on every start. Two more features read keys that were already taken (push to talk on the ping key). After each merge, list for every pair of features: the keys they read, the settings keys they write, the global names they define, the screen space they draw in and the window or input hooks they change. Then route keys through one rebindable table. Check also that the test run actually loads the new build: here every game had quietly loaded the system DLL instead of the proxy. Example: MDK2 HD (mdk2-coop merges 2026-10-22, games/mdk2-hd/NOTES.md).
If every machine already runs the same scripted sequence, sync the start, the skip and the end, not the sequence. MDK2 HD plays a cutscene as a task list on a gob, and every game runs its own copy, so mirroring frames would be work for nothing. Pause each copy when it starts, let the host say go once the others are ready, make skipping a vote the host tallies (a majority, with a countdown, so one idle player cannot hold the rest), and wrap the sequence's completion callback so the next level waits until everyone has reached the end. For the parts that are interactive (a skydive minigame) let each player play their own copy and show the others as ghosts sent at a few Hz. Check two traps: the key you want to hijack (here the game's abort-movie key) can be quietly rebound by another feature of your own mod and then persisted in the game's settings file, so read the binding back in the test; and messages sent from inside the level-load call carry the old level epoch and are dropped. Example: MDK2 HD (tools/qa_cutscene.sh, games/mdk2-hd/NOTES.md).
Parallel QA needs separate evidence as well as separate games. Give every lane its own copies, ports, helper folder and PID file, acquire all lanes before starting a phase, and release partial reservations when capacity is unavailable. Keep machine-specific provisioning behind an optional private command/environment interface. Serialize suites that write shared per-user settings, and isolate a motion probe if contention drops it below its required frame count rather than lowering its assertions. Evidence folders need unique suffixes: two MDK2 projectile variants started in the same second and would otherwise write the same snapshots. Compare the named gate results with a sequential baseline, and force rare event orders directly instead of relying on a broad sweep to reproduce a race. Example: MDK2 HD (tools/qa_all.py, tests/qa_runner_test.py, games/mdk2-hd/NOTES.md, 2026-10-23).
Start a grace window when the player acts, not when the event happens, and test past it. MDK2 HD frees a hero switched in at a wall by moving it to nearby spots while a move key is held, for a few seconds. Those seconds were counted from the switch, so a player who waited before walking stayed stuck for good. The test pressed keys about a second inside the window and passed until a busy PC delayed the presses. Count a window like this from the first attempt it is meant to help with, and make the test wait longer than the old window so the broken version fails every time, not only under load. Example: MDK2 HD (tools/qa_char.sh, games/mdk2-hd/NOTES.md, 2026-10-23).
A test that passes can hide a crash after its checks: also check the run lasted its full length. Jericho's split screen button tests passed for two releases while the game crashed in 4 runs of 6, a second after player 2's melee hit; every assertion had already been met. Fail a run whose game ends early or leaves a crash dump, and log the faulting address with registers from a vectored exception handler in the proxy (the game's own handler shows only its exit path). Here the address led to an engine lookup of "the user of this first-person model" without a bounds check, fed by a trick of the mod's own (hiding a model from one view by giving it the id of a user that does not exist), and to a second one from the game's scripts. Look for every place a fake value you plant can be read back, not only the one you meant it for. Example: Clive Barker's Jericho (native/split.c fp_user_lookup, lua/coop.lua Coop.usrGuard in jericho-coop).
Test an update from the state a real player has, not from what the test setup makes. Jericho Co-op+'s installer recorded its shortcuts as one joined line whenever there were two (desktop and Start menu) and as an empty line with none; the next update then stopped with an error. Every install and update test had put one shortcut in a test folder, which records correctly, so three releases passed. Install the previous published version the way a player does (every shortcut, default options) or reproduce its record exactly, then update, uninstall, and check the record and the files after each step.
Never put a process-local handle on the wire. Jedi Outcast's movers sent the host's sound handle numbers to joiners. In the joiner's process the same number was a different, streamed voice file that can't loop, and the joiner was dropped on every Bespin level. Indexes into per-process tables (sound and model handles, string-table slots, pointers) mean nothing in another process even on the same PC with the same files. Send the name, or a slot in a table that both sides fill from the same names, and resolve it locally (CG_CoopBModelSound in jedi-outcast-coop's fork).
"The player" may be a hard-coded index, on both sides. In Raven's single-player code the player is entity 0 (s.number == 0, g_entities[0], client == 0), and the same check means "the host" on a joiner, whose own slot is 1 to 3. Script names (kyle) and per-frame updates (only entity 0's script task manager ran) followed the same rule, so scripted doors and troop spawns silently never ran once a joiner had joined. Before fixing bugs one by one, grep for every form of "player index 0" on the server and the client side and decide what each one should mean in co-op. Example: Jedi Outcast (games/jedi-outcast).
Never key a texture (or any GPU object) by its address alone. A mod swapped a game's font textures for sharper copies, keyed by the Direct3D texture pointer. After a level load the game freed a font and the allocator reused the address for a decal, so letters appeared on the ground under a character. Key by content (a hash at upload time) or drop the entry when the game releases the object. From: Obscure 1.5, reverted in 1.6.
Measure every graphics option before shipping it. A negative mip LOD bias changed nothing in Obscure because the game sets the mip filter to NONE (logged sampler states), while 16x anisotropic filtering added about 4% detail on angled ground (mean gradient over the same region, A/B with one variable). Log the render state the game really sets, then compare screenshots numerically.
Supersampling an old Direct3D 9 game from a proxy DLL works, if the game's layout follows the bigger buffer. Enlarge the back buffer, shrink it in Present into an extra window-sized swap chain (StretchRect LINEAR at exactly 2:1 is a box filter), and make the game believe its window is bigger (its GetClientRect, plus its windowed code path), or it draws into a corner. Text drawn 1:1 in pixels needs its own scale (scale each string's quads about its anchor). ReShade then sees the extra swap chain as a second runtime (ReShade2.ini) and may need DepthCopyBeforeClears=1 to find depth. From: Obscure 1.2-1.4.
In Unity the product name, not the folder, decides where saves and settings go. A copy of Gedonia in a test folder still wrote the real PlayerPrefs (resolution, fullscreen) into the registry: Application.persistentDataPath and the PlayerPrefs key HKCU\Software\<company>\<product> both come from PlayerSettings.productName in globalgamemanagers, while app.info only moves Player.log. Give each test copy its own product name by patching that serialized string in place (a name of the same length class keeps every offset; check the company string next to it before writing), drop a marker file in the copy, and make the plugin quit if a marked copy still resolves to the real save folder. Compare the real saves and registry key with a backup after every run. Example: Gedonia (scripts/set-product-name.ps1 in gedonia-coop).
Parse numbers from arguments and files with the invariant culture. On a German Windows float.TryParse("0.05") returned 5, so a "5 % packet loss" test option dropped every packet and the run looked like a broken transport. Use CultureInfo.InvariantCulture for every number you parse or print in logs, test steps and protocols, and assert a parsed test parameter by logging it back at start. Example: Gedonia (Plugin.ParseFloat in gedonia-coop).
Sync sounds where the engine plays them, not per feature. Remote players and mirrored enemies were silent: player sounds come from animation events (footsteps, swings) and code (jumps, damage), and a mirror skips the AI that plays attack and hurt sounds. A postfix on the engine's play calls (Unity AudioSource.PlayOneShot(clip, volume) and non-looping Play()) on objects this machine simulates sends the actor id, the source's path below it, the clip name, volume and pitch; others play the clip on the puppet or on the mirror's own source at that path. Skip a clip that source already played locally within a second, because some events (deaths) run everywhere, and cap the rate. Example: Gedonia (SoundSync.cs in gedonia-coop, scenario fight).
A hook on a notification is not a veto. In Code Vein the function that writes the visible co-op state is the session state machine's "on enter state" callback. Skipping it when it announced "end co-op" kept the visible state for 11 ms, then the state machine, which had already stored the new internal state, ran its next step and ended the session anyway. Before vetoing, check the previous-state argument of the next call, or read the code that stores the state: block the function that requests the change (here the one that sets the end request flag), not the one that reports it. Example: Code Vein (games/code-vein).
UE4SS key binds reach every running copy of the game. With two instances on one PC, each F-key press ran in both: the host's "join" tore down its own listen server just as the guest connected, and the guest's "host" re-hosted. Several "join fails" runs were really this. Give each instance a role (lock it on the first host or join key, or read it from a file or the command line) and make every key check it. Example: Genesis Alpha One (main.lua role lock in genesis-alpha-one-coop).
A fully black screen with a working engine is often a HUD widget, not rendering. In Genesis Alpha One both windows turned black when the guest joined. Camera fades, SetEnableWorldRendering, post-processing and view targets were all fine; the engine's own HighResShot (scene only, no UMG) showed the normal view while the window was black. The cause was a full-screen opaque Border (FadeOutBorder) inside the single-player HUD that the join left on. When the scene renders but the window is black, list every visible Image/Border under widgets in the viewport (nested user widgets report IsInViewport() == false, climb their outers) before touching the renderer. Also: in UE 4.23 StopCameraFade leaves the fade at its target value, so it cannot clear a fade to black; use SetManualCameraFade(0, ...).
In single-player Blueprint games the joining player's character may never be initialized. Movement replicated out of the box, but the guest stood frozen: the game's load flow sets IsPlayerInitialized, unblocks input and gives the movement speed only for the player it loaded. Dump the Blueprint bool and number variables of the host's and the guest's pawns side by side (ForEachProperty on the class chain) and diff them; the difference is the to-do list. Example: Genesis Alpha One, IsPlayerInitialized, BlockActionFPSPlayerInput, PlayerMovementSpeed, inactive CharacterMovement.
Runtime level instances do not replicate, but their names can line up. A ship built from LoadLevelInstance calls exists only on the host. LevelStreamingDynamic names its packages <Short>_LevelInstance_<N> from a per-process counter, and UE 4.23 has no name override, so the guest gets the host's names only if it made the same number of instances before and loads the list in the same order. Log the names on both sides before blaming a disconnect on them. Example: Genesis Alpha One, 31 instances _32.._62 identical on both sides.
Add native UE4SS hooks one at a time. Registering a dozen RegisterHook calls on engine UFunctions at once crashed Genesis Alpha One at the main menu ("Tried to execute UFunction::FuncPtr hook but there was no function map entry"), most likely because one patched exec thunk is shared with another function. RegisterBeginPlayPostHook, NotifyOnNewObject and RegisterProcessConsoleExecPreHook gave the same information (what spawns, which widgets appear, which console commands run when a player joins) without that risk. Syntax-check every Lua file before copying it to the game: code appended after a module's return disables the whole mod with one red line in the log.
Check each engine function's ret N before calling it from a hook. In Thief 2 NewDark's MP exe, the "get physics model" function (0x516c90) looks like a plain one-argument function in Ghidra but ends in ret 4 (stdcall). Called as cdecl, every call drifted ESP by 4 and both games crashed on the first in-mission frame. Hooks called from MSVC code also only get a 4-byte-aligned stack, while clang/gcc for i686 assume 16 and emit movapd to the stack: build hook DLLs with -mstackrealign. That fault shows up as an access violation at address 0xffffffff. Example: Thief 2 (games/thief-2/NOTES.md). The opposite mismatch (a callee that pops an argument you never pushed) shows up when the calling function returns: MSVC's stack check reads its cookie one slot off and the game aborts with 0xc0000409 in __report_gsfailure, the same address every time. In Fallout Tactics the Load Game screen function (0x4ca7b0, ret 4) was called with no argument from a pump hook, and every session played from that screen crashed on its way back to the menu; the WER minidump showed the cookie slot holding the return address one slot up (games/fallout-tactics).
Measure remote movement against the sender's true path, not only against the same instant. For player avatars, compare each frame of the displayed position with the nearest point of the real path over the last ~1.5 s (path deviation) as well as with where the player is now (lag). Lag is expected, but leaving the path means passing through walls, floating or being stuck. Also log how far the sender travelled and fail any run where they barely moved: a bot script loaded after the game started gave a perfect, meaningless 0. Example: Thief 2 (tools/qa/analyze.py in T2MPFix).
A hook can change a function it only passes through, if that function reads stack it never wrote. In Thief 2 NewDark's MP exe, the remote player steering code (0x553dc0) decides whether to pull a moving avatar toward its predicted position by testing three stack floats it never writes on that path, so the leftovers of earlier calls decide. A pure refactor of the hook in front of it, with no change to what it passed on, made avatars trail 3x further behind, and the "stock" control measured through that hook had never been the real game (path deviation max 3.0 through the old hook, 7.7 in the plain game). When a refactor changes a stock or control number, A/B old and new builds split by machine, then read the function's machine code for loads from locals that aren't written first. Measure the stock control with nothing hooked in that call chain (here telemetry walks the engine's ghost table instead), and fix such a bug as its own switchable patch rather than by luck of stack layout. Example: Thief 2 (fix_steer.c and docs/NOTES.md in T2MPFix, games/thief-2/NOTES.md).
Stopping a background shell task may leave its child running. On Windows, stopping an agent's background Bash task ended the bash wrapper but not the sh.exe running the script. The script kept sending stick input to a test copy, and when that copy was restarted it steered the title menu: the copy loaded another campaign and autosaved it into its own save folder. Before restarting a copy, list and stop leftover script processes (Shadow of War).
If the game reaches "the player" through one function, swap what it returns instead of reimplementing actions. Forager reads the player through PlayerFind(0) in 365 functions (about 350 calls per frame). Hooking it to return the current actor, normally the local player and a guest's puppet while the host performs that guest's hit, build or purchase, made the game's own code use the guest's tool damage, skills, buffs and gear; outputs aimed at the player (GainXp, GainItem, coins, damage) are then forwarded to the guest instead of applied on the host. Keep the hook cheap and restore the actor on every exit path. Example: Forager (games/forager).
A replica that takes damage locally must report it, or the authority's next update erases it. In Forager a guest's arrows, staffs and sword swings are separate objects that hit the guest's copy of an enemy; the host never heard of the hit, and its next hp update undid it, so only the pickaxe (which sends an explicit hit) worked. Before writing the authority's hp into a replica, compare the replica's hp with the value written last time and send any drop as a hit, and do not count the authority's own hp updates as damage. Example: Forager (mobs.cpp in forager-coop, games/forager/NOTES.md).
When an effect "does nothing", log every hop on the live game before reading more decompile. In Thief 2 NewDark's MP exe, lava did no damage. The decompile of the stimulus code had dropped blocks as "unreachable" and pointed the wrong way. A throwaway logging build then logged each step in the running game: flow propagator, stimulus, receptron, damage effect, damage model, damage filters. Every step worked until a multiplayer-only damage filter, three layers further down, zeroed all damage from Local Only objects. List the chain from the trigger to the visible result, log each hop with its arguments and return value, and read the decompile only for the hop where the value disappears. Example: Thief 2 (fix_damage.c and docs/NOTES.md in T2MPFix, games/thief-2/NOTES.md).
A locked Windows session fails visual and input tests silently. With the PC locked, Direct3D can't present, so Thief 2 test instances drew nothing and injected input did nothing, and no error said why. Before a live suite, check that the session is unlocked (no LogonUI.exe process) and fail the run with that reason if it isn't. Example: Thief 2 (tools/qa in T2MPFix).
A fixed resend timer without a send window only breaks on a real link. On loopback (about 1 ms) every test passed; over the internet through Porthole (about 150 ms round trip) the host's game filled its upload: every unacked datagram was resent every 150 ms, so everything went out two or three times, and bursts snowballed. Cap the unacked datagrams in flight, resend after the measured round trip with backoff per datagram, and send state as changes, not whole (the parts stock, keyed by item JSON, went whole on every change). Keep a headless test on a fake link with a delay that asserts bytes sent per byte of data (here 1.47x before, 1.03x after at 200 ms), and log network use per minute so a player's log shows it. Example: ReStory (games/restory-chill-electronics-repairs).
Over a link that names players by account (Steam), tag each datagram with the game run. With UDP a restarted client comes back on a new port, so the host sees a new peer. With Steam it comes back with the same account id, and the host's late reliable datagrams for the old run sat in the new run's reorder buffer and would have been delivered out of order. Put a random number per game start on every datagram, plus the number of the run it is meant for, and drop datagrams meant for another run. Example: ReStory (IdLink in restory-multiplayer, with a fake-Steam headless test).
Events catch what players do, not what the game does. Syncing drops left out everything the game makes or moves on its own: a customer's device, bills, cash, licences, a device put on the bench by a button or by unpacking. Add a periodic room list from the host (id plus a short signature per object; the guest asks for what it lacks and removes what the host no longer has, never touching its own hand, bench or unconfirmed moves), and a twice-a-second watcher that reports any change of an object's signature made locally. Example: ReStory (CoopCensus, WatchForChanges).
Hidden is not removed when the game saves everything it keeps. Applying "gone" by hiding the object left it in the room registry, so it went into the host's save and every thrown-away box came back the next morning on both sides. Remove gone objects the way the game does (unregister, destroy), and hide only what the other player holds. Example: ReStory.
Decide derived states from the game's own record, in one function. "On this player's bench" was read from the box's state, which stays "stored" after the box is unpacked on the bench, while the report and the signature computed it differently; the device stayed on the other player's desk and the two copies traded "on a bench" and "stored" back and forth. Use the game's record (DeviceService.PlacedDeviceContainer) and one function for reports, signatures and guards; after applying a remote change, give it a moment before reporting it again. Example: ReStory.
Test copies of a Steam build talk to the developers. Copies of the Steam build ran the game's Unity Services Analytics and wrote engine session files and registry values into the real game's data on every bot run. Switch analytics and crash reports off in the plugin when test flags are present, give test copies their own product name (lesson 51), and compare saves strictly while leaving engine bookkeeping out of the "real data unchanged" check. Example: ReStory (TestPrivacy, scripts/real_state.py in restory-multiplayer).
A loaded save is not the state it was saved from. In BattleTech a combat save loaded on the guest put a 'Mech's components back in a different order, so the same seeded critical hit picked a different component, and the full-stat comparison flagged it at seq 207. If the guest joins from a save, have the host load that same save too, so both run from identical post-load state (one extra load on the host), or take the save before anything has happened and have both sides start fresh from it. BattleTech did the first up to 0.0.12; 0.0.13 takes the save at the briefing and the guest loads it as a "restart mission", which launches the contract from scratch, so the host needs no reload and both players see the opening. Example: BattleTech (battletech-coop, "Campaign missions by save").
"Loaded" needs the engine's own signal. After a Unity save load began, a fresh SimGameState existed within 0.1 s, with the default company and 0 funds, long before the save was applied; a check for "new state object exists" passed on the placeholder. Wait for the game's own ready point (here SimGameState._OnAttachUXComplete) and assert on a value that only the loaded state has (the company name you sent). The ship-view test compared the guest's company with the host's and caught it in the first run. Example: BattleTech.
IDs from a counter diverge between copies. New pilots and 'Mechs got "SGRef_" + ++UIDCount, so the same hire on host and guest made two different IDs, and a later dismissal sent by ID found nothing. Send the counter value the originating side started from with each change, apply the change from that counter (restore your own after), and give each copy its own range (BattleTech: 1e9 x player index), so IDs made on different copies never collide. Example: BattleTech (ShipRequests.Company.cs).
A Harmony postfix sees reassigned arguments. SimGameState.ReadyMech(slot, id) overwrites id (chassis ID to storage stat ID) before it returns, and the postfix read the new value, so the host was asked to ready an ID that does nothing. The digest still matched because the guest's test then cancelled the order; only the "refused" check failed. Read the arguments you send in the prefix (__state), and fail a test on any refusal, not only on a final-state mismatch. Example: BattleTech.
A layer without commands still syncs well at the function the UI calls. BattleTech's campaign screens call SimGameState methods directly (refits, pilot training, hiring, the store), so there is nothing to lockstep. Patching those game functions (not UI events), sending their arguments as data, and repeating them through the same functions on the other side gave the game's own costs and timers on both copies. Two traps: these functions call each other (ReadyMech calls ScrapInactiveMech), so send only the outermost call; and suppress capture while applying a remote change, or it echoes back. Example: BattleTech.
A "restart" re-rolls whatever was random. BattleTech's restart-mission path reloads the mission definition and picks its "any 'Mech with these tags" enemies again, so the guest's restart built 7 actors where the host's launch had 6. When you start a mission on both sides from the same save, send what the host's launch picked (here lance, unit and pilot per spawn point) and write it into the guest's definition before its launch rolls; compare unit counts and IDs on both sides in the test. Example: BattleTech (LancePins.cs).
A modding layer can bring its own randomness into a lockstep game. Fallout Tactics' simulation draws from a per-frame seeded generator, identical on every copy, and yet every fight with NPCs split the copies: FTSE (the community Lua extension) overrides burst fire in OnBurstAttack and rolled with Lua's math.random, seeded from the clock. Hunting inside the game's bullet code found nothing because the deciding draw was in a script. List every hook a script extender or other mod installs, and make each random draw it makes come from a seed every copy shares (games/fallout-tactics).
In lockstep, a private screen must not hand out entities while shared commands do. A Fallout Tactics shop pauses nobody, and the shopper's copy creates and destroys item entities with every offer; the other copies only catch up when the shop closes. A give issued by another player meanwhile ran on every copy at the same frame, but took a different slot off each copy's free list, so from then on the copies disagreed about which item was which. The entity log showed it in two lines: the same command, made 702 ... destroyed 1095 on one copy and made 702 ... destroyed 702 on the other. Hold the commands that create or destroy entities from the frame such a screen opens until its changes have reached every copy, then run them in order on every copy at the same frame (games/fallout-tactics).
Tie a drift check's bookkeeping to the loaded save, and log when it matches. Fallout Tactics' drift check skipped every hash at or before the frame of its last correction. A load restarts the frame count, so one correction early in an evening switched the check off for the rest of it, and a real inventory split went unnoticed. Because a match logged nothing, "all fine" and "not checking" looked the same in the logs. Keep every "skip until" marker per loaded generation, and log a line about once a minute when the copies match (games/fallout-tactics).
DirectPlay's TCP provider can block Release() for about 3 seconds. In Fallout Tactics, closing the Multiplayer screen's session search froze the game for 3 s: dpwsockx waits (WaitForSingleObject(thread, INFINITE)) for one of its own threads, which takes that long to notice the shutdown. Sampling the main thread's stack from a helper thread (suspend, read ESP, map return addresses to modules) found it in one run. Releasing on a thread of its own keeps the game responsive, but creating the next DirectPlay object still waits on the first, so only the parts that don't need DirectPlay (here the waiting message and the save download) get faster. Fixed polling intervals (1-2 s) were the bigger, easier win (games/fallout-tactics).
On Unity 5.x, BepInEx 5's default entrypoint can break every scene script. Unity 5.4 loads the game's scripts into a child domain, and BepInEx's default entrypoint (Application..cctor) ran the chainloader before Assembly-CSharp was loaded: every scene logged "The referenced script on this Behaviour is missing!" and the game's Steam manager and other scripts stopped working. Setting [Preloader.Entrypoint] Type = SceneManager, Method = Internal_SceneLoaded starts the chainloader at the first scene load, after the scripts are bound. Ship that BepInEx.cfg with the mod. Example: Rick and Morty: Virtual Rick-ality.
A VR game can be lab-tested without a headset. Unity's -vrmode None keeps the VR runtime from loading. Then fix only what the game needed from it (platform detection, a play-space size wait, the first scene load), and look for the developer's debug input: Virtual Rick-ality's rig has a debug hand controller whose public isGrabbing flag drives the real grab, joint and release code. A bot moving the head and hands and setting that flag tests items, hand-offs and levers with several minimized copies on one PC, and the same layer becomes a flat-screen keyboard-and-mouse mode for players. Example: Rick and Morty: Virtual Rick-ality.
Strip a copied game object's scripts in RequireComponent order. The other player's puppet is often a copy of the game's own player model, with its scripts removed. Removing them in any order makes Unity refuse some ("Can't remove TextContainer because TextMeshPro depends on it"), which logs an error each time and leaves those scripts running on the copy. Remove in passes, each pass taking only the scripts no remaining script on the same object requires, and check in the lab that the copy has no scripts left and the game logged no errors. Example: Rick and Morty: Virtual Rick-ality (Avatars.Inert).
A pointer seen in one session is not a pointer path. In Persona 5 Royal, a static (P5R.exe+0x293EEF8) pointed at the battle object in every snapshot of one probe and read 0 in the next launch; no code references it. Snapshot one battle in two separate launches, keep only the static -> offset chains that resolve to the right value in both (a depth-3 scan cut 481 chains to 23), prefer statics that code actually references, and read several independent chains with a vote so one stale chain cannot decide (games/persona-5-royal).
A memory read that returns 0 on failure hides your bug as "nothing there". A Reloaded-II mod kept Process.GetCurrentProcess().Handle as a raw handle. When the Process object was finalized the handle closed, every ReadProcessMemory failed with error 6, and the helper returned 0, which looked exactly like "not in battle" for two playtests. Use the current-process pseudo-handle (-1), and log when a read of a static fails, since a static must always be readable (games/persona-5-royal).
Diff a UI-made object against a code-made one when your replay is invisible. ZooTogether's replayed Zoo Tycoon buildings were in the world list (the layout hash matched) but never drawn. Dumping 400 bytes of a building the player bought and of one placed from code, side by side, showed a screen position at +0xb4/+0xb8 only the first had: the buy cursor fills it while the player moves the object (BFEntity::snapToGrid). Calling that before placing fixed it, and the same diff later showed snapping moves edge objects (fences) half a tile, so their exact position goes back afterwards. Example: Zoo Tycoon (games/zoo-tycoon).
After replaying the other player's action, clear what the game remembers about it. The host replayed the client's hired keeper, its move and its firing; the host's undo history and map-view selection still pointed at the freed keeper, and the host crashed on its next own build once that memory was reused (3 of 3 runs; bisected by skipping one lab step at a time). Clear the replaying player's undo history and selection after every replayed place, move or remove; undoing the other player's work would be wrong anyway. Example: Zoo Tycoon (forget_undo in ZooTogether's coop.rs).
A confirmation dialog inside a game call runs nested frames. Zoo Tycoon's sell asks "are you sure?" and closing an exhibit asks for its name; both pump the game loop until answered, so the per-frame hook that runs your console and network tick is entered again on the same thread and a blocking mutex deadlocks (the stack sample showed call_next_command waiting on its own lock). Make per-frame entry points try_lock and skip a re-entered frame, and for replays find what skips the dialog: a "confirm before selling" byte cleared around the call, a detour that skips the name prompt. Example: Zoo Tycoon.
When mirroring the host's AI actors, replace drifted ones instead of moving them. Writing a Zoo Tycoon animal's position directly left it registered on its old map tile, and 2 of 11 lab runs crashed on a later path build; moving it with the game's pick-up and drop left the joining player carrying it, so their next click dropped a copy that was sent back (a loop of keepers). Removing the drifted actor and creating one at the host's spot through the same paths as births and deaths was stable. Also leave alone what the player is carrying and what they placed in the last few seconds, and send only player-initiated removals so each copy's own deaths don't kill the other copy's animals. Example: Zoo Tycoon (reconcile_living).
Driving hidden game windows: activate, fake the mouse for drags, and don't trust captures. With two Zoo Tycoon copies parked off-screen (not minimized, which stops the game loop), posted clicks were ignored by every copy but the last one started until WM_ACTIVATEAPP/WM_ACTIVATE were posted first. Drags (fences, paths, terrain) and picking the object under the mouse read GetCursorPos, not the messages, so a lab-only GetCursorPos detour returns a position the test sets (only during drags; left on, it broke plain clicks). PrintWindow on the DirectDraw window often returned an old frame (the start screen until the first click), so reading panel state from captures toggled panels shut; click deterministically and keep pixel checks only where a retry is harmless. Example: Zoo Tycoon (zt_launch.py, lab_coop.py).
Count your Harmony patches at start-up, and fail loudly on zero. In Superflight the plugin called harmony.PatchAll(typeof(Patches)) with every patch as a nested class of Patches; that overload patches only the type it is given, so no hook was live, while the log still said "patches applied" and three test runs passed because none of their checks needed a hook. The first test that did (a portal flow) failed on every patched behaviour at once. Use PatchAll(typeof(Plugin).Assembly) (or PatchAll()), log harmony.GetPatchedMethods() with each method's name, error when the count is 0, and give every patch at least one test that fails without it. Example: Superflight (Plugin.cs in superflight-coop, games/superflight/NOTES.md).
Battle bots need real player stats, and a person watching catches what asserts miss. BLACK SOULS II: a bot that skipped the intro played the placeholder actor (about 27 times normal HP and attack), so every fight one-shot, enemy hits showed "not affected", and a battle suite "passed" without exercising multi-turn sync. A tester watching the visible test windows spotted it, and also noticed that auto-battling bots never open the command menu. Load the real starting characters at a realistic level (read the class curves and the damage formula from the data), keep bot runs visible when someone wants to watch, and add one scenario that uses the real input menus instead of auto-battle.

56 KiB Raw Permalink Blame History

Hard-learned lessons

56 KiB

Raw Permalink Blame History