horizon/docs/benchmark.md

Benchmark: Horizon vs vanilla 26.3

tools/perf_bench.py measures Horizon, the vanilla 26.3 client, and vanilla with performance mods on the same machine, world, and camera path, computing every number with the same code. This page says what it measures, how it keeps the comparison fair, what it can't control, and how to reproduce it. The results are in Results.

What is measured

One game is started per window size (1280x720 and 1920x1080 by default) and render distance (8, 16 and 32 chunks by default), each from a fresh world, so nothing generated by an earlier run is reused. In each game:

1. Start to the title screen. Wall-clock time from starting the process to the first frame of the title screen with no loading overlay.
2. Create the world. A creative world named PerfBench, seed 1234, default world type and difficulty, created the way the game's own Create World does. Start to in-world is the time from starting the process until the player has spawned and every chunk around the spawn is loaded and drawn.
3. Four scenes, the paths of tools/graphics_bench.py: straight 192-block flights over a village, a forest, the ocean and a mountain. For each scene the player is teleported to the start (time set to noon), held there flying, and the chunk load time is the time from the teleport until every chunk in view distance is loaded and drawn. After 2 seconds of settling, the player flies the path at constant speed in 12 seconds (16 blocks per second, faster than sprint-flying), facing along it at the scene's pitch, placed on the line every frame. Chunks are generated and drawn as they come into range during the flight, as in play.
4. Frame times. Every frame of every flight is recorded (time between the starts of consecutive frames). From them: average FPS, 1% and 0.1% lows (the average frame time of the slowest 1% or 0.1% of frames, as FPS: a single long freeze shows), the median, p95, p99 and p99.9 frame times, the slowest frame, and the share of frames in buckets (under 4.17 ms = 240 FPS, 8.33, 16.67, 33.33, 50, 100 ms, and over).
5. Memory. The process's resident set size (RSS), sampled 4 times a second from outside the process (ps on macOS and Linux, the working set, as Task Manager shows it, on Windows): peak over the whole run (startup included) and steady, the median during the flights. For vanilla this counts the whole JVM (heap, JIT, native buffers, the integrated server), for Horizon the whole process. GPU memory where it can be read: Horizon's own Vulkan allocator total, and what nvidia-smi attributes to the process on NVIDIA. On Apple Silicon the GPU shares RAM and no per-process figure is available.

How it is kept fair

• Same machine, same run conditions. Games run one at a time; the load average before and after each run (Windows has none, so it is left out there) and the machine (CPU, RAM, OS) are stored with the results. Runs on a machine doing other work are marked as such and not published.
• Same settings, one file. Every game gets the same options.txt, written by the driver: render distance, simulation distance 12, unlimited frame rate (maxFps:260), VSync off, FOV 70, automatic GUI scale, windowed, no pause on lost focus, and inactivityFpsLimit:minimized (both games otherwise drop to 30 FPS after a minute without input). Everything else stays at vanilla's defaults: Horizon reads vanilla's options.txt keys and its defaults match vanilla's (the options harness checks it), and Horizon's own settings (horizon-graphics.txt) are left at their defaults, which render as vanilla does.
• Same resolution. Both games get the same window size, --width and --height (vanilla's own launch arguments, which Horizon takes too): 1280x720, the size results were first published at, then 1920x1080, the commonest screen; --sizes picks others. The size each game actually renders at is stored with the results: a compositor or screen smaller than the window can shrink it.
• Same world and path. Same seed and world type; same paths, yaw and pitch, same flight speed; the same steps in the same order. Vanilla is driven by a small dev-only Fabric mod (mods/horizon/src/bench), which only calls the game's own code: it creates the world with WorldOpenFlows.createFreshLevel as the Create World screen does, teleports with /tp, sets the time with /time set, and places the player on the path each frame as Horizon's horizon-ctl fly does.
• Warm-up excluded. Each scene's flight starts only after its chunks are loaded and drawn and 2 seconds have passed; the load itself is reported separately. JIT warm-up of the JVM during the first scene is part of vanilla's real behaviour and is not removed.
• The same statistics code. Horizon returns its raw frame times (horizon-ctl fly ... raw) and the vanilla mod writes its own; stats() in the driver computes every figure for both.
• Vanilla as players run it. Your own 26.3 client jar and libraries (as the official launcher installs them for your OS), the official launcher's default JVM arguments (-Xmx2G and its G1 options) and the version manifest's JVM flags (-XX:StackShadowPages=32, without which vanilla crashes on about half its starts on Windows), Java 25. The game's assets index is not used, so neither game plays sound (Horizon runs with HORIZON_NO_AUDIO=1, vanilla with OpenAL's null device); textures, models and shaders come from the client jar as usual.
• Performance mods. --game vanilla-mods adds the Fabric builds for 26.3 of Sodium, Lithium, FerriteCore and ImmediatelyFast, the versions pinned with their hashes in tools/benchmark-deps/mods.lock.json (downloaded from Modrinth, never committed), plus Fabric API if one of them needs it.

Known caveats

• Fabric Loader is present in the vanilla runs, to load the driver mod. The only patch it applies is the driver's frame hook, so its cost after startup is negligible, but its mod discovery and class transformation add to vanilla's startup time (the modded runs use the same loader).
• Readiness is detected differently. For Horizon the driver waits until nothing is queued, the chunk counts are unchanged for a second and meshing has gone quiet (fewer than 100 section meshes a second: blocks that change keep being remeshed), and takes the time of the last of these; sections wait for their neighbours' light before they are meshed, so the chunk counts alone stop the clock before the last sections are drawn. For vanilla it waits until the loaded chunk count is unchanged for a second with nothing left to compile, and takes the time of the last change. Both stop the clock within a frame or two of the last section drawn.
• Locked or headless screens. A window nobody can see is not composited. On a Mac whose session is locked, Horizon draws only with --offscreen (it renders into a texture and does not wait for the window), and vanilla keeps rendering into its window without it being shown. Such runs measure rendering without presentation and are marked offscreen; numbers for players come from runs with the window on a real screen.
• Different graphics APIs. Horizon renders with wgpu (Vulkan on Linux and Windows by default, OpenGL when chosen or when Vulkan has no driver, Metal on macOS); vanilla 26.3 uses OpenGL (or its Vulkan backend when chosen). The backend each game used is stored with the results. This is part of what is being compared.
• World generation is the same algorithm but not the same threads. Both generate the same terrain (the parity harnesses check it), with their own threading; the work per chunk differs, which is also part of what is being compared.
• Variance. Flights are short (12 seconds), so each configuration runs --repeats times (3 by default), each from a fresh world, and every figure is reported as the mean over the repeats with its standard deviation and 95% confidence interval (Student's t). Differences inside the intervals are noise. Lows are the noisiest figures.

Benchmark your own machine

The game runs the Horizon half of this benchmark by itself, with nothing else installed:

horizon-client --benchmark standard --render-distance 16

It starts at the title screen, creates the world horizon-benchmark (seed 1234, creative; the next run replaces it, and it never touches a world it didn't create), flies the four scenes as above, writes a report to benchmarks/ in the game folder and quits. --benchmark village (or forest, ocean, mountain) flies one scene. For the run only, VSync is off, the frame rate unlimited and the game doesn't pause or slow down without focus; your options are not changed. Your other settings (render distance unless given, graphics, window size) are used as they are, and stored in the report, so compare reports with the same conditions.

The report is the benchmark report format: startup to the title screen and to in-world, each scene's chunk load time, chunks generated and section meshes built, every flight's frame figures and histogram, resident and GPU memory, the binary's size, and the machine (OS, CPU, RAM, GPU, driver, backend, display), the load average before the run and whether it ran on battery. If you allowed benchmark uploads (the launcher's data collection choices, docs/telemetry.md), the same report, dated to the day and without local fields, is kept for the launcher to send; otherwise it stays on your machine.

Reproduce it

You need Horizon built (cargo build --release -p horizon-client -p horizon-client-lib --bins) and its resources prepared from your own copy of 26.3 (which also provides .local/minecraft/26.3, the vanilla client the driver starts), Java 25 with javac, and Python 3.10 or newer.

python3 tools/perf_bench.py --game horizon --label horizon-mine
python3 tools/perf_bench.py --game vanilla --label vanilla-mine
python3 tools/perf_bench.py --game vanilla-mods --label mods-mine
python3 tools/perf_bench.py --table horizon-mine vanilla-mine mods-mine

Options: --distances 8,16, --sizes 1920x1080, --scenes village,ocean, --seconds 20, --repeats 5, --heap 4G (vanilla), --offscreen (Horizon, for locked or headless sessions). Results go to benchmarks/perf/<label>-rd<N>.json at 1280x720 and <label>-1080p-rd<N>.json at 1920x1080 (other sizes <label>-<W>x<H>-rd<N>.json; --table takes those labels), one benchmark report per window size and render distance (the same format the game's --benchmark and the launcher write), with the load average before each game; each game's log, and the vanilla driver's event log, stay in .local/perf-bench/. Keep the machine otherwise idle, plugged in, and the window on a screen. PERF_BENCH_TRACE=<file> appends each load's timeline to that file, one JSON line per sample: Horizon's status every 50 ms after a line for each teleport (resident and generated chunks, chunks still pending, stage counts), and the vanilla driver's loaded chunk count and the chunks it holds within the render distance minus one; compare when the columns arrive with when the load ends.

Many-entity stress scenes

--scenes stress (or some of stress_empty, stress_mobs, stress_items, stress_frames, stress_armor_stands, stress_chests, stress_signs, stress_banners, stress_particles) runs the scenes of tools/stress_scenes.py instead of flying: 1000 still mobs of 8 kinds, 2000 dropped items, 512 item frames, 500 armoured armor stands, 800 chests and shulker boxes, a wall of 687 written signs, 300 banners, and steady particles from repeating command blocks (flames, smoke, end rods, firework bursts, TNT in the air). Each is built high above the terrain by the same vanilla commands in every game (Horizon through horizon-ctl command, vanilla and vanilla-mods through the driver mod), settles 6 seconds and is held still for --seconds; stress_empty is the same camera and floor without them. Entities are drawn only within a distance that grows with the render distance up to 20 (Entity.shouldRenderAtSqrDistance), so run them at 16 or more:

python3 tools/perf_bench.py --game horizon --scenes stress --distances 16 --label stress-horizon
python3 tools/perf_bench.py --game vanilla --scenes stress --distances 16 --label stress-vanilla
python3 tools/perf_bench.py --table stress-horizon stress-vanilla

Besides the FPS table, --table then gives each scene's median frame CPU time (Horizon), client tick time and resident memory. Chests stay closed: opening them needs a player, which neither driver has. The stress scenes use 26.3's command syntax, so they are not for --game neoforge.

On Windows, run python tools\perf_bench.py ... the same way. Horizon's control endpoint there is a localhost TCP port, which the game writes to a .json file in the temporary folder. --build can point at an installed release (%APPDATA%\horizon-launcher\versions\vX.Y.Z, with --resources its .local\horizon-client\26.3), and the report then gives that release's version.

Frame pacing

tools/pacing_bench.py measures smoothness rather than throughput: every frame of the sessions where players see hitches, a fast flight (64 blocks a second over new terrain), eight long teleports, trips to the Nether and back, and the inventory and pause menu opened and closed twenty times. For each it reports the median, p99 and p99.9 frame times, the slowest frame, the 1% and 0.1% lows and the frames over 33 ms. The game runs with HORIZON_SPIKES, so every slow frame also logs its timed spans (wall and thread CPU time): the report keeps the slowest ones next to the figures, which says whether a hitch was work (a span's CPU time) or waiting (the GPU, a lock, a file). Renderer's frame and entity_geometry time their steps as spans too (spans::laps).

python3 tools/pacing_bench.py --build target/fast --label main --distances 32
python3 tools/pacing_bench.py --compare main mybranch

--size 320x180 keeps a software renderer (Mesa's lavapipe on a machine with no GPU) from hiding the CPU side; there only the spans' CPU times mean anything.

Results

Saved runs go to benchmarks/perf/; --table prints them as Markdown.

• Horizon v0.2.97 vs vanilla 26.3 on Windows 11, Ryzen 7 9800X3D + RTX 5080 (2026-10-29)