mirror of
https://github.com/feder-cr/Jobs_Applier_AI_Agent.git
synced 2026-11-09 10:09:48 -10:00
image-builder, engine, shared: the low findings of the second browser review (L5)
The review of199e8391and6b48fb1bleft one medium and seven low findings, each checked on the code. Finding -> outcome: 1. (medium) The browser error code to status table had a second hand-kept copy in the scheduler: already fixed by39ac3052(IDENTITY_ERROR_STATUS in packages/shared is the one owner, mirrored by protocol.py, parity tested). Checked, nothing redone. 2. GeoIP "rejected without an owner decision": confirmed, and fixed the way the owner decided (prefetch it in the golden image, design B7: a golden digest input). daijro/geoip-all-in-one is now pinned in pins.json by its exact release URL and the sha256 GitHub shows as the digest of the asset (measured: the file downloaded from the pinned URL hashes to it). The host downloads it with the other golden inputs and refuses a file that does not hash to the pin, before it boots anything; the seed carries the archive and GEOIP_TAG, GEOIP_ARCHIVE, GEOIP_SHA256; build-browser-env.sh checks the hash again, unpacks the archive into invisible_core's cache (cache_root()/geoip/<tag>) as dot and fails when the browser's own lookup would not find that file; provision.sh refuses an image whose cache holds another release than the pin. pins.json is part of the digest, so another release is another image, and the manifest records pinned.geoip. The script no longer calls ensure_geoip_mmdb, so a build fetches nothing by itself; the browser smoke fetches the same pinned archive and runs the same script (browser smoke 52 passed, the file landed in geoip/2026.09.30). A pin that upstream deleted answers 404 (it keeps only its latest two releases: measured, 2026.09.16 and older are gone): the build stops and says what to put into pins.json. What stays unpinned is what invisible-playwright itself does at a launch (fetch a newer release, keep the pinned one when GitHub cannot be reached); the README says so. The L2 text that called the file unpinned and unhashed on purpose is gone. 3. The proxy scrub replaced only the literal stored URL: confirmed, fixed at the origin (_scrub of browser.py, which every server answer passes through). The server splits the URL and unquotes the password, so the password alone (as written, decoded, encoded again), the user and the password together, and the Basic credentials of a header are now found with the URL in one pass, longest first, and put out in the redacted form. Tests: the fake MCP server can say each form (proxy_forms, echo_proxy_as); one test per form checks a failed launch, a failed navigate and a read; they all failed before the change. A proxy without a password is left alone. A short password garbles the page text it also occurs in: accepted, written in the code and in architecture 6. 4. A test that asserts nothing (test_a_provider_without_on_terminated_ignores_the_end_of_the_transport): confirmed, replaced by test_a_provider_without_on_terminated_does_nothing_about_the_end_of_the_transport, which checks no reconnect, no new connection, nothing closed, _connections unchanged and no task left running. Checked against a mutation of _transport_ended that starts a reconnect without on_terminated: the new test fails, the old one passed. 5. launch_failed in the scheduler's pass-through: confirmed, and the cause was the table, not the scheduler: no route of the engine launches (list, create, get, delete, frame, close raise none of it), so IDENTITY_ERROR_STATUS named a status no answer carries. launch_failed is out of the shared table and its mirror; browser.py has RouteErrorCode (what a route can fail with) and ErrorCode = RouteErrorCode plus launch_failed (what the model's tools fail with); test_protocol.py checks that the table's keys are RouteErrorCode and that launch_failed is the one code without a status. The scheduler test runs the codes of the table that are not 4xx, now busy, crashed and frame_failed. architecture 5.3 says it. 6. The postgres floor not raised: confirmed. Measured on postgres 18.4 with DATABASE_URL (packages/database packages/events apps/api apps/scheduler): 213 passed, 91 matching \bpg\b, against 205 and 84. Raised to what the job reaches. 7. The prepare-sleep browser budget had no shared owner or test: confirmed. PREPARE_SLEEP_TIMEOUT_MS (60 000) is in packages/shared protocol.ts and is the default of the scheduler's lifecycle and of GuestClient.prepareSleep (two literals gone); protocol.py mirrors it as PREPARE_SLEEP_TIMEOUT_S, tests/repo/vendored-nanobot.test.ts keeps them equal, and test_engine.py checks that STOP_GRACE_S + _CANCEL_WAIT_S + _CLOSE_BROWSERS_ON_SLEEP_S plus a 5 s margin fit inside it (today exactly 60). 8. Doc and smoke wording: confirmed. The sentence of architecture 8.7 is rewritten, and the smoke check label that said "the delete below" before the idle kill says "the idle kill below". Not findings, left as they are: the four unproven points of the review (idle death with the real server, what the real server echoes of the proxy, a SIGTERM during a prepare-sleep close, the window between the transport end and _close_ended). The scrub no longer depends on what the real server echoes: every form it can echo is hidden. Removed tests: test_a_provider_without_on_terminated_ignores_the_end_of_the_transport (replaced by the stronger test of finding 4, same behavior); test_every_error_code_of_a_browser_identity_has_one_status of test_protocol.py (replaced by the route-code version of finding 5); the GeoIP check of guest-files.test.ts that required ensure_geoip_mmdb (the script no longer fetches by itself; replaced by the checks of the pinned unpacking). Tests: engine pytest 2409 -> 2417 (7 scrub, 1 prepare-sleep budget), vitest 744 -> 750 on win32 (2 pins, 3 golden, 1 guest files), go unchanged, engine smoke 214 passed, browser smoke 52 passed. Floors raised in .github/test-floors.json: pytest 2409 -> 2417, vitest linux 741 -> 747 and win32 744 -> 750, postgres 205 -> 213 with \bpg\b 84 -> 91. No floor lowered.
This commit is contained in:
feder-cr committed
1 parent
39ac3052d6
commit
2e32efa27b
31 files changed
+446
-86
No files matched your search
@@ -1,17 +1,17 @@
|
||||
{
|
||||
"//": "What each suite must reach on each host, read by .github/scripts/test-guard.mjs in CI (tests.yml) and in the pre-push hook. Raise a floor when tests are added; lowering one is a decision to write down in the commit that does it. Linux skips only the tests that drive a real Windows PowerShell; a Windows host compiles dot-agentd's unix-only test files out and skips the two Go tests that need a Linux guest. The pytest floor is the Python engine's (invisible_engine_dots, the engine job of tests.yml, Linux only, nothing may skip).",
|
||||
"vitest": {
|
||||
"linux": { "min_passed": 741, "allow_skipped": "^generated PowerShell on a real Windows PowerShell " },
|
||||
"win32": { "min_passed": 744, "allow_skipped": "" }
|
||||
"linux": { "min_passed": 747, "allow_skipped": "^generated PowerShell on a real Windows PowerShell " },
|
||||
"win32": { "min_passed": 750, "allow_skipped": "" }
|
||||
},
|
||||
"postgres": {
|
||||
"linux": { "min_passed": 205, "allow_skipped": "", "require": { "\\bpg\\b": 84 } }
|
||||
"linux": { "min_passed": 213, "allow_skipped": "", "require": { "\\bpg\\b": 91 } }
|
||||
},
|
||||
"go": {
|
||||
"linux": { "min_passed": 98, "allow_skipped": "" },
|
||||
"win32": { "min_passed": 62, "allow_skipped": "^agentd (TestSystemRoute|TestPowerOffStartsTheCommandAndAnswers202)$" }
|
||||
},
|
||||
"pytest": {
|
||||
"linux": { "min_passed": 2409, "allow_skipped": "" }
|
||||
"linux": { "min_passed": 2417, "allow_skipped": "" }
|
||||
}
|
||||
}
|
||||
@@ -15,6 +15,7 @@ import type { Database } from "@invisible-dots/database";
|
||||
import type { EventLog } from "@invisible-dots/events";
|
||||
import {
|
||||
computerResources,
|
||||
PREPARE_SLEEP_TIMEOUT_MS,
|
||||
toRuntimeConfig,
|
||||
type DotState,
|
||||
type HealthAnswer,
|
||||
@@ -36,7 +37,7 @@ export interface LifecycleOptions {
|
||||
/** First retry delay of the event pump after a failure; doubles up to `pumpMaxRetryMs`. */
|
||||
pumpRetryMs: number;
|
||||
pumpMaxRetryMs: number;
|
||||
/** Time the agent gets to flush its state before a shutdown (section 9.5). */
|
||||
/** Time the agent gets to flush its state before a shutdown (section 9.5). Default PREPARE_SLEEP_TIMEOUT_MS of packages/shared, which the engine's own steps fit inside. */
|
||||
prepareSleepTimeoutMs: number;
|
||||
}
|
||||
|
||||
@@ -46,7 +47,7 @@ export const DEFAULT_LIFECYCLE_OPTIONS: LifecycleOptions = {
|
||||
healthRequestTimeoutMs: 5_000,
|
||||
pumpRetryMs: 1_000,
|
||||
pumpMaxRetryMs: 30_000,
|
||||
prepareSleepTimeoutMs: 60_000,
|
||||
prepareSleepTimeoutMs: PREPARE_SLEEP_TIMEOUT_MS,
|
||||
};
|
||||
|
||||
export interface LifecycleDeps {
|
||||
|
||||
@@ -398,7 +398,7 @@ describe.each(testAdapters())("Scheduler with a fake driver and a fake guest (%s
|
||||
guest.launchIdentity(identity.id);
|
||||
|
||||
const browserAnswers = Object.entries(IDENTITY_ERROR_STATUS).filter(([, status]) => status >= 500);
|
||||
expect(browserAnswers.map(([code]) => code).sort()).toEqual(["busy", "crashed", "frame_failed", "launch_failed"]);
|
||||
expect(browserAnswers.map(([code]) => code).sort()).toEqual(["busy", "crashed", "frame_failed"]);
|
||||
for (const [code, status] of browserAnswers) {
|
||||
guest.identityFault = code as IdentityErrorCode;
|
||||
await expect(scheduler.identityFrame(dot.id, identity.id)).rejects.toMatchObject({ status, code });
|
||||
|
||||
@@ -17,6 +17,7 @@ import {
|
||||
isFatalGuestError,
|
||||
parseOutboundEvent,
|
||||
pollGuestHealth,
|
||||
PREPARE_SLEEP_TIMEOUT_MS,
|
||||
SseParser,
|
||||
type AgentHealthAnswer,
|
||||
type AgentStateAnswer,
|
||||
@@ -328,7 +329,7 @@ export class GuestClient {
|
||||
}
|
||||
|
||||
/** Flushes state and closes browser sessions; can take a while with several browsers open. */
|
||||
prepareSleep(timeoutMs = 60_000): Promise<void> {
|
||||
prepareSleep(timeoutMs = PREPARE_SLEEP_TIMEOUT_MS): Promise<void> {
|
||||
return this.noContent({ method: "POST", path: this.agentPath(AGENT_ROUTES.prepareSleep), timeoutMs });
|
||||
}
|
||||
|
||||
|
||||
+13
-6
@@ -647,10 +647,11 @@ command detached into another session outlives it.
|
||||
The identity routes and the model's identity tools are one code path, the
|
||||
`BrowserManager` (section 6), so its limits hold for both. A route answers an
|
||||
error as `{ error: <code>, message }` with the status of its code: `invalid`
|
||||
400, `not_found` 404, `limit` and `not_open` 409, `busy` 503, `launch_failed`,
|
||||
`crashed` and `frame_failed` 502. The engine has no route that launches an
|
||||
400, `not_found` 404, `limit` and `not_open` 409, `busy` 503, `crashed` and
|
||||
`frame_failed` 502. The engine has no route that launches an
|
||||
identity: only the model's tools open a browser, so `browser.identity.launch`
|
||||
alone decides whether one starts. The host may look at an open identity (the
|
||||
alone decides whether one starts, and `launch_failed` is an answer of those tools,
|
||||
never of a route. The host may look at an open identity (the
|
||||
frame) and close it (or delete it); these are the owner's actions, not the Dot's, so
|
||||
no permission of the Dot applies to them. A frame is no use of the identity: it does
|
||||
not move it in the least-recently-used order, does not touch `last_used_at`,
|
||||
@@ -1123,8 +1124,10 @@ starts it again. What it guarantees:
|
||||
flight and give a tool in flight up to 20 seconds to finish and commit its
|
||||
result, then close the open browsers: at most 4 s on SIGTERM, which leaves about
|
||||
6 of systemd's `TimeoutStopSec=30` to checkpoint the database, and up to a
|
||||
browser's own 30 s close on prepare-sleep, which only the host's 60 s bound (20 s
|
||||
grace, 5 s cancel wait and 30 s stay inside it). A tool cut at
|
||||
browser's own 30 s close on prepare-sleep: only the host's 60 s bounds it, and
|
||||
the 20 s of grace, the 5 s of cancel wait and the 30 s close fit inside, with 5 s
|
||||
to spare for the flush (`PREPARE_SLEEP_TIMEOUT_MS` of packages/shared, with the sum
|
||||
checked by a test of the engine). A tool cut at
|
||||
the grace keeps its intent, and the next entry of its unit reports it as
|
||||
interrupted. Measured in the engine smoke before the browsers existed, with a
|
||||
task's `sleep 70` still running at SIGTERM and the event stream connected: the
|
||||
@@ -1317,7 +1320,11 @@ state.
|
||||
when it happens, from the client's transport, so a process that dies while idle
|
||||
is closed at once, frees its slot of `max_open`, and the next action says
|
||||
`not_open`; a file never claims an open browser for a process that is gone. Text
|
||||
a page tool returns, and its errors, have the proxy in its redacted form. A close calls
|
||||
a page tool returns, and its errors, have the proxy in its redacted form and its
|
||||
password in none: the server splits the URL into a user and a decoded password, so
|
||||
the password alone, as written, decoded or encoded again, the user with it, and the
|
||||
Basic credentials of a header are hidden as well (a short password garbles the
|
||||
page text it also occurs in, which is the price of one that leaves nowhere). A close calls
|
||||
`browser_close` first, so Firefox flushes its profile, then ends the process.
|
||||
Every `browser.identity.*` event commits with the row change it describes.
|
||||
The model's identity and page tools (`browser_tools.py`) and the routes of
|
||||
|
||||
@@ -15,16 +15,17 @@ image's SHA-256; `invisible-dots doctor` checks an image against it with
|
||||
`buildGoldenImage({ qemu, accelerator, runner })`:
|
||||
|
||||
1. Downloads the Ubuntu 24.04 cloud image pinned in
|
||||
`virtualization/images/base.json`, and the Node and uv tarballs pinned in
|
||||
`pins.json`, with Node's fetch. Each published checksum list (`SHA256SUMS`,
|
||||
`SHASUMS256.txt`, uv's `.sha256`) must name the pinned hash before the
|
||||
download starts, and the downloaded bytes must hash to it (Node crypto).
|
||||
`virtualization/images/base.json`, the Node and uv tarballs and the GeoIP
|
||||
database pinned in `pins.json`, with Node's fetch. Each published checksum list
|
||||
(`SHA256SUMS`, `SHASUMS256.txt`, uv's `.sha256`) must name the pinned hash
|
||||
before the download starts, and the downloaded bytes must hash to it (Node
|
||||
crypto). The GeoIP archive has no published list, so its pin alone decides.
|
||||
Cached copies are re-hashed before every build.
|
||||
2. Copies the cloud image and grows it with `qemu-img resize` (default 10G;
|
||||
each Dot's overlay is larger and cloud-init grows the filesystem).
|
||||
3. Writes the builder seed with `@invisible-dots/iso`: one ISO labelled
|
||||
`cidata` holding `user-data`, `meta-data`, `provision.sh`, `pins.env`,
|
||||
`mcp-requirements.lock`, both tarballs, `engine-requirements.lock`,
|
||||
`mcp-requirements.lock`, both tarballs, the GeoIP archive, `engine-requirements.lock`,
|
||||
`build-engine-env.sh` and `build-browser-env.sh`. The engine's own source is
|
||||
not on it: it is ours, so it travels on the runtime ISO.
|
||||
4. Boots it once with the QEMU the host runs Dots with, on the accelerator
|
||||
@@ -33,24 +34,34 @@ image's SHA-256; `invisible-dots doctor` checks an image against it with
|
||||
from the vm-manager's `machineArgs()`). `provision.sh` installs the
|
||||
apt packages (Xvfb, a minimal XFCE, the browser's libraries, ImageMagick
|
||||
for dot-agentd's screenshots), Node, uv, then builds the Dot's browser with
|
||||
`build-browser-env.sh <lock> ~dot/.local/share/invisible-dots/mcp`, as user
|
||||
`build-browser-env.sh <lock> ~dot/.local/share/invisible-dots/mcp <GeoIP
|
||||
archive> <tag> <sha256>`, as user
|
||||
`dot`: a virtual environment filled with
|
||||
`uv pip install --require-hashes -r mcp-requirements.lock`, its
|
||||
`invisible-playwright-mcp` linked into `~/.local/bin`, and
|
||||
`invisible-playwright fetch` run from that environment, so the cached browser
|
||||
engine is the one the MCP server expects. The same script then fetches the
|
||||
GeoIP database that a launch with the timezone left to `auto` needs (the
|
||||
latest build of `daijro/geoip-all-in-one` on the day of the build), so a Dot's
|
||||
first launch downloads nothing; its release is recorded in the manifest as
|
||||
`geoip-database`, and a launch checks for a newer one and keeps this one when
|
||||
GitHub cannot be reached. This one artifact is not pinned and not hashed, on
|
||||
purpose: `daijro/geoip-all-in-one` is rebuilt weekly and keeps only its latest
|
||||
releases, so a pinned tag answers 404 within weeks and would fail every
|
||||
build, and invisible-playwright itself never pins it (it resolves the latest
|
||||
tag on every launch). It is a data file for the timezone lookup, not code,
|
||||
and it is not an input of the golden digest, so two builds from one seed can
|
||||
carry different weeks of it; the manifest records which one each did. The
|
||||
browser smoke runs the same script. The only
|
||||
engine is the one the MCP server expects. The same script then unpacks the
|
||||
GeoIP database that a launch with the timezone left to `auto` needs, so a
|
||||
Dot's first launch downloads nothing. It is one release of
|
||||
`daijro/geoip-all-in-one`, pinned in `pins.json` by its exact URL and the
|
||||
SHA-256 of that file (the `digest` GitHub shows for the release asset): the
|
||||
host checks the hash when it downloads the archive, the script checks it
|
||||
again before it unpacks it into the cache directory the browser reads (and
|
||||
fails when the browser's own lookup would not find that file), and the pin is
|
||||
part of the golden digest, so another release is another image. The release is
|
||||
recorded in the manifest (`pinned.geoip`, and `geoip-database` among the
|
||||
installed components). What is not pinned is what happens later: a launch
|
||||
asks GitHub for the latest release and fetches it when it is newer than the
|
||||
one in the cache (a data file for the timezone lookup, fetched by
|
||||
invisible-playwright itself), and keeps the pinned one when GitHub cannot be
|
||||
reached. The cost of the pin: `daijro/geoip-all-in-one` is rebuilt weekly and
|
||||
keeps only its latest two releases, so a pin that is not refreshed answers
|
||||
404 within weeks. The build then stops and says what to change: put the
|
||||
current tag, URL and `digest` of `geoip-aio-all.mmdb.zip`
|
||||
(`https://api.github.com/repos/daijro/geoip-all-in-one/releases/latest`) into
|
||||
`pins.json`. A host that already built with the pin keeps the verified archive
|
||||
in `images/.cache` and does not need the network for it. The
|
||||
browser smoke runs the same script on the same pinned archive. The only
|
||||
browser a Dot has is that server: a test refuses any other browser or browser
|
||||
library among the apt packages, the lock and the build scripts. `provision.sh`
|
||||
then builds the engine's Python
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
# builder VM, and the browser smoke runs it on a plain ubuntu:24.04 container, so both build
|
||||
# the same thing. The Dot's browser is this server and nothing else (owner's rule).
|
||||
#
|
||||
# build-browser-env.sh <lock> <environment directory>
|
||||
# build-browser-env.sh <lock> <environment directory> <GeoIP zip> <GeoIP release tag> <GeoIP sha256>
|
||||
#
|
||||
# The command is linked into ~dot/.local/bin, which the units put on the PATH of dot-agentd and
|
||||
# of the engine.
|
||||
@@ -15,25 +15,34 @@
|
||||
# --require-hashes), so a file that changed on the index, or a dependency missing from the
|
||||
# lock, fails the build instead of entering the image. The engine is fetched with the
|
||||
# invisible-playwright of that environment, so the cached engine is the one the server's seal
|
||||
# expects; it and the GeoIP file land in ~dot/.cache/invisible-playwright. The GeoIP file is
|
||||
# the latest build of daijro/geoip-all-in-one on the day of the build: a launch checks for a
|
||||
# newer one and keeps this one when it cannot reach GitHub. It is deliberately not pinned or
|
||||
# hashed (upstream deletes old releases, so a pin would break the build; see the README).
|
||||
# expects; it and the GeoIP file land in ~dot/.cache/invisible-playwright. The GeoIP file is the
|
||||
# release of daijro/geoip-all-in-one that pins.json names: whoever runs this (the image builder on
|
||||
# the host, the smoke on its own) fetches it and checks its SHA-256, and it is checked against the
|
||||
# same hash here again before it is unpacked into the cache directory the browser reads. A launch
|
||||
# may later fetch a newer release by itself and keeps this one when it cannot reach GitHub (the
|
||||
# README says so).
|
||||
set -Eeuo pipefail
|
||||
|
||||
usage="usage: build-browser-env.sh <lock> <environment directory>"
|
||||
usage="usage: build-browser-env.sh <lock> <environment directory> <GeoIP zip> <GeoIP release tag> <GeoIP sha256>"
|
||||
lock="${1:?$usage}"
|
||||
env_dir="${2:?$usage}"
|
||||
geoip_zip="${3:?$usage}"
|
||||
geoip_tag="${4:?$usage}"
|
||||
geoip_sha256="${5:?$usage}"
|
||||
bin_dir=/home/dot/.local/bin
|
||||
[ -f "$lock" ] || { echo "build-browser-env: no lock at $lock" >&2; exit 1; }
|
||||
[ -f "$geoip_zip" ] || { echo "build-browser-env: no GeoIP archive at $geoip_zip" >&2; exit 1; }
|
||||
echo "$geoip_sha256 $geoip_zip" | sha256sum --check --status - || { echo "build-browser-env: $geoip_zip is not the GeoIP release pinned as $geoip_tag (sha256 $geoip_sha256)" >&2; exit 1; }
|
||||
id dot >/dev/null 2>&1 || { echo "build-browser-env: user dot does not exist" >&2; exit 1; }
|
||||
|
||||
as_dot() { sudo -u dot -H env PATH="/home/dot/.local/bin:/usr/local/bin:/usr/bin:/bin" "$@"; }
|
||||
|
||||
# dot reads the lock from a file of its own reach.
|
||||
dot_lock="$(mktemp)"
|
||||
trap 'rm -f "$dot_lock"' EXIT
|
||||
dot_geoip="$(mktemp)"
|
||||
trap 'rm -f "$dot_lock" "$dot_geoip"' EXIT
|
||||
install -m 0644 "$lock" "$dot_lock"
|
||||
install -m 0644 "$geoip_zip" "$dot_geoip"
|
||||
|
||||
as_dot mkdir -p "$bin_dir" "$(dirname "$env_dir")"
|
||||
as_dot uv venv --quiet --python /usr/bin/python3 "$env_dir"
|
||||
@@ -42,5 +51,25 @@ as_dot uv pip install --python "$env_dir/bin/python" --require-hashes -r "$dot_l
|
||||
[ -x "$env_dir/bin/invisible-playwright-mcp" ] || { echo "build-browser-env: no invisible-playwright-mcp in $env_dir/bin" >&2; exit 1; }
|
||||
as_dot ln -sfn "$env_dir/bin/invisible-playwright-mcp" "$bin_dir/invisible-playwright-mcp"
|
||||
as_dot "$env_dir/bin/invisible-playwright" fetch
|
||||
as_dot "$env_dir/bin/python" -c 'from invisible_core.download import ensure_geoip_mmdb; print(ensure_geoip_mmdb())'
|
||||
# The pinned database goes where the browser looks for it (invisible_core's cache, one directory per
|
||||
# release tag), as dot, and the browser's own lookup has to find exactly that file: a layout that
|
||||
# moved fails here, not at a launch.
|
||||
as_dot "$env_dir/bin/python" - "$dot_geoip" "$geoip_tag" <<'PYTHON'
|
||||
import sys
|
||||
import zipfile
|
||||
|
||||
from invisible_core.constants import GEOIP_MMDB_NAME
|
||||
from invisible_core.download import cache_root, geoip_mmdb_path
|
||||
|
||||
archive, tag = sys.argv[1], sys.argv[2]
|
||||
target = cache_root() / "geoip" / tag
|
||||
with zipfile.ZipFile(archive) as bundle:
|
||||
if bundle.namelist() != [GEOIP_MMDB_NAME]:
|
||||
raise SystemExit(f"build-browser-env: {archive} holds {bundle.namelist()}, not just {GEOIP_MMDB_NAME}")
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
bundle.extract(GEOIP_MMDB_NAME, target)
|
||||
if geoip_mmdb_path() != target / GEOIP_MMDB_NAME:
|
||||
raise SystemExit(f"build-browser-env: the browser finds {geoip_mmdb_path()}, not the pinned {target / GEOIP_MMDB_NAME}")
|
||||
print(geoip_mmdb_path())
|
||||
PYTHON
|
||||
as_dot sh -c 'command -v invisible-playwright-mcp' >/dev/null || { echo "build-browser-env: invisible-playwright-mcp is not on the PATH of dot" >&2; exit 1; }
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# Runs as root inside the builder VM, from the builder's seed disk (label
|
||||
# cidata), which also carries pins.env and the Node and uv tarballs. Installs
|
||||
# cidata), which also carries pins.env, the Node and uv tarballs and the GeoIP archive. Installs
|
||||
# everything the golden image carries (architecture section 3.3), then wipes
|
||||
# the instance state so every Dot boots as a fresh cloud-init instance, and
|
||||
# powers off.
|
||||
@@ -62,19 +62,19 @@ step "installing invisible-playwright-mcp $MCP_VERSION, fetching the browser eng
|
||||
# lock names and with a file whose SHA-256 it lists, nothing resolved from the index. The engine is
|
||||
# fetched with the invisible-playwright of that environment, so the cached engine is the one the
|
||||
# server's seal expects, and the GeoIP database that a launch with the timezone left to "auto" needs
|
||||
# is fetched beside it, so a Dot's first launch downloads nothing. Both land in
|
||||
# ~dot/.cache/invisible-playwright.
|
||||
# is unpacked beside it from the pinned release the seed carries (hash-checked on the host and again by
|
||||
# the script), so a Dot's first launch downloads nothing. Both land in ~dot/.cache/invisible-playwright.
|
||||
as_dot() { sudo -u dot -H env PATH="/home/dot/.local/bin:/usr/local/bin:/usr/bin:/bin" "$@"; }
|
||||
mcp_env=/home/dot/.local/share/invisible-dots/mcp
|
||||
bash "$payload/$BROWSER_BUILD" "$payload/$PYTHON_LOCK" "$mcp_env"
|
||||
bash "$payload/$BROWSER_BUILD" "$payload/$PYTHON_LOCK" "$mcp_env" "$payload/$GEOIP_ARCHIVE" "$GEOIP_TAG" "$GEOIP_SHA256"
|
||||
# `version` prints the wrapper on its first line and the engine on the line
|
||||
# that starts with "engine" (tag, Firefox version, BuildID); the engine line
|
||||
# is the one that says which browser the image carries.
|
||||
engine_version="$(as_dot "$mcp_env/bin/invisible-playwright" version | sed -n 's/^engine[[:space:]]*//p')"
|
||||
[ -n "$engine_version" ] || { console "invisible-playwright version printed no engine line"; false; }
|
||||
# The GeoIP database is named by its release tag (a date), the directory it was cached in.
|
||||
# The GeoIP database is named by its release tag (a date), the directory it was cached in: the pinned one.
|
||||
geoip_database="$(as_dot "$mcp_env/bin/python" -c 'from invisible_core.download import geoip_mmdb_path; print(geoip_mmdb_path().parent.name)')"
|
||||
[ -n "$geoip_database" ] || { console "no GeoIP database in the cache of invisible-playwright"; false; }
|
||||
[ "$geoip_database" = "$GEOIP_TAG" ] || { console "the cache of invisible-playwright holds GeoIP release '$geoip_database', not the pinned $GEOIP_TAG"; false; }
|
||||
|
||||
step "installing the engine's Python environment"
|
||||
# The engine's third-party dependencies come from the hashed lock on the seed,
|
||||
|
||||
@@ -13,6 +13,11 @@
|
||||
"shasums_entry": "uv-x86_64-unknown-linux-gnu.tar.gz",
|
||||
"sha256": "b9980552309f09c15172b8be828555e375097f16deb459795ce7bfd200380f0b"
|
||||
},
|
||||
"geoip": {
|
||||
"tag": "2026.09.30",
|
||||
"url": "https://github.com/daijro/geoip-all-in-one/releases/download/2026.09.30/geoip-aio-all.mmdb.zip",
|
||||
"sha256": "09af448e51e2f012c87e4ade1b39f93f999c32ba6205e73e25b98e3033cf95ee"
|
||||
},
|
||||
"apt_packages": [
|
||||
"xvfb",
|
||||
"xfce4-session",
|
||||
|
||||
@@ -10,10 +10,10 @@ import { basename, join } from "node:path";
|
||||
import { writeIso } from "@invisible-dots/iso";
|
||||
import { hostPaths, replaceFile, type HostPaths } from "@invisible-dots/shared";
|
||||
import { BUILDER_BROWSER_BUILD, BUILDER_ENGINE_BUILD, BUILDER_ENGINE_LOCK, BUILDER_PROVISION, BUILDER_PYTHON_LOCK, BUILDER_USER_DATA, defaultAssetRoot, readGuestAsset } from "./assets.js";
|
||||
import { fetchVerified, sha256File, type Fetch, type FetchVerifiedOptions } from "./download.js";
|
||||
import { DownloadError, fetchVerified, sha256File, type Fetch, type FetchVerifiedOptions } from "./download.js";
|
||||
import { acquireLock, type Lock } from "./lock.js";
|
||||
import { manifestPathFor, writeManifest, type GoldenManifest } from "./manifest.js";
|
||||
import { BASE_IMAGE, downloadFileName, GUEST_PINS, type BaseImagePin, type GuestPins, type PinnedDownload } from "./pins.js";
|
||||
import { BASE_IMAGE, downloadFileName, GUEST_PINS, type BaseImagePin, type GeoipPin, type GuestPins, type PinnedDownload } from "./pins.js";
|
||||
import { waitForExit, type ProcessRunner } from "./process.js";
|
||||
import { parseHashedLock, parsePythonLock, type PythonLock } from "./python-lock.js";
|
||||
import { builderQemuArgs, type Accelerator, type QemuPrograms } from "./qemu.js";
|
||||
@@ -179,6 +179,7 @@ async function buildLocked(options: GoldenBuildOptions & Resolved, target: Targe
|
||||
};
|
||||
const nodeTarball = await fetchPinned(pins.node);
|
||||
const uvTarball = await fetchPinned(pins.uv);
|
||||
const geoipArchive = await fetchGeoip(pins.geoip, join(cacheDir, downloadFileName(pins.geoip)), downloads);
|
||||
|
||||
// Next to the images, not in the system temp directory: the disk grows to
|
||||
// several GiB and the final rename stays on one filesystem.
|
||||
@@ -207,6 +208,7 @@ async function buildLocked(options: GoldenBuildOptions & Resolved, target: Targe
|
||||
python: target.python,
|
||||
nodeTarball,
|
||||
uvTarball,
|
||||
geoipArchive,
|
||||
engineLock: target.engineLock,
|
||||
engineBuild: target.engineBuild,
|
||||
browserBuild: target.browserBuild,
|
||||
@@ -239,6 +241,7 @@ async function buildLocked(options: GoldenBuildOptions & Resolved, target: Targe
|
||||
pinned: {
|
||||
node: { version: pins.node.version, sha256: pins.node.sha256, url: pins.node.url },
|
||||
uv: { version: pins.uv.version, sha256: pins.uv.sha256, url: pins.uv.url },
|
||||
geoip: { ...pins.geoip },
|
||||
"invisible-playwright-mcp": target.python.mcpVersion,
|
||||
"invisible-playwright": target.python.playwrightVersion,
|
||||
"mcp-requirements.lock": createHash("sha256").update(target.pythonLock).digest("hex"),
|
||||
@@ -263,6 +266,27 @@ async function buildLocked(options: GoldenBuildOptions & Resolved, target: Targe
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The pinned GeoIP release, hashed against its pin like the other inputs. The project publishes no checksum list, so
|
||||
* the pin alone decides; and it keeps only its latest releases, so a 404 means the pin has to be refreshed.
|
||||
*/
|
||||
async function fetchGeoip(pin: GeoipPin, dest: string, options: FetchVerifiedOptions): Promise<string> {
|
||||
try {
|
||||
await fetchVerified({ url: pin.url, sha256: pin.sha256 }, dest, options);
|
||||
} catch (error) {
|
||||
if (error instanceof DownloadError && error.message.includes("HTTP 404")) {
|
||||
throw new DownloadError(
|
||||
`${error.message}: the pinned GeoIP release ${pin.tag} is gone from GitHub (daijro/geoip-all-in-one keeps only its latest releases). ` +
|
||||
`Pin the current one in guest/image-builder/pins.json: its tag, its URL and the "digest" GitHub shows for geoip-aio-all.mmdb.zip at ` +
|
||||
"https://api.github.com/repos/daijro/geoip-all-in-one/releases/latest",
|
||||
{ cause: error },
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
return dest;
|
||||
}
|
||||
|
||||
/** Boots the builder VM, reports its progress, and returns the whole serial console once it has stopped. */
|
||||
async function runBuilderVm(
|
||||
options: GoldenBuildOptions & Resolved,
|
||||
|
||||
@@ -21,7 +21,7 @@ export {
|
||||
type ImageManifest,
|
||||
type RuntimeManifest,
|
||||
} from "./manifest.js";
|
||||
export { BASE_IMAGE, GUEST_PINS, type BaseImagePin, type GuestPins, type PinnedDownload } from "./pins.js";
|
||||
export { BASE_IMAGE, GUEST_PINS, type BaseImagePin, type GeoipPin, type GuestPins, type PinnedDownload } from "./pins.js";
|
||||
export { normalizePackageName, parseHashedLock, parsePythonLock, type PythonLock } from "./python-lock.js";
|
||||
export type { ProcessRunner, RunningProcess } from "./process.js";
|
||||
export { builderQemuArgs, type Accelerator, type QemuPrograms } from "./qemu.js";
|
||||
|
||||
@@ -8,7 +8,7 @@ import { randomBytes } from "node:crypto";
|
||||
import { readFile, rm, stat, writeFile } from "node:fs/promises";
|
||||
import { basename, dirname, join } from "node:path";
|
||||
import { sha256File } from "./download.js";
|
||||
import type { BaseImagePin } from "./pins.js";
|
||||
import type { BaseImagePin, GeoipPin } from "./pins.js";
|
||||
import { replaceFile } from "@invisible-dots/shared";
|
||||
|
||||
export interface PinnedComponent {
|
||||
@@ -34,6 +34,8 @@ export interface GoldenManifest {
|
||||
pinned: {
|
||||
node: PinnedComponent;
|
||||
uv: PinnedComponent;
|
||||
/** The GeoIP release the image carries, checked against this hash at the build (the browser's own launch may fetch a newer one). */
|
||||
geoip: GeoipPin;
|
||||
"invisible-playwright-mcp": string;
|
||||
"invisible-playwright": string;
|
||||
/** SHA-256 of builder/mcp-requirements.lock: the whole Python environment, transitive packages included. */
|
||||
|
||||
@@ -18,6 +18,19 @@ export interface PinnedDownload {
|
||||
sha256: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* The GeoIP database a launch with the timezone left to "auto" needs: one release of daijro/geoip-all-in-one, by its
|
||||
* exact URL and the SHA-256 of that file. The project publishes no checksum list, so the pin is the only record: it
|
||||
* is the `digest` GitHub shows for the release asset (api.github.com/repos/daijro/geoip-all-in-one/releases/latest).
|
||||
* Upstream keeps only its latest releases, so a pin that is not refreshed answers 404 within weeks (see the README).
|
||||
*/
|
||||
export interface GeoipPin {
|
||||
/** The release tag, a date; it names the directory the database is cached in. */
|
||||
tag: string;
|
||||
url: string;
|
||||
sha256: string;
|
||||
}
|
||||
|
||||
export interface BaseImagePin {
|
||||
name: string;
|
||||
release: string;
|
||||
@@ -37,6 +50,7 @@ export interface BaseImagePin {
|
||||
export interface GuestPins {
|
||||
node: PinnedDownload;
|
||||
uv: PinnedDownload;
|
||||
geoip: GeoipPin;
|
||||
apt_packages: string[];
|
||||
}
|
||||
|
||||
@@ -89,8 +103,21 @@ function parseDownload(value: unknown, where: string): PinnedDownload {
|
||||
return pin;
|
||||
}
|
||||
|
||||
/** The release asset that holds the database, and where a release of the project lives. */
|
||||
const GEOIP_ASSET = "geoip-aio-all.mmdb.zip";
|
||||
const GEOIP_RELEASES = "https://github.com/daijro/geoip-all-in-one/releases/download";
|
||||
|
||||
function parseGeoip(value: unknown, where: string): GeoipPin {
|
||||
const o = asObject(value, where);
|
||||
const tag = plainWord(field(o, "tag", where), `${where}.tag`);
|
||||
const url = https(field(o, "url", where), `${where}.url`);
|
||||
const expected = `${GEOIP_RELEASES}/${tag}/${GEOIP_ASSET}`;
|
||||
if (url !== expected) throw new Error(`${where}.url must be ${expected}`);
|
||||
return { tag, url, sha256: sha256(field(o, "sha256", where), `${where}.sha256`) };
|
||||
}
|
||||
|
||||
/** The name a pinned download is stored under in the cache and on the builder seed. */
|
||||
export function downloadFileName(pin: PinnedDownload): string {
|
||||
export function downloadFileName(pin: { url: string }): string {
|
||||
return new URL(pin.url).pathname.split("/").pop() ?? "";
|
||||
}
|
||||
|
||||
@@ -120,6 +147,7 @@ export function parseGuestPins(value: unknown): GuestPins {
|
||||
return {
|
||||
node: parseDownload(o.node, `${where} node`),
|
||||
uv: parseDownload(o.uv, `${where} uv`),
|
||||
geoip: parseGeoip(o.geoip, `${where} geoip`),
|
||||
apt_packages: apt.map((name, i) => plainWord(typeof name === "string" ? name : "", `${where} apt_packages[${i}]`)),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -27,6 +27,8 @@ export interface BuilderSeedInput {
|
||||
/** Host paths of the verified tarballs, streamed into the image. */
|
||||
nodeTarball: string;
|
||||
uvTarball: string;
|
||||
/** The verified zip of the GeoIP database (pins.geoip), streamed into the image. */
|
||||
geoipArchive: string;
|
||||
/** builder/engine-requirements.lock and builder/build-engine-env.sh, as read. */
|
||||
engineLock: Uint8Array;
|
||||
engineBuild: Uint8Array;
|
||||
@@ -46,6 +48,9 @@ export function pinsEnv(pins: GuestPins, python: PythonLock): string {
|
||||
["NODE_TARBALL", downloadFileName(pins.node)],
|
||||
["UV_VERSION", pins.uv.version],
|
||||
["UV_TARBALL", downloadFileName(pins.uv)],
|
||||
["GEOIP_TAG", pins.geoip.tag],
|
||||
["GEOIP_ARCHIVE", downloadFileName(pins.geoip)],
|
||||
["GEOIP_SHA256", pins.geoip.sha256],
|
||||
["MCP_VERSION", python.mcpVersion],
|
||||
["PLAYWRIGHT_VERSION", python.playwrightVersion],
|
||||
["PYTHON_LOCK", SEED_PYTHON_LOCK],
|
||||
@@ -76,6 +81,7 @@ export function builderSeedEntries(input: BuilderSeedInput): IsoEntry[] {
|
||||
{ path: SEED_PYTHON_LOCK, data: input.pythonLock },
|
||||
{ path: downloadFileName(input.pins.node), file: input.nodeTarball },
|
||||
{ path: downloadFileName(input.pins.uv), file: input.uvTarball },
|
||||
{ path: downloadFileName(input.pins.geoip), file: input.geoipArchive },
|
||||
{ path: SEED_ENGINE_LOCK, data: input.engineLock },
|
||||
{ path: SEED_ENGINE_BUILD, data: input.engineBuild },
|
||||
{ path: SEED_BROWSER_BUILD, data: input.browserBuild },
|
||||
|
||||
@@ -15,8 +15,12 @@ import { sha256, startFakeHttp, type FakeHttp } from "./http-fixture.js";
|
||||
const BASE = Buffer.from("QFI\xfb pretend qcow2 cloud image ".repeat(4000), "latin1");
|
||||
const NODE = Buffer.from("node tarball ".repeat(1000));
|
||||
const UV = Buffer.from("uv tarball ".repeat(1000));
|
||||
const GEOIP = Buffer.from("geoip archive ".repeat(1000));
|
||||
/** The next weekly release of the database, which the server also has. */
|
||||
const GEOIP_NEXT = Buffer.from("the next week of geoip ".repeat(1000));
|
||||
const NODE_FILE = "node-v24.21.0-linux-x64.tar.xz";
|
||||
const UV_FILE = "uv-x86_64-unknown-linux-gnu.tar.gz";
|
||||
const GEOIP_FILE = "geoip-aio-all.mmdb.zip";
|
||||
|
||||
const OK_CONSOLE = [
|
||||
"[ 0.000000] Linux version 6.8.0",
|
||||
@@ -43,6 +47,10 @@ beforeEach(async () => {
|
||||
"/node/SHASUMS256.txt": { body: `${sha256(NODE)} ${NODE_FILE}\n` },
|
||||
[`/uv/${UV_FILE}`]: { body: UV },
|
||||
[`/uv/${UV_FILE}.sha256`]: { body: `${sha256(UV)} *${UV_FILE}\n` },
|
||||
[`/geoip/2026.09.30/${GEOIP_FILE}`]: { body: GEOIP },
|
||||
[`/geoip/2026.10.07/${GEOIP_FILE}`]: { body: GEOIP_NEXT },
|
||||
// A release whose bytes are not the ones its pin names.
|
||||
[`/geoip/swapped/${GEOIP_FILE}`]: { body: Buffer.from("not the release that was pinned") },
|
||||
});
|
||||
home = mkdtempSync(join(tmpdir(), "idots-golden-"));
|
||||
paths = hostPaths({ INVISIBLE_DOTS_HOME: home });
|
||||
@@ -59,6 +67,7 @@ beforeEach(async () => {
|
||||
pins = {
|
||||
node: { version: "24.21.0", url: http.url(`/node/${NODE_FILE}`), shasums_url: http.url("/node/SHASUMS256.txt"), shasums_entry: NODE_FILE, sha256: sha256(NODE) },
|
||||
uv: { version: "0.12.22", url: http.url(`/uv/${UV_FILE}`), shasums_url: http.url(`/uv/${UV_FILE}.sha256`), shasums_entry: UV_FILE, sha256: sha256(UV) },
|
||||
geoip: { tag: "2026.09.30", url: http.url(`/geoip/2026.09.30/${GEOIP_FILE}`), sha256: sha256(GEOIP) },
|
||||
apt_packages: ["xvfb", "imagemagick"],
|
||||
};
|
||||
logs = [];
|
||||
@@ -112,8 +121,8 @@ describe("buildGoldenImage", () => {
|
||||
expect(runner.spawns[0]!.args).toEqual(
|
||||
expect.arrayContaining(["-accel", "kvm", "-cpu", "host", "-m", String(GOLDEN_DEFAULTS.memoryMib), "-smp", String(GOLDEN_DEFAULTS.cpus)]),
|
||||
);
|
||||
// The seed carried both tarballs.
|
||||
expect(seedSize).toBeGreaterThan(NODE.length + UV.length);
|
||||
// The seed carried both tarballs and the GeoIP archive.
|
||||
expect(seedSize).toBeGreaterThan(NODE.length + UV.length + GEOIP.length);
|
||||
|
||||
expect(await readFile(result.image)).toEqual(BASE);
|
||||
expect((await stat(result.image)).mode & 0o222).toBe(0);
|
||||
@@ -130,6 +139,7 @@ describe("buildGoldenImage", () => {
|
||||
pinned: {
|
||||
node: { version: "24.21.0", sha256: sha256(NODE) },
|
||||
uv: { version: "0.12.22", sha256: sha256(UV) },
|
||||
geoip: { tag: "2026.09.30", sha256: sha256(GEOIP), url: http.url(`/geoip/2026.09.30/${GEOIP_FILE}`) },
|
||||
"invisible-playwright-mcp": "0.70.2",
|
||||
"invisible-playwright": "0.25.7",
|
||||
apt_packages: ["xvfb", "imagemagick"],
|
||||
@@ -147,7 +157,7 @@ describe("buildGoldenImage", () => {
|
||||
expect((await readdir(paths.imagesDir)).sort()).toEqual(
|
||||
[".cache", "noble-server-cloudimg-amd64.img", `golden-${result.version}.json`, `golden-${result.version}.qcow2`].sort(),
|
||||
);
|
||||
expect((await readdir(join(paths.imagesDir, ".cache"))).sort()).toEqual([NODE_FILE, UV_FILE]);
|
||||
expect((await readdir(join(paths.imagesDir, ".cache"))).sort()).toEqual([GEOIP_FILE, NODE_FILE, UV_FILE]);
|
||||
});
|
||||
|
||||
it("does nothing when an image for the same inputs exists", async () => {
|
||||
@@ -167,6 +177,37 @@ describe("buildGoldenImage", () => {
|
||||
expect(second.version).not.toBe(first.version);
|
||||
});
|
||||
|
||||
it("builds a new version when only the GeoIP release changes, and records the release", async () => {
|
||||
const first = await buildGoldenImage(options({ console: OK_CONSOLE, exit: 0 }).opts);
|
||||
const geoip = { tag: "2026.10.07", url: http.url(`/geoip/2026.10.07/${GEOIP_FILE}`), sha256: sha256(GEOIP_NEXT) };
|
||||
|
||||
const second = await buildGoldenImage(options({ console: OK_CONSOLE, exit: 0 }, { pins: { ...pins, geoip } }).opts);
|
||||
|
||||
expect(second.created).toBe(true);
|
||||
expect(second.version).not.toBe(first.version);
|
||||
expect(((await readManifest(second.manifest)) as GoldenManifest).pinned.geoip).toEqual(geoip);
|
||||
});
|
||||
|
||||
it("refuses a GeoIP file that does not hash to its pin, before booting anything", async () => {
|
||||
const swapped = { ...pins.geoip, url: http.url(`/geoip/swapped/${GEOIP_FILE}`) };
|
||||
const { runner, opts } = options({ console: OK_CONSOLE, exit: 0 }, { pins: { ...pins, geoip: swapped } });
|
||||
|
||||
await expect(buildGoldenImage(opts)).rejects.toThrow(/geoip-aio-all\.mmdb\.zip hashes to [0-9a-f]{64}, but the pin is/);
|
||||
|
||||
expect(runner.spawns).toHaveLength(0);
|
||||
expect((await readdir(paths.imagesDir)).filter((name) => name.startsWith("golden-"))).toEqual([]);
|
||||
});
|
||||
|
||||
it("says what to do when the pinned GeoIP release is gone from GitHub", async () => {
|
||||
// Upstream keeps only its latest releases: the answer for a release it deleted is a 404.
|
||||
const gone = { ...pins.geoip, url: http.url(`/geoip/2026.01.01/${GEOIP_FILE}`) };
|
||||
const { runner, opts } = options({ console: OK_CONSOLE, exit: 0 }, { pins: { ...pins, geoip: gone } });
|
||||
|
||||
await expect(buildGoldenImage(opts)).rejects.toThrow(/HTTP 404.*pinned GeoIP release 2026.09.30 is gone.*guest\/image-builder\/pins\.json.*releases\/latest/s);
|
||||
|
||||
expect(runner.spawns).toHaveLength(0);
|
||||
});
|
||||
|
||||
/** A copy of the guest files the digest reads, for a test that changes one of them. */
|
||||
async function assetCopy(): Promise<string> {
|
||||
const assetRoot = join(home, "assets");
|
||||
|
||||
@@ -111,15 +111,31 @@ describe("the Dot's browser", () => {
|
||||
expect(build).toContain('as_dot "$env_dir/bin/invisible-playwright" fetch');
|
||||
});
|
||||
|
||||
it("has the GeoIP database fetched beside the engine, so a first launch downloads nothing, and records its release", () => {
|
||||
expect(build).toContain("from invisible_core.download import ensure_geoip_mmdb");
|
||||
// The fetch comes after the engine's, from the same environment, as dot (the cache is dot's).
|
||||
expect(build.indexOf("ensure_geoip_mmdb")).toBeGreaterThan(build.indexOf('"$env_dir/bin/invisible-playwright" fetch'));
|
||||
expect(build).toMatch(/as_dot "\$env_dir\/bin\/python" -c 'from invisible_core\.download import ensure_geoip_mmdb/);
|
||||
it("has the pinned GeoIP database unpacked beside the engine, so a first launch downloads nothing, and records its release", () => {
|
||||
// The script takes the archive, its tag and its hash from the seed, and checks the hash before it unpacks anything.
|
||||
expect(build).toContain('geoip_sha256="${5:?$usage}"');
|
||||
expect(build).toContain('echo "$geoip_sha256 $geoip_zip" | sha256sum --check --status -');
|
||||
expect(build.indexOf("sha256sum --check")).toBeLessThan(build.indexOf("uv venv"));
|
||||
// Nothing is resolved from the network by the script: the browser's own "latest release" fetch is not used.
|
||||
expect(build).not.toContain("ensure_geoip_mmdb");
|
||||
// The unpacking comes after the engine's fetch, from the same environment, as dot (the cache is dot's), and it
|
||||
// fails when the browser's own lookup would not find the file it wrote.
|
||||
expect(build.indexOf("bundle.extract")).toBeGreaterThan(build.indexOf('"$env_dir/bin/invisible-playwright" fetch'));
|
||||
expect(build).toMatch(/as_dot "\$env_dir\/bin\/python" - "\$dot_geoip" "\$geoip_tag" <<'PYTHON'/);
|
||||
expect(build).toContain("if geoip_mmdb_path() != target / GEOIP_MMDB_NAME:");
|
||||
// provision.sh hands the three values over, and refuses an image whose cache holds another release than the pin.
|
||||
expect(provision).toContain('"$payload/$GEOIP_ARCHIVE" "$GEOIP_TAG" "$GEOIP_SHA256"');
|
||||
expect(provision).toContain("from invisible_core.download import geoip_mmdb_path");
|
||||
expect(provision).toContain('[ "$geoip_database" = "$GEOIP_TAG" ]');
|
||||
expect(provision).toContain('component geoip-database "$geoip_database"');
|
||||
});
|
||||
|
||||
it("is unpacked the same way by the browser smoke, from the pin and with the same script", () => {
|
||||
const prepare = text("test/smoke/prepare-engine.sh");
|
||||
expect(prepare).toContain(".geoip | \"\\(.tag) \\(.url) \\(.sha256)\"");
|
||||
expect(prepare).toContain('"$geoip_dir/geoip-aio-all.mmdb.zip" "$GEOIP_TAG" "$GEOIP_SHA"');
|
||||
});
|
||||
|
||||
it("the lock in this checkout pins every package with hashes, both top-level packages included", () => {
|
||||
const lock = parsePythonLock(text(BUILDER_PYTHON_LOCK));
|
||||
expect(lock.mcpVersion).toMatch(/^\d+\.\d+\.\d+$/);
|
||||
|
||||
@@ -23,6 +23,14 @@ describe("the pins in this checkout", () => {
|
||||
expect(Object.keys(GUEST_PINS)).not.toContain("python_packages");
|
||||
});
|
||||
|
||||
it("pin the GeoIP database to one release of daijro/geoip-all-in-one by its exact URL and SHA-256", () => {
|
||||
const { geoip } = GUEST_PINS;
|
||||
expect(geoip.tag).toMatch(/^\d{4}\.\d{2}\.\d{2}$/);
|
||||
expect(geoip.url).toBe(`https://github.com/daijro/geoip-all-in-one/releases/download/${geoip.tag}/geoip-aio-all.mmdb.zip`);
|
||||
expect(geoip.sha256).toMatch(/^[0-9a-f]{64}$/);
|
||||
expect(downloadFileName(geoip)).toBe("geoip-aio-all.mmdb.zip");
|
||||
});
|
||||
|
||||
it("install the desktop, the browser libraries and the screenshot tool dot-agentd calls", () => {
|
||||
expect(GUEST_PINS.apt_packages).toEqual(
|
||||
expect.arrayContaining([
|
||||
@@ -50,6 +58,15 @@ describe("pin validation", () => {
|
||||
expect(() => parseGuestPins({ ...good, node: { ...good.node, sha256: "ABC" } })).toThrow(/node.sha256/);
|
||||
});
|
||||
|
||||
it("rejects a GeoIP pin that is not a hash, not a release of the project, or whose URL names another tag", () => {
|
||||
expect(() => parseGuestPins({ ...good, geoip: { ...good.geoip, sha256: "latest" } })).toThrow(/geoip.sha256/);
|
||||
expect(() => parseGuestPins({ ...good, geoip: { ...good.geoip, url: "https://example.org/geoip-aio-all.mmdb.zip" } })).toThrow(/geoip.url/);
|
||||
expect(() => parseGuestPins({ ...good, geoip: { ...good.geoip, tag: "2026.01.01" } })).toThrow(/geoip.url/);
|
||||
expect(() => parseGuestPins({ ...good, geoip: { ...good.geoip, url: good.geoip.url.replace("https:", "http:") } })).toThrow(/not an https URL/);
|
||||
const { geoip: _removed, ...without } = good;
|
||||
expect(() => parseGuestPins(without)).toThrow(/geoip must be an object/);
|
||||
});
|
||||
|
||||
it("rejects plain http", () => {
|
||||
expect(() => parseGuestPins({ ...good, uv: { ...good.uv, url: good.uv.url.replace("https:", "http:") } })).toThrow(/not an https URL/);
|
||||
});
|
||||
|
||||
@@ -27,6 +27,10 @@ describe("pins.env", () => {
|
||||
expect(text).toContain(`NODE_TARBALL='node-v${GUEST_PINS.node.version}-linux-x64.tar.xz'\n`);
|
||||
expect(text).toContain(`UV_VERSION='${GUEST_PINS.uv.version}'\n`);
|
||||
expect(text).toContain("UV_TARBALL='uv-x86_64-unknown-linux-gnu.tar.gz'\n");
|
||||
// The GeoIP release the image carries, and the hash the guest checks it against again before it unpacks it.
|
||||
expect(text).toContain(`GEOIP_TAG='${GUEST_PINS.geoip.tag}'\n`);
|
||||
expect(text).toContain("GEOIP_ARCHIVE='geoip-aio-all.mmdb.zip'\n");
|
||||
expect(text).toContain(`GEOIP_SHA256='${GUEST_PINS.geoip.sha256}'\n`);
|
||||
// The two Python versions come from the lock, the one place they are written.
|
||||
expect(text).toContain(`MCP_VERSION='${python.mcpVersion}'\n`);
|
||||
expect(text).toContain(`PLAYWRIGHT_VERSION='${python.playwrightVersion}'\n`);
|
||||
@@ -48,11 +52,13 @@ describe("pins.env", () => {
|
||||
});
|
||||
|
||||
describe("the builder seed", () => {
|
||||
it("holds the NoCloud files, the provisioner, its pins, the Python lock, both tarballs, the engine's lock with the script that builds its environment, and the script that builds the browser", async () => {
|
||||
it("holds the NoCloud files, the provisioner, its pins, the Python lock, both tarballs, the GeoIP archive, the engine's lock with the script that builds its environment, and the script that builds the browser", async () => {
|
||||
const nodeTarball = join(dir, "node.tar.xz");
|
||||
const uvTarball = join(dir, "uv.tar.gz");
|
||||
await writeFile(nodeTarball, "node bytes");
|
||||
await writeFile(uvTarball, "uv bytes");
|
||||
const geoipArchive = join(dir, "geoip.zip");
|
||||
await writeFile(geoipArchive, "geoip bytes");
|
||||
const engineLock = await readGuestAsset(defaultAssetRoot(), BUILDER_ENGINE_LOCK);
|
||||
const engineBuild = await readGuestAsset(defaultAssetRoot(), BUILDER_ENGINE_BUILD);
|
||||
const browserBuild = await readGuestAsset(defaultAssetRoot(), BUILDER_BROWSER_BUILD);
|
||||
@@ -67,6 +73,7 @@ describe("the builder seed", () => {
|
||||
python,
|
||||
nodeTarball,
|
||||
uvTarball,
|
||||
geoipArchive,
|
||||
engineLock,
|
||||
engineBuild,
|
||||
browserBuild,
|
||||
@@ -80,6 +87,7 @@ describe("the builder seed", () => {
|
||||
"mcp-requirements.lock",
|
||||
`node-v${GUEST_PINS.node.version}-linux-x64.tar.xz`,
|
||||
"uv-x86_64-unknown-linux-gnu.tar.gz",
|
||||
"geoip-aio-all.mmdb.zip",
|
||||
"engine-requirements.lock",
|
||||
"build-engine-env.sh",
|
||||
"build-browser-env.sh",
|
||||
@@ -87,13 +95,14 @@ describe("the builder seed", () => {
|
||||
expect(entries[0]).toEqual({ path: "user-data", data: userData });
|
||||
expect(entries[4]).toEqual({ path: "mcp-requirements.lock", data: pythonLock });
|
||||
expect(entries[5]).toEqual({ path: entries[5]!.path, file: nodeTarball });
|
||||
expect(entries[7]).toEqual({ path: "engine-requirements.lock", data: engineLock });
|
||||
expect(entries[8]).toEqual({ path: "build-engine-env.sh", data: engineBuild });
|
||||
expect(entries[9]).toEqual({ path: "build-browser-env.sh", data: browserBuild });
|
||||
expect(entries[7]).toEqual({ path: "geoip-aio-all.mmdb.zip", file: geoipArchive });
|
||||
expect(entries[8]).toEqual({ path: "engine-requirements.lock", data: engineLock });
|
||||
expect(entries[9]).toEqual({ path: "build-engine-env.sh", data: engineBuild });
|
||||
expect(entries[10]).toEqual({ path: "build-browser-env.sh", data: browserBuild });
|
||||
|
||||
// The real writer accepts it under the label NoCloud looks for.
|
||||
const summary = await writeIso(join(dir, "seed.iso"), entries, { volumeId: SEED_VOLUME_ID });
|
||||
expect(summary.files).toBe(10);
|
||||
expect(summary.files).toBe(11);
|
||||
const image = await readFile(join(dir, "seed.iso"));
|
||||
// Primary volume descriptor at sector 16: the label at offset 40.
|
||||
expect(image.toString("latin1", 16 * 2048 + 40, 16 * 2048 + 46)).toBe("cidata");
|
||||
|
||||
@@ -69,7 +69,7 @@ The real server and a real Firefox are not run by this smoke: they are the brows
|
||||
`browser/smoke.sh` runs the Dot's real browser. The golden image's apt packages (`pins.json`:
|
||||
the desktop, Firefox's libraries, ImageMagick) are installed, the browser is built by
|
||||
`builder/build-browser-env.sh` on the hashed `builder/mcp-requirements.lock` (the MCP server's
|
||||
environment, the browser engine, the GeoIP database: the script `provision.sh` runs), the
|
||||
environment, the browser engine, the GeoIP release `pins.json` pins and the script checks against its SHA-256: the script `provision.sh` runs), the
|
||||
runtime disk's `dot-desktop` script starts Xvfb and an XFCE session on `:0` as dot, and the Dot
|
||||
runs as in the engine smoke with no stand-in for the browser: the engine finds
|
||||
`invisible-playwright-mcp` on its PATH. Only the model is a stand-in, and the pages the browser
|
||||
|
||||
@@ -73,7 +73,12 @@ if [ "$suite" = browser ]; then
|
||||
# The packages of the golden image (pins.json), as provision.sh installs them.
|
||||
apt-get install -y -qq --no-install-recommends $(jq -r '.apt_packages[]' "$TREE/guest/image-builder/pins.json") >/dev/null
|
||||
# The Dot's browser, with the script provision.sh runs on the same lock. dot reaches its home, not the tree.
|
||||
bash "$builder/build-browser-env.sh" "$builder/mcp-requirements.lock" /home/dot/.local/share/invisible-dots/mcp
|
||||
# The GeoIP release pins.json names, fetched here (the image builder fetches it on the host) and checked
|
||||
# against its SHA-256 by the script itself.
|
||||
geoip_dir=$(mktemp -d)
|
||||
read -r GEOIP_TAG GEOIP_URL GEOIP_SHA < <(jq -r '.geoip | "\(.tag) \(.url) \(.sha256)"' "$TREE/guest/image-builder/pins.json")
|
||||
curl -fsSL "$GEOIP_URL" -o "$geoip_dir/geoip-aio-all.mmdb.zip"
|
||||
bash "$builder/build-browser-env.sh" "$builder/mcp-requirements.lock" /home/dot/.local/share/invisible-dots/mcp "$geoip_dir/geoip-aio-all.mmdb.zip" "$GEOIP_TAG" "$GEOIP_SHA"
|
||||
fi
|
||||
|
||||
exec bash "$checks"
|
||||
@@ -436,7 +436,7 @@ check "the frame was the server's browser_watch with the browser role main, and
|
||||
CLOSED_BEFORE=$(closed_total $ID5)
|
||||
check "POST /browser-identities/:id/close answers 204: the browser is closed, the server ends, closed is emitted once, the profile stays" "[ \"\$(status_of $A/browser-identities/$ID5/close /dev/null -X POST)\" = 204 ] && wait_fakes 0 && wait_closed_total $ID5 $((CLOSED_BEFORE + 1)) && [ -d $BROWSERS/$ID5/profile ] && [ \"\$(api $A/browser-identities/$ID5 | jq -r .status)\" = available ] && health_is 4 0"
|
||||
check "closing a closed identity answers 204 and emits nothing; an unknown one is 404" "[ \"\$(status_of $A/browser-identities/$ID5/close /dev/null -X POST)\" = 204 ] && [ \"\$(status_of $A/browser-identities/nobody-abc123/close /dev/null -X POST)\" = 404 ] && [ \"\$(closed_total $ID5)\" = $((CLOSED_BEFORE + 1)) ]"
|
||||
check "the model launches it again, so the delete below meets an open identity" "tool_turn 12 browser_identity_launch '{\"identity_id\":\"$ID5\"}' && wait_fakes 1 && health_is 4 1"
|
||||
check "the model launches it again, so the idle kill below meets an open identity" "tool_turn 12 browser_identity_launch '{\"identity_id\":\"$ID5\"}' && wait_fakes 1 && health_is 4 1"
|
||||
# A server that dies while nothing calls it (its relay ends with it): the engine hears of it when it happens.
|
||||
CLOSED_BEFORE=$(closed_total $ID5)
|
||||
check "a server killed while idle closes its identity at once, with no call: closed is emitted once and /health counts none open" "pkill -9 -u dot -f fake_mcp_server.py; wait_fakes 0 && wait_closed_total $ID5 $((CLOSED_BEFORE + 1)) && health_is 4 0 && [ \"\$(api $A/browser-identities/$ID5 | jq -r .status)\" = available ]"
|
||||
|
||||
@@ -32,12 +32,14 @@ from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import functools
|
||||
import posixpath
|
||||
import re
|
||||
import sqlite3
|
||||
from collections.abc import Callable, Coroutine, Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Literal
|
||||
from urllib.parse import quote, unquote, urlsplit
|
||||
|
||||
from loguru import logger
|
||||
|
||||
@@ -76,7 +78,12 @@ FRAME_WAIT_S = 5.0
|
||||
|
||||
SessionState = Literal["opening", "open", "closing"]
|
||||
|
||||
ErrorCode = Literal["not_found", "invalid", "limit", "not_open", "busy", "launch_failed", "crashed", "frame_failed"]
|
||||
# What an identity route (list, create, get, delete, frame, close) can fail with: IDENTITY_ERROR_STATUS of
|
||||
# protocol.py names the HTTP status of each.
|
||||
RouteErrorCode = Literal["not_found", "invalid", "limit", "not_open", "busy", "crashed", "frame_failed"]
|
||||
# `launch_failed` is the launch's alone: the model's browser_identity_launch and browser_open fail with it, and
|
||||
# no route launches (architecture section 5.3), so no HTTP status names it.
|
||||
ErrorCode = Literal[RouteErrorCode, "launch_failed"]
|
||||
|
||||
|
||||
class BrowserIdentityError(Exception):
|
||||
@@ -746,11 +753,43 @@ class BrowserManager:
|
||||
logger.info("browser identity {} closed", identity_id)
|
||||
|
||||
|
||||
@functools.lru_cache(maxsize=64)
|
||||
def _proxy_scrubber(proxy: str) -> tuple[re.Pattern[str], dict[str, str]]:
|
||||
"""What to find in a text of the server and what to put there, for one stored proxy URL.
|
||||
|
||||
The server splits the URL into a server, a user and a password (percent-decoded), so what it or Firefox
|
||||
repeats is not only the URL as stored: it is the password alone, as written, decoded or encoded again,
|
||||
the user and the password together, or the Basic credentials of a `Proxy-Authorization` header. All of
|
||||
them are found in one pass, longest first, so that the text put in place of one is never searched again.
|
||||
A password too short to be told from the text around it is hidden too: the page text it garbles is the
|
||||
price of a password that leaves nowhere.
|
||||
"""
|
||||
replacements = {proxy: redact_proxy(proxy)}
|
||||
try:
|
||||
parts = urlsplit(proxy.strip())
|
||||
written_user, written_password = parts.username or "", parts.password or ""
|
||||
except ValueError:
|
||||
written_user = written_password = "" # the server's own parse fails too: only the URL can be named
|
||||
if written_password:
|
||||
user, password = unquote(written_user), unquote(written_password)
|
||||
users = {written_user, user, quote(user, safe="")}
|
||||
passwords = {form for form in (written_password, password, quote(password, safe="")) if form}
|
||||
for form in users:
|
||||
for secret in passwords:
|
||||
replacements[f"{form}:{secret}"] = f"{form}:***"
|
||||
replacements[base64.b64encode(f"{user}:{password}".encode()).decode()] = "***"
|
||||
for secret in passwords:
|
||||
replacements.setdefault(secret, "***")
|
||||
pattern = re.compile("|".join(re.escape(found) for found in sorted(replacements, key=len, reverse=True)))
|
||||
return pattern, replacements
|
||||
|
||||
|
||||
def _scrub(text: str, proxy: str | None) -> str:
|
||||
"""A text of the server with the proxy it may echo in its redacted form."""
|
||||
if proxy and proxy in text:
|
||||
return text.replace(proxy, redact_proxy(proxy))
|
||||
return text
|
||||
"""A text of the server with the proxy it may echo, and its password in any form, redacted."""
|
||||
if not proxy:
|
||||
return text
|
||||
pattern, replacements = _proxy_scrubber(proxy)
|
||||
return pattern.sub(lambda found: replacements[found.group(0)], text)
|
||||
|
||||
|
||||
def _scrub_result(result: Any, proxy: str | None) -> Any:
|
||||
|
||||
@@ -33,16 +33,20 @@ AGENT_ROUTES = {
|
||||
# `browserIdentityFrame(id)` and `browserIdentityClose(id)`.
|
||||
BROWSER_IDENTITY_ACTIONS = ("frame", "close")
|
||||
|
||||
# How long the host waits for `POST /prepare-sleep` before it stops the guest anyway, in seconds
|
||||
# (PREPARE_SLEEP_TIMEOUT_MS in packages/shared protocol.ts; tests/repo/vendored-nanobot.test.ts keeps the two
|
||||
# equal, and tests/dots/test_engine.py checks that the engine's steps of a prepare-sleep fit inside it).
|
||||
PREPARE_SLEEP_TIMEOUT_S = 60
|
||||
|
||||
# The HTTP status of each error code the identity routes answer with. IDENTITY_ERROR_STATUS of
|
||||
# packages/shared has the same table; tests/repo/vendored-nanobot.test.ts reads it with a regex, so it is one
|
||||
# `"code": status` line each, and test_protocol.py checks that its keys are the codes of `browser.ErrorCode`.
|
||||
# `"code": status` line each, and test_protocol.py checks that its keys are the codes of `browser.RouteErrorCode`.
|
||||
IDENTITY_ERROR_STATUS = {
|
||||
"invalid": 400,
|
||||
"not_found": 404,
|
||||
"limit": 409,
|
||||
"not_open": 409,
|
||||
"busy": 503,
|
||||
"launch_failed": 502,
|
||||
"crashed": 502,
|
||||
"frame_failed": 502,
|
||||
}
|
||||
|
||||
@@ -1688,11 +1688,27 @@ async def test_a_provider_with_on_terminated_hears_an_idle_servers_end_only_whil
|
||||
assert ended == ["srv"]
|
||||
|
||||
|
||||
async def test_a_provider_without_on_terminated_ignores_the_end_of_the_transport() -> None:
|
||||
async def test_a_provider_without_on_terminated_does_nothing_about_the_end_of_the_transport(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
provider = MCPProvider({"srv": MCPServerConfig(command="fake")}, ToolRegistry())
|
||||
provider._connections["srv"] = SimpleNamespace() # type: ignore[assignment]
|
||||
connection = SimpleNamespace(aclose=AsyncMock())
|
||||
provider._connections["srv"] = connection # type: ignore[assignment]
|
||||
reconnect = AsyncMock()
|
||||
provider._refresh_terminated_server = reconnect # type: ignore[method-assign]
|
||||
connect = AsyncMock()
|
||||
monkeypatch.setattr(mcp_mod, "connect_mcp_servers", connect)
|
||||
tasks_before = asyncio.all_tasks()
|
||||
|
||||
provider._transport_ended("srv")
|
||||
await asyncio.sleep(0)
|
||||
|
||||
# No reconnect, no new connection, nothing closed, nothing left running in the background.
|
||||
reconnect.assert_not_awaited()
|
||||
connect.assert_not_awaited()
|
||||
connection.aclose.assert_not_awaited()
|
||||
assert provider._connections == {"srv": connection}
|
||||
assert asyncio.all_tasks() == tasks_before
|
||||
|
||||
|
||||
@pytest.mark.parametrize("params, error", [
|
||||
|
||||
@@ -18,7 +18,7 @@ from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from fakes.fake_mcp_server import install_fake_mcp, read_record, write_control
|
||||
from fakes.fake_mcp_server import install_fake_mcp, proxy_forms, read_record, write_control
|
||||
from fakes.local_computer import LocalComputer
|
||||
|
||||
from nanobot.agent.tools.base import ToolResult
|
||||
@@ -354,6 +354,46 @@ async def test_what_a_page_tool_returns_of_the_proxy_is_redacted_in_results_and_
|
||||
assert "s3cret" not in result_text(result)
|
||||
|
||||
|
||||
# The credentials of this proxy are written percent-encoded, with a lowercase escape, so the password as
|
||||
# written, decoded and encoded again are three different strings.
|
||||
ENCODED_PROXY = "http://us%40er:p%40ss%3a[email protected]:8080"
|
||||
PASSWORD_FORMS = ("password_as_written", "password_decoded", "password_encoded_again", "basic_credentials")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("form", sorted(proxy_forms(ENCODED_PROXY)))
|
||||
async def test_the_password_leaves_in_no_form_a_server_can_repeat_it(env: Env, form: str) -> None:
|
||||
manager = env.manager()
|
||||
identity = await manager.create("forms", ENCODED_PROXY)
|
||||
mcp_home = env.mcp_home(identity.id)
|
||||
secrets = [proxy_forms(ENCODED_PROXY)[name] for name in PASSWORD_FORMS]
|
||||
assert len(set(secrets)) == len(secrets)
|
||||
|
||||
write_control(mcp_home, fail_open=True, echo_proxy_as=form)
|
||||
with pytest.raises(BrowserIdentityError) as failed_launch:
|
||||
await manager.launch(identity.id)
|
||||
write_control(mcp_home, echo_proxy_as=form)
|
||||
await manager.launch(identity.id)
|
||||
failed = await manager.call_tool(identity.id, "browser_navigate", {"url": "https://example.org/"})
|
||||
read = await manager.call_tool(identity.id, "browser_read_text", {})
|
||||
|
||||
assert result_is_error(failed) and not result_is_error(read)
|
||||
for said in (failed_launch.value.message, result_text(failed), result_text(read)):
|
||||
assert "***" in said
|
||||
for secret in secrets:
|
||||
assert secret not in said
|
||||
|
||||
|
||||
async def test_a_proxy_without_a_password_leaves_what_the_server_says_as_it_is(env: Env) -> None:
|
||||
manager = env.manager()
|
||||
identity = await manager.create("open proxy", "http://proxy.test:8080")
|
||||
write_control(env.mcp_home(identity.id), echo_proxy_on_pages=True)
|
||||
await manager.launch(identity.id)
|
||||
|
||||
read = await manager.call_tool(identity.id, "browser_read_text", {})
|
||||
|
||||
assert result_text(read) == "page behind http://proxy.test:8080"
|
||||
|
||||
|
||||
async def test_reports_a_command_that_cannot_be_started_without_its_arguments(env: Env) -> None:
|
||||
manager = env.manager(mcp_command=str(env.tmp_path / "no-such-mcp-binary"))
|
||||
identity = await manager.create("missing", "http://user:[email protected]:8080")
|
||||
|
||||
@@ -37,7 +37,7 @@ from nanobot.dots.engine import (
|
||||
approval_granted_continuation,
|
||||
approval_rejected_continuation,
|
||||
)
|
||||
from nanobot.dots.protocol import DotsConfigError
|
||||
from nanobot.dots.protocol import PREPARE_SLEEP_TIMEOUT_S, DotsConfigError
|
||||
from nanobot.dots.transcript_outbox import CLOSED, CLOSED_INTERRUPTED, INBOUND_ID
|
||||
from nanobot.providers.base import LLMResponse
|
||||
|
||||
@@ -1442,6 +1442,16 @@ class TestStoppingWithBrowsersOpen:
|
||||
await h.engine.stop()
|
||||
assert closed == []
|
||||
|
||||
def test_the_steps_of_a_prepare_sleep_fit_inside_the_hosts_wait_with_a_margin(self) -> None:
|
||||
# The host gives up on a prepare-sleep after PREPARE_SLEEP_TIMEOUT_S and stops the guest, which would cut
|
||||
# the browsers mid-close: the grace, the wait for the cancelled turns and the closes have to end before
|
||||
# that, with room for the flush of the state and the answer.
|
||||
steps = (
|
||||
engine_module.STOP_GRACE_S + engine_module._CANCEL_WAIT_S + engine_module._CLOSE_BROWSERS_ON_SLEEP_S
|
||||
)
|
||||
margin_s = 5
|
||||
assert steps + margin_s <= PREPARE_SLEEP_TIMEOUT_S
|
||||
|
||||
async def test_a_config_applies_the_browser_limits_to_the_manager(self, make_engine: MakeEngine) -> None:
|
||||
h = started(make_engine([]))
|
||||
assert h.browser.limits == (3, 20)
|
||||
|
||||
@@ -8,7 +8,7 @@ from typing import Any, get_args
|
||||
|
||||
import pytest
|
||||
|
||||
from nanobot.dots.browser import ErrorCode
|
||||
from nanobot.dots.browser import ErrorCode, RouteErrorCode
|
||||
from nanobot.dots.protocol import (
|
||||
AGENT_ROUTES,
|
||||
AGENT_STATES,
|
||||
@@ -55,10 +55,12 @@ class TestSharedNames:
|
||||
assert len(set(AGENT_ROUTES.values())) == len(AGENT_ROUTES)
|
||||
assert AGENT_ROUTES["events_stream"] == "/events/stream"
|
||||
|
||||
def test_every_error_code_of_a_browser_identity_has_one_status(self) -> None:
|
||||
assert set(IDENTITY_ERROR_STATUS) == set(get_args(ErrorCode))
|
||||
def test_every_error_code_a_route_answers_has_one_status_and_the_launch_has_none(self) -> None:
|
||||
assert set(IDENTITY_ERROR_STATUS) == set(get_args(RouteErrorCode))
|
||||
# No route launches (only the model's tools do), so `launch_failed` is the one code with no status.
|
||||
assert set(get_args(ErrorCode)) - set(IDENTITY_ERROR_STATUS) == {"launch_failed"}
|
||||
assert IDENTITY_ERROR_STATUS["busy"] == 503
|
||||
assert {IDENTITY_ERROR_STATUS[code] for code in ("launch_failed", "crashed", "frame_failed")} == {502}
|
||||
assert {IDENTITY_ERROR_STATUS[code] for code in ("crashed", "frame_failed")} == {502}
|
||||
|
||||
def test_the_cancel_event_name(self) -> None:
|
||||
assert TASK_CANCELLED_EVENT == "task.cancelled"
|
||||
|
||||
@@ -17,12 +17,14 @@ so a parameter renamed in a call fails here and not only inside a Dot.
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import quote, unquote, urlparse
|
||||
|
||||
FIXTURE = Path(__file__).resolve().parents[1] / "fixtures" / "mcp-tools-0.70.2.json"
|
||||
SCRIPT = Path(__file__).resolve()
|
||||
@@ -32,6 +34,27 @@ PNG = "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA6
|
||||
JPEG = "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgICAgMCAgIDAwMDBAYEBAQEBAgGBgUGCQgKCgkICQkKDA8MCgsOCwkJDRENDg8QEBEQCgwSExIQEw8QEBD/yQALCAABAAEBAREA/8wABgAQEAX/2gAIAQEAAD8A0s8g/9k="
|
||||
|
||||
|
||||
def proxy_forms(proxy: str) -> dict[str, str]:
|
||||
"""The forms in which the proxy URL's credentials can show up in what a server or Firefox says.
|
||||
|
||||
The real wrapper splits the URL with `urlparse` and unquotes the user and the password
|
||||
(`invisible_playwright_mcp/mcp/proxy.py`), so what it hands on and what a proxy's answer or a log line
|
||||
repeats is the password as written, decoded or encoded again, with the user in front of it, or the
|
||||
Basic credentials of a `Proxy-Authorization` header.
|
||||
"""
|
||||
parsed = urlparse(proxy)
|
||||
written_user, written_password = parsed.username or "", parsed.password or ""
|
||||
user, password = unquote(written_user), unquote(written_password)
|
||||
return {
|
||||
"password_as_written": written_password,
|
||||
"password_decoded": password,
|
||||
"password_encoded_again": quote(password, safe=""),
|
||||
"userinfo_as_written": f"{written_user}:{written_password}",
|
||||
"userinfo_decoded": f"{user}:{password}",
|
||||
"basic_credentials": base64.b64encode(f"{user}:{password}".encode()).decode(),
|
||||
}
|
||||
|
||||
|
||||
def install_fake_mcp(directory: Path) -> Path:
|
||||
"""Write an executable that runs this file with the interpreter of the tests; returns its path."""
|
||||
script = directory / f"fake-mcp-{uuid.uuid4().hex}"
|
||||
@@ -55,6 +78,8 @@ def write_control(mcp_home: Path, **control: Any) -> None:
|
||||
fail_open: `browser_open` fails the way the real server does when Firefox does not start.
|
||||
echo_proxy: the failure of `browser_open` names the proxy it was started with, as a server may.
|
||||
echo_proxy_on_pages: `browser_navigate` fails and `browser_read_text` answers naming that proxy too.
|
||||
echo_proxy_as: the name of one of `proxy_forms`: `browser_open` (with `fail_open`), `browser_navigate` and
|
||||
`browser_read_text` say that form of the proxy's credentials instead of the whole URL.
|
||||
lose_browser_once: the first page action after opening reports the browser gone, as after a Firefox crash.
|
||||
lose_browser_always: every page action does.
|
||||
refuse_close: `browser_close` fails.
|
||||
@@ -96,6 +121,8 @@ async def _serve() -> None:
|
||||
"url": "about:blank",
|
||||
}
|
||||
|
||||
echoed_form = proxy_forms(os.environ.get("STEALTHFOX_PROXY", ""))[control["echo_proxy_as"]] if control.get("echo_proxy_as") else None
|
||||
|
||||
def text(value: str, *, error: bool = False) -> types.CallToolResult:
|
||||
return types.CallToolResult(content=[types.TextContent(type="text", text=value)], isError=error)
|
||||
|
||||
@@ -111,6 +138,8 @@ async def _serve() -> None:
|
||||
if name == "browser_open":
|
||||
if control.get("fail_open"):
|
||||
proxy = f" ({os.environ.get('STEALTHFOX_PROXY')})" if control.get("echo_proxy") else ""
|
||||
if echoed_form is not None:
|
||||
proxy = f" (credentials {echoed_form})"
|
||||
return text(f"the {role} browser did NOT start: proxy refused the connection{proxy}", error=True)
|
||||
if state["download_left"] > 0:
|
||||
state["download_left"] -= 1
|
||||
@@ -143,6 +172,8 @@ async def _serve() -> None:
|
||||
os._exit(3)
|
||||
if control.get("echo_proxy_on_pages"):
|
||||
return text(f"navigation failed through {os.environ.get('STEALTHFOX_PROXY')}", error=True)
|
||||
if echoed_form is not None:
|
||||
return text(f"navigation failed: the proxy refused the credentials {echoed_form}", error=True)
|
||||
state["url"] = str(args["url"])
|
||||
return text(f"200 {state['url']}")
|
||||
if name == "browser_snapshot":
|
||||
@@ -150,6 +181,8 @@ async def _serve() -> None:
|
||||
if name == "browser_read_text":
|
||||
if control.get("echo_proxy_on_pages"):
|
||||
return text(f"page behind {os.environ.get('STEALTHFOX_PROXY')}")
|
||||
if echoed_form is not None:
|
||||
return text(f"page behind the credentials {echoed_form}")
|
||||
return text(f"text of {args.get('selector', 'body')}")
|
||||
if name == "browser_take_screenshot" or name == "browser_click_at":
|
||||
return image(PNG, "image/png")
|
||||
|
||||
@@ -165,10 +165,19 @@ export const AGENT_ROUTES = {
|
||||
prepareSleep: "/prepare-sleep",
|
||||
} as const;
|
||||
|
||||
/**
|
||||
* How long the host waits for `POST /prepare-sleep` before it gives up and stops the guest anyway (architecture
|
||||
* section 9.5). The engine's work inside it (the grace of a tool in flight, the wait for the cancelled turns,
|
||||
* the close of every open browser) has to fit; nanobot/dots/protocol.py holds the same number in seconds
|
||||
* and tests/dots/test_engine.py checks the sum of those steps against it.
|
||||
*/
|
||||
export const PREPARE_SLEEP_TIMEOUT_MS = 60_000;
|
||||
|
||||
/**
|
||||
* The HTTP status of each error code the engine answers on the identity routes (architecture section 5.3):
|
||||
* `invalid`, `not_found`, `limit` and `not_open` are the caller's, `busy` is a call holding the browser, and
|
||||
* `launch_failed`, `crashed` and `frame_failed` are the browser's. nanobot/dots/protocol.py holds the same
|
||||
* `crashed` and `frame_failed` are the browser's. There is no `launch_failed` here: no route launches (the model's
|
||||
* tools do), so no answer of a route carries it. nanobot/dots/protocol.py holds the same
|
||||
* table and serves its answers from it; the control plane passes an answer with one of these pairs through
|
||||
* as it is, so the UI can tell a busy or crashed browser from an unreachable computer.
|
||||
*/
|
||||
@@ -178,7 +187,6 @@ export const IDENTITY_ERROR_STATUS = {
|
||||
limit: 409,
|
||||
not_open: 409,
|
||||
busy: 503,
|
||||
launch_failed: 502,
|
||||
crashed: 502,
|
||||
frame_failed: 502,
|
||||
} as const;
|
||||
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
identityPaths,
|
||||
isIdentityAnswer,
|
||||
OPENROUTER_KEY_RULE,
|
||||
PREPARE_SLEEP_TIMEOUT_MS,
|
||||
truncateText,
|
||||
vmName,
|
||||
} from "../src/browser.js";
|
||||
@@ -20,6 +21,8 @@ describe("protocol constants", () => {
|
||||
expect(AGENT_ROUTES.browserIdentity("a b")).toBe("/browser-identities/a%20b");
|
||||
expect(AGENT_ROUTES.browserIdentityFrame("a b")).toBe("/browser-identities/a%20b/frame");
|
||||
expect(AGENT_ROUTES.browserIdentityClose("a b")).toBe("/browser-identities/a%20b/close");
|
||||
// Section 9.5: the engine's steps of a prepare-sleep (20 s of grace, 5 s of cancel wait, a 30 s browser close) fit inside.
|
||||
expect(PREPARE_SLEEP_TIMEOUT_MS).toBe(60_000);
|
||||
});
|
||||
|
||||
it("lays out an identity directory", () => {
|
||||
@@ -40,7 +43,6 @@ describe("identity error answers", () => {
|
||||
limit: 409,
|
||||
not_open: 409,
|
||||
busy: 503,
|
||||
launch_failed: 502,
|
||||
crashed: 502,
|
||||
frame_failed: 502,
|
||||
});
|
||||
|
||||
@@ -26,6 +26,7 @@ import {
|
||||
OPENROUTER_KEY_PATTERN,
|
||||
OPENROUTER_KEY_RULE,
|
||||
OUTBOUND_EVENT_TYPES,
|
||||
PREPARE_SLEEP_TIMEOUT_MS,
|
||||
TASK_CANCELLED_SYSTEM_EVENT,
|
||||
TOOL_TARGET_MAX,
|
||||
} from "@invisible-dots/shared";
|
||||
@@ -151,6 +152,9 @@ describe("the vendored nanobot fork", () => {
|
||||
const engineStatus = Object.fromEntries([...statusMatch![1]!.matchAll(/"(\w+)":\s*(\d+)/g)].map((m) => [m[1]!, Number(m[2])]));
|
||||
expect(engineStatus).toEqual({ ...IDENTITY_ERROR_STATUS });
|
||||
|
||||
// How long the host waits for a prepare-sleep, which the engine's steps of one have to fit inside.
|
||||
expect(Number(/^PREPARE_SLEEP_TIMEOUT_S = (\d+)$/m.exec(protocol)?.[1]) * 1000).toBe(PREPARE_SLEEP_TIMEOUT_MS);
|
||||
|
||||
expect(/^TASK_CANCELLED_EVENT = "([^"]+)"/m.exec(protocol)?.[1]).toBe(TASK_CANCELLED_SYSTEM_EVENT);
|
||||
// The longest `target` of a `tool.called`: the engine cuts to it, the host's schema refuses more.
|
||||
expect(Number(/^TOOL_TARGET_MAX = (\d+)$/m.exec(protocol)?.[1])).toBe(TOOL_TARGET_MAX);
|
||||
|
||||
Reference in new issue
Block a user