diff --git a/.agents/skills/rea-tool-design/SKILL.md b/.agents/skills/rea-tool-design/SKILL.md index 3dd31ffdb1..6d36fa7aac 100644 --- a/.agents/skills/rea-tool-design/SKILL.md +++ b/.agents/skills/rea-tool-design/SKILL.md @@ -22,7 +22,9 @@ parity. For an audit, follow [audit existing tools](../../../docs/tool-design.md#audit-existing-tools): scope representative tasks, exercise the live workflow, and distinguish reproduced bugs, supported limitations, design opportunities and unverified -claims. Report working alternatives and evidence limits as well as findings. +claims. Check relevant rendered MCP prompts, guided workflows and packaged skill +advice against the live contracts and exercise their suggested handoffs. Report +working alternatives and evidence limits as well as findings. For a design request, return the proposed boundary, input/result shape, Evidence semantics, nearest alternative, and verification approach. Implement when diff --git a/.agents/skills/reverse-engineer-anything/SKILL.md b/.agents/skills/reverse-engineer-anything/SKILL.md index a974b8c698..da90c287a6 100644 --- a/.agents/skills/reverse-engineer-anything/SKILL.md +++ b/.agents/skills/reverse-engineer-anything/SKILL.md @@ -137,6 +137,9 @@ explicit path or endpoint and do not need it. - Agent environment (AgentEnv bundle) of a web application: read [environments](references/environments.md) before recording; it records with `capture_browser_scenario` and writes the bundle with `export_environment_bundle`. +- Task for a desktop application's exported environment, from a + `capture-desktop-application` folder: read + [desktop application tasks](references/environments.md#tasks-for-a-desktop-application-environment). - Fiddler Everywhere traffic or a native `.saz` capture: read the [Fiddler workflow](references/runtime-observation.md#fiddler-everywhere-traffic-debugging). Inspect saved SAZ with `inspect_web_network_capture` and `format: "saz"`; @@ -273,6 +276,15 @@ explicit path or endpoint and do not need it. linker declarations, not proof of enforcement. A missing directory is observed absence of that declaration. See the [PE load configuration guide](https://github.com/morluto/rea/blob/main/docs/pe-load-config.md). +- ELF, PE, and Mach-O hardening: `inspect_binary_hardening` with the explicit + local file. This built-in target-free inspection requires no pwntools or other + host tool. One image per Mach-O slice. Each property is `enabled`, `disabled`, + `partial`, `unknown`, or `not_applicable`, with the header field or import + behind it. A missing stack-protector or fortify import is `unknown`, not + proof the protection was omitted. Authenticode and Mach-O code signatures + report presence, not validity. Hardened runtime stays unknown and names + `inspect_signature`. The image is not executed. See the + [binary hardening guide](https://github.com/morluto/rea/blob/main/docs/binary-hardening.md). - ELF note tables: `inspect_elf_notes` with the explicit local ELF32/ELF64 file. This built-in target-free inspection requires no disassembler. Preserve PT_NOTE and SHT_NOTE ranges, owners, types, descriptor SHA-256, and GNU @@ -394,6 +406,10 @@ describe possible providers, while top-level session capabilities describe the current route and `analysis_provider_binding` names the selected provider. Do not treat those scopes as interchangeable. +For editable Ghidra analysis that must survive a later agent process, select +the [owned Program checkpoint workflow](references/native-and-artifacts.md#continue-editable-ghidra-analysis-across-processes) +before making metadata edits. It is separate from Evidence/Unknown persistence. + For a Ghidra startup timeout on a large native target, follow [deadline recovery](references/native-and-artifacts.md#ghidra-startup-deadline-recovery) before abandoning deep analysis or repeating the same cold import. The default @@ -521,6 +537,8 @@ as complete while required questions remain open. - Agent environments (AgentEnv bundles) of a web application, recorded from a live instance or a HAR: [references/environments.md](references/environments.md) +- Tasks for a desktop application's exported environment: + [references/environments.md](references/environments.md#tasks-for-a-desktop-application-environment) ## Finish the task diff --git a/.agents/skills/reverse-engineer-anything/references/environments.md b/.agents/skills/reverse-engineer-anything/references/environments.md index 0be64fbab2..327ceab171 100644 --- a/.agents/skills/reverse-engineer-anything/references/environments.md +++ b/.agents/skills/reverse-engineer-anything/references/environments.md @@ -8,7 +8,9 @@ confirmed. The workflow records the application once with again if a hint names a gap, then writes the bundle with `export_environment_bundle`. For a HAR the user already has, skip to [export](#export-and-hand-off) through `inspect_web_network_capture`, or use -`rea environment-build `. +`rea environment-build `. For a task in a desktop +application's exported environment, read +[its section](#tasks-for-a-desktop-application-environment). ## Before recording @@ -285,3 +287,90 @@ See [agent environments](https://github.com/morluto/rea/blob/main/docs/agent-environments.md) for what the state, replay and the bundle establish, and [evidence workflows](evidence-workflows.md) for the manifest and state views. + +## Tasks for a desktop application environment + +Use this when the user has a folder from `rea capture-desktop-application` and +wants a task for the environment `rea export-desktop-application` makes from +it: a prompt for a computer-use agent and checks that grade what the agent +left in the app's files. You write the prompt and the checks; REA validates +the task against the capture and exports it. These commands are CLI only. + +### Describe the change + +Run `rea describe-desktop-application-task CAPTURE --format json`. Read: + +- `owned_paths`: the only paths whose files the app serves, so the only files + a check can read. +- `files`: each file there whose state differs between the capture's first + and last version, with its `role`, the `baseline` and `final` versions and + the net `change`. Text changes list each added and removed line once, and + `line_counts` gives lines found at both ends whose count changed. +- `task`: a draft whose `checks` are suggestions from the net change of the + documents the user worked on, or of the app's own state when no document + changed. + +### Write the prompt + +Say what the user wants done, in their words and the app's terms: "Add buy +milk to my to-do list and save it". Never say how the file stores it: no +paths the user would not name, line formats, XML, keys or SQL, and never the +checks. Replace the draft's placeholder; the export refuses it. + +### Pick the checks + +Checks must express only the observed change: + +- Keep a suggestion only if it says what the user did. Drop those that follow + from how the app saves: a format rewrite, renumbered lines, counters. A + LibreCAD drawing gaining four holes was rewritten from DXF R12 to 2007, and + the draft suggests counts of `SECTION`, `ENDSEC` and `CIRCLE`; only + `CIRCLE` equals 9 is the user's change. +- Never check unchanged content, whole files, or values the app stamps: + times, ids it assigns, window sizes. REA refuses ISO 8601 and `asctime` + date-times (`Thu Nov 5 09:20:31 2026`); the + rest is your judgment. +- Prefer a few robust checks: whole lines over positions, rows by meaningful + columns over generated ids, `at_least` when the user may add more. + +Kinds: `file_present` and `file_absent`; `lines_present` and `lines_absent` +(`lines`); `line_count` (`exact` or `contains`, and `equals`, `at_least` or +`at_most`); `value_equals` and `value_absent` (a JSON `pointer` into the view, +such as `/groups/GROUP/KEY` for a key file); `text_equals` and +`text_contains` (an XML element's whole text at a `pointer`, optional +`normalize_whitespace`); `row_present` and `row_absent` (`table`, `where` by +column) and `row_count`. Each names its `file` as a path under the home. A +document saved as a zip archive, such as FreeCAD's `.FCStd`, reads as its +entries: point into an entry's view, and name array members by selector +rather than position, as +`/entries/[name=Document.xml]/view/root/children/[name=Object][@name=Hole]`, +so the check survives an app version that writes them in another order. Give +`value_equals` a `tolerance` to compare a number the app writes as text, such +as `"5.0000000000000000"`, with a number. Check what the user wrote in a +document with `text_equals` on its paragraph element, not `value_equals` on a +text node: apps such as AbiWord and Writer split a paragraph into runs as they +save, so its first text node holds only part of it; use `text_contains` for a +value within a sentence, such as a time. The +[desktop application tasks](https://github.com/morluto/rea/blob/main/docs/desktop-application-tasks.md) +guide has the exact rules. + +### Export the task + +Write the task as JSON with a `name`, the `prompt` and the `checks`, then run +`rea export-desktop-application CAPTURE DIRECTORY --task TASK.json`. REA keeps +the task only when its checks fail on the seed and all pass on the captured +end state. A refusal names the check: one that fails on the end state does not +match what the user left; "every check already passes on the seed" means no +check captures the change. + +When the export refuses an application that no apt repository installs, ask +the user for the `.deb` file they installed it from and capture with +`--package-file`, then ask where the computer may download that file and pass +it as `--package-url`. Never fetch a vendor's download page yourself: such +sites often refuse automated downloads, and the copy and its licence are the +user's to choose. + +Tell the user the task's name, prompt and checks, and how to measure a solve +rate: run `agent-env run DIRECTORY --task NAME --agent claude-code` with +`--model MODEL` three times and count the runs reported `passed`. Do not run +agents unless asked. The bundle's seed is the user's real data: keep it local. diff --git a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md index 5e854285d2..6371fdab07 100644 --- a/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md +++ b/.agents/skills/reverse-engineer-anything/references/native-and-artifacts.md @@ -155,6 +155,18 @@ mitigations carry `evidence_kind: "inferred"`; enforcement remains unobserved. See [PE TLS callbacks and load configuration](https://github.com/morluto/rea/blob/main/docs/pe-load-config.md). +## Binary hardening + +Use `inspect_binary_hardening` for an explicit local ELF, PE, or Mach-O file +when the question concerns PIE or ASLR, NX, RELRO, stack-protector or FORTIFY +signs, control-flow flags, or stripped state. No `open_binary`, pwntools, or +target execution is required. The CLI equivalent is +`inspect-binary-hardening PATH`. Universal Mach-O files return one image per +slice. A missing import is `unknown`, not proof the protection was omitted. +Authenticode and Mach-O code-signature rows report presence, not validity. +`hardened_runtime` stays unknown and names `inspect_signature`. See +[binary hardening](https://github.com/morluto/rea/blob/main/docs/binary-hardening.md). + ## ELF note tables Use `inspect_elf_notes` for an explicit local ELF32/ELF64 image when the @@ -308,6 +320,17 @@ cross-references. Addresses and recovered pseudocode are analysis observations, not original source. Provider unavailability and unsupported metadata remain unknown rather than false. +For large pseudocode, request `procedure_pseudo_code` with `detail: "summary"` +on the first MCP call. It retains the complete observation and returns the +original selection, text availability and length without the body. Use +`normalized_result.parent_evidence_id` with `inspect_analysis_view` and +`view: {kind: "native", facet: "pseudocode", offset: 0, limit: 20000}`; +follow `coverage.next_offset` for later pages. Offsets and limits count UTF-16 +code units, not lines or tokens. Pages reuse the same observation without +repeating decompilation. Null and empty text remain distinct. Omitted or +`complete` detail preserves full inline delivery. Summary delivery requires a +retaining session and does not reduce the work or memory of decompilation. + For a completed `analyze_function` dossier, use `inspect_analysis_view` with `view: {kind: "native", facet: "value_flow_operations", offset: 0, limit: 32}` and the exact retained Evidence ID to read selected facts without reanalysis. @@ -358,6 +381,28 @@ operation's full ordered list. Candidate capabilities, returned with complete detail, describe possible providers. Top-level session capabilities describe the current route, and `analysis_provider_binding` identifies the selected provider. +## Continue editable Ghidra analysis across processes + +For a task spanning agent sessions, check `binary_session` after selecting +Ghidra: the current provider's `capabilities[].shared_limitations` states whether +Program checkpoints are enabled or the session is ephemeral. Successful native +queries distinguish initial analysis from an actually restored packed Program; +do not infer that restoration happened merely because checkpoint mode is enabled. + +In ephemeral mode, choose a private absolute directory and set +`REA_GHIDRA_CHECKPOINT_DIR` in the environment starting REA. Restart the MCP server +after changing that launch configuration; no current tool call or runtime +analysis-setting update enables it. Stock ELF/PE/Mach-O imports on Linux/macOS +are supported; see the [checkpoint setup and limitations](https://github.com/morluto/rea/blob/main/docs/installation.md#automatically-checkpoint-an-owned-ghidra-program). + +With the setting enabled, normal opens restore compatible saved Programs +automatically. Initial completed analysis is saved, and `close_binary` saves +later metadata edits; no separate save/restore tool is needed. Repair a reported +save failure and retry close. Cancellation or process failure can lose edits +after the last successful save; unfinished initial analysis cannot resume. +Keep the same checkpoint configuration for later processes. Evidence/Unknown +history and immutable REA query snapshots have separate persistence workflows. + ## Reuse caller-selected Ghidra analysis When the caller supplies an expanded project, configure `REA_GHIDRA_PROJECT_PATH` @@ -381,7 +426,7 @@ When the error names `ghidra.startup_timeout_ms`, call `update_analysis_settings` with `action: "set"` and a larger value within the reported bounds. Resolve retained cleanup before the next startup attempt. No REA restart is needed; an already accepted startup keeps its original -budget. A new attempt repeats import. Preserve useful Evidence before closing +budget. Follow the error's session-specific restart guidance, and preserve useful Evidence before closing and reopening the target when remediation requires that lifecycle step. For a per-function timeout, update `ghidra.decompile_timeout_seconds` and retry @@ -637,7 +682,7 @@ numbering gaps leave coverage unknown; a DfuSe suffix CRC mismatch follows ### Function annotations in Ghidra Fresh-import sessions admit annotations; existing-project reuse is read-only. -Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic and session-scoped; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits until close. CLI `annotate-native-function` returns the updated analysis before discarding the session. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations. For naming only, `set_address_name` / `set_addresses_names` also work with Ghidra: they name function entries or data addresses (primary labels) and return compact per-address success booleans instead of a dossier. Ghidra rejects batch addresses that request the same name or a name another selected address holds; `raw_result.failures` gives each failure's reason. +Use `annotate_native_function` for one function name and/or entry comments, with at least one explicit change. Review its annotation readback and refreshed dossier inline. Changes are atomic; empty comments clear them, omitted fields preserve them. Later MCP calls observe edits. Normal close saves them when owned Program checkpoint mode is enabled; ephemeral mode discards them. CLI `annotate-native-function` follows the same setting when closing after its updated result. Select the [checkpoint workflow](#continue-editable-ghidra-analysis-across-processes) before edits that must survive a later process. Original executable bytes are unchanged; edits invalidate immutable snapshots. Windows P0 does not admit database mutations. For naming only, `set_address_name` / `set_addresses_names` also work with Ghidra: they name function entries or data addresses (primary labels) and return compact per-address success booleans instead of a dossier. Ghidra rejects batch addresses that request the same name or a name another selected address holds; `raw_result.failures` gives each failure's reason. For `compare_functions`, obtain `analyze_function` Evidence before and after the edit while the MCP session is open; annotation Evidence is not a comparison diff --git a/.github/workflows/real-agentenv.yml b/.github/workflows/real-agentenv.yml index a252eb691c..6121fea671 100644 --- a/.github/workflows/real-agentenv.yml +++ b/.github/workflows/real-agentenv.yml @@ -54,10 +54,11 @@ jobs: - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 with: python-version: "3.12" - # Public ECR caps anonymous pulls by data per source IP and hosted runners share IPs, so the - # framework's image builds can fail here for hours. mirror.gcr.io serves the same Docker - # official images under the same digests, so each base image the pinned framework builds from - # is pulled there and named as its Dockerfile names it, as the framework's own CI does. + # The pinned framework builds its gateway and service-db images only when pulling them from + # ghcr.io fails, from public ECR base images; public ECR caps anonymous pulls by data per source + # IP and hosted runners share IPs, so such a build can fail here for hours. mirror.gcr.io serves + # the same Docker official images under the same digests, so each base image those Dockerfiles + # name is pulled there and named as its Dockerfile names it, as the framework's own CI does. # Digest-pinned names are set in containerd, whose images Docker reads only on the containerd # image store; switching stores restarts dockerd, so it happens before anything uses Docker. - name: Use the containerd image store @@ -72,16 +73,16 @@ jobs: run: | runner_root="${RUNNER_TEMP}/rea-agentenv-runner" python -m venv "${runner_root}" - "${runner_root}/bin/python" -m pip install --require-virtualenv --no-cache-dir 'agentenv-framework==0.9.1334' + "${runner_root}/bin/python" -m pip install --require-virtualenv --no-cache-dir 'agentenv-framework==0.9.1431' "${runner_root}/bin/agent-env" --version docker version --format '{{.Server.Version}}' echo "REA_AGENT_ENV_COMMAND=${runner_root}/bin/agent-env" >> "${GITHUB_ENV}" echo "REA_DOCKER_COMMAND=$(command -v docker)" >> "${GITHUB_ENV}" - - name: Pull the pinned framework's public ECR base images from mirror.gcr.io + - name: Pull the gateway and service-db public ECR base images from mirror.gcr.io shell: bash run: | package="$("${REA_AGENT_ENV_COMMAND%/agent-env}/python" -c 'import agent_env, pathlib; print(pathlib.Path(agent_env.__file__).parent)')" - grep -rhoE '^FROM +public\.ecr\.aws/docker/library/[^ ]+' --include='Dockerfile*' "${package}" | awk '{print $2}' | sort -u | + grep -rhoE '^FROM +public\.ecr\.aws/docker/library/[^ ]+' --include='Dockerfile*' "${package}/env/gateway" "${package}/env/envs/service_db" | awk '{print $2}' | sort -u | while read -r image; do name=${image#public.ecr.aws/docker/library/} source=mirror.gcr.io/library/$name diff --git a/README.md b/README.md index a6343e167a..4a3ea462e7 100644 --- a/README.md +++ b/README.md @@ -205,6 +205,7 @@ Additional tools and host support depend on the target: | PE certificates | Authenticode WIN_CERTIFICATE entries and PKCS#7 signer/X.509 identity fields | Built-in static PE inspection; [PE certificates](docs/pe-certificates.md) | | PE imports | Import and delay-load DLL names, ordinals, IAT RVAs and pefile import hash | Built-in static PE inspection; [PE imports](docs/pe-imports.md) | | ELF notes | PT_NOTE/SHT_NOTE owners, types, GNU build-id and ABI-tag fields | Built-in static ELF inspection; [ELF notes](docs/elf-notes.md) | +| Binary hardening | PIE, NX, RELRO, CFG and related header flags for ELF, PE and Mach-O | Built-in static header inspection; [binary hardening](docs/binary-hardening.md) | | Apple IPS crashes | .ips metadata, threads, images, registers and documented crash classes | Built-in static JSON inspection; [Apple IPS](docs/apple-ips.md) | | Linux kernel modules | .modinfo keys, __versions CRCs, this_module presence, and signature trailer | Built-in static ELF inspection; [kernel modules](docs/kernel-modules.md) | | Erlang BEAM | IFF chunks, atoms, exports, imports, lambdas and the Code header | Built-in static BEAM inspection; [Erlang BEAM](docs/erlang-beam.md) | diff --git a/bridge/agentenv/capture.py b/bridge/agentenv/capture.py new file mode 100644 index 0000000000..b1cb06354e --- /dev/null +++ b/bridge/agentenv/capture.py @@ -0,0 +1,262 @@ +"""Run REA inside an agentenv build computer for REA on the host: capture the app, draft its task and export it. + +REA sends this file's source through the computer's ``computer-exec/v1`` route as +``python3 -I -c ``; each command prints one JSON line and exits. ``start`` +leaves the capture running on its own: its stdin is a FIFO it holds open for reading and writing, so it never +sees the end of its input, and ``mark`` and ``stop`` write their requests there and wait for the capture's own +acknowledgement on its progress log. A small shell around the capture writes its exit status to a file, the +mark a later command reads. REA runs as the session user in the home, with no ``AGENTENV_*`` variable, so +neither REA nor the app it launches holds the computer's control token, and with ``APT_CONFIG`` naming the +package lists ``start`` downloads as that user, so the capture finds the package's sources. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import shutil +import subprocess +import sys +import time + +POLL_S = 0.2 +TAIL_BYTES = 4096 + + +def _say(value: dict) -> int: + print(json.dumps(value), flush=True) + return 0 + + +def _tail(path: str) -> str: + try: + with open(path, "rb") as handle: + handle.seek(max(0, os.path.getsize(path) - TAIL_BYTES)) + return handle.read().decode("utf-8", "replace") + except OSError: + return "" + + +def _environment(work: str) -> dict: + environment = {name: value for name, value in os.environ.items() if not name.startswith("AGENTENV_")} + environment["APT_CONFIG"] = os.path.join(work, "apt", "apt.conf") + return environment + + +def _apt(work: str, timeout: float) -> dict: + """Download the package lists into the work folder as the session user, until one update succeeds.""" + root = os.path.join(work, "apt") + lists = os.path.join(root, "lists") + updated = os.path.join(root, "updated") + os.makedirs(os.path.join(lists, "partial"), exist_ok=True) + os.makedirs(os.path.join(root, "cache", "archives", "partial"), exist_ok=True) + with open(os.path.join(root, "apt.conf"), "w", encoding="utf-8") as conf: + conf.write( + f'Dir::State::Lists "{lists}";\n' + f'Dir::Cache "{root}/cache";\n' + 'Debug::NoLocking "true";\n' + 'APT::Update::Error-Mode "any";\n' + ) + # apt keeps lists compressed as it fetched them, such as *_Packages.lz4 on Ubuntu. + count = lambda: len([name for name in os.listdir(lists) if re.search(r"_Packages(\.\w+)?$", name)]) # noqa: E731 + # A marker left by an update that fetched nothing is not a success; download again. + if os.path.exists(updated) and count() > 0: + return {"lists": count(), "updated": False} + log = os.path.join(root, "update.log") + try: + with open(log, "wb") as output: + code = subprocess.run(["apt-get", "update"], env=_environment(work), stdin=subprocess.DEVNULL, + stdout=output, stderr=subprocess.STDOUT, timeout=timeout).returncode + except (OSError, subprocess.TimeoutExpired) as error: + return {"lists": count(), "updated": True, "error": f"{type(error).__name__}: {error}"} + lists_fetched = count() + if code == 0 and lists_fetched > 0: + # Only a whole update that fetched lists is kept, so a failed or empty one runs again. + open(updated, "w").close() + return {"lists": lists_fetched, "updated": True, "exit_code": code} + if code == 0: + return {"lists": lists_fetched, "updated": True, "exit_code": code, + "error": "apt-get update exited 0 without a package list"} + return {"lists": lists_fetched, "updated": True, "exit_code": code, + "error": f"apt-get update exited {code}: {_tail(log)[-1500:].strip()}"} + + +def _exit_code(capture: str): + try: + with open(os.path.join(capture, "exit"), encoding="utf-8") as handle: + return int(handle.read().strip() or "-1") + except (OSError, ValueError): + return None + + +def _event_line(path: str, offset: int, kinds: set, name): + """The first progress line after ``offset`` whose event is of one of ``kinds``, for ``name`` when given.""" + try: + with open(path, "rb") as handle: + handle.seek(offset) + lines = handle.read().decode("utf-8", "replace").splitlines() + except OSError: + return None + for line in lines: + try: + event = json.loads(line)["rea_progress"]["event"] + except (ValueError, KeyError, TypeError): + continue + if isinstance(event, dict) and event.get("kind") in kinds and (name is None or event.get("name") == name): + return line + return None + + +def _wait(capture: str, offset: int, kinds: set, name, timeout: float) -> dict: + """Wait for the capture's event, or for it to exit, or for time to run out.""" + log = os.path.join(capture, "progress.log") + deadline = time.monotonic() + timeout + while True: + line = _event_line(log, offset, kinds, name) + if line is not None: + return {"line": line} + code = _exit_code(capture) + if code is not None: + return {"exited": code, "progress": _tail(log), "output": _tail(os.path.join(capture, "capture.out"))} + if time.monotonic() > deadline: + return {"timed_out": True, "progress": _tail(log)} + time.sleep(POLL_S) + + +def start(arguments: dict) -> int: + work, capture = os.path.abspath(arguments["work"]), os.path.abspath(arguments["capture"]) + rea = shutil.which("rea") + if rea is None: + return _say({"error": "REA is not installed in this computer; start the computer from a request, which installs it."}) + if os.path.exists(capture): + return _say({"error": f"the capture folder {capture} is already used"}) + apt = _apt(work, arguments.get("apt_timeout_s", 600)) + try: + os.makedirs(capture) + except FileExistsError: + return _say({"error": f"a stop claimed {capture} before the capture launched", "apt": apt}) + control = os.path.join(capture, "control") + os.mkfifo(control, 0o600) + argv = [rea, "capture-desktop-application", "--desktop-entry", arguments["desktop_entry"], + "--output", os.path.join(capture, "capture"), "--exclude", work, "--json"] + for value in arguments.get("arguments", []): + argv += ["--argument", value] + if arguments.get("package_file"): + argv += ["--package-file", arguments["package_file"]] + # Reading and writing, so the capture never reads the end of its input between requests. + reader = os.open(control, os.O_RDWR) + try: + with open(os.path.join(capture, "capture.out"), "wb") as out, \ + open(os.path.join(capture, "progress.log"), "wb") as progress: + process = subprocess.Popen( + ["sh", "-c", 'done=$1; shift; "$@"; code=$?; printf "%s" "$code" > "$done.tmp" && mv "$done.tmp" "$done"', + "sh", os.path.join(capture, "exit"), *argv], + cwd=os.path.expanduser("~"), env=_environment(work), stdin=reader, stdout=out, stderr=progress, + start_new_session=True) + finally: + os.close(reader) + waited = _wait(capture, 0, {"window_open"}, None, arguments.get("timeout_s", 120)) + return _say({"pid": process.pid, "apt": apt, **waited}) + + +def _request(capture: str, line: str) -> bool: + """Write one request to the running capture; False when nothing reads its FIFO any more.""" + try: + fd = os.open(os.path.join(capture, "control"), os.O_WRONLY | os.O_NONBLOCK) + except OSError: + return False + try: + os.write(fd, (line + "\n").encode("utf-8")) + finally: + os.close(fd) + return True + + +def mark(arguments: dict) -> int: + capture, name = os.path.abspath(arguments["capture"]), arguments["name"] + code = _exit_code(capture) + log = os.path.join(capture, "progress.log") + if code is not None: + return _say({"exited": code, "progress": _tail(log)}) + # A checkpoint already taken, whose answer was lost, is answered from the capture's own log. + taken = _event_line(log, 0, {"checkpoint"}, name) + if taken is not None: + return _say({"line": taken}) + offset = os.path.getsize(log) + if not _request(capture, f"checkpoint {name}"): + return _say({"error": "the capture is not reading its requests"}) + return _say(_wait(capture, offset, {"checkpoint", "checkpoint_refused"}, name, arguments.get("timeout_s", 600))) + + +def stop(arguments: dict) -> int: + capture = os.path.abspath(arguments["capture"]) + if _exit_code(capture) is None and not os.path.exists(os.path.join(capture, "control")): + # Never launched: claim the folder, so a start still on its way finds it used and launches nothing. + os.makedirs(capture, exist_ok=True) + return _say({"absent": True}) + if _exit_code(capture) is None: + _request(capture, "stop") + deadline = time.monotonic() + arguments.get("timeout_s", 600) + while (code := _exit_code(capture)) is None and time.monotonic() < deadline: + time.sleep(POLL_S) + log = os.path.join(capture, "progress.log") + if code is None: + return _say({"timed_out": True, "progress": _tail(log)}) + output = os.path.join(capture, "capture.out") + return _say({"exit_code": code, "progress": _tail(log), "output_bytes": os.path.getsize(output)}) + + +def run(arguments: dict) -> int: + """Run one REA command with its output in a file, after writing the files it reads.""" + rea = shutil.which("rea") + if rea is None: + return _say({"error": "REA is not installed in this computer"}) + for path, text in arguments.get("files", {}).items(): + os.makedirs(os.path.dirname(path), exist_ok=True) + with open(path, "w", encoding="utf-8") as handle: + handle.write(text) + with open(arguments["stdout"], "wb") as out: + try: + done = subprocess.run([rea, *arguments["argv"]], cwd=os.path.expanduser("~"), + env=_environment(os.path.abspath(arguments["work"])), stdin=subprocess.DEVNULL, stdout=out, + stderr=subprocess.PIPE, timeout=arguments.get("timeout_s", 1800)) + except subprocess.TimeoutExpired: + return _say({"timed_out": True}) + return _say({"exit_code": done.returncode, "stdout_bytes": os.path.getsize(arguments["stdout"]), + "stderr": done.stderr[-TAIL_BYTES:].decode("utf-8", "replace")}) + + +def listing(arguments: dict) -> int: + """Every regular file under a folder, by its path relative to the home, with its size and SHA-256. + + The list goes to the file ``output`` names, outside the folder, and the answer gives its size: a large + bundle's list outgrows the 1 MiB of stdout a command answers with, so REA reads it through ``data/get``. + """ + home = os.path.expanduser("~") + files, skipped = [], [] + for directory, folders, names in os.walk(os.path.abspath(arguments["root"])): + folders.sort() + for name in sorted(names): + path = os.path.join(directory, name) + relative = os.path.relpath(path, home) + if os.path.islink(path) or not os.path.isfile(path): + skipped.append(relative) + continue + digest = hashlib.sha256() + with open(path, "rb") as handle: + for chunk in iter(lambda: handle.read(1 << 20), b""): + digest.update(chunk) + files.append({"path": relative, "size": os.path.getsize(path), "sha256": digest.hexdigest()}) + output = os.path.abspath(arguments["output"]) + os.makedirs(os.path.dirname(output), exist_ok=True) + with open(output, "w", encoding="utf-8") as handle: + json.dump({"files": files, "skipped": skipped}, handle) + return _say({"listing_bytes": os.path.getsize(output)}) + + +COMMANDS = {"start": start, "mark": mark, "stop": stop, "run": run, "list": listing} + +if __name__ == "__main__": + sys.exit(COMMANDS[sys.argv[1]](json.loads(sys.argv[2]))) diff --git a/bridge/agentenv/computer.py b/bridge/agentenv/computer.py new file mode 100644 index 0000000000..fc1540e29b --- /dev/null +++ b/bridge/agentenv/computer.py @@ -0,0 +1,83 @@ +"""Keep one agentenv computer up for REA until stdin closes or SIGTERM arrives. + +REA runs ``computer.py [task]`` with the agentenv runner's own interpreter, so ``agent_env`` +imports. It deploys the bundle's task, ``smoke`` by default, on this machine with +``run_bundle(keep=True, emulate=False)``, prints one JSON line naming the computer's address, MCP URL, control +token and sandbox id, or ``{"error": ...}``, and tears the computer down once its stdin closes or it gets SIGTERM +or SIGINT. Everything else it or agentenv prints, a build's output included, goes to stderr. It exits 0 once +the teardown left nothing up, 1 when it did, and 2 when no computer came up. +""" + +from __future__ import annotations + +import json +import os +import signal +import sys +import threading + + +def _protocol(): + """A private copy of stdout for the JSON line; fd 1 then points at stderr, as do children that inherit it.""" + protocol = os.fdopen(os.dup(1), "w", encoding="utf-8") + os.dup2(2, 1) + sys.stdout = sys.stderr + return protocol + + +def _say(protocol, value: dict) -> None: + protocol.write(json.dumps(value) + "\n") + protocol.flush() + + +def _computer(result): + """The run's deployed computer, or why there is none.""" + run = result.runs[0] if result.runs else None + if run is None: + return None, "the bundle ran no task" + if run.error is not None or run.failed_steps: + return None, f"the task failed: {run.error or run.failed_steps}" + for env in run.context.deployed_envs: + if env.environment_url and env.mcp_url and getattr(env, "sandbox_id", None): + return env, None + return None, "the task deployed no computer" + + +def main(argv: list[str]) -> int: + protocol = _protocol() + bundle, task = argv[1], argv[2] if len(argv) > 2 else "smoke" + try: + from agent_env.bundle import run_bundle + except ImportError as error: + _say(protocol, {"error": f"agent_env does not import with {sys.executable}: {error}"}) + return 2 + try: + result = run_bundle(bundle, tasks=[task], keep=True, emulate=False, sandbox="local", + on_progress=lambda line: print(line, file=sys.stderr, flush=True)) + except BaseException as error: # noqa: BLE001 - a signal's RunInterrupted too; run_bundle tore its runs down + _say(protocol, {"error": f"{type(error).__name__}: {error}"}) + return 2 + stop = threading.Event() + for signum in (signal.SIGTERM, signal.SIGINT): + signal.signal(signum, lambda *_: stop.set()) + env, problem = _computer(result) + try: + if env is None: + _say(protocol, {"error": problem}) + else: + _say(protocol, {"base_url": env.environment_url, "mcp_url": env.mcp_url, + "control_token": env.control_token, "sandbox_id": env.sandbox_id}) + threading.Thread(target=lambda: (sys.stdin.buffer.read(), stop.set()), daemon=True).start() + while not stop.wait(0.5): + pass + finally: + torn_down = result.teardown() + still_up = [sandbox for run in torn_down.runs if run.torn_down is not None for sandbox in run.torn_down.still_up] + if still_up: + print(f"still up after the teardown: {still_up}", file=sys.stderr, flush=True) + return 1 + return 0 if env is not None else 2 + + +if __name__ == "__main__": + sys.exit(main(sys.argv)) diff --git a/bridge/ghidra/ReaGhidraBridge.java b/bridge/ghidra/ReaGhidraBridge.java index 6b294efc18..9235e24f85 100644 --- a/bridge/ghidra/ReaGhidraBridge.java +++ b/bridge/ghidra/ReaGhidraBridge.java @@ -187,6 +187,8 @@ public final class ReaGhidraBridge extends HeadlessScript { private static final int MAX_DECOMPILE_TIMEOUT_SECONDS = Integer.MAX_VALUE / 1000; private final ReaGhidraExtensions analysisExtensions = new ReaGhidraExtensions(); private static AddressSpace sessionDefaultAddressSpace; + private Path checkpointDirectory; + private long checkpointModification; @Override public void run() throws Exception { @@ -195,6 +197,7 @@ public final class ReaGhidraBridge extends HeadlessScript { throw new IllegalArgumentException("REA bridge requires one session descriptor"); } Path descriptorPath = Path.of(arguments[0]); + checkpointDirectory = descriptorPath.toAbsolutePath().normalize().getParent(); SessionDescriptor descriptor = readDescriptor(descriptorPath); decompileTimeoutSeconds = descriptor.decompileTimeoutSeconds(); // Windows retains a native immutable bearer descriptor lease until @@ -247,6 +250,7 @@ public final class ReaGhidraBridge extends HeadlessScript { throw new IOException("Windows Ghidra P0 does not admit metadata recovery or database mutation."); analysisExtensions.analyze(descriptor.analysisExtensions, currentProgram, monitor); initializeDecompiler(); + checkpointModification = currentProgram.getModificationNumber(); serve(descriptor); } finally { @@ -451,6 +455,7 @@ public final class ReaGhidraBridge extends HeadlessScript { if (descriptor.existingProject && isMutatingOperation(request.method)) throw new RequestFailure("method_unavailable", "Existing Ghidra project sessions are read-only"); return switch (request.method) { + case "checkpoint_program" -> checkpointProgram(request.params, descriptor); case "read_binary_inventory_summary" -> binaryInventorySummary(request.params); case "inspect_native_load_image" -> inspectNativeLoadImage(request.params); case "read_bytes" -> readMemoryBytes(request.params); @@ -506,6 +511,31 @@ public final class ReaGhidraBridge extends HeadlessScript { return method.equals("annotate_native_function") || method.equals("set_address_name") || method.equals("set_addresses_names"); } + /** Internal lifecycle request: the destination is inside this owned runtime. */ + private JsonObject checkpointProgram(JsonObject params, SessionDescriptor descriptor) throws Exception { + requireKeys(params, Set.of("path", "force")); + if (descriptor.existingProject || !descriptor.transport.equals("unix-socket")) + throw new RequestFailure("method_unavailable", "This Program has no checkpoint authority"); + Path output = Path.of(requireString(params, "path")).toAbsolutePath().normalize(); + if (!output.getParent().equals(checkpointDirectory) || + !output.getFileName().toString().matches("checkpoint-[a-f0-9-]{36}\\.gzf")) + throw new RequestFailure("invalid_request", "Checkpoint destination must belong to this runtime"); + JsonElement forced = params.get("force"); + if (!forced.isJsonPrimitive() || !forced.getAsJsonPrimitive().isBoolean()) + throw new RequestFailure("invalid_request", "Checkpoint force must be a boolean"); + boolean force = forced.getAsBoolean(); + JsonObject result = new JsonObject(); + boolean changed = currentProgram.getModificationNumber() != checkpointModification; + result.addProperty("saved", force || changed); + if (force || changed) { + if (Files.exists(output)) throw new RequestFailure("invalid_request", "Checkpoint destination already exists"); + currentProgram.saveToPackedFile(output.toFile(), monitor); + checkpointModification = currentProgram.getModificationNumber(); + result.addProperty("path", output.toString()); + } + return result; + } + private JsonObject sessionInfo(SessionDescriptor descriptor) throws Exception { boolean timedOut = analysisTimeoutOccurred(); JsonObject provider = new JsonObject(); @@ -1296,6 +1326,12 @@ public final class ReaGhidraBridge extends HeadlessScript { if (component.getDataType() instanceof ghidra.program.model.data.BitFieldDataType bitfield) { field.addProperty("bit_size", bitfield.getBitSize()); field.addProperty("bit_offset", bitfield.getBitOffset()); + JsonObject bitField = new JsonObject(); + bitField.addProperty("base_type_id", bitfield.getBaseDataType().getPathName()); + bitField.addProperty("declared_bit_size", bitfield.getDeclaredBitSize()); + bitField.addProperty("component_type_lookup", typeLookupResult(bitfield)); + bitField.addProperty("base_type_lookup", typeLookupResult(bitfield.getBaseDataType())); + field.add("bit_field", bitField); } field.addProperty("status", "observed"); field.addProperty("flexible_tail", "unknown"); @@ -1326,7 +1362,7 @@ public final class ReaGhidraBridge extends HeadlessScript { result.add("members", members); result.addProperty("total_fields", totalFields); result.addProperty("truncated", totalFields > 20_000); - result.add("limitations", GSON.toJsonTree(List.of("Layouts are Ghidra data-type database observations; original debug/source authority and flexible-tail semantics are not inferred. Nested and recursive types retain category-path identities for separate inspection.", "At most 20000 fields or enum members are serialized per type to bound provider output. total_fields and truncated report omissions. Enum values preserve Ghidra's signed long representation."))); + result.add("limitations", GSON.toJsonTree(List.of("Layouts are Ghidra data-type database observations; original debug/source authority and flexible-tail semantics are not inferred. Component type paths do not necessarily identify separately registered types; bit-field lookups report whether the manager resolves equivalent types.", "At most 20000 fields or enum members are serialized per type to bound provider output. total_fields and truncated report omissions. Enum values preserve Ghidra's signed long representation."))); if (type != null) { JsonObject metadata = analysisExtensions.inspectDataType(currentProgram, type, address, monitor); if (metadata != null) result.add("metadata_recovery", metadata); @@ -1334,6 +1370,11 @@ public final class ReaGhidraBridge extends HeadlessScript { return result; } + private String typeLookupResult(DataType type) { + DataType resolved = currentProgram.getDataTypeManager().getDataType(type.getPathName()); + return resolved == null ? "not-found" : resolved.isEquivalent(type) ? "equivalent" : "different"; + } + private JsonObject instructionReference(Reference reference) { JsonObject item = new JsonObject(); JsonObject kind = referenceKind(reference); diff --git a/docs/.vitepress/config.ts b/docs/.vitepress/config.ts index d644d593eb..e7dcd1acbb 100644 --- a/docs/.vitepress/config.ts +++ b/docs/.vitepress/config.ts @@ -36,6 +36,7 @@ export default defineConfig({ { text: "PE imports", link: "/pe-imports" }, { text: "PE TLS/load config", link: "/pe-load-config" }, { text: "ELF notes", link: "/elf-notes" }, + { text: "Binary hardening", link: "/binary-hardening" }, { text: "Apple IPS crash reports", link: "/apple-ips" }, { text: "Linux kernel modules", link: "/kernel-modules" }, { text: "CPython pyc files", link: "/python-pyc" }, @@ -61,6 +62,30 @@ export default defineConfig({ { text: "Electron observation", link: "/electron-observation" }, { text: "Electron fuses", link: "/electron-fuses" }, { text: "Process capture", link: "/process-capture" }, + { + text: "Desktop application capture", + link: "/desktop-application-capture", + }, + { + text: "Desktop application state", + link: "/desktop-application-state", + }, + { + text: "Desktop application export", + link: "/desktop-application-export", + }, + { + text: "Desktop application tasks", + link: "/desktop-application-tasks", + }, + { + text: "Desktop build computer", + link: "/desktop-build-computer", + }, + { + text: "Building and witnessing", + link: "/desktop-build-witness", + }, { text: ".NET assemblies", link: "/managed-code-analysis" }, { text: "Python packaged applications", diff --git a/docs/adb-device-analysis.md b/docs/adb-device-analysis.md index efb8bf2b34..79c74fd2fb 100644 --- a/docs/adb-device-analysis.md +++ b/docs/adb-device-analysis.md @@ -25,7 +25,21 @@ REA never installs the Android SDK, an emulator, or a system image. Readiness is `rea inspect-adb-client`: it runs `adb version` and reports the resolved binary, its version, and where it was selected from. A missing binary reports `capability_unavailable` with the `REA_ADB_PATH` remediation -rather than attempting an installation. +rather than attempting an installation. A path that cannot resolve to a file +(`ENOENT`, `ENOTDIR`, `ELOOP` or `ENAMETOOLONG`) is reported as missing. A +selected path the host refuses to execute, such as a directory or a file +without execute permission (`EACCES`) or one the host does not permit +(`EPERM`), and any other spawn error report `capability_unavailable` naming +that path. Exhausted host capacity (`EAGAIN`, `EMFILE`, `ENFILE` or `ENOMEM`) +is not a configuration failure. `inspect_adb_client`, and any device +operation whose adb cannot start at its version probe or at a later adb +command, report a retryable `capacity` resource constraint rather than a +capability refusal or an adb protocol failure; the `binary_session` inventory +reports the provider as `runtime_unavailable`. Both say the host could not +start adb and ask for a retry after freeing processes or descriptors, without +blaming `REA_ADB_PATH`. When adb never started, the refusal diagnostics record +the spawn error as `filesystem_error` with `path_source`, never as an +`exit_code`. The `binary_session` inventory uses the same bounded client-version probe; it does not query devices, start the adb server, or establish that a requested serial is reachable. Device state is checked by each device operation. @@ -210,9 +224,11 @@ and during shutdown. them in `raw_result.device_path_probes`. Incomplete startup, extra diagnostics, truncation and cancellation do not establish absence. The check is not atomic, so a different process can create the path before `adb push` writes it. When available, the device's - `sha256sum` verifies the transfer. A missing command or output without a - recognizable digest reports `device_digest_source: "unavailable"` with an - unverified-transfer limitation; installs preserve the + `sha256sum` verifies the transfer. A missing command, output without a + recognizable digest, or host capacity that prevents the digest check from + starting reports `device_digest_source: "unavailable"` with an + unverified-transfer limitation rather than a retryable error; installs + preserve the device's `Failure [...]` reason verbatim. - **Producer identity stays attached.** Every result embeds the producing adb client (`binary_path`, `path_source`, `version`, `installed_path`). diff --git a/docs/agent-environments-quickstart.md b/docs/agent-environments-quickstart.md index 251afd1447..4e64305714 100644 --- a/docs/agent-environments-quickstart.md +++ b/docs/agent-environments-quickstart.md @@ -19,7 +19,7 @@ records the app you provide; you start it and prepare its test data. To run the exported environment, install the [AgentEnv framework](https://github.com/scaleapi/agentenv-framework#install) and start Docker. REA's [compatibility check](testing.md#exported-environment-bundles) -uses AgentEnv version `0.9.1334` with Python 3.12. +uses AgentEnv version `0.9.1431` with Python 3.12. There are two ways to build an environment: ask your agent to record the app, or start from a recording you already have. diff --git a/docs/agent-environments.md b/docs/agent-environments.md index 71fd35bd65..be11df05db 100644 --- a/docs/agent-environments.md +++ b/docs/agent-environments.md @@ -1049,8 +1049,9 @@ unknown recorded status stays unknown; it is not borrowed from another record). A bundle with no such tool yields a verifier that checks the count only and says so. `tasks/smoke.json` deploys the env on the local sandbox and runs that verifier against the env's MCP endpoint; the first run -also builds the framework's own infrastructure, and the run tears everything -down when it ends. +also pulls the framework's published infrastructure images from `ghcr.io`, +building one only if its pull fails, and the run tears everything down when it +ends. The README written into the bundle records the capture artifact's SHA-256, the capture and manifest Evidence ids, the manifest view digest, the served tool diff --git a/docs/api-contracts.md b/docs/api-contracts.md new file mode 100644 index 0000000000..bc9e752db8 --- /dev/null +++ b/docs/api-contracts.md @@ -0,0 +1,178 @@ +# Offline API contract inventory + +`inspect_api_contract` / `rea inspect-api-contract` supports local OpenAPI JSON +and GraphQL SDL (`format: "graphql-sdl"`). Neither profile contacts endpoints. + +## OpenAPI JSON + +The default profile reads one local OpenAPI +3.0, 3.1 or 3.2 JSON document. It inventories inline operation slots under +`paths`; the facts describe declarations, not implemented, reachable or +authorized runtime behavior. YAML, AsyncAPI, callbacks/webhooks, schema graphs, +contract diffs and traffic correlation are outside this version. + +The OpenAPI importer never resolves references, opens sibling documents, evaluates +examples or contacts declared servers. OpenAPI 3.2 `query` and +`additionalOperations` follow the [Path Item contract](https://spec.openapis.org/oas/v3.2.1.html#path-item-object), +including case-sensitive additional HTTP method spelling. + +## Select a useful answer + +MCP defaults to a summary retaining the complete inventory. An optional view +returns selected facts immediately, taking precedence over `detail`: + +```json +{ + "path": "/artifacts/openapi.json", + "view": { + "kind": "api-contract", + "collection": "operations", + "methods": ["GET"], + "limit": 20 + } +} +``` + +`operations` supports exact `operation_ids`, `operation_keys`, `methods`, +`paths`, `declared_ids` and `states` filters. Filters combine by intersection; +empty arrays select nothing. `fields` chooses columns; identity, source pointer, +state and unknown reason remain. The other collections are +`path-item-references` (optional `reference_ids`) and `diagnostics` (optional +`codes`), also with fields and offset/limit. The default page contains 64 rows. + +Query a retained result with `inspect_analysis_view`, using the same view and +`source: {"kind":"retained-evidence","evidence_id":"..."}`. Copy the complete +parent ID from `normalized_result.parent_evidence_id`, not the summary's own ID. +These pages use immutable retained facts without rereading the specification. +Complete Evidence remains available through `get_evidence_bundle`. + +CLI defaults to complete Evidence. For a selected answer, save that complete +Evidence in an absent bundle path: + +```sh +rea inspect-api-contract ./openapi.json --detail summary \ + --view '{"kind":"api-contract","collection":"operations","methods":["GET"],"limit":20}' \ + --evidence_output ./contract.evidence.json --json +``` + +Later `rea inspect-analysis-view` can read that bundle through an exact +`evidence-file` source selection, even after the original specification is +unavailable. Publication never overwrites an existing file. CLI preserves its +original path spelling in `caller_input_path`; analysis locations and MCP inputs +use absolute filesystem paths. + +## Identity, provenance and coverage + +Each operation has an artifact-scoped `operation_id`, an exact method/path +`operation_key`, an optional `declared_id` from `operationId`, and its original +JSON Pointer. Keys can correlate exact declarations in a caller-selected pair +of revisions; they do not establish equivalence across unrelated APIs, renamed +path templates or runtime handlers. Declared IDs are not assumed unique. +The artifact includes the verified byte length, SHA-256 and selected path. + +Malformed siblings retain useful declarations plus diagnostics. A Path Item +`$ref`, including a local fragment, remains unresolved; adjacent inline slots +are unknown because referenced fields may overlap. Missing or malformed paths +are explicit, and missing required OpenAPI 3.0 paths does not become a complete +empty inventory. `source_coverage` is limited to inline paths and records that +reference resolution and full schema validation were not performed. Page +exhaustion describes the selected rows and never upgrades source coverage. + +Structured transport userinfo credentials are excluded from reference URIs; +`credential_redacted` reports that change, and an ambiguous credential span leaves +the displayed reference unknown. Ordinary reference query/fragment text and +names are retained. Original bytes remain in the caller's artifact; complete +inventory Evidence does not embed the entire source JSON. Duplicate JSON members +use the shared parser's last-value-wins behavior, recorded in source coverage. + +If decoding or later validation fails after a verified read, the typed error +retains a compact `file-snapshot-hash` observation with the exact path, digest +and byte length. Syntax diagnostics do not quote source JSON. Cancellation +does not replace a parser failure that has already completed. + +## Resource bounds and verification + +The file-byte allowance and conservative expanded-facts allowance are each +32 MiB resource policies, not OpenAPI format limits. `max_file_bytes` can lower +the input allowance. Native JSON parsing additionally checks available heap +against the shared representation-expansion estimate before decoding. The +shared JSON nesting limit is 1,000, including uninterpreted schema regions. +Capacity refusals are typed; reply paging does not increase parser allowances. + +The committed [Petstore fixture](https://github.com/morluto/rea/blob/b7c8caedd4166e6032c49024cdc6fb1154f1a2df/tests/fixtures/openapi/README.md) is an +original document from the published application, replayed offline through +public CLI/MCP tests. These check advertised schemas, complete/selected parity, +retained paging, partial references, zero requests to a loopback reference +server and unchanged source bytes. They do not establish runtime endpoint +behavior or full OpenAPI conformance. + +## GraphQL SDL and operation documents + +Select `format: "graphql-sdl"` for SDL. Optionally supply `operations_path` to +one local document containing operations and fragments. Both MCP paths are +absolute; CLI accepts relative spellings: + +```sh +rea inspect-api-contract ./schema.graphql --contractFormat graphql-sdl \ + --operationsPath ./queries.graphql --detail summary \ + --view '{"kind":"graphql-contract","collection":"references","limit":20}' \ + --evidence_output ./graphql.evidence.json --json +``` + +Collections are `types`, `fields`, `definitions`, `references`, `fragment-spreads` +and `diagnostics`. Each supports `fields`, `offset` and `limit` (default 64). +Exact batch filters include type/field/definition/reference IDs, type names, +`field_keys` such as `Query.viewer`, owning type names, binding/spread states +and diagnostic codes. Filters intersect; an empty array selects nothing. +Identity, source and applicable uncertainty fields always remain. Retained pages +and bundle-based queries use the same `graphql-contract` views without rereading +SDL or rebuilding its schema model. + +Types and fields retain individual declaration occurrences, including extensions, +list/non-null wrappers, implemented interfaces and union member names. Operation +and fragment definitions own field references with aliases and parent-reference +IDs. Each reference links to a supplied SDL field occurrence, remains `model-only` +for a library-provided field such as `__typename`, or records an exact unknown +reason. Rejected schema construction keeps declarations but cannot supply trusted +bindings. Missing or duplicate fragment names remain explicit. Fragments are +visited once in their declared type scope and linked by spread edges; cycles +are never recursively expanded. + +The versioned `graphql-sdl-field-bindings-v1` profile records the GraphQL library +version. It performs syntax parsing and schema construction for field resolution; +it does not establish whole-schema/operation validity, spread applicability, +reachability, authorization or endpoint behavior. Arguments, defaults, +descriptions, directive behavior, enum value inventory, introspection JSON, +federation composition and version comparison remain outside this profile. + +Every fact has a document role, SHA-256, UTF-16 start/end offsets and parser token +line/column. Offsets address decoded UTF-8 text, including an optional BOM, rather +than byte positions. The two files are independently verified snapshots, not an +atomic pair. Fact IDs identify source occurrences; `field_key` separately supports +exact name-based joins and does not assert equivalence across revisions. CLI +retains its auxiliary path spelling in `caller_input_paths.operations_path`, +outside semantic identity, as well as its primary `caller_input_path`. + +Malformed operation text retains the SDL inventory with partial coverage and a +source-located diagnostic. A primary parser refusal or later resource/cancellation +failure retains known source identities and available interpreted Evidence; +syntax diagnostics do not quote source text. Complete Evidence contains facts, +not original document bytes. An exhausted page never upgrades source coverage. + +Each file has the same configurable maximum of 32 MiB. Conservatively charged +facts share a 32 MiB allowance; text/AST allocation is admitted against available +heap. The parser admits at most 262,144 lexical tokens and 65,536 comments per +document; subsequent iterative AST admission limits traversal depth to 128. +These are REA resource policies, not GraphQL language limits. Very deep input can +be refused by the parser before AST admission. Selected replies reduce transport +and retained queries reuse immutable facts; they do not avoid the initial parse. + +The committed SWAPI SDL is an unchanged public application fixture with provenance +in `tests/fixtures/graphql/README.md`. It supplies 53 types and 243 fields; +verification does not execute its API. + +Large complete inventories can exceed the default MCP response budget even when +selected pages fit. Use `export_evidence_bundle` with a caller-selected local +path for complete retained Evidence, or CLI complete JSON/bundle publication. +A transport refusal does not erase a completed retained analysis. Narrow pages +and source coverage remain available on the same continuing connection. diff --git a/docs/binary-hardening.md b/docs/binary-hardening.md new file mode 100644 index 0000000000..2d54d1cda9 --- /dev/null +++ b/docs/binary-hardening.md @@ -0,0 +1,62 @@ +# Binary hardening + +Use `inspect_binary_hardening` to read compile-time and link-time hardening from +an explicit local ELF, PE, or Mach-O file. No disassembler, pwntools install, or +target execution is required. Universal Mach-O files return one image per slice. + +```sh +rea inspect-binary-hardening /absolute/path/to/program --json +rea inspect-binary-hardening /absolute/path/to/program --max-file-bytes 1048576 --json +``` + +MCP accepts `{ "path": "/absolute/path/to/program" }`. CLI paths may be relative +to the current directory; MCP paths must be absolute. + +Each image has one row per property: `property`, `state`, `confidence`, `basis`, +`location`, `note`, and `next_probe`. States are `enabled`, `disabled`, +`partial`, `unknown`, and `not_applicable`. `confidence` is `observed` for a +single header or import, and `derived` when the state combines several fields, +such as ELF PIE or full RELRO. An image's `architecture` is its CPU family, or +null when REA has no family for it. `machine` and `slice.architecture` keep the +exact identity, such as `arm64e`. + +ELF rows cover PIE (`ET_DYN` with `PT_INTERP` or `DF_1_PIE`), NX (`PT_GNU_STACK`; +a missing header is `unknown` and the architecture default is only noted), RELRO +(`PT_GNU_RELRO` plus `BIND_NOW`), stack protector and FORTIFY undefined dynamic +imports, CET and AArch64 BTI/PAC GNU property notes, `DT_RPATH`, `DT_RUNPATH`, and +whether a `SHT_SYMTAB` section is present. Only an undefined dynamic import +counts. Imports come from the table `DT_SYMTAB` declares. A `SHT_DYNSYM` section +only supplies the symbol count when `DT_HASH` and `DT_GNU_HASH` leave it open, +and only if it names the same symbol and string tables. Without that count the +import rows are `unknown` and give the reason. A missing `__stack_chk_fail` or fortify import is `unknown`. That +absence does not establish that the protection was left out of a static or +stripped build. A present import is a sign that some code uses the protection, +not proof that all code does. NX, RELRO, `DT_RPATH`, and `DT_RUNPATH` are +`not_applicable` when `e_type` is neither `ET_EXEC` nor `ET_DYN`, and `unknown` +for a linked image with no readable program headers. + +PE rows read `DllCharacteristics` for dynamic base, high-entropy VA, NX +compatibility, force integrity, no SEH, and guard CF. `cfg_instrumented` is the +load-config `GuardFlags` linker claim when `Size` covers that field. An +unreadable load configuration, including a TLS failure in that parser, leaves +only `cfg_instrumented` `unknown`. Authenticode and debug rows report certificate +and debug directory presence, not signature validity or debug content. A +directory whose address and size disagree is `unknown`. + +Mach-O rows read `MH_PIE`, `MH_ALLOW_STACK_EXECUTION`, `MH_NO_HEAP_EXECUTION`, +an `__RESTRICT` segment, `LC_CODE_SIGNATURE` presence, encryption from the +existing encryption reader, the arm64e subtype, and a stack-protector import. +`hardened_runtime` is `unknown` with `next_probe` `inspect_signature`. +CodeDirectory flags stay with that tool. + +The report states presence of build-time features. It does not state +exploitability. `inspect_binary_layout` is unchanged. + +The default read budget is 64 MiB (`max_file_bytes`, ceiling 512 MiB). A changed +file is refused. A malformed header, a Mach-O segment or encryption command that fails admission, or a program, section, +or segment payload declared outside the file returns no partial report, except that one bad slice in a universal Mach-O leaves the other slices. A symbol table or +symbol name over its budget leaves the import rows `unknown` and keeps the +header rows. A GNU property note over its budget, or a compressed note section +that may hold one, leaves the CET and BTI/PAC rows `unknown` and keeps the header +rows. A dynamic-table budget still fails the ELF report. The image is never +executed. diff --git a/docs/cli.md b/docs/cli.md index db22675191..162bae2f49 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -349,7 +349,10 @@ unknown names, resource limits and retained views. Compressed bytes stay stored. Zstandard (standard method 93). They retain the shared ZIP CRC, declared-size, overlap and archive expansion checks. The native streaming decoder keeps the existing 128 MiB Zstandard window cap; frames requiring a larger window are -refused. Deprecated method 20 and other unregistered ZIP codecs remain unsupported. +reported as a resource limit (`reason: limit`, `boundary: memory`), rather than +as archive corruption. The cap cannot be raised at runtime: recompress with a +window of at most 128 MiB, or decompress externally before retrying. +Deprecated method 20 and other unregistered ZIP codecs remain unsupported. Inventory and extract UF2 or DfuSe regions: @@ -473,7 +476,8 @@ applicable. gzip, xz, and zstd containers are unsupported and name utility, then inspect the decompressed file. See [Linux kernel modules](kernel-modules.md). -Inspect a CPython 3.7–3.13 `.pyc` header and nested marshal code objects +Inspect a CPython 3.7–3.13 `.pyc` header and nested marshal code objects (3.14 and +3.15 headers are qualified but their marshal layout is not pinned) without executing bytecode or installing Python: ```bash @@ -494,6 +498,15 @@ The workflow reports IFF chunk names, atom tables, exports, imports, lambdas, and the Code chunk header. Missing optional tables stay null. See [Erlang BEAM files](erlang-beam.md). +Complete `rea decompile --json` Evidence can be saved and inspected through +`inspect-analysis-view` with an inline source and +`view: {kind: "native", facet: "pseudocode", offset: 0, limit: 20000}`. +The projection reads that saved observation without starting a provider; +offsets/limits count UTF-16 code units. A summary view preserves the original +selector and distinguishes null from empty text. The MCP tool additionally +supports `detail: "summary"` on its first call to retain the complete source +without returning the body inline; see [native investigation](native-investigation.md). + Configure [Hopper or Ghidra](installation.md#hopper), the [IDA adapter](ida-provider.md), or [Kuna](kuna-provider.md) before analyzing a native target. Substitute @@ -734,10 +747,12 @@ not read artifact contents: the inventories of two builds are unresolved records, or one opaque change when you pair them. Use `compare-artifacts`, described at the end of this section, to compare their contents. -`--record_pairs` supplies an explicit JSON array of observation pairs, matching -MCP `compare_bundles.record_pairs`. Pair IDs must exist on their named sides and -be one-to-one. Without pairs, one-sided membership remains unknown; REA does -not infer correspondence from position or display names. +`--record_pairs` accepts an inline JSON array or the path to a local JSON +file containing that array, matching MCP `compare_bundles.record_pairs`. For a +file, use `rea compare left.json right.json --record_pairs pairs.json --json`. +Pair IDs must exist on their named sides and be one-to-one. Without pairs, +one-sided membership remains unknown; REA does not infer correspondence from +position or display names. Exports preserve an existing destination unless `--overwrite` is explicit. @@ -784,7 +799,8 @@ from the bundle for every supported view kind. Bundles exported from MCP work for these commands. Its inline source form cannot also select a bundle. A missing ID is refused with `details.reason: "missing"`; supplying a bundle does not make incompatible Evidence valid for a workflow. -`compare-managed-members` keeps its two assembly paths, and +`compare-managed-members` keeps its two assembly paths and an optional +repeatable `--type` selection applied to both sides, and `compare-process-captures` keeps its two capture Evidence JSON paths. The bundle option above selects an input; producer options that write a new bundle, such as `inspect-managed-members --evidence-bundle`, retain their existing meaning. @@ -822,6 +838,7 @@ Import an older source tree to compare with the current artifact: ```bash rea import-reference-source /absolute/path/to/source rea import-reference-source /absolute/path/to/source --detail summary --outline-depth 2 --json +rea import-reference-source /absolute/path/to/source --exclude-paths '["vendor/","*.generated","!keep.generated"]' ``` The default (`--detail full`) returns the complete `HistoricalSourceGraph`. @@ -968,6 +985,10 @@ Choose the guide for the target and intended interaction: - [Electron fuses](electron-fuses.md): read packaged fuse bytes before expecting `--inspect` to reach an Electron main process. - [Node/Electron Inspector](javascript-runtime-observation.md): record script locations and execution contexts. - [Process capture](process-capture.md): run an executable and compare terminal, exit and filesystem observations. +- [Desktop application capture](desktop-application-capture.md): launch a Linux desktop application and record its windows and home-directory state at checkpoints while it is used. +- [Desktop application state](desktop-application-state.md): read a capture folder into the application's state files, typed views of each version and kinds of change. +- [Desktop application export](desktop-application-export.md): write a capture folder as an agentenv computer app bundle with its class, skill and seed. +- [Desktop application tasks](desktop-application-tasks.md): describe how a capture's files changed, then validate an agent-written task against the capture and export it with its verifier. Runtime requests name the target and actions. Launched targets run with your user permissions; consult the chosen guide for host requirements and effects. @@ -1179,6 +1200,10 @@ such as `UNSUPPORTED_OUTPUT_COMBINATION` for `--json` with `--token-limit`. Use Commands reject surplus positional arguments before execution, including commands that accept no positionals. Use `rea --help` for the declared argument shape; a final variadic argument continues to accept multiple values. +`rea mcp add` accepts options only and rejects unknown options before reading or +backing up client settings or starting registration. Its `--agent` and +`--command`/`-c` options accept spaced and equals forms. Values must be nonempty; a following flag does not satisfy a spaced option's +missing value. `rea mcp doctor --json` checks production MCP startup and reports child logs in `diagnostics`. It drains stderr while retaining at most its first 64 KiB of raw @@ -1319,7 +1344,9 @@ the command's signal handling. Commands routed through the shared `withCommandCancellation` handler in the CLI set status `130` for SIGINT or `143` for SIGTERM after their operation settles. Current source registrations use that handler for process capture and comparison (`capture-process`, -`compare-process-captures`), JavaScript application analysis (`analyze` when it +`compare-process-captures`), desktop application capture and state +(`capture-desktop-application`, `inspect-desktop-application-state`, +`export-desktop-application`, `describe-desktop-application-task`), JavaScript application analysis (`analyze` when it selects the JavaScript lane, and `analyze-javascript-application`), selected browser, web and Electron commands (including `analyze-web-bundle`, `observe-web-execution`, `capture-browser-scenario` and @@ -1365,7 +1392,11 @@ status with the analysis failure status, so it exits `1`. Setup has its own cont structured result and process status as separate signals when scripting cancellation. -`setup --dry-run` returns `planned` and exits `0`; a cancelled setup also exits +`setup --dry-run` returns `planned` and exits `0`. For `setup` and +`environment-build`, explicit `--dry-run=true` and `--dry-run=false` are +accepted; empty or other values fail with `PARSE_ERROR` before any changes. +Every supplied value is checked even when a later occurrence overrides it. +A cancelled setup also exits `0`. Setup returns `1` for `needs_confirmation` or `needs_human`. `doctor` returns `1` when required checks in its selected readiness scope fail; unavailable optional providers remain informational for unrelated tasks. @@ -1410,6 +1441,16 @@ Reports TLS callbacks and size-covered load-config fields without execution. Use `--max-callbacks` and `--max-file-bytes` to set read budgets. See [PE TLS and load configuration](pe-load-config.md). +### Binary hardening + +```sh +rea inspect-binary-hardening /absolute/path/to/program --json +``` + +Reports ELF, PE, and Mach-O hardening rows without execution. A missing import +is `unknown`, not `disabled`. Use `--max-file-bytes` to set the read budget. See +[binary hardening](binary-hardening.md). + Bun standalone executables use the existing `inspect-artifact` and `extract-artifact` commands to inventory and materialize embedded source, sourcemaps and bytecode without running the executable. See diff --git a/docs/client-configuration.md b/docs/client-configuration.md index 5f04937b10..b2a8856ac9 100644 --- a/docs/client-configuration.md +++ b/docs/client-configuration.md @@ -24,8 +24,8 @@ the registration uses them. Clearing either makes startup fail without resolving another package from the client's project. Restore the same installation before repairing it; an unidentified registration remains untouched. See [installation and recovery](installation.md#package-runners-and-global-installation). -Setup does not copy your ambient environment into client -settings. To refresh an older registration, preview +Setup forwards supported, explicitly selected REA settings into client +configuration, but does not copy unrelated ambient environment values. To refresh an older registration, preview `rea setup --client --skill=false --dry-run`, then run the same command without `--dry-run` and approve the displayed changes. Setup preserves unrelated settings, backs up changed configuration and leaves an aligned registration @@ -49,6 +49,31 @@ launcher; see [verification scope](client-verification.md#launch-environment-ver the registration's file does not carry as a [non-blocking unknown](installation.md#check-readiness-for-your-task). +## Setting launch variables in PowerShell + +Shell examples that prefix a command with `REA_MCP_INPUT_SCHEMA_PROFILE=compact` +use POSIX shell syntax. In PowerShell, set the process environment before +launching REA: + +```powershell +$env:REA_MCP_INPUT_SCHEMA_PROFILE = "compact" +rea mcp doctor --json +``` + +The assignment applies to the current PowerShell process and commands it launches. +Restore the previous value when finished, or use +`Remove-Item Env:REA_MCP_INPUT_SCHEMA_PROFILE` if it was previously unset. It does +not update an already running MCP server or desktop client. + +To select this profile for an agent, set `REA_MCP_INPUT_SCHEMA_PROFILE` in that +client's existing REA registration `env` object (`environment` for OpenCode) and +restart the client, preserving its other settings. A successful terminal doctor run does not verify the client's +launch environment. See [launch-environment verification](client-verification.md#launch-environment-verification) +and the client-specific instructions below. REA setup forwards supported selections, +including `REA_MCP_INPUT_SCHEMA_PROFILE`, from its environment into the registration; +it does not copy unrelated environment values. Restoring the shell's previous +value does not undo a selection already saved by setup. + ## Runtime settings and client deadlines `update_analysis_settings` changes curated defaults inside the running REA @@ -77,6 +102,79 @@ does not repair a client's naming behavior or infer permission drift from an unfamiliar prefix. The unavailable fork reported in [#5512](https://github.com/morluto/rea/issues/5512) has not been reverified. +## Hot/cold catalogs + +For a client that supports MCP catalog refresh, set +`REA_MCP_TOOL_CATALOG_PROFILE=hot-cold` in REA's server environment and restart +the connection. REA then advertises a small initial catalog and loads other +named tools through `discover_tools`. Setup preserves this explicit setting. +See the [discovery contract](mcp-contracts.md#hotcold-catalog-discovery) for +pagination, identity and family-scope semantics. + +Analysis tools, including `inspect_analysis_view`, load on demand. For example, +load `inspect_api_contract` and supply its `view` to answer an initial OpenAPI +or GraphQL question directly. When later questions need retained facts, load +`inspect_analysis_view` and use the complete `parent_evidence_id`; source files +need not be reopened. If both tools are needed from the start, request both +names in one `discover_tools` batch to avoid another catalog refresh. The same +pattern applies to selected Playwright trace and endpoint-comparison results. +This defers unused schema delivery and client validator preparation; it does +not reduce parsing work, alter Evidence or shorten requested facts. Clients +without catalog refresh should keep the default `full` profile. + +Codex CLI 0.165.0 requires its experimental `mcp_tool_list_changed` feature for +this workflow. Merge these entries into the existing `~/.codex/config.toml` +without replacing other feature or REA settings: + +```toml +[features] +mcp_tool_list_changed = true + +[mcp_servers.rea.env] +REA_MCP_TOOL_CATALOG_PROFILE = "hot-cold" +``` + +The server setting alone does not enable client refresh. If a successful +discovery call is followed by a missing tool, check the client's refresh feature +and tool allow/deny lists. Clients without refresh should use the default +`REA_MCP_TOOL_CATALOG_PROFILE=full`; reconnect after changing it. The +[FastMCP HTTP trial](#fastmcp-and-unsloth-sse-catalog-limits) below covers +explicit refreshes; other clients' automatic refresh remains unverified. + +## FastMCP and Unsloth SSE catalog limits + +If an HTTP client reports `Server-sent event exceeded the 1048576 byte limit` +during `tools/list`, merge these settings into the **REA server process's** +environment, then reconnect: + +```text +REA_MCP_INPUT_SCHEMA_PROFILE=compact +REA_MCP_OUTPUT_SCHEMA_PROFILE=compact +REA_MCP_TOOL_CATALOG_PROFILE=full +``` + +This advertises all tools without requiring catalog refresh. Compact output +schemas widen client-side validation; REA still validates complete results +against the canonical contracts. See [output schemas](mcp-contracts.md#output-schemas). + +On main `3e0599a94`, a real FastMCP 4.1.6 / Python MCP 2.3.0 HTTP client with +the default 1 MiB SSE event limit rejected the 4,108,975-byte full catalog. +The settings above delivered all 240 tools in 611,305 bytes and a +`binary_session` call succeeded. These are measured SSE response sizes for +that catalog, not a size guarantee for future catalogs or other profiles. + +For a client that refreshes tools, combine `hot-cold` with the compact schema +settings to reduce initial context further: the same trial delivered six +initial tools in 33,903 bytes, then explicitly refreshed through 34 loaded +tools. Hot/cold alone later exceeded the SSE limit during refresh. Follow +the [refresh requirements](#hotcold-catalogs). + +The Unsloth report [#9093](https://github.com/morluto/rea/issues/9093) fails at +catalog delivery; the Unsloth application itself was not exercised in this +trial. Schema compaction does not bound analysis results. Continue to use +summary/detail selection and tool-specific paging for larger results; client +receive limits and full Evidence access still apply. + ## Fiddler Everywhere alongside REA Connect Fiddler Everywhere's MCP server separately for HTTP traffic debugging, @@ -130,7 +228,7 @@ in the file the installed app reads: - Without Claude Desktop's MSIX package storage, setup uses `%APPDATA%\Claude\claude_desktop_config.json`. -- The MSIX build (package family `Claude_pzs8sxrjxfjjc`, installed under +- The legacy MSIX storage layout (package family `Claude_pzs8sxrjxfjjc`, installed under `C:\Program Files\WindowsApps\Claude__x64__pzs8sxrjxfjjc`) virtualizes AppData writes. Once it has saved its configuration, it reads the package-private copy @@ -139,6 +237,11 @@ in the file the installed app reads: private copy when it exists. Before the package has saved one, they use the `%APPDATA%` file it still reads; with neither file, setup creates the private copy. +- If `%LOCALAPPDATA%\Claude-Data\claude_pzs8sxrjxfjjc` exists, setup, doctor + and uninstall report an unresolved location. Migrated MSIX installations + have been observed reading `claude_desktop_config.json` in this directory. + Quit Claude Desktop and verify its active configuration before registering + manually. See [#8968](https://github.com/morluto/rea/issues/8968). - If `%LOCALAPPDATA%\Packages` holds a `Claude_` package from another publisher, an `AnthropicPBC.Claude_` package, or cannot be read, REA cannot establish which file Claude @@ -277,7 +380,7 @@ This headroom is a practical default, not a guarantee for every analysis duration. Setup preserves positive timeout values already in the `rea` entry. Setup also forwards explicitly selected input/output schema profiles, -`REA_MCP_MAX_RESPONSE_BYTES`, `REA_MCP_MAX_INPUT_ELEMENTS`, provider paths and supported Ghidra, JADX and Kuna +`REA_MCP_MAX_RESPONSE_BYTES`, `REA_MCP_MAX_INPUT_ELEMENTS`, the Windows ChangeTime settle settings `REA_WINDOWS_CHANGE_TIME_DRIFT_MS`, `REA_WINDOWS_CHANGE_TIME_SETTLE_POLL_MS`, and `REA_WINDOWS_CHANGE_TIME_SETTLE_MAX_POLLS`, provider paths and supported Ghidra, JADX and Kuna settings. It also forwards explicit executable and resource selections for ADB, apktool, packet capture, GDB/Rizin/objdump, Cutter, WABT, Wakaru, firmware and signal tools, browser inspection, and pwntools/pwndbg. This includes their @@ -606,6 +709,18 @@ The user-level MCP paths follow the editor's VS Code layout. Trae's official documentation describes the raw JSON editor without specifying those paths; loading this registration in Trae remains unverified. +## CodeBuddy + +For CodeBuddy, setup writes `mcpServers.rea` into the global +`~/.codebuddy/mcp.json` (a standard `mcpServers` JSON file at the user home +root on every platform). Entries carry `command`, `args`, and `env`; stdio is +the default transport. CodeBuddy discovers personal skills from +`~/.codebuddy/skills`, so setup installs its skill copy there. + +The global MCP path is a plain JSON file; setup preserves every other server +and top-level key when it upserts `rea`. Loading this registration, the +bundled skill, and real REA tool calls through CodeBuddy are verified. + ## WorkBuddy For [WorkBuddy](https://www.workbuddy.cn/docs/workbuddy/From-Beginner-to-Expert-Guide/Function-Description/MCP-Guide), setup writes `mcpServers.rea` into the @@ -724,6 +839,103 @@ Harness integration, with live DeepSeek API compatibility remaining unverified. See the [Harness MCP guide](https://github.com/deepseek-ai/deepseek-harness/blob/639ed015397290b3745d163aafe02ffee4aa3f84/packages/mcp/mcp-client/README.md) for its native configuration contract. + + +## llama.cpp + +Use REA from llama.cpp's local Web UI through `llama-server`'s native stdio +MCP support. REA supplies the analysis tools; llama.cpp runs your chosen model. +It is not a replacement for a native analysis provider such as Ghidra or Hopper. + +Install REA in a persistent location so the configuration below can use a stable +entrypoint. For a project-local installation, run this in your chosen directory: + +```bash +npm install --save-exact rea-agents +``` + +Or install it globally: + +```bash +npm install --global rea-agents +``` + +See [package runners and global installation](installation.md#package-runners-and-global-installation) +for runtime requirements and installation options. Running `npx rea-agents setup` +alone does not create either installation; do not use its temporary package-cache path. +Use a llama.cpp build whose `llama-server --help` lists +`--mcp-servers-config`; older builds without that option cannot use this route. +The configuration below uses an already installed REA package, so tool discovery +does not depend on downloading an npm package during server startup. + +Save a JSON file, for example `rea-mcp.json`: + +```json +{ + "mcpServers": { + "rea": { + "command": "/absolute/path/to/node", + "args": ["/absolute/path/to/rea-agents/scripts/rea.mjs", "mcp"], + "timeout_ms": 450000 + } + } +} +``` + +Replace both paths with your installation's actual paths. For a project-local npm +installation, run `npm root` in that project; the script is under +`$(npm root)/rea-agents/scripts/rea.mjs`. For a global installation, use +`$(npm root -g)/rea-agents/scripts/rea.mjs` instead. Write the resulting absolute +paths as JSON strings; llama-server does not expand shell expressions there. +On Windows, use forward slashes, such as `"C:/Program Files/nodejs/node.exe"`, +or double each backslash, as in `"C:\\Program Files\\nodejs\\node.exe"`. +Apply this to both `command` and the script path in `args`. Single backslashes +can make the JSON invalid or change the path through escape sequences. +`node -p "JSON.stringify(process.execPath)"` prints your current Node.js executable +as a quoted, escaped JSON string that can be used as the `command` value. +Use the version supported by REA. Preserve any other +servers if you are adding REA to an existing configuration. `timeout_ms` is +llama.cpp's per-tool-call timeout, including cold engine startup. The example +allows REA's default 330-second Ghidra startup budget plus 120 seconds for the +operation. Increase it if you configure a longer provider allowance or need more +operation time; see [provider deadlines](mcp-contracts.md#ghidra-first-query-deadlines-and-recovery). + +Start the server with your own tool-capable GGUF model: + +```bash +llama-server -m /absolute/path/to/model.gguf \ + --host 127.0.0.1 --port 8080 --parallel 1 --ctx-size 16384 \ + --mcp-servers-config /absolute/path/to/rea-mcp.json +``` + +Open `http://127.0.0.1:8080`. REA tools appear with a `rea_` prefix, such as +`rea_analyze_javascript_application`. Select only the tools needed for the +conversation in the Web UI's tool settings: discovery of REA's complete catalog +does not mean that its schemas and results fit the selected model's context. +Choose a context size your model and machine support. For a first check, enable +`rea_analyze_javascript_application` and ask it to inspect a small JavaScript +directory you own. This task needs no native analysis engine. Inspect the +returned Evidence and limitations rather than relying only on the model's summary. + +If the model prints a proposed call as ordinary text, REA has not been called. +Check the selected model and its chat template against llama.cpp's function-calling +guidance. A working MCP connection does not establish reliable tool selection by +every model. + +`--tools all` is not needed; it enables llama.cpp's separate built-in tools. +`--ui-mcp-proxy` is also not needed for this server-side stdio connection. +REA's setup wizard does not register llama.cpp automatically; pass the JSON +file to llama-server explicitly and restart it after changes. + +The MCP child runs with llama-server's permissions. Loopback binding does not +sandbox it or make every REA operation read-only. Enable only trusted commands, +keep the server local, and select targets and operations you are authorized to use. + +See llama.cpp's [MCP server configuration](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md#mcp-servers) +and [function-calling guidance](https://github.com/ggml-org/llama.cpp/blob/master/docs/function-calling.md). +The [verification record](client-verification.md#llamacpp) distinguishes connection +checks from model behavior and lists the tested versions. + ## Other client settings For OpenCode, setup writes the V1 `mcp.rea` entry, which OpenCode V1 and V2 diff --git a/docs/client-verification.md b/docs/client-verification.md index 86b387cfcc..76f0d45b09 100644 --- a/docs/client-verification.md +++ b/docs/client-verification.md @@ -43,6 +43,30 @@ does not establish broader Pi compatibility. See the pinned upstream [`declarations.ts`](https://github.com/earendil-works/pi/blob/b45e0ebcc5ae2fc42a6f2f23104d10e656e6bedb/packages/codemode/src/declarations.ts) and [renderer usage notes](https://github.com/earendil-works/pi/blob/b45e0ebcc5ae2fc42a6f2f23104d10e656e6bedb/packages/codemode/README.md#declarations-for-the-model). + + +## llama.cpp + +The [manual stdio connection](client-configuration.md#llamacpp) was checked on +macOS arm64 with Node.js 24.18.0, llama-server b11715 (`9d1878a43`), and the +published `rea-agents@6.5.0` package. `GET /tools` exposed 207 REA tools without +enabling llama.cpp's built-in tools. With the source checkout at `9e372c6`, all +207 forwarded names, descriptions and input schemas matched a direct MCP client. + +Calls through `POST /tools` returned session metadata and JavaScript analysis +Evidence for a fixture at a Unicode path. A missing target returned REA's input +error, and a subsequent session call still succeeded. The owned llama-server +process exited successfully after the checks. No native analysis engine or +Windows host was exercised. + +A local Qwen3-1.7B Q8_0 run selected the JavaScript analysis tool and received its +Evidence, but its follow-up answer reached the output-token limit. Other small +Qwen3 attempts returned proposed calls as ordinary JSON text rather than tool +calls. These runs do not establish dependable model-driven investigations. +A request with the full catalog and task history exceeded a 16,384-token context; +select tools for the task instead of assuming that discovery implies model fit. +The checks exercised HTTP and stdio, not browser interaction with the Web UI. + ## End-to-end, integration and golden evidence The optional `verify:qwen-client`, `verify:pi-client`, and `verify:hermes-client` diff --git a/docs/design/limits-and-recovery.md b/docs/design/limits-and-recovery.md index c6ea9dde98..ad3afb24d1 100644 --- a/docs/design/limits-and-recovery.md +++ b/docs/design/limits-and-recovery.md @@ -125,6 +125,19 @@ raise REA's reserved string limit or Node's string ceiling. Like the regex deadline, it is in `SELECTED_REGISTRATION_SETTINGS`, so setup forwards an explicit selection into the client registration. +Windows ChangeTime admission waits out a fresh timestamp before recording a +baseline. `REA_WINDOWS_CHANGE_TIME_DRIFT_MS` (default 2000 ms, floor 1 ms, +maximum 60000 ms), `REA_WINDOWS_CHANGE_TIME_SETTLE_POLL_MS` (default 25 ms, +floor 1 ms, maximum 1000 ms), and `REA_WINDOWS_CHANGE_TIME_SETTLE_MAX_POLLS` +(default 40, floor 1, maximum 1000) are launch settings. Unset keeps those +defaults. An invalid value is a configuration error: it names the setting, the +rejected value, source `setting`, the default, the floor, and the maximum, and +says to set a positive decimal integer in the environment that starts REA and +then restart REA. There is no runtime override. Setup forwards an explicit +selection. When the poll budget runs out, admission keeps the last observation +instead of failing the readable file. Raising a setting only changes how long +that wait can last. It does not change Windows ChangeTime itself. + A per-call budget can have a runtime default without having an environment setting. JavaScript `max_heap_mb` (CLI `--max-heap-mb`) is an integer from 128 to 16384 MiB, default 1024. It diff --git a/docs/desktop-application-capture.md b/docs/desktop-application-capture.md new file mode 100644 index 0000000000..f68d8da4e8 --- /dev/null +++ b/docs/desktop-application-capture.md @@ -0,0 +1,379 @@ +# Desktop application capture + +`rea capture-desktop-application` launches an installed Linux desktop +application, watches it while a person or a driver uses it, and records what +it is, which windows it opened and how the files under the home directory +changed. The result is the starting point for turning the application into an +agent environment: the inventory says how to install and launch it, the windows +say how to recognise it, and the checkpoints hold the state it read and wrote. + +The application runs with the current user's permissions, in the current +desktop session, against the user's real files. This is not a sandbox. The +capture folder keeps copies of real files from the home directory, so treat it +as private data and keep it local. + +## Host support + +Capture runs on Linux with an X11 display: `DISPLAY` must be set, and `xprop` +(from `x11-utils`) must be installed, or `REA_XPROP_COMMAND` must name it. +The window manager must support the EWMH client list (`_NET_CLIENT_LIST`, +listed in `_NET_SUPPORTED`), as xfwm4, Mutter, KWin and Openbox do; without it +the capture is refused, because no window could be seen. A supported list that +is unset, as xfwm4 leaves it while no window is open, means no windows. Applications that run under XWayland are visible; +native Wayland windows are not. Other hosts are refused before anything is +launched. + +## Capture an application + +Name the application by its desktop entry id, the entry's absolute path, or an +executable: + +```sh +rea capture-desktop-application --desktop-entry org.xfce.mousepad --output ./mousepad-capture +rea capture-desktop-application --executable gnucash --argument /home/ada/books.gnucash --output ./gnucash-capture +``` + +Desktop entries are looked up in `$XDG_DATA_HOME/applications` and then each +`$XDG_DATA_DIRS` directory; an entry installed as a symbolic link, as flatpak +and Nix do, is followed to its file. The entry's `Exec` line is expanded with no files; +`--argument` values are appended in its place, and the application runs in the +entry's `Path` or the home directory. When the line starts with +`env NAME=value`, the program is found on the `PATH` those assignments set, +with an empty or relative entry read from the folder the application runs +in, and their XDG folders name the state directories. Otherwise the program is +looked up on the host's `PATH` in the order the launch searches it, and one +found only through a relative or empty entry is refused, since the exported +computer would not have that entry. An `--executable` runs from +the current directory, as it would from a shell, and a relative path to it is +resolved there. + +Before writing anything the command refuses to start if a window of the +application is already open, or if the window list cannot be read. A launch +beside a running instance can hand off to it, and that instance's unsaved +state would be missing from the baseline. It then records the starting state, +launches the application in its own process group with a run token in its +environment, and reads commands from stdin, one per line. The application gets +REA's environment without the `AGENTENV_*` variables an agent environment sets, +such as its control token: an app the exported computer launches never has +them, and an app captured inside a computer must not hold a credential to it. + +| Line | Effect | +| ------------------------------------------- | ----------------------------------------------------------- | +| empty, `checkpoint`, or `checkpoint ` | Record the current state as a checkpoint | +| any other text except `stop` | Record a checkpoint named by that text | +| `stop`, or the end of stdin | Record a final checkpoint, close the application and finish | + +An unnamed checkpoint is named `checkpoint-` after its time in the +session. A name that repeats an earlier checkpoint, or is one of the capture's +own names (`baseline`, `stop`, `time-limit`, `exit`), is refused with a +progress line on stderr; no checkpoint is taken and the session continues. + +Name the checkpoints a later step reads by what they mark. The +[export](desktop-application-export.md) seeds the bundle from the baseline +unless `--seed-checkpoint` names another, and a +[task](desktop-application-tasks.md) is checked against the last checkpoint +unless `--end-checkpoint` names another. So one capture can build a seed and +then the change: launch the app, create and save the document, send +`checkpoint seed`, make the edit and save it, send `checkpoint end`, then +`stop`. + +The session also finishes when the launched process has exited and the +application's processes and windows are all gone in two consecutive samples, +or when `--max-duration-seconds` passes. Before deciding the processes are +gone, the capture looks for any process still carrying the run token, so a +worker whose launcher exited before it was sampled keeps the session open. A +sample whose window list could not be read is not taken as an empty one; the +limitations say how many there were. One queued line is handled between +samples, and reading stdin pauses while 16 lines wait. + +Progress lines on stderr are JSON, `{"rea_progress": {"phase": +"desktop_capture", "message": ...}}`. Those a driver waits on also carry an +`event`: `{"kind": "window_open"}` once the window opens; `{"kind": +"checkpoint", "name", "trigger", "at_ms", "entries", "changes", +"retained_files", "retained_bytes"}` once a checkpoint has been taken, the +baseline, `stop` and `exit` included, with the paths walked, the paths +changed since the checkpoint before and the files retained; and `{"kind": +"checkpoint_refused", "name", "reason"}` for a requested name it refused. A +driver that sends `checkpoint ` waits for the event naming it rather +than for a fixed time. + +When the session ends the capture sends SIGTERM to the launched process, its +process group and every process that carries the run token, waits up to five +seconds for them to exit, and then removes every process that still carries +the token, whether or not it stayed in the process group. If that cleanup +cannot be verified after a second attempt, the capture is still written, and +the command then fails, naming the run token and process group that may +remain. Interrupting the command (Ctrl-C) ends the session the same way +without a further checkpoint; the capture is still written with end reason +`cancelled`, and the command exits with status 130. Interrupted before the +application is launched, the command writes no capture and removes the +content it had retained. + +If the capture fails for any other reason before `capture.json` is written, +the application is closed the same way and the content already retained is +removed from the capture folder. The error names any process or file that +could not be removed. + +A person types these lines in the terminal. An agent can drive the +application with a tool such as `xdotool` and write the lines to a FIFO that +it passes as stdin. Because the end of stdin stops the session, the driver +must keep the FIFO open for writing between checkpoints, for example with +`exec 3>capture.fifo` in a shell, rather than open and close it for each +line. + +| Option | Default | Meaning | +| ------------------------ | ------- | ----------------------------------------------------------------------------------------- | +| `--state-path` | none | File or directory whose starting content is retained too (repeatable) | +| `--exclude` | none | Path left out of every checkpoint walk (repeatable); the output folder is always left out | +| `--max-duration-seconds` | 3600 | Stop the session after this long | +| `--max-entries` | 250,000 | Paths recorded per checkpoint walk, its roots included | +| `--max-file-bytes` | 64 MiB | Largest file whose content is retained | +| `--max-content-bytes` | 1 GiB | Total retained content | +| `--package-file` | none | The `.deb` file the application was installed from; its facts are recorded, not the file | + +`--state-path`, `--exclude` and `--output` refuse an empty value, which +would otherwise name the current directory. + +## The capture folder + +The output folder must be new, or empty and owned by the current user; the +command makes it private (mode 0700). It holds: + +- `capture.json`: the Evidence record, also printed on stdout. +- `content/`: the bytes of each retained file, stored once per + distinct content. + +The Evidence operation is `capture_desktop_application` and its predicate type +is `rea.desktop-application-capture`. Its `host` names the platform, +architecture, display and home directory, and the host's name (`hostname`) +and the user's (`user`), which captured state can hold, as in a mail +template's `ada@e81232be457d`; the export removes them from the bundle. A +capture written before they were recorded has neither, and one is left out +when the system cannot say, as for a user with no passwd entry. The rest of +the result has four parts. + +**Application.** The desktop entry (id, path, name, `Exec`, icon, +`StartupWMClass`, categories, MIME types), the launch command and working +directory, the executable as found on `PATH` and after resolving links, and the +package that installed it. A Debian package records its name, version, +architecture and the apt sources that list the installed version, with the +origin (`o=`) apt reports for each source in `source_origins`, such as `Ubuntu` +for the Ubuntu archive and its mirrors or `LP-PPA-` for a PPA; an empty +source list means the version came from a local `.deb` or the package lists +are missing, which `package_lists_present` tells apart. apt's lists folder, +and the trusted keys below, are where `apt-config` resolves +`Dir::State::Lists`, `Dir::Etc::TrustedParts` and `Dir::Etc::Trusted` under +`Dir`, so an `APT_CONFIG` file that moves them is followed: on a computer +without lists, an ordinary user can run `apt-get update` with an `APT_CONFIG` +naming writable `Dir::State::Lists` and `Dir::Cache` folders and capture with +the same `APT_CONFIG`. Without `apt-config`, or when it fails, they are +`/var/lib/apt/lists`, `/etc/apt/trusted.gpg.d` and `/etc/apt/trusted.gpg`. + +The Debian package is the one `dpkg-query -S` names as the owner of the +desktop entry the capture launched, or else of the executable, since the +entry's package is what installs the application: LibreOffice Writer's entry +belongs to `libreoffice-writer`, while the `/usr/bin/libreoffice` it runs +belongs to `libreoffice-common`, which does not bring Writer with it. When the +two differ, `executable_package` names the executable's package, which the +export installs beside the entry's, since the entry's package need not depend +on it (`vim-common` owns Vim's entry but does not depend on `vim`). +`executable_package_reason` says why the computer could not install it by +name, by the rule for library packages below: neither the Ubuntu 24.04 archive +nor a vendor repository the export declares offers it, or `apt-cache policy` +did not say. When a package owns the entry but none owns its program, such as +a `~/.local/bin/` found first on `PATH`, the entry's package is recorded +without `executable_package`, and `executable_package_reason` says no package +owns the program, so the export refuses rather than install a package that +does not bring it. A program link that dpkg does not own counts as owned when +dpkg owns the file it resolves to, as an alternatives link does. An +`--executable` launch, or an entry no package owns, such as +one in `~/.local/share/applications`, records the executable's package. When +`dpkg-query` does not finish or exits with an error for the executable or the +entry, the package is `unknown` and the limitations say which path, since a +failed lookup is not proof that no package owns it. The sources, vendor +repositories and `--package-file` match below are the recorded package's; the +library walk still reads the executable. + +Each source whose origin is not `Ubuntu` is also recorded in +`apt_repositories`. `apt-get indextargets` names the sources file entry apt +read it from and its release, so flat repositories match too, whether their +suite is `./` (Typora's `deb https://downloads.typora.io/linux ./`) or `/` +(pkgs.k8s.io's). The record holds that entry (`sources_file`, `uri`, `suite`, +`components`, none for a flat repository, and `architectures`, the entry's +`arch=` or `Architectures:` restriction, empty when it has none), `release`, +the release's Suite (apt's `a=`) or else its Codename, and the key that signs +it. A user name or password in an entry's URI (`https://user:password@host/…`) +is never recorded: REA removes it from every URI it reads, in the sources file +and in `apt-get indextargets` and `apt-cache policy` output alike, so entries +match whether or not apt prints it, and sets `uri_credentials_removed` on a repository whose URI held +one (a capture written before this was recorded omits the field). The key is +found by verification, not by name: the candidates are the +entry's `signed-by` keyrings, or else every `.gpg` and `.asc` key in apt's trusted +parts folder and its trusted keyring, and each is checked with +`gpgv` against the release's `InRelease` (or `Release` with `Release.gpg`) +beside its index in apt's lists folder. A deb822 entry whose `Signed-By` field +holds the armored key itself, as `add-apt-repository ppa:/` +writes it on Ubuntu 24.04, has that key as its only candidate, as apt trusts +no other for it: REA reads the field's continuation lines as deb822 writes +them, a `.` line standing for an empty one, from the key block's `BEGIN` line +to its last `END` line, and records the key's path as the entry's sources +file. A key counts when gpgv reports a valid signature by it and no bad, +expired, revoked or unverifiable signature by it, whatever gpgv's exit status: +a release also signed by a key the host lacks makes gpgv fail. Each candidate +is copied, an armored key decoded to a binary keyring, into a private +temporary folder that is removed afterwards; a folder REA cannot remove +fails the capture with the folder as its cleanup resource. The key that +verifies is recorded in `signing_key` with its path, +SHA-256, format (`binary` or `armored`) and primary fingerprint, and its bytes +are retained under `content/`. A key file holding one key is retained as it +is; from one holding several, such as a legacy `/etc/apt/trusted.gpg`, only +the transferable key with the verifying fingerprint is retained, in binary, so +the bundle trusts no other key. When no candidate verifies, `gpgv` is +missing, apt holds no signed release, no sources entry matches, the key the +entry's `Signed-By` holds does not verify the release, or the key is larger +than `--max-file-bytes` or the content budget, `signing_key` is null and +`signing_key_reason` says why. Ubuntu Pro's ESM repositories (origins such as +`UbuntuESMApps`, host `esm.ubuntu.com`) are not checked: apt reaches them only +with the host's Ubuntu Pro credentials, which a computer does not have. + +Unless the installed version comes from the Ubuntu 24.04 archive, so for a +vendor repository's package and for a local install that no apt source lists, +`library_packages` lists the packages that own the shared libraries the +executable loads at start; the walk runs beside the repository check, and not +at all for an archive package, whose export does not use it. REA reads +DT_NEEDED from the resolved executable and from each library it finds, and +looks each soname up as ld.so does: in the RPATH of the needing image and of +each image up its chain of loaders to the executable when the needing image +has no RUNPATH, then in its RUNPATH (with `$ORIGIN` expanded and paths +normalized), and then in the `ldconfig -p` cache entries for the executable's +machine. It asks `dpkg-query -S` for each library's package. Libraries in the +executable's own folder belong to the application: they are walked but not +listed, and neither is the application's own package. One batched +`apt-cache policy` then keeps only the packages the Ubuntu 24.04 archive or a +recorded vendor repository with a verified key offers, in any component its +sources entry lists, since the export declares them all; the rest, such as +packages from a PPA or a local `.deb`, are listed in +`uninstallable_library_packages`. The same query checks `executable_package`. +Each recorded repository's `supplies` lists the kept packages, and the +executable's package, that the archive does not offer and its package files +list, so the export can leave out those only a repository it does not declare +supplies, such as one whose key the content budget later dropped. Sonames +found nowhere are listed in +`missing_libraries`, and libraries no package owns in `unpackaged_libraries`. +`library_packages_reason` says why the lists are incomplete: the executable is +not an ELF image (a launcher script, for example), `ldconfig`, `dpkg-query` or +`apt-cache` is missing, a library could not be read or names more than 512 +libraries, or the walk stopped after 512 libraries. + +A sources file, key or library whose handle REA cannot close fails the capture +with the file as its cleanup resource. When the repository check and the +library walk both fail, the capture reports both failures and every folder or +file either could not release. + +An application installed from a `.deb` file you downloaded, such as VariCAD +from varicad.com, has no apt source a computer could install it from. Name +that file with `--package-file` so the export can have the computer download +it instead. REA reads the file's control fields with +`dpkg-deb --field FILE Package Version Architecture Depends Pre-Depends`, and +when its Package, Version and Architecture are the installed package's, it +reads the file once more for its SHA-256 and size and records `package_file`: +the file's `name` without its folder, `sha256`, `size`, `package`, `version`, +`architecture`, and the `depends` and `pre_depends` entries as the file +declares them. Otherwise `package_file_reason` says why, for example that +`dpkg-deb` is not installed, that the file holds another version than the +installed one, or that the file changed between the two reads (its device, +inode, size and times are compared), since its fields and its hash could then +describe different files. The capture never copies the file: it is the vendor's, and only +these facts are kept. + +Snaps, flatpaks and +AppImages are recognised from their paths, and flatpaks also from a +`flatpak run` launch, whose application id is the first operand after +flatpak's options and their values. Electron applications are marked by +the fuse sentinel in their binary or a `resources/app.asar` beside it. State +directories are the XDG config, data, state and cache folders and the +dot-folder named after the package, the executable, the desktop entry's full +id and its last part, and the entry's name, with whether each existed at the +start. A folder in those places whose name is one of these in another letter +case joins them, such as GIMP's `~/.config/GIMP` for the `gimp` package; only +the whole name is compared, so `~/.config/GIMPshop` does not. Dotfiles in the +home named after the application (`.rc`, or +`.` followed by `-`, `.` or `_`, for a name of three or more characters) +that exist at the start join them, since an application may only read such a +file during the capture. A `flatpak run` launch is named after its application +id rather than the `flatpak` launcher, and adds `~/.var/app/`. + +**Session.** When it started, how long it ran and why it ended; the launched +process's exit status; every process seen in the application's tree; every +client window with its class pair, titles, owner pid and type, marked when it +belongs to the application; when the first application window appeared; the +most common class pair of the application's windows, for matching its window +later; files the application's processes held open under the walked roots; the first +8 KiB of the launched process's stderr; and whether cleanup was verified. Each +process records whether the capture started it: the launched process, a +process carrying the capture's run token, or a descendant of one, even when +its environment cannot be read, as for a zombie or a sandboxed renderer. +Processes are told apart by their start time, so a pid reused by another +process is not counted as the application's. + +A window belongs to the application when its owner pid is one of the +application's live processes or its class matches the desktop entry's +`StartupWMClass`, the executable's name, the entry's full id or its last part, +or a flatpak's application id. A flatpak's windows report pids from its +sandbox, so they are matched by class only. +Ownership is checked again at every sample, because a window can appear before +the process that owns it is found. A matching window's pid joins the +application's processes, so a launch handed to a process the capture did not +start, such as a D-Bus activated service, is still followed. That process does +not carry the run token, so the capture never signals it and leaves it +running. + +**State.** The walked roots (the home directory plus any content root +outside it), the content roots (state directories, `--state-path` values and +absolute `--argument` values, except `/` or a directory that contains the home +directory, which are named in the limitations instead; a selected path that is +not a directory brings the `-wal` and `-journal` paths beside it, so a database +selected on its own is captured with its logs), the state directories +as walked (their physical spelling, which differs from the inventory's when one +is a symlink), the paths every walk left out (the output folder and +`--exclude` values), and the checkpoints in order: `baseline` before +launch, each requested checkpoint, `stop` or `time-limit`, and `exit` after the +application has gone. Each checkpoint records how many paths its walk saw, +whether it was truncated, how many paths it could not read or directories on +another device it did not enter (`unchecked_paths`), which paths were created, modified, deleted or +changed type since the previous checkpoint, and which files it retained. A +path counts as created or deleted only where the walk listed its nearest +existing directory completely; differences under a directory that could not be +listed, or that a truncated walk did not reach, are counted as unconfirmed +instead. A walk does not enter a directory on another filesystem than its +root, such as an NFS, sshfs or other FUSE mount under the home directory; the +limitations list each such mount point. The +baseline retains every file under the content roots; later checkpoints retain +every changed file. A change records the file's type, size, mode, +modification time and, when known, the SHA-256 of its content before and +after, plus whether it lies in a state directory and whether the application +held it open. A file is retained only if it is still the version the walk +recorded and does not change while it is read. A file larger than the +remaining content budget is not read, unless stored content of the same size +could make it a duplicate. Files that could not be +retained are listed with the reason: too large, over the content budget, +changed since the walk, changed during the read, or unreadable. +Directories the application held files open in, but whose starting content was +not retained, are listed so a second capture can pass them with +`--state-path`. + +**Limitations.** What the capture could not establish, such as a truncated +walk or a missing window. + +## What the capture does not establish + +Changes are found by comparing walks, using each path's size, modification +time, inode and mode, so a change undone between two checkpoints is not seen +and a rewrite that keeps all four is missed. A change is not attributed to the +process that made it: other desktop services write under the home directory +too, such as the dconf database, the window manager's settings and GTK's +recently used list. The markers for state directories and open files are +evidence for attribution, not proof. + +Clicks and keystrokes are not recorded. Window titles are sampled every half +second, so short-lived titles can be missed. diff --git a/docs/desktop-application-export.md b/docs/desktop-application-export.md new file mode 100644 index 0000000000..f9a21e5b84 --- /dev/null +++ b/docs/desktop-application-export.md @@ -0,0 +1,342 @@ +# Desktop application export + +`rea export-desktop-application ` turns a folder +written by [desktop application capture](desktop-application-capture.md) into an +agentenv bundle holding one computer app environment: the real application, +installed from the distribution archive, its vendor's apt repository or the +`.deb` file it was installed from on the `ubuntu-24.04` computer base, launched +at boot, seeded with the user's state and serving that state through agentenv's +data operations. + +```sh +rea export-desktop-application ./mousepad-capture ./mousepad-bundle +agent-env run ./mousepad-bundle --dry-run +``` + +The export reads the capture and its [state view](desktop-application-state.md); +it does not run the application. `--name` sets the environment name; otherwise +it comes from the desktop entry id or the program, with an `app-` prefix when +that name would not be a portable bundle name (for example all digits). The +output directory must be new or empty. `--task` adds a task an agent wrote for +the capture, once REA has validated it against the capture; see +[desktop application tasks](desktop-application-tasks.md). `--package-url` +says where the computer downloads the package of an application installed from +a `.deb` file; see [Applications from a .deb file](#applications-from-a-deb-file). +`--seed-checkpoint` names the checkpoint whose state seeds the bundle, the +baseline by default, and `--end-checkpoint` the one a task must pass on, the +capture's last by default; see [The seed checkpoint](#the-seed-checkpoint). + +The command returns Evidence with operation `export_desktop_application` and +predicate type `rea.desktop-application-export`, listing the environment name, +the seed and end `checkpoints`, owned paths, seed files, the seeds it +rewrote to remove the capture host's name and user (`scrubbed_files`), the +bundle files that still hold them (`unscrubbed_files`), and written files, +the last three by their path in the bundle. A failed export, cancellation +included, removes the bundle it had not yet published, so no partial copy of the +user's data remains. If the bundle was published but its output handles could +not be closed, the command reports a cleanup error that carries the export +Evidence; the bundle itself is complete. + +## The bundle + +| Path | Contents | +| -------------------------------- | -------------------------------------------------------------------------------------------------- | +| `envs//env.toml` | `type = "computer_app"`, the module and class, `computer = "ubuntu-24.04"`, `driver = "reference"` | +| `envs//.py` | The `ComputerAppEnvironment` class | +| `envs//SKILL.md` | What the app is, its window class and where its files live | +| `envs//seed/` | The application's files at the seed checkpoint, by home-relative path | +| `envs//apt-keys/` | The signing key of each vendor apt repository the app installs from | +| `envs//install-package.sh` | With `--package-url`: the run script that installs the downloaded package with dpkg | +| `tasks/smoke.json` | A task that deploys the app | +| `tasks/.json` | With `--task`: the agent's task, its verifier in `artifacts/` and the eval in `evals/` | +| `README.md` | What was exported, from which capture, and what to keep in mind | + +The class file is `_app.py` with hyphens as underscores, for example +`mousepad_app.py`: agentenv imports it under its stem, so a plain `.py` +could shadow a standard library module such as `calendar`. A class name that +would collide with a name the module imports or defines, such as `Path`, takes +an `App` suffix. + +The class declares: + +- `Packages(apt=[...])` with the package the capture found. A vendor package + is named `/`, its repository's release (the Suite apt + reports as `a=`, or else its Codename), so apt takes it from that repository + even when the Ubuntu archive offers a higher version or epoch, as noble's + transitional `firefox` does over packages.mozilla.org's; it is named alone + when the release names neither, or when the capture was written before + releases were recorded. Its library packages follow, so a package that + leaves out dependencies on the libraries it loads, as Typora's Electron + package does, still starts; library packages the capture found neither the + Ubuntu 24.04 archive nor a declared repository offers are left out, with a + limitation, and so are those only repositories the export leaves out supply + (a repository's `supplies`), such as one whose key the capture could not + retain. An archive package's declared dependencies are trusted as they + are. When the desktop entry's package is not the one that owns the program + it runs, as `vim-common`'s entry runs `vim`'s program, the program's package + (`executable_package`) follows the app's by name, since the entry's package + need not depend on it; a capture that found no archive or declared + repository offering it records why, and the export refuses it with that + reason, as it does when only repositories it leaves out supply it, or when + no package owns the program at all (`executable_package_reason` without + `executable_package`). The package must come from the Ubuntu 24.04 archive + or a mirror of it, + or from a vendor apt repository whose signing key the capture verified and + retained. A source counts as the archive when apt reports its origin as + `Ubuntu` for a noble suite; a capture made before origins were recorded + counts only Canonical's own archive hosts. +- For a vendor repository, `sources=[AptSource(apt=..., suite=..., + components=[...], key="apt-keys/.gpg")]`, with the retained key copied + to that path (`.asc` when it is armored, which agentenv tells by its + content). A PPA added with `add-apt-repository`, whose sources entry holds its + key in `Signed-By`, exports that key as `apt-keys/.asc`. apt takes one + key per URI and suite, so sources that share them, such as one repository + listed for two architectures, are declared once, with their components + merged; one with a different key is left out. A repository + whose key the capture did not verify, an Ubuntu Pro (ESM) repository, one + whose sources entry's URI held a user name or password (which the capture + did not record and the export never writes; the reason names neither them + nor the URI), one not reached over http or https, or one whose `arch=` + restriction leaves out the capture host's architecture is refused with its + reason (a capture + written before restrictions were recorded counts as unrestricted); when + another usable repository lists the same version it is left out instead, + with a limitation. The export also refuses snaps, flatpaks, AppImages, and a + package no recorded repository lists unless it was installed from a `.deb` + file the capture recorded and `--package-url` is given, since the computer + could not install them by name, and a capture written before credentials + were removed whose repository URI still holds them, since it counted that + repository's libraries as installable; capture it again. +- `@computer_app(supports=...)` with both `amd64` and `arm64` for an archive + package, which the archive serves for both, and only the capture host's + architecture for a vendor package, since its repository may serve no other. + A vendor package captured on any other architecture is refused. A package + installed from its `.deb` file supports only that file's architecture. +- A launch at boot with the captured command, ready when a window of the + captured class appears and one of them is listed in the taskbar and is not + typed as a splash: a splash screen or a dialog of the class skips the + taskbar or says it is a splash, and an app whose splash shows first would + otherwise count as ready before its own window opens. Arguments that are the captured home or under it, and + `