Prog-Prober/docs/architecture.md

Prog-Prober — Architecture Specification

Status: accepted. Derived from the design grilling captured in docs/adr/. This document is the single source of truth for the system. GOAL.md is the original brief and is retained for context only.

1. Purpose

Prog-Prober is an open-source, Linux-hosted, local-first desktop workbench for analyzing untrusted binaries — primarily Windows PE (x64), secondarily ELF (x64). It unifies four capabilities under one agent-drivable interface:

1. Static execution graphing — call graphs (CG) and control-flow graphs (CFG) from entrypoints to high-risk API sinks.
2. Isolated dynamic execution — detached detonation inside disposable Windows guests on a Linux host.
3. Network sinkholing — an inline, default-deny interception layer that logs and mocks outbound traffic.
4. Defense probing — instrumentation and detection of anti-analysis behavior, plus structured crash telemetry.

It is agent-first and human-steered: the primary consumer of the tool API is an LLM agent (via MCP), with a human analyst steering through the UI.

2. Goals and non-goals

Goals

• Reproducible analysis: every derived artifact is cache-keyed by tool and configuration, so results are explainable and invalidate cleanly.
• Safety by construction: untrusted code never executes on the host.
• Interactivity at scale: 10k functions interactive, 50k+ via clustering and level-of-detail; no canvas stutter.
• One tool API exposed three ways: renderer UI, CLI, and MCP.

Non-goals (for now)

• Hosted/multi-tenant operation. The workbench is local-first and desktop-shaped; a shared signature/RAG corpus may be read from later.
• Mobile or Windows-host support.
• Automated malware attribution or family classification beyond the RAG corpus.
• Kernel-level stealth (hypervisor memory introspection) in v1.

3. Operating assumptions and safety contract

Linux host, /dev/kvm available. Samples are authorized and offline. The following invariants are hard requirements; every component must satisfy them and any change that weakens them requires an ADR.

#	Invariant
S1	Samples are read-only and content-addressed (samples/<sha256>).
S2	Nothing untrusted ever executes on the host.
S3	Detonation happens only inside a disposable sandbox.
S4	Egress is default-deny; only the sinkhole proxy is reachable.
S5	The sandbox shares no host network namespace.
S6	Teardown is verified, and a teardown attestation is recorded per run.
S7	Artifacts follow an explicit retention policy (declared per case).

4. System context

┌─────────────────────────────────────────────────────────────┐
│ Electron (main + renderer)                                   │
│   React UI ── @cosmos.gl/graph (GPU force layout)            │
└───────────────▲──────────────────────────────────────────────┘
                │ localhost WebSocket + JSON (types generated by ts-rs)
┌───────────────┴──────────────────────────────────────────────┐
│ progprober-core  (Rust sidecar daemon)                        │
│   ├─ Tool API + MCP adapter (rmcp)                            │
│   ├─ Job queue: cancel · progress · per-case resource limits  │
│   ├─ Case store: SQLite + LadybugDB · samples/<sha256>        │
│   └─ Runner supervisor (Podman)                               │
└───────▲───────────────────────▲───────────────────────────────┘
        │ stdio JSON-RPC        │ Podman / OCI
┌───────┴────────────┐   ┌──────┴──────────────────────────────┐
│ Python workers     │   │ progprober-runtime (OCI image)       │
│ PyGhidra · capa    │   │   Ghidra+JDK · emulators · mitmproxy │
│ Speakeasy · Qiling │   │   runs as disposable containers      │
└────────────────────┘   └──────▲───────────────────────────────┘
                                 │ isolated bridge, default-deny
                         ┌───────┴───────────┐
                         │ Sinkhole sidecar   │
                         │ mitmproxy + INetSim│
                         └────────────────────┘

5. Process topology and protocols

• Electron hosts the UI. The main process spawns progprober-core as a long-lived sidecar; it never loads Rust as a native addon (ADR-0002).
• Renderer ↔ core: localhost WebSocket carrying JSON. Rust structs are the source of truth; TypeScript types are generated (ts-rs). No hand-written DTOs.
• Core ↔ Python workers: stdio JSON-RPC, job-scoped. A worker crash cannot take down the core; workers are disposable.
• Core → containers: Podman (rootless where possible) drives the OCI runtime. The core, not the worker, owns container lifecycle and teardown.
• MCP: the rmcp adapter lives inside the core and speaks the same Tool API (ADR-0009). It is one more face on one API, not a second backend.

6. Tool API

The internal API is the contract. Both the UI and MCP call it.

Tool	Purpose	Phase
inspect_call_graph	Return the function-level call graph, with filters (depth, sink class).	1
extract_strings	Return/floss-decoded strings with provenance and offsets.	1
trace_function_refs	Callers/callees/xrefs for one function; drill into its CFG.	1 (drill-down 1.5)
trigger_sandbox_run	Queue a detonation run; stream events.	2
get_crash_telemetry	Fetch crash dumps, registers, stack, exception records.	3

Every tool call is a job: cancellable, progress-streamed, and cache-aware. Long analyses (Ghidra) must be resumable from cache.

7. Static analysis pipeline

1. Ingest: resolve workspace folder, detect entrypoints and bundled libraries/sidecars (LIEF for fast PE/ELF parsing).
2. Analyze: Ghidra headless via PyGhidra extracts functions, call relationships, CFG, decompilation, xrefs.
3. Enrich: capa attaches capability matches (MITRE ATT&CK/MBC); FLOSS recovers obfuscated strings; LIEF contributes headers/imports.
4. Graph: build the canonical graph in LadybugDB; project to flat typed arrays for @cosmos.gl/graph; hold the working set in graphology.
5. Serve: Tool API queries with viewport-scoped deltas.

Engine selection and rationale: ADR-0003. Rizin/angr are future optional backends, not v1.

8. Dynamic sandbox and network sinkhole

• Runner interface: a pluggable abstraction over execution backends. v1 implements a QEMU/KVM microVM provisioned dockur/windows-style (ADR-0004). DRAKVUF (Xen VMI, agentless, stealthy) is a later backend behind the same interface; Wine/Proton is explicitly rejected as trivially fingerprinted.
• Sinkhole: mitmproxy in transparent mode provides inline TLS/HTTP interception and TLS key logging for PCAP decryption; a local DNS sinkhole redirects name resolution; INetSim/FakeNet-NG listeners simulate protocols. All of it runs on an isolated bridge with a default-deny gateway (ADR-0005, safety S4/S5).
• Event streaming: syscall/API/file/registry/network events stream to the UI and MCP as they occur.

9. Emulation, instrumentation, and probing

• Emulation first: Speakeasy (MIT, Windows-focused) runs in the runtime image with no guest agent to detect — ideal for triage (ADR-0008). Qiling (GPL-2.0-or-later) is an optional backend, compatible with our AGPL-3.0 license and kept process-separated.
• Instrumentation second: Frida hooks for in-guest tracing where emulation is insufficient.
• Defense probing: detect timing checks (RDTSC), PEB inspection, hardware breakpoint scanning, API-hook detection, and hypervisor/VM footprint checks. VMAware, al-khaser, and pafish are the hardening test corpus — they tell us what malware will check, so we can measure our own detectability.
• Fault triage: structured crash dumps, unhandled exceptions, and register states.

10. Data model

Case layout

<case>/
  case.db                 # SQLite: metadata, sample index, cache keys
  graph.ladybug           # LadybugDB: nodes, edges, vectors, FTS
  samples/<sha256>        # immutable originals
  artifacts/              # derived output, cache-keyed
  run-attestations/       # teardown attestations (S6)

Cache key

Every derived artifact is keyed by (sample_hash, tool, tool_version, config_hash). A tool bump or config change cleanly invalidates.

Graph schema

Canonical JSON Schema: schemas/graph.schema.json.

• Node kinds: binary, function, basicblock, instruction, import, api_sink, string.
• Edge kinds: call, cfg, ref, import, data_flow.
• Every node/edge carries id, kind, binary_id, provenance{tool,version}, and optional confidence.

The canonical graph is normalized in LadybugDB (Cypher). The renderer receives a flattened, typed-array projection at the edge; graphology holds the in-memory working set (ADR-0006, ADR-0007).

Storage split

Store	Owns
SQLite	case metadata, sample index, cache keys, job history
LadybugDB	nodes, edges, vector index, full-text index
graphology (memory)	renderer working set
DuckDB	omitted until an analytics query demands it

11. Runtime packaging

One versioned OCI image, progprober-runtime, bundles Ghidra + JDK + Python + emulators + sinkhole tooling. The Rust core drives Podman to run it. This makes analysis reproducible, keeps the host clean, and makes version-based cache invalidation meaningful (ADR-0011).

12. Tooling audit

Layer	Candidate	License	Notes
Static engine	Ghidra headless (PyGhidra)	Apache-2.0	Primary; best decompiler; capa integrates via PyGhidra
Fast parsing	LIEF	Apache-2.0	PE/ELF/Mach-O triage
Capabilities	capa	Apache-2.0	Static + dynamic (CAPE/DRAKVUF reports)
Strings	FLOSS	Apache-2.0	Obfuscated string recovery
Prior art	pyghidra-mcp, ghidra-headless-mcp, Arkana, LCRE	mixed	Patterns to learn from, not dependencies
Win-on-Linux	dockur/windows (QEMU/KVM)	MIT	v1 Runner; DRAKVUF later
Sinkhole	mitmproxy, INetSim, FakeNet-NG	MIT / GPL / Apache	Transparent inline + service simulation
Emulation	Speakeasy	MIT	Primary
Emulation (optional)	Qiling	GPL-2.0-or-later	AGPL-compatible; process-separated
Graph render	@cosmos.gl/graph	MIT	GPU force layout; Cosmograph branding is CC-BY-NC — do not use
Graph store	LadybugDB (Kuzu successor)	MIT	Kuzu archived Oct 2025
Relational	SQLite (+ sqlite-vec if needed)	Public domain	Case metadata
MCP	rmcp (Rust)	MIT/Apache	In-core adapter
Desktop	Electron + React	MIT	Rejected Tauri by choice

Full prior-art matrix with activity dates is in the ADRs and the issue tracker.

13. Phasing

Phase 1 — Static workspace and graph MVP (+ Tool API and thin MCP adapter). Exit criteria: open a folder → select a PE → Ghidra analyzes inside the OCI runtime → capa capabilities attach → functions-only call graph renders in @cosmos.gl/graph → the same path works via MCP inspect_call_graph → the case persists and reopens without re-analysis. Per-function CFG drill-down is Phase 1.5 if it needs to be cut.

Phase 2 — Dynamic sandbox and network sinkhole. Exit: detonate a sample in a disposable guest, stream events, capture decrypted traffic, and verify teardown.

Phase 3 — Emulation, probing, and crash telemetry. Exit: Speakeasy run produces traces; anti-analysis detectors fire and are attributable; crash telemetry is retrievable via MCP.

Phase 4 — RAG signature intelligence. Exit: vector-indexed packer/protector and anti-tamper corpus enriches findings, local-only by default.

14. Roadblocks and mitigations

Roadblock	Mitigation
Wine/KVM evasion artifacts	QEMU/KVM with a hardening track; DRAKVUF backend later; measure against VMAware/al-khaser/pafish
Graph memory at 50k+ instructions	GPU layout, clustering, level-of-detail, viewport-scoped deltas
Deterministic teardown / state restoration	Disposable VM per run; teardown attestation recorded (S6)
Slow Ghidra analyses	Job queue with cancellation; aggressive artifact caching
Tool-version drift invalidating results	Cache key includes tool version and config hash

15. Testing strategy

• Unit: Rust core (job queue, cache keys, protocol framing), Python workers (parsers, JSON-RPC handlers).
• Contract: the Tool API is tested once and shared by UI/CLI/MCP.
• Integration: a known-benign PE fixture flows end-to-end through the OCI runtime in CI (no live malware in CI).
• Safety: automated assertion that host network namespaces are never shared and that teardown attestations are written.
• Malware samples are never committed and never executed on the host or in CI.

16. Decision index

ADR	Decision
0001	Electron desktop shell
0002	Rust sidecar core (not a native addon)
0003	Ghidra headless as the static engine
0004	QEMU/KVM sandbox behind a Runner interface
0005	mitmproxy sinkhole on an isolated bridge
0006	@cosmos.gl/graph as the graph renderer
0007	SQLite + LadybugDB storage split
0008	Speakeasy primary, Qiling optional
0009	Agent-first Tool API with an in-core MCP adapter
0010	AGPL-3.0-or-later license
0011	Single versioned OCI runtime image