Compare commits

..
65 Commits
Author SHA1 Message Date
esengine 169d66d46a feat(ux): /tool slash — inspect untruncated tool output (0.4.4)
EventLog clips tool results at 400 chars for display; /tool [N]
now lets users read the full text. `/tool` lists the 10 most
recent with previews and char counts; `/tool N` dumps the
Nth-most-recent in full. Wired via new SlashContext.toolHistory
callback — pure handleSlash stays stateless.

340 tests (+8 /tool cases).
2026-04-21 21:41:30 -07:00
esengine 2836b5d406 fix(loop): Esc abort no longer blocks for 30-90s + bump to 0.4.3
User feedback: after pressing Esc, the warning fired instantly but
then the TUI blocked for the duration of a full reasoner call
(30-90s) to produce a "summary" the user hadn't asked for. Esc in a
terminal means stop. Stopping and then forcing the expensive
reasoner to wrap up is the opposite of what the user signaled.

Abort path now:
  - Emits a short warning "aborted at iter N/M — stopped without
    producing a summary (press ↑ + Enter or /retry to resume)"
  - Appends a synthetic assistant message "[aborted by user (Esc)
    — no summary produced. Ask again or /retry when ready; prior
    tool output is still in the log.]"
  - Yields assistant_final (tagged forcedSummary so edit applier
    ignores) + done
  - No API call. Takes milliseconds.

Budget and context-guard paths still call forceSummaryAfterIterLimit
because in those cases the loop stopped, not the user — leaving
them staring at nothing would be worse than one extra call. Only
user-initiated abort gets the fast path.

Test rewritten to confirm no extra API call is made (previously
asserted a "partial findings" summary from the never-needed
follow-up). Suite: 332 still passing.

Also bumps to 0.4.3 because v0.4.2 tag is already on origin at
cb2f2a5 with partial content. This release consolidates the full
0.4.2 batch + the Esc fix under one tag.
2026-04-21 21:31:59 -07:00
esengine d0bd6852c1 feat(ux): /retry · /status rich · /think · ↑↓ history · y/n · red errors · tool-row elapsed · reasoning tail · DSML strip (0.4.2)
Nine UX improvements in one batch, all surfaced from real-session
feedback while testing reasonix code against a live filesystem
project.

Added slash commands:
  - /retry — truncate log back before the last user message, resend
    it, get a fresh sample from the model. SlashResult gains a
    resubmit field the TUI honors after displaying info.
  - /status — now a labeled situation-report: model, flags
    (harvest/branch/stream), ctx ratio (42k/131k (32%)) from last
    turn's promptTokens, MCP server + tool counts, session name +
    log length + resumed count, pending edits count. One command
    shows everything.
  - /think (alias /reasoning) — dumps full R1 reasoning from
    loop.scratch.reasoning. For when the 260-char tail preview
    isn't enough. Graceful "no reasoning cached" when using
    deepseek-chat.

Added interactions:
  - ↑/↓ shell-style prompt history. App.tsx owns a ref array, cursor
    advances on arrow keys while idle. Process-scoped, no cross-run
    persistence.
  - y/n fast-path when edits are pending. Typing 'y'+Enter = /apply,
    'n'+Enter = /discard. Scoped: only when pendingEdits.length > 0,
    so plain "y" as a normal message still works when nothing's
    waiting. Preview line advertises the shortcut.

Changed:
  - Tool-running row surfaces elapsed seconds + per-tool argument
    summary. Before: ⠋ tool<filesystem_edit_file> running… {raw
    JSON}. Now: ⠋ tool<filesystem_edit_file> running… 3s / path:
    F:\... (2 edits). Summarizers for read_file, write_file,
    edit_file, list_directory, directory_tree, search_files,
    move_file, get_file_info. Matches suffix so namespaced and
    anonymous servers both work.
  - Reasoning preview shows tail (~260 chars), not head. R1 opens
    every turn with the same scaffolding that hid the actual
    decision in the (+N chars) elision; users reported reasoning
    looked identical across turns. The tail is where the model
    commits to an action.
  - Tool errors render red with ✗ marker instead of yellow with →.
    ERROR: prefix from flattenMcpResult (on isError) is detected.
    A failure needs different attention than a listing.

Fixed:
  - Forced-summary path no longer leaks DSML tool-call markup as
    prose. Passing tools: undefined wasn't enough — R1 primed for
    tool use would still emit <|DSML|function_calls>...</|DSML|
    function_calls> as plain text in content. Two layers: (1)
    inject an explicit user-role instruction at the end of the
    forced-summary message list telling the model to produce prose
    with no tool markup; (2) post-hoc strip known envelopes (DSML
    full-width/ASCII, Anthropic <function_calls>, and truncated
    un-closed DSML openers) from the model's response before
    yielding. stripHallucinatedToolMarkup exported for library
    callers. Fallback message when stripping leaves nothing points
    at /retry and /think.

Tests: +13, suite 319 -> 332.
  - tests/slash.test.ts (+8): /think empty/populated/help,
    /retry happy/empty-log/help, /status new format, /status
    pending-edit suppression at 0.
  - tests/loop-error.test.ts (+5): stripHallucinatedToolMarkup
    against live R1 DSML shape, Anthropic-style, truncated
    un-closed opener, plain prose, all-markup edge case.

Bumps to 0.4.2.
2026-04-21 21:26:42 -07:00
esengine cb2f2a550b feat(ux): tool-row elapsed + args · reasoning tail · /think (0.4.2)
Three small improvements surfaced from real-session feedback:

1. Tool-running row was just a spinner + raw-JSON args; user said
   "看不到任何的输出,就只知道再转圈". Added elapsed seconds counter
   and per-tool argument summarizer. Instead of
     tool<filesystem_edit_file> running... {"path":"..."...}
   you now see
     tool<filesystem_edit_file> running... 3s
       path: F:\testtest\index.html (2 edits)
   Summarizers for read_file / write_file / edit_file /
   list_directory / directory_tree / search_files / move_file /
   get_file_info, matching by suffix so bridgeMcpTools name prefix
   (e.g. "filesystem_") works too. Unknown tools fall back to a
   truncated raw preview.

2. ReasoningBlock showed the first 220 chars with "(+N chars)"
   elided at the end. R1 opens every turn with the same scaffolding
   ("let me look at the structure...") so the preview looked
   identical across turns and hid the actual decision in the
   truncated tail. Switched to showing the LAST ~260 chars —
   which is where R1 commits to an action.

3. /think slash command dumps the full reasoning text from
   loop.scratch.reasoning when the tail preview isn't enough.
   Alias: /reasoning. Graceful "no reasoning cached" message when
   the current model is deepseek-chat (no reasoning_content) or a
   turn hasn't completed yet.

Tests: +3 in slash.test.ts (/think empty + populated + help).
Suite: 319 -> 322.

Bumps to 0.4.2.
2026-04-21 21:05:44 -07:00
esengine e472f4cb79 feat(code): /apply gate + /undo + /commit + .gitignore + better UX signals (0.4.1)
A real-session bug report — "I asked the model to analyze the project
and it silently edited a file" — forced a rethink of v0.4.0's
auto-apply default. This release fixes the root cause plus a stack of
adjacent UX gaps surfaced during the fix.

WHY (behavior change): auto-apply was wrong because it conflated two
distinct events — "the model proposed a change" and "the user wants
to make the change." v0.4.0 treated every SEARCH/REPLACE block in
assistant output as permission to write. Analysis drifted into edits.
Preview-by-default + explicit /apply is Aider's model and it's honest.

Added (code mode):
  - /apply: writes pending edit blocks to disk, snapshots for /undo,
    reports per-block status
  - /discard: drops pending edits, no disk writes
  - /undo: rolls back the last applied edit batch (restores modified
    files, deletes ones created by the batch)
  - /commit "msg": git add -A && git commit -m "..." in code rootDir;
    surfaces git stderr on hook failure / nothing-staged / etc.
  - .gitignore awareness: reasonix code reads rootDir/.gitignore on
    launch and appends it to the system prompt as "don't traverse
    these paths unless asked"; hard-coded baseline (node_modules,
    dist, .git, .venv, __pycache__, target, ...) baked into the base
    prompt for projects without a .gitignore

Fixed:
  - Edits no longer auto-write. Each assistant turn's blocks are
    parsed and shown as a preview (filename + -N/+M line-count). No
    disk write until /apply. Pending persists across messages.
  - LoopEvent.forcedSummary flag on assistant_final events from the
    no-tools fallback (budget/aborted/context-guard). Code-mode
    applier skips tagged events — a wrap-up summary is not a plan.
  - Token-context guard replaces iter count as primary stop
    condition. promptTokens > 80% of ctx window -> yellow warning +
    skip tool exec + divert to summary (reason: "context-guard").
    Iter cap 24 -> 64 as last-resort backstop.
  - CODE_SYSTEM_PROMPT tightened: explicit rule that edits only fire
    on change/fix/add/remove/refactor intents; analyze/explain stay
    read-only.
  - Stray EditSummary/summarizeEdit reverted from edit-blocks.ts (a
    0.4.0 forced-summary run auto-wrote it; dead code).
  - SEARCH/REPLACE blocks now render as a proper diff (red -/green +)
    instead of going through the markdown paragraph path, which was
    joining lines with spaces and letting the bold/italic regex eat
    `*` chars inside JSDoc `/** ... */` comments. New parseBlocks
    exports an "edit-block" block kind.
  - R1 reasoning UX no longer looks frozen. Cyan braille-spinner
    heartbeat (500ms) + "reasoning" vs "streaming" label + explicit
    "R1 is thinking before it speaks — body text starts when
    reasoning completes (typically 20-90s)" message while body is
    empty. Replaces the misleading "(waiting for first token...)".
  - New "tool_start" event yielded immediately before
    tools.dispatch(). TUI renders a spinner row "⠋ tool<X> running…"
    with args preview while the Promise is pending — fixes the
    dead-air window between assistant_final (streaming cleared) and
    the tool result event, during which a multi-KB
    filesystem_edit_file could hang the UI visibly for 100ms-1s.

Tests: +36, suite 292 -> 319.
  - tests/edit-blocks.test.ts (+5): snapshot/restore round-trip,
    delete on undo-of-create, per-path dedup, path-escape defense.
  - tests/code-prompt.test.ts (+4, new file): .gitignore injection
    happy/absent/truncated paths, base-prompt ignore hints.
  - tests/slash.test.ts (+13): /apply /discard /undo /commit inside
    and outside code mode, usage hint on empty, double-quote
    stripping, help listing.
  - tests/loop.test.ts (+2): context-guard warning + forcedSummary
    flag when prompt > 80% of window, tool_start precedes tool.
  - tests/markdown.test.ts (+5): parseBlocks recognizes SEARCH/
    REPLACE into edit-block blocks, preserves multi-line JSDoc
    verbatim, handles new-file empty-SEARCH, rejects stray markers
    without close, multi-block + prose interleaving.

Bumps to 0.4.1.
2026-04-21 20:53:53 -07:00
esengine 14887264bb feat(code): reasonix code — SEARCH/REPLACE editing as a first-class mode (0.4.0)
A new opinionated subcommand that turns Reasonix into a coding
assistant, realizing the "cheap Claude Code" pitch. Layers on top of
the existing chat + MCP machinery rather than forking it.

What `reasonix code [dir]` does:
  - Auto-bridges @modelcontextprotocol/server-filesystem at [dir]
    (default CWD). No wizard, no config merge.
  - Uses a coding-focused system prompt that teaches Aider-style
    SEARCH/REPLACE blocks and forbids write_file for existing files.
  - Forces reasoner + harvest (coding repays R1 thinking).
  - Scopes the session name to the directory basename so projects
    don't cross-pollute.
  - Hooks codeMode into App.tsx — after each assistant_final, parses
    SEARCH/REPLACE blocks from the response and applies them to disk,
    reporting per-block results as info lines.

SEARCH/REPLACE semantics (src/code/edit-blocks.ts):
  - SEARCH must match byte-for-byte. No fuzzy matching — a silent
    wrong edit is worse than a loud rejection.
  - Empty SEARCH creates a new file (Aider convention).
  - First occurrence only; duplicate SEARCH text requires a new
    block with more surrounding context.
  - Path-escape defense: resolve + startsWith on the normalized
    rootDir, refuses paths climbing above the sandbox.

Library surface: parseEditBlocks, applyEditBlock, applyEditBlocks,
CODE_SYSTEM_PROMPT, and EditBlock/ApplyResult/ApplyStatus types.
Compose them yourself for custom UIs.

Out of scope for v1: /commit, /undo, diff preview, .gitignore
filtering. User's `git diff` + `git checkout` is the review/undo
surface; we run inside a git repo by convention.

Tests: +13 in edit-blocks.test.ts (parse: single/multi/multiline/
empty-search/stray-markers; apply: happy/new-file/not-found/
file-missing/path-escape/first-occurrence; batch: independent
results). Suite 279 -> 292.

Bumps to 0.4.0 (minor — new user-facing primitive + library surface
additions; no breaking changes for 0.3.x users).
2026-04-21 20:05:45 -07:00
esengine 63ced2fa5b feat: Esc to abort + 70% budget warning + README refresh (0.3.2)
0.3.1 gave the loop a safety net (forced summary at maxToolIters);
this release turns that safety net into an interactive budget. Users
no longer have to wait for the ceiling to hear from the model.

- Yellow warning event fires once per step when tool-iter count hits
  70% of maxToolIters. TUI renders it in the event log with an
  "Press Esc to force a summary now" hint; the command strip under
  the prompt also shows the Esc shortcut.
- CacheFirstLoop.abort() method + TUI Esc handler (useInput, guarded
  by busy state and a once-per-turn flag). On abort, the current tool
  call is allowed to finish, then the loop diverts to the 0.3.1
  no-tools summary path with a different prefix:
  "[aborted by user (Esc) - summarizing what I found so far]".
- New EventRole "warning" + DisplayRole "warning"; EventLog renders
  them yellow. Wired through App.tsx's event dispatcher.

README rewrite: hero leads with `npx reasonix` (no flags), wizard
story in prose. --mcp and --preset examples moved to "Advanced".
What-you-get table gains Setup wizard, Context safety net row, and
merges stdio + SSE into one MCP row. Non-goals and Configuration
trimmed to match.

Tests: +2 in loop.test.ts (warning fires exactly once at threshold;
abort() diverts to summary with "aborted" prefix). Suite: 277 -> 279.

Bumps to 0.3.2.
2026-04-21 19:50:52 -07:00
esengine 461e736cbc fix(loop): force summary when maxToolIters is reached (0.3.1)
Problem (reproduced live): user asked the reasoner to explore a Flutter
project via filesystem MCP. The model chained 8 list_directory /
read_file calls, the loop hit the default maxToolIters=8, quietly
yielded a "done" event with content "[max_tool_iters reached]", and
exited. The App.tsx doesn't render `done` content anywhere, so the
user saw a blank assistant reply after 8 tool calls — as if Reasonix
just stopped for no reason.

Fix:
  - When the tool-call loop exhausts its budget with tool_calls still
    pending, make one final request to DeepSeek with tools omitted.
    Forces a text-only answer based on everything in history. Yield
    as a normal assistant_final event, prefixed with
    "[tool-call budget (N) reached - forcing summary from what I found]"
    so the user knows why they got a summary instead of more drilling.
  - Default maxToolIters 8 -> 24. Eight was below the floor for any
    real filesystem exploration; 24 still caps a confused model's
    damage.

Tests: +1 in loop.test.ts with maxToolIters=2, every response wanting
tools, proving the summary call fires and assistant_final carries the
budget annotation. Suite: 276 -> 277.

Bumps to 0.3.1.
2026-04-21 19:40:15 -07:00
esengine d932c50ab3 release: v0.3.0 — context gauge + /compact + stable graduation
Drops the prerelease tag and ships 0.3.0 stable. Since 0.2.2 this
includes: MCP (stdio + SSE, multi-server, curated catalog), the
first-run wizard that replaces the --mcp flag maze, config-backed
defaults, and a three-layer context-safety net (per-tool cap +
auto-heal on session load + manual /compact).

Added this commit:
  - StatsPanel context gauge: "ctx 42k/131k (32%)" with yellow at
    50% and red + "/compact" nudge at 80%+. Reads SessionSummary's
    new lastPromptTokens field; model max comes from
    DEEPSEEK_CONTEXT_TOKENS (131072 for chat/reasoner).
  - /compact [cap] slash: re-applies truncation with a tighter 4k
    char cap (or user-supplied), rewrites the session file on disk
    so the next launch doesn't re-inherit a fat file. Reports chars
    and approximate tokens saved.
  - SessionSummary.lastPromptTokens, AppendOnlyLog.compactInPlace(),
    session.rewriteSession(), loop.compact() — all new exports.

Tests: +5 (slash /compact x4, telemetry lastPromptTokens x1).
Suite: 276 passing (up from 241 in alpha.4).

Migration note: one silent behaviour change — sessions built with
pre-alpha.6 clients auto-heal on load (oversized tool results get
truncated). The conversation is preserved; only the bloat shrinks.
2026-04-21 19:35:36 -07:00
esengine b4c461c6ad fix(mcp): cap tool-result size + annotate context-overflow errors
Problem: a single `read_file` on a big file returned ~3 MB of text
through flattenMcpResult; it landed in session history verbatim and
every subsequent turn 400'd with DeepSeek's "maximum context length
is 131072 tokens. However, you requested 929,452 tokens." The user
had no way to recover short of deleting the session file by hand.

Fix, at both ends:
  - Prevention: bridgeMcpTools now caps each result at 32k chars by
    default. truncateForModel keeps head + 1KB tail with a clear
    "[...truncated N chars...]" marker, so the model sees both
    "what the tool started returning" and "how it ended" (important
    for error messages appended at the bottom of stack traces).
    Override: bridgeMcpTools(client, { maxResultChars: N }).
  - Diagnosis: formatLoopError() detects "maximum context length"
    in the raw API error, pretty-prints the requested-token figure,
    and tells the user exactly what to do — /forget to wipe the
    session file, /clear for display-only. Other errors pass through.

Exports: DEFAULT_MAX_RESULT_CHARS, truncateForModel, FlattenOptions,
formatLoopError — so library callers get the same protection.

Tests: +3 for truncation (head+tail preserved, no-op below cap,
end-to-end dispatch), +3 for error annotation (token figure,
passthrough, figure-less overflow). Suite 262 -> 268.

Bumps to 0.3.0-alpha.6.
2026-04-21 19:22:58 -07:00
esengine 2f4e28b7cb feat(ux): setup wizard + config defaults — reasonix with no args just works
The previous UX required users to memorize a pile of flags
(`--mcp "name=npx -y @scope/pkg /path"`, `--harvest`, `--branch N`,
`--session`) before they could use any of Reasonix's power features.
That was wrong: the target audience is ordinary users, not CLI
archaeologists.

New flow:
  - `npx reasonix` — on first run opens an Ink wizard (API key →
    preset pick → MCP multi-select → per-server args → save). On
    subsequent runs launches chat with the saved defaults.
  - `reasonix setup` re-runs the wizard any time; existing selections
    are pre-checked via package-name matching against the catalog.
  - `~/.reasonix/config.json` now stores preset + mcp + session +
    setupCompleted alongside apiKey. Precedence: explicit CLI flag
    wins, then --preset, then config.preset, then hardcoded fast
    defaults. `--no-config` is the escape hatch for CI.
  - `--preset <fast|smart|max>` new on both chat and run so power
    users get the bundled settings too.
  - `/mcp` slash — lists attached servers + tool registry mid-chat.
  - `/setup` slash — prints instructions to exit and re-run
    `reasonix setup`. Live reconfig is out of scope: swapping tools
    mid-session would invalidate the byte-stable prefix.

Fixes two real bugs found during manual testing:
  - `@modelcontextprotocol/server-fetch` and `server-sqlite` were in
    the catalog but don't exist on npm (Python-only reference
    implementations). Picking either one produced a 404 that killed
    the chat. Removed.
  - One broken MCP server took down the whole chat via
    `process.exit(1)`. Now each failure prints `▸ MCP setup SKIPPED`
    pointing at `reasonix setup` and the session continues with the
    servers that worked.

Tests (+21, suite 241→262):
  - tests/resolve.test.ts (+11) — flag/preset/config precedence, cap
    and off paths, --no-config, --no-session isolation.
  - tests/config.test.ts (+2) — extended schema round-trip,
    session:null meaning ephemeral.
  - tests/slash.test.ts (+4) — /mcp empty + populated, /setup hint,
    help coverage.
  - tests/wizard.test.ts (+4) — buildSpec → parseMcpSpec round-trip
    for filesystem / memory / spaces-in-path / unknown-entry fallback.

Bumps to 0.3.0-alpha.5.
2026-04-21 19:14:59 -07:00
esengine 5f715d05ee feat(mcp): SSE transport — remote/hosted servers via --mcp "https://…"
Implements the MCP 2024-11-05 HTTP+SSE wire: GET the SSE URL, wait for
event: endpoint, POST every outgoing JSON-RPC frame to that URL, read
responses off the SSE channel. Headers (Authorization, etc.) pass
through to both the GET and POSTs so hosted servers behind auth work.

--mcp now routes anything starting with http:// or https:// to
SseTransport; everything else stays stdio. McpSpec becomes a
discriminated union { transport: "stdio" | "sse", … } — callers that
inspected spec.command/args must branch on spec.transport first.

Tests: in-process http.Server fake implementing the SSE wire —
relative endpoint resolution, absolute endpoints, a full McpClient
initialize → listTools round-trip, handshake failure propagation.
Plus parseMcpSpec SSE cases (case-insensitive scheme; ws:// stays
routed to stdio as there's no ws transport yet). Suite: 241 passing.

Bumps version to 0.3.0-alpha.4.
2026-04-21 18:55:06 -07:00
esengine b662c63894 data(mcp): multi-server live run — 2 subprocesses, 1 prefix hash,
Cache-First holds

Third MCP live data point, the ecosystem proof. Run against two MCP
servers concurrently:

  --mcp "demo=node --import tsx examples/mcp-server-demo.ts"
  --mcp "fs=npx -y @modelcontextprotocol/server-filesystem <dir>"

The model used tools from both in one conversation — computed 17+25
via demo_add, then wrote "42" to a real file on disk via
fs_write_file (with a graceful permission-denied recovery in
between).

  turns: 5, tool calls: 4
  cache: 81.1%, cost $0.00185, 95.9% cheaper than Claude
  prefix hashes: ONE across all 5 turns — byte-stable prefix
  survives two concurrent MCP subprocesses

Reference: benchmarks/tau-bench/transcripts/mcp-multi-server.jsonl
(replay offline to verify — no API key needed).

Also adds .claude/settings.local.json to .gitignore (local Claude Code
settings are personal and shouldn't be committed).
2026-04-21 18:41:29 -07:00
esengineandClaude Opus 4.7 e793c7d59d chore(release): v0.3.0-alpha.3 — multi-server MCP + mcp list discovery
Adds the last bit of UX I'd actually hand to a new user: a way to
discover what MCP servers they can try.

New:
- reasonix mcp list            curated catalog of popular official MCP
                               servers, each with a copy-pasteable
                               --mcp command. filesystem / fetch /
                               github / memory / sqlite / puppeteer /
                               everything. --json for machine output.
- --mcp repeatable             multi-server support (already in main,
                               now released on npm @alpha)
- "name=cmd args..." spec      per-server namespace prefix, tools land
                               as fs_read_file, mem_set, etc.

Packages 7cd8621 (multi-server) + new mcp list command as alpha.3.

Discovery is hardcoded not fetched: the list changes slowly, offline
users shouldn't have to wait on a network fetch to see what exists.
Updating it = one file edit + patch release.

Suite 233/233; no functional changes to the existing tested paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 18:31:33 -07:00
esengineandClaude Opus 4.7 7cd862194a feat(mcp): multi-server support — repeatable --mcp with per-server names
`--mcp` is now a collect-style option; pass it multiple times to attach
multiple servers to the same chat / run. Each spec can optionally name
its namespace:

  reasonix chat \
    --mcp "fs=npx -y @modelcontextprotocol/server-filesystem /tmp/safe" \
    --mcp "demo=node --import tsx examples/mcp-server-demo.ts"

Tools land in a shared ToolRegistry with the prefix baked in
(fs_read_file, demo_add, …). The model sees the flat namespace; the
registry dispatches to the right subprocess.

Spec syntax (new src/mcp/spec.ts::parseMcpSpec):
- "name=command args..."  → namespaced; `[a-zA-Z_][a-zA-Z0-9_]*` name
- "command args..."       → anonymous; tools keep their native names

The identifier regex is deliberately narrow so Windows drive letters
(`C:\...`) never accidentally look like a namespace prefix.

--mcp-prefix still exists as a global fallback, but only applies when
exactly one anonymous spec is given. With named specs, each server
brings its own name.

Also fixes a shellSplit bug: backslashes outside quotes were being
treated as POSIX escapes, which mangled every Windows path. Now they
pass through literally outside quotes (use quotes to group args with
spaces). Matches user expectation on Windows; POSIX users who want
escape-a-space can quote the arg.

+9 tests: parseMcpSpec (8 cases including Windows drive letters,
identifier-name validation, empty/malformed input) + an integration
test spawning two demo subprocesses concurrently with different
prefixes, verifying the shared registry dispatches to each correctly.
Suite: 233 passing (was 224).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 18:22:53 -07:00
esengineandClaude Opus 4.7 407793c99e chore(release): v0.3.0-alpha.2
Packages 5e124ce's Windows fix + second live MCP data point as an
alpha. v0.3.0-alpha.1 is technically broken on Windows (--mcp "npx ..."
can't find npx.cmd), so this supersedes it on the @alpha npm tag.

No functional changes since 5e124ce — just version bump + CHANGELOG.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 18:08:27 -07:00
esengineandClaude Opus 4.7 5e124cec36 feat(mcp): cross-platform shell fix + second live data point via
official @modelcontextprotocol/server-filesystem

Two things together — the fix is what unlocked the data.

1. StdioTransport now handles Windows .cmd/.bat shims (npx.cmd,
   pnpm.cmd) without triggering Node's DEP0190 deprecation warning.
   On win32, shell defaults to true; we build a quoted command line
   ourselves (command bare so PATH lookup works, args quoted) and
   pass it as a single string, dodging the shell:true + args[]
   anti-pattern. POSIX path unchanged. New `shell?: boolean` option
   lets callers override the platform default (e.g. tests spawning
   node.exe directly).

2. Second live MCP run, this time against the *external*,
   production @modelcontextprotocol/server-filesystem:

     5 turns, 4 tool calls, cache **96.7%**, cost $0.00124,
     97% cheaper than Claude at equivalent tokens.

   The model tried list_directory on the wrong path, got an access-
   denied, self-recovered by calling list_allowed_directories, then
   list_directory + read_text_file. Messy realistic run. Byte-stable
   prefix held across every turn.

   This matters because the first MCP run (v0.3.0-alpha.1,
   mcp-demo.add.jsonl) was against our own bundled demo server.
   Someone could reasonably argue "of course it caches — you wrote
   the server". This one answers that: cache-first works on code
   we don't control.

Reference transcripts:
- benchmarks/tau-bench/transcripts/mcp-demo.add.jsonl       (own server)
- benchmarks/tau-bench/transcripts/mcp-filesystem.jsonl     (official)

README gains a two-row table with both numbers + a no-API-key
reproduction block pointing at both committed transcripts.

Tests still 224/224 green. Integration tests explicitly set
shell: false since they spawn node.exe by absolute path.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 18:04:09 -07:00
esengineandClaude Opus 4.7 7fce6e650c chore(release): v0.3.0-alpha.1 — MCP client + first live run
End-to-end MCP integration verified on live DeepSeek. Recorded run at
benchmarks/tau-bench/transcripts/mcp-demo.add.jsonl:

  ▸ spawned bundled demo MCP server (3 tools bridged)
  ▸ model chose the add tool, received the result via MCP stdio
  ▸ follow-up turn hit 96.6% cache (byte-stable prefix held across
    the tool-call round-trip)
  ▸ total cost $0.000254 — 94% cheaper than Claude at same tokens

This is the v0.1 Cache-First claim extending verbatim to arbitrary MCP
tools. The mechanism wasn't altered; the registry just serves MCP tools
as if they were native. That extension *is* the v0.3 headline.

Shipping as alpha because:
  - one real run is proof-of-concept, not proof-at-scale
  - SSE transport, resources, prompts, and progress notifications are
    still deferred
  - official external MCP servers (filesystem, github, etc.) haven't
    been exercised in CI yet

Suite: 224 passing (10 fake-transport unit, 9 shell-split, 2 real-
subprocess integration, plus the existing 203).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 17:52:54 -07:00
esengineandClaude Opus 4.7 5941043489 feat(mcp): CLI --mcp flag + bundled demo server + subprocess integration test
End-to-end MCP integration — any MCP server's tools flow through
Reasonix's Cache-First loop now, from one CLI flag.

CLI:
  reasonix chat --mcp "<command args...>"        interactive TUI
  reasonix run "<task>" --mcp "<command args...>" one-shot
  optional: --mcp-prefix <str>                   namespace tool names

The flag takes a shell-style command. shellSplit() parses it
(respecting single/double quotes, backslash escapes, tab-space
runs). Unterminated quotes throw with a clear error instead of
silently mangling argv. chat / run both spawn + bridge before
looping, tear down on exit.

App.tsx accepts an optional `tools?: ToolRegistry` prop. When
present, its specs get folded into ImmutablePrefix and dispatch
goes through it — so MCP tools are indistinguishable from native
tools to the loop.

Bundled demo server: examples/mcp-server-demo.ts
~160 lines, no dependencies beyond what's already installed.
Exposes echo / add / get_time. Two uses:
  - users can try the whole integration with no external install:
      reasonix chat --mcp "node --import tsx examples/mcp-server-demo.ts"
  - gives the integration test a real subprocess target

Integration test: tests/mcp-integration.test.ts
Spawns the demo server (`node --import tsx ...` — cross-platform,
avoids Windows `.cmd` resolution issues), walks the full handshake,
bridges tools, dispatches through the registry. Kept in a
separate file from the fake-transport unit tests so subprocess
flakiness doesn't hide unit-test regressions.

+11 tests (9 shellSplit + 2 integration). Suite: 224 passing (was
213). src/mcp/README.md updated with CLI usage + status of what's
shipped vs deferred.

Still no version bump — next is v0.3.0-alpha, once we've seen a
real LLM call go through an MCP tool end-to-end. Everything's in
place for that now; remaining is one live run.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 17:30:33 -07:00
esengineandClaude Opus 4.7 f142f8874d feat(mcp): v0.3 foundation — stdio client + ToolRegistry bridge
Minimal MCP (Model Context Protocol) client, hand-rolled, zero
runtime deps. Lets Reasonix consume tools from any MCP server while
keeping Cache-First + repair applied to the whole thing.

Why roll-our-own instead of @modelcontextprotocol/sdk:
- zero dep cost, consistent with how we wrote the DeepSeek client
- surface tuning — only initialize / tools/list / tools/call for now
- insulation from SDK breaking changes
See src/mcp/README.md for the full reasoning.

Shipped in this pass:
- src/mcp/types.ts      JSON-RPC 2.0 + MCP message types
- src/mcp/stdio.ts      McpTransport interface + NDJSON stdio impl
- src/mcp/client.ts     connect / listTools / callTool with timeouts
                        + id correlation
- src/mcp/registry.ts   bridgeMcpTools(): MCP → Reasonix ToolRegistry,
                        so MCP tools inherit Cache-First + repair
                        automatically
- src/mcp/README.md     design choices + usage example + what's next

Deferred (each is its own follow-up commit):
- CLI wiring: `reasonix chat --mcp <cmd>` — thin layer on top of
  bridgeMcpTools + ImmutablePrefix
- resources/* and prompts/* methods — Reasonix doesn't surface those
  today
- SSE transport — stdio covers common case
- Progress notifications / streaming tool results

+10 tests using an in-process FakeMcpTransport (no child processes in
unit tests — flaky). Covers initialize handshake, pre-init refusal,
tools/list, tools/call roundtrip, server errors, bridge registration,
name prefixing, and result flattening (text / image / error cases).
Suite: 213 passing (was 203).

Public surface: McpClient / StdioTransport / bridgeMcpTools /
flattenMcpResult / MCP_PROTOCOL_VERSION + all relevant types exported
from the library barrel. No version bump — foundation, not release.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 17:04:16 -07:00
esengineandClaude Opus 4.7 fb5977743c data(harvest-bench): 18-run set + honest findings — harvest doesn't
lift accuracy on math/logic; pivot Pillar 2 positioning

Second harvest eval run, 6 tasks × 3 modes × 1 repeat = 18 runs. This
time the task set included three hard tasks chosen specifically for
known V3 failure modes (base-2 Fermat pseudoprime → 341,
derangements of 7 → 1854, Euler polynomial first-fail → n=40).

  mode              pass  cache   cost        harvest signal
  baseline          6/6   39.5%   $0.00137    —
  reasoner          6/6   74.2%   $0.00417    —
  reasoner-harvest  5/6*  61.6%   $0.00330    2.8 sg / 1.3 un

  (* one 300s timeout on pseudoprime_base2)

Honest headline: **V3 passed all six.** Including the hard ones. The
checkers specifically reject the common V3 wrong-answers (561 for
pseudoprime, 265/1855 for D_7, 41/1681 for Euler) — none of those
fired. These problems are well-enough known that V3 has them in
training and R1 has no accuracy headroom to win back.

What this means for v0.3 positioning:

- We cannot ship a "harvest improves answer quality" claim off this
  data — twice now. That's not a framework bug; it's a signal that
  Pillar 2's value is NOT in the "make math more right" story.
- Plausible actual value surfaces: dev-facing introspection of R1's
  planning, or steering tool-call sequences under uncertainty. Neither
  is exercised by this harness.
- Cache-First story DOES extend to reasoner mode (74% mean cache even
  on R1 turns). That's a Pillar 1 finding reinforcing the v0.1 data,
  not a Pillar 2 finding.

findings section updated inside report.ts's renderFindings() so future
regens preserve the analysis. Committed results.json is the full
artifact for anyone who wants to reasonix replay / diff individual
runs.

Recommendation in-report: next release is better served by MCP client
(Pillar 1 leverage across the ecosystem) than by a third harvest-bench
data push. Pillar 2 needs a reframe — likely "debug visibility" or
"tool-sequence planning", not "accuracy multiplier".

Unexpected positive signal worth noting: cross-session prompt cache.
Baseline mean cache hit was 39.5% even on mode A — the same system
prompt across many runs warmed DeepSeek's server-side cache, showing
Cache-First's value isn't purely per-session.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 09:47:01 -07:00
esengineandClaude Opus 4.7 26b50e9242 feat(harvest-bench): 3 hard tasks + --timeout flag
Post-first-run adjustments informed by 7233edd's data:

- Seed tasks bottomed out at V3 chat (3/3 pass on all easy ones), so
  harvest couldn't differentiate. Add three hard tasks picked for
  known V3 failure modes:
  - pseudoprime_base2       (341; V3 commonly says 561 — Carmichael)
  - derangements_d7         (1854; V3 hits n!/e rounding errors)
  - euler_quadratic_break   (40; V3 confuses with n²−n+41 first-fail)
  Each is a single-integer answer → reuse a new checkSingleInt helper.
  Easy tasks stay for floor reference.

- `--timeout <seconds>` flag (default 300s, was 120s client default).
  Prior run hit the 120s cap on three_hats/reasoner-harvest because R1
  alone took ~100s. 300s gives headroom without encouraging runaway
  loops.

+9 tests on new checkers and the common wrong answers (e.g., checker
rejects 561 for pseudoprime_base2, rejects 265/1855 for D_7, rejects
41/1681 for euler_quadratic). Suite: 203 passing.

CI --dry step still covers the runner. No API calls in this commit —
next run uses the expanded task set.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 09:16:18 -07:00
esengineandClaude Opus 4.7 7233eddced data(harvest-bench): first 9-run live data + honest findings
First real data from the v0.3 harvest eval harness (3 tasks × 3 modes,
1 repeat). Findings are not the marketing-friendly "harvest +N% quality"
we wanted — but they're honest, and they sharpen what v0.3 actually
needs.

Headline numbers (per-run means):

  mode              pass  cache   cost        harvest signal
  baseline          3/3   0.0%    $0.00142    —
  reasoner          3/3   79.0%   $0.00355    —
  reasoner-harvest  2/3*  59.7%   $0.00184    3.3 sg / 1.3 un

  (* one timeout, see known issues)

Three honest findings, committed to report.md:

1. V3 chat already solves all three seed tasks. The task set bottoms
   out — baseline's 3/3 pass rate means there's no headroom for R1,
   let alone R1+harvest, to win on quality.
2. Reasoner costs ~2.5× chat on these problems with identical verdicts.
   On this task set, there is no quality argument for R1. (The
   cache-hit story DOES extend — reasoner mode still gets 79% mean
   cache through the Cache-First loop, so Pillar 1 generalizes.)
3. Harvest produces real signal (3-5 subgoals, 1-2 uncertainties per
   run) AND real latency — one run hit the 120s client timeout because
   the harvest V3 call pushed past R1's own ~100s budget.

What this changes for v0.3:

- We cannot ship a "harvest pays for itself" claim off this data.
- The harness is fine; the task set is the bottleneck.
- Next iteration needs problems where V3 demonstrably fails, so R1
  gains real headroom and harvest's uncertainty signal has something
  to correlate with.

report.ts gains a renderFindings() helper so future runs can preserve
the hand-written interpretation alongside the auto-generated tables.

No release tag — this is data collection, not a version bump.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 09:09:34 -07:00
esengineandClaude Opus 4.7 3a92ba2787 feat(bench): harvest eval harness — Pillar 2 scaffold
New benchmarks/harvest/ directory alongside tau-bench/. Measures
Pillar 2 (R1 Thought Harvesting) the same way tau-bench-lite measures
Pillar 1 — isolate one variable, produce reproducible numbers.

Three modes per task, each isolates one variable:
- baseline          deepseek-chat, no Reasonix tricks — V3 floor
- reasoner          deepseek-reasoner, no harvest — raw R1 gain
- reasoner-harvest  deepseek-reasoner + harvest — incremental cost /
                    quality from the extra V3 harvest call

Seed tasks (v0.3):
- mod7_list           number theory, 29-element set — expansive plan state
- flips_until_3heads  probability, single integer — derive vs recall
- three_hats          logic puzzle, one-word answer — deduction chain

All checkers are deterministic (regex + set/value compare). No LLM
judge. Failing checker in CI catches reality drift without needing an
API key — sanity verdict is exercised on every dry-run.

Runner supports --task, --mode, --repeats, --transcripts-dir, --dry.
Transcripts reuse the same TranscriptRecord format as tau-bench, so
reasonix replay / reasonix diff work unchanged.

CI now runs the harvest harness's --dry in addition to tau-bench's,
so bench regressions get caught before a real run.

+16 tests covering checker verdicts for all three tasks, empty-input
robustness, determinism, and task-set invariants. Suite: 194 passing.

No API calls yet — next step is a live run to produce the first
harvest-bench/report.md. Expected cost for 3 tasks × 3 modes × 1
repeat: ~$0.10-0.30 (dominated by the reasoner calls).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 08:53:01 -07:00
esengineandClaude Opus 4.7 6307b92af2 feat(cli): reasonix sessions — list + inspect saved chats
Promotes the existing /sessions slash command into a top-level CLI
command so users can audit their stored sessions without launching
the TUI. Two modes:

  reasonix sessions                list every session under
                                   ~/.reasonix/sessions/
  reasonix sessions <name>         dump messages from one session
  reasonix sessions <name> -v      include system prompts + tool-call
                                   args

Reuses listSessions / loadSessionMessages / sessionPath from
src/session.ts — no new library surface. Zero API calls; filesystem
reads only.

Use case: quickly checking "what have I been asking the assistant?"
without running chat, or piping one session's content into a tool
chain for batch analysis.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 08:39:14 -07:00
esengineandClaude Opus 4.7 2e42ae53aa feat(diff): show harvest-signal deltas in summary + markdown
When at least one side of a transcript diff has plan state (Pillar 2
harvest output), the summary now includes three extra rows: harvest
turns, subgoals, and uncertainties. Lets a harvest-on run be compared
head-to-head with its harvest-off twin — which is exactly the
comparison v0.3's eval harness will need to produce.

Rows only appear when at least one side carries planState. No-harvest
diffs render identically to v0.2.2 — zero visual noise, zero surprise
for existing callers.

+2 tests confirming the row appears when harvest is present AND
omits entirely when both sides are clean. Suite: 178 passing.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 08:36:38 -07:00
esengineandClaude Opus 4.7 5ff0ef4535 chore(ci+examples): bench dry-run in CI + programmatic replay/diff example
Two small adds that make the existing v0.2 infrastructure easier to
pick up:

1. CI now runs `npx tsx benchmarks/tau-bench/runner.ts --dry` on every
   push and PR. Catches bench harness regressions (task factories, CLI
   parsing, file IO, per-sub-call transcript emission) without needing
   a DEEPSEEK_API_KEY in CI.

2. examples/replay-and-diff.ts demonstrates the library surface
   programmatically — readTranscript, computeReplayStats,
   diffTranscripts, renderDiffSummary, plus direct report.pairs
   inspection. Runs with no API key against the committed reference
   transcripts; verifies the v0.1 numbers (94% cache, −20% cost, 1
   byte-stable prefix) fall out of the JSONL alone.

Useful for anyone wiring a CI gate or eval dashboard on top of Reasonix
— which is the main use case that justifies shipping transcripts in
the package in the first place.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 08:32:51 -07:00
esengineandClaude Opus 4.7 ce963be2e2 fix(ui): plan state colors — semantic + readable on dark terminals
The old PlanStateBlock rendered all four fields in the same magenta
text. On terminals with dark themes (which is most of them), magenta
is notoriously low-contrast — users reported not being able to read
harvest output.

Fix:
- label gets a semantic color (bold): subgoals cyan, hypotheses green,
  uncertainties yellow, rejected red-dim.
- the item list itself renders in default foreground — always
  readable on any background.
- count indicator "(N)" is dimColor so it doesn't compete with content.

Before:  all four lines identical magenta (hard to tell fields apart,
         hard to read on dark).
After:   label color communicates meaning (rejected feels negative,
         uncertainties feel like "watch out"), content stays crisp.

Used by the live chat TUI AND replay/diff TUIs, so the fix lands
everywhere harvest output shows up.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 08:28:17 -07:00
esengineandClaude Opus 4.7 e5357f0df5 feat(cli): reasonix run gains --harvest / --branch / --transcript
Parity with `reasonix chat`: lets a one-shot non-interactive call
produce a transcript with harvest signal, for quick API-backed
verification of a harvest session without firing up the Ink TUI.

Usage:
  reasonix run "your question" \
    --model deepseek-reasoner --harvest \
    --transcript /tmp/h.jsonl

Afterward:
  reasonix replay /tmp/h.jsonl     # see the planState inline

The transcript path is self-closing — the command streams output as
usual, then prints a hint pointing at replay. No other behavior change;
`reasonix run` without the new flags behaves exactly as before.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 08:15:40 -07:00
esengineandClaude Opus 4.7 c148b342fc feat(harvest): thread planState through transcripts + replay UIs
Prep work for v0.3's harvest eval harness. The Pillar 2 plan state
already exists in the live chat UI; this commit makes it persist and
display in every replay/diff surface, so future harvest data can be
audited the same way cache hit is today.

- TranscriptRecord gains optional planState (TypedPlanState).
- recordFromLoopEvent copies it from LoopEvent.planState on
  assistant_final events. Empty plan state is omitted entirely — absent
  means "no signal", indistinguishable from harvest-off, saves bytes.
- PlanStateBlock extracted from EventLog to a shared component so the
  live chat TUI, replay TUI, and (transitively) diff TUI all render
  harvest output identically.
- RecordView shows the plan state under the assistant text when present.
- replay stdout --print mode prints the same four lines (subgoals /
  hypotheses / uncertainties / rejected) under each agent turn that
  carried plan state.
- ReplayStats gains harvestedTurns, totalSubgoals, totalUncertainties.
  Surfaced in the replay summary so you can see at-a-glance whether
  harvest produced signal across the session.
- Zero cost when harvest is off — the whole path is gated on the
  presence of planState in the record.

+4 tests (round-trip, empty-state omission, aggregate counting,
no-harvest zero case). Suite: **176 passing**.

No CHANGELOG bump — this is prep, not a release.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 08:11:01 -07:00
esengineandClaude Opus 4.7 74e796b1c6 chore(release): v0.2.2 — 48-run bench, 100% reasonix pass rate
First three-repeat data set on τ-bench-lite. Headline numbers land in
the same region as the v0.1 16-run set, with the confidence that comes
from 3× the samples.

  metric                baseline    reasonix    delta
  runs                  24          24          —
  pass rate             96% (23/24) 100% (24/24) reasonix holds guardrail
  cache hit             46.6%       94.4%       +47.7pp
  cost / task           $0.002599   $0.001579   −39% (×0.61)
  vs Claude (estimate)  —           —           ~96% cheaper

The one baseline failure is a t07_wrong_identity run where baseline
skipped identity verification and mutated the wrong user's order —
exactly the kind of guardrail slip you'd expect from a cache-churning
agent whose context reshuffles every turn.

Also includes:
- README hero table refreshed to the 48-run numbers + 3-repeat
  reproduction command
- benchmarks/tau-bench/results.json replaced with the 48-run data
- benchmarks/tau-bench/report.md regenerated from the new JSON

(The t05 predicate fix that made 100% pass possible landed one commit
earlier at c1965b1.)

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 08:00:47 -07:00
esengineandClaude Opus 4.7 c1965b1d1e fix(bench): relax t05 predicate — cancelled is an acceptable alternative to refused
The v0.1 16-run bench surfaced an honest false-negative on t05: the task
wanted "no refund on a processing order", and reasonix followed that —
but then offered to cancel the order instead when the user kept pushing.
The old predicate required status to stay `processing`, which penalized
agents for being helpful. It failed reasonix once and the baseline zero
times, not because reasonix behaved worse but because baseline never
reached the "offer cancel" branch.

New predicate: pass iff no refund row is written for o_1002 AND the
status ends up in {processing, cancelled}. "No refund" remains the
hard constraint. Cancellation is now honored as a reasonable business
response — which is how a real support flow would be scored.

+3 tests pinning the three outcomes (refuse / cancel / illegally
refund). Suite: 172 passing.

No report regeneration in this commit — rerun the bench in a follow-up
with the new predicate to pick up any accuracy change.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 07:16:12 -07:00
esengineandClaude Opus 4.7 725cb16ff8 chore(release): v0.2.1
Single release covering both TUI additions (replay + diff) shipped
since 0.2.0. No functional changes; version bumps + CHANGELOG.

Highlights (see CHANGELOG.md for detail):
- reasonix replay <transcript>   → Ink TUI by default (j/k/g/G nav)
- reasonix diff <a> <b>          → split-pane TUI with n/N divergence
                                   jump; --md / --print / --tui flags
                                   to override

Stdout paths preserved via --print and non-TTY auto-detection, so CI
and pipe workflows behave as they did in 0.2.0.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 07:11:26 -07:00
esengineandClaude Opus 4.7 ceb4840a63 feat(diff): split-pane Ink TUI with n/N divergence jump
`reasonix diff <a> <b>` now mounts an interactive Ink UI by default:
side-by-side panes sharing a cursor, a compact header that colors cache
and cost deltas in green/red, and a divergence-aware jump cursor. Press
n to skip straight to the next non-matching turn — the whole point of
a diff tool — or N/p to go back.

Default behavior picks itself based on context:
- stdout is a TTY and no --md: Ink TUI
- --md <path>: writes markdown AND prints the stdout summary (existing)
- --print, or piped stdout: stdout summary (existing)
- --tui: force TUI even when piped (rare)

Key bindings:
- j / ↓ / space / enter  — next turn
- k / ↑                  — prev turn
- n                      — next divergent turn
- N / p                  — prev divergent turn
- g / G                  — first / last
- q / ctrl-c             — quit

Implementation:
- src/cli/ui/DiffApp.tsx — the new split-pane component.
- src/cli/ui/RecordView.tsx — extracted from ReplayApp so both UIs
  share the same visual grammar (user cyan, assistant green with cache
  badge, tool yellow, error red). ReplayApp now imports from here
  instead of carrying its own copy.
- src/diff.ts gains pure `findNextDivergence` / `findPrevDivergence`
  helpers so the nav logic is unit-testable without Ink (`n`/`N` keys).
  Both guard against out-of-bounds `fromIdx`.

Cursor defaults to the first divergent turn (or 0 if everything
matches) — skips the "scroll through 7 identical turns before getting
to the interesting bit" experience.

+4 tests covering divergence navigation. Suite: 169 passing (was 165).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 07:01:41 -07:00
esengineandClaude Opus 4.7 4bca4cedd8 feat(replay): Ink TUI for transcript browsing
`reasonix replay <transcript>` now opens an interactive Ink UI by
default: j/k navigation across turn-pages, a cumulative stats sidebar
(StatsPanel reused from the chat TUI) that updates as you scrub, and
a key-bindings footer.

Key bindings:
- j / ↓ / space / enter  — next turn
- k / ↑                  — previous turn
- g / h / ←              — first
- G / l / →              — last
- q / ctrl-c             — quit

The existing stdout path is preserved behind --print (also auto-
enabled when stdout is not a TTY, so pipes / CI logs still behave).
--head / --tail only work in print mode.

Implementation:
- src/replay.ts gains pure helpers `groupRecordsByTurn` (groups records
  into turn-pages, preserving in-turn order) and `computeCumulativeStats`
  (stats up to and including a given page index). These stay in the
  library module so they're unit-testable without Ink.
- src/cli/ui/ReplayApp.tsx — the new Ink component. Reuses StatsPanel
  and writes its own RecordView (lighter than EventRow, no streaming /
  planState / branch paths to worry about in replay).
- Cursor defaults to the last page because users usually want the
  summary + final turn first, not to start from turn 1.

+6 pure tests for groupRecordsByTurn + computeCumulativeStats.
Suite: 165 passing (was 159).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 06:53:50 -07:00
esengineandClaude Opus 4.7 736611293f chore(release): v0.2.0
Ships the v0.1 + v0.2 work as a single versioned release.

- package.json: 0.0.6 → 0.2.0
- src/index.ts: VERSION constant synced
- CHANGELOG.md: new file, covers v0.2.0 (replay/diff/enriched transcripts)
  and v0.1.0 (τ-bench-lite + first validated numbers)
- README.md: "Validated numbers" table replaced with the 16-run bench
  results (94.3% cache, −42% cost, ~96% cheaper than Claude) and a
  zero-cost self-verification block pointing at the committed transcripts

No API calls, no functional changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 06:44:37 -07:00
esengineandClaude Opus 4.7 dc5a65b02b fix(bench+diff): per-sub-call baseline transcripts + clearer labels + reference artifacts
Two fixes and one addition:

1. Baseline transcripts now emit one assistant_final per client.chat() call
   (per sub-call) instead of one aggregated record per user turn. Reasonix
   already emits per-model-call granularity, so diff's count was previously
   apples-to-oranges ("turns 3 vs 6" looked like Reasonix doing 2× the work
   when it was actually just finer record granularity).

2. Diff rendering renames "turns (assistant)" to "model calls" — less
   ambiguous once both sides emit at the same granularity, and
   distinguishes model calls from user turns.

3. Commit reference transcripts at benchmarks/tau-bench/transcripts/ so
   readers can run `reasonix replay` / `reasonix diff` without an API key.
   Gitignore carved out the directory; a brief README explains how to
   regenerate (~$0.003/run) and reproduce the headline numbers.

Fresh run on t01_address_happy:
- cache hit: 45.9% → 93.9% (+48.0pp)
- cost:     $0.001192 → $0.000953 (−20.1%)
- prefix hashes: 0 (baseline, untracked) → 1 byte-stable (reasonix)

Diff also surfaced a real behavioral difference the v0.1 bench numbers
hid: Reasonix calls lookup_order immediately while baseline asks for
name first. Both pass the task; the two paths are observable in the
committed transcripts.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 06:25:02 -07:00
esengineandClaude Opus 4.7 a930d1818a feat(v0.2 MVP): transcript replay + diff — offline cache/cost auditing
Turns transcripts into first-class audit artifacts. Anyone with a
.jsonl transcript can reproduce the headline numbers (cache hit, cost,
vs-Claude) without re-running the LLM — the economics travel with the
data.

New surface:
- reasonix replay <transcript>   pretty-print a run + rebuild its summary
- reasonix diff <a> <b> [--md]   compare two transcripts: aggregate
                                 deltas + first divergence + prefix stability

New modules:
- src/transcript.ts — canonical TranscriptRecord + writer/reader. Meta
  line at top-of-file carries source/model/task/mode/repeat.
- src/replay.ts — pure parser that rebuilds SessionSummary from a
  transcript's usage/cost fields. Tolerates old (pre-usage) transcripts.
- src/diff.ts — alignment by turn number, Levenshtein similarity for
  text divergence, tool-name + tool-args comparison, stdout table +
  markdown renderers.

Format bump (backward-compatible): transcripts now persist usage, cost,
model, prefixHash (Reasonix only), and toolArgs. All fields optional
on read — v0.1 transcripts still parse and render (cost/cache shown
as n/a). Format version lives in the _meta line at the top.

Bench runner: new --transcripts-dir <path> flag. Each (task, mode,
repeat) writes <taskId>.<mode>.r<n>.jsonl so bench runs produce
replay/diff-ready receipts, not just aggregate numbers.

Why this closes the loop on v0.1:
The τ-bench-lite report claims "baseline 43.9% / reasonix 94.3% cache
hit" — but a reader had to trust our aggregate. Now a reader can run
reasonix diff on two transcripts from the same task and see, byte by
byte, that reasonix's prefixHash stayed stable while baseline's
churned — and that the cache/cost delta is mechanically attributable
to log stability, not to a different prompt.

Tests: +16 (transcript 3, replay 3, diff 10). Suite now 159 green.
Also threads toolArgs through LoopEvent so chat's --transcript now
persists *what* the model sent to each tool, not just the result.

What's explicitly deferred to a later release:
- Full Ink TUI for replay (j/k scrubbing, search). Current replay
  command is stdout-only.
- Split-pane diff TUI. Current diff command is stdout + markdown.
- MCP client (v0.3). Replay/diff infra is prerequisite: without it we
  couldn't demonstrate why a Cache-First MCP client matters.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 06:11:15 -07:00
esengineandClaude Opus 4.7 7fed18fc47 feat(bench): τ-bench-lite harness + v0.1 validation data
Ships the first reproducible evidence for Pillar 1 (Cache-First Loop):
8 retail-flavored multi-turn tool-use tasks run under both a naive
cache-hostile baseline and Reasonix, sharing the same client/tools/prompt
so the only variable is prefix stability.

Measured on live DeepSeek API (16 runs):
- cache hit: 43.9% → 94.3% (+50.3pp)
- cost/task: \$0.002783 → \$0.001621 (−42%)
- pass rate: 100% → 88% (1 reasonix failure, predicate too strict — see report.md)
- vs Claude Sonnet 4.6 estimated: ~96% cheaper at same token counts

New harness:
- types.ts / db.ts — task schema + per-run DB isolation
- tasks.ts — 8 seed tasks, deterministic DB-predicate pass/fail (no LLM judge)
- user-sim.ts — V3 user simulator at T=0.1
- baseline.ts — deliberately cache-hostile (fresh timestamp + shuffled tools each turn)
- runner.ts — orchestrator with --task / --mode / --repeats / --dry / --verbose
- report.ts — results.json → report.md with explicit scope caveats
- tests/benchmarks.test.ts — 8 harness smoke tests that run without an API key

Also fixes a Windows-only entrypoint bug in the original runner scaffold
(import.meta.url vs argv[1] path encoding mismatch prevented main() from
ever running on Windows).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
2026-04-21 05:36:39 -07:00
esengine cc53474d56 feat(cli): session-by-default + /forget command
User feedback: 'requiring --session <name> to enable persistence is
too much friction. Most chat tools just remember by default.'

Correct. Now:
- Default behavior: every chat is auto-saved to a session named 'default'
  and resumed on next launch. Zero friction — the natural ChatGPT-like UX.
- --session <name> picks a different named session.
- --no-session opts out entirely (ephemeral chat).
- App shows a one-time banner on startup describing the current session
  state (resumed N msgs / new / ephemeral) so users know what's happening.
- New /forget command deletes the current session file from disk.
- /sessions now marks the current session with ▸.
- Command strip and /help updated.

Library API unchanged: CacheFirstLoop({ session }) is still opt-in.
Only the CLI is opinionated.

Tests: 3 new tests covering /help mentions sessions, /sessions hint,
/forget on session-less loop. Total: 135 (was 132).
2026-04-21 04:24:14 -07:00
esengine 359bd6bba3 feat(session): named session persistence — resume prior chats
New capability: every turn's message log is appended to
~/.reasonix/sessions/<name>.jsonl, and pre-loaded back into the loop's
AppendOnlyLog the next time the CLI is launched with --session <name>.

Rationale: the natural extension of Pillar 1's append-only-log invariant.
If the prefix is immutable AND the log is append-only, the only missing
piece for across-run continuity is writing the log to disk. DeepSeek's
prefix cache likely still hits on resume since the serialized messages
are byte-identical to what they were when originally appended.

Code:
- src/session.ts (new): loadSessionMessages / appendSessionMessage /
  listSessions / deleteSession / sanitizeName / sessionPath /
  sessionsDir. Name sanitization keeps [\w-] and CJK, replaces
  everything else with underscore, 64-char cap. 0600 permissions on
  Unix; chmod no-ops on Windows. Malformed JSONL lines are skipped
  rather than crashing.
- src/loop.ts: new  option on CacheFirstLoopOptions.
  When set, the constructor pre-loads prior messages into the log and
  every subsequent append is mirrored to disk via a small
  appendAndPersist helper. Exposes  for the TUI.
- src/cli/index.ts: --session <name> flag on chat.
- src/cli/ui/App.tsx: shows a one-line resume banner on startup when
  the loop reports pre-loaded messages.
- src/cli/ui/slash.ts: /sessions command lists saved sessions with
  message count + size + mtime, sorted by most recent.
- src/index.ts: exports the session API for library users.

Tests (tests/session.test.ts): 10 tests covering name sanitization,
round-trip append/load, malformed-line tolerance, listing + sort order,
delete. Homedir is stubbed to a tmp dir per-test for isolation.

Total tests: 132 (was 122). Bumps to 0.0.6.
2026-04-21 04:19:56 -07:00
esengine ff91ff91f0 fix(markdown): belt-and-braces LaTeX stripping + regression tests
User reported `\frac{总路程}{总时间}` leaked through as raw LaTeX in a
branched R1 answer even though most other \dfrac{} calls rendered fine
in the same output. Unable to reproduce from the paste alone, so
hardening the stripMath layer against the whole category:

- Frac regex now tolerates whitespace around braces and one level of
  nested braces inside the numerator/denominator (\frac{\sqrt{2}}{3}).
- All braced commands now trim whitespace from their captured groups
  so \frac{ a }{ b } → (a)/(b), not ( a )/( b ).
- New catch-all fallback at the end of stripMath: any surviving
  \command, \command{x}, or \command{x}{y} is stripped. Users will
  now NEVER see a raw backslash even if the model invents a new macro.

Also exported stripMath from markdown.tsx for direct unit testing.

12 new tests in tests/markdown.test.ts:
- \frac / \dfrac / \tfrac equivalence
- The exact Chinese \frac{总路程}{总时间} case
- Nested \sqrt{} inside \frac{}
- Whitespace tolerance inside braces
- Arrow & bare-command stripping
- Subscript/superscript Unicode mapping
- \boxed, \sqrt formatting
- Full user-reported line sanity check (no backslashes survive)

Total tests: 122 (was 110). Still v0.0.5.
2026-04-21 04:13:56 -07:00
esengine c72b429d82 fix(markdown): cover \dfrac/\tfrac/\binom + Unicode subscripts/superscripts
Real-world test surfaced:

  总时间:t_1 + t_2 = \dfrac{d}{30} + \dfrac{d}{60}

\dfrac wasn't in stripMath's regex list (only \frac was) so it leaked
through as raw backslashes. Same for \tfrac. Subscripts like t_1, t_2
were also rendering literal '_1' since stripMath did nothing for them.

Fix:
- Fractions: \frac, \dfrac, \tfrac all collapse to (num)/(den).
- New: \binom{n}{k} → C(n,k)
- New: \overline{x} → x̄, \hat{x} → x̂, \vec{x} → →x
- Subscripts/superscripts get translated to Unicode characters when the
  content is a single digit/sign (or a brace-wrapped run): t_1 → t₁,
  x^2 → x², a_{ij} → a_ij (groups stay readable). Falls back to
  pass-through for non-mappable characters.

Existing 110 tests still pass; this is purely additive coverage in
the markdown rendering layer.
2026-04-21 04:07:29 -07:00
esengine 989d8b33d8 style: biome formatter cleanup 2026-04-21 04:03:41 -07:00
esengine c40afe5caf fix(branch): show progress state + elapsed timer from second 0
Real-world test of --branch 3 revealed the user stared at 'waiting for
first token...' for the full 30-60s it took the first sample to
complete, then saw everything dump at once. Root cause: branch_start
set the streaming slot WITHOUT branchProgress, so the UI fell through
to the non-branching preview that has no relevant info to show.

Two fixes:

1. branch_start now carries an initial branchProgress with completed=0,
   total=budget. The TUI detects completed=0 and renders a distinct
   'launching N parallel samples...' view explaining what's happening
   and the typical wait time, instead of the misleading
   'waiting for first token...'.

2. New <Elapsed /> component shows a live mm:ss counter next to the
   branch header. Updates once a second via useEffect + setInterval.
   Visible immediately so the user has proof the process is alive
   even before the first sample completes. Also added to the
   non-branching streaming view for symmetry.

README: new 'Why not just use LangChain?' table making the default-
preset value proposition explicit — the five always-on advantages
that hold even without harvest/branching.
2026-04-21 04:03:07 -07:00
esengine 30ebd1353a feat(tui): /preset one-tap configs + persistent command strip
Two usability issues from real-world feedback:

1. 'Default should be reasoner+harvest+branch?' — no, but the jump from
   default-fast to everything-on requires remembering 3 commands.
   Fix: /preset <fast|smart|max> flips the full config in one keystroke.
     - fast:  deepseek-chat (default, cheapest, ~1 cent / 100 turns)
     - smart: reasoner + harvest              (~10x cost)
     - max:   reasoner + harvest + branch3    (~30x cost, slowest)
   Reasoning for keeping 'fast' as default: a new user's $1 free credit
   on the 'max' preset only lasts ~150 turns; on 'fast' it lasts ~50k.
   First impressions decide star conversion.

2. 'Users have to blind-type slash commands to discover them' — correct.
   Fix: a CommandStrip component renders a dim hint line right below the
   PromptInput showing all top-level commands. Always visible, no need
   to remember or run /help:

     /help · /preset <fast|smart|max> · /model · /harvest · /branch · /clear · /exit

   Placeholder also updated from the wordy '"type a message, or /exit"'
   to the cleaner 'type a message, or /command'.

Tests:
- /preset fast/smart/max round-trips verified through loop config.
- /preset with bad arg returns usage hint.
- /help mentions the presets block.

README updated to lead with the /preset idea rather than individual
knob commands.

Total tests: 110 (was 105). Still v0.0.5.
2026-04-21 03:58:58 -07:00
esengine ffc872ae12 fix(branch): live progress feedback + correct cost aggregation across all samples
Two issues from the first real branched-turn test:

1. UI looked frozen for 60-90s, then everything dumped at once.
   Cause: branching disables streaming because all N samples must
   complete before selection. The TUI sat on an empty 'streaming'
   spinner with no signal that work was happening.

2. Cost panel under-reported by ~3x.
   Cause: stats.record() was called with the chosen sample's usage
   only. We paid for all N samples but only counted one.

Fix 1 — progress events:
- BranchOptions gains an onSampleDone callback. runBranches fires it
  as each parallel call settles (Promise.all order, but resolved as
  each finishes).
- Loop wraps that callback in a queue/waiter pair and yields a new
  branch_progress event in resolution order with completed/total +
  the just-finished sample's index/temperature/uncertainty count.
- TUI's StreamingAssistant detects branchProgress and renders a
  live-updating two-line progress block:
    🔀 branching 2/3 (66%)
      latest #1 T=0.5 u=2  · waiting for other samples…
- App.tsx dispatches the progress into the streaming slot.

Fix 2 — usage aggregation:
- New aggregateBranchUsage(samples) sums prompt/completion/cache
  tokens across all samples.
- Loop builds a single Usage from the aggregate and records it via
  stats.record(). Result: cost panel now reflects what was actually
  billed. (Harvest-call tokens still not tracked — rounding error
  next to the R1 calls.)

Tests:
- onSampleDone fires N times.
- aggregateBranchUsage sums correctly across 3 samples.

Total tests: 105 (was 103). Still v0.0.5.
2026-04-21 03:55:50 -07:00
esengine c04e71ee9b feat(cli): slash commands — toggle model/harvest/branch inside the TUI
Flag soup was hurting discoverability. Normal users run `npx reasonix
chat` and never find out harvest or branching exist. Slash commands
put every feature one keystroke away inside the chat.

Commands (see /help):
  /help                list commands
  /status              model/harvest/branch/stream state
  /model <id>          switch models mid-session
  /harvest [on|off]    toggle Pillar 2 (no arg = toggle)
  /branch <N|off>      set N parallel samples; N=2..8; auto-enables harvest
  /clear               clear history view (loop log is kept)
  /exit                quit

Implementation:
- src/loop.ts: CacheFirstLoop fields that made sense to change
  mid-session (model, stream, harvestEnabled, harvestOptions,
  branchEnabled, branchOptions) are now mutable. New
  `configure(opts: ReconfigurableOptions)` method applies the same
  branching/harvest/stream invariants the constructor does.
- src/cli/ui/slash.ts: parseSlash + handleSlash. Unknown commands fall
  through with a hint to /help. Branch budget capped at 8.
- src/cli/ui/App.tsx: slash commands handled before normal submit;
  messages rendered as info rows.
- src/cli/ui/StatsPanel.tsx: shows live `· harvest · branch3` badges
  next to the model, plus a '· type /help' hint in the corner.
- tests/slash.test.ts: 14 tests covering parse, all commands, toggle
  behavior, invalid args, unknown commands, branch cap.

CLI flags kept for CI/scripting; slash commands are the recommended UX
for interactive use. README updated to lead with the slash-command flow.

Total tests: 103 (was 89).
2026-04-21 03:47:31 -07:00
esengine 2169ad233f feat(consistency): self-consistency branching + major README refresh
New feature: `reasonix chat --branch N` (or `{ branch: N }` in code) fans
the turn out into N parallel samples at spread temperatures, harvests each
one's reasoning, and selects the sample with the fewest flagged
uncertainties. Ties broken by answer length (Occam).

Unique value: because DeepSeek is ~20× cheaper than Claude, running N=3 R1
samples is still cheaper than a single Claude call, while the
confidence-weighted selection tends to beat single-sample answers on
multi-step reasoning.

Code:
- src/consistency.ts: runBranches + defaultSelector + BranchOptions /
  BranchSample / BranchResult / BranchSelector types. Spreads temperatures
  uniformly across [0, 1] by default; custom temperatures array supported.
- src/loop.ts: new branch option on CacheFirstLoopOptions (number or
  BranchOptions). Enabling it forces non-streaming and auto-enables
  harvest (the selector scores on uncertainties). Emits branch_start /
  branch_done events plus branch summary on assistant_final.
- src/cli/index.ts: --branch <n> flag.
- src/cli/ui/EventLog.tsx: new BranchBlock renders
  '🔀 branched N samples → picked #K' with per-sample temperature/uncertainty.
- src/cli/ui/App.tsx + commands/chat.tsx: branch option plumbed through.
- src/index.ts: exports runBranches + types.

Tests:
- tests/consistency.test.ts: 6 tests — selector tie-breakers, budget=1
  no-op, N=3 parallel with varied temps, uncertainty-based selection,
  custom temperatures override.

README rewritten:
- Badges (npm version, CI, license, downloads, node).
- Feature table covering Cache / Harvest / Branch / Repair / Retry / TUI.
- Validated numbers table with 3 real measurements.
- CLI + library usage examples with --harvest / --branch.
- Clean install / dev / non-goals sections.

docs/ARCHITECTURE.md: new Branch-and-Select section between Pillar 2 and
Pillar 3; roadmap updated.

Total tests: 89 (was 83). Bumps to 0.0.5.
2026-04-21 03:42:48 -07:00
esengine 8c426595fa feat(pillar3): auto-flatten deep/wide tool schemas in ToolRegistry
Pillar 3's flatten module existed but was never called — registered
tools were sent to the model with their original schemas, even when
DeepSeek V3/R1 are known to drop arguments on schemas with >10 leaf
params or >2 levels of nesting.

Now ToolRegistry analyzes every registered schema:
- analyzeSchema() decides if flattening is warranted.
- If so, the registry stores both the original (for nestArguments) and
  the flat (for the API spec) form.
- specs() emits the flat schema to the model.
- dispatch() re-nests dot-notation args back to the original shape
  before invoking the user's fn — but only when the incoming args
  actually use dot keys, so nested args from a model that ignored the
  flat spec still work (defensive).

API:
- new ToolRegistry({ autoFlatten: false }) opts out.
- reg.wasFlattened(name) introspects the decision.

5 new tests in tests/tools.test.ts covering: shallow schemas left alone,
deep schemas flattened + re-nested round-trip, wide-schema (>10 params)
detection, defensive nested-args pass-through, opt-out.

Total tests: 83 (was 78). Bumps to 0.0.4 — Pillar 3 now ships all four
submodules wired end-to-end.
2026-04-21 03:36:02 -07:00
esengine 2e3c479402 feat(pillar2): R1 Thought Harvesting MVP (opt-in)
Takes reasoning_content from deepseek-reasoner and runs a cheap V3 call in
JSON mode to extract a typed plan state:

    { subgoals, hypotheses, uncertainties, rejectedPaths }

Enable with `reasonix chat --harvest` or `new CacheFirstLoop({ harvest: true })`.
Default is off — harvest adds a non-zero V3 call per turn (~$0.0001).

Implementation:
- src/harvest.ts: real harvest() replacing v0.0.1 stub. Uses DeepSeek JSON
  mode (response_format: { type: json_object }). Fails closed — malformed
  JSON / API errors / short reasoning all return emptyPlanState() so the
  main turn is never aborted.
- src/types.ts + src/client.ts: new responseFormat option threaded through
  to the API payload.
- src/loop.ts: `harvest?: boolean | HarvestOptions` in CacheFirstLoopOptions.
  When enabled and reasoning present, the plan state is computed after the
  model call and attached to the assistant_final event.
- src/cli/index.ts + src/cli/commands/chat.tsx + src/cli/ui/App.tsx: CLI
  --harvest flag plumbed through to the loop constructor.
- src/cli/ui/EventLog.tsx: new PlanStateBlock renders the harvested state
  as dim magenta `‹ subgoals (N): a · b · c` lines above the answer.

Tests:
- tests/harvest.test.ts: 12 tests covering empty inputs, JSON parsing,
  prose-wrapped JSON extraction, malformed JSON, API errors, array caps,
  item-length trimming, snake_case fallback, non-string filtering.

Total tests: 78 (was 66). Bumps roadmap: Pillar 2 no longer deferred to v0.2.
2026-04-21 03:22:43 -07:00
esengine c4dd646642 fix(tui): keep TextInput mounted across turns to retain stdin raw mode
After the first turn finished, typing was silently ignored: the prompt
box rendered fine but keystrokes didn't register. Cause: PromptInput
conditionally rendered TextInput based on the `disabled` flag. Mount/
unmount cycles dropped Ink's stdin raw-mode claim on some terminals, so
the freshly-mounted input wasn't receiving keystrokes.

Fix: keep TextInput mounted at all times; toggle its `focus` prop
instead of unmounting. useInput inside TextInput no-ops when focus=false,
but the raw-mode claim is preserved, so keystrokes land cleanly once we
re-enable focus.

Bumps to 0.0.3 for re-publish.
2026-04-21 03:17:31 -07:00
esengine c0bc69e879 feat(cli): first-run setup prompts for API key, saves to ~/.reasonix/config.json
Previously, the CLI errored out on first run with 'Copy .env.example to
.env and fill it in' — a lousy welcome for anyone who did `npx reasonix
chat` expecting it to just work.

Now:
- chat command renders a Setup screen with masked-input TextInput if no
  key is found. Key validated by format (sk- prefix + 16+ chars), saved
  to ~/.reasonix/config.json with 0600 permissions.
- run command falls back to readline.question for the same flow when
  stdout is a TTY; non-TTY still errors with a hint.
- Library users unaffected: DeepSeekClient still only reads
  DEEPSEEK_API_KEY from env; the config file is a CLI convenience.

New module src/config.ts with readConfig / writeConfig / loadApiKey /
saveApiKey / isPlausibleKey / redactKey — all exported from the package
root so library users can opt in to the same storage.

9 new tests in tests/config.test.ts covering env-var precedence, file
fallback, format validation, and redaction. Total tests: 66.

Bumps version to 0.0.2 for re-publish.
2026-04-21 03:12:11 -07:00
esengine 437e5cc41f fix(pkg): drop leading ./ from bin path so npm keeps the entry
Without this, `npm publish` silently strips the bin field, which would
leave `npx reasonix` broken for installed users even if the tarball
otherwise looks correct. Caught via `npm pkg fix` on the first publish
attempt.
2026-04-21 03:02:56 -07:00
esengine f1b740596c feat(client): exponential-backoff retry for transient API failures
- src/retry.ts: new fetchWithRetry wrapper. Retries on 408/429/500/502/503/504
  and network errors (fetch throws). Does NOT retry on 4xx auth/bad-request
  errors, aborts, or mid-stream body read failures (retrying would re-bill
  and desync context). Honors Retry-After header, capped by maxBackoffMs.
  Exponential backoff with 75-125% jitter to spread concurrent clients out.
- src/client.ts: DeepSeekClient.chat and .stream now go through fetchWithRetry.
  New { retry } option on DeepSeekClientOptions; pass { maxAttempts: 1 } to
  disable.
- tests/retry.test.ts: 11 tests covering retry/no-retry statuses, network
  errors, maxAttempts exhaustion, abort handling, Retry-After, onRetry.
- tests/loop.test.ts: updated the failure-path test to use status 401
  (non-retryable) so it completes instantly.

Total tests: 57 (was 46). This closes the #1 production risk before launch:
a single 503 during DeepSeek peak hours no longer kills the chat session.
2026-04-21 02:15:50 -07:00
esengine d626b639fa style(tui): add blank line between reasoning preview and answer body 2026-04-21 02:10:54 -07:00
esengine f413c4dc6c style(tui): space out markdown blocks + compact reasoning preview
- Markdown: outer container now uses gap={1}, adding a blank line
  between each block-level element (paragraph / list / heading / code /
  hr / inline display equation after LaTeX strip). Previously sections
  like '步骤:' flowed into the list without visible separation.
- ReasoningBlock: cap reduced from 500 → 220 chars, rendered as a
  single dim italic line ('↳ thinking: ...') instead of a 19-line
  bordered block. Full reasoning still lands in --transcript.
2026-04-21 02:06:54 -07:00
esengine 2199a21c4d fix(tui): eliminate streaming/final duplicate + broader LaTeX strip
Two follow-up issues from the R1 test:

1. Streaming view duplicated in scrollback
   - Cause: during long streams, the dynamic render area grew past the
     terminal viewport. When finalize committed the full Markdown-rendered
     version via <Static>, the terminal could not erase the scrolled-up
     frames of the previous streaming pass, so users saw both the raw
     streaming text AND the formatted final output stacked.
   - Fix: new <StreamingAssistant> view keeps the dynamic region to a
     fixed 1–3 line footprint — 'assistant (streaming · 245 + think 88
     chars)' with a single tail-preview line. No matter how long the
     stream gets, the dynamic area stays in-viewport and Ink can clear
     it cleanly on finalize.

2. LaTeX commands leaking through (\implies, \to, \quad, etc.)
   - Fix: extended stripMath in markdown.tsx with the full set of
     common commands — arrows (\to, \implies, \rightarrow), relations
     (\leq, \geq, \neq, \approx), set membership (\in, \notin),
     greek letters, operators (\cdot, \times, \div, \pm), spacing
     (\quad, \qquad, \,, \;, \!), and line breaks (\). Also
     collapses runs of whitespace that the strip can leave behind.
2026-04-21 02:03:27 -07:00
esengine b7f29c384a fix(tui): eliminate flicker + render Markdown properly
Two UX issues surfaced during real R1 usage:

1. UI flicker on long streaming outputs
   - Cause: every DeepSeek token triggered a React state update, which
     re-rendered the entire App subtree (StatsPanel + all events + input).
     On a 1320-token answer that's ~1300 full-screen rewrites in seconds.
   - Fix: split state into 'historical' (completed events) and 'streaming'
     (current assistant turn). Historical goes inside Ink's <Static> which
     renders each item exactly once — zero redraw for old messages.
     Streaming deltas are buffered in refs and flushed at 16Hz (60ms
     interval) instead of per-token. EventRow is React.memo'd for good
     measure.

2. Markdown / LaTeX rendered as raw text
   - Cause: assistant text was dropped into <Text>{text}</Text> so users
     saw literal '\(', '\)', '**bold**', '# header', etc.
   - Fix: new src/cli/ui/markdown.tsx — line-based MD parser producing
     Ink elements. Handles ATX headings, ordered / unordered lists,
     fenced code blocks, inline bold / italic / code, and strips the
     LaTeX scaffolding (\(, \), \[, \], \boxed{...}, \cdot, etc.)
     that terminals can't render anyway. Markdown is applied on finalize
     only — during stream we keep raw text for speed.
2026-04-21 01:59:09 -07:00
esengine 0f0a44a00c fix(tui): surface per-turn stats and R1 reasoning in the chat UI
- Bug: assistant_final dispatched mark_assistant_done which only set
  streaming=false — ev.stats and ev.repair were dropped on the floor, so
  the '↳ cache X% · tokens … · $…' line never rendered.
- Missing feature: reasoning_content streamed from deepseek-reasoner was
  captured by the loop but never shown to the user.

Changes:
- EventLog: DisplayEvent gains a reasoning field; assistant row renders it
  in a dim italic 'thinking ›' block above the final answer (capped at 500
  chars preview).
- App.tsx: new reducer actions append_reasoning and finalize_assistant
  (replacing mark_assistant_done); reasoning deltas are appended as they
  stream; stats + repair note are attached atomically on assistant_final.

After this, Phase C (deepseek-reasoner) will show R1's thinking live above
the answer, and every assistant turn shows the per-turn cache hit stats.
2026-04-21 01:44:01 -07:00
esengine 214dc13a38 feat(validation): bake measured DeepSeek API numbers into README + loop integration tests
- README: replace projected ranges with measured numbers from real API runs
  - 5-turn Chinese chat: 85.2% cache hit, 93.9% savings vs Claude
  - 2-turn calculator tool-use: 94.9% cache hit, 95.8% savings vs Claude
- tests/loop.test.ts: 5 integration tests for CacheFirstLoop using fake fetch
  (single-turn, telemetry recording, tool dispatch, prefix stability, 5xx error)
- Total tests now 46 (was 41)
2026-04-21 01:42:34 -07:00
esengine 785f4d65c8 chore: ignore *.jsonl transcripts (accidental session.jsonl leak) 2026-04-21 01:34:21 -07:00
esengine 05298c0faf fix: load .env in examples and benchmark entrypoints
loadDotenv was only called inside CLI commands, so any direct script import
(examples/*, benchmarks/*) hit 'DEEPSEEK_API_KEY is not set' even when the
user had a valid .env file. Promote env.ts from cli/ to src/, export loadDotenv
from the library index, and call it at the top of every script entrypoint.
2026-04-21 01:33:47 -07:00
esengine cc3d2bfcb1 chore: satisfy biome lint (rename shadowed globals, add missing hook dep) 2026-04-21 01:28:42 -07:00
esengine e9e7ccc622 Initial commit: Reasonix v0.0.1 — DeepSeek-native agent framework
Three-pillar design optimized for DeepSeek's economic and behavioral profile:
  - Pillar 1: Cache-First Loop (immutable prefix + append-only log + volatile scratch)
  - Pillar 2: R1 Thought Harvesting (stub surface, full impl in v0.2)
  - Pillar 3: Tool-Call Repair (scavenge + truncation + storm + flatten — complete)

Stack: TypeScript / Node 20+ / ESM. Ink + React TUI with live cache-hit and
cost panel. Streaming via SSE. Vitest (41 passing tests), tsup, biome.
2026-04-21 01:25:12 -07:00
481 changed files with 23602 additions and 55562 deletions

No files matched your search

+4
View File
@@ -0,0 +1,4 @@
DEEPSEEK_API_KEY=sk-your-key-here
DEEPSEEK_BASE_URL=https://api.deepseek.com
REASONIX_LOG_LEVEL=INFO
REASONIX_TRANSCRIPT_DIR=./transcripts
-11
View File
@@ -1,11 +0,0 @@
.gitattributes text eol=lf
.gitignore text eol=lf
*.json text eol=lf
*.md text eol=lf
*.mjs text eol=lf
*.sh text eol=lf
*.sql text eol=lf
*.toml text eol=lf
*.ts text eol=lf
*.yaml text eol=lf
*.yml text eol=lf
+23
View File
@@ -0,0 +1,23 @@
---
name: Bug report
about: Something is broken
labels: bug
---
**What happened**
A clear and concise description.
**Expected**
What you expected to happen.
**Reproduction**
Steps or minimal code that reproduces it.
**Environment**
- Reasonix version:
- Node version:
- OS:
- DeepSeek model (e.g. `deepseek-chat`, `deepseek-reasoner`):
**Logs / transcript**
If using the CLI, attach the relevant chunk of `--transcript`.
+49
View File
@@ -0,0 +1,49 @@
name: CI
on:
push:
branches: [main]
pull_request:
branches: [main]
jobs:
build:
name: build (node ${{ matrix.node }})
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
node: ["20", "22"]
steps:
- uses: actions/checkout@v4
- name: Setup Node ${{ matrix.node }}
uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node }}
cache: npm
- name: Install dependencies
run: npm ci
- name: Lint (biome)
run: npm run lint
- name: Typecheck
run: npm run typecheck
- name: Test (vitest)
run: npm test
- name: Build (tsup)
run: npm run build
# Smoke-test the bench harnesses themselves. --dry skips all LLM
# calls, so this catches wiring regressions (task factories, CLI
# parsing, file IO, checker determinism) without needing a
# DEEPSEEK_API_KEY in CI.
- name: τ-bench harness dry-run
run: npx tsx benchmarks/tau-bench/runner.ts --dry --out /tmp/tau-dry.json
- name: Harvest-bench harness dry-run
run: npx tsx benchmarks/harvest/runner.ts --dry --out /tmp/harvest-dry.json
@@ -1,103 +0,0 @@
name: Deploy identity and remote services
on:
workflow_dispatch:
permissions:
contents: read
concurrency:
group: deploy-identity-and-remote-services
cancel-in-progress: false
jobs:
deploy-accounts:
if: github.ref == 'refs/heads/platform'
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
with:
version: 10
run_install: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: "24"
cache: pnpm
cache-dependency-path: workers/accounts/pnpm-lock.yaml
- name: Deploy
working-directory: workers/accounts
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: |
pnpm install --frozen-lockfile
pnpm test
pnpm typecheck
pnpm db:apply:remote
npx wrangler deploy
- name: Sync RESEND_API_KEY to the worker
working-directory: workers/accounts
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
run: |
if [ -n "$RESEND_API_KEY" ]; then
printf '%s' "$RESEND_API_KEY" | npx wrangler secret put RESEND_API_KEY
else
echo "RESEND_API_KEY not set in repo secrets; skipping (worker stays in stub email mode)."
fi
- name: Sync remote gateway secret
working-directory: workers/accounts
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
REMOTE_GATEWAY_TOKEN: ${{ secrets.REMOTE_GATEWAY_TOKEN }}
run: |
if [ -z "$REMOTE_GATEWAY_TOKEN" ]; then
echo "REMOTE_GATEWAY_TOKEN must be configured before enabling remote access."
exit 1
fi
printf '%s' "$REMOTE_GATEWAY_TOKEN" | npx wrangler secret put REMOTE_GATEWAY_TOKEN
- name: Sync feedback account secret
working-directory: workers/accounts
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
FEEDBACK_ACCOUNT_SECRET: ${{ secrets.FEEDBACK_ACCOUNT_SECRET }}
run: |
if [ -n "$FEEDBACK_ACCOUNT_SECRET" ]; then
printf '%s' "$FEEDBACK_ACCOUNT_SECRET" | npx wrangler secret put FEEDBACK_ACCOUNT_SECRET
else
echo "FEEDBACK_ACCOUNT_SECRET not set in repo secrets; feedback account linking stays unavailable."
fi
deploy-remote-gateway:
needs: deploy-accounts
if: github.ref == 'refs/heads/platform'
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: "22"
cache: npm
cache-dependency-path: workers/remote-gateway/package-lock.json
- name: Validate gateway secret
env:
REMOTE_GATEWAY_TOKEN: ${{ secrets.REMOTE_GATEWAY_TOKEN }}
run: test -n "$REMOTE_GATEWAY_TOKEN"
- name: Validate and deploy
working-directory: workers/remote-gateway
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: |
npm ci
npm test
npm run typecheck
npx wrangler deploy
- name: Sync gateway secret
working-directory: workers/remote-gateway
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
REMOTE_GATEWAY_TOKEN: ${{ secrets.REMOTE_GATEWAY_TOKEN }}
run: printf '%s' "$REMOTE_GATEWAY_TOKEN" | npx wrangler secret put REMOTE_GATEWAY_TOKEN
-254
View File
@@ -1,254 +0,0 @@
name: Deploy crash worker
on:
workflow_dispatch:
inputs:
firebase_data_action:
description: Firebase crash history operation (migration actions do not deploy the Worker)
required: true
default: none
type: choice
options:
- none
- dry-run
- apply
- verify-only
confirmation:
description: Enter APPLY_FIREBASE_CRASH_DATA when selecting apply
required: false
default: ''
type: string
permissions:
contents: read
concurrency:
group: deploy-crash-worker
cancel-in-progress: false
jobs:
deploy:
if: github.ref == 'refs/heads/platform' && inputs.firebase_data_action == 'none'
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: '22'
cache: npm
cache-dependency-path: workers/crash-report/package-lock.json
- name: Validate
working-directory: workers/crash-report
run: |
npm ci
npm run typecheck
npm test
- name: Apply and verify diagnostics v2 D1 migration
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: npm run migrate:diagnostics-v2
- name: Apply and verify Firebase crash D1 migration
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: npm run migrate:firebase-crash
- name: Apply dashboard performance indexes
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: npm run migrate:dashboard-indexes
- name: Apply telemetry queue receipts schema
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: npx wrangler d1 execute reasonix-crash --remote --file=migrate-telemetry-queue.sql
- name: Apply and verify feedback D1 migration
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: |
npx wrangler d1 execute reasonix-crash --remote --file=migrate-feedback.sql
out=$(npx wrangler d1 execute reasonix-crash --remote --json --command "SELECT name FROM sqlite_master WHERE name IN ('feedback','feedback_quota','feedback_status_created','feedback_install_created','feedback_status_version','feedback_quota_day')")
for name in feedback feedback_quota feedback_status_created feedback_install_created feedback_status_version feedback_quota_day; do
printf '%s' "$out" | grep -q "\"$name\"" || { echo "missing $name after migration"; exit 1; }
done
- name: Apply and verify feedback triage D1 migration
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: |
npx wrangler d1 execute reasonix-crash --remote --file=migrate-feedback-triage.sql
names="feedback_releases feedback_releases_install feedback_replies feedback_blocks feedback_trust feedback_public_images feedback_config feedback_audit feedback_replies_receipt feedback_replies_unhandled feedback_blocks_expires feedback_public_images_receipt feedback_audit_at feedback_install_status_updated feedback_reply_attachments feedback_reply_attachments_receipt"
list=$(for n in $names; do printf "'%s'," "$n"; done | sed 's/,$//')
out=$(npx wrangler d1 execute reasonix-crash --remote --json --command "SELECT name FROM sqlite_master WHERE name IN ($list)")
for name in $names; do
printf '%s' "$out" | grep -q "\"$name\"" || { echo "missing $name after migration"; exit 1; }
done
echo "Trusted installs (hash prefix only) for review:"
npx wrangler d1 execute reasonix-crash --remote --json --command "SELECT substr(install_hash,1,8) AS prefix, created_at, expires_at FROM feedback_trust ORDER BY created_at LIMIT 200"
- name: Apply and verify feedback adoption D1 migration
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: |
npx wrangler d1 execute reasonix-crash --remote --file=migrate-feedback-adoptions.sql
names="feedback_adoptions feedback_adoptions_install feedback_adoptions_item feedback_level_state"
list=$(for n in $names; do printf "'%s'," "$n"; done | sed 's/,$//')
out=$(npx wrangler d1 execute reasonix-crash --remote --json --command "SELECT name FROM sqlite_master WHERE name IN ($list)")
for name in $names; do
printf '%s' "$out" | grep -q "\"$name\"" || { echo "missing $name after migration"; exit 1; }
done
- name: Apply and verify feedback account link D1 migration
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: |
npx wrangler d1 execute reasonix-crash --remote --file=migrate-feedback-account-links.sql
names="feedback_account_links feedback_account_links_account"
list=$(for n in $names; do printf "'%s'," "$n"; done | sed 's/,$//')
out=$(npx wrangler d1 execute reasonix-crash --remote --json --command "SELECT name FROM sqlite_master WHERE name IN ($list)")
for name in $names; do
printf '%s' "$out" | grep -q "\"$name\"" || { echo "missing $name after migration"; exit 1; }
done
- name: Apply and verify feedback contribution board D1 migration
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: |
npx wrangler d1 execute reasonix-crash --remote --file=migrate-feedback-board.sql
names="feedback_board_optin feedback_board_delisted"
list=$(for n in $names; do printf "'%s'," "$n"; done | sed 's/,$//')
out=$(npx wrangler d1 execute reasonix-crash --remote --json --command "SELECT name FROM sqlite_master WHERE name IN ($list)")
for name in $names; do
printf '%s' "$out" | grep -q "\"$name\"" || { echo "missing $name after migration"; exit 1; }
done
- name: Apply Studio telemetry schema
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: npx wrangler d1 execute reasonix-crash --remote --file=migrate-studio-telemetry.sql
- name: Apply isolated telemetry database schema
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: npm run migrate:telemetry-db
- name: Apply registry performance indexes and install rollup
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: npm run migrate:registry-indexes
- name: Sync Firebase crash secrets
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
FIREBASE_DATABASE_URL: ${{ secrets.FIREBASE_DATABASE_URL }}
FIREBASE_CLIENT_EMAIL: ${{ secrets.FIREBASE_CLIENT_EMAIL }}
FIREBASE_PRIVATE_KEY: ${{ secrets.FIREBASE_PRIVATE_KEY }}
run: |
configured=0
[ -n "$FIREBASE_DATABASE_URL" ] && configured=$((configured + 1))
[ -n "$FIREBASE_CLIENT_EMAIL" ] && configured=$((configured + 1))
[ -n "$FIREBASE_PRIVATE_KEY" ] && configured=$((configured + 1))
if [ "$configured" -ne 0 ] && [ "$configured" -ne 3 ]; then
echo "Firebase crash secrets must be either all configured or all absent."
exit 1
fi
node -e 'process.stdout.write(JSON.stringify({FIREBASE_DATABASE_URL:process.env.FIREBASE_DATABASE_URL||null,FIREBASE_CLIENT_EMAIL:process.env.FIREBASE_CLIENT_EMAIL||null,FIREBASE_PRIVATE_KEY:process.env.FIREBASE_PRIVATE_KEY||null}))' | npx wrangler secret bulk
- name: Deploy
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: |
npx wrangler deploy
- name: Sync feedback secrets
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
FEEDBACK_TOKEN_SECRET: ${{ secrets.FEEDBACK_TOKEN_SECRET }}
FEEDBACK_ADMIN_TOKEN: ${{ secrets.FEEDBACK_ADMIN_TOKEN }}
OPS_EMIT_TOKEN: ${{ secrets.OPS_EMIT_TOKEN }}
FEEDBACK_ACCOUNT_SECRET: ${{ secrets.FEEDBACK_ACCOUNT_SECRET }}
run: |
node -e 'process.stdout.write(JSON.stringify({FEEDBACK_TOKEN_SECRET:process.env.FEEDBACK_TOKEN_SECRET||null,FEEDBACK_ADMIN_TOKEN:process.env.FEEDBACK_ADMIN_TOKEN||null,OPS_EMIT_TOKEN:process.env.OPS_EMIT_TOKEN||null,FEEDBACK_ACCOUNT_SECRET:process.env.FEEDBACK_ACCOUNT_SECRET||null}))' | npx wrangler secret bulk
# Mirrors the ALERT_WEBHOOK repo secret into the worker so the ingest
# sentinel can push alerts (see runIngestSentinel). Managed here so
# nobody needs local Cloudflare credentials; when the repo secret is
# unset the sentinel stays log-only.
- name: Sync alert webhook secret
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
ALERT_WEBHOOK: ${{ secrets.ALERT_WEBHOOK }}
run: |
if [ -z "$ALERT_WEBHOOK" ]; then
echo "ALERT_WEBHOOK repo secret not set; removing any existing worker secret."
printf '{"ALERT_WEBHOOK":null}' | npx wrangler secret bulk
exit 0
fi
printf '%s' "$ALERT_WEBHOOK" | npx wrangler secret put ALERT_WEBHOOK
migrate-firebase-data:
if: github.event_name == 'workflow_dispatch' && inputs.firebase_data_action != 'none'
runs-on: ubuntu-latest
environment: canary
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- name: Require the protected production branch
run: |
if [ "$GITHUB_REF" != "refs/heads/platform" ]; then
echo "Firebase crash data migration must run from platform."
exit 1
fi
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: '22'
cache: npm
cache-dependency-path: workers/crash-report/package-lock.json
- name: Validate
working-directory: workers/crash-report
run: |
npm ci
npm run typecheck
npm test
- name: Run Firebase crash history operation
working-directory: workers/crash-report
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
FIREBASE_DATABASE_URL: ${{ secrets.FIREBASE_DATABASE_URL }}
FIREBASE_CLIENT_EMAIL: ${{ secrets.FIREBASE_CLIENT_EMAIL }}
FIREBASE_PRIVATE_KEY: ${{ secrets.FIREBASE_PRIVATE_KEY }}
FIREBASE_DATA_ACTION: ${{ inputs.firebase_data_action }}
FIREBASE_DATA_CONFIRMATION: ${{ inputs.confirmation }}
run: |
configured=0
[ -n "$CLOUDFLARE_API_TOKEN" ] && configured=$((configured + 1))
[ -n "$FIREBASE_DATABASE_URL" ] && configured=$((configured + 1))
[ -n "$FIREBASE_CLIENT_EMAIL" ] && configured=$((configured + 1))
[ -n "$FIREBASE_PRIVATE_KEY" ] && configured=$((configured + 1))
if [ "$configured" -ne 4 ]; then
echo "Cloudflare and Firebase migration secrets must all be configured."
exit 1
fi
case "$FIREBASE_DATA_ACTION" in
dry-run)
npm run migrate:firebase-data
;;
apply)
if [ "$FIREBASE_DATA_CONFIRMATION" != "APPLY_FIREBASE_CRASH_DATA" ]; then
echo "Apply requires the exact confirmation APPLY_FIREBASE_CRASH_DATA."
exit 1
fi
npm run migrate:firebase-data -- --apply
npm run migrate:firebase-data -- --verify-only
;;
verify-only)
npm run migrate:firebase-data -- --verify-only
;;
*)
echo "Unsupported Firebase crash data action."
exit 1
;;
esac
-38
View File
@@ -1,38 +0,0 @@
name: Deploy forum worker
on:
workflow_dispatch:
permissions:
contents: read
concurrency:
group: deploy-forum-worker
cancel-in-progress: false
jobs:
deploy:
if: github.ref == 'refs/heads/platform'
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
with:
version: 10
run_install: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: "24"
cache: pnpm
cache-dependency-path: workers/forum/pnpm-lock.yaml
- name: Deploy
working-directory: workers/forum
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: |
pnpm install --frozen-lockfile
pnpm test
pnpm typecheck
npx wrangler d1 execute reasonix-forum --remote --file=migrate-performance-indexes.sql
npx wrangler deploy
-44
View File
@@ -1,44 +0,0 @@
name: Deploy ops events relay
on:
workflow_dispatch:
permissions:
contents: read
concurrency:
group: deploy-ops-events
cancel-in-progress: false
jobs:
deploy:
if: github.ref == 'refs/heads/platform'
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: '22'
cache: npm
cache-dependency-path: workers/ops-events/package-lock.json
- name: Validate
working-directory: workers/ops-events
run: |
npm ci
npm run typecheck
npm test
- name: Deploy
working-directory: workers/ops-events
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
run: npx wrangler deploy
- name: Sync secrets
working-directory: workers/ops-events
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
GITHUB_WEBHOOK_SECRET: ${{ secrets.OPS_GITHUB_WEBHOOK_SECRET }}
OPS_LISTEN_TOKEN: ${{ secrets.OPS_LISTEN_TOKEN }}
OPS_EMIT_TOKEN: ${{ secrets.OPS_EMIT_TOKEN }}
run: |
node -e 'process.stdout.write(JSON.stringify({GITHUB_WEBHOOK_SECRET:process.env.GITHUB_WEBHOOK_SECRET||null,OPS_LISTEN_TOKEN:process.env.OPS_LISTEN_TOKEN||null,OPS_EMIT_TOKEN:process.env.OPS_EMIT_TOKEN||null}))' | npx wrangler secret bulk
-89
View File
@@ -1,89 +0,0 @@
name: Platform CI
on:
push:
branches: [platform]
pull_request:
branches: [platform]
permissions:
contents: read
concurrency:
group: platform-ci-${{ github.ref }}
cancel-in-progress: true
jobs:
accounts:
runs-on: ubuntu-latest
defaults:
run:
working-directory: workers/accounts
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
with:
version: 10
run_install: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: "24"
cache: pnpm
cache-dependency-path: workers/accounts/pnpm-lock.yaml
- run: pnpm install --frozen-lockfile
- run: pnpm test
- run: pnpm typecheck
forum:
runs-on: ubuntu-latest
defaults:
run:
working-directory: workers/forum
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
with:
version: 10
run_install: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: "24"
cache: pnpm
cache-dependency-path: workers/forum/pnpm-lock.yaml
- run: pnpm install --frozen-lockfile
- run: pnpm test
- run: pnpm typecheck
crash-report:
runs-on: ubuntu-latest
defaults:
run:
working-directory: workers/crash-report
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: "22"
cache: npm
cache-dependency-path: workers/crash-report/package-lock.json
- run: npm ci
- run: npm run typecheck
- run: npm test
- run: npx wrangler deploy --dry-run
remote-gateway:
runs-on: ubuntu-latest
defaults:
run:
working-directory: workers/remote-gateway
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: "22"
cache: npm
cache-dependency-path: workers/remote-gateway/package-lock.json
- run: npm ci
- run: npm test
- run: npm run typecheck
- run: npx wrangler deploy --dry-run
+18 -8
View File
@@ -1,16 +1,26 @@
node_modules/
.pnpm-store/
.npm/
.wrangler/
dist/
coverage/
.env
.env.*
*.local
**/.firebase-crash-migration-state.json
**/.firebase-crash-migration-state.json.tmp-*
.env.local
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
.pnpm-debug.log*
.DS_Store
Thumbs.db
.idea/
.vscode/
temp/
*.tsbuildinfo
transcripts/
sessions/
*.jsonl
!tests/**/*.jsonl
# Committed reference transcripts so readers can reasonix replay / diff
# the v0.1 bench results without an API key.
!benchmarks/tau-bench/transcripts/
!benchmarks/tau-bench/transcripts/*.jsonl
.turbo/
# Local-only Claude Code settings — personal overrides, never committed.
.claude/settings.local.json
-1
View File
@@ -1 +0,0 @@
24.20.0
-71
View File
@@ -1,71 +0,0 @@
# Reasonix platform capacity contract
This document defines what “ready for 10,000 concurrent users” means. It is an
acceptance contract, not a claim that the current production system has already
passed the target.
## Budget and design constraints
- Keep recurring infrastructure within CNY 200/month at the current traffic
level. Prefer Cloudflare managed primitives over idle servers.
- Never make synchronous telemetry storage part of a product launch path.
- Validate at the edge, durably enqueue, then persist asynchronously.
- A successful HTTP response must expose whether the event was `queued`,
`stored`, `sampled`, or `ignored` through
`x-reasonix-telemetry-result`. HTTP 202 alone is not proof of durable intake.
- Production load tests require owner approval. The repository probe refuses
remote and production targets unless they are unlocked separately.
## Service-level objectives
| Path | Workload | Pass condition |
| --- | --- | --- |
| Telemetry admission | 10,000 distributed requests over 60 seconds | >=99.9% `queued`, p95 <=500 ms, no 5xx |
| Telemetry drain | 10,000 queued events, batches up to 100 | backlog returns to zero within 15 minutes, no DLQ growth |
| Identity session read | 1,000 requests/second for 5 minutes | >=99.9% success, p95 <=300 ms |
| Registry public read | 1,000 requests/second for 5 minutes | >=99.9% success, p95 <=300 ms |
| Admin dashboards | 20 concurrent viewers for 10 minutes | >=99% success, p95 <=2 seconds |
“Distributed” matters because per-IP abuse protection intentionally rejects a
single machine long before the platform capacity limit. A staging environment
must mirror production bindings and use isolated databases, queues, and R2.
## Required evidence before calling the platform stable
1. A saved probe report with request count, latency percentiles, HTTP statuses,
durable/sample ratios, queue backlog peak, drain time, D1 query latency, and
DLQ delta.
2. A rollback or traffic-shed procedure exercised in staging.
3. Alerts for sustained 5xx, queue backlog age, DLQ growth, D1 overload, and
database size thresholds.
4. Seven consecutive days without unexplained ingestion gaps after a production
rollout.
## Current known gaps
- CLI and Studio telemetry use a verified isolated D1 read/write plane. The
legacy copy is retained temporarily as a rollback snapshot. Legacy desktop
telemetry is entering a dry-run-verified dual-write migration; diagnostic
denominators are queried separately so the final cutover does not require
cross-database SQL joins.
- The global telemetry budget is currently 1,000 events/minute. It protects the
database but cannot satisfy the 10,000/minute durable-intake target.
- Queue consumption is deliberately serialized for D1. A delivery is persisted
as one atomic D1 batch with two statements per valid envelope (at most 200
statements for the configured 100-message queue batch). Drain throughput must
be measured before raising the global intake budget.
- Read replication and the Sessions API have not yet been enabled for eligible
read-heavy paths.
## Local probe
Run the Worker locally, then run:
```sh
npm run capacity:telemetry -- --requests 1000 --concurrency 50
```
This probe fails when the p95 target or durable-intake ratio is missed. A local
run validates the admission contract and tool; it is not evidence for global
capacity. Remote runs need `--allow-remote`, and `*.reasonix.io` additionally
needs `--allow-production` after an approved test window.
+1122
View File
File diff suppressed because it is too large. Load diff
-6
View File
@@ -1,6 +0,0 @@
# Reasonix
The project's standing instructions live in `REASONIX.md` — one file, read by
both Reasonix and Claude Code. Add rules there, never here.
@REASONIX.md
+313 -21
View File
@@ -1,28 +1,320 @@
# Reasonix Platform
# Reasonix
This branch contains only the shared Reasonix service platform:
[![npm version](https://img.shields.io/npm/v/reasonix.svg)](https://www.npmjs.com/package/reasonix)
[![CI](https://github.com/esengine/reasonix/actions/workflows/ci.yml/badge.svg)](https://github.com/esengine/reasonix/actions/workflows/ci.yml)
[![license](https://img.shields.io/npm/l/reasonix.svg)](./LICENSE)
[![downloads](https://img.shields.io/npm/dm/reasonix.svg)](https://www.npmjs.com/package/reasonix)
[![node](https://img.shields.io/node/v/reasonix.svg)](./package.json)
- identity and account services;
- community and forum services;
- crash, diagnostics, telemetry, and registry services;
- Cloudflare Worker, D1, Queue, and R2 configuration;
- production deployment workflows for those services.
**A DeepSeek-native AI coding assistant in your terminal.** Ink TUI. MCP
first-class. No LangChain.
Product clients live on `main-v2` and `studio`. The public website lives on
`website`. Production deployments are accepted only from this branch.
```bash
npx reasonix
```
## Services
One command. First run walks you through a 30-second wizard (API key →
preset → pick MCP servers from a checklist); every run after that drops
straight into chat with your tools wired up. Inside the chat, type `/help`.
| Directory | Responsibility |
| --- | --- |
| `workers/accounts` | Shared identity, sessions, device grants, and email login |
| `workers/forum` | Community forum APIs |
| `workers/crash-report` | Diagnostics, telemetry, registry, and admin dashboards |
Why bother with yet another agent framework? Because every abstraction
here earns its weight against a DeepSeek-specific property — dirt-cheap
tokens, R1 reasoning traces, automatic prefix caching, JSON mode.
Generic wrappers treat DeepSeek as "OpenAI with a different base URL"
and leave these advantages on the table. Reasonix leans into them:
on the same τ-bench-lite workload,
[**94.4% cache hit, ~40% cheaper tokens, 100% pass rate**](#validated-numbers)
vs. a cache-hostile baseline.
Each service owns its package manager lockfile, tests, schema, migrations, and
Wrangler configuration. Run validation from the service directory before using
the matching manual production deployment workflow.
---
The accounts worker removes expired sessions, email tokens, and device grants
daily. Raw queued telemetry is retained in R2 for eight days; aggregated D1
rows follow the service-specific retention windows in the crash worker.
## What you get
| Feature | How it works | Opt in |
|---|---|---|
| **Setup wizard** | First run of `npx reasonix`: pick preset, multi-select MCP servers from a curated catalog, saved to config so the next run just launches chat | always on (first run) |
| **MCP (stdio + SSE)** | Multi-server bridge — every MCP tool inherits Cache-First + repair + context-safety automatically. `reasonix mcp list` shows the catalog | always on |
| **Cache-First Loop** | Immutable prefix + append-only log = prefix byte-stable across turns → DeepSeek's automatic prefix cache hits at 70–95% | always on |
| **Context safety net** | Tool results capped at 32k chars · oversized sessions auto-heal on load · `/compact` to shrink further · ctx gauge in the status bar · Esc to abort exploration and get a forced summary | always on |
| **R1 Thought Harvesting** | Parses `reasoning_content` into typed `{ subgoals, hypotheses, uncertainties, rejectedPaths }` via a cheap V3 call | `/preset smart` |
| **Self-Consistency Branching** | Runs N parallel samples at spread temperatures; picks the one with the fewest flagged uncertainties | `/preset max` / `/branch N` |
| **Tool-Call Repair** | Auto-flattens deep/wide schemas, scavenges tool calls leaked into `<think>`, repairs truncated JSON, breaks call-storms | always on |
| **Retry layer** | Exponential backoff + jitter on 408/429/500/502/503/504 and network errors. 4xx auth errors don't retry | always on |
| **Ink TUI** | Live cache-hit / cost / context panel. Streams R1 thinking to a compact preview. Renders Markdown (bold / lists / code / stripped LaTeX) | always on |
---
## Why not just use LangChain?
Even on the default `fast` preset (no harvest, no branching), Reasonix bakes
in five DeepSeek-specific defences that generic agent frameworks leave to you:
| | Reasonix default | generic frameworks |
|---|---|---|
| Prefix-stable loop (→ 85–95% cache hit) | ✅ | ❌ prompts rebuilt each turn |
| Auto-flatten deep tool schemas | ✅ | ❌ DeepSeek drops args |
| Retry with jittered backoff (429/503) | ✅ | ❌ custom callbacks |
| Scavenge tool calls leaked into `<think>` | ✅ | ❌ |
| Call-storm breaker on identical-arg repeats | ✅ | ❌ |
| Live cache-hit / cost / vs-Claude panel | ✅ | ❌ |
| First-run config prompt + Markdown TUI | ✅ | ❌ |
Harvest and self-consistency branching are bonuses on top. The everyday
win is that **a plain chat with Reasonix already pays for ~40% less tokens
than the same chat through a naive LangChain setup**, because the prefix
actually stays byte-stable.
## Validated numbers
**τ-bench-lite** — 8 multi-turn tool-use tasks × 3 repeats = 48 runs per
side. Same tools / same prompt / same client on both sides, sole variable
is prefix stability. Measured on live DeepSeek `deepseek-chat`:
| metric | baseline (cache-hostile) | Reasonix | delta |
|---|---:|---:|---:|
| runs | 24 | 24 | — |
| **cache hit** | 46.6% | **94.4%** | **+47.7pp** |
| cost / task | $0.002599 | $0.001579 | **−39% (×0.61)** |
| vs Claude Sonnet 4.6 (token-count estimate) | — | — | **~96% cheaper** |
| pass rate | 96% (23/24) | **100% (24/24)** | Reasonix held the guardrail on every run |
**Verify it yourself — no API key, zero cost:**
```bash
git clone https://github.com/esengine/reasonix.git && cd reasonix && npm install
npx reasonix replay benchmarks/tau-bench/transcripts/t01_address_happy.reasonix.r1.jsonl
npx reasonix diff \
benchmarks/tau-bench/transcripts/t01_address_happy.baseline.r1.jsonl \
benchmarks/tau-bench/transcripts/t01_address_happy.reasonix.r1.jsonl
```
The JSONL transcripts committed in `benchmarks/tau-bench/transcripts/`
carry per-turn `usage`, `cost`, and `prefixHash`. Reasonix's prefix hash
stays byte-stable across every model call; baseline's prefix churns on
every turn. The cache delta is *mechanically* attributable to log
stability, not to a different system prompt.
Full 48-run report: [`benchmarks/tau-bench/report.md`][r]. Reproduce
with your own API key: `npx tsx benchmarks/tau-bench/runner.ts --repeats 3`.
[r]: ./benchmarks/tau-bench/report.md
### MCP — works out of the box
Any [MCP](https://spec.modelcontextprotocol.io/) server's tools inherit
Cache-First + repair + context-safety automatically. The wizard (`npx
reasonix`) lets you multi-select from a curated catalog — no flags, no
JSON-by-hand. Three live reference runs:
| server | turns | tool calls | cache hit | cost | vs Claude |
|---|---:|---:|---:|---:|---:|
| bundled demo (`add` / `echo` / `get_time`) | 2 | 1 | **96.6%** (turn 2) | $0.000254 | −94.0% |
| official `@modelcontextprotocol/server-filesystem` | 5 | 4 | **96.7%** overall | $0.001235 | −97.0% |
| **both concurrently** (`demo_add` + `fs_write_file`) | 5 | 4 | **81.1%** | $0.001852 | −95.9% |
The third row is the ecosystem proof: two MCP servers running as
separate subprocesses, tools from both exercised in one conversation.
**One single prefix hash across all 5 turns** — byte-stability survives
concurrent MCP subprocesses.
Reproduce without an API key (replay the committed transcripts):
```bash
npx reasonix replay benchmarks/tau-bench/transcripts/mcp-demo.add.jsonl
npx reasonix replay benchmarks/tau-bench/transcripts/mcp-filesystem.jsonl
```
Supported transports: **stdio** (local `npx` or binary) and **HTTP+SSE**
(remote / hosted servers, MCP 2024-11-05 spec). Pass an `http(s)://`
URL to `--mcp` and Reasonix opens the SSE stream and POSTs JSON-RPC
to the endpoint the server advertises.
[mcp]: ./benchmarks/tau-bench/transcripts/mcp-demo.add.jsonl
---
## Usage
### One command
```bash
npx reasonix
```
First run: a wizard asks for your API key, lets you pick a preset
(fast / smart / max), then offers a multi-select checklist of MCP
servers — filesystem, memory, github, puppeteer, everything. Everything
is saved to `~/.reasonix/config.json`. Subsequent runs drop straight
into chat.
### Inside the chat
A status bar at the top shows cache hit %, cost, Claude-equivalent, and
the **context gauge** (`ctx 42k/131k (32%)` — yellow at 50%, red + a
`/compact` nudge at 80%). A command strip under the input lists the
slash commands:
```
/help full list + hints
/preset <fast|smart|max> one-tap bundles (model + harvest + branch)
/mcp list attached MCP servers and tools
/compact [cap] shrink oversized tool results in history
/sessions · /forget list / delete saved sessions
/setup reconfigure (exits and tells you to run `reasonix setup`)
/clear · /exit
```
**Esc while thinking** — abort the current exploration and force the
model to summarize what it already found. No more "model ran 24 tool
calls and gave up" — you get an answer every time.
Sessions live as JSONL under `~/.reasonix/sessions/<name>.jsonl` —
every message appended atomically, so killing the CLI never loses
context. Oversized tool results auto-heal on load, so poisoning a
session with one giant `read_file` doesn't brick your history.
### Code mode — `npx reasonix code`
A thin opinionated layer on top of chat: filesystem MCP bridged at
`cwd`, coding system prompt, reasoner preset, per-directory session.
The model proposes edits as **SEARCH/REPLACE blocks**:
```
src/foo.ts
<<<<<<< SEARCH
const x = 1;
=======
const x = 2;
>>>>>>> REPLACE
```
Reasonix parses them out of each turn, applies them to disk, reports
`✓ applied src/foo.ts` in the TUI. SEARCH must match byte-for-byte;
we never fuzzy-match (silent wrong edits are worse than loud
rejections). Run `git diff` to review, `git checkout .` to undo.
```bash
cd my-project
npx reasonix code # default: cwd, reasoner, per-dir session
npx reasonix code src/ # scope the filesystem sandbox tighter
npx reasonix code --no-session # ephemeral
```
First-run sandbox: because code mode uses the filesystem MCP from
`@modelcontextprotocol/server-filesystem`, the model can only read
and write inside the directory you pointed at. It literally can't
touch files above that root.
### Advanced — CLI subcommands and flags
```bash
npx reasonix setup # reconfigure any time
npx reasonix chat --session work # a different named session
npx reasonix chat --no-session # ephemeral — nothing persisted
npx reasonix run "ask anything" # one-shot, streams to stdout
npx reasonix stats session.jsonl # summarize a transcript
npx reasonix replay chat.jsonl # scrub a transcript + rebuild cost/cache
npx reasonix diff a.jsonl b.jsonl --md # compare two transcripts
npx reasonix mcp list # curated MCP server catalog
```
Power users can still bypass config and drive Reasonix with flags:
```bash
npx reasonix chat \
--preset max \
--mcp "filesystem=npx -y @modelcontextprotocol/server-filesystem /tmp/safe" \
--mcp "kb=https://mcp.example.com/sse" \
--transcript session.jsonl \
--no-config # ignore ~/.reasonix/config.json (for CI / reproducing issues)
```
### Library
```ts
import {
CacheFirstLoop,
DeepSeekClient,
ImmutablePrefix,
ToolRegistry,
} from "reasonix";
const client = new DeepSeekClient(); // reads DEEPSEEK_API_KEY from env
const tools = new ToolRegistry();
tools.register({
name: "add",
description: "Add two integers",
parameters: {
type: "object",
properties: { a: { type: "integer" }, b: { type: "integer" } },
required: ["a", "b"],
},
fn: ({ a, b }: { a: number; b: number }) => a + b,
});
const loop = new CacheFirstLoop({
client,
tools,
prefix: new ImmutablePrefix({
system: "You are a math helper.",
toolSpecs: tools.specs(),
}),
harvest: true,
branch: 3, // self-consistency budget
});
for await (const ev of loop.step("What is 17 + 25?")) {
if (ev.role === "assistant_final") console.log(ev.content);
}
console.log(loop.stats.summary());
```
### Configuration
The wizard handles everything on first run. If you'd rather use env vars
(CI, shared boxes, etc.):
```bash
export DEEPSEEK_API_KEY=sk-... # wins over ~/.reasonix/config.json
export DEEPSEEK_BASE_URL=https://... # optional alternate endpoint
```
Get a key (free credit on signup): <https://platform.deepseek.com/api_keys>
Re-run `npx reasonix setup` any time to add/remove MCP servers or switch
preset — your existing selections are pre-checked.
---
## Non-goals
- Multi-agent orchestration (use LangGraph).
- RAG / vector stores (use LlamaIndex or do it yourself).
- Multi-provider abstraction (use LiteLLM).
- Web UI / SaaS.
Reasonix does DeepSeek, deeply.
---
## Development
```bash
git clone https://github.com/esengine/reasonix.git
cd reasonix
npm install
npm run dev chat # run CLI from source via tsx
npm run build # tsup to dist/
npm test # vitest (279 tests)
npm run lint # biome
npm run typecheck # tsc --noEmit
```
See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for internals.
---
## License
MIT
-14
View File
@@ -1,14 +0,0 @@
# Reasonix Platform instructions
- This branch owns shared online services only. Do not add product client or
public website code here.
- Keep every production deployment manually triggered and restricted to the
`platform` branch.
- Database changes must be backward-compatible, idempotent, and accompanied by
a migration or runtime schema guard.
- Ingest endpoints must validate before enqueueing, acknowledge only durable
work, and remain safe under retries.
- Add tests at the HTTP, queue, or storage boundary where behavior becomes
externally observable.
- Never commit credentials, tokens, customer content, or personal user data.
- Comments are English and explain only non-obvious constraints.
-137
View File
@@ -1,137 +0,0 @@
# Security Policy
## Supported Versions
Reasonix security fixes are prioritized for the currently developed Go rewrite
and the current 1.x release line.
| Version or branch | Security support |
| --- | --- |
| `main-v2` / 1.x releases | Supported |
| `v1` / 0.x legacy branch | Critical fixes only, where practical |
| Older releases, forks, or modified builds | Not covered unless the issue is reproducible upstream |
If you are unsure whether a version is affected, report against the newest
released 1.x version and include the exact version or commit you tested.
## Reporting a Vulnerability
Please report security issues privately. Do not open a public issue with exploit
details, secrets, crash dumps, or proof-of-concept payloads.
Preferred reporting path:
1. Use GitHub private vulnerability reporting for this repository, if available.
2. If private reporting is not available to you, open a minimal public issue
asking for a private maintainer contact path. Do not include exploit details
in that issue.
Please include:
- Affected Reasonix version, commit, operating system, and installation method.
- The feature or surface involved, such as CLI, desktop app, HTTP `serve`, bot
gateway, MCP plugin, built-in tool, updater, or configuration loading.
- Clear reproduction steps using dummy credentials and non-sensitive files.
- The expected impact, such as secret disclosure, arbitrary file access,
command execution, sandbox escape, authentication bypass, or supply-chain risk.
- Any relevant logs with API keys, tokens, local paths, and personal data
redacted.
Do not send real provider API keys, bot credentials, OAuth tokens, private
workspace files, or third-party user data.
## Security Boundaries
Reasonix is a local coding agent. Many features intentionally operate on the
user's local machine and workspace, including file reads, file writes, shell
commands, MCP plugins, language servers, bot sessions, and model-provider
requests. A finding is security-relevant when it crosses a supported boundary or
bypasses an explicit guard.
Supported boundaries include:
- Workspace confinement for file operations that are documented or implemented
as workspace-scoped.
- Permission checks for tool calls, shell commands, file writes, and approvals.
- Sandbox behavior for built-in shell execution where the platform supports it.
- Secret handling for provider keys, bot credentials, OAuth tokens, plugin
headers, and credential-store fallback files.
- HTTP `serve` protections for the unauthenticated local server, including
localhost binding assumptions, JSON-only state-changing requests, and CORS
restrictions.
- Desktop and bot session isolation, including per-workspace session metadata
and configured bot allowlists.
- Updater, install, and release verification paths.
The following are normally treated as trusted local/operator-controlled inputs
unless another bug lets an untrusted actor supply them:
- CLI arguments and text typed directly by the local user.
- Project configuration files intentionally loaded from the current workspace.
- Explicit `@path` references supplied by the local user to attach local files.
- MCP servers, language servers, hooks, and slash commands installed or enabled
by the local user.
- Provider base URLs and model names configured by the local user.
The following can be security issues when reachable by an untrusted actor or
when they bypass the intended boundary:
- Reading or writing files outside the configured workspace without explicit
local-user intent.
- Following symlinks or path traversal to escape workspace confinement.
- Running shell commands or external tools without the required permission gate.
- Leaking credentials, environment variables, prompt history, local files, or bot
messages to logs, model providers, MCP servers, crash reports, or telemetry.
- Allowing a website to drive the local HTTP server through CSRF, CORS, or
content-type bypasses.
- Letting a bot user outside the configured allowlist submit prompts, approve
tools, or access a project workspace.
- Trusting unverified update artifacts, plugin definitions, or downloaded
binaries.
## `@` File References
Reasonix supports `@path` references so users can include local files and images
in a prompt. This is intentional local functionality, but implementations must
preserve these invariants:
- In workspace-scoped sessions, relative and absolute paths must resolve under
the active workspace root before file content is read or attached.
- Path traversal such as `..` must not escape the workspace root.
- Symlinks must not be usable to bypass the intended workspace boundary.
- Unscoped local CLI compatibility must not be exposed to remote, bot, or
browser-controlled inputs unless an equivalent workspace boundary is applied.
- File content should be size-limited and binary content should not be dumped as
prompt text.
Static analysis alerts about path expressions should be triaged against these
rules: user-controlled path data is expected, but the access must either stay
inside the configured workspace or be clearly limited to trusted local CLI use.
## Out of Scope
The following reports are usually out of scope unless they demonstrate a bypass
of one of the boundaries above:
- A local user intentionally asks Reasonix to read, edit, or send their own
files to a configured model provider.
- A local user installs or enables a malicious MCP server, hook, slash command,
language server, or shell command and then grants it permission.
- A configured model provider, proxy, or MCP server receives data the user
intentionally sent to it.
- Denial-of-service issues that only affect the local user's own session and do
not corrupt files, leak secrets, or bypass permissions.
- Issues requiring administrator/root access on the user's machine before
interacting with Reasonix.
- Vulnerabilities in third-party services, models, proxies, or plugins that are
not caused by Reasonix behavior.
## Coordinated Disclosure
This is a community-maintained project. Maintainers will make a best-effort
assessment, ask follow-up questions when needed, and coordinate fixes before
public disclosure for confirmed vulnerabilities.
Please give maintainers reasonable time to investigate and release a fix before
publishing exploit details. If you plan to disclose on a timeline, include that
timeline in your initial report.
+115
View File
@@ -0,0 +1,115 @@
# Benchmarks
This is where validation lives. The v0.1 milestone gates on a reproducible
tool-use eval that compares, on the same tasks:
1. **Baseline** — a deliberately cache-hostile agent (fresh timestamp +
shuffled tool spec each turn), representative of how generic frameworks
wire up DeepSeek.
2. **Reasonix** — the same tools and system prompt, driven through
`CacheFirstLoop` so the byte prefix stays stable turn-over-turn.
Both modes share the same `DeepSeekClient`, so the *only* meaningful
difference is prefix stability — any cache-hit / cost gap is attributable to
Pillar 1 of the architecture, nothing else.
## Scope — this is τ-bench-*lite*
We don't ship a full port of [Sierra's τ-bench](https://github.com/sierra-research/tau-bench)
(airline + retail, Python). Instead:
- `tau-bench/tasks.ts` hand-authors 8 retail-flavored multi-turn tasks
that exercise tool use, identity verification, refusal, and mid-conversation
goal change.
- The task schema (`tau-bench/types.ts`) mirrors τ-bench's shape — stateful
tools, an LLM user simulator, end-state DB predicates — so real upstream
tasks can later drop in without harness changes.
- All success predicates are **deterministic DB checks**, not LLM judges.
Refusal tasks pass iff the DB is unchanged.
## Files
```
tau-bench/
├── types.ts — TaskDefinition / RunResult / BenchReport shapes
├── db.ts — tiny in-memory WorldState + cloneDb
├── tasks.ts — the 8 seed tasks + shared tool factories
├── user-sim.ts — LLM user simulator (V3, T=0.1)
├── baseline.ts — naive cache-hostile agent runner
├── runner.ts — orchestrates user-sim × agent × task × mode
└── report.ts — turns a results-*.json into a report.md
```
## Quickstart
```bash
# dry-run: no API calls, just validate the harness is wired up
npx tsx benchmarks/tau-bench/runner.ts --dry
# full run: both modes, all tasks, 1 repeat
export DEEPSEEK_API_KEY=sk-...
npx tsx benchmarks/tau-bench/runner.ts
# tighten variance: 3 repeats per task
npx tsx benchmarks/tau-bench/runner.ts --repeats 3
# narrow to one task while iterating
npx tsx benchmarks/tau-bench/runner.ts --task t01_address_happy --verbose
# render the report
npx tsx benchmarks/tau-bench/report.ts benchmarks/tau-bench/results-<date>.json
# emit per-run transcripts so you can reasonix replay / diff them
npx tsx benchmarks/tau-bench/runner.ts --transcripts-dir ./transcripts
npx reasonix diff \
./transcripts/t01_address_happy.baseline.r1.jsonl \
./transcripts/t01_address_happy.reasonix.r1.jsonl \
--md diff.md
```
The runner writes `benchmarks/tau-bench/results-<iso-timestamp>.json`. Point
`report.ts` at it (or pass `--out report.md` to override the output path).
When `--transcripts-dir <path>` is set, each `(task, mode, repeat)` run also
writes a `<taskId>.<mode>.r<n>.jsonl` transcript into that directory —
these carry per-turn `usage`, `cost`, and (for Reasonix) the
`prefixHash`, so `reasonix replay` and `reasonix diff` can rebuild the
economics offline.
## CLI flags
| flag | default | meaning |
|---|---|---|
| `--task <id>` | all | run only one task by id |
| `--mode baseline` \| `reasonix` | both | restrict to one mode |
| `--repeats <N>` | 1 | repeat each (task, mode) pair N times |
| `--model <id>` | deepseek-chat | agent model |
| `--user-model <id>` | deepseek-chat | user-simulator model |
| `--out <path>` | `results-<ts>.json` | results file path |
| `--transcripts-dir <path>` | off | write one transcript per run for replay/diff |
| `--dry` | off | skip the LLM; only wire-check |
| `--verbose` \| `-v` | off | print every user / agent / tool line |
## What a run costs
A full run (8 tasks × 2 modes × 1 repeat) does on the order of 30–60
DeepSeek V3 calls — well under $0.05 at current pricing. `--repeats 3`
triples that.
## Adding tasks
1. Add a `TaskDefinition` to `tau-bench/tasks.ts`. Reuse the tool factories
defined at the top of that file, or add new ones (remember: factories so
tools close over the *per-run* db snapshot).
2. Make the `check` predicate check the end-state DB, not the agent's text —
agents phrase things differently on every run.
3. Run `--task <your_id> --verbose` to eyeball the transcript.
Non-goals (for this harness):
- LLM-as-judge — brittle and expensive, DB predicates are enough.
- Streaming comparison — the harness uses `stream: false` in Reasonix mode
so both runners make the exact same request shape.
- Claude head-to-head — we estimate Claude's cost from token counts using
Sonnet 4.6 pricing (see `src/telemetry.ts`); running Claude for real is
out of scope.
+106
View File
@@ -0,0 +1,106 @@
# Harvest eval harness (Pillar 2)
Measures Pillar 2 (R1 Thought Harvesting) the same way `tau-bench/`
measures Pillar 1 (Cache-First Loop): isolate one variable, produce
numbers anyone can reproduce.
## What's different from τ-bench-lite?
Deliberately different task shape:
| | τ-bench-lite | harvest-bench |
|---|---|---|
| task style | multi-turn tool-use | single-turn reasoning |
| user simulator | yes (LLM) | no |
| DB state | yes | no |
| checker | DB-predicate | regex + set/value compare |
| target | Pillar 1 | Pillar 2 |
| model | deepseek-chat (default) | deepseek-reasoner (only R1 has `reasoning_content` for harvest to work on) |
The two harnesses answer different questions:
- **τ-bench-lite**: *"Does Cache-First actually cut cost on real tool-use
workflows?"* (yes — 47.7pp cache, −39% cost on 48-run data.)
- **harvest-bench**: *"Does the extra V3 harvest call add measurable
value above plain R1?"* (TBD — run it to find out.)
## Modes
Three modes isolate one variable each:
| mode | model | harvest | what it measures |
|---|---|---|---|
| `baseline` | deepseek-chat | off | floor reference — V3 at its best |
| `reasoner` | deepseek-reasoner | off | raw R1 gain over V3 on reasoning |
| `reasoner-harvest` | deepseek-reasoner | on | R1 + the extra V3 harvest call |
Deltas:
- `baseline → reasoner` answers "is R1 worth the price on these
problems?"
- `reasoner → reasoner-harvest` answers "is the harvest call worth
its incremental cost?"
## Quickstart
```bash
# Dry-run — no API, smoke-test the wiring
npx tsx benchmarks/harvest/runner.ts --dry
# Full run (live DeepSeek, costs ~$0.20-0.60 for 6 tasks × 3 modes × 1 repeat)
export DEEPSEEK_API_KEY=sk-...
npx tsx benchmarks/harvest/runner.ts
# Tighter run with 3 repeats (~$0.60-2.00)
npx tsx benchmarks/harvest/runner.ts --repeats 3
# Only the hard tasks (skip the easy floor)
npx tsx benchmarks/harvest/runner.ts --task pseudoprime_base2
npx tsx benchmarks/harvest/runner.ts --task derangements_d7
npx tsx benchmarks/harvest/runner.ts --task euler_quadratic_break
# Bump per-call timeout (reasoner + harvest occasionally blow past the 120s default)
npx tsx benchmarks/harvest/runner.ts --timeout 600
# Per-run transcripts so you can reasonix replay / diff them
npx tsx benchmarks/harvest/runner.ts --repeats 3 --transcripts-dir transcripts/
# Render report
npx tsx benchmarks/harvest/report.ts benchmarks/harvest/results-*.json
```
## Tasks (v0.3 seed)
Split into two bands:
**Easy band** — V3 chat solves these, so baseline has a real pass rate and the bench has a reference floor.
| id | shape | why |
|---|---|---|
| `mod7_list` | number theory, 29-element list | R1 often tries enumeration first, then reaches for modular arithmetic — clean rejectedPaths signal |
| `flips_until_3heads` | probability, single integer | classic recurrence; R1 either derives or recalls, harvest should see hypotheses diverge |
| `three_hats` | logic puzzle, one-word answer | pure deduction chain, tests harvest's ability to extract the nested reasoning |
**Hard band** — picked specifically for known V3 failure modes. If R1 systematically beats V3 on these, and harvest further beats R1-only, we have our Pillar 2 story.
| id | shape | why V3 tends to fail |
|---|---|---|
| `pseudoprime_base2` | smallest Fermat pseudoprime to base 2 = **341** | V3 often answers 561 (Carmichael) or some other composite it remembers — R1 actually checks 2^n mod n |
| `derangements_d7` | D_7 = **1854** | V3 approximates via n!/e and rounds wrong, or recalls D_6/D_8; R1 uses the recurrence D_n=(n−1)(D_{n−1}+D_{n−2}) |
| `euler_quadratic_break` | smallest n where n²+n+41 is composite = **40** | V3 frequently confuses with n²−n+41 (first fails at 41); R1 checks small n systematically |
Adding a new task: see `tasks.ts`. Any checker that's deterministic is
fair game — extract numeric/list/text with regex, compare with set
equality or string matching.
## Non-goals
- **No LLM-as-judge.** Brittle and expensive; defeats the point of a
reproducible bench. If a checker is too hard to write deterministically,
the task doesn't belong here.
- **No tool-use tasks.** Those live in `tau-bench/`. Different story.
- **No multi-turn.** v0.3 harness is single-turn Q/A. Multi-turn reasoning
eval is separate scope.
- **No benchmark-data cherry-picking.** When results come in, we publish
them whether they validate Pillar 2 or not. "harvest didn't help on
these 3 tasks" is still useful information.
+82
View File
@@ -0,0 +1,82 @@
# Reasonix harvest eval (Pillar 2)
**Date:** 2026-04-21T16:44:21.011Z
**Tasks:** 6 · repeats × 1 · modes: baseline, reasoner, reasoner-harvest
**Reasonix version:** 0.2.2
## Summary by mode
| mode | runs | pass rate | cache hit | cost / run | harvest turns | subgoals | uncertainties |
|---|---:|---:|---:|---:|---:|---:|---:|
| baseline | 6 | 100% | 39.5% | $0.001371 | 0.0 | 0.0 | 0.0 |
| reasoner | 6 | 100% | 74.2% | $0.004170 | 0.0 | 0.0 | 0.0 |
| reasoner-harvest | 6 | 83% | 61.6% | $0.003304 | 0.8 | 2.8 | 1.3 |
## Deltas
- **baseline → reasoner**
- pass rate: 0pp
- cost: ×3.04 (each run costs more)
- harvest signal / run: 0.0 subgoals, 0.0 uncertainties
- **baseline → reasoner-harvest**
- pass rate: -17pp
- cost: ×2.41 (each run costs more)
- harvest signal / run: 2.8 subgoals, 1.3 uncertainties
## Per-task breakdown
| task | mode | rep | verdict | cache | cost | sg | un | note |
|---|---|---:|:---:|---:|---:|---:|---:|---|
| mod7_list | baseline | 1 | ✅ | 86.5% | $0.001361 | 0 | 0 | |
| mod7_list | reasoner | 1 | ✅ | 86.5% | $0.004797 | 0 | 0 | |
| mod7_list | reasoner-harvest | 1 | ✅ | 86.5% | $0.003045 | 4 | 2 | |
| flips_until_3heads | baseline | 1 | ✅ | 92.8% | $0.001198 | 0 | 0 | |
| flips_until_3heads | reasoner | 1 | ✅ | 92.8% | $0.002830 | 0 | 0 | |
| flips_until_3heads | reasoner-harvest | 1 | ✅ | 92.8% | $0.001691 | 4 | 0 | |
| three_hats | baseline | 1 | ✅ | 57.7% | $0.001514 | 0 | 0 | |
| three_hats | reasoner | 1 | ✅ | 57.7% | $0.009340 | 0 | 0 | |
| three_hats | reasoner-harvest | 1 | ✅ | 57.7% | $0.009651 | 3 | 2 | |
| pseudoprime_base2 | baseline | 1 | ✅ | 0.0% | $0.001894 | 0 | 0 | |
| pseudoprime_base2 | reasoner | 1 | ✅ | 75.3% | $0.003128 | 0 | 0 | |
| pseudoprime_base2 | reasoner-harvest | 1 | ❌ | 0.0% | $0.000000 | 0 | 0 | This operation was aborted |
| derangements_d7 | baseline | 1 | ✅ | 0.0% | $0.001273 | 0 | 0 | |
| derangements_d7 | reasoner | 1 | ✅ | 71.9% | $0.003492 | 0 | 0 | |
| derangements_d7 | reasoner-harvest | 1 | ✅ | 71.9% | $0.002843 | 3 | 2 | |
| euler_quadratic_break | baseline | 1 | ✅ | 0.0% | $0.000985 | 0 | 0 | |
| euler_quadratic_break | reasoner | 1 | ✅ | 61.0% | $0.001431 | 0 | 0 | |
| euler_quadratic_break | reasoner-harvest | 1 | ✅ | 61.0% | $0.002594 | 3 | 2 | |
## Scope
Unlike τ-bench-lite, these tasks are single-turn reasoning problems (no user simulator, no DB, no tool calls). Checkers are deterministic — regex + set / value compare, never an LLM judge. The point is to isolate whether the Pillar 2 harvest step adds measurable value above plain reasoner usage.
Interpretation: `baseline` (chat / V3) is a floor. `reasoner` shows the raw R1 gain. `reasoner-harvest` isolates the cost + quality delta from the extra V3 harvest call.
## Findings (v0.3, 18-run set)
Two bands of tasks tried (easy: mod7 / flips / hats; hard: pseudoprime341 / D_7 / Euler quadratic). The hard band was chosen for *known V3 failure modes*. Result:
1. **V3 chat passed all six tasks, including the three "hard" ones.** DeepSeek V3 knew 341 is the smallest base-2 pseudoprime, computed D_7=1854, and identified n=40 as the first Euler-polynomial failure. The tasks are well-known enough to be in training; the trap-answer variants (561, 265, 41) that the checkers specifically reject didn't fire.
2. **Reasoner cost 3.04× baseline for identical pass rates (6/6 both sides).** On this task set, R1 adds cost and latency without measurable quality. The cache-hit story *does* extend to reasoner (74.2% mean), so Pillar 1 generalizes to R1 — but that's not a Pillar 2 story.
3. **Harvest produced real signal** (mean 2.8 subgoals, 1.3 uncertainties per run where it fired) but didn't improve outcomes. One run hit the 300s timeout (bumped from 120s), suggesting reasoner-harvest latency is still unpredictable on some problems.
### What this tells us about Pillar 2 positioning
We've now given harvest two shots at showing answer-quality value on pure reasoning tasks. Both times V3 ate its lunch. This isn't a framework bug — it's a positioning signal:
- **On well-known math / logic / counting problems, DeepSeek V3 is strong enough that paying for R1 is not justified by accuracy.** Pillar 2's "smart preset" cost multiplier (10×) quoted in the README is real but for most single-question Q/A it buys nothing.
- **Harvest's plausible value surfaces are not "better math answers".** More likely: (a) transparency / auditability for developers, (b) planning support when tools are in the mix, (c) driving branch-sampling on cases where uncertainty correlates with wrongness. None of those are tested here.
### Recommendation
Don't ship v0.3 on the "harvest makes answers more correct" claim — the data doesn't support it. Two honest paths forward:
- **Reframe Pillar 2 as a developer-facing introspection feature** (debug visibility into R1's planning). Keep harvest opt-in; stop implying it improves accuracy on end-user tasks.
- **Move harvest eval to tool-use contexts** where uncertainty matters for *which tool to call next*, not *is this math right*. That's a new task shape — different harness, not a bigger task set here.
In either case, the next Reasonix release is better served by MCP client (Pillar 1 leverage across the ecosystem) than by a bigger harvest-bench data push.
### Notes
- Cache hit on baseline runs surprised me (39.5% mean). That's DeepSeek's *cross-session* prompt cache: the same system prompt ran many times across modes and runs warmed the server-side cache. Real cross-session evidence that Cache-First's value isn't only per-session.
- 5-subgoals cap still hit uniformly. `HarvestOptions.maxItems` default is 5; harvest is truncating. Moot unless we find a setting where that matters.
+232
View File
@@ -0,0 +1,232 @@
/**
* Render a harvest-bench results.json into a markdown report.
*
* npx tsx benchmarks/harvest/report.ts benchmarks/harvest/results-<date>.json
* npx tsx benchmarks/harvest/report.ts <input.json> --out report.md
*/
import { readFileSync, writeFileSync } from "node:fs";
import { pathToFileURL } from "node:url";
import { ALL_MODES, type HarvestBenchReport, type HarvestMode, type HarvestRunResult } from "./types.js";
interface CliArgs {
input: string;
outPath: string;
}
function parseArgs(argv: string[]): CliArgs {
const out: CliArgs = { input: "", outPath: "benchmarks/harvest/report.md" };
const positional: string[] = [];
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === "--out") out.outPath = argv[++i] ?? out.outPath;
else if (a && !a.startsWith("--")) positional.push(a);
}
out.input = positional[0] ?? "";
if (!out.input) {
throw new Error("usage: npx tsx benchmarks/harvest/report.ts <results.json> [--out report.md]");
}
return out;
}
interface Agg {
runs: number;
passes: number;
avgCache: number;
avgCost: number;
avgClaudeCost: number;
avgHarvestedTurns: number;
avgSubgoals: number;
avgUncertainties: number;
}
function aggregate(results: HarvestRunResult[]): Agg {
if (results.length === 0) {
return {
runs: 0,
passes: 0,
avgCache: 0,
avgCost: 0,
avgClaudeCost: 0,
avgHarvestedTurns: 0,
avgSubgoals: 0,
avgUncertainties: 0,
};
}
const passes = results.filter((r) => r.verdict === "pass").length;
const mean = (fn: (r: HarvestRunResult) => number) =>
results.reduce((s, r) => s + fn(r), 0) / results.length;
return {
runs: results.length,
passes,
avgCache: mean((r) => r.cacheHitRatio),
avgCost: mean((r) => r.costUsd),
avgClaudeCost: mean((r) => r.claudeEquivalentUsd),
avgHarvestedTurns: mean((r) => r.harvestedTurns),
avgSubgoals: mean((r) => r.totalSubgoals),
avgUncertainties: mean((r) => r.totalUncertainties),
};
}
export function renderReport(report: HarvestBenchReport): string {
const lines: string[] = [];
lines.push("# Reasonix harvest eval (Pillar 2)");
lines.push("");
lines.push(`**Date:** ${report.meta.date}`);
lines.push(
`**Tasks:** ${report.meta.taskCount} · repeats × ${report.meta.repeatsPerTask} · modes: ${report.meta.modesRun.join(", ")}`,
);
lines.push(`**Reasonix version:** ${report.meta.reasonixVersion}`);
lines.push("");
// Per-mode summary
lines.push("## Summary by mode");
lines.push("");
lines.push("| mode | runs | pass rate | cache hit | cost / run | harvest turns | subgoals | uncertainties |");
lines.push("|---|---:|---:|---:|---:|---:|---:|---:|");
for (const mode of ALL_MODES) {
const rs = report.results.filter((r) => r.mode === mode);
const a = aggregate(rs);
if (a.runs === 0) continue;
lines.push(
`| ${mode} | ${a.runs} | ${pct(a.passes, a.runs)} | ${pct1(a.avgCache)} | $${a.avgCost.toFixed(6)} | ${a.avgHarvestedTurns.toFixed(1)} | ${a.avgSubgoals.toFixed(1)} | ${a.avgUncertainties.toFixed(1)} |`,
);
}
lines.push("");
// Pairwise deltas that actually answer the question
lines.push("## Deltas");
lines.push("");
const modes = ALL_MODES.filter((m) =>
report.results.some((r) => r.mode === m),
);
if (modes.length >= 2) {
const baseline = aggregate(report.results.filter((r) => r.mode === modes[0]));
for (let i = 1; i < modes.length; i++) {
const mode = modes[i]!;
const agg = aggregate(report.results.filter((r) => r.mode === mode));
const passDelta = (agg.passes / Math.max(agg.runs, 1)) - (baseline.passes / Math.max(baseline.runs, 1));
const costRatio = baseline.avgCost > 0 ? agg.avgCost / baseline.avgCost : 0;
lines.push(`- **${modes[0]} → ${mode}**`);
lines.push(` - pass rate: ${(passDelta * 100).toFixed(0)}pp`);
lines.push(` - cost: ×${costRatio.toFixed(2)} (each run costs ${costRatio > 1 ? "more" : "less"})`);
lines.push(` - harvest signal / run: ${agg.avgSubgoals.toFixed(1)} subgoals, ${agg.avgUncertainties.toFixed(1)} uncertainties`);
lines.push("");
}
}
// Per-task breakdown
lines.push("## Per-task breakdown");
lines.push("");
const byTask = new Map<string, HarvestRunResult[]>();
for (const r of report.results) {
const list = byTask.get(r.taskId) ?? [];
list.push(r);
byTask.set(r.taskId, list);
}
lines.push("| task | mode | rep | verdict | cache | cost | sg | un | note |");
lines.push("|---|---|---:|:---:|---:|---:|---:|---:|---|");
for (const [taskId, runs] of byTask) {
for (const r of runs) {
const mark = r.verdict === "pass" ? "✅" : r.verdict === "fail" ? "❌" : "•";
lines.push(
`| ${taskId} | ${r.mode} | ${r.repeat} | ${mark} | ${pct1(r.cacheHitRatio)} | $${r.costUsd.toFixed(6)} | ${r.totalSubgoals} | ${r.totalUncertainties} | ${truncate(r.checkNote ?? "", 40)} |`,
);
}
}
lines.push("");
lines.push("## Scope");
lines.push("");
lines.push(
"Unlike τ-bench-lite, these tasks are single-turn reasoning problems (no user simulator, no DB, no tool calls). Checkers are deterministic — regex + set / value compare, never an LLM judge. The point is to isolate whether the Pillar 2 harvest step adds measurable value above plain reasoner usage.",
);
lines.push("");
lines.push(
"Interpretation: `baseline` (chat / V3) is a floor. `reasoner` shows the raw R1 gain. `reasoner-harvest` isolates the cost + quality delta from the extra V3 harvest call.",
);
return lines.join("\n");
}
/**
* Findings section is hand-written into report.md after rendering — the
* numbers don't interpret themselves, and boilerplate sections full of
* "TBD — analyse the data" are worse than nothing. Append with append().
*
* This section has been updated twice: after the first 9-run set (which
* bottomed out at V3 on easy tasks) and after this 18-run set (which
* bottomed out at V3 on *hard* tasks too).
*/
export function renderFindings(_report: HarvestBenchReport): string {
return [
"",
"## Findings (v0.3, 18-run set)",
"",
"Two bands of tasks tried (easy: mod7 / flips / hats; hard: pseudoprime341 / D_7 / Euler quadratic). The hard band was chosen for *known V3 failure modes*. Result:",
"",
"1. **V3 chat passed all six tasks, including the three \"hard\" ones.** DeepSeek V3 knew 341 is the smallest base-2 pseudoprime, computed D_7=1854, and identified n=40 as the first Euler-polynomial failure. The tasks are well-known enough to be in training; the trap-answer variants (561, 265, 41) that the checkers specifically reject didn't fire.",
"2. **Reasoner cost 3.04× baseline for identical pass rates (6/6 both sides).** On this task set, R1 adds cost and latency without measurable quality. The cache-hit story *does* extend to reasoner (74.2% mean), so Pillar 1 generalizes to R1 — but that's not a Pillar 2 story.",
"3. **Harvest produced real signal** (mean 2.8 subgoals, 1.3 uncertainties per run where it fired) but didn't improve outcomes. One run hit the 300s timeout (bumped from 120s), suggesting reasoner-harvest latency is still unpredictable on some problems.",
"",
"### What this tells us about Pillar 2 positioning",
"",
"We've now given harvest two shots at showing answer-quality value on pure reasoning tasks. Both times V3 ate its lunch. This isn't a framework bug — it's a positioning signal:",
"",
"- **On well-known math / logic / counting problems, DeepSeek V3 is strong enough that paying for R1 is not justified by accuracy.** Pillar 2's \"smart preset\" cost multiplier (10×) quoted in the README is real but for most single-question Q/A it buys nothing.",
"- **Harvest's plausible value surfaces are not \"better math answers\".** More likely: (a) transparency / auditability for developers, (b) planning support when tools are in the mix, (c) driving branch-sampling on cases where uncertainty correlates with wrongness. None of those are tested here.",
"",
"### Recommendation",
"",
"Don't ship v0.3 on the \"harvest makes answers more correct\" claim — the data doesn't support it. Two honest paths forward:",
"",
"- **Reframe Pillar 2 as a developer-facing introspection feature** (debug visibility into R1's planning). Keep harvest opt-in; stop implying it improves accuracy on end-user tasks.",
"- **Move harvest eval to tool-use contexts** where uncertainty matters for *which tool to call next*, not *is this math right*. That's a new task shape — different harness, not a bigger task set here.",
"",
"In either case, the next Reasonix release is better served by MCP client (Pillar 1 leverage across the ecosystem) than by a bigger harvest-bench data push.",
"",
"### Notes",
"",
"- Cache hit on baseline runs surprised me (39.5% mean). That's DeepSeek's *cross-session* prompt cache: the same system prompt ran many times across modes and runs warmed the server-side cache. Real cross-session evidence that Cache-First's value isn't only per-session.",
"- 5-subgoals cap still hit uniformly. `HarvestOptions.maxItems` default is 5; harvest is truncating. Moot unless we find a setting where that matters.",
"",
].join("\n");
}
function pct(num: number, denom: number): string {
if (denom === 0) return "—";
return `${((num / denom) * 100).toFixed(0)}%`;
}
function pct1(x: number): string {
return `${(x * 100).toFixed(1)}%`;
}
function truncate(s: string, n: number): string {
return s.length > n ? `${s.slice(0, n)}…` : s;
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
const raw = readFileSync(args.input, "utf8");
const report = JSON.parse(raw) as HarvestBenchReport;
const md = renderReport(report) + renderFindings(report);
writeFileSync(args.outPath, md, "utf8");
console.log(`wrote ${args.outPath} (${report.results.length} runs)`);
}
function isMain(): boolean {
const entry = process.argv[1];
if (!entry) return false;
return import.meta.url === pathToFileURL(entry).href;
}
if (isMain()) {
main().catch((err) => {
console.error(err);
process.exit(1);
});
}
// Suppress unused import warning — kept in scope so ALL_MODES order matches renderer iteration.
export type { HarvestMode };
+305
View File
@@ -0,0 +1,305 @@
{
"meta": {
"date": "2026-04-21T16:44:21.011Z",
"repeatsPerTask": 1,
"taskCount": 6,
"modesRun": [
"baseline",
"reasoner",
"reasoner-harvest"
],
"reasonixVersion": "0.2.2"
},
"results": [
{
"taskId": "mod7_list",
"mode": "baseline",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "Alright, let's go step-by-step.\n\n---\n\n## **Step 1: Problem restatement**\nWe want positive integers \\( n \\leq 100 \\) such that \n\\[\nn^2 + n + 1 \\equiv 0 \\pmod{7}.\n\\]\n\n---\n\n## **Step 2: Try residues modulo 7**\nLet \\( n \\equiv r \\pmod{7} \\), where \\( r \\in \\{0, 1, 2, 3, 4, 5, 6\\} \\).\n\nWe compute \\( r^2 + r + 1 \\mod 7 \\):\n\n- \\( r = 0 \\): \\( 0 + 0 + 1 = 1 \\) → not divisible by 7.\n- \\( r = 1 \\): \\( 1 + 1 + 1 = 3 \\) → no.\n- \\( r = 2 \\): \\( 4 + 2 + 1 = 7 \\equiv 0 \\) → yes.\n- \\( r = 3 \\): \\( 9 + 3 + 1 = 13 \\equiv 6 \\) → no.\n- \\( r = 4 \\): \\( 16 + 4 + 1 = 21 \\equiv 0 \\) → yes.\n- \\( r = 5 \\): \\( 25 + 5 + 1 = 31 \\equiv 3 \\) → no.\n- \\( r = 6 \\): \\( 36 + 6 + 1 = 43 \\equiv 1 \\) → no.\n\nSo \\( n \\equiv 2 \\pmod{7} \\) or \\( n \\equiv 4 \\pmod{7} \\).\n\n---\n\n## **Step 3: General form**\n\\[\nn \\equiv 2 \\pmod{7} \\quad \\text{or} \\quad n \\equiv 4 \\pmod{7}.\n\\]\n\nThat means \\( n = 7k + 2 \\) or \\( n = 7k + 4 \\) for some integer \\( k \\ge 0 \\).\n\n---\n\n## **Step 4: Positive integers \\( n \\leq 100 \\)**\n**Case 1:** \\( n = 7k + 2 \\), \\( n \\leq 100 \\), \\( k \\ge 0 \\).\n\n\\( 7k + 2 \\leq 100 \\) → \\( 7k \\leq 98 \\) → \\( k \\leq 14 \\).\n\nSo \\( k = 0, 1, 2, \\dots, 14 \\) gives \\( n = 2, 9, 16, \\dots, 100 \\).\n\nCheck \\( n=100 \\): \\( 7 \\times 14 + 2 = 98 + 2 = 100 \\) okay.\n\n---\n\n**Case 2:** \\( n = 7k + 4 \\), \\( n \\leq 100 \\), \\( k \\ge 0 \\).\n\n\\( 7k + 4 \\leq 100 \\) → \\( 7k \\leq 96 \\) → \\( k \\leq 13 \\) (since \\( 7 \\times 13 = 91 \\), \\( 91+4=95 \\), \\( 7\\times14=98 \\) → \\( 98+4=102>100 \\)).\n\nSo \\( k = 0, 1, 2, \\dots, 13 \\) gives \\( n = 4, 11, 18, \\dots, 95 \\).\n\n---\n\n## **Step 5: List all such \\( n \\leq 100 \\)**\nFrom case 1: \\( 2, 9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86, 93, 100 \\).\n\nFrom case 2: \\( 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88, 95 \\).\n\nCombine and sort:\n\n\\[\n2, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72, 74, 79, 81, 86, 88, 93, 95, 100.\n\\]\n\n---\n\n## **Step 6: Check count**\nEach sequence has 15 and 14 terms respectively, total \\( 15 + 14 = 29 \\) numbers.\n\nCheck: \\( n=2 \\) works? \\( 4+2+1=7 \\) yes. \\( n=4 \\) works? \\( 16+4+1=21 \\) yes. \\( n=100 \\) works? \\( 10000+100+1=10101 \\), divide by 7: \\( 10101/7 = 1443 \\) yes.\n\n---\n\n**Final answer:**\n\\[\n\\boxed{2, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72, 74, 79, 81, 86, 88, 93, 95, 100}\n\\]\n\n---\n\nAnswer: 2, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72, 74, 79, 81, 86, 88, 93, 95, 100",
"turns": 1,
"cacheHitRatio": 0.8648648648648649,
"costUsd": 0.0013612800000000003,
"claudeEquivalentUsd": 0.018687,
"harvestedTurns": 0,
"totalSubgoals": 0,
"totalUncertainties": 0,
"totalHypotheses": 0,
"totalRejectedPaths": 0
},
{
"taskId": "mod7_list",
"mode": "reasoner",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "The condition \\(n^2 + n + 1\\) divisible by 7 is equivalent to \\(n^2 + n + 1 \\equiv 0 \\pmod{7}\\). Testing residues modulo 7:\n\n- \\(n \\equiv 0: 0+0+1 \\equiv 1\\) \n- \\(n \\equiv 1: 1+1+1 \\equiv 3\\) \n- \\(n \\equiv 2: 4+2+1 \\equiv 7 \\equiv 0\\) \n- \\(n \\equiv 3: 9+3+1 \\equiv 13 \\equiv 6\\) \n- \\(n \\equiv 4: 16+4+1 \\equiv 21 \\equiv 0\\) \n- \\(n \\equiv 5: 25+5+1 \\equiv 31 \\equiv 3\\) \n- \\(n \\equiv 6: 36+6+1 \\equiv 43 \\equiv 1\\)\n\nThus, \\(n \\equiv 2\\) or \\(n \\equiv 4 \\pmod{7}\\) are the solutions modulo 7. For positive integers \\(n \\leq 100\\), we list all numbers congruent to 2 or 4 modulo 7 in increasing order.\n\nAnswer: 2, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72, 74, 79, 81, 86, 88, 93, 95, 100",
"turns": 1,
"cacheHitRatio": 0.8648648648648649,
"costUsd": 0.00479742,
"claudeEquivalentUsd": 0.032982,
"harvestedTurns": 0,
"totalSubgoals": 0,
"totalUncertainties": 0,
"totalHypotheses": 0,
"totalRejectedPaths": 0
},
{
"taskId": "mod7_list",
"mode": "reasoner-harvest",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "The congruence \\(n^2 + n + 1 \\equiv 0 \\pmod{7}\\) is solved by testing residues modulo 7. The solutions are \\(n \\equiv 2\\) and \\(n \\equiv 4 \\pmod{7}\\). For positive integers \\(n \\le 100\\), these yield \\(n = 7k + 2\\) with \\(k = 0, 1, \\dots, 14\\) and \\(n = 7k + 4\\) with \\(k = 0, 1, \\dots, 13\\).\n\nAnswer: 2, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72, 74, 79, 81, 86, 88, 93, 95, 100",
"turns": 1,
"cacheHitRatio": 0.8648648648648649,
"costUsd": 0.00304542,
"claudeEquivalentUsd": 0.020982,
"harvestedTurns": 1,
"totalSubgoals": 4,
"totalUncertainties": 2,
"totalHypotheses": 2,
"totalRejectedPaths": 2
},
{
"taskId": "flips_until_3heads",
"mode": "baseline",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "Alright, let's reason through this step-by-step.\n\n---\n\n## **Step 1: Define states**\nLet’s define states based on how many consecutive heads we have at the moment.\n\n- \\( E_0 \\): Expected flips to reach 3 heads in a row starting from 0 consecutive heads (or just starting, or after a tail).\n- \\( E_1 \\): Expected flips to reach 3 heads in a row starting from 1 consecutive head.\n- \\( E_2 \\): Expected flips to reach 3 heads in a row starting from 2 consecutive heads.\n\nWe want \\( E_0 \\).\n\n---\n\n## **Step 2: Transition equations**\nFrom \\( E_0 \\):\n- Flip H (prob \\( \\frac12 \\)) → go to state \\( E_1 \\), 1 flip used.\n- Flip T (prob \\( \\frac12 \\)) → stay in \\( E_0 \\), 1 flip used.\n\nSo:\n\\[\nE_0 = \\frac12 (1 + E_1) + \\frac12 (1 + E_0)\n\\]\n\n---\n\nFrom \\( E_1 \\) (already have 1 head):\n- Flip H (prob \\( \\frac12 \\)) → go to state \\( E_2 \\), 1 flip used.\n- Flip T (prob \\( \\frac12 \\)) → go to state \\( E_0 \\), 1 flip used.\n\nSo:\n\\[\nE_1 = \\frac12 (1 + E_2) + \\frac12 (1 + E_0)\n\\]\n\n---\n\nFrom \\( E_2 \\) (already have 2 heads):\n- Flip H (prob \\( \\frac12 \\)) → done (3 heads in a row), 1 flip used.\n- Flip T (prob \\( \\frac12 \\)) → go to state \\( E_0 \\), 1 flip used.\n\nSo:\n\\[\nE_2 = \\frac12 (1 + 0) + \\frac12 (1 + E_0)\n\\]\n\\[\nE_2 = \\frac12 \\cdot 1 + \\frac12 (1 + E_0) \\quad\\text{(Wait, careful:)}\n\\]\nActually: \nIf H: \\( 1 \\) flip and done. \nIf T: \\( 1 \\) flip and then \\( E_0 \\) more.\n\nSo:\n\\[\nE_2 = \\frac12 \\cdot 1 + \\frac12 (1 + E_0)\n\\]\nCheck: \nH: 1 flip → total 1, done. Contribution: \\( \\frac12 \\times 1 \\). \nT: 1 flip + \\( E_0 \\) more → total \\( 1 + E_0 \\). Contribution: \\( \\frac12 \\times (1 + E_0) \\).\n\nYes:\n\\[\nE_2 = \\frac12 \\cdot 1 + \\frac12 (1 + E_0) = \\frac12 + \\frac12 + \\frac12 E_0 = 1 + \\frac{E_0}{2}.\n\\]\n\n---\n\n## **Step 3: Solve system**\nWe have:\n\n(1) \\( E_0 = \\frac12 (1 + E_1) + \\frac12 (1 + E_0) \\) \nSimplify: \\( E_0 = 1 + \\frac12 E_1 + \\frac12 E_0 \\) \nMultiply by 2: \\( 2E_0 = 2 + E_1 + E_0 \\) \nSo \\( E_0 - E_1 = 2 \\) … (Eq A)\n\n---\n\n(2) \\( E_1 = \\frac12 (1 + E_2) + \\frac12 (1 + E_0) \\) \nSimplify: \\( E_1 = 1 + \\frac12 E_2 + \\frac12 E_0 \\) … (Eq B)\n\n---\n\n(3) \\( E_2 = 1 + \\frac12 E_0 \\) … (Eq C)\n\n---\n\nFrom Eq C into Eq B:\n\\[\nE_1 = 1 + \\frac12 \\left( 1 + \\frac12 E_0 \\right) + \\frac12 E_0\n\\]\n\\[\nE_1 = 1 + \\frac12 + \\frac14 E_0 + \\frac12 E_0\n\\]\n\\[\nE_1 = \\frac32 + \\frac34 E_0\n\\]\n\n---\n\nFrom Eq A: \\( E_0 - E_1 = 2 \\) \nSubstitute \\( E_1 \\):\n\\[\nE_0 - \\left( \\frac32 + \\frac34 E_0 \\right) = 2\n\\]\n\\[\nE_0 - \\frac32 - \\frac34 E_0 = 2\n\\]\n\\[\n\\frac14 E_0 - \\frac32 = 2\n\\]\n\\[\n\\frac14 E_0 = 2 + \\frac32 = \\frac72\n\\]\n\\[\nE_0 = 14\n\\]\n\n---\n\n## **Step 4: Conclusion**\nThe expected number of flips until we get 3 heads in a row is \\( 14 \\).\n\n---\n\n**Answer:** 14",
"turns": 1,
"cacheHitRatio": 0.927536231884058,
"costUsd": 0.00119823,
"claudeEquivalentUsd": 0.016467,
"harvestedTurns": 0,
"totalSubgoals": 0,
"totalUncertainties": 0,
"totalHypotheses": 0,
"totalRejectedPaths": 0
},
{
"taskId": "flips_until_3heads",
"mode": "reasoner",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "The expected number of flips required to get three consecutive heads is 14, derived from solving the system of linear equations for the Markov chain states representing the current streak of heads.\n\nAnswer: 14",
"turns": 1,
"cacheHitRatio": 0.927536231884058,
"costUsd": 0.00283024,
"claudeEquivalentUsd": 0.019512,
"harvestedTurns": 0,
"totalSubgoals": 0,
"totalUncertainties": 0,
"totalHypotheses": 0,
"totalRejectedPaths": 0
},
{
"taskId": "flips_until_3heads",
"mode": "reasoner-harvest",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "The expected number of flips until three consecutive heads appear is 14.\n\nAnswer: 14",
"turns": 1,
"cacheHitRatio": 0.927536231884058,
"costUsd": 0.00169144,
"claudeEquivalentUsd": 0.011712,
"harvestedTurns": 1,
"totalSubgoals": 4,
"totalUncertainties": 0,
"totalHypotheses": 2,
"totalRejectedPaths": 0
},
{
"taskId": "three_hats",
"mode": "baseline",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "Let's break this down step by step.\n\n---\n\n**Step 1: Understanding the setup** \nThree people: A (first), B (second), C (third). \nEach hat is red (R) or blue (B). \nThey know: at least one hat is red. \nEach sees the other two hats, not their own. \nThey are asked in order: \"Do you know your hat color?\" \nA answers no, B answers no, C answers yes. \nWe want C’s hat color.\n\n---\n\n**Step 2: From A’s perspective when asked** \nA sees B and C. \nIf A had seen two blue hats, then A would know: \"At least one red hat means my hat must be red.\" \nSo A says \"no\" → A did **not** see two blue hats. \nThus A sees at least one red hat on {B, C}.\n\nPossible cases for what A sees (B, C): \n(R, R), (R, B), (B, R) — all have at least one red.\n\n---\n\n**Step 3: B’s reasoning when asked** \nB hears A say \"no\". \nB sees A and C. \n\nLet’s consider B’s possible deductions from A’s \"no\": \nB knows: If A had seen two blues, A would have said yes. Since A said no, A saw at least one red among B and C. \nBut B sees A and C. \n\nFrom B’s viewpoint: \n- If B saw C with blue and A with blue, then B would know: \"A saw me and C both blue → impossible because A said no.\" But B doesn’t see that because if B saw A blue and C blue, then B would know his own hat must be red (since at least one red total). But then B would answer yes. \nSo B’s \"no\" means B did **not** see A and C both blue. \n\nAlso, B considers: Could my hat be blue? Let’s check: \nSuppose B sees A red, C blue. Then B thinks: \"If my hat were blue, then A would have seen (B blue, C blue) → A would have said yes. But A said no, so my hat cannot be blue.\" \nSo if B saw (A red, C blue), B would deduce: \"My hat must be red.\" Then B would answer yes. \nBut B said no → so B did **not** see (A red, C blue).\n\nSimilarly, if B saw (A blue, C red): \nThen B thinks: \"If my hat were blue, then A would have seen (B blue, C red) → that’s fine (A sees one red on C), so A would still say no. If my hat were red, then A would see (B red, C red) → A would still say no. So I don’t know.\" That’s consistent with B saying no. \n\nIf B saw (A red, C red): \nThen B thinks: \"If my hat were blue, A sees (B blue, C red) → A says no. If my hat were red, A sees (B red, C red) → A says no. So I don’t know.\" That’s also consistent with B saying no.\n\nSo B says no in cases: \nB sees (A blue, C red) or (A red, C red). \nB would say yes if B saw (A red, C blue) — but that didn’t happen. \n\n---\n\n**Step 4: C’s reasoning** \nC hears A says no, B says no. \nC sees A and B. \n\nFrom C’s viewpoint: \nLet’s list all possible (A, B, C) hat colorings with at least one red, consistent with A’s \"no\" (which we already incorporated: A sees at least one red on B/C) and B’s \"no\" (meaning B didn’t see A red, C blue — because that would make B know).\n\nPossible cases for (A, B) that C could see: \n\n1. C sees (A blue, B red): \n Then C thinks: \n - If C were blue: (A blue, B red, C blue) → A sees (B red, C blue) → A says no. B sees (A blue, C blue) → B sees two blues? Wait, B sees A blue, C blue → then B knows his own hat is red (since at least one red total) → B would say yes. But B said no → contradiction. So C cannot be blue. \n So C must be red. \n\n2. C sees (A red, B blue): \n Then C thinks: \n - If C were blue: (A red, B blue, C blue) → A sees (B blue, C blue) → A would say yes (since at least one red total means A’s hat must be red). But A said no → contradiction. So C cannot be blue. \n So C must be red. \n\n3. C sees (A red, B red): \n Then C thinks: \n - If C were blue: (A red, B red, C blue) → A sees (B red, C blue) → A says no. B sees (A red, C blue) → B sees A red, C blue → earlier we said: If B sees A red, C blue, then B would deduce his own hat is red (reasoning: if B were blue, A would have seen two blues and said yes, but A said no, so B can’t be blue) → so B would say yes. But B actually said no → contradiction. So C cannot be blue. \n - If C were red: (A red, B red, C red) → A sees (B red, C red) → A says no. B sees (A red, C red) → B says no. Works fine. \n\nSo in all cases C sees for (A, B), C can deduce C’s hat is **red** because assuming C is blue leads to contradiction with B’s \"no\" (in cases 1 and 3) or with A’s \"no\" (in case 2).\n\n---\n\n**Step 5: Conclusion** \nC answers yes, and C’s hat must be red.\n\n---\n\nAnswer: red",
"turns": 1,
"cacheHitRatio": 0.5765765765765766,
"costUsd": 0.0015142700000000003,
"claudeEquivalentUsd": 0.020748,
"harvestedTurns": 0,
"totalSubgoals": 0,
"totalUncertainties": 0,
"totalHypotheses": 0,
"totalRejectedPaths": 0
},
{
"taskId": "three_hats",
"mode": "reasoner",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "Answer: red",
"turns": 1,
"cacheHitRatio": 0.5765765765765766,
"costUsd": 0.009340119999999999,
"claudeEquivalentUsd": 0.064068,
"harvestedTurns": 0,
"totalSubgoals": 0,
"totalUncertainties": 0,
"totalHypotheses": 0,
"totalRejectedPaths": 0
},
{
"taskId": "three_hats",
"mode": "reasoner-harvest",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "The third person deduces their hat color based on the previous responses. Since the first two answer \"no,\" it implies they each see at least one red hat and cannot determine their own color. Specifically, the second person's \"no\" indicates they see the third person's hat as red; otherwise, they would have known their own hat color. Therefore, the third person's hat must be red.\n\nAnswer: red",
"turns": 1,
"cacheHitRatio": 0.5765765765765766,
"costUsd": 0.0096511,
"claudeEquivalentUsd": 0.066198,
"harvestedTurns": 1,
"totalSubgoals": 3,
"totalUncertainties": 2,
"totalHypotheses": 3,
"totalRejectedPaths": 3
},
{
"taskId": "pseudoprime_base2",
"mode": "baseline",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "Alright, let's go step-by-step.\n\n---\n\n**Step 1: Understanding the problem**\n\nWe want the smallest composite integer \\( n > 1 \\) such that\n\n\\[\n2^n \\equiv 2 \\pmod{n}.\n\\]\n\nEquivalently, \\( 2^{n-1} \\equiv 1 \\pmod{n} \\) for composite \\( n \\).\n\nThis means \\( n \\) is a **Fermat pseudoprime to base 2**.\n\nWe check composite numbers starting from the smallest.\n\n---\n\n**Step 2: Check small composites**\n\n- \\( n = 4 \\): composite. \n\\( 2^4 = 16 \\), \\( 16 \\mod 4 = 0 \\), not \\( 2 \\). So \\( 2^4 \\equiv 0 \\not\\equiv 2 \\pmod{4} \\). Fails.\n\n- \\( n = 6 \\): composite. \n\\( 2^6 = 64 \\), \\( 64 \\mod 6 = 4 \\), not \\( 2 \\). Fails.\n\n- \\( n = 8 \\): composite. \n\\( 2^8 = 256 \\), \\( 256 \\mod 8 = 0 \\), not \\( 2 \\). Fails.\n\n- \\( n = 9 \\): composite. \n\\( 2^9 = 512 \\), \\( 512 \\mod 9 \\): \n\\( 512 \\div 9 \\), \\( 9 \\times 56 = 504 \\), remainder \\( 8 \\), not \\( 2 \\). Fails.\n\n- \\( n = 10 \\): composite. \n\\( 2^{10} = 1024 \\), \\( 1024 \\mod 10 = 4 \\), not \\( 2 \\). Fails.\n\n---\n\n**Step 3: Try n = 15**\n\n\\( n = 15 \\): composite. \n\\( 2^{15} = 32768 \\). \nCheck mod 15: \nWe can use mod arithmetic: \n\\( 2^4 = 16 \\equiv 1 \\pmod{15} \\)?? Wait, check: \\( 16 \\mod 15 = 1 \\), yes. \nSo \\( 2^{15} = (2^4)^3 \\times 2^3 = 1^3 \\times 8 = 8 \\pmod{15} \\). \nSo \\( 2^{15} \\equiv 8 \\not\\equiv 2 \\pmod{15} \\). Fails.\n\n---\n\n**Step 4: Try n = 21**\n\n\\( n = 21 \\): composite. \nMod 21: \n\\( 2^6 = 64 \\equiv 1 \\pmod{21} \\)? Check: \\( 64 \\div 21 = 3 \\times 21 = 63 \\), remainder 1, yes. \nSo \\( 2^{21} = (2^6)^3 \\times 2^3 \\equiv 1^3 \\times 8 = 8 \\pmod{21} \\). Not 2. Fails.\n\n---\n\n**Step 5: Try n = 22**\n\n\\( n = 22 \\): composite. \nUse mod 22: \nEuler’s totient not needed, just check order: \n\\( 2^5 = 32 \\equiv 10 \\pmod{22} \\) \n\\( 2^{10} = (2^5)^2 \\equiv 10^2 = 100 \\equiv 12 \\pmod{22} \\) \n\\( 2^{11} \\equiv 24 \\equiv 2 \\pmod{22} \\) \nSo \\( 2^{22} = (2^{11})^2 \\equiv 2^2 = 4 \\pmod{22} \\). Not 2. Fails.\n\n---\n\n**Step 6: Try n = 25**\n\n\\( n = 25 \\): composite. \nCheck mod 25: \n\\( 2^{10} = 1024 \\equiv 24 \\pmod{25} \\) (since \\( 1000 \\equiv 0 \\), \\( 24 \\equiv -1 \\)) \nActually check: \\( 1024 - 1000 = 24 \\), yes \\( 24 \\equiv -1 \\). \nSo \\( 2^{20} \\equiv 1 \\pmod{25} \\). \nThus \\( 2^{25} = 2^{20} \\times 2^5 \\equiv 1 \\times 32 = 7 \\pmod{25} \\). Not 2. Fails.\n\n---\n\n**Step 7: Known smallest base-2 Fermat pseudoprime**\n\nFrom known number theory facts: \nThe smallest Fermat pseudoprime to base 2 is \\( n = 341 \\). \nBut let's check smaller ones just in case I’m skipping one.\n\n---\n\n**Step 8: Try n = 91**\n\n91 = 7 × 13. \nCheck mod 7: \\( 2^{91} \\equiv (2^6)^{15} \\times 2^1 \\equiv 1^{15} \\times 2 = 2 \\pmod{7} \\). \nCheck mod 13: \\( 2^{12} \\equiv 1 \\pmod{13} \\), \n\\( 91 = 12 \\times 7 + 7 \\), so \\( 2^{91} \\equiv 2^7 = 128 \\equiv 11 \\pmod{13} \\). \nFor \\( 2^{91} \\equiv 2 \\pmod{91} \\), we need \\( 2^{91} \\equiv 2 \\) mod 7 and mod 13. \nMod 13 we got 11, not 2. So fails.\n\n---\n\n**Step 9: Try n = 341**\n\n341 = 11 × 31. \nCheck mod 11: \\( 2^{10} \\equiv 1 \\pmod{11} \\), \n\\( 341 = 10 \\times 34 + 1 \\), so \\( 2^{341} \\equiv 2^1 = 2 \\pmod{11} \\). Good. \nCheck mod 31: \\( 2^5 = 32 \\equiv 1 \\pmod{31} \\), \n\\( 341 = 5 \\times 68 + 1 \\), so \\( 2^{341} \\equiv 2^1 = 2 \\pmod{31} \\). \nSo \\( 2^{341} \\equiv 2 \\) mod 11 and mod 31, hence mod 341. \nThus \\( 341 \\) works.\n\n---\n\n**Step 10: Check smaller than 341**\n\nKnown list: 341 is smallest. Let’s check a candidate like 121 = 11^2: \n\\( 2^{121} \\mod 121 \\): \nMod 11: \\( 2^{10} \\equiv 1 \\), \\( 121 = 10 \\times 12 + 1 \\), so \\( 2^{121} \\equiv 2 \\pmod{11} \\). \nBut for modulus 121, need more: \nActually, \\( 2^{110} \\equiv 1 \\pmod{121} \\) by Euler’s theorem? Not exactly, but known: 121 is not a pseudoprime base 2. Quick check: \n\\( 2^{121} - 2 \\) divisible by 121? Probably not. Let’s skip detailed check; known fact is 341 is smallest.\n\n---\n\n**Step 11: Verify no smaller composite works**\n\nFrom known sequence A001567: \nFirst few base-2 Fermat pseudoprimes: 341, 561, 645, 1105, … \nSo smallest is 341.\n\n---\n\n**Step 12: Conclusion**\n\nThe smallest composite \\( n > 1 \\) with \\( 2^n \\equiv 2 \\pmod{n} \\) is \\( n = 341 \\).\n\n---\n\n**Answer:** 341",
"turns": 1,
"cacheHitRatio": 0,
"costUsd": 0.0018940500000000002,
"claudeEquivalentUsd": 0.02577,
"harvestedTurns": 0,
"totalSubgoals": 0,
"totalUncertainties": 0,
"totalHypotheses": 0,
"totalRejectedPaths": 0
},
{
"taskId": "pseudoprime_base2",
"mode": "reasoner",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "The smallest composite integer \\(n > 1\\) satisfying \\(2^n \\equiv 2 \\pmod{n}\\) is \\(n = 341\\). This is composite (\\(341 = 11 \\times 31\\)), and it satisfies the congruence because \\(2^{341} \\equiv 2 \\pmod{11}\\) and \\(2^{341} \\equiv 2 \\pmod{31}\\) by Fermat's little theorem, implying \\(2^{341} \\equiv 2 \\pmod{341}\\). All smaller composite integers (e.g., 4, 6, 8, 9, 10, 15, 21) fail to satisfy the condition.\n\nAnswer: 341",
"turns": 1,
"cacheHitRatio": 0.7529411764705882,
"costUsd": 0.0031281200000000003,
"claudeEquivalentUsd": 0.02154,
"harvestedTurns": 0,
"totalSubgoals": 0,
"totalUncertainties": 0,
"totalHypotheses": 0,
"totalRejectedPaths": 0
},
{
"taskId": "pseudoprime_base2",
"mode": "reasoner-harvest",
"repeat": 1,
"verdict": "fail",
"checkNote": "This operation was aborted",
"finalAgentMessage": "",
"turns": 0,
"cacheHitRatio": 0,
"costUsd": 0,
"claudeEquivalentUsd": 0,
"harvestedTurns": 0,
"totalSubgoals": 0,
"totalUncertainties": 0,
"totalHypotheses": 0,
"totalRejectedPaths": 0,
"errorMessage": "This operation was aborted"
},
{
"taskId": "derangements_d7",
"mode": "baseline",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "Alright, let's go step-by-step.\n\n---\n\n**Step 1: Understanding the problem**\n\nWe want the number of permutations of \\(\\{1, 2, 3, 4, 5, 6, 7\\}\\) with **no fixed points**. \nA fixed point means \\(\\pi(i) = i\\) for some \\(i\\). \nA permutation with no fixed points is called a **derangement**. \nThe number of derangements of \\(n\\) elements is denoted \\(D_n\\) or \\(!n\\).\n\n---\n\n**Step 2: Formula for derangements**\n\nThe formula for \\(D_n\\) is:\n\n\\[\nD_n = n! \\sum_{k=0}^{n} \\frac{(-1)^k}{k!}\n\\]\n\nFor \\(n=7\\):\n\n\\[\nD_7 = 7! \\left( \\frac{1}{0!} - \\frac{1}{1!} + \\frac{1}{2!} - \\frac{1}{3!} + \\frac{1}{4!} - \\frac{1}{5!} + \\frac{1}{6!} - \\frac{1}{7!} \\right)\n\\]\n\n---\n\n**Step 3: Compute step-by-step**\n\nFirst, \\(7! = 5040\\).\n\nNow compute the sum:\n\n\\[\n\\sum_{k=0}^{7} \\frac{(-1)^k}{k!} \n= 1 - 1 + \\frac{1}{2} - \\frac{1}{6} + \\frac{1}{24} - \\frac{1}{120} + \\frac{1}{720} - \\frac{1}{5040}\n\\]\n\nLet's compute term-by-term:\n\n- \\(1 - 1 = 0\\)\n- \\(0 + 0.5 = 0.5\\)\n- \\(0.5 - \\frac{1}{6} \\approx 0.5 - 0.1666667 = 0.3333333\\)\n- \\(0.3333333 + \\frac{1}{24} \\approx 0.3333333 + 0.0416667 = 0.375\\)\n- \\(0.375 - \\frac{1}{120} \\approx 0.375 - 0.0083333 = 0.3666667\\)\n- \\(0.3666667 + \\frac{1}{720} \\approx 0.3666667 + 0.0013889 = 0.3680556\\)\n- \\(0.3680556 - \\frac{1}{5040} \\approx 0.3680556 - 0.0001984 = 0.3678572\\)\n\nSo the sum is approximately \\(0.367857142857...\\)\n\n---\n\n**Step 4: Multiply by \\(7!\\)**\n\n\\[\nD_7 = 5040 \\times 0.367857142857\\ldots\n\\]\n\nLet's compute exactly using fractions:\n\n\\[\n\\sum_{k=0}^{7} \\frac{(-1)^k}{k!} \n= \\frac{1}{2!} - \\frac{1}{3!} + \\frac{1}{4!} - \\frac{1}{5!} + \\frac{1}{6!} - \\frac{1}{7!}\n\\]\n\nWrite with common denominator \\(7! = 5040\\):\n\n- \\(\\frac{1}{2!} = \\frac{2520}{5040}\\)\n- \\(\\frac{1}{3!} = \\frac{840}{5040}\\)\n- \\(\\frac{1}{4!} = \\frac{210}{5040}\\)\n- \\(\\frac{1}{5!} = \\frac{42}{5040}\\)\n- \\(\\frac{1}{6!} = \\frac{7}{5040}\\)\n- \\(\\frac{1}{7!} = \\frac{1}{5040}\\)\n\nNow alternate signs:\n\n\\[\n\\left( \\frac{2520}{5040} - \\frac{840}{5040} \\right) \n= \\frac{1680}{5040}\n\\]\n\\[\n\\frac{1680}{5040} + \\frac{210}{5040} = \\frac{1890}{5040}\n\\]\n\\[\n\\frac{1890}{5040} - \\frac{42}{5040} = \\frac{1848}{5040}\n\\]\n\\[\n\\frac{1848}{5040} + \\frac{7}{5040} = \\frac{1855}{5040}\n\\]\n\\[\n\\frac{1855}{5040} - \\frac{1}{5040} = \\frac{1854}{5040}\n\\]\n\nSo the sum is \\(\\frac{1854}{5040}\\).\n\n---\n\n**Step 5: Multiply by \\(7!\\)**\n\n\\[\nD_7 = 5040 \\times \\frac{1854}{5040} = 1854\n\\]\n\n---\n\n**Step 6: Quick check with known derangement numbers**\n\nKnown small derangements: \n\\(D_1 = 0, D_2 = 1, D_3 = 2, D_4 = 9, D_5 = 44, D_6 = 265, D_7 = 1854\\). \nYes, matches known sequence.\n\n---\n\n**Final answer:**\n\nAnswer: 1854",
"turns": 1,
"cacheHitRatio": 0,
"costUsd": 0.00127253,
"claudeEquivalentUsd": 0.017292,
"harvestedTurns": 0,
"totalSubgoals": 0,
"totalUncertainties": 0,
"totalHypotheses": 0,
"totalRejectedPaths": 0
},
{
"taskId": "derangements_d7",
"mode": "reasoner",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "The number of derangements of \\(n\\) distinct objects, denoted \\(D_n\\), counts permutations with no fixed points. For \\(n = 7\\), this can be computed using the recurrence \\(D_n = (n-1)(D_{n-1} + D_{n-2})\\) with initial values \\(D_1 = 0\\) and \\(D_2 = 1\\). \nThus: \n\\(D_3 = 2(1+0) = 2\\), \n\\(D_4 = 3(2+1) = 9\\), \n\\(D_5 = 4(9+2) = 44\\), \n\\(D_6 = 5(44+9) = 265\\), \n\\(D_7 = 6(265+44) = 1854\\).\n\nAnswer: 1854",
"turns": 1,
"cacheHitRatio": 0.7191011235955056,
"costUsd": 0.00349167,
"claudeEquivalentUsd": 0.024027,
"harvestedTurns": 0,
"totalSubgoals": 0,
"totalUncertainties": 0,
"totalHypotheses": 0,
"totalRejectedPaths": 0
},
{
"taskId": "derangements_d7",
"mode": "reasoner-harvest",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "The number of derangements of \\(n\\) elements, denoted \\(D_n\\), counts permutations with no fixed points. For \\(n=7\\), this can be computed using the recurrence \\(D_n = (n-1)(D_{n-1} + D_{n-2})\\) with initial values \\(D_0 = 1\\) and \\(D_1 = 0\\). Applying the recurrence yields \\(D_2 = 1\\), \\(D_3 = 2\\), \\(D_4 = 9\\), \\(D_5 = 44\\), \\(D_6 = 265\\), and finally \\(D_7 = 6 \\times (265 + 44) = 1854\\).\n\nAnswer: 1854",
"turns": 1,
"cacheHitRatio": 0.7191011235955056,
"costUsd": 0.0028434299999999997,
"claudeEquivalentUsd": 0.019587,
"harvestedTurns": 1,
"totalSubgoals": 3,
"totalUncertainties": 2,
"totalHypotheses": 2,
"totalRejectedPaths": 2
},
{
"taskId": "euler_quadratic_break",
"mode": "baseline",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "Let’s check \\( f(n) = n^2 + n + 41 \\) for small \\( n \\):\n\n- \\( n = 0 \\): \\( f(0) = 41 \\) (prime) \n- \\( n = 1 \\): \\( f(1) = 1 + 1 + 41 = 43 \\) (prime) \n- \\( n = 2 \\): \\( f(2) = 4 + 2 + 41 = 47 \\) (prime) \n- \\( n = 3 \\): \\( f(3) = 9 + 3 + 41 = 53 \\) (prime) \n- \\( n = 4 \\): \\( f(4) = 16 + 4 + 41 = 61 \\) (prime) \n- \\( n = 5 \\): \\( f(5) = 25 + 5 + 41 = 71 \\) (prime) \n- \\( n = 6 \\): \\( f(6) = 36 + 6 + 41 = 83 \\) (prime) \n- \\( n = 7 \\): \\( f(7) = 49 + 7 + 41 = 97 \\) (prime) \n- \\( n = 8 \\): \\( f(8) = 64 + 8 + 41 = 113 \\) (prime) \n- \\( n = 9 \\): \\( f(9) = 81 + 9 + 41 = 131 \\) (prime) \n- \\( n = 10 \\): \\( f(10) = 100 + 10 + 41 = 151 \\) (prime) \n\nWe can keep going, but the first known counterexample for this famous Euler polynomial is \\( n = 40 \\), because \n\\[\nf(40) = 40^2 + 40 + 41 = 1600 + 40 + 41 = 1681.\n\\] \nCheck if \\( 1681 \\) is prime: \n\\( 1681 = 41^2 \\) (since \\( 41 \\times 41 = 1681 \\)), so it’s composite.\n\nBut wait — let’s check if there’s a smaller \\( n \\) where \\( f(n) \\) is composite. \nWe can test \\( n = 41 \\): \n\\( f(41) = 41^2 + 41 + 41 = 1681 + 82 = 1763 \\), which is \\( 41 \\times 43 \\) (since \\( 1763/41 = 43 \\)), so composite. But \\( n = 41 \\) is larger than \\( n = 40 \\).\n\nCheck \\( n = 39 \\): \\( f(39) = 1521 + 39 + 41 = 1601 \\). Test divisibility: not divisible by primes up to \\( \\sqrt{1601} \\approx 40 \\), seems prime. \n\nCheck \\( n = 40 \\) is indeed the first composite? Let’s check \\( n = 1 \\) to \\( 39 \\) quickly for a smaller counterexample: \nWe can note \\( f(n) = n(n+1) + 41 \\). If \\( 41 \\mid f(n) \\), then \\( 41 \\mid n(n+1) \\). \nThat happens when \\( n \\equiv 0 \\pmod{41} \\) or \\( n \\equiv 40 \\pmod{41} \\). \nThe smallest positive such \\( n \\) is \\( n = 40 \\) (since \\( n=41 \\) is larger). \nSo for \\( n = 40 \\), \\( f(40) = 40 \\times 41 + 41 = 41 \\times 41 \\) is composite. \n\nCheck \\( n = 0 \\) to \\( 39 \\): no multiple of 41 except \\( n=0 \\) (which gives 41, prime). So \\( n=40 \\) is indeed the first nonnegative integer where \\( f(n) \\) is composite.\n\nThus smallest \\( n \\) is \\( 40 \\), \\( f(40) = 1681 = 41^2 \\).\n\n**Answer:** 40",
"turns": 1,
"cacheHitRatio": 0,
"costUsd": 0.00098535,
"claudeEquivalentUsd": 0.013365,
"harvestedTurns": 0,
"totalSubgoals": 0,
"totalUncertainties": 0,
"totalHypotheses": 0,
"totalRejectedPaths": 0
},
{
"taskId": "euler_quadratic_break",
"mode": "reasoner",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "The smallest nonnegative integer \\(n\\) for which \\(f(n) = n^2 + n + 41\\) is composite is \\(n = 40\\). At this value, \\(f(40) = 40^2 + 40 + 41 = 1681 = 41^2\\), which is not prime.\n\nAnswer: 40",
"turns": 1,
"cacheHitRatio": 0.6095238095238096,
"costUsd": 0.0014309199999999998,
"claudeEquivalentUsd": 0.0099,
"harvestedTurns": 0,
"totalSubgoals": 0,
"totalUncertainties": 0,
"totalHypotheses": 0,
"totalRejectedPaths": 0
},
{
"taskId": "euler_quadratic_break",
"mode": "reasoner-harvest",
"repeat": 1,
"verdict": "pass",
"finalAgentMessage": "The polynomial \\(f(n) = n^2 + n + 41\\) is known to produce primes for \\(n = 0, 1, 2, \\dots, 39\\). At \\(n = 40\\), we have \\(f(40) = 40^2 + 40 + 41 = 1681\\), and \\(1681 = 41^2\\), which is composite. Therefore, \\(n = 40\\) is the smallest nonnegative integer where \\(f(n)\\) fails to be prime.\n\nAnswer: 40",
"turns": 1,
"cacheHitRatio": 0.6095238095238096,
"costUsd": 0.0025938099999999998,
"claudeEquivalentUsd": 0.017865,
"harvestedTurns": 1,
"totalSubgoals": 3,
"totalUncertainties": 2,
"totalHypotheses": 3,
"totalRejectedPaths": 2
}
]
}
+312
View File
@@ -0,0 +1,312 @@
/**
* Harvest-bench runner.
*
* Usage:
* npx tsx benchmarks/harvest/runner.ts # all tasks, all modes, 1 repeat
* npx tsx benchmarks/harvest/runner.ts --task mod7_list
* npx tsx benchmarks/harvest/runner.ts --mode reasoner-harvest --repeats 3
* npx tsx benchmarks/harvest/runner.ts --transcripts-dir transcripts/
* npx tsx benchmarks/harvest/runner.ts --dry # smoke test wiring, no API
*
* Writes results.json. Render report.md with `report.ts`.
*/
import { type WriteStream, mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { pathToFileURL } from "node:url";
import {
CacheFirstLoop,
DeepSeekClient,
ImmutablePrefix,
claudeEquivalentCost,
loadDotenv,
} from "../../src/index.js";
import { openTranscriptFile, recordFromLoopEvent, writeRecord } from "../../src/transcript.js";
import { TASKS } from "./tasks.js";
import type {
HarvestBenchMeta,
HarvestBenchReport,
HarvestMode,
HarvestRunResult,
HarvestTask,
} from "./types.js";
import { ALL_MODES } from "./types.js";
loadDotenv();
interface CliArgs {
taskFilter: string | null;
modes: HarvestMode[];
repeats: number;
outPath: string | null;
transcriptsDir: string | null;
/** Per-HTTP-call timeout in ms. Default 300_000 (5 min) — reasoner + harvest runs legitimately need this. */
timeoutMs: number;
dry: boolean;
verbose: boolean;
}
function parseArgs(argv: string[]): CliArgs {
const out: CliArgs = {
taskFilter: null,
modes: [...ALL_MODES],
repeats: 1,
outPath: null,
transcriptsDir: null,
timeoutMs: 300_000,
dry: false,
verbose: false,
};
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === "--task") out.taskFilter = argv[++i] ?? null;
else if (a === "--mode") {
const v = (argv[++i] ?? "").toLowerCase() as HarvestMode;
if ((ALL_MODES as string[]).includes(v)) out.modes = [v];
} else if (a === "--repeats") out.repeats = Number.parseInt(argv[++i] ?? "1", 10);
else if (a === "--out") out.outPath = argv[++i] ?? null;
else if (a === "--transcripts-dir") out.transcriptsDir = argv[++i] ?? null;
else if (a === "--timeout") {
// Accept seconds for humans, convert to ms internally.
const secs = Number.parseInt(argv[++i] ?? "300", 10);
if (Number.isFinite(secs) && secs > 0) out.timeoutMs = secs * 1000;
} else if (a === "--dry") out.dry = true;
else if (a === "--verbose" || a === "-v") out.verbose = true;
}
return out;
}
// ---------- model resolution ----------
function modelForMode(mode: HarvestMode): string {
if (mode === "baseline") return "deepseek-chat";
return "deepseek-reasoner";
}
function harvestForMode(mode: HarvestMode): boolean {
return mode === "reasoner-harvest";
}
// ---------- run one task in one mode ----------
async function runOnce(
task: HarvestTask,
mode: HarvestMode,
rep: number,
client: DeepSeekClient,
transcriptStream: WriteStream | null,
): Promise<HarvestRunResult> {
const model = modelForMode(mode);
const harvest = harvestForMode(mode);
const prefix = new ImmutablePrefix({ system: task.systemPrompt });
const loop = new CacheFirstLoop({
client,
prefix,
model,
harvest,
stream: false,
});
const prefixHash = prefix.fingerprint;
let harvestedTurns = 0;
let totalSubgoals = 0;
let totalUncertainties = 0;
let totalHypotheses = 0;
let totalRejectedPaths = 0;
let finalText = "";
let errorMessage: string | undefined;
if (transcriptStream) {
writeRecord(transcriptStream, {
ts: new Date().toISOString(),
turn: 1,
role: "user",
content: task.prompt,
});
}
try {
for await (const ev of loop.step(task.prompt)) {
if (transcriptStream && ev.role !== "assistant_delta") {
writeRecord(transcriptStream, recordFromLoopEvent(ev, { model, prefixHash }));
}
if (ev.role === "assistant_final") {
finalText = ev.content;
if (ev.planState) {
if (
ev.planState.subgoals.length +
ev.planState.hypotheses.length +
ev.planState.uncertainties.length +
ev.planState.rejectedPaths.length >
0
) {
harvestedTurns++;
totalSubgoals += ev.planState.subgoals.length;
totalHypotheses += ev.planState.hypotheses.length;
totalUncertainties += ev.planState.uncertainties.length;
totalRejectedPaths += ev.planState.rejectedPaths.length;
}
}
} else if (ev.role === "done") {
finalText = ev.content || finalText;
break;
} else if (ev.role === "error") {
throw new Error(ev.error ?? "loop error");
}
}
} catch (err) {
errorMessage = (err as Error).message;
}
const verdict = errorMessage ? { verdict: "fail" as const, note: errorMessage } : task.check(finalText);
return {
taskId: task.id,
mode,
repeat: rep + 1,
verdict: verdict.verdict,
checkNote: verdict.note,
finalAgentMessage: finalText,
turns: loop.stats.turns.length,
cacheHitRatio: loop.stats.aggregateCacheHitRatio,
costUsd: loop.stats.totalCost,
claudeEquivalentUsd: loop.stats.turns.reduce((s, t) => s + claudeEquivalentCost(t.usage), 0),
harvestedTurns,
totalSubgoals,
totalUncertainties,
totalHypotheses,
totalRejectedPaths,
errorMessage,
};
}
// ---------- dry mode ----------
function runDry(args: CliArgs): HarvestBenchReport {
const tasks = filterTasks(args.taskFilter);
const results: HarvestRunResult[] = [];
for (const task of tasks) {
// Exercise each checker with a deliberately wrong reply so we know
// the wiring is live — not a pass on empty.
const sanity = task.check("Answer: this is not the right answer at all");
console.log(`[${task.id}] checker exercises ok (sanity verdict=${sanity.verdict})`);
for (const mode of args.modes) {
results.push({
taskId: task.id,
mode,
repeat: 1,
verdict: "inconclusive",
checkNote: "dry-run (no model call)",
finalAgentMessage: "[dry-run]",
turns: 0,
cacheHitRatio: 0,
costUsd: 0,
claudeEquivalentUsd: 0,
harvestedTurns: 0,
totalSubgoals: 0,
totalUncertainties: 0,
totalHypotheses: 0,
totalRejectedPaths: 0,
});
}
}
return { meta: buildMeta(args, tasks.length), results };
}
// ---------- main ----------
function filterTasks(filter: string | null): HarvestTask[] {
if (!filter) return TASKS;
const t = TASKS.find((x) => x.id === filter);
if (!t) throw new Error(`unknown task: ${filter}`);
return [t];
}
function buildMeta(args: CliArgs, taskCount: number): HarvestBenchMeta {
return {
date: new Date().toISOString(),
repeatsPerTask: args.repeats,
taskCount,
modesRun: args.modes,
reasonixVersion: "0.2.2",
};
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
if (args.dry) {
const report = runDry(args);
writeReport(report, args.outPath);
return;
}
const client = new DeepSeekClient({ timeoutMs: args.timeoutMs });
const tasks = filterTasks(args.taskFilter);
if (args.transcriptsDir) mkdirSync(args.transcriptsDir, { recursive: true });
const results: HarvestRunResult[] = [];
for (const task of tasks) {
for (let rep = 0; rep < args.repeats; rep++) {
for (const mode of args.modes) {
let stream: WriteStream | null = null;
if (args.transcriptsDir) {
const fname = `${task.id}.${mode}.r${rep + 1}.jsonl`;
stream = openTranscriptFile(join(args.transcriptsDir, fname), {
version: 1,
source: `harvest-bench/${mode}`,
model: modelForMode(mode),
task: task.id,
mode,
repeat: rep + 1,
startedAt: new Date().toISOString(),
});
}
const started = Date.now();
let result: HarvestRunResult;
try {
result = await runOnce(task, mode, rep, client, stream);
} finally {
stream?.end();
}
const elapsed = ((Date.now() - started) / 1000).toFixed(1);
console.log(
`[${task.id}/${mode}/r${rep + 1}] verdict=${result.verdict}` +
` cache=${(result.cacheHitRatio * 100).toFixed(1)}%` +
` cost=$${result.costUsd.toFixed(6)}` +
` harvest=${result.harvestedTurns}:${result.totalSubgoals}s/${result.totalUncertainties}u` +
` (${elapsed}s)` +
(result.checkNote ? ` ${result.checkNote}` : ""),
);
results.push(result);
}
}
}
const report: HarvestBenchReport = { meta: buildMeta(args, tasks.length), results };
writeReport(report, args.outPath);
}
function writeReport(report: HarvestBenchReport, outPath: string | null): void {
const path =
outPath ?? `benchmarks/harvest/results-${new Date().toISOString().replace(/[:.]/g, "-")}.json`;
writeFileSync(path, `${JSON.stringify(report, null, 2)}\n`, "utf8");
console.log(`wrote ${path}`);
}
function isMain(): boolean {
const entry = process.argv[1];
if (!entry) return false;
return import.meta.url === pathToFileURL(entry).href;
}
if (isMain()) {
main().catch((err) => {
console.error(err);
process.exit(1);
});
}
export { main as runHarvestBench };
+171
View File
@@ -0,0 +1,171 @@
/**
* Seed reasoning tasks for the harvest eval harness.
*
* Each task has:
* - a minimal system prompt (so the comparison is about the model, not
* the prompt)
* - a single user question
* - a deterministic checker (regex + set/value compare)
*
* When adding a new task, prefer problems where:
* - the answer has a clean shape (a number, a small set, a yes/no +
* explanation) — easier to check
* - reasoning actually matters (so harvest has signal to extract);
* trivia questions are out of scope
* - there's a known-good answer; judgment calls are out of scope
*/
import type { HarvestTask } from "./types.js";
const CONCISE_SYSTEM = `You are a precise reasoner. Think carefully, then state your final answer on a clearly-labeled line starting with "Answer:".`;
/**
* Parse all non-negative integers out of a string. Handles commas,
* ranges written as "2-4" (expands to 2,3,4), and ignores numbers that
* look like formula references (e.g. "n^2", "mod 7") by only accepting
* integers after we strip common LaTeX decorations.
*/
function extractIntegers(text: string): number[] {
// Strip LaTeX delimiters so n_1 / n^2 / \{ ... \} don't confuse us.
const cleaned = text
.replace(/\\[a-zA-Z]+/g, " ") // \frac, \mod etc.
.replace(/[{}\\]/g, " ")
.replace(/[_^]\d+/g, " "); // subscript/superscript numbers
// Expand ranges: "2-4" → "2,3,4" (only for small gaps)
const rangeExpanded = cleaned.replace(/(\d+)\s*[-–]\s*(\d+)/g, (_, a, b) => {
const lo = Number.parseInt(a, 10);
const hi = Number.parseInt(b, 10);
if (!Number.isFinite(lo) || !Number.isFinite(hi) || hi - lo > 50) return `${a} ${b}`;
const out: number[] = [];
for (let i = Math.min(lo, hi); i <= Math.max(lo, hi); i++) out.push(i);
return out.join(" ");
});
const matches = rangeExpanded.match(/-?\d+/g) ?? [];
return matches.map((m) => Number.parseInt(m, 10)).filter((n) => Number.isFinite(n));
}
/** Extract content after the last "Answer:" label (case-insensitive). */
function extractAnswerLine(text: string): string {
const matches = [...text.matchAll(/answer\s*[::]\s*([^\n]+)/gi)];
if (matches.length === 0) return text;
return matches[matches.length - 1]![1]!.trim();
}
// ---------- expected answers ----------
/** Positive n ≤ 100 such that n^2 + n + 1 ≡ 0 (mod 7). */
const MOD7_EXPECTED: number[] = (() => {
const out: number[] = [];
for (let n = 1; n <= 100; n++) {
if ((n * n + n + 1) % 7 === 0) out.push(n);
}
return out;
})();
/**
* Single-integer checker. Pulls the first plausible integer from the last
* "Answer:" line; compares for exact equality.
*/
function checkSingleInt(expected: number) {
return (reply: string) => {
const answerLine = extractAnswerLine(reply);
const nums = extractIntegers(answerLine);
if (nums.length === 0) return { verdict: "fail" as const, note: "no number in Answer line" };
const n = nums[0]!;
if (n === expected) return { verdict: "pass" as const };
return { verdict: "fail" as const, note: `got ${n}, expected ${expected}` };
};
}
// ---------- tasks ----------
export const TASKS: HarvestTask[] = [
{
id: "mod7_list",
description:
"Number theory — find all positive n ≤ 100 with n^2+n+1 ≡ 0 (mod 7). 29-element set. Classic R1 scratch-work territory.",
systemPrompt: CONCISE_SYSTEM,
prompt:
"Find all positive integers n with n ≤ 100 such that n^2 + n + 1 is divisible by 7. Briefly justify, then end with the full list on a line starting with \"Answer:\".",
check: (reply) => {
const answerLine = extractAnswerLine(reply);
const found = new Set(extractIntegers(answerLine).filter((n) => n >= 1 && n <= 100));
const expected = new Set(MOD7_EXPECTED);
if (found.size === 0) {
return { verdict: "fail", note: "no integers extracted from Answer line" };
}
// Exact set equality — both containment directions.
for (const n of expected) if (!found.has(n)) return { verdict: "fail", note: `missing n=${n}` };
for (const n of found) if (!expected.has(n)) return { verdict: "fail", note: `extra n=${n}` };
return { verdict: "pass" };
},
},
{
id: "flips_until_3heads",
description:
"Probability — expected coin flips until 3 consecutive heads. Answer: 14. Tests whether R1 derives via recurrence or via memorized result.",
systemPrompt: CONCISE_SYSTEM,
prompt:
"A fair coin is flipped repeatedly until you see 3 heads in a row. What is the expected number of total flips? Give a single integer on a line starting with \"Answer:\".",
check: (reply) => {
const answerLine = extractAnswerLine(reply);
const nums = extractIntegers(answerLine);
if (nums.length === 0) return { verdict: "fail", note: "no number in Answer line" };
// Accept the first sensible number — sometimes agents write "Answer: 14 flips".
const n = nums[0]!;
if (n === 14) return { verdict: "pass" };
return { verdict: "fail", note: `got ${n}, expected 14` };
},
},
{
id: "three_hats",
description:
"Logic — three people wear red/blue hats, at least one red, they're asked in order. First two say no, third says yes. What color? Answer: red (deducible because if third were blue, the first two could see one red + one unknown and wouldn't have said no on their full first round, for most configurations the third can only be certain if red).",
systemPrompt: CONCISE_SYSTEM,
prompt:
'Three people wear hats that are each red or blue. They are told at least one hat is red. Each can see the other two hats but not their own. They are asked in order: "Do you know your hat color?" The first two answer no. The third answers yes. What color is the third person\'s hat? Give a one-word answer on a line starting with "Answer:".',
check: (reply) => {
const answerLine = extractAnswerLine(reply).toLowerCase();
// Accept "red" as a standalone word (not "redirected" etc).
if (/\bred\b/.test(answerLine) && !/\bblue\b/.test(answerLine)) {
return { verdict: "pass" };
}
if (/\bblue\b/.test(answerLine)) return { verdict: "fail", note: "answered blue" };
return { verdict: "fail", note: "no clear color in Answer line" };
},
},
// ---------- hard tasks (picked for known V3 failure modes) ----------
{
id: "pseudoprime_base2",
description:
"Number theory — smallest composite n > 1 such that 2^n ≡ 2 (mod n). This is the smallest Fermat pseudoprime to base 2. Answer: 341 (= 11·31). V3 frequently answers 561 (Carmichael / wrong base-2 property) or 9 (not actually a pseudoprime to base 2, verifying the model isn't computing). R1 usually gets this right.",
systemPrompt: CONCISE_SYSTEM,
prompt:
'Find the smallest composite integer n > 1 such that 2^n is congruent to 2 modulo n (i.e., n is a Fermat pseudoprime to base 2). Briefly justify, then give a single integer on a line starting with "Answer:".',
check: checkSingleInt(341),
},
{
id: "derangements_d7",
description:
"Combinatorics — number of derangements of 7 elements. Answer: 1854 (D_7 = 7! · (1 − 1/1! + 1/2! − 1/3! + … ± 1/7!) = 5040 · 0.367857… = 1854). V3 sometimes approximates via n!/e and rounds wrong, or confuses with D_6=265 or D_8=14833. R1's explicit recurrence D_n = (n-1)(D_{n-1}+D_{n-2}) avoids the mistake.",
systemPrompt: CONCISE_SYSTEM,
prompt:
'How many permutations of {1, 2, 3, 4, 5, 6, 7} have no fixed points? (This is D_7, the derangement number.) Briefly justify, then give a single integer on a line starting with "Answer:".',
check: checkSingleInt(1854),
},
{
id: "euler_quadratic_break",
description:
"Number theory — smallest nonneg integer n where n^2 + n + 41 is NOT prime. Euler's prime-generating polynomial famously produces primes for n = 0..39 and first fails at n = 40 (40²+40+41 = 1681 = 41²). V3 often confuses this with the related n²-n+41 (first fails at 41) or cites the wrong threshold. R1 checks small n systematically.",
systemPrompt: CONCISE_SYSTEM,
prompt:
'Consider the polynomial f(n) = n^2 + n + 41 for n = 0, 1, 2, …. What is the smallest nonneg integer n at which f(n) fails to be prime? Briefly justify (including the value of f(n) at that point), then give just the integer n on a line starting with "Answer:".',
check: checkSingleInt(40),
},
];
+82
View File
@@ -0,0 +1,82 @@
/**
* Types for the harvest eval harness.
*
* Scope: this is deliberately NOT τ-bench-lite. These tasks:
* - are single-turn Q/A (no user simulator, no DB)
* - target reasoning-heavy problems (math, logic, planning)
* - use deterministic checkers (regex extraction + set/value compare),
* never an LLM judge
*
* The point is to measure Pillar 2 (harvest) the same way τ-bench-lite
* measures Pillar 1 (cache-first): isolate one variable, produce numbers
* anyone can reproduce.
*/
export type HarvestMode =
/** deepseek-chat, no Reasonix tricks. Floor reference for answer quality on reasoning tasks (V3 at its best). */
| "baseline"
/** deepseek-reasoner, no harvest. Isolates "does reasoner help over chat on these problems?" */
| "reasoner"
/** deepseek-reasoner + harvest. Isolates "does the extra V3 harvest call add quality / change behavior?" */
| "reasoner-harvest";
export const ALL_MODES: HarvestMode[] = ["baseline", "reasoner", "reasoner-harvest"];
/**
* Checker verdict. We don't use boolean directly because "maybe" is real
* — e.g. "answer in the right shape but can't verify without running the
* program". For v0.3 we only use {pass, fail}, but the shape leaves room.
*/
export type CheckVerdict = "pass" | "fail" | "inconclusive";
export interface HarvestTask {
id: string;
/** One-line human description, not shown to the model. */
description: string;
/** System prompt. Kept minimal so harvest signal can be attributed to the model, not to prompt engineering. */
systemPrompt: string;
/** The single question. */
prompt: string;
/**
* Check the agent's final reply. Returns pass/fail/inconclusive plus an
* optional one-line explanation the report can surface.
*
* IMPORTANT: must be pure and deterministic. No LLM calls, no file I/O.
*/
check: (agentReply: string) => { verdict: CheckVerdict; note?: string };
}
export interface HarvestRunResult {
taskId: string;
mode: HarvestMode;
repeat: number;
verdict: CheckVerdict;
checkNote?: string;
finalAgentMessage: string;
/** Turns the loop completed (usually 1 for a single-shot Q/A, > 1 if tool-calls happen). */
turns: number;
cacheHitRatio: number;
costUsd: number;
claudeEquivalentUsd: number;
/** Harvest-specific — only non-zero on reasoner-harvest mode. */
harvestedTurns: number;
totalSubgoals: number;
totalUncertainties: number;
totalHypotheses: number;
totalRejectedPaths: number;
/** Optional error message if the run crashed (model down, timeout, etc). */
errorMessage?: string;
}
export interface HarvestBenchMeta {
date: string;
repeatsPerTask: number;
taskCount: number;
modesRun: HarvestMode[];
reasonixVersion: string;
}
export interface HarvestBenchReport {
meta: HarvestBenchMeta;
results: HarvestRunResult[];
}
+178
View File
@@ -0,0 +1,178 @@
/**
* Naive baseline runner — deliberately does NOT use CacheFirstLoop.
*
* For the benchmark comparison to be honest, this runner must reproduce
* what a "typical" agent framework does that breaks DeepSeek's automatic
* prefix cache:
*
* 1. Inject a fresh timestamp into the system prompt every turn.
* 2. Re-serialize the tool list each turn with a shuffled key order.
* 3. Rebuild the full message array each turn rather than appending.
*
* These are all things frameworks like LangChain/LangGraph do by default
* (either directly, or because tool specs are regenerated from Python dicts
* with non-deterministic ordering, or because a "current_time" placeholder
* is recommended in common system-prompt recipes).
*
* Reuses DeepSeekClient, ToolRegistry, and SessionStats so the *only*
* difference from the Reasonix run is prefix stability.
*/
import {
type ChatMessage,
type DeepSeekClient,
SessionStats,
type ToolCall,
type ToolDefinition,
ToolRegistry,
type ToolSpec,
Usage,
} from "../../src/index.js";
import type { Turn } from "./types.js";
export interface BaselineRunnerOptions {
client: DeepSeekClient;
systemPrompt: string;
tools: ToolDefinition[];
model?: string;
maxToolIters?: number;
}
export interface BaselineSubCall {
/** Assistant text from this sub-call (often empty when the response is tool-calls-only). */
content: string;
/** Usage for this single client.chat() call. */
usage: Usage;
/** Tools the model chose to call on the back of this response. */
toolCalls: { name: string; args: string; result: string }[];
}
export interface BaselineTurnResult {
assistantMessage: string;
toolCallsExecuted: { name: string; args: string; result: string }[];
/**
* Per-sub-call breakdown, one entry per client.chat() invocation. Gives
* downstream tools (bench transcripts) the same granularity as Reasonix
* loop events — without it, diff can't compare model-call counts
* apples-to-apples.
*/
subCalls: BaselineSubCall[];
/** Turn number (1-based) assigned by the agent. */
turnNo: number;
}
export class BaselineAgent {
readonly client: DeepSeekClient;
readonly stats = new SessionStats();
private readonly systemPrompt: string;
private readonly registry: ToolRegistry;
private readonly model: string;
private readonly maxToolIters: number;
/**
* Previous-turn messages. A naive framework keeps these, but we still
* rebuild the prefix around them each turn so the byte prefix churns.
*/
private history: ChatMessage[] = [];
private turnNo = 0;
constructor(opts: BaselineRunnerOptions) {
this.client = opts.client;
this.systemPrompt = opts.systemPrompt;
this.model = opts.model ?? "deepseek-chat";
this.maxToolIters = opts.maxToolIters ?? 6;
this.registry = new ToolRegistry({ autoFlatten: false });
for (const t of opts.tools) this.registry.register(t);
}
/**
* Run one user-turn: sends the user message, lets the model do tool
* calls until it stops, returns the final assistant text.
*
* Intentionally non-cache-friendly: rebuilds the prefix with a fresh
* timestamp + re-shuffled tool specs every turn.
*/
async userTurn(userMessage: string, transcript: Turn[]): Promise<BaselineTurnResult> {
this.turnNo++;
// Naive pattern #1: current-time placeholder in the system prompt.
const churnedSystem = `${this.systemPrompt}\nCurrent time: ${new Date().toISOString()}`;
// Naive pattern #2: shuffle tool spec order each turn (simulates
// frameworks that materialize tools from Python dicts / maps).
const shuffledTools = shuffle(this.registry.specs(), this.turnNo);
this.history.push({ role: "user", content: userMessage });
const toolExecutions: { name: string; args: string; result: string }[] = [];
const subCalls: BaselineSubCall[] = [];
for (let iter = 0; iter < this.maxToolIters; iter++) {
// Naive pattern #3: always rebuild the full message array.
const messages: ChatMessage[] = [{ role: "system", content: churnedSystem }, ...this.history];
const resp = await this.client.chat({
model: this.model,
messages,
tools: shuffledTools,
});
this.stats.record(this.turnNo, this.model, resp.usage);
const assistantMessage: ChatMessage = { role: "assistant", content: resp.content };
if (resp.toolCalls.length > 0) assistantMessage.tool_calls = resp.toolCalls;
this.history.push(assistantMessage);
if (resp.toolCalls.length === 0) {
subCalls.push({ content: resp.content, usage: resp.usage, toolCalls: [] });
return {
assistantMessage: resp.content,
toolCallsExecuted: toolExecutions,
subCalls,
turnNo: this.turnNo,
};
}
const subToolCalls: { name: string; args: string; result: string }[] = [];
for (const tc of resp.toolCalls) {
const name = tc.function?.name ?? "";
const args = tc.function?.arguments ?? "{}";
const result = await this.registry.dispatch(name, args);
toolExecutions.push({ name, args, result });
subToolCalls.push({ name, args, result });
this.history.push({
role: "tool",
tool_call_id: tc.id ?? "",
name,
content: result,
});
}
subCalls.push({ content: resp.content, usage: resp.usage, toolCalls: subToolCalls });
}
const lastAssistant = [...this.history].reverse().find((m) => m.role === "assistant");
return {
assistantMessage: lastAssistant?.content ?? "[max_tool_iters reached]",
toolCallsExecuted: toolExecutions,
subCalls,
turnNo: this.turnNo,
};
}
}
/**
* Deterministic Fisher–Yates shuffle seeded by an integer. Same turn number
* → same ordering, so the baseline is reproducible across runs while still
* being cache-hostile (different turns get different orderings).
*/
function shuffle<T>(arr: T[], seed: number): T[] {
const out = [...arr];
let s = seed * 9301 + 49297;
for (let i = out.length - 1; i > 0; i--) {
s = (s * 9301 + 49297) % 233280;
const j = Math.floor((s / 233280) * (i + 1));
[out[i], out[j]] = [out[j]!, out[i]!];
}
return out;
}
// Re-export ToolCall, ToolSpec so caller files don't need to import both places.
export type { ToolCall, ToolSpec };
+30
View File
@@ -0,0 +1,30 @@
import type { WorldState } from "./types.js";
/**
* Deep-clone a WorldState so per-run mutations stay isolated.
* structuredClone is enough — WorldState is JSON-shaped by contract.
*/
export function cloneDb(db: WorldState): WorldState {
return structuredClone(db);
}
export function getRow(
db: WorldState,
table: string,
id: string,
): Record<string, unknown> | undefined {
return db[table]?.[id];
}
export function setField(
db: WorldState,
table: string,
id: string,
field: string,
value: unknown,
): boolean {
const row = db[table]?.[id];
if (!row) return false;
row[field] = value;
return true;
}
+100
View File
@@ -0,0 +1,100 @@
# Reasonix tool-use eval (τ-bench-lite)
**Date:** 2026-04-21T14:57:44.906Z
**Agent model:** `deepseek-chat`
**User-simulator model:** `deepseek-chat`
**Tasks:** 8, repeats × 3
**Reasonix version:** 0.2.1
## Summary
| metric | baseline | reasonix | delta |
|---|---:|---:|---:|
| runs | 24 | 24 | — |
| pass rate | 96% | 100% | +4pp |
| cache hit | 46.6% | 94.4% | **+47.7pp** |
| mean cost / task | $0.002599 | $0.001579 | ×0.61 |
| mean turns | 4.4 | 4.6 | — |
| mean tool calls | 3.0 | 2.7 | — |
**Reasonix vs Claude Sonnet 4.6 (estimated, same token counts):**
Claude would cost ~$0.038203 / task, so Reasonix saves ~96.0%.
(This is a *token-count-based estimate*, not a head-to-head quality comparison.)
## Per-task breakdown
| task | mode | pass | turns | tools | cache | cost |
|---|---|:---:|---:|---:|---:|---:|
| t01_address_happy | baseline | ✅ | 3 | 3 | 53.1% | $0.001383 |
| t01_address_happy | reasonix | ✅ | 3 | 3 | 94.4% | $0.000896 |
| t01_address_happy | baseline | ✅ | 3 | 3 | 53.3% | $0.001405 |
| t01_address_happy | reasonix | ✅ | 3 | 2 | 95.0% | $0.000756 |
| t01_address_happy | baseline | ✅ | 3 | 2 | 44.0% | $0.001193 |
| t01_address_happy | reasonix | ✅ | 3 | 2 | 92.6% | $0.000848 |
| t02_address_not_allowed | baseline | ✅ | 8 | 2 | 23.6% | $0.004647 |
| t02_address_not_allowed | reasonix | ✅ | 8 | 4 | 95.6% | $0.003067 |
| t02_address_not_allowed | baseline | ✅ | 8 | 3 | 29.9% | $0.005585 |
| t02_address_not_allowed | reasonix | ✅ | 8 | 2 | 96.2% | $0.002822 |
| t02_address_not_allowed | baseline | ✅ | 8 | 7 | 50.7% | $0.007021 |
| t02_address_not_allowed | reasonix | ✅ | 8 | 3 | 95.8% | $0.003140 |
| t03_cancel_processing | baseline | ✅ | 3 | 3 | 53.3% | $0.001349 |
| t03_cancel_processing | reasonix | ✅ | 3 | 3 | 94.9% | $0.000803 |
| t03_cancel_processing | baseline | ✅ | 2 | 3 | 64.6% | $0.001041 |
| t03_cancel_processing | reasonix | ✅ | 2 | 2 | 93.8% | $0.000563 |
| t03_cancel_processing | baseline | ✅ | 3 | 3 | 53.4% | $0.001332 |
| t03_cancel_processing | reasonix | ✅ | 2 | 2 | 93.8% | $0.000566 |
| t04_refund_delivered | baseline | ✅ | 2 | 2 | 55.9% | $0.001007 |
| t04_refund_delivered | reasonix | ✅ | 2 | 3 | 91.0% | $0.000911 |
| t04_refund_delivered | baseline | ✅ | 3 | 2 | 44.8% | $0.001268 |
| t04_refund_delivered | reasonix | ✅ | 3 | 3 | 93.2% | $0.000941 |
| t04_refund_delivered | baseline | ✅ | 3 | 3 | 53.5% | $0.001469 |
| t04_refund_delivered | reasonix | ✅ | 3 | 3 | 94.4% | $0.000952 |
| t05_refund_not_delivered | baseline | ✅ | 8 | 4 | 41.7% | $0.005369 |
| t05_refund_not_delivered | reasonix | ✅ | 7 | 2 | 96.3% | $0.002212 |
| t05_refund_not_delivered | baseline | ✅ | 8 | 3 | 30.6% | $0.004762 |
| t05_refund_not_delivered | reasonix | ✅ | 8 | 4 | 95.1% | $0.003159 |
| t05_refund_not_delivered | baseline | ✅ | 6 | 2 | 24.8% | $0.003128 |
| t05_refund_not_delivered | reasonix | ✅ | 6 | 2 | 94.7% | $0.001927 |
| t06_multi_order_lookup | baseline | ✅ | 2 | 2 | 52.4% | $0.000913 |
| t06_multi_order_lookup | reasonix | ✅ | 3 | 2 | 92.7% | $0.000988 |
| t06_multi_order_lookup | baseline | ✅ | 3 | 2 | 41.2% | $0.001530 |
| t06_multi_order_lookup | reasonix | ✅ | 3 | 2 | 93.4% | $0.000996 |
| t06_multi_order_lookup | baseline | ✅ | 3 | 4 | 60.9% | $0.001920 |
| t06_multi_order_lookup | reasonix | ✅ | 3 | 2 | 92.2% | $0.000947 |
| t07_wrong_identity | baseline | ❌ | 2 | 2 | 54.9% | $0.000815 |
| t07_wrong_identity | reasonix | ✅ | 8 | 2 | 95.5% | $0.002777 |
| t07_wrong_identity | baseline | ✅ | 8 | 3 | 26.6% | $0.004953 |
| t07_wrong_identity | reasonix | ✅ | 7 | 2 | 95.6% | $0.002356 |
| t07_wrong_identity | baseline | ✅ | 8 | 3 | 24.0% | $0.005210 |
| t07_wrong_identity | reasonix | ✅ | 8 | 2 | 96.3% | $0.002587 |
| t08_address_then_cancel | baseline | ✅ | 3 | 3 | 53.4% | $0.001465 |
| t08_address_then_cancel | reasonix | ✅ | 3 | 4 | 94.6% | $0.001174 |
| t08_address_then_cancel | baseline | ✅ | 3 | 4 | 62.1% | $0.001677 |
| t08_address_then_cancel | reasonix | ✅ | 3 | 5 | 94.1% | $0.001371 |
| t08_address_then_cancel | baseline | ✅ | 3 | 5 | 66.5% | $0.001933 |
| t08_address_then_cancel | reasonix | ✅ | 3 | 4 | 93.5% | $0.001142 |
## Scope & caveats
This is **τ-bench-lite**, not a port of Sierra's upstream τ-bench. Specifically:
- Tasks are hand-authored in the retail domain; the schema mirrors τ-bench
(stateful tools, LLM user-sim, DB-end-state success predicates), so upstream
tasks can later be dropped in without harness changes.
- Every pass/fail judgment is a deterministic DB predicate — no LLM judge.
Refusal tasks pass iff the DB is unchanged.
- The "baseline" deliberately reproduces cache-hostile patterns common in
generic agent frameworks: fresh timestamp in the system prompt each turn,
re-shuffled tool spec ordering per turn. It is **not** a benchmark of
LangChain specifically.
- Claude comparison is a *token-count-based cost estimate* using Anthropic's
public pricing, not a head-to-head quality run.
- User simulator is DeepSeek V3 at T=0.1. Some run-to-run drift is expected;
rerun with `--repeats N` to get a tighter mean.
## Reproducing
1. `export DEEPSEEK_API_KEY=sk-...`
2. `npm install`
3. `npx tsx benchmarks/tau-bench/runner.ts --repeats 3`
4. `npx tsx benchmarks/tau-bench/report.ts benchmarks/tau-bench/results-*.json`
+225
View File
@@ -0,0 +1,225 @@
/**
* Render a results-*.json into a human-readable report.md.
*
* npx tsx benchmarks/tau-bench/report.ts benchmarks/tau-bench/results-<date>.json
* npx tsx benchmarks/tau-bench/report.ts <input.json> --out report.md
*/
import { readFileSync, writeFileSync } from "node:fs";
import { pathToFileURL } from "node:url";
import type { BenchReport, RunMode, RunResult } from "./types.js";
interface CliArgs {
input: string;
outPath: string;
}
function parseArgs(argv: string[]): CliArgs {
const out: CliArgs = { input: "", outPath: "benchmarks/tau-bench/report.md" };
const positional: string[] = [];
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === "--out") out.outPath = argv[++i] ?? out.outPath;
else if (a && !a.startsWith("--")) positional.push(a);
}
out.input = positional[0] ?? "";
if (!out.input) {
throw new Error(
"usage: npx tsx benchmarks/tau-bench/report.ts <results.json> [--out report.md]",
);
}
return out;
}
interface Agg {
runs: number;
passes: number;
avgCache: number;
avgCost: number;
avgClaudeCost: number;
avgTurns: number;
avgToolCalls: number;
}
function aggregate(results: RunResult[]): Agg {
if (results.length === 0) {
return {
runs: 0,
passes: 0,
avgCache: 0,
avgCost: 0,
avgClaudeCost: 0,
avgTurns: 0,
avgToolCalls: 0,
};
}
const passes = results.filter((r) => r.pass).length;
const mean = (fn: (r: RunResult) => number) =>
results.reduce((s, r) => s + fn(r), 0) / results.length;
return {
runs: results.length,
passes,
avgCache: mean((r) => r.cacheHitRatio),
avgCost: mean((r) => r.costUsd),
avgClaudeCost: mean((r) => r.claudeEquivalentUsd),
avgTurns: mean((r) => r.turns),
avgToolCalls: mean((r) => r.toolCalls),
};
}
function renderSummary(report: BenchReport): string {
const byMode: Record<RunMode, RunResult[]> = { baseline: [], reasonix: [] };
for (const r of report.results) byMode[r.mode].push(r);
const b = aggregate(byMode.baseline);
const rx = aggregate(byMode.reasonix);
const costRatio = b.avgCost > 0 ? rx.avgCost / b.avgCost : 0;
const claudeSavings = b.avgClaudeCost > 0 ? (1 - rx.avgCost / b.avgClaudeCost) * 100 : 0;
return `
## Summary
| metric | baseline | reasonix | delta |
|---|---:|---:|---:|
| runs | ${b.runs} | ${rx.runs} | — |
| pass rate | ${pct(b.passes, b.runs)} | ${pct(rx.passes, rx.runs)} | ${signPct(rx.passes, rx.runs, b.passes, b.runs)} |
| cache hit | ${pct1(b.avgCache)} | ${pct1(rx.avgCache)} | **${signPctAbs(rx.avgCache - b.avgCache)}** |
| mean cost / task | $${fmt(b.avgCost, 6)} | $${fmt(rx.avgCost, 6)} | ${costRatio > 0 ? `×${fmt(costRatio, 2)}` : "—"} |
| mean turns | ${fmt(b.avgTurns, 1)} | ${fmt(rx.avgTurns, 1)} | — |
| mean tool calls | ${fmt(b.avgToolCalls, 1)} | ${fmt(rx.avgToolCalls, 1)} | — |
**Reasonix vs Claude Sonnet 4.6 (estimated, same token counts):**
Claude would cost ~$${fmt(rx.avgClaudeCost, 6)} / task, so Reasonix saves ~${fmt(
claudeSavings,
1,
)}%.
(This is a *token-count-based estimate*, not a head-to-head quality comparison.)
`.trim();
}
function renderPerTask(report: BenchReport): string {
const rowsByTask = new Map<string, RunResult[]>();
for (const r of report.results) {
const list = rowsByTask.get(r.taskId) ?? [];
list.push(r);
rowsByTask.set(r.taskId, list);
}
const rows: string[] = [
"| task | mode | pass | turns | tools | cache | cost |",
"|---|---|:---:|---:|---:|---:|---:|",
];
for (const [taskId, runs] of rowsByTask) {
for (const r of runs) {
rows.push(
`| ${taskId} | ${r.mode} | ${r.pass ? "✅" : "❌"} | ${r.turns} | ${r.toolCalls} | ${pct1(
r.cacheHitRatio,
)} | $${fmt(r.costUsd, 6)} |${r.errorMessage ? ` err: ${truncate(r.errorMessage, 40)}` : ""}`,
);
}
}
return `## Per-task breakdown\n\n${rows.join("\n")}`;
}
function renderHeader(report: BenchReport): string {
const m = report.meta;
return `# Reasonix tool-use eval (τ-bench-lite)
**Date:** ${m.date}
**Agent model:** \`${m.model}\`
**User-simulator model:** \`${m.userSimModel}\`
**Tasks:** ${m.taskCount}, repeats × ${m.repeatsPerTask}
**Reasonix version:** ${m.reasonixVersion}
`;
}
function renderCaveats(): string {
return `## Scope & caveats
This is **τ-bench-lite**, not a port of Sierra's upstream τ-bench. Specifically:
- Tasks are hand-authored in the retail domain; the schema mirrors τ-bench
(stateful tools, LLM user-sim, DB-end-state success predicates), so upstream
tasks can later be dropped in without harness changes.
- Every pass/fail judgment is a deterministic DB predicate — no LLM judge.
Refusal tasks pass iff the DB is unchanged.
- The "baseline" deliberately reproduces cache-hostile patterns common in
generic agent frameworks: fresh timestamp in the system prompt each turn,
re-shuffled tool spec ordering per turn. It is **not** a benchmark of
LangChain specifically.
- Claude comparison is a *token-count-based cost estimate* using Anthropic's
public pricing, not a head-to-head quality run.
- User simulator is DeepSeek V3 at T=0.1. Some run-to-run drift is expected;
rerun with \`--repeats N\` to get a tighter mean.
## Reproducing
1. \`export DEEPSEEK_API_KEY=sk-...\`
2. \`npm install\`
3. \`npx tsx benchmarks/tau-bench/runner.ts --repeats 3\`
4. \`npx tsx benchmarks/tau-bench/report.ts benchmarks/tau-bench/results-*.json\`
`;
}
export function renderReport(report: BenchReport): string {
return [
renderHeader(report),
renderSummary(report),
"",
renderPerTask(report),
"",
renderCaveats(),
].join("\n");
}
// ---------- formatting helpers ----------
function pct(num: number, denom: number): string {
if (denom === 0) return "—";
return `${((num / denom) * 100).toFixed(0)}%`;
}
function signPct(num1: number, denom1: number, num2: number, denom2: number): string {
if (denom1 === 0 || denom2 === 0) return "—";
const d = num1 / denom1 - num2 / denom2;
const s = (d * 100).toFixed(0);
return `${d >= 0 ? "+" : ""}${s}pp`;
}
function signPctAbs(diff: number): string {
const s = (diff * 100).toFixed(1);
return `${diff >= 0 ? "+" : ""}${s}pp`;
}
function pct1(x: number): string {
return `${(x * 100).toFixed(1)}%`;
}
function fmt(x: number, digits: number): string {
return x.toFixed(digits);
}
function truncate(s: string, n: number): string {
return s.length > n ? `${s.slice(0, n)}…` : s;
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
const raw = readFileSync(args.input, "utf8");
const report = JSON.parse(raw) as BenchReport;
const md = renderReport(report);
writeFileSync(args.outPath, md, "utf8");
console.log(`wrote ${args.outPath} (${report.results.length} runs)`);
}
function isMain(): boolean {
const entry = process.argv[1];
if (!entry) return false;
return import.meta.url === pathToFileURL(entry).href;
}
if (isMain()) {
main().catch((err) => {
console.error(err);
process.exit(1);
});
}
+684
View File
@@ -0,0 +1,684 @@
{
"meta": {
"date": "2026-04-21T14:57:44.906Z",
"model": "deepseek-chat",
"userSimModel": "deepseek-chat",
"taskCount": 8,
"repeatsPerTask": 3,
"reasonixVersion": "0.2.1"
},
"results": [
{
"taskId": "t01_address_happy",
"mode": "baseline",
"pass": true,
"turns": 3,
"toolCalls": 3,
"cacheHitRatio": 0.531435349940688,
"costUsd": 0.0013829700000000003,
"claudeEquivalentUsd": 0.023388000000000003,
"promptTokens": 5901,
"completionTokens": 379,
"truncated": false,
"finalAgentMessage": "Success! I've updated the shipping address for your order o_1002 from \"22 Oak Rd, NYC, NY 10001\" to \"5 Birch Rd, NYC, NY 10001\". \n\nYour order is still in processing status and will now be shipped to the new address."
},
{
"taskId": "t01_address_happy",
"mode": "reasonix",
"pass": true,
"turns": 3,
"toolCalls": 3,
"cacheHitRatio": 0.9444063143445436,
"costUsd": 0.0008962600000000001,
"claudeEquivalentUsd": 0.023259000000000002,
"promptTokens": 5828,
"completionTokens": 385,
"truncated": false,
"finalAgentMessage": "Success! I've updated the shipping address for your order o_1002 to **5 Birch Rd, NYC, NY 10001**. \n\nYour order is still in processing status and will now be shipped to the new address. Is there anything else you need help with today?"
},
{
"taskId": "t01_address_happy",
"mode": "baseline",
"pass": true,
"turns": 3,
"toolCalls": 3,
"cacheHitRatio": 0.533242645808536,
"costUsd": 0.00140507,
"claudeEquivalentUsd": 0.023703,
"promptTokens": 5881,
"completionTokens": 404,
"truncated": false,
"finalAgentMessage": "Great! I've successfully updated the shipping address for your order o_1002. The new address is now: **5 Birch Rd, NYC, NY 10001**.\n\nThe order is still in \"processing\" status, so it will be shipped to this new address. Is there anything else I can help you with?"
},
{
"taskId": "t01_address_happy",
"mode": "reasonix",
"pass": true,
"turns": 3,
"toolCalls": 2,
"cacheHitRatio": 0.9502183406113537,
"costUsd": 0.0007556,
"claudeEquivalentUsd": 0.01905,
"promptTokens": 4580,
"completionTokens": 354,
"truncated": false,
"finalAgentMessage": "Success! I've updated the shipping address for your order o_1002 to \"5 Birch Rd, NYC, NY 10001\". Your order is still in processing status, and it will now be shipped to the new address. Is there anything else I can help you with today?"
},
{
"taskId": "t01_address_happy",
"mode": "baseline",
"pass": true,
"turns": 3,
"toolCalls": 2,
"cacheHitRatio": 0.44005156854318866,
"costUsd": 0.00119348,
"claudeEquivalentUsd": 0.018687000000000002,
"promptTokens": 4654,
"completionTokens": 315,
"truncated": false,
"finalAgentMessage": "Perfect! I've successfully updated the shipping address for your order o_1002. The new shipping address is now:\n\n**5 Birch Rd, NYC, NY 10001**\n\nYour order is still in processing status and will be shipped to this new address. Is there anything else I can help you with?"
},
{
"taskId": "t01_address_happy",
"mode": "reasonix",
"pass": true,
"turns": 3,
"toolCalls": 2,
"cacheHitRatio": 0.9257604764943629,
"costUsd": 0.00084767,
"claudeEquivalentUsd": 0.020223,
"promptTokens": 4701,
"completionTokens": 408,
"truncated": false,
"finalAgentMessage": "Success! I've updated the shipping address for your order o_1002. The new shipping address is now: **5 Birch Rd, NYC, NY 10001**.\n\nYour order is still in processing status, and the address change has been recorded. Is there anything else I can help you with today?"
},
{
"taskId": "t02_address_not_allowed",
"mode": "baseline",
"pass": true,
"turns": 8,
"toolCalls": 2,
"cacheHitRatio": 0.2358892823588928,
"costUsd": 0.0046473899999999995,
"claudeEquivalentUsd": 0.06284100000000001,
"promptTokens": 13837,
"completionTokens": 1422,
"truncated": true,
"finalAgentMessage": "I understand you're determined to get this resolved, but I need to be completely transparent about what I can and cannot do:\n\n1. **I cannot update the address** - As shown by the system error, I'm physically unable to change addresses for orders with \"shipped\" status. The tool returns an error and won't allow the change.\n\n2. **I cannot transfer you to a supervisor** - I don't have access to any communication tools, phone numbers, or transfer capabilities. I'm an automated support agent with a specific set of order management functions.\n\n3. **I cannot escalate your case** - I don't have escalation procedures or access to human customer service teams.\n\nI'm literally unable to perform either of the actions you're requesting. Continuing to ask won't change my technical limitations - I can only work with the tools I've been given.\n\nThe most helpful thing I can do at this point is to suggest you:\n- Check your email for order confirmation messages (they usually contain customer service contact info)\n- Visit our website and look for \"Contact Us\" or \"Help\" sections\n- Try calling any phone numbers you might have from previous interactions with our company\n\nI apologize for the frustration this situation is causing, but I'm being honest about my limitations. Is there anything else I can help you with using the tools I do have available?"
},
{
"taskId": "t02_address_not_allowed",
"mode": "reasonix",
"pass": true,
"turns": 8,
"toolCalls": 4,
"cacheHitRatio": 0.9560694920880823,
"costUsd": 0.0030673800000000006,
"claudeEquivalentUsd": 0.07663199999999999,
"promptTokens": 18074,
"completionTokens": 1494,
"truncated": true,
"finalAgentMessage": "I understand you want this address change request noted, but I need to be honest - I don't have any tools to add notes to accounts or orders. My available tools are limited to:\n- Looking up orders and user information\n- Updating addresses (only for processing orders)\n- Canceling orders (only for processing orders)\n- Issuing refunds (only for delivered orders)\n- Listing user orders\n\nI can't add notes, comments, or annotations to your account or order. The best I can do is suggest that when the package is delivered, you can contact me again and I can help you with a refund if needed (using the refund_order tool).\n\nFor now, I would strongly recommend:\n1. Contacting the shipping carrier directly with your tracking information\n2. Reaching out to our customer service team via email or phone\n3. Keeping a record of your request for 99 New St, SF, CA\n\nI apologize that I can't provide the note-taking functionality you're looking for, but I want to be transparent about my limitations."
},
{
"taskId": "t02_address_not_allowed",
"mode": "baseline",
"pass": true,
"turns": 8,
"toolCalls": 3,
"cacheHitRatio": 0.29918476711704006,
"costUsd": 0.00558507,
"claudeEquivalentUsd": 0.078513,
"promptTokens": 17541,
"completionTokens": 1726,
"truncated": true,
"finalAgentMessage": "I understand your plan, but I need to be completely clear: I don't have any tools to make notes or annotations on orders or accounts. When you come back for a refund after delivery, I won't have any record of this conversation or your address change request.\n\nHowever, once the order is marked as \"delivered,\" you can contact me again and I'll be able to process a refund for you. At that time, you'll need to provide:\n1. Your name and order ID again\n2. A reason for the refund (like \"delivered to wrong address\")\n\nI recommend you keep track of:\n- Your order ID: o_1001\n- Your desired address: 99 New St, SF, CA\n- The fact that you wanted to change the address before delivery\n\nWhen you return for the refund, just let me know the situation again, and I'll be able to help you with the refund process using the `refund_order` tool.\n\nIs there anything else I can help you with right now?"
},
{
"taskId": "t02_address_not_allowed",
"mode": "reasonix",
"pass": true,
"turns": 8,
"toolCalls": 2,
"cacheHitRatio": 0.9616535218928475,
"costUsd": 0.0028220600000000004,
"claudeEquivalentUsd": 0.06702899999999999,
"promptTokens": 14708,
"completionTokens": 1527,
"truncated": true,
"finalAgentMessage": "As expected, the system returned an error: \"cannot edit: status=shipped\". The update_address tool is designed to only work for orders in \"processing\" status, and since order o_1001 is already marked as \"shipped\", the system won't allow the address change through this automated process.\n\nI've tried exactly what you asked, but the system's rules prevent this change. For shipped orders, address modifications would need to be handled through carrier intercept services or other logistics processes that I don't have access to.\n\nIs there anything else I can help you with using my available tools?"
},
{
"taskId": "t02_address_not_allowed",
"mode": "baseline",
"pass": true,
"turns": 8,
"toolCalls": 7,
"cacheHitRatio": 0.5073297860788359,
"costUsd": 0.0070210900000000015,
"claudeEquivalentUsd": 0.115131,
"promptTokens": 27627,
"completionTokens": 2150,
"truncated": true,
"finalAgentMessage": "As you can see, the refund tool also fails because your order is \"shipped\" (not \"delivered\").\n\nI genuinely don't have any way to record your address change request to \"99 New St, SF, CA\" in the system. I'm an automated agent with very specific, limited tools, and none of them allow me to add notes, escalate issues, or contact human support.\n\nYour only option is to contact customer service directly through the contact methods available on our website or in your account."
},
{
"taskId": "t02_address_not_allowed",
"mode": "reasonix",
"pass": true,
"turns": 8,
"toolCalls": 3,
"cacheHitRatio": 0.958138999251454,
"costUsd": 0.0031397900000000004,
"claudeEquivalentUsd": 0.07635600000000001,
"promptTokens": 17367,
"completionTokens": 1617,
"truncated": true,
"finalAgentMessage": "I can see you have two orders:\n1. o_1001 - wool sweater M - shipped (the one you're asking about)\n2. o_1005 - notebook pack - delivered\n\nUnfortunately, both orders are either shipped or delivered, so I cannot update addresses on either of them. The system only allows address updates for orders with \"processing\" status.\n\nI truly wish I could help you, but I've exhausted all my available tools and functions. I don't have any other methods to try - I can't override the system, contact carriers, escalate to supervisors, or provide contact information.\n\nThe only thing I could potentially do is issue a refund if the order was delivered and you never received it, but that would require the order to be in \"delivered\" status and you'd need to provide a reason for the refund.\n\nI'm sorry I can't be more helpful with your address change request."
},
{
"taskId": "t03_cancel_processing",
"mode": "baseline",
"pass": true,
"turns": 3,
"toolCalls": 3,
"cacheHitRatio": 0.5333333333333333,
"costUsd": 0.0013487000000000002,
"claudeEquivalentUsd": 0.022935000000000004,
"promptTokens": 5880,
"completionTokens": 353,
"truncated": false,
"finalAgentMessage": "Perfect! I've successfully cancelled your order o_1004. The order has been updated to \"cancelled\" status. \n\nYou're all set - no further action is needed on your part. Is there anything else I can help you with?"
},
{
"taskId": "t03_cancel_processing",
"mode": "reasonix",
"pass": true,
"turns": 3,
"toolCalls": 3,
"cacheHitRatio": 0.9489849955869374,
"costUsd": 0.00080305,
"claudeEquivalentUsd": 0.02175,
"promptTokens": 5665,
"completionTokens": 317,
"truncated": false,
"finalAgentMessage": "Your order #o_1004 has been successfully cancelled! No trouble at all - we're happy to help. The cancellation is complete and you should receive a confirmation email shortly. Is there anything else I can assist you with?"
},
{
"taskId": "t03_cancel_processing",
"mode": "baseline",
"pass": true,
"turns": 2,
"toolCalls": 3,
"cacheHitRatio": 0.6460567823343849,
"costUsd": 0.00104125,
"claudeEquivalentUsd": 0.019334999999999998,
"promptTokens": 4755,
"completionTokens": 338,
"truncated": false,
"finalAgentMessage": "Success! I've cancelled your order o_1004. The order has been successfully cancelled and is no longer in processing. You should receive a confirmation email shortly. Is there anything else I can help you with today?"
},
{
"taskId": "t03_cancel_processing",
"mode": "reasonix",
"pass": true,
"turns": 2,
"toolCalls": 2,
"cacheHitRatio": 0.9379310344827586,
"costUsd": 0.0005629000000000001,
"claudeEquivalentUsd": 0.014204999999999999,
"promptTokens": 3480,
"completionTokens": 251,
"truncated": false,
"finalAgentMessage": "Perfect! I've successfully cancelled your order o_1004. The order has been updated to \"cancelled\" status. You should receive a confirmation email shortly. Is there anything else I can help you with?"
},
{
"taskId": "t03_cancel_processing",
"mode": "baseline",
"pass": true,
"turns": 3,
"toolCalls": 3,
"cacheHitRatio": 0.5341509112587294,
"costUsd": 0.0013319700000000002,
"claudeEquivalentUsd": 0.022713,
"promptTokens": 5871,
"completionTokens": 340,
"truncated": false,
"finalAgentMessage": "Your order o_1004 has been successfully cancelled! The status is now showing as \"cancelled\". You should receive a confirmation email shortly.\n\nIs there anything else I can help you with today?"
},
{
"taskId": "t03_cancel_processing",
"mode": "reasonix",
"pass": true,
"turns": 2,
"toolCalls": 2,
"cacheHitRatio": 0.9376615914966964,
"costUsd": 0.00056647,
"claudeEquivalentUsd": 0.014253000000000002,
"promptTokens": 3481,
"completionTokens": 254,
"truncated": false,
"finalAgentMessage": "Perfect! I've successfully cancelled your order (o_1004). The order has been updated to \"cancelled\" status. You should receive a confirmation email shortly. Is there anything else I can help you with today?"
},
{
"taskId": "t04_refund_delivered",
"mode": "baseline",
"pass": true,
"turns": 2,
"toolCalls": 2,
"cacheHitRatio": 0.5592572364827962,
"costUsd": 0.00100704,
"claudeEquivalentUsd": 0.016821000000000003,
"promptTokens": 3662,
"completionTokens": 389,
"truncated": false,
"finalAgentMessage": "Perfect! I've successfully processed your refund request for order o_1003. A refund of $55 has been issued for your desk lamp that arrived broken. The refund has been recorded in our system with the reason \"Item arrived broken/damaged.\"\n\nYou should see the refund reflected in your original payment method within 5-10 business days, depending on your bank or payment processor. Is there anything else I can help you with today?"
},
{
"taskId": "t04_refund_delivered",
"mode": "reasonix",
"pass": true,
"turns": 2,
"toolCalls": 3,
"cacheHitRatio": 0.9100794646591385,
"costUsd": 0.0009113400000000001,
"claudeEquivalentUsd": 0.021036,
"promptTokens": 4782,
"completionTokens": 446,
"truncated": false,
"finalAgentMessage": "Great news! I've successfully processed your refund for order o_1003. A refund of $55 has been issued for your desk lamp that arrived broken. The refund has been recorded in our system and should be processed back to your original payment method according to your bank's processing times.\n\nIs there anything else I can help you with today?"
},
{
"taskId": "t04_refund_delivered",
"mode": "baseline",
"pass": true,
"turns": 3,
"toolCalls": 2,
"cacheHitRatio": 0.4484076433121019,
"costUsd": 0.0012684000000000003,
"claudeEquivalentUsd": 0.019845,
"promptTokens": 4710,
"completionTokens": 381,
"truncated": false,
"finalAgentMessage": "Excellent! Your refund has been successfully processed for order o_1003. A refund of $55 has been issued for your desk lamp that arrived broken. The refund has been recorded in our system. You should see the refund reflected in your original payment method within the next 3-5 business days, depending on your bank or payment provider's processing time.\n\nIs there anything else I can help you with today?"
},
{
"taskId": "t04_refund_delivered",
"mode": "reasonix",
"pass": true,
"turns": 3,
"toolCalls": 3,
"cacheHitRatio": 0.9323050556983719,
"costUsd": 0.0009406500000000001,
"claudeEquivalentUsd": 0.023685,
"promptTokens": 5835,
"completionTokens": 412,
"truncated": false,
"finalAgentMessage": "Great news, Cai Lin! I've successfully processed your refund for order o_1003. A refund of $55 has been issued for your desk lamp that arrived broken. The refund has been recorded in our system and should be processed back to your original payment method within the next few business days.\n\nIs there anything else I can help you with today?"
},
{
"taskId": "t04_refund_delivered",
"mode": "baseline",
"pass": true,
"turns": 3,
"toolCalls": 3,
"cacheHitRatio": 0.5346700083542189,
"costUsd": 0.0014687500000000002,
"claudeEquivalentUsd": 0.024675,
"promptTokens": 5985,
"completionTokens": 448,
"truncated": false,
"finalAgentMessage": "Great news! I've successfully processed a refund for your order o_1003. A refund of $55 has been issued for the desk lamp that arrived broken. The refund has been recorded with the reason \"Item arrived broken.\"\n\nYou should see the refund reflected in your original payment method within 3-10 business days, depending on your bank or payment provider's processing time.\n\nIs there anything else I can help you with today?"
},
{
"taskId": "t04_refund_delivered",
"mode": "reasonix",
"pass": true,
"turns": 3,
"toolCalls": 3,
"cacheHitRatio": 0.944082332761578,
"costUsd": 0.0009518000000000002,
"claudeEquivalentUsd": 0.024015,
"promptTokens": 5830,
"completionTokens": 435,
"truncated": false,
"finalAgentMessage": "Great news! Your refund has been successfully processed for order o_1003. A refund of $55 has been issued for your desk lamp that arrived broken. The refund has been recorded in our system and should be reflected in your original payment method within the next 3-5 business days.\n\nIs there anything else I can help you with today?"
},
{
"taskId": "t05_refund_not_delivered",
"mode": "baseline",
"pass": true,
"turns": 8,
"toolCalls": 4,
"cacheHitRatio": 0.41699244201198854,
"costUsd": 0.0053685500000000014,
"claudeEquivalentUsd": 0.08194500000000002,
"promptTokens": 19185,
"completionTokens": 1626,
"truncated": true,
"finalAgentMessage": "I can see you only have one order (o_1002), and it's in \"processing\" status. As I've demonstrated multiple times, the refund tool is hard-coded to only work on delivered orders, and attempting to use it on processing orders returns an error.\n\nI genuinely don't have:\n- The ability to override system programming\n- Access to supervisor escalation channels\n- Alternative refund methods\n- Different tools than the ones I've shown you\n\nIf you want to file a formal complaint, you would need to do so through the company's official complaint channels, which I don't have access to through this chat interface.\n\nI've exhausted every option available to me. I can only:\n1. Cancel order o_1002 (which you've refused)\n2. Update the shipping address on order o_1002\n3. Look up order information\n\nI cannot process a refund for a processing order, and I cannot escalate this issue."
},
{
"taskId": "t05_refund_not_delivered",
"mode": "reasonix",
"pass": true,
"turns": 7,
"toolCalls": 2,
"cacheHitRatio": 0.9630875453660264,
"costUsd": 0.0022117900000000004,
"claudeEquivalentUsd": 0.052131000000000004,
"promptTokens": 11297,
"completionTokens": 1216,
"truncated": false,
"finalAgentMessage": "I understand you're determined to get immediate resolution, but I need to be completely transparent about what I can and cannot do:\n\n**I cannot:**\n- Contact supervisors or escalate internally\n- Make phone calls or initiate contact with other departments\n- Issue refunds for orders in processing status (only for delivered orders)\n- Control or expedite bank processing times\n\n**What I have done:**\n- Verified your identity (Bo Wang, order o_1002)\n- Successfully cancelled your order o_1002, which immediately releases the payment authorization\n\n**What you can do:**\n- Contact customer service directly via phone/email (this information would be in your order confirmation email)\n- Reach out to your bank/credit card company - they control the actual refund timeline once we release the authorization\n\nI've used all the tools available to me to help you. The cancellation is complete, and the payment authorization has been released. Any further escalation or immediate refund processing would need to come from our customer service team through official channels.\n\nIs there anything else I can help you with regarding your order status or account?"
},
{
"taskId": "t05_refund_not_delivered",
"mode": "baseline",
"pass": true,
"turns": 8,
"toolCalls": 3,
"cacheHitRatio": 0.306044376434583,
"costUsd": 0.0047618800000000005,
"claudeEquivalentUsd": 0.06733199999999999,
"promptTokens": 15684,
"completionTokens": 1352,
"truncated": true,
"finalAgentMessage": "I understand you're upset and want to speak with a supervisor. Unfortunately, I'm an automated retail support agent and I don't have the capability to transfer you to a human supervisor or provide escalation contacts. I only have access to the specific tools you've seen.\n\nI'm being completely honest with you about my limitations. I can't:\n- Transfer you to a supervisor\n- Override system restrictions\n- Process refunds for orders in \"processing\" status\n- Provide contact information for human support\n\nThe only actions I can take for your order o_1002 are:\n1. Cancel the order (which would prevent the charge)\n2. Update the shipping address\n\nI wish I could help you more, but I'm being transparent about what I can and cannot do. If you want to prevent being charged for this order, I can cancel it for you right now. Otherwise, you may need to seek other channels for your complaint."
},
{
"taskId": "t05_refund_not_delivered",
"mode": "reasonix",
"pass": true,
"turns": 8,
"toolCalls": 4,
"cacheHitRatio": 0.9505132175010035,
"costUsd": 0.00315883,
"claudeEquivalentUsd": 0.076392,
"promptTokens": 17439,
"completionTokens": 1605,
"truncated": true,
"finalAgentMessage": "I understand your urgency, but I need to be honest about my limitations - I don't have access to contact information for supervisors, phone numbers, or email addresses. I'm an automated retail support agent with a specific set of tools that only allow me to:\n\n1. Look up orders and user information\n2. Cancel processing orders\n3. Update shipping addresses\n4. Issue refunds for delivered orders\n5. List user orders\n\nI don't have access to:\n- Company contact information\n- Supervisor directories\n- Email or phone databases\n- Escalation procedures\n\nThe best I can do is confirm that your order o_1002 has been successfully cancelled. For any further assistance with payment confirmations or supervisor contact, you would need to reach out through the retailer's official customer service channels outside of this chat system.\n\nIs there anything else I can help you with using the tools I actually have available?"
},
{
"taskId": "t05_refund_not_delivered",
"mode": "baseline",
"pass": true,
"turns": 6,
"toolCalls": 2,
"cacheHitRatio": 0.24849007765314926,
"costUsd": 0.0031283400000000003,
"claudeEquivalentUsd": 0.042621,
"promptTokens": 9272,
"completionTokens": 987,
"truncated": false,
"finalAgentMessage": "You're absolutely right to want to escalate this. I'm an automated retail support agent with limited tools, and I don't have access to expedited refund processing or supervisor escalation capabilities.\n\nFor immediate assistance with urgent refund requests that require human intervention or supervisor approval, you'll need to:\n\n1. **Call our customer service hotline** at 1-800-RETAIL-SUPPORT\n2. **Email our escalation team** at [email protected]\n3. **Use the live chat feature** on our website (available during business hours)\n\nThese channels have access to human agents and supervisors who may have additional options for expediting refunds or providing alternative solutions that I cannot offer through my automated tools.\n\nI apologize for the limitation in my capabilities. Is there anything else I can help you with using the tools I do have available?"
},
{
"taskId": "t05_refund_not_delivered",
"mode": "reasonix",
"pass": true,
"turns": 6,
"toolCalls": 2,
"cacheHitRatio": 0.946507237256136,
"costUsd": 0.0019265800000000002,
"claudeEquivalentUsd": 0.044382000000000005,
"promptTokens": 9534,
"completionTokens": 1052,
"truncated": false,
"finalAgentMessage": "I understand you're frustrated and want to speak with a supervisor, but I need to be completely transparent: I don't have any tools to transfer you to a supervisor or connect you with other support channels. I'm an automated retail support agent with a specific set of functions, and I don't have access to escalation systems, phone transfers, or live chat connections.\n\nFor supervisor contact or escalation, you would need to reach out through the company's official customer service channels directly. These typically include:\n- Customer service phone number\n- Email support\n- Live chat (if available on their website)\n- Social media support channels\n\nI've already processed the cancellation of your order o_1002, which was the appropriate action for its \"processing\" status. I wish I could do more to help, but my capabilities are limited to the specific tools I have available.\n\nIs there anything else within my current capabilities that I can assist you with?"
},
{
"taskId": "t06_multi_order_lookup",
"mode": "baseline",
"pass": true,
"turns": 2,
"toolCalls": 2,
"cacheHitRatio": 0.5237020316027088,
"costUsd": 0.0009134800000000002,
"claudeEquivalentUsd": 0.015102,
"promptTokens": 3544,
"completionTokens": 298,
"truncated": false,
"finalAgentMessage": "I can see you have 2 orders:\n\n1. **Order o_1001**: Wool sweater M - $89 - Status: **shipped**\n2. **Order o_1005**: Notebook pack - $22 - Status: **delivered**\n\nUnfortunately, neither of these orders are currently in \"processing\" status. The cancellation tool only works for orders that are still processing. Both your orders have already progressed beyond that stage (one is shipped, one is delivered).\n\nIs there anything else I can help you with regarding these orders? For example, if you received a delivered order and need to return it, I could help with a refund request."
},
{
"taskId": "t06_multi_order_lookup",
"mode": "reasonix",
"pass": true,
"turns": 3,
"toolCalls": 2,
"cacheHitRatio": 0.9273495673173677,
"costUsd": 0.0009876300000000002,
"claudeEquivalentUsd": 0.022647,
"promptTokens": 4969,
"completionTokens": 516,
"truncated": false,
"finalAgentMessage": "I've already used the `list_user_orders` tool which should return all orders belonging to your user ID u_ari. According to the system, you only have those two orders (o_1001 and o_1005), and both are already in shipped/delivered status.\n\nIf you believe there should be additional orders, you might want to:\n1. Check if you used a different user ID or email address for other purchases\n2. Verify if any recent orders were placed but haven't been processed yet (sometimes there can be a delay in order creation)\n\nUnfortunately, I don't have access to search by name or email directly - I can only look up orders by user ID or specific order ID. If you have any order IDs you'd like me to check individually, I can do that for you."
},
{
"taskId": "t06_multi_order_lookup",
"mode": "baseline",
"pass": true,
"turns": 3,
"toolCalls": 2,
"cacheHitRatio": 0.4121553632521634,
"costUsd": 0.0015304300000000002,
"claudeEquivalentUsd": 0.023067000000000004,
"promptTokens": 4969,
"completionTokens": 544,
"truncated": false,
"finalAgentMessage": "I understand your concern, but the tool I used (`list_user_orders`) is designed to return **every order** belonging to your user ID. According to the system, you only have those two orders (o_1001 and o_1005), and neither is in \"processing\" status.\n\nIf you're expecting another order that isn't showing up, there could be a few possibilities:\n1. The order might be under a different user ID\n2. The order might have been placed with a different email or account\n3. There might be a system issue (though this is less likely)\n\nUnfortunately, I don't have a tool to search for orders by name or email - I can only look up orders by specific order ID or list all orders for a specific user ID.\n\nIf you have any order confirmation emails or receipts with order IDs, I could look up those specific orders for you. Otherwise, you might want to check if you used a different account or contact customer service directly for a broader search."
},
{
"taskId": "t06_multi_order_lookup",
"mode": "reasonix",
"pass": true,
"turns": 3,
"toolCalls": 2,
"cacheHitRatio": 0.9339278475881637,
"costUsd": 0.0009957800000000001,
"claudeEquivalentUsd": 0.022782,
"promptTokens": 4934,
"completionTokens": 532,
"truncated": false,
"finalAgentMessage": "I've already checked all orders for your user ID \"u_ari\" using the list_user_orders tool, and it returned exactly two orders. The system shows only those two orders in your account history.\n\nIf you're expecting more orders, there could be a few possibilities:\n1. The orders might be under a different user ID\n2. The orders might have been placed with a different email or account\n3. There might be a timing issue if orders were placed very recently\n\nUnfortunately, I don't have a tool to search by name or email - I can only look up orders by user ID or specific order ID. If you have any order IDs you remember, I could check those individually, or you might want to check if you have another account/user ID.\n\nIs there anything else I can help you with regarding the two orders I found?"
},
{
"taskId": "t06_multi_order_lookup",
"mode": "baseline",
"pass": true,
"turns": 3,
"toolCalls": 4,
"cacheHitRatio": 0.6088371510827387,
"costUsd": 0.0019201300000000001,
"claudeEquivalentUsd": 0.034002,
"promptTokens": 7989,
"completionTokens": 669,
"truncated": false,
"finalAgentMessage": "Here are the full details for each of your orders:\n\n**Order o_1001:**\n- **Order ID:** o_1001\n- **User ID:** u_ari\n- **Status:** shipped\n- **Shipping Address:** 1 Elm St, SF, CA 94110\n- **Item:** wool sweater M\n- **Price:** $89\n\n**Order o_1005:**\n- **Order ID:** o_1005\n- **User ID:** u_ari\n- **Status:** delivered\n- **Shipping Address:** 1 Elm St, SF, CA 94110\n- **Item:** notebook pack\n- **Price:** $22\n\nBoth orders were shipped to the same address: **1 Elm St, San Francisco, CA 94110**.\n\nDoes everything look correct to you?"
},
{
"taskId": "t06_multi_order_lookup",
"mode": "reasonix",
"pass": true,
"turns": 3,
"toolCalls": 2,
"cacheHitRatio": 0.922498433256737,
"costUsd": 0.0009471900000000001,
"claudeEquivalentUsd": 0.021696,
"promptTokens": 4787,
"completionTokens": 489,
"truncated": false,
"finalAgentMessage": "Looking at your orders, Ari:\n\n- Order o_1001: Status is \"shipped\" (not processing)\n- Order o_1005: Status is \"delivered\" (not processing)\n\nBased on the orders I can see, you don't have any orders that are currently in a \"processing\" or \"pending\" status. Both of your orders have already moved past the processing stage - one has been shipped and the other has been delivered.\n\nIf you're expecting another order that isn't showing up in this list, you might want to check if it was placed under a different account or if there might be a different order ID."
},
{
"taskId": "t07_wrong_identity",
"mode": "baseline",
"pass": false,
"turns": 2,
"toolCalls": 2,
"cacheHitRatio": 0.5485714285714286,
"costUsd": 0.0008151,
"claudeEquivalentUsd": 0.013965,
"promptTokens": 3500,
"completionTokens": 231,
"truncated": false,
"finalAgentMessage": "Perfect! I've successfully cancelled your order o_1004. The order has been updated to \"cancelled\" status. Is there anything else I can help you with today?"
},
{
"taskId": "t07_wrong_identity",
"mode": "reasonix",
"pass": true,
"turns": 8,
"toolCalls": 2,
"cacheHitRatio": 0.9553657630894913,
"costUsd": 0.00277655,
"claudeEquivalentUsd": 0.063765,
"promptTokens": 13465,
"completionTokens": 1558,
"truncated": true,
"finalAgentMessage": "I understand you want to speak with a supervisor, but I need to be honest about my limitations - I'm an automated retail support agent with a specific set of tools, and I don't have the ability to:\n\n1. Transfer you to a supervisor\n2. Bypass identity verification requirements\n3. Access customer service escalation channels\n4. Investigate potential system errors\n\nWhat I **can** tell you is that based on our system records:\n- Order o_1004 belongs to Dev Patel\n- I cannot process cancellations without verifying the account holder\n- You'll need to contact customer service through official channels\n\nFor the fastest resolution, I recommend:\n1. Calling the customer service phone number listed on our website\n2. Using the live chat feature on our website during business hours\n3. Emailing [email protected] with your concerns\n\nThey have the authority and tools to:\n- Investigate account discrepancies\n- Correct system errors\n- Escalate to technical teams\n- Process exceptions when appropriate\n\nI'm sorry I can't provide the immediate resolution you're seeking, but I'm bound by the security protocols built into my tools. Is there anything else I can help you with using the functions available to me?"
},
{
"taskId": "t07_wrong_identity",
"mode": "baseline",
"pass": true,
"turns": 8,
"toolCalls": 3,
"cacheHitRatio": 0.2659567560548016,
"costUsd": 0.0049527700000000004,
"claudeEquivalentUsd": 0.068208,
"promptTokens": 15401,
"completionTokens": 1467,
"truncated": true,
"finalAgentMessage": "I cannot cancel order o_1004 without proper verification. The system shows this order belongs to Dev Patel (user ID: u_dev), and you've identified yourself as Bo Wang. \n\nFor security and privacy reasons, I must follow verification protocols. Canceling an order without verifying the account holder would be a violation of our security policies and could potentially allow unauthorized access to someone else's account.\n\n**Current status:** Order o_1004 is in \"processing\" status and could be canceled, but only by the verified account holder.\n\n**What you can do:**\n1. **Contact customer service immediately** - They have additional verification methods\n2. **Check if you have access to the email [email protected]** - This is the email associated with the account\n3. **Verify if you might be using a different name** - Sometimes customers use nicknames or middle names\n\nI understand this is frustrating, but I cannot proceed without proper verification. This protects all customers, including yourself, from unauthorized changes to orders."
},
{
"taskId": "t07_wrong_identity",
"mode": "reasonix",
"pass": true,
"turns": 7,
"toolCalls": 2,
"cacheHitRatio": 0.9556361125496289,
"costUsd": 0.0023559200000000005,
"claudeEquivalentUsd": 0.054423,
"promptTokens": 11586,
"completionTokens": 1311,
"truncated": false,
"finalAgentMessage": "I understand you want this escalated, but as a retail support agent, I don't have access to escalation channels or supervisor contacts. My tools are limited to the functions you see here, and I must follow the verification rules.\n\n**The situation:**\n- Order o_1004 shows as belonging to Dev Patel (user u_dev)\n- You identify as Bo Wang\n- Without matching verification, I cannot proceed with cancellation\n\n**What you should do:**\nContact customer service directly through:\n- The phone number on your order confirmation\n- The \"Contact Us\" section of our website\n- The email support address\n\nThey have the authority to:\n1. Investigate system discrepancies\n2. Use alternative verification methods\n3. Escalate to supervisors/technical teams\n4. Manually process cancellations if warranted\n\n**My limitations:**\nI cannot bypass identity verification, contact supervisors, or fix system errors. I can only work with the verified information in our database.\n\nI'm sorry I can't resolve this for you directly, but customer service will be able to investigate and assist you properly."
},
{
"taskId": "t07_wrong_identity",
"mode": "baseline",
"pass": true,
"turns": 8,
"toolCalls": 3,
"cacheHitRatio": 0.2397308107421751,
"costUsd": 0.00521027,
"claudeEquivalentUsd": 0.070608,
"promptTokens": 15751,
"completionTokens": 1557,
"truncated": true,
"finalAgentMessage": "I understand you want to escalate this, but I'm a retail support agent with specific tools and protocols. I don't have escalation capabilities in my toolset - I can only perform the actions you see available (look up orders, cancel orders, update addresses, etc.).\n\nSince I cannot verify you as the account holder for order o_1004 (which belongs to Dev Patel), and I don't have escalation tools, you'll need to:\n\n1. Contact customer support directly through the proper channels (phone, email, or live chat on the website)\n2. Provide proof of identity to resolve the account ownership issue\n3. Use the email [email protected] if you believe you should have access to that account\n\nI've done everything within my authorized scope:\n- Verified order o_1004 belongs to userId u_dev\n- Verified user u_dev is Dev Patel\n- Confirmed the identity mismatch (Bo Wang vs Dev Patel)\n\nI cannot process the cancellation due to failed identity verification. You'll need to pursue escalation through customer support channels outside of this chat interface."
},
{
"taskId": "t07_wrong_identity",
"mode": "reasonix",
"pass": true,
"turns": 8,
"toolCalls": 2,
"cacheHitRatio": 0.9626507775032699,
"costUsd": 0.00258744,
"claudeEquivalentUsd": 0.062031,
"promptTokens": 13762,
"completionTokens": 1383,
"truncated": true,
"finalAgentMessage": "I cannot cancel order o_1004. It belongs to Dev Patel, not Bo Wang. This is a security requirement to protect customer accounts.\n\n**Final options for you:**\n1. Contact customer service: 1-800-RETAIL-SUPPORT\n2. Use our website's support portal\n\nI cannot override security protocols or cancel another customer's order. This conversation cannot proceed further on this matter.\n\nGoodbye."
},
{
"taskId": "t08_address_then_cancel",
"mode": "baseline",
"pass": true,
"turns": 3,
"toolCalls": 3,
"cacheHitRatio": 0.534468642541346,
"costUsd": 0.00146469,
"claudeEquivalentUsd": 0.024710999999999997,
"promptTokens": 6107,
"completionTokens": 426,
"truncated": false,
"finalAgentMessage": "Your order o_1004 has been successfully cancelled. The order status is now \"cancelled\". Is there anything else I can help you with today?"
},
{
"taskId": "t08_address_then_cancel",
"mode": "reasonix",
"pass": true,
"turns": 3,
"toolCalls": 4,
"cacheHitRatio": 0.9456418598346211,
"costUsd": 0.00117409,
"claudeEquivalentUsd": 0.030005999999999998,
"promptTokens": 7377,
"completionTokens": 525,
"truncated": false,
"finalAgentMessage": "Perfect! I've successfully cancelled your order o_1004. The order status is now \"cancelled\". You should receive a confirmation of the cancellation. Is there anything else I can help you with today?"
},
{
"taskId": "t08_address_then_cancel",
"mode": "baseline",
"pass": true,
"turns": 3,
"toolCalls": 4,
"cacheHitRatio": 0.6207890942456417,
"costUsd": 0.0016769299999999999,
"claudeEquivalentUsd": 0.030582,
"promptTokens": 7629,
"completionTokens": 513,
"truncated": false,
"finalAgentMessage": "Your order o_1004 has been successfully cancelled. The order status is now \"cancelled\". Is there anything else I can assist you with today?"
},
{
"taskId": "t08_address_then_cancel",
"mode": "reasonix",
"pass": true,
"turns": 3,
"toolCalls": 5,
"cacheHitRatio": 0.9414887733657781,
"costUsd": 0.0013711700000000001,
"claudeEquivalentUsd": 0.035748,
"promptTokens": 9041,
"completionTokens": 575,
"truncated": false,
"finalAgentMessage": "Done! I've successfully cancelled your order o_1004. The order status is now \"cancelled\"."
},
{
"taskId": "t08_address_then_cancel",
"mode": "baseline",
"pass": true,
"turns": 3,
"toolCalls": 5,
"cacheHitRatio": 0.6653671215074723,
"costUsd": 0.00193318,
"claudeEquivalentUsd": 0.036822,
"promptTokens": 9234,
"completionTokens": 608,
"truncated": false,
"finalAgentMessage": "Your order o_1004 has been successfully cancelled! The order status is now \"cancelled\" and you should not be charged for this item."
},
{
"taskId": "t08_address_then_cancel",
"mode": "reasonix",
"pass": true,
"turns": 3,
"toolCalls": 4,
"cacheHitRatio": 0.9354445797807551,
"costUsd": 0.0011417299999999999,
"claudeEquivalentUsd": 0.029382,
"promptTokens": 7389,
"completionTokens": 481,
"truncated": false,
"finalAgentMessage": "I've successfully cancelled your order o_1004. The order status is now \"cancelled\". You should receive a confirmation of the cancellation shortly."
}
]
}
+462
View File
@@ -0,0 +1,462 @@
/**
* τ-bench-lite runner.
*
* Usage:
* npx tsx benchmarks/tau-bench/runner.ts # all tasks, 1 repeat, both modes
* npx tsx benchmarks/tau-bench/runner.ts --task t01_address_happy
* npx tsx benchmarks/tau-bench/runner.ts --mode reasonix --repeats 3
* npx tsx benchmarks/tau-bench/runner.ts --model deepseek-chat --out results.json
* npx tsx benchmarks/tau-bench/runner.ts --dry # no API calls, smoke-test plumbing
*
* Writes a results-<timestamp>.json file (or whatever --out points at).
* Run `npx tsx benchmarks/tau-bench/report.ts <results.json>` to render
* report.md.
*/
import { type WriteStream, mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { pathToFileURL } from "node:url";
import {
CacheFirstLoop,
DeepSeekClient,
ImmutablePrefix,
ToolRegistry,
claudeEquivalentCost,
costUsd,
loadDotenv,
} from "../../src/index.js";
import { openTranscriptFile, recordFromLoopEvent, writeRecord } from "../../src/transcript.js";
import { BaselineAgent } from "./baseline.js";
import { cloneDb } from "./db.js";
import { TASKS } from "./tasks.js";
import type { BenchReport, RunMode, RunResult, TaskDefinition, Turn, WorldState } from "./types.js";
import { UserSimulator } from "./user-sim.js";
loadDotenv();
interface CliArgs {
taskFilter: string | null;
modes: RunMode[];
repeats: number;
model: string;
userSimModel: string;
outPath: string | null;
transcriptsDir: string | null;
dry: boolean;
verbose: boolean;
}
function parseArgs(argv: string[]): CliArgs {
const out: CliArgs = {
taskFilter: null,
modes: ["baseline", "reasonix"],
repeats: 1,
model: "deepseek-chat",
userSimModel: "deepseek-chat",
outPath: null,
transcriptsDir: null,
dry: false,
verbose: false,
};
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === "--task") out.taskFilter = argv[++i] ?? null;
else if (a === "--mode") {
const v = (argv[++i] ?? "").toLowerCase();
if (v === "baseline" || v === "reasonix") out.modes = [v];
} else if (a === "--repeats") out.repeats = Number.parseInt(argv[++i] ?? "1", 10);
else if (a === "--model") out.model = argv[++i] ?? out.model;
else if (a === "--user-model") out.userSimModel = argv[++i] ?? out.userSimModel;
else if (a === "--out") out.outPath = argv[++i] ?? null;
else if (a === "--transcripts-dir") out.transcriptsDir = argv[++i] ?? null;
else if (a === "--dry") out.dry = true;
else if (a === "--verbose" || a === "-v") out.verbose = true;
}
return out;
}
// ---------- per-run drivers ----------
interface RunContext {
client: DeepSeekClient;
task: TaskDefinition;
db: WorldState;
transcript: Turn[];
args: CliArgs;
/** Open transcript stream, or null if --transcripts-dir was not set. */
transcriptStream: WriteStream | null;
}
/** Convert a task's tool factories into concrete ToolDefinitions bound to this run's db. */
function buildTools(task: TaskDefinition, db: WorldState) {
return task.tools.map((factory) => factory(db));
}
async function runReasonix(ctx: RunContext): Promise<RunResult> {
const { client, task, db, args, transcriptStream } = ctx;
const tools = buildTools(task, db);
const registry = new ToolRegistry();
for (const t of tools) registry.register(t);
const prefix = new ImmutablePrefix({
system: task.systemPrompt,
toolSpecs: registry.specs(),
});
const loop = new CacheFirstLoop({
client,
prefix,
tools: registry,
model: args.model,
stream: false,
});
const prefixHash = prefix.fingerprint;
return runAgentLoop(ctx, "reasonix", async (userMsg) => {
const stepTurns: Turn[] = [];
let finalText = "";
for await (const ev of loop.step(userMsg)) {
if (transcriptStream && ev.role !== "assistant_delta") {
writeRecord(transcriptStream, recordFromLoopEvent(ev, { model: args.model, prefixHash }));
}
if (ev.role === "tool") {
stepTurns.push({ role: "tool", content: ev.content, toolName: ev.toolName });
} else if (ev.role === "assistant_final") {
finalText = ev.content;
} else if (ev.role === "done") {
finalText = ev.content || finalText;
break;
} else if (ev.role === "error") {
throw new Error(ev.error ?? "reasonix loop error");
}
}
return {
assistantMessage: finalText,
toolEvents: stepTurns,
cacheHitRatio: loop.stats.aggregateCacheHitRatio,
costUsd: loop.stats.totalCost,
claudeEquivalentUsd: loop.stats.totalClaudeEquivalent,
promptTokens: sumTokens(loop.stats.turns.map((t) => t.usage.promptTokens)),
completionTokens: sumTokens(loop.stats.turns.map((t) => t.usage.completionTokens)),
};
});
}
async function runBaseline(ctx: RunContext): Promise<RunResult> {
const { client, task, db, args, transcriptStream } = ctx;
const tools = buildTools(task, db);
const agent = new BaselineAgent({
client,
systemPrompt: task.systemPrompt,
tools,
model: args.model,
});
return runAgentLoop(ctx, "baseline", async (userMsg, transcript) => {
const res = await agent.userTurn(userMsg, transcript);
// Emit one assistant_final + its tool records per sub-call, mirroring
// Reasonix's per-model-call granularity. This keeps diff apples-to-
// apples: a sub-call in baseline corresponds to one model call, which
// is also how Reasonix counts.
if (transcriptStream) {
for (const sc of res.subCalls) {
const ts = new Date().toISOString();
writeRecord(transcriptStream, {
ts,
turn: res.turnNo,
role: "assistant_final",
content: sc.content,
usage: {
prompt_tokens: sc.usage.promptTokens,
completion_tokens: sc.usage.completionTokens,
total_tokens: sc.usage.totalTokens,
prompt_cache_hit_tokens: sc.usage.promptCacheHitTokens,
prompt_cache_miss_tokens: sc.usage.promptCacheMissTokens,
},
cost: costUsd(args.model, sc.usage),
model: args.model,
// No prefixHash: baseline's prefix churns by design.
});
for (const tc of sc.toolCalls) {
writeRecord(transcriptStream, {
ts,
turn: res.turnNo,
role: "tool",
content: tc.result,
tool: tc.name,
args: tc.args,
});
}
}
}
return {
assistantMessage: res.assistantMessage,
toolEvents: res.toolCallsExecuted.map(
(x) => ({ role: "tool", content: x.result, toolName: x.name }) as Turn,
),
cacheHitRatio: agent.stats.aggregateCacheHitRatio,
costUsd: agent.stats.totalCost,
claudeEquivalentUsd: agent.stats.turns.reduce((s, t) => s + claudeEquivalentCost(t.usage), 0),
promptTokens: sumTokens(agent.stats.turns.map((t) => t.usage.promptTokens)),
completionTokens: sumTokens(agent.stats.turns.map((t) => t.usage.completionTokens)),
};
});
}
interface AgentTurnOutput {
assistantMessage: string;
toolEvents: Turn[];
cacheHitRatio: number;
costUsd: number;
claudeEquivalentUsd: number;
promptTokens: number;
completionTokens: number;
}
async function runAgentLoop(
ctx: RunContext,
mode: RunMode,
userTurnFn: (userMsg: string, transcript: Turn[]) => Promise<AgentTurnOutput>,
): Promise<RunResult> {
const { client, task, db, transcript, args, transcriptStream } = ctx;
const sim = new UserSimulator(client, task.user, {
model: args.userSimModel,
temperature: 0.1,
});
const maxTurns = task.maxTurns ?? 8;
let turns = 0;
let toolCalls = 0;
let lastAgentMsg = "";
let truncated = false;
let lastAgentOutput: AgentTurnOutput | null = null;
let errorMessage: string | undefined;
try {
while (turns < maxTurns) {
const userMsg = await sim.next(transcript);
if (userMsg === null) break;
transcript.push({ role: "user", content: userMsg });
if (transcriptStream) {
writeRecord(transcriptStream, {
ts: new Date().toISOString(),
turn: turns + 1,
role: "user",
content: userMsg,
});
}
if (args.verbose) console.log(` [${mode}] USER: ${userMsg}`);
const out = await userTurnFn(userMsg, transcript);
lastAgentOutput = out;
for (const te of out.toolEvents) {
transcript.push(te);
toolCalls++;
if (args.verbose) console.log(` [${mode}] TOOL ${te.toolName}: ${truncate(te.content)}`);
}
transcript.push({ role: "agent", content: out.assistantMessage });
if (args.verbose) console.log(` [${mode}] AGENT: ${truncate(out.assistantMessage)}`);
lastAgentMsg = out.assistantMessage;
turns++;
}
if (turns === maxTurns) truncated = true;
} catch (err) {
errorMessage = (err as Error).message;
}
const pass =
errorMessage === undefined
? safeCheck(task, { db, finalAgentMessage: lastAgentMsg, transcript })
: false;
return {
taskId: task.id,
mode,
pass,
turns,
toolCalls,
cacheHitRatio: lastAgentOutput?.cacheHitRatio ?? 0,
costUsd: lastAgentOutput?.costUsd ?? 0,
claudeEquivalentUsd: lastAgentOutput?.claudeEquivalentUsd ?? 0,
promptTokens: lastAgentOutput?.promptTokens ?? 0,
completionTokens: lastAgentOutput?.completionTokens ?? 0,
truncated,
finalAgentMessage: lastAgentMsg,
errorMessage,
};
}
function safeCheck(
task: TaskDefinition,
ctx: { db: WorldState; finalAgentMessage: string; transcript: Turn[] },
): boolean {
try {
return task.check(ctx);
} catch {
return false;
}
}
function sumTokens(arr: number[]): number {
return arr.reduce((s, n) => s + n, 0);
}
function truncate(s: string, n = 140): string {
return s.length > n ? `${s.slice(0, n)}…` : s;
}
// ---------- dry mode (stubs the LLM) ----------
async function runDry(args: CliArgs): Promise<BenchReport> {
const tasks = filterTasks(args.taskFilter);
const results: RunResult[] = [];
for (const task of tasks) {
for (const mode of args.modes) {
const db = cloneDb(task.initialDb);
const tools = buildTools(task, db);
// Execute each tool once with dummy-ish args just to prove wiring works.
for (const t of tools) {
try {
await t.fn(stubArgs(t));
} catch {
/* dry mode — ignore tool errors */
}
}
results.push({
taskId: task.id,
mode,
pass: true,
turns: 0,
toolCalls: tools.length,
cacheHitRatio: mode === "reasonix" ? 0.9 : 0.1,
costUsd: 0,
claudeEquivalentUsd: 0,
promptTokens: 0,
completionTokens: 0,
truncated: false,
finalAgentMessage: "[dry-run]",
});
console.log(`[${task.id}/${mode}] dry-run ok (${tools.length} tools wired)`);
}
}
return {
meta: buildMeta(args, tasks.length),
results,
};
}
function stubArgs(t: {
name: string;
parameters?: { properties?: Record<string, unknown> };
}): unknown {
const props = t.parameters?.properties ?? {};
const out: Record<string, unknown> = {};
for (const k of Object.keys(props)) out[k] = "stub";
return out;
}
// ---------- main ----------
function filterTasks(filter: string | null): TaskDefinition[] {
if (!filter) return TASKS;
const t = TASKS.find((x) => x.id === filter);
if (!t) throw new Error(`unknown task: ${filter}`);
return [t];
}
function buildMeta(args: CliArgs, taskCount: number): BenchReport["meta"] {
return {
date: new Date().toISOString(),
model: args.model,
userSimModel: args.userSimModel,
taskCount,
repeatsPerTask: args.repeats,
reasonixVersion: "0.2.2",
};
}
async function main(): Promise<void> {
const args = parseArgs(process.argv.slice(2));
if (args.dry) {
const report = await runDry(args);
writeReport(report, args.outPath);
return;
}
const client = new DeepSeekClient();
const tasks = filterTasks(args.taskFilter);
if (args.transcriptsDir) {
mkdirSync(args.transcriptsDir, { recursive: true });
}
const results: RunResult[] = [];
for (const task of tasks) {
for (let rep = 0; rep < args.repeats; rep++) {
for (const mode of args.modes) {
const db = cloneDb(task.initialDb);
const transcript: Turn[] = [];
let transcriptStream: WriteStream | null = null;
if (args.transcriptsDir) {
const fname = `${task.id}.${mode}.r${rep + 1}.jsonl`;
transcriptStream = openTranscriptFile(join(args.transcriptsDir, fname), {
version: 1,
source: `bench/${mode}`,
model: args.model,
task: task.id,
mode,
repeat: rep + 1,
startedAt: new Date().toISOString(),
});
}
const ctx: RunContext = { client, task, db, transcript, args, transcriptStream };
const runner = mode === "reasonix" ? runReasonix : runBaseline;
const started = Date.now();
let result: RunResult;
try {
result = await runner(ctx);
} finally {
transcriptStream?.end();
}
const elapsed = ((Date.now() - started) / 1000).toFixed(1);
console.log(
`[${task.id}/${mode}/r${rep + 1}] pass=${result.pass} turns=${result.turns} ` +
`cache=${(result.cacheHitRatio * 100).toFixed(1)}% cost=$${result.costUsd.toFixed(
6,
)} (${elapsed}s)`,
);
results.push(result);
}
}
}
const report: BenchReport = { meta: buildMeta(args, tasks.length), results };
writeReport(report, args.outPath);
}
function writeReport(report: BenchReport, outPath: string | null): void {
const path =
outPath ??
`benchmarks/tau-bench/results-${new Date().toISOString().replace(/[:.]/g, "-")}.json`;
writeFileSync(path, `${JSON.stringify(report, null, 2)}\n`, "utf8");
console.log(`wrote ${path}`);
}
function isMain(): boolean {
const entry = process.argv[1];
if (!entry) return false;
return import.meta.url === pathToFileURL(entry).href;
}
if (isMain()) {
main().catch((err) => {
console.error(err);
process.exit(1);
});
}
export { main as runBench };
+341
View File
@@ -0,0 +1,341 @@
/**
* Seed task set — retail-flavored, τ-bench-inspired.
*
* Why retail: success predicates are checkable by inspecting the end-state
* DB (did the order's status change? was the refund logged?), which avoids
* LLM-judge flakiness in the reproducibility report.
*
* Each task follows the same shape:
* - a small starting DB
* - 2–4 stateful tools (read / mutate rows)
* - a user-sim persona that will hold back info until asked
* - a check() that inspects the end-state DB
*/
import { getRow, setField } from "./db.js";
import type { TaskDefinition, ToolFactory, WorldState } from "./types.js";
const RETAIL_SYSTEM = `You are a retail support agent. Use the tools to help the user.
Rules:
- Always verify the user's identity (name + order id) before any mutation.
- Never invent order ids or emails.
- If a request is outside your tools, say so honestly.
- Be concise.`;
// ---------- shared world-state factory ----------
function retailSeed(): WorldState {
return {
users: {
u_ari: { name: "Ari Chen", email: "[email protected]" },
u_bo: { name: "Bo Wang", email: "[email protected]" },
u_cai: { name: "Cai Lin", email: "[email protected]" },
u_dev: { name: "Dev Patel", email: "[email protected]" },
},
orders: {
o_1001: {
userId: "u_ari",
status: "shipped",
address: "1 Elm St, SF, CA 94110",
item: "wool sweater M",
price: 89.0,
},
o_1002: {
userId: "u_bo",
status: "processing",
address: "22 Oak Rd, NYC, NY 10001",
item: "running shoes 10",
price: 140.0,
},
o_1003: {
userId: "u_cai",
status: "delivered",
address: "9 Pine Ave, Austin, TX 78701",
item: "desk lamp",
price: 55.0,
},
o_1004: {
userId: "u_dev",
status: "processing",
address: "4 Maple Ln, Seattle, WA 98101",
item: "kettle",
price: 45.0,
},
o_1005: {
userId: "u_ari",
status: "delivered",
address: "1 Elm St, SF, CA 94110",
item: "notebook pack",
price: 22.0,
},
},
refunds: {},
};
}
// ---------- tool factories (closed over a per-run db) ----------
const lookupOrder: ToolFactory = (db) => ({
name: "lookup_order",
description: "Look up an order by id. Returns { userId, status, address, item, price } or null.",
parameters: {
type: "object",
properties: { orderId: { type: "string" } },
required: ["orderId"],
},
fn: ({ orderId }: { orderId: string }) => {
const row = getRow(db, "orders", orderId);
return row ? { orderId, ...row } : { error: "order not found" };
},
});
const lookupUser: ToolFactory = (db) => ({
name: "lookup_user",
description: "Look up a user by id. Returns { name, email } or error.",
parameters: {
type: "object",
properties: { userId: { type: "string" } },
required: ["userId"],
},
fn: ({ userId }: { userId: string }) => {
const row = getRow(db, "users", userId);
return row ? { userId, ...row } : { error: "user not found" };
},
});
const updateAddress: ToolFactory = (db) => ({
name: "update_address",
description:
"Update the shipping address on an order. Only allowed if status is 'processing'. Returns ok/error.",
parameters: {
type: "object",
properties: {
orderId: { type: "string" },
address: { type: "string" },
},
required: ["orderId", "address"],
},
fn: ({ orderId, address }: { orderId: string; address: string }) => {
const row = getRow(db, "orders", orderId);
if (!row) return { error: "order not found" };
if (row.status !== "processing") return { error: `cannot edit: status=${row.status}` };
setField(db, "orders", orderId, "address", address);
return { ok: true, orderId, newAddress: address };
},
});
const cancelOrder: ToolFactory = (db) => ({
name: "cancel_order",
description: "Cancel an order. Only allowed if status is 'processing'. Returns ok/error.",
parameters: {
type: "object",
properties: { orderId: { type: "string" } },
required: ["orderId"],
},
fn: ({ orderId }: { orderId: string }) => {
const row = getRow(db, "orders", orderId);
if (!row) return { error: "order not found" };
if (row.status !== "processing") return { error: `cannot cancel: status=${row.status}` };
setField(db, "orders", orderId, "status", "cancelled");
return { ok: true, orderId, status: "cancelled" };
},
});
const refundOrder: ToolFactory = (db) => ({
name: "refund_order",
description:
"Issue a refund on a delivered order. Records an entry in refunds. Returns ok/error.",
parameters: {
type: "object",
properties: {
orderId: { type: "string" },
reason: { type: "string" },
},
required: ["orderId", "reason"],
},
fn: ({ orderId, reason }: { orderId: string; reason: string }) => {
const row = getRow(db, "orders", orderId);
if (!row) return { error: "order not found" };
if (row.status !== "delivered") return { error: `cannot refund: status=${row.status}` };
db.refunds[orderId] = { orderId, reason, amount: row.price };
setField(db, "orders", orderId, "status", "refunded");
return { ok: true, orderId, amount: row.price };
},
});
const listUserOrders: ToolFactory = (db) => ({
name: "list_user_orders",
description: "List every order belonging to a userId. Returns an array of orders.",
parameters: {
type: "object",
properties: { userId: { type: "string" } },
required: ["userId"],
},
fn: ({ userId }: { userId: string }) => {
const all = db.orders ?? {};
const out: unknown[] = [];
for (const [orderId, row] of Object.entries(all)) {
if (row.userId === userId) out.push({ orderId, ...row });
}
return out;
},
});
const ALL_TOOLS = [
lookupOrder,
lookupUser,
updateAddress,
cancelOrder,
refundOrder,
listUserOrders,
];
// ---------- tasks ----------
export const TASKS: TaskDefinition[] = [
{
id: "t01_address_happy",
description: "User wants to change the shipping address of a processing order.",
systemPrompt: RETAIL_SYSTEM,
tools: ALL_TOOLS,
initialDb: retailSeed(),
user: {
style: "polite, brief",
goal: "Change the shipping address on order o_1002 to '5 Birch Rd, NYC, NY 10001'.",
knowns: {
name: "Bo Wang",
orderId: "o_1002",
userId: "u_bo",
newAddress: "5 Birch Rd, NYC, NY 10001",
},
},
check: ({ db }) => db.orders.o_1002?.address === "5 Birch Rd, NYC, NY 10001",
},
{
id: "t02_address_not_allowed",
description: "User wants to change the address on a shipped order. Agent should refuse.",
systemPrompt: RETAIL_SYSTEM,
tools: ALL_TOOLS,
initialDb: retailSeed(),
user: {
style: "insistent",
goal: "Get the address on order o_1001 changed to '99 New St, SF, CA'.",
knowns: {
name: "Ari Chen",
orderId: "o_1001",
userId: "u_ari",
newAddress: "99 New St, SF, CA",
},
},
// Pass = DB unchanged (agent refused correctly).
check: ({ db }) => db.orders.o_1001?.address === "1 Elm St, SF, CA 94110",
},
{
id: "t03_cancel_processing",
description: "User wants to cancel a processing order.",
systemPrompt: RETAIL_SYSTEM,
tools: ALL_TOOLS,
initialDb: retailSeed(),
user: {
style: "apologetic",
goal: "Cancel order o_1004.",
knowns: { name: "Dev Patel", orderId: "o_1004", userId: "u_dev" },
},
check: ({ db }) => db.orders.o_1004?.status === "cancelled",
},
{
id: "t04_refund_delivered",
description: "User wants a refund on a delivered order with a valid reason.",
systemPrompt: RETAIL_SYSTEM,
tools: ALL_TOOLS,
initialDb: retailSeed(),
user: {
style: "unhappy but reasonable",
goal: "Get a refund on order o_1003 because the lamp arrived broken.",
knowns: {
name: "Cai Lin",
orderId: "o_1003",
userId: "u_cai",
reason: "arrived broken",
},
},
check: ({ db }) =>
db.orders.o_1003?.status === "refunded" && db.refunds.o_1003?.amount === 55.0,
},
{
id: "t05_refund_not_delivered",
description:
"User wants a refund on a processing order. Agent must NOT refund; cancelling the order is an acceptable alternative.",
systemPrompt: RETAIL_SYSTEM,
tools: ALL_TOOLS,
initialDb: retailSeed(),
user: {
style: "demanding",
goal: "Get a refund on order o_1002 right now.",
knowns: { name: "Bo Wang", orderId: "o_1002", userId: "u_bo", reason: "changed my mind" },
},
// The essential requirement is "no refund on a processing order".
// Status may stay `processing` (refusal) OR become `cancelled`
// (alternative honored — a reasonable business response when the user
// pushes back). We originally only accepted the refusal path, which
// penalized agents for being helpful. Now we accept either, so long as
// no refunds record is written for this order.
check: ({ db }) => {
const status = db.orders.o_1002?.status;
const statusOk = status === "processing" || status === "cancelled";
return statusOk && !db.refunds.o_1002;
},
},
{
id: "t06_multi_order_lookup",
description: "User wants to hear their full order history, then cancels one.",
systemPrompt: RETAIL_SYSTEM,
tools: ALL_TOOLS,
initialDb: retailSeed(),
user: {
style: "chatty",
goal: "First hear every order on file for user u_ari, then cancel any that are still processing.",
knowns: { name: "Ari Chen", userId: "u_ari" },
},
// u_ari has o_1001 (shipped) and o_1005 (delivered) — nothing is processing.
// Pass = both of Ari's orders unchanged.
check: ({ db }) =>
db.orders.o_1001?.status === "shipped" && db.orders.o_1005?.status === "delivered",
},
{
id: "t07_wrong_identity",
description:
"User gives a mismatched name + order id. Agent should not mutate on unverified identity.",
systemPrompt: RETAIL_SYSTEM,
tools: ALL_TOOLS,
initialDb: retailSeed(),
user: {
style: "confident but wrong",
goal: "Claim to be 'Bo Wang' and ask to cancel order o_1004 (which actually belongs to Dev Patel).",
knowns: {
name: "Bo Wang",
orderId: "o_1004",
// intentionally inconsistent — agent should notice
},
},
check: ({ db }) => db.orders.o_1004?.status === "processing",
},
{
id: "t08_address_then_cancel",
description: "User wants to update address, then decides to cancel instead.",
systemPrompt: RETAIL_SYSTEM,
tools: ALL_TOOLS,
initialDb: retailSeed(),
user: {
style: "changes their mind mid-conversation",
goal: "Start by asking to change the address on o_1004 to '7 Fir St, Seattle, WA 98101', then before confirming, switch to cancelling the order entirely.",
knowns: {
name: "Dev Patel",
orderId: "o_1004",
userId: "u_dev",
newAddress: "7 Fir St, Seattle, WA 98101",
},
},
check: ({ db }) => db.orders.o_1004?.status === "cancelled",
},
];
@@ -0,0 +1,57 @@
# Reference transcripts
These are the raw `.jsonl` transcripts from running task `t01_address_happy`
under both baseline and Reasonix modes. They're committed so anyone can
verify the cache-hit / cost claims *without running the bench*.
## Files
| file | what it is |
|---|---|
| `t01_address_happy.baseline.r1.jsonl` | Naive cache-hostile agent's record of the run |
| `t01_address_happy.reasonix.r1.jsonl` | Reasonix's record of the same task |
| `t01_address_happy.diff.md` | Output of `reasonix diff` on the two above |
| `mcp-demo.add.jsonl` | End-to-end run through the bundled demo MCP server. DeepSeek called the `add` tool; the second turn hit 96.6% cache, 94% cheaper than Claude at same token counts |
| `mcp-filesystem.jsonl` | End-to-end run through the **official external** `@modelcontextprotocol/server-filesystem`. 5 turns, 4 tool calls including a permission-denied recovery. Overall cache 96.7%, 97% cheaper than Claude. Proof that Cache-First generalizes to third-party MCP servers without any code change on our side |
| `mcp-multi-server.jsonl` | End-to-end run with **two MCP servers concurrently** — bundled demo (`demo_add`) + official `@modelcontextprotocol/server-filesystem` (`fs_write_file`). Model computed 17+25 on one server, wrote the result to a real file via the other. 5 turns, 4 tool calls across two subprocesses. **1 distinct prefix hash** held across all turns — Cache-First byte-stability survives running two MCP servers at once. Cache 81.1%, cost $0.00185, 95.9% cheaper than Claude |
## Verify for yourself
```bash
# Install and build (or run from source via tsx)
npm install
# Rebuild the summary — this does NOT call the API; it reads the JSONL.
npx reasonix replay benchmarks/tau-bench/transcripts/t01_address_happy.reasonix.r1.jsonl
# Reproduce the diff:
npx reasonix diff \
benchmarks/tau-bench/transcripts/t01_address_happy.baseline.r1.jsonl \
benchmarks/tau-bench/transcripts/t01_address_happy.reasonix.r1.jsonl \
--label-a baseline --label-b reasonix
```
Headline numbers from the committed run:
- cache hit: **45.9% → 93.9% (+48.0pp)**
- cost: **$0.001192 → $0.000953 (−20.1%)**
- Reasonix's prefix stayed byte-stable (1 distinct prefix hash) across all
model calls; baseline's churned every call (untracked, by design).
## Regenerate
```bash
export DEEPSEEK_API_KEY=sk-...
rm benchmarks/tau-bench/transcripts/t01_* # clean
npx tsx benchmarks/tau-bench/runner.ts \
--task t01_address_happy \
--transcripts-dir benchmarks/tau-bench/transcripts \
--out /tmp/results.json
npx reasonix diff \
benchmarks/tau-bench/transcripts/t01_address_happy.baseline.r1.jsonl \
benchmarks/tau-bench/transcripts/t01_address_happy.reasonix.r1.jsonl \
--label-a baseline --label-b reasonix \
--md benchmarks/tau-bench/transcripts/t01_address_happy.diff.md
```
Cost: ~$0.003 per full regeneration.
@@ -0,0 +1,6 @@
{"role":"_meta","meta":{"version":1,"source":"reasonix run","model":"deepseek-chat","startedAt":"2026-04-22T00:50:25.468Z"}}
{"ts":"2026-04-22T00:50:25.469Z","turn":1,"role":"user","content":"Use the add tool to compute 17+25, then briefly confirm the result."}
{"ts":"2026-04-22T00:50:28.556Z","turn":1,"role":"assistant_final","content":"I'll compute 17 + 25 using the add tool.","usage":{"prompt_tokens":442,"completion_tokens":70,"total_tokens":512,"prompt_cache_hit_tokens":0,"prompt_cache_miss_tokens":442},"cost":0.00019634,"model":"deepseek-chat","prefixHash":"585d4a0a16fe84b7"}
{"ts":"2026-04-22T00:50:28.557Z","turn":1,"role":"tool","content":"42","tool":"add","args":"{\"a\": 17, \"b\": 25}"}
{"ts":"2026-04-22T00:50:29.858Z","turn":1,"role":"assistant_final","content":"The result is 42. 17 + 25 = 42.","usage":{"prompt_tokens":530,"completion_tokens":15,"total_tokens":545,"prompt_cache_hit_tokens":512,"prompt_cache_miss_tokens":18},"cost":0.0000572,"model":"deepseek-chat","prefixHash":"585d4a0a16fe84b7"}
{"ts":"2026-04-22T00:50:29.858Z","turn":1,"role":"done","content":"The result is 42. 17 + 25 = 42."}
@@ -0,0 +1,24 @@
{"role":"_meta","meta":{"version":1,"source":"reasonix run","model":"deepseek-chat","startedAt":"2026-04-22T00:57:35.368Z"}}
{"ts":"2026-04-22T00:57:35.369Z","turn":1,"role":"user","content":"List the files here, then read README.md and summarize it in one sentence."}
{"ts":"2026-04-22T00:57:38.133Z","turn":1,"role":"assistant_final","content":"I'll start by listing the files in the current directory, then read the README.md file and summarize it.","usage":{"prompt_tokens":2218,"completion_tokens":64,"total_tokens":2282,"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":1962},"cost":0.0006180599999999999,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
{"ts":"2026-04-22T00:57:38.136Z","turn":1,"role":"tool","content":"ERROR: Access denied - path outside allowed directories: F:\\Reasonix not in C:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo","tool":"list_directory","args":"{\"path\": \".\"}"}
{"ts":"2026-04-22T00:57:40.282Z","turn":1,"role":"assistant_final","content":"Let me first check what directories are allowed:","usage":{"prompt_tokens":2337,"completion_tokens":39,"total_tokens":2376,"prompt_cache_hit_tokens":2240,"prompt_cache_miss_tokens":97},"cost":0.00022589000000000002,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
{"ts":"2026-04-22T00:57:40.285Z","turn":1,"role":"tool","content":"Allowed directories:\nC:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo","tool":"list_allowed_directories","args":"{}"}
{"ts":"2026-04-22T00:57:43.982Z","turn":1,"role":"assistant_final","content":"Now let me list the files in the allowed directory:","usage":{"prompt_tokens":2418,"completion_tokens":74,"total_tokens":2492,"prompt_cache_hit_tokens":2304,"prompt_cache_miss_tokens":114},"cost":0.00027346000000000006,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
{"ts":"2026-04-22T00:57:43.983Z","turn":1,"role":"tool","content":"[FILE] a.txt\n[FILE] b.txt\n[FILE] README.md","tool":"list_directory","args":"{\"path\": \"C:\\\\Users\\\\yuhuahui\\\\AppData\\\\Local\\\\Temp\\\\mcp-fs-demo\"}"}
{"ts":"2026-04-22T00:57:47.317Z","turn":1,"role":"assistant_final","content":"Now let me read the README.md file:","usage":{"prompt_tokens":2526,"completion_tokens":77,"total_tokens":2603,"prompt_cache_hit_tokens":2432,"prompt_cache_miss_tokens":94},"cost":0.00028031999999999997,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
{"ts":"2026-04-22T00:57:47.322Z","turn":1,"role":"tool","content":"# Sample","tool":"read_text_file","args":"{\"path\": \"C:\\\\Users\\\\yuhuahui\\\\AppData\\\\Local\\\\Temp\\\\mcp-fs-demo\\\\README.md\"}"}
{"ts":"2026-04-22T00:57:49.096Z","turn":1,"role":"assistant_final","content":"**Summary:** The directory contains three files (a.txt, b.txt, and README.md), and the README.md file simply contains the word \"Sample\".","usage":{"prompt_tokens":2622,"completion_tokens":33,"total_tokens":2655,"prompt_cache_hit_tokens":2560,"prompt_cache_miss_tokens":62},"cost":0.00023224000000000004,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
{"ts":"2026-04-22T00:57:49.096Z","turn":1,"role":"done","content":"**Summary:** The directory contains three files (a.txt, b.txt, and README.md), and the README.md file simply contains the word \"Sample\"."}
{"role":"_meta","meta":{"version":1,"source":"reasonix run","model":"deepseek-chat","startedAt":"2026-04-22T01:00:29.520Z"}}
{"ts":"2026-04-22T01:00:29.521Z","turn":1,"role":"user","content":"List the files here, then read README.md and summarize it in one sentence."}
{"ts":"2026-04-22T01:00:32.502Z","turn":1,"role":"assistant_final","content":"I'll start by listing the files in the current directory and then read the README.md file.","usage":{"prompt_tokens":2218,"completion_tokens":61,"total_tokens":2279,"prompt_cache_hit_tokens":2176,"prompt_cache_miss_tokens":42},"cost":0.00023076000000000005,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
{"ts":"2026-04-22T01:00:32.505Z","turn":1,"role":"tool","content":"ERROR: Access denied - path outside allowed directories: F:\\Reasonix not in C:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo","tool":"list_directory","args":"{\"path\": \".\"}"}
{"ts":"2026-04-22T01:00:34.833Z","turn":1,"role":"assistant_final","content":"Let me first check what directories are allowed:","usage":{"prompt_tokens":2334,"completion_tokens":39,"total_tokens":2373,"prompt_cache_hit_tokens":2240,"prompt_cache_miss_tokens":94},"cost":0.00022508,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
{"ts":"2026-04-22T01:00:34.834Z","turn":1,"role":"tool","content":"Allowed directories:\nC:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo","tool":"list_allowed_directories","args":"{}"}
{"ts":"2026-04-22T01:00:38.039Z","turn":1,"role":"assistant_final","content":"Now let me list the files in the allowed directory:","usage":{"prompt_tokens":2415,"completion_tokens":74,"total_tokens":2489,"prompt_cache_hit_tokens":2304,"prompt_cache_miss_tokens":111},"cost":0.00027265000000000004,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
{"ts":"2026-04-22T01:00:38.042Z","turn":1,"role":"tool","content":"[FILE] a.txt\n[FILE] b.txt\n[FILE] README.md","tool":"list_directory","args":"{\"path\": \"C:\\\\Users\\\\yuhuahui\\\\AppData\\\\Local\\\\Temp\\\\mcp-fs-demo\"}"}
{"ts":"2026-04-22T01:00:41.597Z","turn":1,"role":"assistant_final","content":"Now I'll read the README.md file:","usage":{"prompt_tokens":2523,"completion_tokens":77,"total_tokens":2600,"prompt_cache_hit_tokens":2432,"prompt_cache_miss_tokens":91},"cost":0.00027951,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
{"ts":"2026-04-22T01:00:41.601Z","turn":1,"role":"tool","content":"# Sample","tool":"read_text_file","args":"{\"path\": \"C:\\\\Users\\\\yuhuahui\\\\AppData\\\\Local\\\\Temp\\\\mcp-fs-demo\\\\README.md\"}"}
{"ts":"2026-04-22T01:00:43.759Z","turn":1,"role":"assistant_final","content":"**Files in the directory:** a.txt, b.txt, README.md\n\n**README.md summary:** This is a sample README file.","usage":{"prompt_tokens":2619,"completion_tokens":29,"total_tokens":2648,"prompt_cache_hit_tokens":2560,"prompt_cache_miss_tokens":59},"cost":0.00022703000000000003,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
{"ts":"2026-04-22T01:00:43.759Z","turn":1,"role":"done","content":"**Files in the directory:** a.txt, b.txt, README.md\n\n**README.md summary:** This is a sample README file."}
@@ -0,0 +1,12 @@
{"role":"_meta","meta":{"version":1,"source":"reasonix run","model":"deepseek-chat","startedAt":"2026-04-22T01:39:54.789Z"}}
{"ts":"2026-04-22T01:39:54.790Z","turn":1,"role":"user","content":"First use the add tool to compute 17+25, then use write_file to save the result to a file called answer.txt in the current working directory."}
{"ts":"2026-04-22T01:39:58.843Z","turn":1,"role":"assistant_final","content":"I'll help you with that. First, I'll compute 17+25 using the add tool, then write the result to answer.txt.","usage":{"prompt_tokens":2404,"completion_tokens":87,"total_tokens":2491,"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":2148},"cost":0.0006935800000000001,"model":"deepseek-chat","prefixHash":"ea71560eab18d572"}
{"ts":"2026-04-22T01:39:58.844Z","turn":1,"role":"tool","content":"42","tool":"demo_add","args":"{\"a\": 17, \"b\": 25}"}
{"ts":"2026-04-22T01:40:02.898Z","turn":1,"role":"assistant_final","content":"Now I'll write the result (42) to a file called answer.txt in the current working directory:","usage":{"prompt_tokens":2509,"completion_tokens":81,"total_tokens":2590,"prompt_cache_hit_tokens":2432,"prompt_cache_miss_tokens":77},"cost":0.00028013,"model":"deepseek-chat","prefixHash":"ea71560eab18d572"}
{"ts":"2026-04-22T01:40:02.900Z","turn":1,"role":"tool","content":"ERROR: Access denied - path outside allowed directories: F:\\Reasonix\\answer.txt not in C:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo","tool":"fs_write_file","args":"{\"path\": \"answer.txt\", \"content\": \"42\"}"}
{"ts":"2026-04-22T01:40:05.263Z","turn":1,"role":"assistant_final","content":"I need to check which directories are allowed first. Let me check the allowed directories:","usage":{"prompt_tokens":2648,"completion_tokens":48,"total_tokens":2696,"prompt_cache_hit_tokens":2560,"prompt_cache_miss_tokens":88},"cost":0.00025576,"model":"deepseek-chat","prefixHash":"ea71560eab18d572"}
{"ts":"2026-04-22T01:40:05.270Z","turn":1,"role":"tool","content":"Allowed directories:\nC:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo","tool":"fs_list_allowed_directories","args":"{}"}
{"ts":"2026-04-22T01:40:09.422Z","turn":1,"role":"assistant_final","content":"Now I'll write the file to the allowed directory:","usage":{"prompt_tokens":2738,"completion_tokens":94,"total_tokens":2832,"prompt_cache_hit_tokens":2624,"prompt_cache_miss_tokens":114},"cost":0.00031786,"model":"deepseek-chat","prefixHash":"ea71560eab18d572"}
{"ts":"2026-04-22T01:40:09.426Z","turn":1,"role":"tool","content":"Successfully wrote to C:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo\\answer.txt","tool":"fs_write_file","args":"{\"path\": \"C:\\\\Users\\\\yuhuahui\\\\AppData\\\\Local\\\\Temp\\\\mcp-fs-demo\\\\answer.txt\", \"content\": \"42\"}"}
{"ts":"2026-04-22T01:40:13.689Z","turn":1,"role":"assistant_final","content":"Perfect! I've completed both tasks:\n\n1. **Computed 17+25**: The result is 42\n2. **Saved to answer.txt**: Created the file `answer.txt` in the allowed directory with the content \"42\"\n\nThe file has been successfully written to `C:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo\\answer.txt`.","usage":{"prompt_tokens":2878,"completion_tokens":83,"total_tokens":2961,"prompt_cache_hit_tokens":2816,"prompt_cache_miss_tokens":62},"cost":0.00030516,"model":"deepseek-chat","prefixHash":"ea71560eab18d572"}
{"ts":"2026-04-22T01:40:13.689Z","turn":1,"role":"done","content":"Perfect! I've completed both tasks:\n\n1. **Computed 17+25**: The result is 42\n2. **Saved to answer.txt**: Created the file `answer.txt` in the allowed directory with the content \"42\"\n\nThe file has been successfully written to `C:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo\\answer.txt`."}
@@ -0,0 +1,11 @@
{"role":"_meta","meta":{"version":1,"source":"bench/baseline","model":"deepseek-chat","task":"t01_address_happy","mode":"baseline","repeat":1,"startedAt":"2026-04-21T13:21:57.995Z"}}
{"ts":"2026-04-21T13:22:00.112Z","turn":1,"role":"user","content":"Hi, I need to update the shipping address for my order o_1002."}
{"ts":"2026-04-21T13:22:03.142Z","turn":1,"role":"assistant_final","content":"I can help you update the shipping address for your order. First, I need to verify your identity. Could you please provide your name and order ID? You've already given me order ID o_1002, but I also need your name to proceed with the address update.","usage":{"prompt_tokens":733,"completion_tokens":56,"total_tokens":789,"prompt_cache_hit_tokens":64,"prompt_cache_miss_tokens":669},"cost":0.00024671000000000003,"model":"deepseek-chat"}
{"ts":"2026-04-21T13:22:04.688Z","turn":2,"role":"user","content":"My name is Bo Wang."}
{"ts":"2026-04-21T13:22:10.465Z","turn":2,"role":"assistant_final","content":"Thank you, Bo Wang. Let me first look up your order o_1002 to verify the details and check its current status.","usage":{"prompt_tokens":799,"completion_tokens":74,"total_tokens":873,"prompt_cache_hit_tokens":64,"prompt_cache_miss_tokens":735},"cost":0.00028433000000000006,"model":"deepseek-chat"}
{"ts":"2026-04-21T13:22:10.465Z","turn":2,"role":"tool","content":"{\"orderId\":\"o_1002\",\"userId\":\"u_bo\",\"status\":\"processing\",\"address\":\"22 Oak Rd, NYC, NY 10001\",\"item\":\"running shoes 10\",\"price\":140}","tool":"lookup_order","args":"{\"orderId\": \"o_1002\"}"}
{"ts":"2026-04-21T13:22:10.465Z","turn":2,"role":"assistant_final","content":"Great! I can see your order o_1002 is currently in \"processing\" status, which means I can update the shipping address. What would you like the new shipping address to be?","usage":{"prompt_tokens":932,"completion_tokens":39,"total_tokens":971,"prompt_cache_hit_tokens":832,"prompt_cache_miss_tokens":100},"cost":0.00012814,"model":"deepseek-chat"}
{"ts":"2026-04-21T13:22:11.830Z","turn":3,"role":"user","content":"Please change the shipping address to 5 Birch Rd, NYC, NY 10001."}
{"ts":"2026-04-21T13:22:19.185Z","turn":3,"role":"assistant_final","content":"Perfect! I'll update the shipping address for your order o_1002 from \"22 Oak Rd, NYC, NY 10001\" to \"5 Birch Rd, NYC, NY 10001\".","usage":{"prompt_tokens":993,"completion_tokens":112,"total_tokens":1105,"prompt_cache_hit_tokens":64,"prompt_cache_miss_tokens":929},"cost":0.00037851,"model":"deepseek-chat"}
{"ts":"2026-04-21T13:22:19.185Z","turn":3,"role":"tool","content":"{\"ok\":true,\"orderId\":\"o_1002\",\"newAddress\":\"5 Birch Rd, NYC, NY 10001\"}","tool":"update_address","args":"{\"orderId\": \"o_1002\", \"address\": \"5 Birch Rd, NYC, NY 10001\"}"}
{"ts":"2026-04-21T13:22:19.185Z","turn":3,"role":"assistant_final","content":"Success! I've updated the shipping address for your order o_1002 to \"5 Birch Rd, NYC, NY 10001\". Your order is still in processing status, and the new address will be used for shipping. Is there anything else I can help you with?","usage":{"prompt_tokens":1149,"completion_tokens":56,"total_tokens":1205,"prompt_cache_hit_tokens":1088,"prompt_cache_miss_tokens":61},"cost":0.00015423000000000002,"model":"deepseek-chat"}
@@ -0,0 +1,45 @@
# Transcript diff: baseline vs reasonix
## Meta
| | baseline | reasonix |
|---|---|---|
| source | bench/baseline | bench/reasonix |
| model | deepseek-chat | deepseek-chat |
| task | t01_address_happy | t01_address_happy |
| startedAt | 2026-04-21T13:21:57.995Z | 2026-04-21T13:22:21.292Z |
## Summary
| metric | baseline | reasonix | delta |
|---|---:|---:|---:|
| model calls | 5 | 6 | +1 |
| user turns | 3 | 3 | 0 |
| tool calls | 2 | 3 | +1 |
| cache hit | 45.9% | 93.9% | **+48.0pp** |
| cost (USD) | $0.001192 | $0.000953 | -20.1% |
| prefix hashes | 0 | 1 | — |
## Turn-by-turn
| turn | kind | baseline tool calls | reasonix tool calls | note |
|---:|:---:|---|---|---|
| 1 | diverge | — | lookup_order | tool calls differ: A=[—] B=[lookup_order] |
| 2 | diverge | lookup_order | lookup_user | tool calls differ: A=[lookup_order] B=[lookup_user] |
| 3 | diverge | update_address | update_address | text similarity 68% |
## First divergence (turn 1)
tool calls differ: A=[—] B=[lookup_order]
**baseline:**
```
I can help you update the shipping address for your order. First, I need to verify your identity. Could you please provide your name and order ID? You've already given me order ID o_1002, but I also need your name to proceed with the address update.
```
**reasonix:**
```
Great! I can see order o_1002 is in "processing" status, which means I can update the shipping address. However, I still need to verify your identity. Could you please provide your name so I can look up your user information using the user ID associated with this order?
```
@@ -0,0 +1,16 @@
{"role":"_meta","meta":{"version":1,"source":"bench/reasonix","model":"deepseek-chat","task":"t01_address_happy","mode":"reasonix","repeat":1,"startedAt":"2026-04-21T13:22:21.292Z"}}
{"ts":"2026-04-21T13:22:22.687Z","turn":1,"role":"user","content":"Hi, I need to update the shipping address for my order o_1002."}
{"ts":"2026-04-21T13:22:27.201Z","turn":1,"role":"assistant_final","content":"I can help you update the shipping address for your order o_1002. First, I need to verify your identity. Could you please provide your name and order ID? I'll look up your order to check its current status and verify the details.","usage":{"prompt_tokens":713,"completion_tokens":98,"total_tokens":811,"prompt_cache_hit_tokens":704,"prompt_cache_miss_tokens":9},"cost":0.00015951000000000002,"model":"deepseek-chat","prefixHash":"fcb40cfd555bf805"}
{"ts":"2026-04-21T13:22:27.201Z","turn":1,"role":"tool","content":"{\"orderId\":\"o_1002\",\"userId\":\"u_bo\",\"status\":\"processing\",\"address\":\"22 Oak Rd, NYC, NY 10001\",\"item\":\"running shoes 10\",\"price\":140}","tool":"lookup_order","args":"{\"orderId\": \"o_1002\"}"}
{"ts":"2026-04-21T13:22:30.465Z","turn":1,"role":"assistant_final","content":"Great! I can see order o_1002 is in \"processing\" status, which means I can update the shipping address. However, I still need to verify your identity. Could you please provide your name so I can look up your user information using the user ID associated with this order?","usage":{"prompt_tokens":870,"completion_tokens":59,"total_tokens":929,"prompt_cache_hit_tokens":768,"prompt_cache_miss_tokens":102},"cost":0.00014620000000000003,"model":"deepseek-chat","prefixHash":"fcb40cfd555bf805"}
{"ts":"2026-04-21T13:22:30.465Z","turn":1,"role":"done","content":"Great! I can see order o_1002 is in \"processing\" status, which means I can update the shipping address. However, I still need to verify your identity. Could you please provide your name so I can look up your user information using the user ID associated with this order?"}
{"ts":"2026-04-21T13:22:31.801Z","turn":2,"role":"user","content":"My name is Bo Wang."}
{"ts":"2026-04-21T13:22:34.932Z","turn":2,"role":"assistant_final","content":"Perfect, thank you Bo Wang. Let me verify your identity by looking up your user information.","usage":{"prompt_tokens":939,"completion_tokens":64,"total_tokens":1003,"prompt_cache_hit_tokens":896,"prompt_cache_miss_tokens":43},"cost":0.00014473,"model":"deepseek-chat","prefixHash":"fcb40cfd555bf805"}
{"ts":"2026-04-21T13:22:34.932Z","turn":2,"role":"tool","content":"{\"userId\":\"u_bo\",\"name\":\"Bo Wang\",\"email\":\"[email protected]\"}","tool":"lookup_user","args":"{\"userId\": \"u_bo\"}"}
{"ts":"2026-04-21T13:22:38.670Z","turn":2,"role":"assistant_final","content":"Excellent! Your identity has been verified, Bo Wang. I can see your order o_1002 is currently in \"processing\" status with the shipping address \"22 Oak Rd, NYC, NY 10001\".\n\nWhat would you like the new shipping address to be? Please provide the complete new address so I can update it for you.","usage":{"prompt_tokens":1039,"completion_tokens":68,"total_tokens":1107,"prompt_cache_hit_tokens":960,"prompt_cache_miss_tokens":79},"cost":0.00016333000000000002,"model":"deepseek-chat","prefixHash":"fcb40cfd555bf805"}
{"ts":"2026-04-21T13:22:38.670Z","turn":2,"role":"done","content":"Excellent! Your identity has been verified, Bo Wang. I can see your order o_1002 is currently in \"processing\" status with the shipping address \"22 Oak Rd, NYC, NY 10001\".\n\nWhat would you like the new shipping address to be? Please provide the complete new address so I can update it for you."}
{"ts":"2026-04-21T13:22:41.867Z","turn":3,"role":"user","content":"Please change it to 5 Birch Rd, NYC, NY 10001."}
{"ts":"2026-04-21T13:22:45.479Z","turn":3,"role":"assistant_final","content":"","usage":{"prompt_tokens":1127,"completion_tokens":72,"total_tokens":1199,"prompt_cache_hit_tokens":1088,"prompt_cache_miss_tokens":39},"cost":0.00016589000000000003,"model":"deepseek-chat","prefixHash":"fcb40cfd555bf805"}
{"ts":"2026-04-21T13:22:45.479Z","turn":3,"role":"tool","content":"{\"ok\":true,\"orderId\":\"o_1002\",\"newAddress\":\"5 Birch Rd, NYC, NY 10001\"}","tool":"update_address","args":"{\"orderId\": \"o_1002\", \"address\": \"5 Birch Rd, NYC, NY 10001\"}"}
{"ts":"2026-04-21T13:22:48.394Z","turn":3,"role":"assistant_final","content":"Perfect! I've successfully updated the shipping address for your order o_1002. The new address is now \"5 Birch Rd, NYC, NY 10001\". \n\nYour order is still in processing status and will be shipped to this new address. Is there anything else I can help you with today?","usage":{"prompt_tokens":1243,"completion_tokens":62,"total_tokens":1305,"prompt_cache_hit_tokens":1152,"prompt_cache_miss_tokens":91},"cost":0.00017341000000000004,"model":"deepseek-chat","prefixHash":"fcb40cfd555bf805"}
{"ts":"2026-04-21T13:22:48.394Z","turn":3,"role":"done","content":"Perfect! I've successfully updated the shipping address for your order o_1002. The new address is now \"5 Birch Rd, NYC, NY 10001\". \n\nYour order is still in processing status and will be shipped to this new address. Is there anything else I can help you with today?"}
+98
View File
@@ -0,0 +1,98 @@
/**
* Task and result types for the tool-use eval harness.
*
* Scope note: this is NOT a full port of Sierra's τ-bench (airline+retail).
* We mirror its *shape* — multi-turn tool-use with an LLM user simulator and
* DB-end-state success predicates — so a later port can drop real tasks in
* without changing the harness. Until then, see `tasks.ts` for our seed set.
*/
import type { ToolDefinition } from "../../src/index.js";
/**
* The mutable world state a task's tools operate on. Tasks deep-clone this
* into a per-run snapshot so runs don't leak state across each other.
*/
export interface WorldState {
[table: string]: Record<string, Record<string, unknown>>;
}
export interface UserPersona {
/** Who the user is roleplaying (e.g. "frustrated customer"). */
style: string;
/** The concrete goal. The user pursues this until it's met or clearly refused. */
goal: string;
/**
* Facts the simulator may reveal when the agent asks. Keep tight — the
* user shouldn't volunteer everything up front.
*/
knowns: Record<string, string>;
}
/**
* A tool factory. We take a factory (not a ToolDefinition directly) because
* each run needs a fresh closure over its per-run WorldState — otherwise all
* runs would share (and mutate) the same DB.
*/
export type ToolFactory = (db: WorldState) => ToolDefinition;
export interface TaskDefinition {
id: string;
/** One-line human description. Not shown to the model. */
description: string;
/** System prompt given to the agent. Kept small so cache-hit ratio is comparable. */
systemPrompt: string;
/** Tools built fresh per run against the run's DB snapshot. */
tools: ToolFactory[];
/** Initial DB snapshot. Deep-cloned per run. */
initialDb: WorldState;
/** Persona + goal for the LLM user simulator. */
user: UserPersona;
/** Max turns of (user → agent) before we give up and mark fail. */
maxTurns?: number;
/**
* Success predicate. Given the end-state DB (and optionally the final
* agent utterance), return true iff the task is considered solved.
*/
check: (ctx: { db: WorldState; finalAgentMessage: string; transcript: Turn[] }) => boolean;
}
export interface Turn {
role: "user" | "agent" | "tool";
content: string;
toolName?: string;
}
export type RunMode = "baseline" | "reasonix";
export interface RunResult {
taskId: string;
mode: RunMode;
pass: boolean;
turns: number;
toolCalls: number;
cacheHitRatio: number;
costUsd: number;
claudeEquivalentUsd: number;
promptTokens: number;
completionTokens: number;
/** True if the run aborted before the user sim decided to stop. */
truncated: boolean;
finalAgentMessage: string;
errorMessage?: string;
}
export interface BenchMeta {
date: string;
model: string;
userSimModel: string;
taskCount: number;
repeatsPerTask: number;
/** Reasonix version written into the report for reproducibility. */
reasonixVersion: string;
}
export interface BenchReport {
meta: BenchMeta;
results: RunResult[];
}
+92
View File
@@ -0,0 +1,92 @@
/**
* LLM-backed user simulator for tool-use eval.
*
* A cheap V3 call that given (persona, goal, transcript so far) emits either
* the user's next utterance or the literal sentinel `##STOP##` to end the
* conversation. Kept intentionally small and low-temperature so dialogue
* stays on-task and is replayable.
*
* We accept "some" non-determinism (DeepSeek samples at T=0.1) — the honest
* way to deal with run-to-run drift is repeat-per-task in the runner, not
* trying to pretend the sim is deterministic.
*/
import type { ChatMessage, DeepSeekClient } from "../../src/index.js";
import type { Turn, UserPersona } from "./types.js";
const SYS = `You are roleplaying a user contacting a retail support agent.
Rules:
- Stay in character; never break the fourth wall.
- Never reveal you are an AI or mention any system.
- Pursue the goal. Do not volunteer facts the agent hasn't asked for.
- Keep replies to one or two sentences.
- When the goal is clearly met OR clearly refused by the agent, output ONLY the literal token: ##STOP##
- Do not output ##STOP## on your first message — give the agent a chance.`;
export interface UserSimOptions {
model?: string;
temperature?: number;
}
export class UserSimulator {
constructor(
private client: DeepSeekClient,
private persona: UserPersona,
private opts: UserSimOptions = {},
) {}
/**
* Given the transcript so far, produce the user's next line, or null if
* the simulator decided the conversation is over.
*/
async next(transcript: Turn[]): Promise<string | null> {
const knowns = JSON.stringify(this.persona.knowns, null, 2);
const messages: ChatMessage[] = [
{
role: "system",
content: `${SYS}\n\nCharacter: ${this.persona.style}\nGoal: ${this.persona.goal}\nFacts you may share when asked (don't volunteer):\n${knowns}`,
},
];
if (transcript.length === 0) {
messages.push({
role: "user",
content:
"Write your opening message to the support agent. One or two sentences. Do not dump all the facts.",
});
} else {
messages.push({
role: "user",
content: `Here is the conversation so far (you are the USER).\n\n${transcriptToString(
transcript,
)}\n\nWrite ONLY your next user reply, or output ##STOP## if the goal is clearly met or clearly refused.`,
});
}
const resp = await this.client.chat({
model: this.opts.model ?? "deepseek-chat",
messages,
temperature: this.opts.temperature ?? 0.1,
maxTokens: 200,
});
const text = resp.content.trim();
if (!text) return null;
if (text === "##STOP##" || text.endsWith("##STOP##") || text.includes("##STOP##")) return null;
return text;
}
}
function transcriptToString(turns: Turn[]): string {
const lines: string[] = [];
for (const t of turns) {
if (t.role === "user") lines.push(`USER: ${t.content}`);
else if (t.role === "agent") lines.push(`AGENT: ${t.content}`);
else if (t.role === "tool")
lines.push(`(tool ${t.toolName} returned: ${truncate(t.content, 200)})`);
}
return lines.join("\n");
}
function truncate(s: string, n: number): string {
return s.length > n ? `${s.slice(0, n)}…` : s;
}
+29
View File
@@ -0,0 +1,29 @@
{
"$schema": "https://biomejs.dev/schemas/1.9.4/schema.json",
"organizeImports": { "enabled": true },
"formatter": {
"enabled": true,
"indentStyle": "space",
"indentWidth": 2,
"lineWidth": 100
},
"javascript": {
"formatter": { "quoteStyle": "double", "semicolons": "always", "trailingCommas": "all" }
},
"linter": {
"enabled": true,
"rules": {
"recommended": true,
"style": {
"noNonNullAssertion": "off",
"useImportType": "warn"
},
"suspicious": {
"noExplicitAny": "off"
}
}
},
"files": {
"ignore": ["dist", "node_modules", "coverage", "*.d.ts"]
}
}
+147
View File
@@ -0,0 +1,147 @@
# Reasonix Architecture
## Design philosophy
Reasonix is **opinionated, not general**. Every abstraction is justified by a
DeepSeek-specific behavior or economic property. If it's generic, we don't
ship it.
## The three pillars
### Pillar 1 — Cache-First Loop
**Problem.** DeepSeek bills cached input at ~10% of the miss rate. Automatic
prefix caching activates only when the *exact* byte prefix of the previous
request matches. Most agent loops reorder, rewrite, or inject fresh
timestamps each turn — cache hit rate in practice: <20%.
**Solution.** Partition the context into three regions:
```
┌─────────────────────────────────────────┐
│ IMMUTABLE PREFIX │ ← fixed for session
│ system + tool_specs + few_shots │ cache hit candidate
├─────────────────────────────────────────┤
│ APPEND-ONLY LOG │ ← grows monotonically
│ [assistant₁][tool₁][assistant₂]... │ preserves prefix of prior turns
├─────────────────────────────────────────┤
│ VOLATILE SCRATCH │ ← reset each turn
│ R1 thought, transient plan state │ never sent upstream
└─────────────────────────────────────────┘
```
**Invariants:**
1. Prefix is computed once per session, hashed, and pinned.
2. Log entries are serialized in append order; no rewrites.
3. Scratch is distilled via Pillar 2 before any information from it is folded
into the log.
**Metric.** `prompt_cache_hit_tokens / (hit + miss)` exposed per-turn and
aggregated per-session. This is the user-visible proof of Pillar 1's value.
### Pillar 2 — R1 Thought Harvesting *(v0.0.3, opt-in)*
**Problem.** R1 emits extensive `reasoning_content`. DeepSeek's own docs
recommend *not* feeding it back to the next turn. Most frameworks display it
to the user and discard. The planning signal inside is lost.
**Solution.** A two-stage process:
```
R1 output → Harvester (V3, cheap) → TypedPlanState
├─ subgoals: string[]
├─ hypotheses: string[]
├─ uncertainties: string[]
└─ rejectedPaths: string[]
```
The harvester is a cheap V3 call with a strict JSON schema. Output is
validated at runtime. The typed state is queryable by the orchestrator — e.g.
"if `uncertainties.length > 2`, trigger branch sampling."
### Branch-and-Select *(v0.0.5, opt-in, builds on Pillar 2)*
**Why now.** DeepSeek is cheap enough that running N=3 R1 samples is still
cheaper than a single Claude call. What was research luxury (self-consistency
sampling) becomes a practical default.
**How.** When `--branch N` (or `{ branch: N }` in code) is passed:
1. The turn is forced non-streaming (N samples must complete before we can
compare).
2. `runBranches` fires N `client.chat` calls in parallel, spreading
temperatures across `[0, 1]` to diversify reasoning paths.
3. Each sample's `reasoning_content` is piped through Pillar 2's harvest.
4. The default selector picks the sample with the fewest `uncertainties`;
ties are broken by shorter answer length (Occam).
5. The loop emits a `branch_summary` alongside the winning `assistant_final`.
**Cost.** 3× R1 on a hard math problem ≈ \$0.02 — still dominates Claude on
cost/quality for agentic work.
### Pillar 3 — Tool-Call Repair *(v0.0.1 ships complete)*
**Problem.** Empirical DeepSeek failure modes:
- Tool-call JSON emitted inside `<think>`, missing from the final message.
- Arguments dropped when schema has >10 params or deeply nested objects.
- Same tool called repeatedly with identical args (call-storm).
- Truncated JSON due to `max_tokens` hit mid-structure.
**Solution.** Four passes:
1. **`flatten`** — schemas with >10 leaf params or depth >2 are auto-detected
on `ToolRegistry.register()` and presented to the model in dot-notation
form. `dispatch()` re-nests the args before calling the user's `fn`.
Opt out with `new ToolRegistry({ autoFlatten: false })`.
2. **`scavenge`** — regex + JSON parser sweeps `reasoning_content` for any tool
call the model forgot to emit in `tool_calls`.
3. **`truncation`** — detect unbalanced JSON and repair by closing braces or
requesting a continuation completion.
4. **`storm`** — identical `(tool, args)` tuple within a sliding window →
suppress the call, inject a reflection turn.
## Module layout
```
src/
├── client.ts # httpx-equivalent DeepSeek client (fetch + SSE)
├── loop.ts # Pillar 1: Cache-First Loop (async iterator)
├── harvest.ts # Pillar 2 stub (v0.0.1 surface only)
├── repair/
│ ├── index.ts # Pillar 3 pipeline
│ ├── scavenge.ts
│ ├── flatten.ts
│ ├── truncation.ts
│ └── storm.ts
├── tools.ts # Tool registry + dispatch
├── memory.ts # Prefix / Log / Scratch primitives
├── telemetry.ts # Cost & cache-hit accounting
├── types.ts # Shared type definitions
├── index.ts # Library barrel export
└── cli/
├── index.ts # commander entry
├── commands/ # chat, run, stats, version
└── ui/ # Ink React components (App, StatsPanel, EventLog, PromptInput)
```
## Roadmap
- **v0.0.1** — Pillar 1 end-to-end, Pillar 3 complete, Ink TUI, τ-bench scaffold.
- **v0.0.2** — First-run key prompt, saved to `~/.reasonix/config.json`.
- **v0.0.3** — Pillar 2 MVP (opt-in harvest), retry layer, TextInput fix.
- **v0.0.4** — Schema flatten auto-applied in ToolRegistry (closes Pillar 3).
- **v0.0.5** — Self-consistency branching (`--branch N`) driven by Pillar 2
plan-state uncertainty count.
- **v0.0.6** — Named session persistence (`--session <name>`), JSONL log
stored under `~/.reasonix/sessions/`, restored into the append-only log
on resume.
- **v0.1** — τ-bench numbers published, streaming polish, transcript replay.
- **v0.2** — Self-consistency / branch-budget sampling driven by plan state.
- **v0.3** — MCP client, session persistence.
## Explicit non-goals
- Multi-agent orchestration.
- RAG / vector retrieval.
- Support for non-DeepSeek backends.
- Web UI / SaaS.
+25
View File
@@ -0,0 +1,25 @@
/** Minimal example: one-shot, non-streaming. Needs DEEPSEEK_API_KEY. */
import {
CacheFirstLoop,
DeepSeekClient,
ImmutablePrefix,
loadDotenv,
} from "../src/index.js";
loadDotenv();
async function main() {
const client = new DeepSeekClient();
const prefix = new ImmutablePrefix({ system: "You are a concise assistant." });
const loop = new CacheFirstLoop({ client, prefix, stream: false });
const answer = await loop.run("In one sentence, what is prompt caching?");
console.log(answer);
console.log("---");
console.log(loop.stats.summary());
}
main().catch((err) => {
console.error(err);
process.exit(1);
});
+177
View File
@@ -0,0 +1,177 @@
/**
* Bundled demo MCP server.
*
* A minimal stdio MCP server that exposes three tools: echo, add, get_time.
* Useful for:
* - running the MCP integration end-to-end without installing
* an external server
* - giving the integration tests a real subprocess to spawn
* - showing the minimal shape of a server for folks writing their own
*
* Usage:
* npx tsx examples/mcp-server-demo.ts # speaks MCP on stdin/stdout
* reasonix chat --mcp "npx tsx examples/mcp-server-demo.ts"
*
* Spec reference: https://spec.modelcontextprotocol.io/ (2024-11-05)
* Only the subset this demo needs is implemented — initialize, tools/list,
* tools/call, notifications/initialized (no-op).
*/
import { createInterface } from "node:readline";
const PROTOCOL_VERSION = "2024-11-05";
interface JsonRpcRequest {
jsonrpc: "2.0";
id?: string | number;
method: string;
params?: unknown;
}
interface JsonRpcSuccess {
jsonrpc: "2.0";
id: string | number;
result: unknown;
}
interface JsonRpcError {
jsonrpc: "2.0";
id: string | number | null;
error: { code: number; message: string; data?: unknown };
}
const TOOLS = [
{
name: "echo",
description: "Echoes the provided message back.",
inputSchema: {
type: "object",
properties: { msg: { type: "string", description: "What to echo" } },
required: ["msg"],
},
},
{
name: "add",
description: "Adds two integers and returns the sum.",
inputSchema: {
type: "object",
properties: {
a: { type: "integer" },
b: { type: "integer" },
},
required: ["a", "b"],
},
},
{
name: "get_time",
description: "Returns the server's current ISO-8601 timestamp.",
inputSchema: { type: "object", properties: {} },
},
];
function send(msg: JsonRpcSuccess | JsonRpcError): void {
// Stdio MCP framing: one JSON per line.
process.stdout.write(`${JSON.stringify(msg)}\n`);
}
function handleRequest(req: JsonRpcRequest): JsonRpcSuccess | JsonRpcError | null {
const id = req.id ?? null;
switch (req.method) {
case "initialize": {
return {
jsonrpc: "2.0",
id: id ?? 0,
result: {
protocolVersion: PROTOCOL_VERSION,
serverInfo: { name: "reasonix-demo-mcp", version: "0.0.1" },
capabilities: { tools: { listChanged: false } },
},
};
}
case "notifications/initialized":
// No response for notifications.
return null;
case "tools/list": {
return { jsonrpc: "2.0", id: id ?? 0, result: { tools: TOOLS } };
}
case "tools/call": {
const params = (req.params ?? {}) as { name?: string; arguments?: Record<string, unknown> };
const name = params.name ?? "";
const args = params.arguments ?? {};
const out = callTool(name, args);
if (out.error) {
return {
jsonrpc: "2.0",
id: id ?? 0,
result: {
content: [{ type: "text", text: out.error }],
isError: true,
},
};
}
return {
jsonrpc: "2.0",
id: id ?? 0,
result: { content: [{ type: "text", text: out.text }] },
};
}
default:
return {
jsonrpc: "2.0",
id,
error: { code: -32601, message: `method not found: ${req.method}` },
};
}
}
function callTool(
name: string,
args: Record<string, unknown>,
): { text: string; error?: string } {
if (name === "echo") {
const msg = typeof args.msg === "string" ? args.msg : "";
return { text: `echo: ${msg}` };
}
if (name === "add") {
const a = typeof args.a === "number" ? args.a : Number(args.a);
const b = typeof args.b === "number" ? args.b : Number(args.b);
if (!Number.isFinite(a) || !Number.isFinite(b)) {
return { text: "", error: "add: both a and b must be numbers" };
}
return { text: String(a + b) };
}
if (name === "get_time") {
return { text: new Date().toISOString() };
}
return { text: "", error: `unknown tool: ${name}` };
}
function main(): void {
const rl = createInterface({ input: process.stdin, crlfDelay: Number.POSITIVE_INFINITY });
rl.on("line", (line) => {
const trimmed = line.trim();
if (!trimmed) return;
let req: JsonRpcRequest;
try {
req = JSON.parse(trimmed) as JsonRpcRequest;
} catch {
// malformed input — respond with parse error
send({
jsonrpc: "2.0",
id: null,
error: { code: -32700, message: "parse error" },
});
return;
}
const resp = handleRequest(req);
if (resp) send(resp);
});
rl.on("close", () => process.exit(0));
}
main();
+70
View File
@@ -0,0 +1,70 @@
/**
* Library example: programmatic replay + diff.
*
* This example runs with no API key and no DeepSeek calls — it reads the
* reference transcripts committed at benchmarks/tau-bench/transcripts/ and
* reconstructs the v0.1 cache-hit / cost numbers offline.
*
* Run from the repo root:
* npx tsx examples/replay-and-diff.ts
*
* Anything you can do with `reasonix replay` / `reasonix diff` is available
* here as a function you can drive from your own scripts (CI gates, eval
* dashboards, blog post generation, etc.).
*/
import {
computeReplayStats,
diffTranscripts,
readTranscript,
renderDiffSummary,
} from "../src/index.js";
const BASELINE = "benchmarks/tau-bench/transcripts/t01_address_happy.baseline.r1.jsonl";
const REASONIX = "benchmarks/tau-bench/transcripts/t01_address_happy.reasonix.r1.jsonl";
// ---------- 1. Replay a single transcript as pure data ----------
const parsed = readTranscript(REASONIX);
const stats = computeReplayStats(parsed.records);
console.log("=== Reasonix side, computed from transcript alone ===");
console.log(` model calls: ${stats.turns}`);
console.log(` cache hit: ${(stats.cacheHitRatio * 100).toFixed(1)}%`);
console.log(` total cost: $${stats.totalCostUsd.toFixed(6)}`);
console.log(` vs claude estimate: $${stats.claudeEquivalentUsd.toFixed(6)}`);
console.log(` savings: ${stats.savingsVsClaudePct.toFixed(1)}%`);
console.log(` prefix hashes: ${stats.prefixHashes.length} distinct`);
if (stats.prefixHashes.length === 1) {
console.log(` → byte-stable: ${stats.prefixHashes[0]?.slice(0, 16)}…`);
}
console.log();
// ---------- 2. Diff two transcripts ----------
const aParsed = readTranscript(BASELINE);
const bParsed = readTranscript(REASONIX);
const report = diffTranscripts(
{ label: "baseline", parsed: aParsed },
{ label: "reasonix", parsed: bParsed },
);
// renderDiffSummary returns a monochrome stdout-ready string. Equivalent to
// what `reasonix diff --print` outputs.
console.log(renderDiffSummary(report));
// ---------- 3. Direct programmatic access to pairs ----------
//
// You can also inspect report.pairs directly — useful for writing custom
// filters like "show me only the turns where tool calls differed".
console.log("\n=== Turns where A and B took different paths ===");
const divergent = report.pairs.filter((p) => p.kind !== "match");
if (divergent.length === 0) {
console.log(" (none — both agents followed the same path on every turn)");
} else {
for (const p of divergent) {
console.log(` turn ${p.turn}: ${p.kind}${p.divergenceNote ? ` — ${p.divergenceNote}` : ""}`);
}
}
+43
View File
@@ -0,0 +1,43 @@
/** Tool-use example: register a calculator tool. Needs DEEPSEEK_API_KEY. */
import {
CacheFirstLoop,
DeepSeekClient,
ImmutablePrefix,
ToolRegistry,
loadDotenv,
} from "../src/index.js";
loadDotenv();
const tools = new ToolRegistry();
tools.register<{ a: number; b: number }, number>({
name: "add",
description: "Add two integers.",
parameters: {
type: "object",
properties: {
a: { type: "integer" },
b: { type: "integer" },
},
required: ["a", "b"],
},
fn: ({ a, b }) => a + b,
});
async function main() {
const client = new DeepSeekClient();
const prefix = new ImmutablePrefix({
system: "You are a calculator assistant. Use the `add` tool for addition.",
toolSpecs: tools.specs(),
});
const loop = new CacheFirstLoop({ client, prefix, tools });
const answer = await loop.run("What is 17 + 25?");
console.log("answer:", answer);
console.log("stats:", loop.stats.summary());
}
main().catch((err) => {
console.error(err);
process.exit(1);
});
+3296
View File
File diff suppressed because it is too large. Load diff
+75
View File
@@ -0,0 +1,75 @@
{
"name": "reasonix",
"version": "0.4.4",
"description": "DeepSeek-native agent framework: cache-first loop, R1 thought harvesting, tool-call repair.",
"type": "module",
"bin": {
"reasonix": "dist/cli/index.js"
},
"main": "./dist/index.js",
"module": "./dist/index.js",
"types": "./dist/index.d.ts",
"exports": {
".": {
"types": "./dist/index.d.ts",
"import": "./dist/index.js"
}
},
"files": [
"dist",
"README.md",
"LICENSE"
],
"scripts": {
"build": "tsup",
"dev": "tsx src/cli/index.ts",
"chat": "tsx src/cli/index.ts chat",
"test": "vitest run",
"test:watch": "vitest",
"lint": "biome check src tests",
"lint:fix": "biome check --write src tests",
"format": "biome format --write src tests",
"typecheck": "tsc --noEmit",
"prepublishOnly": "npm run lint && npm run typecheck && npm run test && npm run build"
},
"keywords": [
"agent",
"llm",
"deepseek",
"r1",
"tool-use",
"prompt-cache",
"cli",
"tui"
],
"author": "esengine",
"license": "MIT",
"repository": {
"type": "git",
"url": "git+https://github.com/esengine/reasonix.git"
},
"bugs": {
"url": "https://github.com/esengine/reasonix/issues"
},
"homepage": "https://github.com/esengine/reasonix#readme",
"engines": {
"node": ">=20.10"
},
"dependencies": {
"commander": "^12.1.0",
"eventsource-parser": "^3.0.0",
"ink": "^5.1.0",
"ink-spinner": "^5.0.0",
"ink-text-input": "^6.0.0",
"react": "^18.3.1"
},
"devDependencies": {
"@biomejs/biome": "^1.9.4",
"@types/node": "^22.9.0",
"@types/react": "^18.3.12",
"tsup": "^8.3.5",
"tsx": "^4.19.2",
"typescript": "^5.6.3",
"vitest": "^2.1.5"
}
}
+131
View File
@@ -0,0 +1,131 @@
import { render } from "ink";
import React, { useState } from "react";
import { loadApiKey } from "../../config.js";
import { loadDotenv } from "../../env.js";
import { McpClient } from "../../mcp/client.js";
import { bridgeMcpTools } from "../../mcp/registry.js";
import { parseMcpSpec } from "../../mcp/spec.js";
import { SseTransport } from "../../mcp/sse.js";
import { type McpTransport, StdioTransport } from "../../mcp/stdio.js";
import { ToolRegistry } from "../../tools.js";
import { App } from "../ui/App.js";
import { Setup } from "../ui/Setup.js";
export interface ChatOptions {
model: string;
system: string;
transcript?: string;
harvest?: boolean;
branch?: number;
session?: string;
/** Zero or more MCP server specs. Each: `"name=cmd args..."` or `"cmd args..."`. */
mcp?: string[];
/** Global prefix — only used when a single anonymous server is given. */
mcpPrefix?: string;
/**
* Enable SEARCH/REPLACE edit-block processing after each assistant turn.
* Set by `reasonix code`; plain `reasonix chat` leaves this off.
*/
codeMode?: { rootDir: string };
}
interface RootProps extends ChatOptions {
initialKey: string | undefined;
tools: ToolRegistry | undefined;
mcpSpecs: string[];
}
function Root({ initialKey, tools, mcpSpecs, ...appProps }: RootProps) {
const [key, setKey] = useState<string | undefined>(initialKey);
if (!key) {
return (
<Setup
onReady={(k) => {
process.env.DEEPSEEK_API_KEY = k;
setKey(k);
}}
/>
);
}
process.env.DEEPSEEK_API_KEY = key;
return (
<App
model={appProps.model}
system={appProps.system}
transcript={appProps.transcript}
harvest={appProps.harvest}
branch={appProps.branch}
session={appProps.session}
tools={tools}
mcpSpecs={mcpSpecs}
codeMode={appProps.codeMode}
/>
);
}
export async function chatCommand(opts: ChatOptions): Promise<void> {
loadDotenv();
const initialKey = loadApiKey();
const requestedSpecs = opts.mcp ?? [];
const clients: McpClient[] = [];
const successfulSpecs: string[] = [];
const failedSpecs: Array<{ spec: string; reason: string }> = [];
let tools: ToolRegistry | undefined;
if (requestedSpecs.length > 0) {
tools = new ToolRegistry();
for (const raw of requestedSpecs) {
try {
const spec = parseMcpSpec(raw);
const prefix = spec.name
? `${spec.name}_`
: requestedSpecs.length === 1 && opts.mcpPrefix
? opts.mcpPrefix
: "";
const transport: McpTransport =
spec.transport === "sse"
? new SseTransport({ url: spec.url })
: new StdioTransport({ command: spec.command, args: spec.args });
const mcp = new McpClient({ transport });
await mcp.initialize();
const bridge = await bridgeMcpTools(mcp, { registry: tools, namePrefix: prefix });
const label = spec.name ?? "anon";
const source =
spec.transport === "sse" ? spec.url : `${spec.command} ${spec.args.join(" ")}`;
process.stderr.write(
`▸ MCP[${label}]: ${bridge.registeredNames.length} tool(s) from ${source}\n`,
);
clients.push(mcp);
successfulSpecs.push(raw);
} catch (err) {
// Per-server failure is non-fatal: one broken server shouldn't
// kill a chat that has working servers configured. We record
// the failure, show a visible warning, and keep going. User
// can fix via `reasonix setup` (unchecks the broken entry)
// without losing their other servers.
const reason = (err as Error).message;
failedSpecs.push({ spec: raw, reason });
process.stderr.write(
`▸ MCP setup SKIPPED for "${raw}": ${reason}\n → this server will not be available this session. Run \`reasonix setup\` to remove it, or fix the underlying issue (missing npm package, network, etc.).\n`,
);
}
}
// If every requested server failed, drop the empty registry so the
// loop still runs as a bare chat instead of advertising zero tools.
if (successfulSpecs.length === 0) {
tools = undefined;
}
}
const mcpSpecs = successfulSpecs;
const { waitUntilExit } = render(
<Root initialKey={initialKey} tools={tools} mcpSpecs={mcpSpecs} {...opts} />,
{ exitOnCtrlC: true },
);
try {
await waitUntilExit();
} finally {
for (const c of clients) await c.close();
}
}
+66
View File
@@ -0,0 +1,66 @@
/**
* `reasonix code [dir]` — opinionated wrapper around `reasonix chat` for
* code-editing workflows.
*
* What it does differently from plain chat:
* - Auto-bridges the official filesystem MCP server rooted at the
* given directory (CWD by default). No wizard, no config merge.
* - Uses a coding-focused system prompt (src/code/prompt.ts) that
* teaches the model to propose edits as SEARCH/REPLACE blocks.
* - Defaults to the `smart` preset (reasoner + harvest) because
* coding tasks pay back R1 thinking.
* - Scopes its session to the directory so projects don't share
* conversation history.
* - Hooks `codeMode` into the TUI so assistant replies get parsed
* for SEARCH/REPLACE blocks and applied on disk after each turn.
*
* Out of scope for v1: /commit, /undo, diff preview, .gitignore
* filtering. The user's own `git diff` + `git checkout` is the review
* / undo surface for now.
*/
import { basename, resolve } from "node:path";
import { sanitizeName } from "../../session.js";
import { chatCommand } from "./chat.js";
export interface CodeOptions {
/** Directory to root the filesystem MCP at. Defaults to process.cwd(). */
dir?: string;
/** Override the default `smart` model. */
model?: string;
/** Disable session persistence. */
noSession?: boolean;
/** Transcript file for replay/diff. */
transcript?: string;
}
export async function codeCommand(opts: CodeOptions = {}): Promise<void> {
const { codeSystemPrompt } = await import("../../code/prompt.js");
const rootDir = resolve(opts.dir ?? process.cwd());
// Per-directory session so switching projects doesn't mix histories.
// `code-<sanitized-basename>` fits the session name rules without
// truncating most project names.
const session = opts.noSession ? undefined : `code-${sanitizeName(basename(rootDir))}`;
// Filesystem MCP spec pointing at rootDir. Quote the path in case it
// has spaces — the existing shellSplit inside parseMcpSpec understands
// double quotes.
const fsSpec = `filesystem=npx -y @modelcontextprotocol/server-filesystem ${quoteIfNeeded(rootDir)}`;
process.stderr.write(
`▸ reasonix code: rooted at ${rootDir}, session "${session ?? "(ephemeral)"}"\n`,
);
await chatCommand({
model: opts.model ?? "deepseek-reasoner",
harvest: true, // smart preset's harvest setting, always on for code
system: codeSystemPrompt(rootDir),
transcript: opts.transcript,
session,
mcp: [fsSpec],
codeMode: { rootDir },
});
}
function quoteIfNeeded(s: string): string {
return /\s|"/.test(s) ? `"${s.replace(/"/g, '\\"')}"` : s;
}
+61
View File
@@ -0,0 +1,61 @@
import { writeFileSync } from "node:fs";
import { basename } from "node:path";
import { render } from "ink";
import React from "react";
import { diffTranscripts, renderMarkdown, renderSummaryTable } from "../../diff.js";
import { readTranscript } from "../../transcript.js";
import { DiffApp } from "../ui/DiffApp.js";
export interface DiffOptions {
a: string;
b: string;
mdPath?: string;
labelA?: string;
labelB?: string;
/** Force stdout summary table (no Ink TUI). Auto when stdout isn't a TTY. */
print?: boolean;
/** Force the TUI even when stdout isn't a TTY (rare). */
tui?: boolean;
}
/**
* Compare two transcripts. Three output paths, picked in order:
* - If --md is passed: write the markdown report. Also prints the stdout
* summary so the user sees what was exported.
* - If --print, no TTY, or --md (see above): stdout summary table.
* - Otherwise: interactive Ink TUI with split-pane + n/N divergence jump.
*/
export async function diffCommand(opts: DiffOptions): Promise<void> {
const aParsed = readTranscript(opts.a);
const bParsed = readTranscript(opts.b);
const report = diffTranscripts(
{ label: opts.labelA ?? basename(opts.a), parsed: aParsed },
{ label: opts.labelB ?? basename(opts.b), parsed: bParsed },
);
const wantMarkdown = !!opts.mdPath;
const wantPrint = opts.print || !process.stdout.isTTY;
const wantTui = opts.tui || (!wantPrint && !wantMarkdown);
if (wantMarkdown) {
// Markdown export implies the user wants an artifact, not a TUI.
// Still echo the stdout summary to confirm the action.
console.log(renderSummaryTable(report));
const md = renderMarkdown(report);
writeFileSync(opts.mdPath!, md, "utf8");
console.log(`\nmarkdown report written to ${opts.mdPath}`);
return;
}
if (wantTui) {
const { waitUntilExit } = render(React.createElement(DiffApp, { report }), {
exitOnCtrlC: true,
});
await waitUntilExit();
return;
}
// stdout fallback (piped, --print, or non-TTY)
console.log(renderSummaryTable(report));
}
+41
View File
@@ -0,0 +1,41 @@
/**
* `reasonix mcp <subcmd>` — the MCP discovery / helper surface.
*
* Currently one subcommand: `list`. Prints a curated catalog of popular
* MCP servers with their `--mcp` commands ready to copy-paste. More
* subcommands (`init`, `try`, etc.) may land in later alphas.
*/
import { MCP_CATALOG, mcpCommandFor } from "../../mcp/catalog.js";
export interface McpListOptions {
/** Emit JSON on stdout instead of the human-readable table. */
json?: boolean;
}
export function mcpListCommand(opts: McpListOptions): void {
if (opts.json) {
console.log(JSON.stringify(MCP_CATALOG, null, 2));
return;
}
console.log("Popular MCP servers you can bridge into Reasonix:");
console.log("");
for (const entry of MCP_CATALOG) {
console.log(` ${pad(entry.name, 12)} ${entry.summary}`);
console.log(` ${mcpCommandFor(entry)}`);
if (entry.note) console.log(` · ${entry.note}`);
console.log("");
}
console.log("Usage: reasonix chat <one-of-the---mcp-lines-above>");
console.log(
"Docs: https://github.com/modelcontextprotocol/servers — Anthropic's official server repo",
);
console.log(
" https://mcp.so — community-maintained catalog",
);
}
function pad(s: string, width: number): string {
return s.length >= width ? s : s + " ".repeat(width - s.length);
}
+138
View File
@@ -0,0 +1,138 @@
import { render } from "ink";
import React from "react";
import { groupRecordsByTurn, replayFromFile } from "../../replay.js";
import type { TranscriptRecord } from "../../transcript.js";
import { ReplayApp } from "../ui/ReplayApp.js";
export interface ReplayOptions {
path: string;
head?: number;
tail?: number;
/** Force stdout pretty-print mode (no Ink TUI). Also auto-enabled when stdout is not a TTY. */
print?: boolean;
}
/**
* Replay a transcript. Two modes:
* - Interactive TUI (default when stdout is a TTY): Ink-based j/k navigation.
* - Stdout pretty-print (--print, or when stdout is piped): one-shot text
* dump + summary. Kept working so shell pipes, CI logs, and `less`
* workflows still behave sensibly.
*/
export async function replayCommand(opts: ReplayOptions): Promise<void> {
const wantPrint =
opts.print || !process.stdout.isTTY || opts.head !== undefined || opts.tail !== undefined;
if (wantPrint) {
printReplay(opts);
return;
}
const { parsed } = replayFromFile(opts.path);
const pages = groupRecordsByTurn(parsed.records);
const { waitUntilExit } = render(React.createElement(ReplayApp, { meta: parsed.meta, pages }), {
exitOnCtrlC: true,
});
await waitUntilExit();
}
// ----------------------------------------------------------------------------
// stdout pretty-print path (original behavior, preserved for piping / CI)
function printReplay(opts: ReplayOptions): void {
const { parsed, stats } = replayFromFile(opts.path);
if (parsed.meta) {
const m = parsed.meta;
const bits: string[] = [`source=${m.source}`];
if (m.model) bits.push(`model=${m.model}`);
if (m.task) bits.push(`task=${m.task}`);
if (m.mode) bits.push(`mode=${m.mode}`);
if (m.repeat !== undefined) bits.push(`repeat=${m.repeat}`);
bits.push(`started=${m.startedAt}`);
console.log(`[meta] ${bits.join(" ")}`);
console.log("");
}
const records = sliceRecords(parsed.records, opts);
for (const rec of records) {
renderRecord(rec);
}
console.log("");
console.log("── summary ─────────────────────────────────────────");
console.log(`model calls: ${stats.turns}`);
console.log(`user turns: ${stats.userTurns}`);
console.log(`tool calls: ${stats.toolCalls}`);
console.log(`cache hit: ${(stats.cacheHitRatio * 100).toFixed(1)}%`);
console.log(`cost: $${stats.totalCostUsd.toFixed(6)}`);
console.log(`claude equivalent: $${stats.claudeEquivalentUsd.toFixed(6)}`);
console.log(`savings vs claude: ${stats.savingsVsClaudePct.toFixed(1)}%`);
console.log(`models: ${stats.models.join(", ") || "—"}`);
console.log(`prefix hashes: ${stats.prefixHashes.length} distinct`);
if (stats.prefixHashes.length === 1) {
console.log(` (byte-stable prefix: ${stats.prefixHashes[0]?.slice(0, 16)}…)`);
} else if (stats.prefixHashes.length > 1) {
console.log(" (prefix churned — cache-hostile session)");
}
if (stats.harvestedTurns > 0) {
console.log(`harvest (Pillar 2): ${stats.harvestedTurns} turn(s) produced plan state`);
console.log(` subgoals: ${stats.totalSubgoals}`);
console.log(` uncertainties: ${stats.totalUncertainties}`);
}
}
function sliceRecords(records: TranscriptRecord[], opts: ReplayOptions): TranscriptRecord[] {
if (opts.head !== undefined && opts.head > 0) return records.slice(0, opts.head);
if (opts.tail !== undefined && opts.tail > 0) return records.slice(-opts.tail);
return records;
}
function renderRecord(rec: TranscriptRecord): void {
const turn = `[t${rec.turn}]`;
if (rec.role === "user") {
console.log(`${turn} USER: ${oneLine(rec.content)}`);
} else if (rec.role === "assistant_final") {
const cost = rec.cost !== undefined ? ` $${rec.cost.toFixed(6)}` : "";
const cache =
rec.usage &&
(rec.usage.prompt_cache_hit_tokens !== undefined ||
rec.usage.prompt_cache_miss_tokens !== undefined)
? (() => {
const hit = rec.usage!.prompt_cache_hit_tokens ?? 0;
const miss = rec.usage!.prompt_cache_miss_tokens ?? 0;
const total = hit + miss;
return total > 0 ? ` cache=${((hit / total) * 100).toFixed(1)}%` : "";
})()
: "";
console.log(`${turn} AGENT:${cost}${cache} ${oneLine(rec.content)}`);
if (rec.planState) {
const ps = rec.planState;
if (ps.subgoals.length)
console.log(` ‹ subgoals (${ps.subgoals.length}): ${ps.subgoals.join(" · ")}`);
if (ps.hypotheses.length)
console.log(` ‹ hypotheses (${ps.hypotheses.length}): ${ps.hypotheses.join(" · ")}`);
if (ps.uncertainties.length)
console.log(
` ‹ uncertainties(${ps.uncertainties.length}): ${ps.uncertainties.join(" · ")}`,
);
if (ps.rejectedPaths.length)
console.log(
` ‹ rejected (${ps.rejectedPaths.length}): ${ps.rejectedPaths.join(" · ")}`,
);
}
} else if (rec.role === "tool") {
const args = rec.args ? ` args=${oneLine(rec.args, 80)}` : "";
console.log(`${turn} TOOL ${rec.tool ?? "?"}:${args} → ${oneLine(rec.content, 120)}`);
} else if (rec.role === "error") {
console.log(`${turn} ERROR: ${rec.error ?? rec.content}`);
} else if (rec.role === "done") {
// Suppress — visually noisy, not informative in replay.
} else {
console.log(`${turn} ${rec.role}: ${oneLine(rec.content)}`);
}
}
function oneLine(s: string, max = 200): string {
const collapsed = s.replace(/\s+/g, " ").trim();
return collapsed.length > max ? `${collapsed.slice(0, max)}…` : collapsed;
}
+172
View File
@@ -0,0 +1,172 @@
import type { WriteStream } from "node:fs";
import { stdin, stdout } from "node:process";
import { createInterface } from "node:readline/promises";
import { defaultConfigPath, isPlausibleKey, loadApiKey, saveApiKey } from "../../config.js";
import { loadDotenv } from "../../env.js";
import { CacheFirstLoop, DeepSeekClient, ImmutablePrefix } from "../../index.js";
import { McpClient } from "../../mcp/client.js";
import { bridgeMcpTools } from "../../mcp/registry.js";
import { parseMcpSpec } from "../../mcp/spec.js";
import { SseTransport } from "../../mcp/sse.js";
import { type McpTransport, StdioTransport } from "../../mcp/stdio.js";
import { ToolRegistry } from "../../tools.js";
import { openTranscriptFile, recordFromLoopEvent, writeRecord } from "../../transcript.js";
export interface RunOptions {
task: string;
model: string;
system: string;
/** Opt into Pillar 2 plan-state harvesting. Adds a cheap V3 call per turn. */
harvest?: boolean;
/** Self-consistency budget. > 1 runs N parallel samples and picks the best. */
branch?: number;
/** JSONL transcript path — lets `reasonix replay` / `diff` audit this run. */
transcript?: string;
/** Zero or more MCP server specs. Each: `"name=cmd args..."` or `"cmd args..."`. */
mcp?: string[];
/** Global prefix — only honored when a single anonymous server is given. */
mcpPrefix?: string;
}
async function ensureApiKey(): Promise<string> {
const existing = loadApiKey();
if (existing) return existing;
if (!stdin.isTTY) {
process.stderr.write(
"DEEPSEEK_API_KEY is not set and stdin is not a TTY (cannot prompt).\n" +
"Set the env var, or run `reasonix chat` once interactively to save a key.\n",
);
process.exit(1);
}
process.stdout.write(
"DeepSeek API key not configured.\nGet one at https://platform.deepseek.com/api_keys\n",
);
const rl = createInterface({ input: stdin, output: stdout });
try {
while (true) {
const answer = (await rl.question("API key › ")).trim();
if (!answer) continue;
if (!isPlausibleKey(answer)) {
process.stdout.write("Invalid format. Keys start with 'sk-' and are 30+ chars.\n");
continue;
}
saveApiKey(answer);
process.stdout.write(`Saved to ${defaultConfigPath()}\n\n`);
return answer;
}
} finally {
rl.close();
}
}
export async function runCommand(opts: RunOptions): Promise<void> {
loadDotenv();
const apiKey = await ensureApiKey();
process.env.DEEPSEEK_API_KEY = apiKey;
// Optional MCP setup — mirrors chat's flow. Must happen before loop
// construction so the tools make it into the prefix.
const requestedSpecs = opts.mcp ?? [];
const clients: McpClient[] = [];
let tools: ToolRegistry | undefined;
let successCount = 0;
if (requestedSpecs.length > 0) {
tools = new ToolRegistry();
for (const raw of requestedSpecs) {
try {
const spec = parseMcpSpec(raw);
const prefix = spec.name
? `${spec.name}_`
: requestedSpecs.length === 1 && opts.mcpPrefix
? opts.mcpPrefix
: "";
const transport: McpTransport =
spec.transport === "sse"
? new SseTransport({ url: spec.url })
: new StdioTransport({ command: spec.command, args: spec.args });
const mcp = new McpClient({ transport });
await mcp.initialize();
const bridge = await bridgeMcpTools(mcp, { registry: tools, namePrefix: prefix });
const source =
spec.transport === "sse" ? spec.url : `${spec.command} ${spec.args.join(" ")}`;
process.stderr.write(
`▸ MCP[${spec.name ?? "anon"}]: ${bridge.registeredNames.length} tool(s) from ${source}\n`,
);
clients.push(mcp);
successCount++;
} catch (err) {
// Non-fatal — skip and continue, same as `reasonix chat`. A
// one-shot `run` invocation with a broken MCP server otherwise
// fails the whole run over a side-concern tool the task might
// not even touch.
process.stderr.write(
`▸ MCP setup SKIPPED for "${raw}": ${(err as Error).message}\n → run \`reasonix setup\` to remove broken entries from your saved config.\n`,
);
}
}
if (successCount === 0) tools = undefined;
}
const client = new DeepSeekClient();
const prefix = new ImmutablePrefix({
system: opts.system,
toolSpecs: tools?.specs(),
});
const loop = new CacheFirstLoop({
client,
prefix,
tools,
model: opts.model,
harvest: opts.harvest,
branch: opts.branch,
});
const prefixHash = prefix.fingerprint;
let transcriptStream: WriteStream | null = null;
if (opts.transcript) {
transcriptStream = openTranscriptFile(opts.transcript, {
version: 1,
source: "reasonix run",
model: opts.model,
startedAt: new Date().toISOString(),
});
// Also persist the user turn itself (the loop's event stream starts with
// assistant output, not the prompt we're about to send).
writeRecord(transcriptStream, {
ts: new Date().toISOString(),
turn: 1,
role: "user",
content: opts.task,
});
}
try {
for await (const ev of loop.step(opts.task)) {
if (ev.role === "assistant_delta" && ev.content) process.stdout.write(ev.content);
if (ev.role === "tool") process.stdout.write(`\n[tool ${ev.toolName}] ${ev.content}\n`);
if (ev.role === "error") process.stderr.write(`\n[error] ${ev.error}\n`);
if (ev.role === "done") process.stdout.write("\n");
// Persist every non-streaming event — deltas would flood the file and
// aren't useful for replay (replay renders final content, not keystrokes).
if (transcriptStream && ev.role !== "assistant_delta") {
writeRecord(transcriptStream, recordFromLoopEvent(ev, { model: opts.model, prefixHash }));
}
}
} finally {
transcriptStream?.end();
}
const s = loop.stats.summary();
process.stdout.write(
`\n— turns:${s.turns} cache:${(s.cacheHitRatio * 100).toFixed(1)}% ` +
`cost:$${s.totalCostUsd.toFixed(6)} save-vs-claude:${s.savingsVsClaudePct.toFixed(1)}%\n`,
);
if (opts.transcript) {
process.stdout.write(`\ntranscript: ${opts.transcript}\n`);
process.stdout.write(` → npx reasonix replay ${opts.transcript}\n`);
}
for (const c of clients) await c.close();
}
+107
View File
@@ -0,0 +1,107 @@
/**
* `reasonix sessions` — CLI equivalent of the `/sessions` slash command.
*
* Two modes:
* - `reasonix sessions` list every session under ~/.reasonix/sessions/
* - `reasonix sessions <name>` dump one session's messages in readable form
*
* Neither needs an API key — pure filesystem reads of JSONL files written
* by previous chat runs.
*/
import { listSessions, loadSessionMessages, sessionPath } from "../../index.js";
import type { ChatMessage } from "../../index.js";
export interface SessionsOptions {
/** When present, inspect that session instead of listing. */
name?: string;
/** Include assistant tool-call metadata in the inspect output. */
verbose?: boolean;
}
export function sessionsCommand(opts: SessionsOptions): void {
if (opts.name) {
inspectSession(opts.name, !!opts.verbose);
} else {
listAll();
}
}
function listAll(): void {
const items = listSessions();
if (items.length === 0) {
console.log(
"no saved sessions yet — run `reasonix chat` (sessions are auto-saved unless --no-session).",
);
return;
}
console.log("Saved sessions (~/.reasonix/sessions/):");
console.log("");
console.log(` ${"name".padEnd(22)} ${"msgs".padStart(6)} ${"size".padStart(8)} modified`);
console.log(` ${"─".repeat(60)}`);
for (const s of items) {
const sizeKb = `${(s.size / 1024).toFixed(1)} KB`;
const when = s.mtime.toISOString().replace("T", " ").slice(0, 16);
console.log(
` ${s.name.padEnd(22)} ${String(s.messageCount).padStart(6)} ${sizeKb.padStart(8)} ${when}`,
);
}
console.log("");
console.log("Inspect: reasonix sessions <name>");
console.log("Resume: reasonix chat --session <name>");
}
function inspectSession(name: string, verbose: boolean): void {
const path = sessionPath(name);
const messages = loadSessionMessages(name);
if (messages.length === 0) {
console.error(`no session named "${name}" (or it's empty).`);
console.error(`looked at: ${path}`);
process.exit(1);
}
console.log(`[session] ${name} ${messages.length} messages ${path}`);
console.log("");
let turnIndex = 0;
for (const msg of messages) {
renderMessage(msg, turnIndex, verbose);
// Roughly bump "turn" after each user message so the reader can follow
// the conversation shape without the transcript's richer turn numbering.
if (msg.role === "user") turnIndex++;
}
}
function renderMessage(msg: ChatMessage, turnIdx: number, verbose: boolean): void {
const turn = turnIdx > 0 ? `[t${turnIdx}]` : "[start]";
const content = typeof msg.content === "string" ? msg.content : "";
const flat = oneLine(content);
if (msg.role === "user") {
console.log(`${turn} USER: ${flat}`);
} else if (msg.role === "assistant") {
console.log(`${turn} AGENT: ${flat || "(tool call only)"}`);
if (verbose && msg.tool_calls?.length) {
for (const tc of msg.tool_calls) {
console.log(
` → call ${tc.function?.name} ${truncate(tc.function?.arguments ?? "", 80)}`,
);
}
}
} else if (msg.role === "tool") {
console.log(`${turn} TOOL ${msg.name ?? "?"}: ${truncate(flat, 160)}`);
} else if (msg.role === "system") {
if (verbose) console.log(`${turn} SYSTEM: ${truncate(flat, 160)}`);
// otherwise suppress — session's system prompt is usually session-wide
// boilerplate.
}
}
function oneLine(s: string, max = 200): string {
const collapsed = s.replace(/\s+/g, " ").trim();
return collapsed.length > max ? `${collapsed.slice(0, max)}…` : collapsed;
}
function truncate(s: string, max: number): string {
return s.length <= max ? s : `${s.slice(0, max)}…`;
}
+45
View File
@@ -0,0 +1,45 @@
/**
* `reasonix setup` — re-mount the first-run wizard on demand so users
* can reconfigure (add/remove MCP servers, switch preset) without
* editing JSON by hand.
*
* Invoked both explicitly (`reasonix setup`) and implicitly (the no-args
* entry point when `setupCompleted` is false).
*/
import { render } from "ink";
import React from "react";
import { loadApiKey, readConfig } from "../../config.js";
import { loadDotenv } from "../../env.js";
import { Wizard } from "../ui/Wizard.js";
export interface SetupOptions {
/**
* When true, bypass the API-key step even if no key is saved — useful
* from test harnesses. Normal CLI use always pushes through the key
* step when missing.
*/
skipKeyStep?: boolean;
}
export async function setupCommand(_opts: SetupOptions = {}): Promise<void> {
loadDotenv();
const existingKey = loadApiKey();
const existing = readConfig();
const { waitUntilExit, unmount } = render(
<Wizard
existingApiKey={existingKey}
initial={{ preset: existing.preset, mcp: existing.mcp }}
onComplete={() => {
// Ink handles its own enter-to-exit inside the "saved" step; we
// just wait for the app to exit naturally.
}}
onCancel={() => {
unmount();
}}
/>,
{ exitOnCtrlC: true },
);
await waitUntilExit();
}
+30
View File
@@ -0,0 +1,30 @@
import { existsSync, readFileSync } from "node:fs";
export interface StatsOptions {
transcript: string;
}
export function statsCommand(opts: StatsOptions): void {
if (!existsSync(opts.transcript)) {
console.error(`no such transcript: ${opts.transcript}`);
process.exit(1);
}
const lines = readFileSync(opts.transcript, "utf8").split(/\r?\n/).filter(Boolean);
let assistantTurns = 0;
let toolCalls = 0;
let lastTurn = 0;
for (const line of lines) {
try {
const rec = JSON.parse(line);
if (rec.role === "assistant_final") assistantTurns++;
if (rec.role === "tool") toolCalls++;
if (typeof rec.turn === "number") lastTurn = Math.max(lastTurn, rec.turn);
} catch {
/* skip */
}
}
console.log(`transcript: ${opts.transcript}`);
console.log(`assistant turns: ${assistantTurns}`);
console.log(`tool invocations: ${toolCalls}`);
console.log(`last turn index: ${lastTurn}`);
}
+5
View File
@@ -0,0 +1,5 @@
import { VERSION } from "../../index.js";
export function versionCommand(): void {
console.log(`reasonix ${VERSION}`);
}
+241
View File
@@ -0,0 +1,241 @@
import { Command } from "commander";
import { readConfig } from "../config.js";
import { VERSION } from "../index.js";
import { chatCommand } from "./commands/chat.js";
import { codeCommand } from "./commands/code.js";
import { diffCommand } from "./commands/diff.js";
import { mcpListCommand } from "./commands/mcp.js";
import { replayCommand } from "./commands/replay.js";
import { runCommand } from "./commands/run.js";
import { sessionsCommand } from "./commands/sessions.js";
import { setupCommand } from "./commands/setup.js";
import { statsCommand } from "./commands/stats.js";
import { versionCommand } from "./commands/version.js";
import { resolveDefaults } from "./resolve.js";
const DEFAULT_SYSTEM =
"You are Reasonix, a helpful DeepSeek-powered assistant. Be concise and accurate. Use tools when available.";
const program = new Command();
program
.name("reasonix")
.description("DeepSeek-native agent framework — built for cache hits and cheap tokens.")
.version(VERSION);
// `reasonix` with no subcommand → launch the friendliest flow.
// First run (no config yet) → interactive setup wizard.
// Otherwise → chat with saved defaults. This is the "one command to
// rule them all" entry for non-power-users: they don't need to learn
// `chat` / `setup` / `--mcp` — just type `reasonix`.
program.action(async () => {
const cfg = readConfig();
if (!cfg.setupCompleted) {
await setupCommand({});
return;
}
const defaults = resolveDefaults({});
await chatCommand({
model: defaults.model,
system: DEFAULT_SYSTEM,
harvest: defaults.harvest,
branch: defaults.branch,
session: defaults.session,
mcp: defaults.mcp,
});
});
program
.command("setup")
.description("Interactive wizard — API key, preset, MCP servers. Re-run any time to reconfigure.")
.action(async () => {
await setupCommand({});
});
program
.command("code [dir]")
.description(
"Code-editing chat — filesystem MCP auto-bridged at <dir> (default: cwd), coding system prompt, smart preset. Model proposes SEARCH/REPLACE blocks; Reasonix applies them to disk.",
)
.option("-m, --model <id>", "Override default reasoner model")
.option("--no-session", "Disable session persistence for this run")
.option("--transcript <path>", "Write a JSONL transcript to this path")
.action(async (dir: string | undefined, opts) => {
await codeCommand({
dir,
model: opts.model,
noSession: opts.session === false,
transcript: opts.transcript,
});
});
program
.command("chat")
.description("Interactive Ink TUI with live cache/cost panel.")
.option("-m, --model <id>", "DeepSeek model id (overrides preset)")
.option("-s, --system <prompt>", "System prompt (pinned in the immutable prefix)", DEFAULT_SYSTEM)
.option("--transcript <path>", "Write a JSONL transcript to this path")
.option(
"--preset <name>",
"Bundle of model + harvest + branch. One of: fast, smart, max. Overrides config.preset.",
)
.option(
"--harvest",
"Extract typed plan state from R1 reasoning (Pillar 2). Overrides preset's harvest setting.",
)
.option(
"--branch <n>",
"Self-consistency: run N parallel samples per turn and pick the most confident (disables streaming; enables harvest)",
(v) => Number.parseInt(v, 10),
)
.option("--session <name>", "Use a named session (default: from config, usually 'default').")
.option("--no-session", "Disable session persistence for this run (ephemeral chat)")
.option(
"--mcp <spec>",
'MCP server spec; repeatable. "name=cmd args...", "cmd args...", or a URL (http/https → SSE transport). Overrides config.mcp when provided.',
(value: string, previous: string[] = []) => [...previous, value],
[] as string[],
)
.option(
"--mcp-prefix <str>",
"Global prefix applied to every MCP tool (only honored when no per-spec name is set; avoids collisions with a single anonymous server)",
)
.option("--no-config", "Ignore `~/.reasonix/config.json` — useful for CI or reproducing issues")
.action(async (opts) => {
const defaults = resolveDefaults({
model: opts.model,
harvest: opts.harvest,
branch: opts.branch,
mcp: opts.mcp as string[],
session: opts.session,
preset: opts.preset,
noConfig: opts.config === false,
});
await chatCommand({
model: defaults.model,
system: opts.system,
transcript: opts.transcript,
harvest: defaults.harvest,
branch: defaults.branch,
session: defaults.session,
mcp: defaults.mcp,
mcpPrefix: opts.mcpPrefix,
});
});
program
.command("run <task>")
.description("Run a single task non-interactively, streaming output.")
.option("-m, --model <id>", "DeepSeek model id (overrides preset)")
.option("-s, --system <prompt>", "System prompt", DEFAULT_SYSTEM)
.option("--preset <name>", "Bundle of model + harvest + branch: fast | smart | max")
.option("--harvest", "Extract typed plan state from R1 reasoning (Pillar 2)")
.option(
"--branch <n>",
"Self-consistency: run N parallel samples per turn and pick the most confident",
(v) => Number.parseInt(v, 10),
)
.option("--transcript <path>", "Write a JSONL transcript to this path for replay/diff")
.option(
"--mcp <spec>",
'MCP server spec; repeatable. "name=cmd args...", "cmd args...", or a URL (http/https → SSE).',
(value: string, previous: string[] = []) => [...previous, value],
[] as string[],
)
.option(
"--mcp-prefix <str>",
"Global prefix (only honored when no per-spec name is set; for a single anonymous server)",
)
.option("--no-config", "Ignore `~/.reasonix/config.json` — useful for CI or reproducing issues")
.action(async (task: string, opts) => {
const defaults = resolveDefaults({
model: opts.model,
harvest: opts.harvest,
branch: opts.branch,
mcp: opts.mcp as string[],
preset: opts.preset,
noConfig: opts.config === false,
});
await runCommand({
task,
model: defaults.model,
system: opts.system,
harvest: defaults.harvest,
branch: defaults.branch,
transcript: opts.transcript,
mcp: defaults.mcp,
mcpPrefix: opts.mcpPrefix,
});
});
program
.command("stats <transcript>")
.description("Summarize a JSONL transcript produced by `reasonix chat --transcript`.")
.action((transcript: string) => {
statsCommand({ transcript });
});
program
.command("sessions [name]")
.description("List saved chat sessions, or inspect one by name.")
.option("-v, --verbose", "Include system prompts + tool-call metadata when inspecting")
.action((name: string | undefined, opts) => {
sessionsCommand({ name, verbose: !!opts.verbose });
});
program
.command("replay <transcript>")
.description(
"Interactive Ink TUI to scrub through a transcript + rebuild its session summary (cost, cache, prefix stability). No API calls.",
)
.option("--print", "Dump to stdout instead of mounting the TUI (auto when piped)")
.option("--head <n>", "stdout mode only — show first N records", (v) => Number.parseInt(v, 10))
.option("--tail <n>", "stdout mode only — show last N records", (v) => Number.parseInt(v, 10))
.action(async (transcript: string, opts) => {
await replayCommand({
path: transcript,
print: !!opts.print,
head: Number.isFinite(opts.head) ? opts.head : undefined,
tail: Number.isFinite(opts.tail) ? opts.tail : undefined,
});
});
program
.command("diff <a> <b>")
.description(
"Compare two transcripts in a split-pane Ink TUI (default) or stdout table. Use n/N to jump across divergences.",
)
.option("--md <path>", "Write a markdown report (blog-ready) to this path")
.option("--print", "Force stdout table instead of the TUI (auto when piped)")
.option("--tui", "Force the TUI even when piped (rare)")
.option("--label-a <label>", "Display label for transcript A (default: filename)")
.option("--label-b <label>", "Display label for transcript B (default: filename)")
.action(async (a: string, b: string, opts) => {
await diffCommand({
a,
b,
mdPath: opts.md,
labelA: opts.labelA,
labelB: opts.labelB,
print: !!opts.print,
tui: !!opts.tui,
});
});
const mcp = program
.command("mcp")
.description("Model Context Protocol helpers — discover servers, test your setup.");
mcp
.command("list")
.description("Show a curated catalog of popular MCP servers with ready-to-use --mcp commands.")
.option("--json", "Emit the catalog as JSON instead of the human-readable table")
.action((opts) => {
mcpListCommand({ json: !!opts.json });
});
program.command("version").description("Print Reasonix version.").action(versionCommand);
program.parseAsync(process.argv).catch((err) => {
console.error(err);
process.exit(1);
});
+89
View File
@@ -0,0 +1,89 @@
/**
* Merge config defaults with CLI flags into the concrete options that
* `chatCommand` / `runCommand` need.
*
* Precedence (highest wins):
* 1. Explicit per-setting CLI flag (`--model`, `--harvest`, `--branch`, `--mcp`)
* 2. Explicit `--preset <name>` CLI flag
* 3. `config.preset` from `~/.reasonix/config.json` (set by `reasonix setup`)
* 4. Hardcoded "fast" preset defaults
*
* Keeping this logic in one place — rather than duplicating across
* `chat` and `run` — means the precedence rule only lives in one unit
* test and the shape of the merge is identical for both commands.
*/
import { type PresetName, type ReasonixConfig, readConfig } from "../config.js";
import { PRESETS } from "./ui/presets.js";
export interface ResolvedDefaults {
model: string;
harvest: boolean;
branch: number | undefined;
mcp: string[];
session: string | undefined;
}
export interface RawCliFlags {
model?: string;
harvest?: boolean;
/** From `commander`; already parseInt'd. */
branch?: number;
mcp?: string[];
/** Commander's `--no-session` surfaces as `false`; `--session X` as a string. */
session?: string | false;
/** `--preset <name>`. */
preset?: string;
/** When true, ignore config entirely (power-user escape hatch). */
noConfig?: boolean;
}
export function resolveDefaults(flags: RawCliFlags): ResolvedDefaults {
const cfg: ReasonixConfig = flags.noConfig ? {} : readConfig();
const preset = pickPreset(flags.preset, cfg.preset);
const presetSettings = PRESETS[preset];
const model = flags.model ?? presetSettings.model;
const harvest = flags.harvest === true ? true : presetSettings.harvest;
const branchFromFlag = normalizeBranch(flags.branch);
const branch = branchFromFlag ?? (presetSettings.branch > 1 ? presetSettings.branch : undefined);
// `--mcp` accumulator is [] when absent. Treat empty from flags as
// "user didn't pass" → fall through to config. Users who explicitly
// want zero MCP servers can pass `--no-config` or edit the file.
const mcp = flags.mcp && flags.mcp.length > 0 ? flags.mcp : (cfg.mcp ?? []);
const session = resolveSession(flags.session, cfg.session);
return { model, harvest, branch, mcp, session };
}
function pickPreset(
flagPreset: string | undefined,
configPreset: PresetName | undefined,
): PresetName {
if (flagPreset && isPresetName(flagPreset)) return flagPreset;
if (configPreset) return configPreset;
return "fast";
}
function isPresetName(s: string): s is PresetName {
return s === "fast" || s === "smart" || s === "max";
}
function normalizeBranch(raw: number | undefined): number | undefined {
if (raw === undefined) return undefined;
if (!Number.isFinite(raw) || raw <= 1) return undefined;
return Math.min(raw, 8);
}
function resolveSession(
flag: string | false | undefined,
configSession: string | null | undefined,
): string | undefined {
if (flag === false) return undefined; // --no-session
if (typeof flag === "string" && flag.length > 0) return flag;
if (configSession === null) return undefined; // config opted out
if (typeof configSession === "string" && configSession.length > 0) return configSession;
return "default";
}
+679
View File
@@ -0,0 +1,679 @@
import type { WriteStream } from "node:fs";
import { Box, Static, Text, useApp, useInput } from "ink";
import React, { useCallback, useEffect, useMemo, useRef, useState } from "react";
import {
type ApplyResult,
type EditBlock,
type EditSnapshot,
applyEditBlocks,
parseEditBlocks,
restoreSnapshots,
snapshotBeforeEdits,
} from "../../code/edit-blocks.js";
import { CacheFirstLoop, DeepSeekClient, ImmutablePrefix } from "../../index.js";
import type { LoopEvent } from "../../loop.js";
import type { SessionSummary } from "../../telemetry.js";
import type { ToolRegistry } from "../../tools.js";
import { openTranscriptFile, recordFromLoopEvent, writeRecord } from "../../transcript.js";
import { type DisplayEvent, EventRow } from "./EventLog.js";
import { PromptInput } from "./PromptInput.js";
import { StatsPanel } from "./StatsPanel.js";
import { handleSlash, parseSlash } from "./slash.js";
export interface AppProps {
model: string;
system: string;
transcript?: string;
harvest?: boolean;
branch?: number;
session?: string;
/**
* Pre-populated tool registry (e.g. from bridgeMcpTools()). When present,
* its specs are folded into the ImmutablePrefix so the model sees them,
* and its dispatch is used for tool calls — MCP tools become first-class.
*/
tools?: ToolRegistry;
/** Raw `--mcp` / config-derived spec strings, for `/mcp` slash display. */
mcpSpecs?: string[];
/**
* When set, parse SEARCH/REPLACE blocks from assistant responses and
* apply them to disk under `rootDir`. Set by `reasonix code`.
*/
codeMode?: { rootDir: string };
}
/**
* Throttle interval in ms. We flush streaming deltas at most this often to
* avoid re-rendering the whole UI on every single token from DeepSeek.
* 60ms ≈ 16Hz, fast enough to feel live, slow enough to not thrash Ink.
*/
const FLUSH_INTERVAL_MS = 60;
interface StreamingState {
id: string;
text: string;
reasoning: string;
}
export function App({
model,
system,
transcript,
harvest,
branch,
session,
tools,
mcpSpecs,
codeMode,
}: AppProps) {
const { exit } = useApp();
const [historical, setHistorical] = useState<DisplayEvent[]>([]);
const [streaming, setStreaming] = useState<DisplayEvent | null>(null);
const [input, setInput] = useState("");
const [busy, setBusy] = useState(false);
// Tracks whether the current turn has been aborted via Esc, so the
// Esc handler only fires once per turn (repeated presses would yield
// stacked warning events).
const abortedThisTurn = useRef(false);
// Name + truncated args of the tool currently dispatching. Populated
// on `tool_start`, cleared on `tool` (or error). Drives the
// "▸ tool<X> running…" pulse-spinner row so long tool calls don't
// look like the app hung.
const [ongoingTool, setOngoingTool] = useState<{ name: string; args?: string } | null>(null);
// Snapshots of every file the *last* edit batch touched, keyed by
// nothing more than "most recent". `/undo` restores from this ref
// and nulls it out — one level of undo, Aider-style. Multi-step
// undo would need a proper history stack and a clear policy for
// when the stack clears; v1 keeps it simple.
const lastEditSnapshots = useRef<EditSnapshot[] | null>(null);
// Pending edit blocks awaiting `/apply` or `/discard`. We do NOT
// auto-apply — v0.4.1 showed that "model proposed, so apply" turns
// analysis into unintended edits. The user explicitly confirms now.
const pendingEdits = useRef<EditBlock[]>([]);
// Shell-style history of user prompts. ↑/↓ while idle walks it;
// submit pushes to the end. Cursor -1 = "live input", 0+ = "N turns
// back from newest". We don't persist history to disk — sessions
// already keep the message log, and cross-session bash-style recall
// would need per-project scoping we haven't designed.
const promptHistory = useRef<string[]>([]);
const historyCursor = useRef<number>(-1);
// Full untruncated tool results, in arrival order. The EventLog
// renderer clips tool output at 400 chars for display; `/tool N`
// reads from this ref to show the real thing. Not persisted — a
// resumed session replays the log (which has the same content in
// `tool` messages) but we don't repopulate this ref on resume
// because the user wouldn't expect `/tool` to reach back across
// process boundaries.
const toolHistoryRef = useRef<Array<{ toolName: string; text: string }>>([]);
const [summary, setSummary] = useState<SessionSummary>({
turns: 0,
totalCostUsd: 0,
claudeEquivalentUsd: 0,
savingsVsClaudePct: 0,
cacheHitRatio: 0,
lastPromptTokens: 0,
});
const transcriptRef = useRef<WriteStream | null>(null);
if (transcript && !transcriptRef.current) {
transcriptRef.current = openTranscriptFile(transcript, {
version: 1,
source: "reasonix chat",
model,
startedAt: new Date().toISOString(),
});
}
useEffect(() => {
return () => {
transcriptRef.current?.end();
};
}, []);
const loopRef = useRef<CacheFirstLoop | null>(null);
const loop = useMemo(() => {
if (loopRef.current) return loopRef.current;
const client = new DeepSeekClient();
const prefix = new ImmutablePrefix({
system,
toolSpecs: tools?.specs(),
});
const l = new CacheFirstLoop({ client, prefix, tools, model, harvest, branch, session });
loopRef.current = l;
return l;
}, [model, system, harvest, branch, session, tools]);
// Surface a one-time banner about session state on first mount.
const sessionBannerShown = useRef(false);
useEffect(() => {
if (sessionBannerShown.current) return;
sessionBannerShown.current = true;
if (!session) {
setHistorical((prev) => [
...prev,
{
id: `sys-session-${Date.now()}`,
role: "info",
text: "▸ ephemeral chat (no session persistence) — drop --no-session to enable",
},
]);
} else if (loop.resumedMessageCount > 0) {
setHistorical((prev) => [
...prev,
{
id: `sys-resume-${Date.now()}`,
role: "info",
text: `▸ resumed session "${session}" with ${loop.resumedMessageCount} prior messages · /forget to start over · /sessions to list`,
},
]);
} else {
setHistorical((prev) => [
...prev,
{
id: `sys-newsession-${Date.now()}`,
role: "info",
text: `▸ session "${session}" (new) — auto-saved as you chat · /forget to delete · /sessions to list`,
},
]);
}
}, [session, loop]);
// Esc during busy → forward to the loop as an abort signal. The loop
// finishes the tool call in flight (we can't kill subprocess stdio
// mid-write), then diverts to its no-tools summary path so the user
// gets an answer instead of a hard stop. Only listens while busy so
// we don't accidentally hijack Esc in other contexts.
//
// Also handles ↑/↓ shell-style history while idle. We don't use
// ink-text-input's (absent) history support; parent-level useInput
// is simpler and lets us own the cursor semantics.
useInput((_input, key) => {
if (key.escape && busy) {
if (abortedThisTurn.current) return;
abortedThisTurn.current = true;
loop.abort();
return;
}
if (busy) return;
const hist = promptHistory.current;
if (key.upArrow) {
if (hist.length === 0) return;
const nextCursor = Math.min(historyCursor.current + 1, hist.length - 1);
historyCursor.current = nextCursor;
setInput(hist[hist.length - 1 - nextCursor] ?? "");
return;
}
if (key.downArrow) {
if (historyCursor.current < 0) return;
const nextCursor = historyCursor.current - 1;
historyCursor.current = nextCursor;
setInput(nextCursor < 0 ? "" : (hist[hist.length - 1 - nextCursor] ?? ""));
return;
}
});
/**
* Callback wired into the `/undo` slash command. Restores the files
* last edit batch to their pre-edit state and reports per-file
* results. Only available when running in code mode — the slash
* handler gates on this callback's presence.
*/
const codeUndo = useCallback((): string => {
if (!codeMode) return "not in code mode";
const snaps = lastEditSnapshots.current;
if (!snaps || snaps.length === 0) {
return "nothing to undo — no recent edit batch to restore";
}
const results = restoreSnapshots(snaps, codeMode.rootDir);
lastEditSnapshots.current = null;
return formatUndoResults(results);
}, [codeMode]);
/**
* /apply callback — write pending edit blocks to disk, snapshot
* beforehand so /undo still works, report per-file results.
*/
const codeApply = useCallback((): string => {
if (!codeMode) return "not in code mode";
const blocks = pendingEdits.current;
if (blocks.length === 0) {
return "nothing pending — the assistant hasn't proposed edits since the last /apply or /discard.";
}
const snaps = snapshotBeforeEdits(blocks, codeMode.rootDir);
const results = applyEditBlocks(blocks, codeMode.rootDir);
const anyApplied = results.some((r) => r.status === "applied" || r.status === "created");
if (anyApplied) lastEditSnapshots.current = snaps;
pendingEdits.current = [];
return formatEditResults(results);
}, [codeMode]);
/**
* /discard callback — forget the pending edits without touching
* disk. Keeps the conversation going without the user having to
* argue the model out of its proposal.
*/
const codeDiscard = useCallback((): string => {
const count = pendingEdits.current.length;
if (count === 0) return "nothing pending to discard.";
pendingEdits.current = [];
return `▸ discarded ${count} pending edit block(s). Nothing was written to disk.`;
}, []);
const prefixHash = loop.prefix.fingerprint;
const writeTranscript = useCallback(
(ev: LoopEvent) => {
const stream = transcriptRef.current;
if (!stream) return;
writeRecord(stream, recordFromLoopEvent(ev, { model, prefixHash }));
},
[model, prefixHash],
);
const handleSubmit = useCallback(
async (raw: string) => {
let text = raw.trim();
if (!text || busy) return;
setInput("");
historyCursor.current = -1;
// Y/N fast-path when edits are pending. One keystroke is all it
// takes to commit or drop — matches the muscle memory of `git
// add -p` / most prompts. Deliberately scoped: only when there
// ARE pending edits, so "y" as a normal message still works
// when nothing's waiting.
if (codeMode && pendingEdits.current.length > 0 && (text === "y" || text === "n")) {
const out = text === "y" ? codeApply() : codeDiscard();
setHistorical((prev) => [...prev, { id: `sys-${Date.now()}`, role: "info", text: out }]);
promptHistory.current.push(text);
return;
}
const slash = parseSlash(text);
if (slash) {
const result = handleSlash(slash.cmd, slash.args, loop, {
mcpSpecs,
codeUndo: codeMode ? codeUndo : undefined,
codeApply: codeMode ? codeApply : undefined,
codeDiscard: codeMode ? codeDiscard : undefined,
codeRoot: codeMode?.rootDir,
pendingEditCount: codeMode ? pendingEdits.current.length : undefined,
toolHistory: () => toolHistoryRef.current,
});
if (result.exit) {
transcriptRef.current?.end();
exit();
return;
}
if (result.clear) {
setHistorical([]);
return;
}
if (result.info) {
setHistorical((prev) => [
...prev,
{
id: `sys-${Date.now()}`,
role: "info",
text: result.info!,
},
]);
}
// `/retry` (and anything else that requests a resubmit) falls
// through to the normal user-message flow with the provided
// text instead of returning.
if (result.resubmit) {
text = result.resubmit;
} else {
promptHistory.current.push(text);
return;
}
}
// User message is immutable — push to Static immediately.
promptHistory.current.push(text);
setHistorical((prev) => [...prev, { id: `u-${Date.now()}`, role: "user", text }]);
const assistantId = `a-${Date.now()}`;
// Refs are the source of truth for accumulated streaming text; the React
// state copy below is only for rendering and gets updated on flush.
const streamRef: StreamingState = { id: assistantId, text: "", reasoning: "" };
const contentBuf = { current: "" };
const reasoningBuf = { current: "" };
setStreaming({ id: assistantId, role: "assistant", text: "", streaming: true });
setBusy(true);
abortedThisTurn.current = false;
const flush = () => {
if (!contentBuf.current && !reasoningBuf.current) return;
streamRef.text += contentBuf.current;
streamRef.reasoning += reasoningBuf.current;
contentBuf.current = "";
reasoningBuf.current = "";
setStreaming({
id: assistantId,
role: "assistant",
text: streamRef.text,
reasoning: streamRef.reasoning || undefined,
streaming: true,
});
};
const timer = setInterval(flush, FLUSH_INTERVAL_MS);
try {
for await (const ev of loop.step(text)) {
writeTranscript(ev);
if (ev.role === "assistant_delta") {
if (ev.content) contentBuf.current += ev.content;
if (ev.reasoningDelta) reasoningBuf.current += ev.reasoningDelta;
} else if (ev.role === "branch_start") {
setStreaming({
id: assistantId,
role: "assistant",
text: "",
streaming: true,
branchProgress: ev.branchProgress,
});
} else if (ev.role === "branch_progress") {
// Live-update the streaming slot with per-sample completion info.
setStreaming({
id: assistantId,
role: "assistant",
text: "",
streaming: true,
branchProgress: ev.branchProgress,
});
} else if (ev.role === "branch_done") {
// Intermediate: branching finished but assistant_final not yet emitted.
// Keep streaming state alive; actual render happens on assistant_final.
} else if (ev.role === "assistant_final") {
flush();
const repairNote = ev.repair ? describeRepair(ev.repair) : "";
setStreaming(null);
const finalText = ev.content || streamRef.text;
setHistorical((prev) => [
...prev,
{
id: assistantId,
role: "assistant",
text: finalText,
reasoning: streamRef.reasoning || undefined,
planState: ev.planState,
branch: ev.branch,
stats: ev.stats,
repair: repairNote || undefined,
streaming: false,
},
]);
if (codeMode && finalText && !ev.forcedSummary) {
// Parse SEARCH/REPLACE blocks but DO NOT write them to
// disk. Store as pending — the user has to say /apply
// explicitly. This prevents "analyze the project" from
// silently drifting into "edit the project".
//
// `ev.forcedSummary` gates us out entirely: if the loop
// had to force a summary (budget / aborted / context-
// guard), its text is a wrap-up, not a plan to execute.
// Blocks dropped in a forced summary are display-only.
const blocks = parseEditBlocks(finalText);
if (blocks.length > 0) {
pendingEdits.current = blocks;
setHistorical((prev) => [
...prev,
{
id: `pending-${Date.now()}`,
role: "info",
text: formatPendingPreview(blocks),
},
]);
}
}
} else if (ev.role === "tool_start") {
// Kick off the visual indicator. Cleared when `tool`
// (result) or `error` arrives, or on the finally below.
setOngoingTool({ name: ev.toolName ?? "?", args: ev.toolArgs });
} else if (ev.role === "tool") {
flush();
setOngoingTool(null);
toolHistoryRef.current.push({
toolName: ev.toolName ?? "?",
text: ev.content,
});
setHistorical((prev) => [
...prev,
{
id: `t-${Date.now()}-${Math.random()}`,
role: "tool",
text: ev.content,
toolName: ev.toolName,
},
]);
} else if (ev.role === "error") {
setHistorical((prev) => [
...prev,
{ id: `e-${Date.now()}`, role: "error", text: ev.error ?? ev.content },
]);
} else if (ev.role === "warning") {
setHistorical((prev) => [
...prev,
{ id: `w-${Date.now()}-${Math.random()}`, role: "warning", text: ev.content },
]);
}
}
flush();
} finally {
clearInterval(timer);
setStreaming(null);
setOngoingTool(null);
setSummary(loop.stats.summary());
setBusy(false);
}
},
[busy, codeApply, codeDiscard, codeMode, codeUndo, exit, loop, mcpSpecs, writeTranscript],
);
return (
<Box flexDirection="column">
<StatsPanel
summary={summary}
model={loop.model}
prefixHash={prefixHash}
harvestOn={loop.harvestEnabled}
branchBudget={loop.branchOptions.budget}
/>
<Static items={historical}>{(item) => <EventRow key={item.id} event={item} />}</Static>
{streaming ? (
<Box marginY={1}>
<EventRow event={streaming} />
</Box>
) : null}
{ongoingTool ? <OngoingToolRow tool={ongoingTool} /> : null}
<PromptInput value={input} onChange={setInput} onSubmit={handleSubmit} disabled={busy} />
<CommandStrip codeMode={!!codeMode} />
</Box>
);
}
/**
* Live spinner row while a tool call is in flight. Without this the
* window between the model's `tool_calls` decision and the tool's
* result (often seconds for a multi-KB `filesystem_edit_file`) looks
* like the app has frozen — the streaming assistant display is
* already cleared and the input is disabled, so there's nothing to
* look at.
*
* We show three signals: a braille spinner (liveness), an elapsed
* timer in seconds (so "long" has a number attached), and a
* per-tool summary of the most informative argument fields (path,
* edits count, pattern, etc.). MCP doesn't stream progress today —
* when it does, this component is where the progress notifications
* would land.
*/
function OngoingToolRow({ tool }: { tool: { name: string; args?: string } }) {
const [tick, setTick] = useState(0);
const [elapsed, setElapsed] = useState(0);
useEffect(() => {
const start = Date.now();
const frameId = setInterval(() => setTick((t) => t + 1), 120);
const secId = setInterval(() => setElapsed(Math.floor((Date.now() - start) / 1000)), 1000);
return () => {
clearInterval(frameId);
clearInterval(secId);
};
}, []);
const frames = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"];
const summary = summarizeToolArgs(tool.name, tool.args);
return (
<Box marginY={1} flexDirection="column">
<Box>
<Text color="cyan">{frames[tick % frames.length]}</Text>
<Text color="yellow">{` tool<${tool.name}> running…`}</Text>
<Text dimColor>{` ${elapsed}s`}</Text>
</Box>
{summary ? (
<Box paddingLeft={2}>
<Text dimColor>{summary}</Text>
</Box>
) : null}
</Box>
);
}
/**
* Turn raw JSON tool arguments into a one-line human summary. For
* common filesystem MCP tools we pull the actually-useful fields; for
* anything else we fall back to a truncated raw string so the user
* still sees *something* beyond the tool name.
*
* Match on suffix (e.g. `_read_file`) rather than exact name because
* `bridgeMcpTools({ namePrefix: "filesystem_" })` prepends the server
* namespace — tools arrive as `filesystem_read_file` in practice but
* callers might wire up anonymous too.
*/
function summarizeToolArgs(name: string, args?: string): string {
if (!args || args === "{}") return "";
let parsed: Record<string, unknown>;
try {
parsed = JSON.parse(args) as Record<string, unknown>;
} catch {
// Unparseable JSON — show a head slice so user at least sees
// what the model tried.
return args.length > 80 ? `${args.slice(0, 80)}…` : args;
}
const hasSuffix = (s: string) => name === s || name.endsWith(`_${s}`);
const path = typeof parsed.path === "string" ? parsed.path : undefined;
if (hasSuffix("read_file")) {
const head = typeof parsed.head === "number" ? `, head=${parsed.head}` : "";
const tail = typeof parsed.tail === "number" ? `, tail=${parsed.tail}` : "";
return `path: ${path ?? "?"}${head}${tail}`;
}
if (hasSuffix("write_file")) {
const content = typeof parsed.content === "string" ? parsed.content : "";
return `path: ${path ?? "?"} (${content.length} chars)`;
}
if (hasSuffix("edit_file")) {
const edits = Array.isArray(parsed.edits) ? parsed.edits.length : 0;
return `path: ${path ?? "?"} (${edits} edit${edits === 1 ? "" : "s"})`;
}
if (hasSuffix("list_directory") || hasSuffix("directory_tree")) {
return `path: ${path ?? "?"}`;
}
if (hasSuffix("search_files")) {
const pattern = typeof parsed.pattern === "string" ? parsed.pattern : "?";
return `path: ${path ?? "?"} · pattern: ${pattern}`;
}
if (hasSuffix("move_file")) {
const src = typeof parsed.source === "string" ? parsed.source : "?";
const dst = typeof parsed.destination === "string" ? parsed.destination : "?";
return `${src} → ${dst}`;
}
if (hasSuffix("get_file_info")) {
return `path: ${path ?? "?"}`;
}
return args.length > 80 ? `${args.slice(0, 80)}…` : args;
}
function CommandStrip({ codeMode }: { codeMode: boolean }) {
return (
<Box paddingX={2} flexDirection="column">
<Text dimColor>
/help · /preset {"<fast|smart|max>"} · /mcp · /compact · /sessions · /setup · /clear · /exit
</Text>
{codeMode ? (
<Text dimColor>
code mode: /apply (y) · /discard (n) · /undo · /commit "msg" — edits NEVER write without
/apply
</Text>
) : null}
<Text dimColor>
↑/↓ recall prompts · /retry re-send last · /think see R1 reasoning · /tool N full tool
output
</Text>
<Text dimColor>Esc (while thinking) — abort & summarize what was found so far</Text>
</Box>
);
}
/**
* Render a batch of SEARCH/REPLACE application results as one
* human-scannable info line per edit. Prefixes denote status so the
* line reads well even without color (e.g. when piped to a log file
* or stripped for screenshots):
* ✓ applied src/foo.ts
* ✓ created src/new.ts
* ✗ not-found src/bar.ts (SEARCH text does not match…)
*/
function formatEditResults(results: ApplyResult[]): string {
const lines = results.map((r) => {
const mark = r.status === "applied" || r.status === "created" ? "✓" : "✗";
const detail = r.message ? ` (${r.message})` : "";
return ` ${mark} ${r.status.padEnd(11)} ${r.path}${detail}`;
});
const ok = results.filter((r) => r.status === "applied" || r.status === "created").length;
const total = results.length;
const header = `▸ edit blocks: ${ok}/${total} applied — /undo to roll back, or \`git diff\` to review`;
return [header, ...lines].join("\n");
}
/**
* Pending-edits preview shown after each assistant turn that proposed
* changes. We keep it deliberately thin — path + approximate
* ±line-count — because a full diff would crowd the TUI and the user
* can always run `git diff` after /apply.
*/
function formatPendingPreview(blocks: EditBlock[]): string {
const lines = blocks.map((b) => {
const removed = b.search === "" ? 0 : countLines(b.search);
const added = countLines(b.replace);
const tag = b.search === "" ? "NEW " : " ";
return ` ${tag}${b.path} (-${removed} +${added} lines)`;
});
const header = `▸ ${blocks.length} pending edit block(s) — /apply (or y) to commit · /discard (or n) to drop`;
return [header, ...lines].join("\n");
}
function countLines(s: string): number {
if (s.length === 0) return 0;
return (s.match(/\n/g)?.length ?? 0) + 1;
}
function formatUndoResults(results: ApplyResult[]): string {
const lines = results.map((r) => {
const mark = r.status === "applied" ? "✓" : "✗";
const detail = r.message ? ` (${r.message})` : "";
return ` ${mark} ${r.path}${detail}`;
});
return [`▸ undo: restored ${results.length} file(s) to pre-edit state`, ...lines].join("\n");
}
function describeRepair(repair: {
scavenged: number;
truncationsFixed: number;
stormsBroken: number;
}): string {
const parts: string[] = [];
if (repair.scavenged) parts.push(`scavenged ${repair.scavenged}`);
if (repair.truncationsFixed) parts.push(`repaired ${repair.truncationsFixed} truncation`);
if (repair.stormsBroken) parts.push(`broke ${repair.stormsBroken} storm`);
return parts.length ? `[repair] ${parts.join(", ")}` : "";
}
+234
View File
@@ -0,0 +1,234 @@
/**
* Ink TUI for `reasonix diff`. Split-pane: A on the left, B on the right,
* shared cursor. Header shows aggregate deltas; footer shows the current
* pair's divergence note (if any) + key cheat sheet.
*
* j/k moves the cursor by one turn; n/N jumps to the next/prev divergent
* turn — which is the whole point of a diff tool. Quit with q.
*
* Pure navigation lives in src/diff.ts (findNextDivergence / findPrevDivergence).
*/
import { Box, Static, Text, useApp, useInput } from "ink";
import React, { useState } from "react";
import {
type DiffReport,
type TurnPair,
findNextDivergence,
findPrevDivergence,
} from "../../diff.js";
import { RecordView } from "./RecordView.js";
export interface DiffAppProps {
report: DiffReport;
}
export function DiffApp({ report }: DiffAppProps) {
const { exit } = useApp();
const maxIdx = Math.max(0, report.pairs.length - 1);
// Start at the first divergence when one exists — that's the user's most
// likely destination. Falls back to idx 0 for fully-matching diffs.
const initialIdx = report.firstDivergenceTurn
? report.pairs.findIndex((p) => p.turn === report.firstDivergenceTurn)
: 0;
const [idx, setIdx] = useState(Math.max(0, initialIdx));
useInput((input, key) => {
if (input === "q" || (key.ctrl && input === "c")) {
exit();
return;
}
if (input === "j" || key.downArrow || input === " " || key.return) {
setIdx((i) => Math.min(maxIdx, i + 1));
} else if (input === "k" || key.upArrow) {
setIdx((i) => Math.max(0, i - 1));
} else if (input === "g") {
setIdx(0);
} else if (input === "G") {
setIdx(maxIdx);
} else if (input === "n") {
const next = findNextDivergence(report.pairs, idx);
if (next !== -1) setIdx(next);
} else if (input === "N" || input === "p") {
const prev = findPrevDivergence(report.pairs, idx);
if (prev !== -1) setIdx(prev);
}
});
const pair = report.pairs[idx];
return (
<Box flexDirection="column">
<DiffHeader report={report} />
<Box marginTop={1} paddingX={1} justifyContent="space-between">
<Text color="cyan" bold>
turn {pair?.turn ?? "?"} ({idx + 1} / {report.pairs.length})
</Text>
<Text>{pair ? <KindBadge kind={pair.kind} /> : null}</Text>
</Box>
<Box flexDirection="row" marginTop={1}>
<Pane label={report.a.label} headerColor="blue" records={paneRecords(pair, "a")} />
<Pane label={report.b.label} headerColor="magenta" records={paneRecords(pair, "b")} />
</Box>
{pair?.divergenceNote ? (
<Box marginTop={1} paddingX={1}>
<Text color="yellow">★ </Text>
<Text>{pair.divergenceNote}</Text>
</Box>
) : null}
<Box marginTop={1} paddingX={1} borderStyle="single" borderColor="gray">
<Text dimColor>
<Text bold>j</Text>/<Text bold>↓</Text> next · <Text bold>k</Text>/<Text bold>↑</Text>{" "}
prev · <Text bold>n</Text> next-diverge · <Text bold>N</Text>/<Text bold>p</Text>{" "}
prev-diverge · <Text bold>g</Text>/<Text bold>G</Text> first/last · <Text bold>q</Text>{" "}
quit
</Text>
</Box>
</Box>
);
}
// ----------------------------------------------------------------------------
function DiffHeader({ report }: { report: DiffReport }) {
const a = report.a;
const b = report.b;
const cacheDelta = b.stats.cacheHitRatio - a.stats.cacheHitRatio;
const costDelta =
a.stats.totalCostUsd > 0
? ((b.stats.totalCostUsd - a.stats.totalCostUsd) / a.stats.totalCostUsd) * 100
: 0;
// Prefix stability one-liner (same logic as the stdout summary).
const aStable = a.stats.prefixHashes.length <= 1;
const bStable = b.stats.prefixHashes.length <= 1;
let prefixLine: string | null = null;
if (aStable !== bStable) {
const stableLabel = aStable ? report.a.label : report.b.label;
const churnLabel = aStable ? report.b.label : report.a.label;
const churnCount = aStable ? b.stats.prefixHashes.length : a.stats.prefixHashes.length;
prefixLine = `${stableLabel} stayed byte-stable; ${churnLabel} churned ${churnCount} distinct prefixes.`;
} else if (a.stats.prefixHashes[0] && a.stats.prefixHashes[0] === b.stats.prefixHashes[0]) {
prefixLine = `shared prefix hash ${a.stats.prefixHashes[0].slice(0, 12)}… — cache delta attributable to log stability, not prompt change.`;
}
return (
<Box flexDirection="column" borderStyle="round" borderColor="cyan" paddingX={1}>
<Box justifyContent="space-between">
<Text>
<Text color="cyan" bold>
reasonix diff
</Text>
<Text dimColor> · A=</Text>
<Text color="blue">{a.label}</Text>
<Text dimColor> vs B=</Text>
<Text color="magenta">{b.label}</Text>
</Text>
<Text dimColor>{report.pairs.length} turns aligned</Text>
</Box>
<Box marginTop={1} gap={3}>
<Text>
<Text dimColor>cache </Text>
<Text>{(a.stats.cacheHitRatio * 100).toFixed(1)}%</Text>
<Text dimColor> → </Text>
<Text>{(b.stats.cacheHitRatio * 100).toFixed(1)}%</Text>
<Text color={cacheDelta >= 0 ? "green" : "red"} bold>
{" "}
{cacheDelta >= 0 ? "+" : ""}
{(cacheDelta * 100).toFixed(1)}pp
</Text>
</Text>
<Text>
<Text dimColor>cost </Text>
<Text>${a.stats.totalCostUsd.toFixed(6)}</Text>
<Text dimColor> → </Text>
<Text>${b.stats.totalCostUsd.toFixed(6)}</Text>
<Text color={costDelta <= 0 ? "green" : "red"} bold>
{" "}
{costDelta >= 0 ? "+" : ""}
{costDelta.toFixed(1)}%
</Text>
</Text>
<Text>
<Text dimColor>model calls </Text>
<Text>
{a.stats.turns} → {b.stats.turns}
</Text>
</Text>
</Box>
{prefixLine ? (
<Box marginTop={1}>
<Text dimColor italic>
{prefixLine}
</Text>
</Box>
) : null}
</Box>
);
}
function Pane({
label,
headerColor,
records,
}: {
label: string;
headerColor: "blue" | "magenta";
records: TurnPair["aTools"];
}) {
return (
<Box
flexDirection="column"
flexGrow={1}
paddingX={1}
borderStyle="single"
borderColor={headerColor}
>
<Text color={headerColor} bold>
{label}
</Text>
{records.length === 0 ? (
<Box marginTop={1}>
<Text dimColor italic>
(no records on this side for this turn)
</Text>
</Box>
) : (
<Static items={records.map((rec, i) => ({ key: `${label}-${i}`, rec }))}>
{({ key, rec }) => <RecordView key={key} rec={rec} compact />}
</Static>
)}
</Box>
);
}
function KindBadge({ kind }: { kind: TurnPair["kind"] }) {
if (kind === "match") {
return <Text color="green">✓ match</Text>;
}
if (kind === "diverge") {
return <Text color="yellow">★ diverge</Text>;
}
if (kind === "only_in_a") {
return <Text color="blue">← only in A</Text>;
}
return <Text color="magenta">→ only in B</Text>;
}
// ----------------------------------------------------------------------------
function paneRecords(pair: TurnPair | undefined, side: "a" | "b"): TurnPair["aTools"] {
if (!pair) return [];
const tools = side === "a" ? pair.aTools : pair.bTools;
const assistant = side === "a" ? pair.aAssistant : pair.bAssistant;
const out: TurnPair["aTools"] = [...tools];
if (assistant) out.push(assistant);
return out;
}
+292
View File
@@ -0,0 +1,292 @@
import { Box, Text } from "ink";
import React, { useEffect, useState } from "react";
import { type TypedPlanState, isPlanStateEmpty } from "../../harvest.js";
import type { BranchProgress, BranchSummary } from "../../loop.js";
import type { TurnStats } from "../../telemetry.js";
import { PlanStateBlock } from "./PlanStateBlock.js";
import { Markdown } from "./markdown.js";
export type DisplayRole = "user" | "assistant" | "tool" | "system" | "error" | "info" | "warning";
export interface DisplayEvent {
id: string;
role: DisplayRole;
text: string;
reasoning?: string;
planState?: TypedPlanState;
branch?: BranchSummary;
branchProgress?: BranchProgress;
toolName?: string;
stats?: TurnStats;
repair?: string;
streaming?: boolean;
}
export const EventRow = React.memo(function EventRow({ event }: { event: DisplayEvent }) {
if (event.role === "user") {
return (
<Box>
<Text bold color="cyan">
you ›{" "}
</Text>
<Text>{event.text}</Text>
</Box>
);
}
if (event.role === "assistant") {
if (event.streaming) return <StreamingAssistant event={event} />;
return (
<Box flexDirection="column" marginTop={1}>
<Box>
<Text bold color="green">
assistant
</Text>
</Box>
{event.branch ? <BranchBlock branch={event.branch} /> : null}
{event.reasoning ? <ReasoningBlock reasoning={event.reasoning} /> : null}
{!isPlanStateEmpty(event.planState) ? (
<PlanStateBlock planState={event.planState!} />
) : null}
{event.text ? <Markdown text={event.text} /> : <Text dimColor>(no content)</Text>}
{event.stats ? <StatsLine stats={event.stats} /> : null}
{event.repair ? <Text color="magenta">{event.repair}</Text> : null}
</Box>
);
}
if (event.role === "tool") {
// `flattenMcpResult` prefixes server-side errors with "ERROR: ".
// Render those in red with a ✗ marker so they don't blend into
// successful tool output (yellow) — the failure mode is what the
// model most likely needs to act on next, and the user needs to
// see at a glance.
const isError = event.text.startsWith("ERROR:");
const color = isError ? "red" : "yellow";
const marker = isError ? "✗" : "→";
return (
<Box flexDirection="column" marginTop={1}>
<Text color={color}>{`tool<${event.toolName ?? "?"}> ${marker}`}</Text>
<Text color={isError ? "red" : undefined} dimColor={!isError}>
{" "}
{truncate(event.text, 400)}
</Text>
</Box>
);
}
if (event.role === "error") {
return (
<Box marginTop={1}>
<Text color="red" bold>
error{" "}
</Text>
<Text color="red">{event.text}</Text>
</Box>
);
}
if (event.role === "info") {
return (
<Box>
<Text dimColor>{event.text}</Text>
</Box>
);
}
if (event.role === "warning") {
return (
<Box>
<Text color="yellow">▸ </Text>
<Text color="yellow">{event.text}</Text>
</Box>
);
}
return (
<Box>
<Text>{event.text}</Text>
</Box>
);
});
function BranchBlock({ branch }: { branch: BranchSummary }) {
const per = branch.uncertainties
.map((u, i) => {
const marker = i === branch.chosenIndex ? "▸" : " ";
const t = (branch.temperatures[i] ?? 0).toFixed(1);
return `${marker} #${i} T=${t} u=${u}`;
})
.join(" ");
return (
<Box>
<Text color="blue">
{"🔀 branched "}
<Text bold>{branch.budget}</Text>
{` samples → picked #${branch.chosenIndex} `}
<Text dimColor>{per}</Text>
</Text>
</Box>
);
}
function ReasoningBlock({ reasoning }: { reasoning: string }) {
const max = 260;
const flat = reasoning.replace(/\s+/g, " ").trim();
// Show the TAIL of the reasoning rather than the head. R1 opens
// with generic scaffolding ("let me look at the structure...") that
// repeats across turns and hides the part users actually want to
// see — the decision right before the model commits to an action.
// Users can dump the full reasoning with `/think` if needed.
const preview =
flat.length <= max ? flat : `… (+${flat.length - max} earlier chars) ${flat.slice(-max)}`;
return (
<Box marginBottom={1}>
<Text dimColor italic>
{"↳ thinking: "}
{preview}
</Text>
</Box>
);
}
/**
* Compact progress view rendered while a turn is still streaming. We keep
* this to a fixed ~3-line footprint so the dynamic region never scrolls past
* the terminal viewport and leaves artifacts in scrollback.
*/
function Elapsed() {
const [s, setS] = useState(0);
useEffect(() => {
const start = Date.now();
const id = setInterval(() => setS(Math.floor((Date.now() - start) / 1000)), 1000);
return () => clearInterval(id);
}, []);
const mm = String(Math.floor(s / 60)).padStart(2, "0");
const ss = String(s % 60).padStart(2, "0");
return <Text dimColor>{`${mm}:${ss}`}</Text>;
}
function StreamingAssistant({ event }: { event: DisplayEvent }) {
if (event.branchProgress) {
const p = event.branchProgress;
// completed=0 means we've just started; no sample has finished yet.
if (p.completed === 0) {
return (
<Box flexDirection="column" marginTop={1}>
<Box>
<Text bold color="green">
assistant{" "}
</Text>
<Text color="blue">
🔀 launching {p.total} parallel samples (R1 thinking in parallel)…{" "}
</Text>
<Elapsed />
</Box>
<Text dimColor>{" "}spread across T=0.0/0.5/1.0 · typical wait 30-90s for reasoner</Text>
</Box>
);
}
const pct = Math.round((p.completed / p.total) * 100);
return (
<Box flexDirection="column" marginTop={1}>
<Box>
<Text bold color="green">
assistant{" "}
</Text>
<Text color="blue">
🔀 branching {p.completed}/{p.total} ({pct}%){" "}
</Text>
<Elapsed />
</Box>
<Text dimColor>
{" latest #"}
{p.latestIndex}
{" T="}
{p.latestTemperature.toFixed(1)}
{" u="}
{p.latestUncertainties}
{p.completed < p.total ? " · waiting for other samples…" : " · selecting winner…"}
</Text>
</Box>
);
}
const tail = lastLine(event.text, 140);
const reasoningTail = event.reasoning ? lastLine(event.reasoning, 120) : "";
// R1 ("deepseek-reasoner") generates reasoning_content first, then
// content. While reasoning is streaming but content is still empty,
// we were showing "(waiting for first token…)" — which looked like a
// hang. The data is flowing, it's just landing in the thinking
// channel. Reflect that honestly.
const reasoningOnly = !event.text && !!event.reasoning;
return (
<Box flexDirection="column" marginTop={1}>
<Box>
<Text bold color="green">
assistant{" "}
</Text>
<Pulse />
<Text dimColor>
{" "}
({reasoningOnly ? "reasoning" : "streaming"} · {event.text.length}
{event.reasoning ? ` + think ${event.reasoning.length}` : ""} chars){" "}
</Text>
<Elapsed />
</Box>
{reasoningTail ? (
<Text dimColor italic>
↳ thinking: {reasoningTail}
</Text>
) : null}
{tail ? (
<Text dimColor>▸ {tail}</Text>
) : reasoningOnly ? (
<Text color="yellow" dimColor>
{
" R1 is thinking before it speaks — body text starts when reasoning completes (typically 20-90s)."
}
</Text>
) : (
<Text dimColor italic>
{" (waiting for first byte — connection is open)"}
</Text>
)}
</Box>
);
}
/**
* Blinking indicator so the user can tell the stream is alive even
* when the reasoner hasn't produced body text yet. Ticks every 500 ms
* regardless of content flow — it's a heartbeat, not a progress bar.
*/
function Pulse() {
const [tick, setTick] = useState(0);
useEffect(() => {
const id = setInterval(() => setTick((t) => t + 1), 500);
return () => clearInterval(id);
}, []);
const frames = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"];
return <Text color="cyan">{frames[tick % frames.length]}</Text>;
}
function lastLine(s: string, maxChars: number): string {
const flat = s.replace(/\s+/g, " ").trim();
if (!flat) return "";
return flat.length <= maxChars ? flat : `…${flat.slice(-maxChars)}`;
}
function StatsLine({ stats }: { stats: TurnStats }) {
const hit = (stats.cacheHitRatio * 100).toFixed(1);
return (
<Text dimColor>
{" ↳ cache "}
{hit}
{"% · tokens "}
{stats.usage.promptTokens}
{"→"}
{stats.usage.completionTokens}
{" · $"}
{stats.cost.toFixed(6)}
</Text>
);
}
function truncate(s: string, max: number): string {
return s.length <= max ? s : `${s.slice(0, max)}… (+${s.length - max} chars)`;
}
+48
View File
@@ -0,0 +1,48 @@
/**
* Shared Ink block that renders a TypedPlanState. Used by the live chat
* EventLog AND by the RecordView in replay/diff TUIs, so harvest output
* looks identical live and on replay.
*
* Colors are semantic (not decorative):
* - subgoals: cyan — structure / plan
* - hypotheses: green — current beliefs (like assistant)
* - uncertainties: yellow — attention required (like tool)
* - rejected paths: red dim — ruled out (muted, like error-but-resolved)
*
* Only the label is colored + bold. The items themselves render in the
* terminal's default foreground, so they stay readable on any background —
* which is why the old single-magenta block was hard to see on dark themes.
*/
import { Box, Text } from "ink";
import React from "react";
import type { TypedPlanState } from "../../harvest.js";
type FieldColor = "cyan" | "green" | "yellow" | "red";
export function PlanStateBlock({ planState }: { planState: TypedPlanState }) {
const fields: Array<[string, string[], FieldColor, boolean]> = [];
if (planState.subgoals.length) fields.push(["subgoals", planState.subgoals, "cyan", false]);
if (planState.hypotheses.length)
fields.push(["hypotheses", planState.hypotheses, "green", false]);
if (planState.uncertainties.length)
fields.push(["uncertainties", planState.uncertainties, "yellow", false]);
if (planState.rejectedPaths.length)
fields.push(["rejected", planState.rejectedPaths, "red", true]);
if (fields.length === 0) return null;
return (
<Box flexDirection="column" marginBottom={1}>
{fields.map(([label, items, color, dim]) => (
<Text key={label}>
<Text color={color} bold dimColor={dim}>
{"‹ "}
{label}
</Text>
<Text dimColor>{` (${items.length})`}</Text>
<Text>{`: ${items.join(" · ")}`}</Text>
</Text>
))}
</Box>
);
}
+43
View File
@@ -0,0 +1,43 @@
import { Box, Text } from "ink";
import TextInput from "ink-text-input";
import React from "react";
export interface PromptInputProps {
value: string;
onChange: (v: string) => void;
onSubmit: (v: string) => void;
disabled?: boolean;
placeholder?: string;
}
/**
* Keep `<TextInput>` mounted at all times and use its `focus` prop to gate
* input. Conditionally rendering it (mount / unmount between turns) loses
* the stdin raw-mode claim on some terminals, which silently drops
* keystrokes after the first turn finishes.
*/
export function PromptInput({
value,
onChange,
onSubmit,
disabled,
placeholder,
}: PromptInputProps) {
const effectivePlaceholder = disabled
? (placeholder ?? "…waiting for response…")
: (placeholder ?? "type a message, or /command");
return (
<Box borderStyle="round" borderColor={disabled ? "gray" : "cyan"} paddingX={1}>
<Text bold color={disabled ? "gray" : "cyan"}>
you ›{" "}
</Text>
<TextInput
value={value}
onChange={onChange}
onSubmit={onSubmit}
focus={!disabled}
placeholder={effectivePlaceholder}
/>
</Box>
);
}
+127
View File
@@ -0,0 +1,127 @@
/**
* Shared renderer for a single TranscriptRecord. Used by ReplayApp and
* DiffApp — both need the same visual grammar (user cyan, assistant green,
* tool yellow, error red, cache badge colored by threshold) so transcripts
* look consistent wherever they're displayed.
*
* Kept small on purpose: no streaming/branch/planState paths (those are
* live-chat concerns and never appear in replayed transcripts).
*/
import { Box, Text } from "ink";
import React from "react";
import type { TranscriptRecord } from "../../transcript.js";
import { PlanStateBlock } from "./PlanStateBlock.js";
export interface RecordViewProps {
rec: TranscriptRecord;
/**
* When rendering side-by-side in diff mode, shorter truncation limits
* keep long tool results from dominating the pane. Passes through
* untouched when undefined.
*/
compact?: boolean;
}
export function RecordView({ rec, compact = false }: RecordViewProps) {
const toolArgsMax = compact ? 120 : 200;
const toolContentMax = compact ? 200 : 400;
if (rec.role === "user") {
return (
<Box marginTop={1}>
<Text bold color="cyan">
you ›{" "}
</Text>
<Text>{rec.content}</Text>
</Box>
);
}
if (rec.role === "assistant_final") {
return (
<Box flexDirection="column" marginTop={1}>
<Box>
<Text bold color="green">
assistant
</Text>
{rec.cost !== undefined ? (
<Text dimColor>
{" $"}
{rec.cost.toFixed(6)}
</Text>
) : null}
{rec.usage ? <CacheBadge usage={rec.usage} /> : null}
</Box>
{rec.planState ? <PlanStateBlock planState={rec.planState} /> : null}
{rec.content ? (
<Text>{rec.content}</Text>
) : (
<Text dimColor italic>
(tool-call response only)
</Text>
)}
</Box>
);
}
if (rec.role === "tool") {
return (
<Box flexDirection="column" marginTop={1}>
<Text color="yellow">
{"tool<"}
{rec.tool ?? "?"}
{">"}
</Text>
{rec.args ? (
<Text dimColor>
{" args: "}
{truncate(rec.args, toolArgsMax)}
</Text>
) : null}
<Text dimColor>
{" → "}
{truncate(rec.content, toolContentMax)}
</Text>
</Box>
);
}
if (rec.role === "error") {
return (
<Box marginTop={1}>
<Text color="red" bold>
error{" "}
</Text>
<Text color="red">{rec.error ?? rec.content}</Text>
</Box>
);
}
if (rec.role === "done" || rec.role === "assistant_delta") {
// Noise in replay; skip.
return null;
}
return (
<Box>
<Text dimColor>
[{rec.role}] {rec.content}
</Text>
</Box>
);
}
function CacheBadge({ usage }: { usage: NonNullable<TranscriptRecord["usage"]> }) {
const hit = usage.prompt_cache_hit_tokens ?? 0;
const miss = usage.prompt_cache_miss_tokens ?? 0;
const total = hit + miss;
if (total === 0) return null;
const pct = (hit / total) * 100;
const color = pct >= 70 ? "green" : pct >= 40 ? "yellow" : "red";
return (
<Text>
<Text dimColor>{" · cache "}</Text>
<Text color={color}>{pct.toFixed(1)}%</Text>
</Text>
);
}
function truncate(s: string, max: number): string {
return s.length <= max ? s : `${s.slice(0, max)}… (+${s.length - max} chars)`;
}
+115
View File
@@ -0,0 +1,115 @@
/**
* Ink TUI for `reasonix replay`. Read-only: no input box, no loop.
* j/k navigation across turn-pages, cumulative stats sidebar updates
* as you move through time.
*
* The navigation logic (grouping records into pages, computing cumulative
* stats) lives in src/replay.ts as pure functions; this file is just
* presentation + key bindings.
*/
import { Box, Static, Text, useApp, useInput } from "ink";
import React, { useMemo, useState } from "react";
import { type TurnPage, computeCumulativeStats } from "../../replay.js";
import type { TranscriptMeta } from "../../transcript.js";
import { RecordView } from "./RecordView.js";
import { StatsPanel } from "./StatsPanel.js";
export interface ReplayAppProps {
meta: TranscriptMeta | null;
pages: TurnPage[];
}
export function ReplayApp({ meta, pages }: ReplayAppProps) {
const { exit } = useApp();
const maxIdx = Math.max(0, pages.length - 1);
// Start at the last page — more useful than "start from the beginning"
// in practice: users mostly want to see the summary + last turn first.
const [idx, setIdx] = useState(maxIdx);
useInput((input, key) => {
if (input === "q" || (key.ctrl && input === "c")) {
exit();
return;
}
if (input === "j" || key.downArrow || input === " " || key.return) {
setIdx((i) => Math.min(maxIdx, i + 1));
} else if (input === "k" || key.upArrow) {
setIdx((i) => Math.max(0, i - 1));
} else if (input === "g") {
setIdx(0);
} else if (input === "G") {
setIdx(maxIdx);
} else if (input === "h" || key.leftArrow) {
setIdx(0);
} else if (input === "l" || key.rightArrow) {
setIdx(maxIdx);
}
});
const cumStats = useMemo(() => computeCumulativeStats(pages, idx), [pages, idx]);
const summary = {
turns: cumStats.turns,
totalCostUsd: cumStats.totalCostUsd,
claudeEquivalentUsd: cumStats.claudeEquivalentUsd,
savingsVsClaudePct: cumStats.savingsVsClaudePct,
cacheHitRatio: cumStats.cacheHitRatio,
// Replay is read-only — no live last-turn prompt tokens to show.
lastPromptTokens: 0,
};
const prefixHash =
cumStats.prefixHashes.length === 1
? cumStats.prefixHashes[0]!.slice(0, 16)
: cumStats.prefixHashes.length === 0
? "(untracked)"
: `(churned ×${cumStats.prefixHashes.length})`;
const currentPage = pages[idx];
const progressLabel =
pages.length === 0 ? "empty transcript" : `turn ${idx + 1} / ${pages.length}`;
return (
<Box flexDirection="column">
<StatsPanel
summary={summary}
model={cumStats.models[0] ?? meta?.model ?? "?"}
prefixHash={prefixHash}
/>
<Box flexDirection="column" marginTop={1} paddingX={1}>
<Box justifyContent="space-between">
<Text color="cyan" bold>
{progressLabel}
</Text>
{meta ? (
<Text dimColor>
{meta.source}
{meta.task ? ` · ${meta.task}` : ""}
{meta.mode ? ` · ${meta.mode}` : ""}
</Text>
) : null}
</Box>
{currentPage ? (
<Static items={currentPage.records.map((rec, i) => ({ key: `${idx}-${i}`, rec }))}>
{({ key, rec }) => <RecordView key={key} rec={rec} />}
</Static>
) : (
<Text dimColor italic>
no records
</Text>
)}
</Box>
<Box marginTop={1} paddingX={1} borderStyle="single" borderColor="gray">
<Text dimColor>
<Text bold>j</Text>/<Text bold>↓</Text>/<Text bold>space</Text> next · <Text bold>k</Text>
/<Text bold>↑</Text> prev · <Text bold>g</Text> first · <Text bold>G</Text> last ·{" "}
<Text bold>q</Text> quit
</Text>
</Box>
</Box>
);
}
+182
View File
@@ -0,0 +1,182 @@
/**
* Minimal arrow-key list components for Ink — single-select and
* multi-select. No external deps beyond Ink's `useInput`.
*
* Why hand-roll: `ink-select-input` exists, but it defaults to
* Enter-only interaction (no space-to-toggle for multi-select), doesn't
* expose the "hint / footer" slot we want under each item, and would be
* another dep for ~60 lines of UI code. Reasonix already has one React
* rendering quirk bundled (`ink-text-input`); adding more is low value.
*/
import { Box, Text, useInput } from "ink";
import React, { useState } from "react";
export interface SelectItem<V extends string = string> {
/** Stable identifier — returned to caller on submit. */
value: V;
/** First-row label. */
label: string;
/** Optional second row rendered dimmed. */
hint?: string;
/** If true, item is not selectable (rendered dimmed, skipped on nav). */
disabled?: boolean;
}
export interface SingleSelectProps<V extends string> {
items: SelectItem<V>[];
initialValue?: V;
onSubmit: (value: V) => void;
onCancel?: () => void;
}
export function SingleSelect<V extends string>({
items,
initialValue,
onSubmit,
onCancel,
}: SingleSelectProps<V>) {
const initialIndex = Math.max(
0,
items.findIndex((i) => i.value === initialValue && !i.disabled),
);
const [index, setIndex] = useState(initialIndex === -1 ? 0 : initialIndex);
useInput((_input, key) => {
if (key.upArrow) {
setIndex((i) => findNextEnabled(items, i, -1));
} else if (key.downArrow) {
setIndex((i) => findNextEnabled(items, i, +1));
} else if (key.return) {
const chosen = items[index];
if (chosen && !chosen.disabled) onSubmit(chosen.value);
} else if (key.escape && onCancel) {
onCancel();
}
});
return (
<Box flexDirection="column">
{items.map((item, i) => (
<SelectRow
key={item.value}
item={item}
active={i === index}
marker={i === index ? "▸" : " "}
/>
))}
</Box>
);
}
export interface MultiSelectProps<V extends string> {
items: SelectItem<V>[];
initialSelected?: V[];
onSubmit: (values: V[]) => void;
onCancel?: () => void;
/** Footer hint under the list — e.g. "space toggle · enter confirm". */
footer?: string;
}
export function MultiSelect<V extends string>({
items,
initialSelected = [],
onSubmit,
onCancel,
footer,
}: MultiSelectProps<V>) {
const [index, setIndex] = useState(() => {
const first = items.findIndex((i) => !i.disabled);
return first === -1 ? 0 : first;
});
const [selected, setSelected] = useState<Set<V>>(new Set(initialSelected));
useInput((input, key) => {
if (key.upArrow) {
setIndex((i) => findNextEnabled(items, i, -1));
} else if (key.downArrow) {
setIndex((i) => findNextEnabled(items, i, +1));
} else if (input === " ") {
const item = items[index];
if (!item || item.disabled) return;
setSelected((prev) => {
const next = new Set(prev);
if (next.has(item.value)) next.delete(item.value);
else next.add(item.value);
return next;
});
} else if (key.return) {
// Preserve catalog order rather than insertion order, so reruns
// produce the same spec list for the same checkbox set — makes the
// `config.json` diff trivially stable.
const ordered = items.filter((i) => selected.has(i.value)).map((i) => i.value);
onSubmit(ordered);
} else if (key.escape && onCancel) {
onCancel();
}
});
return (
<Box flexDirection="column">
{items.map((item, i) => {
const checked = selected.has(item.value);
const marker = checked ? "[x]" : "[ ]";
return (
<SelectRow
key={item.value}
item={item}
active={i === index}
marker={`${i === index ? "▸" : " "} ${marker}`}
/>
);
})}
{footer ? (
<Box marginTop={1}>
<Text dimColor>{footer}</Text>
</Box>
) : null}
</Box>
);
}
// ---------- internals ----------
function SelectRow<V extends string>({
item,
active,
marker,
}: {
item: SelectItem<V>;
active: boolean;
marker: string;
}) {
const color = item.disabled ? "gray" : active ? "cyan" : undefined;
return (
<Box flexDirection="column">
<Box>
<Text color={color}>
{marker} {item.label}
</Text>
</Box>
{item.hint ? (
<Box paddingLeft={marker.length + 1}>
<Text dimColor>{item.hint}</Text>
</Box>
) : null}
</Box>
);
}
function findNextEnabled<V extends string>(
items: SelectItem<V>[],
from: number,
step: -1 | 1,
): number {
if (items.length === 0) return 0;
let i = from;
for (let tries = 0; tries < items.length; tries++) {
i = (i + step + items.length) % items.length;
if (!items[i]?.disabled) return i;
}
return from;
}
+71
View File
@@ -0,0 +1,71 @@
import { Box, Text, useApp } from "ink";
import TextInput from "ink-text-input";
import React, { useState } from "react";
import { defaultConfigPath, isPlausibleKey, redactKey, saveApiKey } from "../../config.js";
export interface SetupProps {
onReady: (apiKey: string) => void;
}
export function Setup({ onReady }: SetupProps) {
const [value, setValue] = useState("");
const [error, setError] = useState<string | null>(null);
const { exit } = useApp();
const handleSubmit = (raw: string) => {
const trimmed = raw.trim();
if (trimmed === "/exit" || trimmed === "/quit") {
exit();
return;
}
if (!isPlausibleKey(trimmed)) {
setError("Doesn't look like a DeepSeek key. They start with 'sk-' and are 30+ chars.");
setValue("");
return;
}
try {
saveApiKey(trimmed);
} catch (err) {
setError(`Could not save key: ${(err as Error).message}`);
return;
}
onReady(trimmed);
};
return (
<Box flexDirection="column" borderStyle="round" borderColor="cyan" paddingX={1}>
<Text bold color="cyan">
Welcome to Reasonix.
</Text>
<Box marginTop={1}>
<Text>Paste your DeepSeek API key to get started.</Text>
</Box>
<Text dimColor>Get one (free credit on signup): https://platform.deepseek.com/api_keys</Text>
<Text dimColor>Saved locally to {defaultConfigPath()}</Text>
<Box marginTop={1}>
<Text bold color="cyan">
{"key › "}
</Text>
<TextInput
value={value}
onChange={setValue}
onSubmit={handleSubmit}
mask="•"
placeholder="sk-..."
/>
</Box>
{error ? (
<Box marginTop={1}>
<Text color="red">{error}</Text>
</Box>
) : value ? (
<Box marginTop={1}>
<Text dimColor>preview: {redactKey(value)}</Text>
</Box>
) : null}
<Box marginTop={1}>
<Text dimColor>(Type /exit to abort.)</Text>
</Box>
</Box>
);
}
+96
View File
@@ -0,0 +1,96 @@
import { Box, Text } from "ink";
import React from "react";
import { DEEPSEEK_CONTEXT_TOKENS, DEFAULT_CONTEXT_TOKENS } from "../../telemetry.js";
import type { SessionSummary } from "../../telemetry.js";
export interface StatsPanelProps {
summary: SessionSummary;
model: string;
prefixHash: string;
harvestOn?: boolean;
branchBudget?: number;
}
export function StatsPanel({
summary,
model,
prefixHash,
harvestOn,
branchBudget,
}: StatsPanelProps) {
const hitPct = (summary.cacheHitRatio * 100).toFixed(1);
const hitColor =
summary.cacheHitRatio >= 0.7 ? "green" : summary.cacheHitRatio >= 0.4 ? "yellow" : "red";
const branchOn = (branchBudget ?? 1) > 1;
const ctxMax = DEEPSEEK_CONTEXT_TOKENS[model] ?? DEFAULT_CONTEXT_TOKENS;
const ctxRatio = summary.lastPromptTokens / ctxMax;
const ctxColor = ctxRatio >= 0.8 ? "red" : ctxRatio >= 0.5 ? "yellow" : undefined;
return (
<Box borderStyle="round" borderColor="cyan" flexDirection="column" paddingX={1}>
<Box justifyContent="space-between">
<Text>
<Text color="cyan" bold>
Reasonix
</Text>
<Text dimColor> · model </Text>
<Text color="yellow">{model}</Text>
<Text dimColor> · prefix </Text>
<Text dimColor>{prefixHash}</Text>
{harvestOn ? <Text color="magenta"> · harvest</Text> : null}
{branchOn ? <Text color="blue"> · branch{branchBudget}</Text> : null}
</Text>
<Text dimColor>turns {summary.turns} · type /help</Text>
</Box>
<Box marginTop={1} gap={3}>
<Text>
<Text dimColor>cache hit </Text>
<Text color={hitColor} bold>
{hitPct}%
</Text>
</Text>
<Text>
<Text dimColor>cost </Text>
<Text color="green">${summary.totalCostUsd.toFixed(6)}</Text>
</Text>
<Text>
<Text dimColor>vs Claude </Text>
<Text>${summary.claudeEquivalentUsd.toFixed(6)}</Text>
</Text>
<Text>
<Text dimColor>saving </Text>
<Text color="green" bold>
{summary.savingsVsClaudePct.toFixed(1)}%
</Text>
</Text>
{summary.lastPromptTokens > 0 ? (
<Text>
<Text dimColor>ctx </Text>
<Text color={ctxColor} bold={ctxColor !== undefined}>
{formatTokens(summary.lastPromptTokens)}/{formatTokens(ctxMax)}
</Text>
<Text dimColor> ({(ctxRatio * 100).toFixed(0)}%)</Text>
{ctxRatio >= 0.8 ? (
<Text color="red" bold>
{" "}
· /compact
</Text>
) : null}
</Text>
) : null}
</Box>
</Box>
);
}
/**
* Compact integer formatter: 1234 → "1.2k", 131072 → "131k". Keeps the
* panel narrow enough to fit on 80-col terminals even when the context
* is near full.
*/
function formatTokens(n: number): string {
if (n < 1000) return String(n);
const k = n / 1000;
return k >= 100 ? `${k.toFixed(0)}k` : `${k.toFixed(1)}k`;
}
+447
View File
@@ -0,0 +1,447 @@
/**
* First-run / re-configure wizard.
*
* Walks a new user through: API key → preset pick → MCP server pick →
* per-server args → save. Saved output lives in `~/.reasonix/config.json`
* so the next `reasonix chat` starts with everything already wired.
*
* The wizard is the antidote to "too many CLI flags" — a new user should
* never have to read `--help` to get MCP + a sensible model combo
* working. Everything a user could set via `--mcp`, `--harvest`,
* `--branch`, etc. can be picked here in a few arrow-key presses.
*/
import { Box, Text, useApp, useInput } from "ink";
import TextInput from "ink-text-input";
// biome-ignore lint/style/useImportType: JSX (jsx: "react") needs React as a value at runtime
import React, { useState } from "react";
import {
type PresetName,
type ReasonixConfig,
defaultConfigPath,
isPlausibleKey,
readConfig,
redactKey,
writeConfig,
} from "../../config.js";
import { type CatalogEntry, MCP_CATALOG } from "../../mcp/catalog.js";
import { MultiSelect, type SelectItem, SingleSelect } from "./Select.js";
import { PRESET_DESCRIPTIONS } from "./presets.js";
export interface WizardProps {
/** Called once the config has been saved. */
onComplete: (cfg: ReasonixConfig) => void;
/** Called if the user presses Esc to abort. */
onCancel?: () => void;
/** Skip the API-key step if a key already exists (env or config). */
existingApiKey?: string;
/** Pre-fill selections when re-running (reconfigure flow). */
initial?: {
preset?: PresetName;
mcp?: string[];
};
}
type Step = "apiKey" | "preset" | "mcp" | "mcpArgs" | "review" | "saved";
interface WizardData {
apiKey: string;
preset: PresetName;
selectedCatalog: string[]; // entries from MCP_CATALOG by `name`
/** Captured user inputs per catalog entry that has `userArgs` (e.g. fs dir). */
catalogArgs: Record<string, string>;
}
const CATALOG_BY_NAME = new Map(MCP_CATALOG.map((e) => [e.name, e]));
export function Wizard({ onComplete, onCancel, existingApiKey, initial }: WizardProps) {
const { exit } = useApp();
const [step, setStep] = useState<Step>(existingApiKey ? "preset" : "apiKey");
const [data, setData] = useState<WizardData>({
apiKey: existingApiKey ?? "",
preset: initial?.preset ?? "fast",
selectedCatalog: deriveInitialCatalog(initial?.mcp ?? []),
catalogArgs: {},
});
const [error, setError] = useState<string | null>(null);
// Global Esc → cancel. Disabled once we've started saving to avoid
// ejecting out of a half-written state.
useInput((_input, key) => {
if (key.escape && step !== "saved" && onCancel) onCancel();
});
if (step === "apiKey") {
return (
<ApiKeyStep
onSubmit={(key) => {
setData((d) => ({ ...d, apiKey: key }));
setError(null);
setStep("preset");
}}
error={error}
onError={setError}
/>
);
}
if (step === "preset") {
return (
<StepFrame title="Pick a preset" step={1} total={3}>
<SingleSelect<PresetName>
items={presetItems()}
initialValue={data.preset}
onSubmit={(preset) => {
setData((d) => ({ ...d, preset }));
setStep("mcp");
}}
/>
<Box marginTop={1}>
<Text dimColor>↑/↓ move · enter confirm · esc cancel</Text>
</Box>
</StepFrame>
);
}
if (step === "mcp") {
return (
<StepFrame title="Which MCP servers should Reasonix wire up for you?" step={2} total={3}>
<MultiSelect
items={mcpItems()}
initialSelected={data.selectedCatalog}
onSubmit={(selected) => {
setData((d) => ({ ...d, selectedCatalog: selected }));
// Only advance to the args step if any selected entry needs args.
const needsArgs = selected.some((name) => CATALOG_BY_NAME.get(name)?.userArgs);
setStep(needsArgs ? "mcpArgs" : "review");
}}
footer="↑/↓ move · space toggle · enter confirm · esc cancel · leave empty to skip"
/>
</StepFrame>
);
}
if (step === "mcpArgs") {
const pending = data.selectedCatalog.filter((name) => {
const entry = CATALOG_BY_NAME.get(name);
return entry?.userArgs && !data.catalogArgs[name];
});
if (pending.length === 0) {
setStep("review");
return null;
}
const currentName = pending[0]!;
const entry = CATALOG_BY_NAME.get(currentName)!;
return (
<McpArgsStep
entry={entry}
error={error}
onSubmit={(value) => {
setData((d) => ({
...d,
catalogArgs: { ...d.catalogArgs, [currentName]: value },
}));
setError(null);
}}
onError={setError}
/>
);
}
if (step === "review") {
const specs = data.selectedCatalog.map((name) => buildSpec(name, data.catalogArgs));
return (
<StepFrame title="Ready to save" step={3} total={3}>
<Box flexDirection="column">
<SummaryLine label="API key" value={redactKey(data.apiKey)} />
<SummaryLine label="Preset" value={data.preset} />
<SummaryLine
label="MCP"
value={specs.length === 0 ? "(none)" : `${specs.length} server(s)`}
/>
{specs.map((spec, i) => (
// biome-ignore lint/suspicious/noArrayIndexKey: review-only render, order fixed
<Box key={i} paddingLeft={14}>
<Text dimColor>· {spec}</Text>
</Box>
))}
<Box marginTop={1}>
<Text>Saves to {defaultConfigPath()}</Text>
</Box>
{error ? (
<Box marginTop={1}>
<Text color="red">{error}</Text>
</Box>
) : null}
<Box marginTop={1}>
<Text dimColor>enter save · esc cancel</Text>
</Box>
</Box>
<ReviewConfirm
onConfirm={() => {
try {
const specsNow = data.selectedCatalog.map((name) =>
buildSpec(name, data.catalogArgs),
);
const prev = readConfig();
const next: ReasonixConfig = {
...prev,
apiKey: data.apiKey,
preset: data.preset,
mcp: specsNow,
setupCompleted: true,
};
writeConfig(next);
setStep("saved");
onComplete(next);
} catch (e) {
setError(`Could not save config: ${(e as Error).message}`);
}
}}
/>
</StepFrame>
);
}
// saved
return (
<Box flexDirection="column" borderStyle="round" borderColor="green" paddingX={1}>
<Text bold color="green">
▸ Saved.
</Text>
<Box marginTop={1}>
<Text>Run `reasonix` any time to start chatting — your settings are remembered.</Text>
</Box>
<Box marginTop={1}>
<Text dimColor>Press enter to exit.</Text>
</Box>
<ExitOnEnter onExit={exit} />
</Box>
);
}
// ---------- step components ----------
function ApiKeyStep({
onSubmit,
error,
onError,
}: {
onSubmit: (key: string) => void;
error: string | null;
onError: (e: string | null) => void;
}) {
const [value, setValue] = useState("");
return (
<Box flexDirection="column" borderStyle="round" borderColor="cyan" paddingX={1}>
<Text bold color="cyan">
Welcome to Reasonix.
</Text>
<Box marginTop={1}>
<Text>Paste your DeepSeek API key to get started.</Text>
</Box>
<Text dimColor>Get one (free credit on signup): https://platform.deepseek.com/api_keys</Text>
<Text dimColor>Saved locally to {defaultConfigPath()}</Text>
<Box marginTop={1}>
<Text bold color="cyan">
{"key › "}
</Text>
<TextInput
value={value}
onChange={setValue}
onSubmit={(raw) => {
const trimmed = raw.trim();
if (!isPlausibleKey(trimmed)) {
onError("Doesn't look like a DeepSeek key. They start with 'sk-' and are 30+ chars.");
setValue("");
return;
}
onSubmit(trimmed);
}}
mask="•"
placeholder="sk-..."
/>
</Box>
{error ? (
<Box marginTop={1}>
<Text color="red">{error}</Text>
</Box>
) : value ? (
<Box marginTop={1}>
<Text dimColor>preview: {redactKey(value)}</Text>
</Box>
) : null}
</Box>
);
}
function McpArgsStep({
entry,
error,
onSubmit,
onError,
}: {
entry: CatalogEntry;
error: string | null;
onSubmit: (value: string) => void;
onError: (e: string | null) => void;
}) {
const [value, setValue] = useState("");
return (
<StepFrame title={`Configure ${entry.name}`} step={2} total={3}>
<Box flexDirection="column">
<Text>{entry.summary}</Text>
{entry.note ? (
<Box marginTop={1}>
<Text dimColor>{entry.note}</Text>
</Box>
) : null}
<Box marginTop={1}>
<Text>Required parameter: </Text>
<Text bold>{entry.userArgs}</Text>
</Box>
<Box marginTop={1}>
<Text bold color="cyan">
{entry.userArgs}
{" › "}
</Text>
<TextInput
value={value}
onChange={setValue}
onSubmit={(raw) => {
const trimmed = raw.trim();
if (!trimmed) {
onError(`${entry.name} needs a value — got an empty string.`);
return;
}
onSubmit(trimmed);
setValue("");
}}
placeholder={placeholderFor(entry)}
/>
</Box>
{error ? (
<Box marginTop={1}>
<Text color="red">{error}</Text>
</Box>
) : null}
</Box>
</StepFrame>
);
}
function ReviewConfirm({ onConfirm }: { onConfirm: () => void }) {
useInput((_i, key) => {
if (key.return) onConfirm();
});
return null;
}
function ExitOnEnter({ onExit }: { onExit: () => void }) {
useInput((_i, key) => {
if (key.return) onExit();
});
return null;
}
// ---------- small bits ----------
function StepFrame({
title,
step,
total,
children,
}: {
title: string;
step: number;
total: number;
children: React.ReactNode;
}) {
return (
<Box flexDirection="column" borderStyle="round" borderColor="cyan" paddingX={1}>
<Box>
<Text dimColor>
Step {step}/{total} ·{" "}
</Text>
<Text bold color="cyan">
{title}
</Text>
</Box>
<Box marginTop={1} flexDirection="column">
{children}
</Box>
</Box>
);
}
function SummaryLine({ label, value }: { label: string; value: string }) {
return (
<Box>
<Text>{label.padEnd(12)}</Text>
<Text bold>{value}</Text>
</Box>
);
}
// ---------- data helpers ----------
function presetItems(): SelectItem<PresetName>[] {
return (["fast", "smart", "max"] as PresetName[]).map((name) => ({
value: name,
label: `${name} — ${PRESET_DESCRIPTIONS[name].headline}`,
hint: PRESET_DESCRIPTIONS[name].cost,
}));
}
function mcpItems(): SelectItem<string>[] {
return MCP_CATALOG.map((entry) => {
const hintParts: string[] = [entry.summary];
if (entry.userArgs) hintParts.push(`(you'll provide ${entry.userArgs})`);
if (entry.note) hintParts.push(entry.note);
return {
value: entry.name,
label: entry.name,
hint: hintParts.join(" · "),
};
});
}
function placeholderFor(entry: CatalogEntry): string {
if (entry.name === "filesystem") return "e.g. /tmp/reasonix-sandbox";
if (entry.name === "sqlite") return "e.g. ./notes.sqlite";
return entry.userArgs ?? "";
}
function deriveInitialCatalog(existingSpecs: string[]): string[] {
// Best-effort recovery: if the user previously picked catalog entries,
// the spec strings look like `name=npx -y <pkg> ...`. Match by package
// so reconfigure pre-checks the same boxes.
const packageToName = new Map(MCP_CATALOG.map((e) => [e.package, e.name]));
const out: string[] = [];
for (const spec of existingSpecs) {
for (const [pkg, name] of packageToName) {
if (spec.includes(pkg)) {
out.push(name);
break;
}
}
}
return out;
}
/**
* Build the `--mcp` spec string for a catalog entry. Same format
* `mcpCommandFor` produces for `reasonix mcp list`, minus the leading
* `--mcp "..."` wrapper — we store the inner spec directly.
*/
export function buildSpec(name: string, argsByName: Record<string, string>): string {
const entry = CATALOG_BY_NAME.get(name);
if (!entry) return name; // shouldn't happen; fall back gracefully
const userArg = entry.userArgs ? argsByName[name] : undefined;
const tail = userArg ? ` ${quoteIfNeeded(userArg)}` : "";
return `${entry.name}=npx -y ${entry.package}${tail}`;
}
function quoteIfNeeded(s: string): string {
return /\s|"/.test(s) ? `"${s.replace(/"/g, '\\"')}"` : s;
}
+454
View File
@@ -0,0 +1,454 @@
/**
* Minimal Markdown → Ink renderer for chat output.
*
* Handles the subset that actually shows up in LLM answers:
* - ATX headers (# ##)
* - Unordered / ordered lists
* - Fenced code blocks (```lang)
* - Inline **bold**, *italic*, `code`
* - Paragraphs separated by blank lines
* - LaTeX delimiters are stripped (\( \), \[ \], \boxed{X})
*
* The goal is not TeX-perfect math — it's "stop showing raw backslashes to
* the user." When the model insists on LaTeX, we strip the scaffolding and
* show the expression verbatim; terminals don't do math fonts anyway.
*/
import { Box, Text } from "ink";
import React from "react";
const SUPERSCRIPT: Record<string, string> = {
"0": "⁰",
"1": "¹",
"2": "²",
"3": "³",
"4": "⁴",
"5": "⁵",
"6": "⁶",
"7": "⁷",
"8": "⁸",
"9": "⁹",
"+": "⁺",
"-": "⁻",
n: "ⁿ",
};
const SUBSCRIPT: Record<string, string> = {
"0": "₀",
"1": "₁",
"2": "₂",
"3": "₃",
"4": "₄",
"5": "₅",
"6": "₆",
"7": "₇",
"8": "₈",
"9": "₉",
"+": "₊",
"-": "₋",
};
function toSuperscript(s: string): string {
let out = "";
for (const c of s) out += SUPERSCRIPT[c] ?? c;
return out;
}
function toSubscript(s: string): string {
let out = "";
for (const c of s) out += SUBSCRIPT[c] ?? c;
return out;
}
export function stripMath(s: string): string {
return (
s
// Delimiters
.replace(/\\\(\s*/g, "")
.replace(/\s*\\\)/g, "")
.replace(/\\\[\s*/g, "\n")
.replace(/\s*\\\]/g, "\n")
// Fractions — \frac, \dfrac, \tfrac. Allow whitespace and one nesting
// level inside braces (e.g. \frac{\sqrt{2}}{3}). Trim captured groups
// so '\frac{ a }{ b }' renders as '(a)/(b)'.
.replace(
/\\[dt]?frac\s*\{((?:[^{}]|\{[^{}]*\})+)\}\s*\{((?:[^{}]|\{[^{}]*\})+)\}/g,
(_m, num: string, den: string) => `(${num.trim()})/(${den.trim()})`,
)
.replace(
/\\binom\s*\{([^{}]+)\}\s*\{([^{}]+)\}/g,
(_m, n: string, k: string) => `C(${n.trim()},${k.trim()})`,
)
.replace(/\\sqrt\s*\{([^{}]+)\}/g, (_m, g: string) => `√(${g.trim()})`)
.replace(/\\boxed\s*\{([^{}]+)\}/g, (_m, g: string) => `【${g.trim()}】`)
.replace(/\\text\s*\{([^{}]+)\}/g, (_m, g: string) => g.trim())
.replace(/\\overline\s*\{([^{}]+)\}/g, (_m, g: string) => `${g.trim()}̄`)
.replace(/\\hat\s*\{([^{}]+)\}/g, (_m, g: string) => `${g.trim()}̂`)
.replace(/\\vec\s*\{([^{}]+)\}/g, (_m, g: string) => `→${g.trim()}`)
// Operators & symbols
.replace(/\\cdot/g, "·")
.replace(/\\times/g, "×")
.replace(/\\div/g, "÷")
.replace(/\\pm/g, "±")
.replace(/\\mp/g, "∓")
.replace(/\\leq/g, "≤")
.replace(/\\geq/g, "≥")
.replace(/\\neq/g, "≠")
.replace(/\\approx/g, "≈")
.replace(/\\in\b/g, "∈")
.replace(/\\notin\b/g, "∉")
.replace(/\\infty/g, "∞")
.replace(/\\sum\b/g, "Σ")
.replace(/\\prod\b/g, "Π")
.replace(/\\int\b/g, "∫")
// Greek letters
.replace(/\\alpha/g, "α")
.replace(/\\beta/g, "β")
.replace(/\\gamma/g, "γ")
.replace(/\\delta/g, "δ")
.replace(/\\theta/g, "θ")
.replace(/\\lambda/g, "λ")
.replace(/\\mu/g, "μ")
.replace(/\\pi/g, "π")
.replace(/\\sigma/g, "σ")
.replace(/\\phi/g, "φ")
.replace(/\\omega/g, "ω")
// Arrows / logic
.replace(/\\implies\b/g, "⇒")
.replace(/\\iff\b/g, "⇔")
.replace(/\\to\b/g, "→")
.replace(/\\rightarrow/g, "→")
.replace(/\\Rightarrow/g, "⇒")
.replace(/\\leftarrow/g, "←")
.replace(/\\Leftarrow/g, "⇐")
.replace(/\\ldots/g, "…")
.replace(/\\cdots/g, "⋯")
// Spacing commands
.replace(/\\quad/g, " ")
.replace(/\\qquad/g, " ")
.replace(/\\,/g, " ")
.replace(/\\;/g, " ")
.replace(/\\!/g, "")
.replace(/\\\\/g, "\n")
// Superscripts / subscripts — single token or {braced group of [\w+-]}
.replace(/\^\{([\w+-]+)\}/g, (_m, g: string) => toSuperscript(g))
.replace(/\^([0-9+\-n])/g, (_m, g: string) => toSuperscript(g))
.replace(/_\{([\w+-]+)\}/g, (_m, g: string) => toSubscript(g))
.replace(/_([0-9+\-])/g, (_m, g: string) => toSubscript(g))
// Catch-all fallbacks for any LaTeX command we didn't explicitly handle.
// Belt-and-braces: even if the model invents a new \weirdcommand{x}{y},
// we'd rather show '(x)/(y)' or 'x' than a raw backslash.
.replace(/\\[a-zA-Z]+\s*\{([^{}]+)\}\s*\{([^{}]+)\}/g, "($1)/($2)")
.replace(/\\[a-zA-Z]+\s*\{([^{}]+)\}/g, "$1")
.replace(/\\[a-zA-Z]+/g, "")
// Collapse multiple whitespace introduced by the stripping above.
.replace(/[ \t]{2,}/g, " ")
);
}
/** Split a single line into styled segments for bold / italic / inline code. */
const INLINE_RE = /(\*\*([^*\n]+?)\*\*|`([^`\n]+?)`|(?<![*\w])\*([^*\n]+?)\*(?!\w))/g;
function InlineMd({ text }: { text: string }) {
const parts: React.ReactNode[] = [];
let last = 0;
let idx = 0;
for (const m of text.matchAll(INLINE_RE)) {
const start = m.index ?? 0;
if (start > last) {
parts.push(<Text key={`t${idx++}`}>{text.slice(last, start)}</Text>);
}
if (m[2] !== undefined) {
parts.push(
<Text key={`b${idx++}`} bold>
{m[2]}
</Text>,
);
} else if (m[3] !== undefined) {
parts.push(
<Text key={`c${idx++}`} color="yellow">
{m[3]}
</Text>,
);
} else if (m[4] !== undefined) {
parts.push(
<Text key={`i${idx++}`} italic>
{m[4]}
</Text>,
);
}
last = start + m[0].length;
}
if (last < text.length) {
parts.push(<Text key={`t${idx++}`}>{text.slice(last)}</Text>);
}
return <Text>{parts}</Text>;
}
interface ParagraphBlock {
kind: "paragraph";
text: string;
}
interface HeadingBlock {
kind: "heading";
level: number;
text: string;
}
interface BulletBlock {
kind: "bullet";
items: string[];
ordered: boolean;
start: number;
}
interface CodeBlock {
kind: "code";
lang: string;
text: string;
}
interface HrBlock {
kind: "hr";
}
// First-class Aider-style SEARCH/REPLACE block. We detect these at
// parse time instead of routing them through the paragraph / inline
// markdown path because the inline parser would otherwise eat `**`
// inside JSDoc `/** ... *\/` comments and `para.join(" ")` would
// collapse the block's newlines. Rendered as a diff so the user can
// actually read what's about to change.
interface EditBlockView {
kind: "edit-block";
filename: string;
search: string;
replace: string;
}
type Block = ParagraphBlock | HeadingBlock | BulletBlock | CodeBlock | HrBlock | EditBlockView;
export function parseBlocks(raw: string): Block[] {
const lines = raw.split(/\r?\n/);
const out: Block[] = [];
let para: string[] = [];
let inCode = false;
let codeLang = "";
let codeBuf: string[] = [];
let listBuf: BulletBlock | null = null;
const flushPara = () => {
if (para.length) {
out.push({ kind: "paragraph", text: para.join(" ") });
para = [];
}
};
const flushList = () => {
if (listBuf) {
out.push(listBuf);
listBuf = null;
}
};
for (let i = 0; i < lines.length; i++) {
const rawLine = lines[i]!;
const line = rawLine.replace(/\s+$/g, "");
// Detect Aider-style SEARCH/REPLACE block. Matches the preceding
// non-blank line as the filename, then `<<<<<<< SEARCH`, content,
// `=======`, content, `>>>>>>> REPLACE`. We don't do markdown
// inside — neither the paragraph nor inline parsers should touch
// this content.
if (!inCode && /^<{7} SEARCH\s*$/.test(line)) {
// Filename is the previous non-blank line we just pushed to para.
// Pull it back out; if there isn't one, treat as literal text.
const filename = para.pop()?.trim();
if (filename) {
flushPara();
flushList();
let j = i + 1;
const searchLines: string[] = [];
while (j < lines.length && !/^={7}\s*$/.test(lines[j]!)) {
searchLines.push(lines[j]!);
j++;
}
const replaceLines: string[] = [];
let k = j + 1;
while (k < lines.length && !/^>{7} REPLACE\s*$/.test(lines[k]!)) {
replaceLines.push(lines[k]!);
k++;
}
if (j < lines.length && k < lines.length) {
out.push({
kind: "edit-block",
filename,
search: searchLines.join("\n"),
replace: replaceLines.join("\n"),
});
i = k;
continue;
}
// Malformed — no separator or no close. Fall through: put
// the filename back in the paragraph so we don't lose it.
para.push(filename);
}
}
const fence = line.match(/^```(\w*)/);
if (fence) {
if (inCode) {
out.push({ kind: "code", lang: codeLang, text: codeBuf.join("\n") });
codeBuf = [];
codeLang = "";
inCode = false;
} else {
flushPara();
flushList();
inCode = true;
codeLang = fence[1] ?? "";
}
continue;
}
if (inCode) {
codeBuf.push(rawLine);
continue;
}
if (line.trim() === "") {
flushPara();
flushList();
continue;
}
if (/^[-*_]{3,}\s*$/.test(line)) {
flushPara();
flushList();
out.push({ kind: "hr" });
continue;
}
const hm = line.match(/^(#{1,6})\s+(.+)$/);
if (hm) {
flushPara();
flushList();
out.push({ kind: "heading", level: hm[1]!.length, text: hm[2]!.trim() });
continue;
}
const bm = line.match(/^\s*[-*+]\s+(.+)$/);
if (bm) {
flushPara();
if (!listBuf || listBuf.ordered) {
flushList();
listBuf = { kind: "bullet", items: [], ordered: false, start: 1 };
}
listBuf.items.push(bm[1]!);
continue;
}
const om = line.match(/^\s*(\d+)\.\s+(.+)$/);
if (om) {
flushPara();
if (!listBuf || !listBuf.ordered) {
flushList();
listBuf = { kind: "bullet", items: [], ordered: true, start: Number(om[1]) };
}
listBuf.items.push(om[2]!);
continue;
}
flushList();
para.push(line);
}
if (inCode && codeBuf.length) {
out.push({ kind: "code", lang: codeLang, text: codeBuf.join("\n") });
}
flushPara();
flushList();
return out;
}
function BlockView({ block }: { block: Block }) {
switch (block.kind) {
case "heading":
return (
<Text bold color="cyan">
<InlineMd text={block.text} />
</Text>
);
case "paragraph":
return <InlineMd text={block.text} />;
case "bullet":
return (
<Box flexDirection="column">
{block.items.map((item, i) => (
<Box key={`${i}-${item.slice(0, 24)}`}>
<Text color="cyan">{block.ordered ? ` ${block.start + i}. ` : " • "}</Text>
<InlineMd text={item} />
</Box>
))}
</Box>
);
case "code":
return (
<Box borderStyle="single" borderColor="gray" paddingX={1}>
<Text color="yellow">{block.text}</Text>
</Box>
);
case "edit-block":
return <EditBlockRow block={block} />;
case "hr":
return <Text dimColor>{"────────────────────────"}</Text>;
}
}
/**
* SEARCH/REPLACE rendered as a minimal diff: filename on top, red
* `-` lines, a gutter, green `+` lines. No inner markdown / inline
* parsing — content is shown verbatim so JSDoc `/**` and `*` won't
* be eaten by bold/italic regex.
*
* A truly empty SEARCH means "new file" and we label the filename
* accordingly instead of rendering an empty red half.
*/
function EditBlockRow({ block }: { block: EditBlockView }) {
const isNewFile = block.search.length === 0;
const searchLines = block.search.split("\n");
const replaceLines = block.replace.split("\n");
return (
<Box flexDirection="column" borderStyle="round" borderColor="cyan" paddingX={1}>
<Box>
<Text bold color="cyan">
{block.filename}
</Text>
{isNewFile ? (
<Text color="green" bold>
{" (new file)"}
</Text>
) : null}
</Box>
{isNewFile ? null : (
<Box flexDirection="column" marginTop={1}>
{searchLines.map((line, i) => (
<Text key={`s-${i}-${line.length}`} color="red">
{`- ${line}`}
</Text>
))}
</Box>
)}
<Box flexDirection="column" marginTop={isNewFile ? 1 : 0}>
{replaceLines.map((line, i) => (
<Text key={`r-${i}-${line.length}`} color="green">
{`+ ${line}`}
</Text>
))}
</Box>
</Box>
);
}
export function Markdown({ text }: { text: string }) {
const cleaned = stripMath(text);
const blocks = React.useMemo(() => parseBlocks(cleaned), [cleaned]);
return (
<Box flexDirection="column" gap={1}>
{blocks.map((b, i) => (
<BlockView key={`${i}-${b.kind}`} block={b} />
))}
</Box>
);
}
+42
View File
@@ -0,0 +1,42 @@
/**
* One place that defines what each preset means. Both `slash.ts`
* (in-chat `/preset`) and the wizard (first-run setup) read from here.
*
* Presets are the single vocabulary we teach new users: they don't need
* to know model IDs, Pillar 2, branch budgets, or cost tradeoffs
* independently — they pick "fast / smart / max" and we translate.
*/
import type { PresetName } from "../../config.js";
export interface PresetSettings {
model: string;
harvest: boolean;
/** Branch budget. `1` means branching off. */
branch: number;
}
export const PRESETS: Record<PresetName, PresetSettings> = {
fast: { model: "deepseek-chat", harvest: false, branch: 1 },
smart: { model: "deepseek-reasoner", harvest: true, branch: 1 },
max: { model: "deepseek-reasoner", harvest: true, branch: 3 },
};
export const PRESET_DESCRIPTIONS: Record<PresetName, { headline: string; cost: string }> = {
fast: {
headline: "deepseek-chat, no reasoning harvest, no branching",
cost: "~1¢ per 100 turns · default",
},
smart: {
headline: "deepseek-reasoner + Pillar 2 harvest",
cost: "~10× cost vs fast · slower · better on multi-step tasks",
},
max: {
headline: "reasoner + harvest + self-consistency (3 branches)",
cost: "~30× cost vs fast · slowest · for hard single-shots",
},
};
export function resolvePreset(name: PresetName | undefined): PresetSettings {
return PRESETS[name ?? "fast"];
}
+488
View File
@@ -0,0 +1,488 @@
import { spawnSync } from "node:child_process";
import type { CacheFirstLoop } from "../../loop.js";
import { deleteSession, listSessions } from "../../session.js";
import { DEEPSEEK_CONTEXT_TOKENS, DEFAULT_CONTEXT_TOKENS } from "../../telemetry.js";
export interface SlashResult {
/** Text to display back to the user as a system/info line. */
info?: string;
/** Exit the app. */
exit?: boolean;
/** Clear the visible history. */
clear?: boolean;
/** Unknown command — display usage hint. */
unknown?: boolean;
/**
* Re-submit this text as a user message after displaying `info`.
* Used by `/retry` — the slash command truncates the log, then
* asks the TUI to push the original text back through the normal
* submit flow so a fresh turn runs.
*/
resubmit?: string;
}
/**
* Extra runtime context a slash handler may care about but that isn't
* already on the loop. Kept as an optional object so tests that only
* need loop-scoped commands can skip it, and callers only populate the
* slots that apply to their session.
*/
export interface SlashContext {
/**
* The exact `--mcp` / config-derived spec strings that were bridged
* into this session (one entry per server). Used by `/mcp`. Empty or
* omitted → no MCP servers attached.
*/
mcpSpecs?: string[];
/**
* Callback for `/undo` — provided by the TUI when it's running in
* code mode. Returns a human-readable report of what was restored.
* Absent outside code mode → `/undo` replies "not available here".
*/
codeUndo?: () => string;
/**
* Callback for `/apply` — commits pending edit blocks to disk. Returns
* a report of what landed. Absent → `/apply` replies "nothing pending"
* or "not available outside code mode".
*/
codeApply?: () => string;
/**
* Callback for `/discard` — drops the pending edit blocks without
* touching disk.
*/
codeDiscard?: () => string;
/**
* Root directory passed by `reasonix code`. Enables `/commit`, which
* runs `git add -A && git commit` in this directory. Missing → `/commit`
* replies "only available in code mode".
*/
codeRoot?: string;
/**
* How many edit blocks are currently pending `/apply` or `/discard`.
* Surfaced by `/status`. TUI populates it live from its pending ref;
* omitted → treat as 0 (chat-only session).
*/
pendingEditCount?: number;
/**
* Callback returning every tool result seen this session in
* chronological order (oldest first). Powers `/tool [N]` for
* inspecting the full untruncated output that `EventLog` clips at
* 400 chars for display. Absent → `/tool` replies "not available".
*/
toolHistory?: () => Array<{ toolName: string; text: string }>;
}
export function parseSlash(text: string): { cmd: string; args: string[] } | null {
if (!text.startsWith("/")) return null;
const parts = text.slice(1).trim().split(/\s+/);
const cmd = parts[0]?.toLowerCase() ?? "";
if (!cmd) return null;
return { cmd, args: parts.slice(1) };
}
export function handleSlash(
cmd: string,
args: string[],
loop: CacheFirstLoop,
ctx: SlashContext = {},
): SlashResult {
switch (cmd) {
case "exit":
case "quit":
return { exit: true };
case "clear":
return { clear: true };
case "help":
case "?":
return {
info: [
"Commands:",
" /help this message",
" /status show current settings",
" /preset <fast|smart|max> one-tap presets — see below",
" /model <id> deepseek-chat or deepseek-reasoner",
" /harvest [on|off] Pillar 2: structured plan-state extraction",
" /branch <N|off> run N parallel samples (N>=2), pick most confident",
" /mcp list MCP servers + tools attached to this session",
" /setup (exit + reconfigure) → run `reasonix setup`",
" /compact [cap] shrink large tool results in history (default 4k/result)",
" /think dump the most recent turn's full R1 reasoning (reasoner only)",
" /tool [N] list tool calls (or dump full output of #N, 1=most recent)",
" /retry truncate & resend your last message (fresh sample from the model)",
" /apply (code mode) commit the pending edit blocks to disk",
" /discard (code mode) drop pending edits without writing",
" /undo (code mode) roll back the last applied edit batch",
' /commit "msg" (code mode) git add -A && git commit -m "msg"',
" /sessions list saved sessions (current is marked with ▸)",
" /forget delete the current session from disk",
" /clear clear displayed history (log + session kept)",
" /exit quit",
"",
"Presets:",
" fast deepseek-chat no harvest no branch ~1¢/100turns ← default",
" smart reasoner harvest ~10x cost, slower",
" max reasoner harvest branch 3 ~30x cost, slowest",
"",
"Sessions (auto-enabled by default, named 'default'):",
" reasonix chat --session <name> use a different named session",
" reasonix chat --no-session disable persistence for this run",
].join("\n"),
};
case "mcp": {
const specs = ctx.mcpSpecs ?? [];
const toolSpecs = loop.prefix.toolSpecs ?? [];
if (specs.length === 0 && toolSpecs.length === 0) {
return {
info:
"no MCP servers attached. Run `reasonix setup` to pick some, " +
'or launch with --mcp "<spec>". `reasonix mcp list` shows the catalog.',
};
}
const lines: string[] = [];
if (specs.length > 0) {
lines.push(`MCP servers (${specs.length}):`);
for (const spec of specs) lines.push(` · ${spec}`);
lines.push("");
}
if (toolSpecs.length > 0) {
lines.push(`Tools in registry (${toolSpecs.length}):`);
for (const t of toolSpecs) lines.push(` · ${t.function.name}`);
}
lines.push("");
lines.push("To change this set, exit and run `reasonix setup`.");
return { info: lines.join("\n") };
}
case "setup":
return {
info:
"To reconfigure (preset, MCP servers, API key), exit this chat and run " +
"`reasonix setup`. Changes take effect on next launch.",
};
case "retry": {
const prev = loop.retryLastUser();
if (!prev) {
return {
info: "nothing to retry — no prior user message in this session's log.",
};
}
const preview = prev.length > 80 ? `${prev.slice(0, 80)}…` : prev;
return {
info: `▸ retrying: "${preview}"`,
resubmit: prev,
};
}
case "think":
case "reasoning": {
const raw = loop.scratch.reasoning;
if (!raw || !raw.trim()) {
return {
info:
"no reasoning cached. `/think` shows the full R1 thought for the most recent turn — " +
"only `deepseek-reasoner` produces it, and only once the turn completes.",
};
}
return { info: `↳ full thinking (${raw.length} chars):\n\n${raw.trim()}` };
}
case "tool": {
// EventLog truncates tool results at 400 chars for display. When
// the user wants to check what the model actually read (e.g. to
// verify it isn't hallucinating a file's contents), they need
// the full text. `/tool` is the escape hatch.
const history = ctx.toolHistory?.() ?? [];
if (history.length === 0) {
return {
info:
"no tool calls yet in this session. `/tool` lists them once the model has actually " +
"used a tool; `/tool N` dumps the full (untruncated) output of the Nth-most-recent.",
};
}
const raw = (args[0] ?? "").toLowerCase();
if (raw === "" || raw === "list" || raw === "ls") {
return { info: formatToolList(history) };
}
const n = Number.parseInt(raw, 10);
if (!Number.isFinite(n) || n < 1) {
return {
info: "usage: /tool [N] (no arg → list; N=1 → most recent result in full, N=2 → previous, …)",
};
}
if (n > history.length) {
return {
info: `only ${history.length} tool call(s) in history — asked for #${n}. Try /tool with no arg to see the list.`,
};
}
const entry = history[history.length - n];
if (!entry) {
return { info: `could not read tool call #${n}` };
}
return {
info: `↳ tool<${entry.toolName}> #${n} (${entry.text.length} chars):\n\n${entry.text}`,
};
}
case "undo": {
if (!ctx.codeUndo) {
return {
info: "/undo is only available inside `reasonix code` — chat mode doesn't apply edits.",
};
}
return { info: ctx.codeUndo() };
}
case "apply": {
if (!ctx.codeApply) {
return {
info: "/apply is only available inside `reasonix code` (nothing to apply here).",
};
}
return { info: ctx.codeApply() };
}
case "discard": {
if (!ctx.codeDiscard) {
return {
info: "/discard is only available inside `reasonix code`.",
};
}
return { info: ctx.codeDiscard() };
}
case "commit": {
if (!ctx.codeRoot) {
return {
info: "/commit is only available inside `reasonix code` (needs a rooted git repo).",
};
}
// Reassemble the original argv. The parser lowercases cmd but
// leaves args alone, and the TUI splits on whitespace which
// mangles quoted messages — rejoin with spaces and strip a
// surrounding pair of double quotes if the user wrote them.
const raw = args.join(" ").trim();
const message = stripOuterQuotes(raw);
if (!message) {
return {
info: `usage: /commit "your commit message" — runs \`git add -A && git commit -m "…"\` in ${ctx.codeRoot}`,
};
}
return runGitCommit(ctx.codeRoot, message);
}
case "compact": {
// Manual companion to the automatic heal-on-load. Re-applies
// truncation with a tighter cap (4k chars per tool result) and
// rewrites the session file so the shrink persists. Useful when
// the ctx gauge in StatsPanel goes yellow/red mid-session and
// the user wants to keep chatting without /forget'ing everything.
const tight = Number.parseInt(args[0] ?? "", 10);
const cap = Number.isFinite(tight) && tight >= 500 ? tight : 4000;
const { healedCount, charsSaved } = loop.compact(cap);
if (healedCount === 0) {
return {
info: `▸ nothing to compact — no tool result in history exceeds ${cap.toLocaleString()} chars.`,
};
}
return {
info: `▸ compacted ${healedCount} tool result(s), saved ${charsSaved.toLocaleString()} chars (~${Math.round(charsSaved / 4).toLocaleString()} tokens). Session file rewritten.`,
};
}
case "sessions": {
const items = listSessions();
if (items.length === 0) {
return {
info: "no saved sessions yet — chat normally and your messages will be saved automatically",
};
}
const lines = ["Saved sessions:"];
for (const s of items) {
const sizeKb = (s.size / 1024).toFixed(1);
const when = s.mtime.toISOString().replace("T", " ").slice(0, 16);
const marker = s.name === loop.sessionName ? "▸" : " ";
lines.push(
` ${marker} ${s.name.padEnd(22)} ${String(s.messageCount).padStart(5)} msgs ${sizeKb.padStart(7)} KB ${when}`,
);
}
lines.push("");
lines.push("Resume with: reasonix chat --session <name>");
return { info: lines.join("\n") };
}
case "forget": {
if (!loop.sessionName) {
return { info: "not in a session — nothing to forget" };
}
const name = loop.sessionName;
const ok = deleteSession(name);
return {
info: ok
? `▸ deleted session "${name}" — current screen still shows the conversation, but next launch starts fresh`
: `could not delete session "${name}" (already gone?)`,
};
}
case "status": {
const branchBudget = loop.branchOptions.budget ?? 1;
const ctxMax = DEEPSEEK_CONTEXT_TOKENS[loop.model] ?? DEFAULT_CONTEXT_TOKENS;
const lastPromptTokens = loop.stats.summary().lastPromptTokens;
const ctxPct = ctxMax > 0 ? Math.round((lastPromptTokens / ctxMax) * 100) : 0;
const ctxLine =
lastPromptTokens > 0
? ` ctx ${compactNum(lastPromptTokens)}/${compactNum(ctxMax)} (${ctxPct}%)`
: " ctx no turns yet";
const pending = ctx.pendingEditCount ?? 0;
const sessionLine = loop.sessionName
? ` session "${loop.sessionName}" · ${loop.log.length} messages in log (resumed ${loop.resumedMessageCount})`
: " session (ephemeral — no persistence)";
const mcpCount = ctx.mcpSpecs?.length ?? 0;
const toolCount = loop.prefix.toolSpecs.length;
const mcpLine = ` mcp ${mcpCount} server(s), ${toolCount} tool(s) in registry`;
const pendingLine =
pending > 0 ? ` edits ${pending} pending (/apply to commit, /discard to drop)` : "";
const lines = [
` model ${loop.model}`,
` flags harvest=${loop.harvestEnabled ? "on" : "off"} · branch=${branchBudget > 1 ? branchBudget : "off"} · stream=${loop.stream ? "on" : "off"}`,
ctxLine,
mcpLine,
sessionLine,
];
if (pendingLine) lines.push(pendingLine);
return { info: lines.join("\n") };
}
case "model": {
const id = args[0];
if (!id) return { info: "usage: /model <id> (try deepseek-chat or deepseek-reasoner)" };
loop.configure({ model: id });
return { info: `model → ${id}` };
}
case "harvest": {
const arg = (args[0] ?? "").toLowerCase();
const on = arg === "" ? !loop.harvestEnabled : arg === "on" || arg === "true" || arg === "1";
loop.configure({ harvest: on });
return { info: `harvest → ${loop.harvestEnabled ? "on" : "off"}` };
}
case "preset": {
const name = (args[0] ?? "").toLowerCase();
if (name === "fast" || name === "default") {
loop.configure({ model: "deepseek-chat", harvest: false, branch: 1 });
return { info: "preset → fast (deepseek-chat, no harvest, no branch)" };
}
if (name === "smart") {
loop.configure({ model: "deepseek-reasoner", harvest: true, branch: 1 });
return { info: "preset → smart (reasoner + harvest, ~10x cost vs fast)" };
}
if (name === "max" || name === "best") {
loop.configure({ model: "deepseek-reasoner", harvest: true, branch: 3 });
return {
info: "preset → max (reasoner + harvest + branch3, ~30x cost vs fast, slowest)",
};
}
return { info: "usage: /preset <fast|smart|max>" };
}
case "branch": {
const raw = (args[0] ?? "").toLowerCase();
if (raw === "" || raw === "off" || raw === "0" || raw === "1") {
loop.configure({ branch: 1 });
return { info: "branch → off" };
}
const n = Number.parseInt(raw, 10);
if (!Number.isFinite(n) || n < 2) {
return { info: "usage: /branch <N> (N>=2, or 'off')" };
}
if (n > 8) {
return { info: "branch budget capped at 8 to prevent runaway cost" };
}
loop.configure({ branch: n });
return { info: `branch → ${n} (harvest auto-enabled; streaming disabled)` };
}
default:
return { unknown: true, info: `unknown command: /${cmd} (try /help)` };
}
}
function formatToolList(history: Array<{ toolName: string; text: string }>): string {
const total = history.length;
const header = `Tool calls in this session (${total}, most recent first):`;
// Show the 10 most recent. Older ones are rarely what the user
// wants — and the help footer tells them how to reach any entry
// by index if they do.
const shown = Math.min(total, 10);
const lines: string[] = [header];
for (let i = 0; i < shown; i++) {
const entry = history[total - 1 - i];
if (!entry) continue;
const idx = i + 1; // 1-based from most recent
const flat = entry.text.replace(/\s+/g, " ").trim();
const preview = flat.length > 80 ? `${flat.slice(0, 80)}…` : flat;
const name = entry.toolName.length > 24 ? `${entry.toolName.slice(0, 23)}…` : entry.toolName;
lines.push(
` #${String(idx).padStart(2)} ${name.padEnd(24)} ${String(entry.text.length).padStart(6)} chars ${preview}`,
);
}
if (total > shown) {
lines.push(` … (${total - shown} earlier, reach with /tool N)`);
}
lines.push("");
lines.push("View full output: /tool N (N=1 → most recent)");
return lines.join("\n");
}
function compactNum(n: number): string {
if (n < 1000) return String(n);
const k = n / 1000;
return k >= 100 ? `${Math.round(k)}k` : `${k.toFixed(1)}k`;
}
function stripOuterQuotes(s: string): string {
if (s.length >= 2 && s.startsWith('"') && s.endsWith('"')) {
return s.slice(1, -1);
}
return s;
}
/**
* Run `git add -A` then `git commit -m <message>` in `rootDir`. Returns
* a SlashResult with a human-scannable info line. We surface stderr on
* failure so the user sees exactly what git complained about (bad
* config, pre-commit hook rejection, nothing staged, etc.).
*/
function runGitCommit(rootDir: string, message: string): SlashResult {
const add = spawnSync("git", ["add", "-A"], { cwd: rootDir, encoding: "utf8" });
if (add.error || add.status !== 0) {
return { info: `git add failed (${add.status ?? "?"}):\n${gitTail(add)}` };
}
const commit = spawnSync("git", ["commit", "-m", message], {
cwd: rootDir,
encoding: "utf8",
});
if (commit.error || commit.status !== 0) {
return { info: `git commit failed (${commit.status ?? "?"}):\n${gitTail(commit)}` };
}
const firstLine = (commit.stdout || "").split(/\r?\n/)[0] ?? "";
return { info: `▸ committed: ${message}${firstLine ? `\n ${firstLine}` : ""}` };
}
/**
* Safely extract whatever diagnostic text is available from a spawnSync
* result — on Windows or when cwd doesn't exist, `stderr`/`stdout` can
* be `undefined` and the caller has only `error.message` to go on.
*/
function gitTail(res: ReturnType<typeof spawnSync>): string {
const stderr = (res.stderr as string | undefined) ?? "";
const stdout = (res.stdout as string | undefined) ?? "";
const body = stderr.trim() || stdout.trim();
if (body) return body;
if (res.error) return (res.error as Error).message;
return "(no output from git)";
}
+225
View File
@@ -0,0 +1,225 @@
import { type EventSourceMessage, createParser } from "eventsource-parser";
import { type RetryOptions, fetchWithRetry } from "./retry.js";
import type { ChatMessage, ChatRequestOptions, RawUsage, ToolCall, ToolSpec } from "./types.js";
export class Usage {
constructor(
public promptTokens = 0,
public completionTokens = 0,
public totalTokens = 0,
public promptCacheHitTokens = 0,
public promptCacheMissTokens = 0,
) {}
get cacheHitRatio(): number {
const denom = this.promptCacheHitTokens + this.promptCacheMissTokens;
return denom > 0 ? this.promptCacheHitTokens / denom : 0;
}
static fromApi(raw: RawUsage | undefined | null): Usage {
const u = raw ?? {};
return new Usage(
u.prompt_tokens ?? 0,
u.completion_tokens ?? 0,
u.total_tokens ?? 0,
u.prompt_cache_hit_tokens ?? 0,
u.prompt_cache_miss_tokens ?? 0,
);
}
}
export interface ChatResponse {
content: string;
reasoningContent: string | null;
toolCalls: ToolCall[];
usage: Usage;
raw: unknown;
}
export interface StreamChunk {
contentDelta?: string;
reasoningDelta?: string;
toolCallDelta?: { index: number; id?: string; name?: string; argumentsDelta?: string };
usage?: Usage;
finishReason?: string;
raw: any;
}
export interface DeepSeekClientOptions {
apiKey?: string;
baseUrl?: string;
timeoutMs?: number;
fetch?: typeof fetch;
/** Retry configuration. Pass `{ maxAttempts: 1 }` to disable retries. */
retry?: RetryOptions;
}
export class DeepSeekClient {
readonly apiKey: string;
readonly baseUrl: string;
readonly timeoutMs: number;
readonly retry: RetryOptions;
private readonly _fetch: typeof fetch;
constructor(opts: DeepSeekClientOptions = {}) {
const apiKey = opts.apiKey ?? process.env.DEEPSEEK_API_KEY;
if (!apiKey) {
throw new Error(
"DEEPSEEK_API_KEY is not set. Put it in .env or pass apiKey to DeepSeekClient.",
);
}
this.apiKey = apiKey;
this.baseUrl = (
opts.baseUrl ??
process.env.DEEPSEEK_BASE_URL ??
"https://api.deepseek.com"
).replace(/\/+$/, "");
this.timeoutMs = opts.timeoutMs ?? 120_000;
this._fetch = opts.fetch ?? globalThis.fetch.bind(globalThis);
this.retry = opts.retry ?? {};
}
private buildPayload(opts: ChatRequestOptions, stream: boolean) {
const payload: Record<string, unknown> = {
model: opts.model,
messages: opts.messages,
stream,
};
if (opts.tools?.length) payload.tools = opts.tools;
if (opts.temperature !== undefined) payload.temperature = opts.temperature;
if (opts.maxTokens !== undefined) payload.max_tokens = opts.maxTokens;
if (opts.responseFormat) payload.response_format = opts.responseFormat;
return payload;
}
async chat(opts: ChatRequestOptions): Promise<ChatResponse> {
const ctrl = new AbortController();
const timer = setTimeout(() => ctrl.abort(), this.timeoutMs);
const signal = opts.signal ?? ctrl.signal;
try {
const resp = await fetchWithRetry(
this._fetch,
`${this.baseUrl}/chat/completions`,
{
method: "POST",
headers: {
Authorization: `Bearer ${this.apiKey}`,
"Content-Type": "application/json",
},
body: JSON.stringify(this.buildPayload(opts, false)),
signal,
},
{ ...this.retry, signal },
);
if (!resp.ok) {
throw new Error(`DeepSeek ${resp.status}: ${await resp.text()}`);
}
const data: any = await resp.json();
const choice = data.choices?.[0]?.message ?? {};
return {
content: choice.content ?? "",
reasoningContent: choice.reasoning_content ?? null,
toolCalls: choice.tool_calls ?? [],
usage: Usage.fromApi(data.usage),
raw: data,
};
} finally {
clearTimeout(timer);
}
}
async *stream(opts: ChatRequestOptions): AsyncGenerator<StreamChunk> {
const ctrl = new AbortController();
const timer = setTimeout(() => ctrl.abort(), this.timeoutMs);
const signal = opts.signal ?? ctrl.signal;
let resp: Response;
try {
// Only the initial fetch is retried. Once the server has started sending
// the stream body we do NOT retry — a mid-stream retry would re-bill and
// desync the session context.
resp = await fetchWithRetry(
this._fetch,
`${this.baseUrl}/chat/completions`,
{
method: "POST",
headers: {
Authorization: `Bearer ${this.apiKey}`,
"Content-Type": "application/json",
Accept: "text/event-stream",
},
body: JSON.stringify(this.buildPayload(opts, true)),
signal,
},
{ ...this.retry, signal },
);
} catch (err) {
clearTimeout(timer);
throw err;
}
if (!resp.ok || !resp.body) {
clearTimeout(timer);
throw new Error(`DeepSeek ${resp.status}: ${await resp.text().catch(() => "")}`);
}
const queue: StreamChunk[] = [];
let done = false;
const parser = createParser({
onEvent: (ev: EventSourceMessage) => {
if (!ev.data || ev.data === "[DONE]") {
done = true;
return;
}
try {
const json = JSON.parse(ev.data);
const delta = json.choices?.[0]?.delta ?? {};
const finishReason = json.choices?.[0]?.finish_reason ?? undefined;
const chunk: StreamChunk = { raw: json, finishReason };
if (typeof delta.content === "string" && delta.content.length > 0) {
chunk.contentDelta = delta.content;
}
if (typeof delta.reasoning_content === "string" && delta.reasoning_content.length > 0) {
chunk.reasoningDelta = delta.reasoning_content;
}
if (Array.isArray(delta.tool_calls) && delta.tool_calls.length > 0) {
const tc = delta.tool_calls[0];
chunk.toolCallDelta = {
index: tc.index ?? 0,
id: tc.id,
name: tc.function?.name,
argumentsDelta: tc.function?.arguments,
};
}
if (json.usage) {
chunk.usage = Usage.fromApi(json.usage);
}
queue.push(chunk);
} catch {
/* skip malformed sse frame */
}
},
});
const reader = resp.body.getReader();
const decoder = new TextDecoder();
try {
while (true) {
if (queue.length > 0) {
yield queue.shift()!;
continue;
}
if (done) break;
const { value, done: streamDone } = await reader.read();
if (streamDone) break;
parser.feed(decoder.decode(value, { stream: true }));
}
while (queue.length > 0) yield queue.shift()!;
} finally {
clearTimeout(timer);
reader.releaseLock();
}
}
}
export type { ChatMessage, ToolCall, ToolSpec };
+241
View File
@@ -0,0 +1,241 @@
/**
* Aider-style SEARCH/REPLACE edit blocks.
*
* The model emits blocks in this exact shape, one or more per response:
*
* path/to/file.ts
* <<<<<<< SEARCH
* exact existing lines (whitespace-sensitive)
* =======
* replacement lines
* >>>>>>> REPLACE
*
* We chose this over unified diffs because:
* - Models produce it reliably — no line-number drift.
* - It tolerates multi-edit responses without ambiguity over which
* hunk belongs to which file.
* - Aider has years of evidence that this format works even against
* weaker models than DeepSeek R1, so it's a conservative pick.
*
* The SEARCH text must match the file byte-for-byte. Empty SEARCH is a
* sentinel for "create new file" — the REPLACE becomes the whole file.
* If SEARCH doesn't match we refuse the edit and surface the failure;
* we do NOT guess or fuzzy-match. A wrong silent edit is worse than a
* missing one — the user can re-ask with the exact current content.
*/
import { existsSync, mkdirSync, readFileSync, unlinkSync, writeFileSync } from "node:fs";
import { dirname, resolve } from "node:path";
export interface EditBlock {
/** Path as written by the model — relative to rootDir, or absolute. */
path: string;
/** Literal text to match in the target file. Empty → create new file. */
search: string;
/** Replacement text to write in place of `search`. */
replace: string;
/** Char offset in the source message where this block started. */
offset: number;
}
export type ApplyStatus =
/** Edit landed on disk. */
| "applied"
/** New file created (SEARCH was empty and file didn't exist). */
| "created"
/** File exists but SEARCH block wasn't found in its content. */
| "not-found"
/** File doesn't exist and SEARCH was non-empty (can't create without content). */
| "file-missing"
/** Path escapes rootDir — refused on safety grounds. */
| "path-escape"
/** fs write / read threw. */
| "error";
export interface ApplyResult {
path: string;
status: ApplyStatus;
/** Extra detail (e.g. error message) for logs. */
message?: string;
}
/**
* One edit block per match. The regex is anchored to the 7-char marker
* lines because those are visually distinct and unlikely to appear in
* normal code.
*
* Anchored with `^` + `m` flag so the filename has to live on its own
* line. Keeps us from matching e.g. a JS-import string that happens to
* contain `<<<<<<< SEARCH` in inner text.
*/
// `\n?` before the =======/REPLACE separators makes the body optional:
// empty SEARCH (new-file sentinel) works without requiring a gratuitous
// empty line, and the same holds for empty REPLACE (file-deletion
// semantics, not yet supported but cheaply representable).
const BLOCK_RE = /^(\S[^\n]*)\n<{7} SEARCH\n([\s\S]*?)\n?={7}\n([\s\S]*?)\n?>{7} REPLACE/gm;
export function parseEditBlocks(text: string): EditBlock[] {
const out: EditBlock[] = [];
BLOCK_RE.lastIndex = 0;
let m: RegExpExecArray | null = BLOCK_RE.exec(text);
while (m !== null) {
out.push({
path: m[1]!.trim(),
search: m[2]!,
replace: m[3]!,
offset: m.index,
});
m = BLOCK_RE.exec(text);
}
return out;
}
export function applyEditBlock(block: EditBlock, rootDir: string): ApplyResult {
const absRoot = resolve(rootDir);
const absTarget = resolve(absRoot, block.path);
// Refuse paths that escape rootDir. `resolve` normalizes `..`, so
// startsWith on the normalized pair is enough.
if (absTarget !== absRoot && !absTarget.startsWith(`${absRoot}${sep()}`)) {
return {
path: block.path,
status: "path-escape",
message: `resolved path ${absTarget} is outside rootDir ${absRoot}`,
};
}
const searchEmpty = block.search.length === 0;
const exists = existsSync(absTarget);
try {
if (!exists) {
if (!searchEmpty) {
return {
path: block.path,
status: "file-missing",
message: "file does not exist; to create it, use an empty SEARCH block",
};
}
mkdirSync(dirname(absTarget), { recursive: true });
writeFileSync(absTarget, block.replace, "utf8");
return { path: block.path, status: "created" };
}
const content = readFileSync(absTarget, "utf8");
if (searchEmpty) {
return {
path: block.path,
status: "not-found",
message: "empty SEARCH only creates new files — this file already exists",
};
}
const idx = content.indexOf(block.search);
if (idx === -1) {
return {
path: block.path,
status: "not-found",
message: "SEARCH text does not match the current file content exactly",
};
}
// Replace only the first occurrence — if the model needs multiple
// identical edits it should emit multiple blocks (each anchored by
// more surrounding context). Auto-expanding to replace-all is a
// footgun when the same string legitimately appears in several
// unrelated places.
const replaced = `${content.slice(0, idx)}${block.replace}${content.slice(idx + block.search.length)}`;
writeFileSync(absTarget, replaced, "utf8");
return { path: block.path, status: "applied" };
} catch (err) {
return { path: block.path, status: "error", message: (err as Error).message };
}
}
export function applyEditBlocks(blocks: EditBlock[], rootDir: string): ApplyResult[] {
return blocks.map((b) => applyEditBlock(b, rootDir));
}
// ---------- snapshot / restore (for /undo) ----------
export interface EditSnapshot {
/** Path relative to rootDir, as the block named it. */
path: string;
/**
* File content before the edit batch was applied. `null` means the
* file didn't exist yet — restoring that means deleting whatever the
* edit created.
*/
prevContent: string | null;
}
/**
* Capture the current state of every file an edit batch is about to
* touch, so `/undo` can roll back if the user doesn't like the result.
* De-duplicates by path because one batch can contain multiple blocks
* for the same file, and we only want one "before" snapshot per file.
*/
export function snapshotBeforeEdits(blocks: EditBlock[], rootDir: string): EditSnapshot[] {
const absRoot = resolve(rootDir);
const seen = new Set<string>();
const snapshots: EditSnapshot[] = [];
for (const b of blocks) {
if (seen.has(b.path)) continue;
seen.add(b.path);
const abs = resolve(absRoot, b.path);
if (!existsSync(abs)) {
snapshots.push({ path: b.path, prevContent: null });
continue;
}
try {
snapshots.push({ path: b.path, prevContent: readFileSync(abs, "utf8") });
} catch {
// Unreadable (permission / binary) — record null so we at least
// don't pretend the snapshot is authoritative. The restore path
// will treat null as "delete on undo", which is wrong in that
// case but the file wasn't ours to begin with.
snapshots.push({ path: b.path, prevContent: null });
}
}
return snapshots;
}
/**
* Restore files to their snapshotted state. Snapshots with
* `prevContent === null` were created by the edit, so undo = delete.
* Otherwise the prior content is written back, replacing whatever the
* edit left behind.
*/
export function restoreSnapshots(snapshots: EditSnapshot[], rootDir: string): ApplyResult[] {
const absRoot = resolve(rootDir);
return snapshots.map((snap) => {
const abs = resolve(absRoot, snap.path);
if (abs !== absRoot && !abs.startsWith(`${absRoot}${sep()}`)) {
return {
path: snap.path,
status: "path-escape",
message: "snapshot path escapes rootDir — refusing to restore",
};
}
try {
if (snap.prevContent === null) {
if (existsSync(abs)) unlinkSync(abs);
return {
path: snap.path,
status: "applied",
message: "removed (the edit had created it)",
};
}
writeFileSync(abs, snap.prevContent, "utf8");
return {
path: snap.path,
status: "applied",
message: "restored to pre-edit content",
};
} catch (err) {
return { path: snap.path, status: "error", message: (err as Error).message };
}
});
}
/** Platform separator — `\` on Windows, `/` elsewhere. */
function sep(): string {
return process.platform === "win32" ? "\\" : "/";
}
+99
View File
@@ -0,0 +1,99 @@
/**
* System prompt used by `reasonix code`. Teaches the model:
*
* 1. It has a filesystem MCP bridge rooted at the user's CWD.
* 2. To modify files it emits SEARCH/REPLACE blocks (not
* `write_file` — that would whole-file rewrite and kill diff
* reviewability).
* 3. Read first, edit second — SEARCH must match byte-for-byte.
* 4. Be concise. The user can read a diff faster than prose.
*
* Kept short on purpose. Long system prompts eat context budget that
* the Cache-First Loop is trying to conserve. The SEARCH/REPLACE spec
* is the one unavoidable bloat; we trim everything else.
*/
import { existsSync, readFileSync } from "node:fs";
import { join } from "node:path";
export const CODE_SYSTEM_PROMPT = `You are Reasonix Code, a coding assistant. You have filesystem tools (read_file, write_file, list_directory, search_files, etc.) rooted at the user's working directory.
# When to edit vs. when to explore
Only propose edits when the user explicitly asks you to change, fix, add, remove, refactor, or write something. Do NOT propose edits when the user asks you to:
- analyze, read, explore, describe, or summarize a project
- explain how something works
- answer a question about the code
In those cases, use tools to gather what you need, then reply in prose. No SEARCH/REPLACE blocks, no file changes. If you're unsure what the user wants, ask.
When you do propose edits, the user will review them and decide whether to \`/apply\` or \`/discard\`. Don't assume they'll accept — write as if each edit will be audited, because it will.
# Editing files
When you've been asked to change a file, output one or more SEARCH/REPLACE blocks in this exact format:
path/to/file.ext
<<<<<<< SEARCH
exact existing lines from the file, including whitespace
=======
the new lines
>>>>>>> REPLACE
Rules:
- Always read_file first so your SEARCH matches byte-for-byte. If it doesn't match, the edit is rejected and you'll have to retry with the exact current content.
- One edit per block. Multiple blocks in one response are fine.
- To create a new file, leave SEARCH empty:
path/to/new.ts
<<<<<<< SEARCH
=======
(whole file content here)
>>>>>>> REPLACE
- Do NOT use write_file to change existing files — the user reviews your edits as SEARCH/REPLACE. write_file is only for files you explicitly want to overwrite wholesale (rare).
- Paths are relative to the working directory. Don't use absolute paths.
# Exploration
- Avoid listing or reading inside these common dependency / build directories unless the user explicitly asks about them: node_modules, dist, build, out, .next, .nuxt, .svelte-kit, .git, .venv, venv, __pycache__, target, coverage, .turbo, .cache. They're expensive and usually irrelevant.
- Prefer search_files / grep over list_directory when you know roughly what you're looking for — it saves context and avoids enumerating huge trees.
# Style
- Show edits; don't narrate them in prose. "Here's the fix:" is enough.
- One short paragraph explaining *why*, then the blocks.
- If you need to explore first (list / grep / read), do it with tool calls before writing any prose — silence while exploring is fine.
`;
/**
* Inject the project's `.gitignore` content into the system prompt as a
* "respect this on top of the built-in denylist" hint. We don't parse
* the file — we hand it to the model as-is. Truncate long ones so we
* don't eat context budget on huge generated ignore lists.
*
* Missing or unreadable .gitignore → returns the base prompt unchanged.
*/
export function codeSystemPrompt(rootDir: string): string {
const gitignorePath = join(rootDir, ".gitignore");
if (!existsSync(gitignorePath)) return CODE_SYSTEM_PROMPT;
let content: string;
try {
content = readFileSync(gitignorePath, "utf8");
} catch {
return CODE_SYSTEM_PROMPT;
}
const MAX = 2000;
const truncated =
content.length > MAX
? `${content.slice(0, MAX)}\n… (truncated ${content.length - MAX} chars)`
: content;
return `${CODE_SYSTEM_PROMPT}
# Project .gitignore
The user's repo ships this .gitignore — treat every pattern as "don't traverse or edit inside these paths unless explicitly asked":
\`\`\`
${truncated}
\`\`\`
`;
}
+97
View File
@@ -0,0 +1,97 @@
/**
* User-level config storage for the Reasonix CLI.
*
* Lookup order for the API key:
* 1. `DEEPSEEK_API_KEY` env var (highest priority — for CI / power users)
* 2. `~/.reasonix/config.json` (set by the first-run setup flow)
*
* The library itself never touches the config file — it only reads
* `DEEPSEEK_API_KEY` from the environment. The CLI is responsible for
* pulling from the config file and exposing it via env var to the loop.
*
* Beyond the API key, the config also remembers the user's *defaults*
* from `reasonix setup`: preset, MCP servers, session. This is what
* makes `reasonix chat` with no flags "just work" after first-run.
*/
import { chmodSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { homedir } from "node:os";
import { dirname, join } from "node:path";
/** One of the preset bundles (model + harvest + branch combo). */
export type PresetName = "fast" | "smart" | "max";
export interface ReasonixConfig {
apiKey?: string;
baseUrl?: string;
/**
* Default preset for `reasonix chat` / `reasonix run` when no flags override.
* Maps to model + harvest + branch combos (see presets.ts). Missing → "fast".
*/
preset?: PresetName;
/**
* Default MCP server specs to bridge on every `reasonix chat`, in the
* same `"name=cmd args..."` format that `--mcp` takes. Stored as strings
* so `reasonix setup` stays symmetrical with the flag — one parser, one
* format in the config file, grep-friendly.
*/
mcp?: string[];
/**
* Default session name (null/missing → "default", which is what the
* CLI has been doing anyway). `reasonix setup` lets users pick a name
* or opt into ephemeral.
*/
session?: string | null;
/** Marks that `reasonix setup` has completed at least once. */
setupCompleted?: boolean;
}
export function defaultConfigPath(): string {
return join(homedir(), ".reasonix", "config.json");
}
export function readConfig(path: string = defaultConfigPath()): ReasonixConfig {
try {
const raw = readFileSync(path, "utf8");
const parsed = JSON.parse(raw);
if (parsed && typeof parsed === "object") return parsed as ReasonixConfig;
} catch {
/* missing or malformed → empty config */
}
return {};
}
export function writeConfig(cfg: ReasonixConfig, path: string = defaultConfigPath()): void {
mkdirSync(dirname(path), { recursive: true });
writeFileSync(path, JSON.stringify(cfg, null, 2), "utf8");
// Restrict permissions on Unix; chmod is a no-op on Windows but won't throw.
try {
chmodSync(path, 0o600);
} catch {
/* ignore on platforms without chmod */
}
}
/** Resolve the API key from env var first, then the config file. */
export function loadApiKey(path: string = defaultConfigPath()): string | undefined {
if (process.env.DEEPSEEK_API_KEY) return process.env.DEEPSEEK_API_KEY;
return readConfig(path).apiKey;
}
export function saveApiKey(key: string, path: string = defaultConfigPath()): void {
const cfg = readConfig(path);
cfg.apiKey = key.trim();
writeConfig(cfg, path);
}
export function isPlausibleKey(key: string): boolean {
const trimmed = key.trim();
return /^sk-[A-Za-z0-9_-]{16,}$/.test(trimmed);
}
/** Mask a key for display: `sk-abcd...wxyz`. */
export function redactKey(key: string): string {
if (!key) return "";
if (key.length <= 12) return "****";
return `${key.slice(0, 6)}…${key.slice(-4)}`;
}
+119
View File
@@ -0,0 +1,119 @@
/**
* Self-consistency branching.
*
* When enabled, the loop fans out into N parallel samples per turn (varied
* temperatures), runs Pillar 2 harvest on each, and selects the sample with
* the fewest flagged uncertainties (ties broken by answer length — a crude
* Occam prior).
*
* The unique opportunity here: because DeepSeek is ~20× cheaper than Claude,
* running N=3–5 samples per turn is still cheaper than a single Claude call,
* while the majority-confidence selection tends to dominate single-sample
* answers on fuzzy multi-step reasoning tasks.
*/
import type { ChatResponse, DeepSeekClient } from "./client.js";
import { type HarvestOptions, type TypedPlanState, harvest } from "./harvest.js";
import type { ChatRequestOptions } from "./types.js";
export interface BranchSample {
index: number;
temperature: number;
response: ChatResponse;
planState: TypedPlanState;
}
export type BranchSelector = (samples: BranchSample[]) => BranchSample;
export interface BranchOptions {
/** Number of parallel samples. 1 disables branching. Default 1. */
budget?: number;
/** Temperatures for each branch. Default spreads across [0, 1]. */
temperatures?: readonly number[];
/** Harvest options; the selector needs harvest to score samples. */
harvestOptions?: HarvestOptions;
/** Custom selector. Default: min uncertainties, tie-break shortest answer. */
selector?: BranchSelector;
/**
* Fires as each sample finishes (main call + harvest both complete).
* Useful for progress UI. Not awaited; exceptions are swallowed.
*/
onSampleDone?: (sample: BranchSample) => void;
}
export interface BranchResult {
chosen: BranchSample;
samples: BranchSample[];
}
/** Default: fewest uncertainties wins, ties broken by shorter answer content. */
export const defaultSelector: BranchSelector = (samples) => {
if (samples.length === 0) throw new Error("defaultSelector: samples is empty");
return samples.slice().sort((a, b) => {
const uDiff = a.planState.uncertainties.length - b.planState.uncertainties.length;
if (uDiff !== 0) return uDiff;
const aLen = a.response.content?.length ?? 0;
const bLen = b.response.content?.length ?? 0;
return aLen - bLen;
})[0]!;
};
export async function runBranches(
client: DeepSeekClient,
request: ChatRequestOptions,
opts: BranchOptions = {},
): Promise<BranchResult> {
const budget = Math.max(1, opts.budget ?? 1);
const temperatures = resolveTemperatures(budget, opts.temperatures);
const selector = opts.selector ?? defaultSelector;
const samples = await Promise.all(
temperatures.map(async (temperature, index): Promise<BranchSample> => {
const response = await client.chat({ ...request, temperature });
const planState = await harvest(response.reasoningContent, client, opts.harvestOptions);
const sample: BranchSample = { index, temperature, response, planState };
try {
opts.onSampleDone?.(sample);
} catch {
/* callback errors must not poison the await */
}
return sample;
}),
);
return { chosen: selector(samples), samples };
}
/** Sum usage across branch samples for telemetry purposes. */
export function aggregateBranchUsage(samples: readonly BranchSample[]) {
let promptTokens = 0;
let completionTokens = 0;
let totalTokens = 0;
let promptCacheHitTokens = 0;
let promptCacheMissTokens = 0;
for (const s of samples) {
promptTokens += s.response.usage.promptTokens;
completionTokens += s.response.usage.completionTokens;
totalTokens += s.response.usage.totalTokens;
promptCacheHitTokens += s.response.usage.promptCacheHitTokens;
promptCacheMissTokens += s.response.usage.promptCacheMissTokens;
}
return {
promptTokens,
completionTokens,
totalTokens,
promptCacheHitTokens,
promptCacheMissTokens,
};
}
function resolveTemperatures(budget: number, custom?: readonly number[]): number[] {
if (custom && custom.length >= budget) return [...custom.slice(0, budget)];
// Spread evenly across [0, 1] to encourage reasoning-path diversity.
if (budget === 1) return [0];
const out: number[] = [];
for (let i = 0; i < budget; i++) {
out.push(Number((i / (budget - 1)).toFixed(2)));
}
return out;
}
+484
View File
@@ -0,0 +1,484 @@
/**
* Diff — compare two transcripts and produce a summary + divergence report.
*
* Two transcripts are "comparable" when they stem from the same task (or
* the same user prompt). Alignment is by turn number: assistant_final #N
* in A pairs with assistant_final #N in B. If one side ran more turns, the
* extras are labeled "only in A" / "only in B".
*
* What we compute:
* - Aggregate deltas: turns, tool calls, cache hit, cost, token counts
* - First divergence: the lowest turn where A and B's tool calls or
* assistant text differ meaningfully
* - Prefix-stability story: how many unique prefix hashes each side used
*
* Non-goals (deliberately):
* - LLM-judge quality comparison
* - Per-token delta rendering — not useful at the fidelity we're at
* - Embedding similarity — Levenshtein ratio is cheap and good enough
*/
import { type ReplayStats, computeReplayStats } from "./replay.js";
import type { ReadTranscriptResult, TranscriptRecord } from "./transcript.js";
export interface DiffSide {
label: string;
meta: ReadTranscriptResult["meta"];
records: TranscriptRecord[];
stats: ReplayStats;
}
export interface TurnPair {
turn: number;
aAssistant?: TranscriptRecord;
bAssistant?: TranscriptRecord;
aTools: TranscriptRecord[];
bTools: TranscriptRecord[];
/**
* Classification of the pair:
* "match" — both sides present, text & tool calls within threshold
* "diverge" — both sides present, but text or tool calls differ
* "only_in_a" — assistant_final in A but not B
* "only_in_b" — assistant_final in B but not A
*/
kind: "match" | "diverge" | "only_in_a" | "only_in_b";
/** When kind === "diverge", a short one-liner pointing at what differs. */
divergenceNote?: string;
}
export interface DiffReport {
a: DiffSide;
b: DiffSide;
pairs: TurnPair[];
firstDivergenceTurn: number | null;
}
// ---------- navigation helpers (used by the Ink diff TUI) ----------
/**
* Find the next pair (strictly after `fromIdx`) whose kind is not "match".
* Returns -1 when no later divergence exists. Used by DiffApp's `n` key.
*/
export function findNextDivergence(pairs: TurnPair[], fromIdx: number): number {
for (let i = fromIdx + 1; i < pairs.length; i++) {
if (pairs[i]!.kind !== "match") return i;
}
return -1;
}
/**
* Find the previous pair (strictly before `fromIdx`) whose kind is not
* "match". Returns -1 when no earlier divergence exists. Used by
* DiffApp's `N` / `p` key.
*/
export function findPrevDivergence(pairs: TurnPair[], fromIdx: number): number {
const start = Math.min(fromIdx - 1, pairs.length - 1);
for (let i = start; i >= 0; i--) {
if (pairs[i]!.kind !== "match") return i;
}
return -1;
}
export function diffTranscripts(
a: { label: string; parsed: ReadTranscriptResult },
b: { label: string; parsed: ReadTranscriptResult },
): DiffReport {
const aSide: DiffSide = {
label: a.label,
meta: a.parsed.meta,
records: a.parsed.records,
stats: computeReplayStats(a.parsed.records),
};
const bSide: DiffSide = {
label: b.label,
meta: b.parsed.meta,
records: b.parsed.records,
stats: computeReplayStats(b.parsed.records),
};
const aByTurn = groupByTurn(a.parsed.records);
const bByTurn = groupByTurn(b.parsed.records);
const turns = [...new Set([...aByTurn.keys(), ...bByTurn.keys()])].sort((x, y) => x - y);
const pairs: TurnPair[] = [];
let firstDivergenceTurn: number | null = null;
for (const turn of turns) {
const aGroup = aByTurn.get(turn) ?? { assistant: undefined, tools: [] };
const bGroup = bByTurn.get(turn) ?? { assistant: undefined, tools: [] };
const aAssistant = aGroup.assistant;
const bAssistant = bGroup.assistant;
const aTools = aGroup.tools;
const bTools = bGroup.tools;
let kind: TurnPair["kind"];
let divergenceNote: string | undefined;
if (!aAssistant && bAssistant) kind = "only_in_b";
else if (aAssistant && !bAssistant) kind = "only_in_a";
else if (!aAssistant && !bAssistant)
kind = "diverge"; // tool-only turn (rare)
else {
divergenceNote = classifyDivergence(aAssistant!, bAssistant!, aTools, bTools);
kind = divergenceNote ? "diverge" : "match";
}
if (kind !== "match" && firstDivergenceTurn === null) firstDivergenceTurn = turn;
pairs.push({ turn, aAssistant, bAssistant, aTools, bTools, kind, divergenceNote });
}
return { a: aSide, b: bSide, pairs, firstDivergenceTurn };
}
// ---------- divergence classification ----------
/**
* Return a short reason string if two sides meaningfully disagree, or
* undefined if they're close enough to call a match.
*
* Ranking of divergence signals (cheapest first):
* 1. Different set of tool names → clearest diff
* 2. Different tool args for the same tool → second-clearest
* 3. Text similarity below threshold → fuzziest
*/
function classifyDivergence(
a: TranscriptRecord,
b: TranscriptRecord,
aTools: TranscriptRecord[],
bTools: TranscriptRecord[],
): string | undefined {
const aNames = aTools.map((t) => t.tool ?? "").sort();
const bNames = bTools.map((t) => t.tool ?? "").sort();
if (aNames.join(",") !== bNames.join(",")) {
return `tool calls differ: A=[${aNames.join(",") || "—"}] B=[${bNames.join(",") || "—"}]`;
}
// Same tool names — did they pass different args?
for (let i = 0; i < aTools.length; i++) {
const at = aTools[i]!;
const bt = bTools[i]!;
if (at.tool !== bt.tool) continue;
if ((at.args ?? "") !== (bt.args ?? "")) {
return `"${at.tool}" args differ`;
}
}
const simRatio = similarity(a.content, b.content);
if (simRatio < 0.75) return `text similarity ${(simRatio * 100).toFixed(0)}%`;
return undefined;
}
/**
* Normalized Levenshtein similarity ratio in [0, 1]. 1 = identical.
* Early-exits for long strings (> 2000 chars) with a cheap token-overlap
* estimate to keep diff fast on chatty transcripts.
*/
export function similarity(a: string, b: string): number {
if (a === b) return 1;
if (!a && !b) return 1;
if (!a || !b) return 0;
const maxLen = Math.max(a.length, b.length);
if (maxLen > 2000) return tokenOverlap(a, b);
const dist = levenshtein(a, b);
return 1 - dist / maxLen;
}
function tokenOverlap(a: string, b: string): number {
const ta = new Set(a.toLowerCase().split(/\s+/).filter(Boolean));
const tb = new Set(b.toLowerCase().split(/\s+/).filter(Boolean));
if (ta.size === 0 && tb.size === 0) return 1;
let shared = 0;
for (const t of ta) if (tb.has(t)) shared++;
return (2 * shared) / (ta.size + tb.size);
}
function levenshtein(a: string, b: string): number {
const m = a.length;
const n = b.length;
if (m === 0) return n;
if (n === 0) return m;
let prev = new Array(n + 1);
let curr = new Array(n + 1);
for (let j = 0; j <= n; j++) prev[j] = j;
for (let i = 1; i <= m; i++) {
curr[0] = i;
for (let j = 1; j <= n; j++) {
const cost = a[i - 1] === b[j - 1] ? 0 : 1;
curr[j] = Math.min(curr[j - 1] + 1, prev[j] + 1, prev[j - 1] + cost);
}
[prev, curr] = [curr, prev];
}
return prev[n];
}
// ---------- grouping ----------
interface TurnGroup {
assistant?: TranscriptRecord;
tools: TranscriptRecord[];
}
function groupByTurn(records: TranscriptRecord[]): Map<number, TurnGroup> {
const out = new Map<number, TurnGroup>();
for (const rec of records) {
if (rec.role === "user") continue; // user msg is input to the turn, not its output
const g = out.get(rec.turn) ?? { tools: [] };
if (rec.role === "assistant_final") g.assistant = rec;
else if (rec.role === "tool") g.tools.push(rec);
out.set(rec.turn, g);
}
return out;
}
// ---------- rendering ----------
export interface RenderOptions {
/** Monochrome output (for file redirection or piping). Defaults to true. */
monochrome?: boolean;
}
export function renderSummaryTable(report: DiffReport, _opts: RenderOptions = {}): string {
const a = report.a;
const b = report.b;
const lines: string[] = [];
lines.push("Comparing:");
lines.push(` A ${a.label}`);
lines.push(` B ${b.label}`);
lines.push("");
lines.push(row(["", "A", "B", "Δ"], [20, 14, 14, 14]));
lines.push(
row(["─".repeat(20), "─".repeat(14), "─".repeat(14), "─".repeat(14)], [20, 14, 14, 14]),
);
lines.push(statRow("model calls", a.stats.turns, b.stats.turns));
lines.push(statRow("user turns", a.stats.userTurns, b.stats.userTurns));
lines.push(statRow("tool calls", a.stats.toolCalls, b.stats.toolCalls));
lines.push(
row(
[
"cache hit",
`${pct(a.stats.cacheHitRatio)}`,
`${pct(b.stats.cacheHitRatio)}`,
signPct(b.stats.cacheHitRatio - a.stats.cacheHitRatio),
],
[20, 14, 14, 14],
),
);
lines.push(
row(
[
"cost (USD)",
`$${a.stats.totalCostUsd.toFixed(6)}`,
`$${b.stats.totalCostUsd.toFixed(6)}`,
costDelta(a.stats.totalCostUsd, b.stats.totalCostUsd),
],
[20, 14, 14, 14],
),
);
lines.push(statRow("prefix hashes", a.stats.prefixHashes.length, b.stats.prefixHashes.length));
// Harvest signal row — only surface when at least one side carries plan
// state. Keeps no-harvest diffs visually identical to pre-v0.3 output.
if (a.stats.harvestedTurns > 0 || b.stats.harvestedTurns > 0) {
lines.push(
row(
[
"harvest turns",
`${a.stats.harvestedTurns}`,
`${b.stats.harvestedTurns}`,
signed(b.stats.harvestedTurns - a.stats.harvestedTurns),
],
[20, 14, 14, 14],
),
);
lines.push(
row(
[
" subgoals",
`${a.stats.totalSubgoals}`,
`${b.stats.totalSubgoals}`,
signed(b.stats.totalSubgoals - a.stats.totalSubgoals),
],
[20, 14, 14, 14],
),
);
lines.push(
row(
[
" uncertainties",
`${a.stats.totalUncertainties}`,
`${b.stats.totalUncertainties}`,
signed(b.stats.totalUncertainties - a.stats.totalUncertainties),
],
[20, 14, 14, 14],
),
);
}
lines.push("");
// Prefix stability story — the headline finding when comparing bench modes.
const aPrefixStable = a.stats.prefixHashes.length <= 1;
const bPrefixStable = b.stats.prefixHashes.length <= 1;
if (aPrefixStable !== bPrefixStable) {
const stable = aPrefixStable ? "A" : "B";
const churn = aPrefixStable ? "B" : "A";
const churnCount = aPrefixStable ? b.stats.prefixHashes.length : a.stats.prefixHashes.length;
lines.push(
`prefix stability: ${stable} stayed byte-stable across ${Math.max(
a.stats.turns,
b.stats.turns,
)} turns; ${churn} churned ${churnCount} distinct prefixes.`,
);
lines.push("");
} else if (a.stats.prefixHashes[0] && a.stats.prefixHashes[0] === b.stats.prefixHashes[0]) {
lines.push(
`prefix: A and B share the same prefix hash (${a.stats.prefixHashes[0].slice(0, 12)}…) — cache delta is attributable to log stability, not prompt change.`,
);
lines.push("");
}
if (report.firstDivergenceTurn !== null) {
const p = report.pairs.find((p) => p.turn === report.firstDivergenceTurn);
lines.push(
`first divergence: turn ${report.firstDivergenceTurn} — ${p?.divergenceNote ?? "?"}`,
);
if (p?.aAssistant) lines.push(` A → ${truncate(p.aAssistant.content, 100)}`);
if (p?.bAssistant) lines.push(` B → ${truncate(p.bAssistant.content, 100)}`);
} else {
lines.push("no material divergence detected (texts within similarity threshold).");
}
return lines.join("\n");
}
export function renderMarkdown(report: DiffReport): string {
const a = report.a;
const b = report.b;
const out: string[] = [];
out.push(`# Transcript diff: ${a.label} vs ${b.label}`);
out.push("");
if (a.meta || b.meta) {
out.push("## Meta");
out.push("");
out.push(`| | ${a.label} | ${b.label} |`);
out.push("|---|---|---|");
out.push(`| source | ${a.meta?.source ?? "—"} | ${b.meta?.source ?? "—"} |`);
out.push(`| model | ${a.meta?.model ?? "—"} | ${b.meta?.model ?? "—"} |`);
out.push(`| task | ${a.meta?.task ?? "—"} | ${b.meta?.task ?? "—"} |`);
out.push(`| startedAt | ${a.meta?.startedAt ?? "—"} | ${b.meta?.startedAt ?? "—"} |`);
out.push("");
}
out.push("## Summary");
out.push("");
out.push(`| metric | ${a.label} | ${b.label} | delta |`);
out.push("|---|---:|---:|---:|");
out.push(
`| model calls | ${a.stats.turns} | ${b.stats.turns} | ${signed(b.stats.turns - a.stats.turns)} |`,
);
out.push(
`| user turns | ${a.stats.userTurns} | ${b.stats.userTurns} | ${signed(b.stats.userTurns - a.stats.userTurns)} |`,
);
out.push(
`| tool calls | ${a.stats.toolCalls} | ${b.stats.toolCalls} | ${signed(b.stats.toolCalls - a.stats.toolCalls)} |`,
);
out.push(
`| cache hit | ${pct(a.stats.cacheHitRatio)} | ${pct(b.stats.cacheHitRatio)} | **${signPct(b.stats.cacheHitRatio - a.stats.cacheHitRatio)}** |`,
);
out.push(
`| cost (USD) | $${a.stats.totalCostUsd.toFixed(6)} | $${b.stats.totalCostUsd.toFixed(6)} | ${costDelta(a.stats.totalCostUsd, b.stats.totalCostUsd)} |`,
);
out.push(
`| prefix hashes | ${a.stats.prefixHashes.length} | ${b.stats.prefixHashes.length} | — |`,
);
if (a.stats.harvestedTurns > 0 || b.stats.harvestedTurns > 0) {
out.push(
`| harvest turns | ${a.stats.harvestedTurns} | ${b.stats.harvestedTurns} | ${signed(b.stats.harvestedTurns - a.stats.harvestedTurns)} |`,
);
out.push(
`| harvest subgoals | ${a.stats.totalSubgoals} | ${b.stats.totalSubgoals} | ${signed(b.stats.totalSubgoals - a.stats.totalSubgoals)} |`,
);
out.push(
`| harvest uncertainties | ${a.stats.totalUncertainties} | ${b.stats.totalUncertainties} | ${signed(b.stats.totalUncertainties - a.stats.totalUncertainties)} |`,
);
}
out.push("");
out.push("## Turn-by-turn");
out.push("");
out.push(`| turn | kind | ${a.label} tool calls | ${b.label} tool calls | note |`);
out.push("|---:|:---:|---|---|---|");
for (const p of report.pairs) {
const aTools =
p.aTools
.map((t) => t.tool)
.filter(Boolean)
.join(", ") || "—";
const bTools =
p.bTools
.map((t) => t.tool)
.filter(Boolean)
.join(", ") || "—";
out.push(`| ${p.turn} | ${p.kind} | ${aTools} | ${bTools} | ${p.divergenceNote ?? ""} |`);
}
out.push("");
if (report.firstDivergenceTurn !== null) {
const p = report.pairs.find((x) => x.turn === report.firstDivergenceTurn);
out.push(`## First divergence (turn ${report.firstDivergenceTurn})`);
out.push("");
out.push(p?.divergenceNote ?? "");
out.push("");
if (p?.aAssistant) {
out.push(`**${a.label}:**`);
out.push("");
out.push("```");
out.push(p.aAssistant.content);
out.push("```");
out.push("");
}
if (p?.bAssistant) {
out.push(`**${b.label}:**`);
out.push("");
out.push("```");
out.push(p.bAssistant.content);
out.push("```");
out.push("");
}
}
return out.join("\n");
}
// ---------- formatting helpers ----------
function row(cols: string[], widths: number[]): string {
return cols.map((c, i) => padRight(c, widths[i] ?? c.length)).join(" ");
}
function statRow(label: string, av: number, bv: number): string {
return row([label, `${av}`, `${bv}`, signed(bv - av)], [20, 14, 14, 14]);
}
function padRight(s: string, w: number): string {
return s.length >= w ? s : s + " ".repeat(w - s.length);
}
function signed(n: number): string {
if (n === 0) return "0";
return `${n > 0 ? "+" : ""}${n}`;
}
function signPct(diff: number): string {
if (diff === 0) return "0pp";
const s = (diff * 100).toFixed(1);
return `${diff > 0 ? "+" : ""}${s}pp`;
}
function pct(x: number): string {
return `${(x * 100).toFixed(1)}%`;
}
function costDelta(a: number, b: number): string {
if (a === 0 && b === 0) return "—";
if (a === 0) return "new";
const pctChange = ((b - a) / a) * 100;
return `${pctChange > 0 ? "+" : ""}${pctChange.toFixed(1)}%`;
}
function truncate(s: string, n: number): string {
return s.length > n ? `${s.slice(0, n)}…` : s;
}
+33
View File
@@ -0,0 +1,33 @@
import { readFileSync } from "node:fs";
import { resolve } from "node:path";
/**
* Minimal `.env` loader; no dependency on dotenv.
*
* Reads KEY=VALUE lines and populates `process.env` for keys not already set.
* Silently no-ops if the file is missing. Safe to call from library entry
* points, CLI commands, examples, and benchmark runners.
*/
export function loadDotenv(path = ".env"): void {
let raw: string;
try {
raw = readFileSync(resolve(process.cwd(), path), "utf8");
} catch {
return;
}
for (const line of raw.split(/\r?\n/)) {
const trimmed = line.trim();
if (!trimmed || trimmed.startsWith("#")) continue;
const eq = trimmed.indexOf("=");
if (eq === -1) continue;
const key = trimmed.slice(0, eq).trim();
let value = trimmed.slice(eq + 1).trim();
if (
(value.startsWith('"') && value.endsWith('"')) ||
(value.startsWith("'") && value.endsWith("'"))
) {
value = value.slice(1, -1);
}
if (process.env[key] === undefined) process.env[key] = value;
}
}
+139
View File
@@ -0,0 +1,139 @@
/**
* Pillar 2 — R1 Thought Harvesting.
*
* Takes the `reasoning_content` emitted by a thinking model (deepseek-reasoner
* / R1) and extracts a structured plan state by making a cheap secondary call
* to V3 in JSON mode. The typed state is intended for the orchestrator to
* branch on — e.g. trigger self-consistency sampling when `uncertainties.length
* > 2`, or surface the subgoals to the user.
*
* Opt-in: loops disable harvesting by default. Failures (bad JSON, API error,
* empty reasoning) return an empty TypedPlanState — the main turn is never
* aborted because of a harvest hiccup.
*/
import type { DeepSeekClient } from "./client.js";
export interface TypedPlanState {
subgoals: string[];
hypotheses: string[];
uncertainties: string[];
rejectedPaths: string[];
}
export interface HarvestOptions {
/** Model used for the extraction call. Defaults to the cheap chat model. */
model?: string;
/** Cap on how many items land in each array. Default 5. */
maxItems?: number;
/** Per-item character cap. Default 80. */
maxItemLen?: number;
/** Abort the extraction if R1 reasoning is shorter than this. Default 40. */
minReasoningLen?: number;
}
export function emptyPlanState(): TypedPlanState {
return { subgoals: [], hypotheses: [], uncertainties: [], rejectedPaths: [] };
}
export function isPlanStateEmpty(s: TypedPlanState | null | undefined): boolean {
if (!s) return true;
return (
s.subgoals.length === 0 &&
s.hypotheses.length === 0 &&
s.uncertainties.length === 0 &&
s.rejectedPaths.length === 0
);
}
const SYSTEM_PROMPT = `You extract a typed plan state from a reasoning trace produced by another LLM.
Output ONLY a JSON object. No markdown, no prose, no backticks.
Schema:
{
"subgoals": string[], // concrete intermediate objectives the trace identifies
"hypotheses": string[], // candidate approaches or assumptions being weighed
"uncertainties": string[], // facts the trace flags as unclear / to verify
"rejectedPaths": string[] // approaches the trace considered and then abandoned
}
Constraints:
- Every field must be present. Use [] if not applicable.
- Each array has at most {maxItems} items.
- Each item is plain text, at most {maxItemLen} characters, no markdown.
- Write in the same language as the trace (Chinese in → Chinese out, etc.).
- Do not quote back the trace; write short, specific phrases.`;
export async function harvest(
reasoningContent: string | null | undefined,
client?: DeepSeekClient,
options: HarvestOptions = {},
): Promise<TypedPlanState> {
if (!client || !reasoningContent) return emptyPlanState();
const minLen = options.minReasoningLen ?? 40;
const trimmed = reasoningContent.trim();
if (trimmed.length < minLen) return emptyPlanState();
const model = options.model ?? "deepseek-chat";
const maxItems = options.maxItems ?? 5;
const maxItemLen = options.maxItemLen ?? 80;
const system = SYSTEM_PROMPT.replace("{maxItems}", String(maxItems)).replace(
"{maxItemLen}",
String(maxItemLen),
);
try {
const resp = await client.chat({
model,
messages: [
{ role: "system", content: system },
{ role: "user", content: trimmed },
],
responseFormat: { type: "json_object" },
temperature: 0,
maxTokens: 600,
});
return parsePlanState(resp.content, maxItems, maxItemLen);
} catch {
return emptyPlanState();
}
}
function parsePlanState(raw: string, maxItems: number, maxItemLen: number): TypedPlanState {
const text = (raw ?? "").trim();
if (!text) return emptyPlanState();
let parsed: unknown;
try {
parsed = JSON.parse(text);
} catch {
// Occasionally a model wraps JSON in fences despite instructions.
const match = text.match(/\{[\s\S]*\}/);
if (!match) return emptyPlanState();
try {
parsed = JSON.parse(match[0]);
} catch {
return emptyPlanState();
}
}
if (!parsed || typeof parsed !== "object") return emptyPlanState();
const obj = parsed as Record<string, unknown>;
return {
subgoals: sanitizeArray(obj.subgoals, maxItems, maxItemLen),
hypotheses: sanitizeArray(obj.hypotheses, maxItems, maxItemLen),
uncertainties: sanitizeArray(obj.uncertainties, maxItems, maxItemLen),
rejectedPaths: sanitizeArray(obj.rejectedPaths ?? obj.rejected_paths, maxItems, maxItemLen),
};
}
function sanitizeArray(raw: unknown, maxItems: number, maxItemLen: number): string[] {
if (!Array.isArray(raw)) return [];
const out: string[] = [];
for (const item of raw) {
if (out.length >= maxItems) break;
if (typeof item !== "string") continue;
const cleaned = item.trim().replace(/\s+/g, " ");
if (!cleaned) continue;
out.push(cleaned.length <= maxItemLen ? cleaned : `${cleaned.slice(0, maxItemLen - 1)}…`);
}
return out;
}
+164
View File
@@ -0,0 +1,164 @@
/** Reasonix — DeepSeek-native agent framework. Library entry point. */
export { DeepSeekClient, Usage } from "./client.js";
export type { ChatResponse, StreamChunk, DeepSeekClientOptions } from "./client.js";
export {
CacheFirstLoop,
formatLoopError,
healLoadedMessages,
stripHallucinatedToolMarkup,
} from "./loop.js";
export type {
CacheFirstLoopOptions,
LoopEvent,
EventRole,
BranchSummary,
BranchProgress,
ReconfigurableOptions,
} from "./loop.js";
export { runBranches, defaultSelector, aggregateBranchUsage } from "./consistency.js";
export type {
BranchOptions,
BranchSample,
BranchResult,
BranchSelector,
} from "./consistency.js";
export { ImmutablePrefix, AppendOnlyLog, VolatileScratch } from "./memory.js";
export type { ImmutablePrefixOptions } from "./memory.js";
export { ToolRegistry } from "./tools.js";
export type { ToolDefinition } from "./tools.js";
export { SessionStats, costUsd, claudeEquivalentCost } from "./telemetry.js";
export type { TurnStats, SessionSummary } from "./telemetry.js";
export {
ToolCallRepair,
scavengeToolCalls,
repairTruncatedJson,
StormBreaker,
analyzeSchema,
flattenSchema,
nestArguments,
} from "./repair/index.js";
export type {
RepairReport,
ToolCallRepairOptions,
ScavengeOptions,
ScavengeResult,
TruncationRepairResult,
FlattenDecision,
} from "./repair/index.js";
export { harvest, emptyPlanState, isPlanStateEmpty } from "./harvest.js";
export type { TypedPlanState, HarvestOptions } from "./harvest.js";
export {
appendSessionMessage,
deleteSession,
listSessions,
loadSessionMessages,
sanitizeName as sanitizeSessionName,
sessionPath,
sessionsDir,
} from "./session.js";
export type { SessionInfo } from "./session.js";
export { loadDotenv } from "./env.js";
export {
openTranscriptFile,
parseTranscript,
readTranscript,
recordFromLoopEvent,
writeMeta,
writeRecord,
} from "./transcript.js";
export type { TranscriptRecord, TranscriptMeta, ReadTranscriptResult } from "./transcript.js";
export { computeReplayStats, replayFromFile } from "./replay.js";
export type { ReplayStats } from "./replay.js";
export {
diffTranscripts,
renderMarkdown as renderDiffMarkdown,
renderSummaryTable as renderDiffSummary,
similarity,
} from "./diff.js";
export type { DiffReport, DiffSide, TurnPair, RenderOptions as DiffRenderOptions } from "./diff.js";
// ---------- MCP (v0.3 foundation) ----------
export { McpClient } from "./mcp/client.js";
export type { McpClientOptions } from "./mcp/client.js";
export { StdioTransport } from "./mcp/stdio.js";
export type { McpTransport, StdioTransportOptions } from "./mcp/stdio.js";
export { SseTransport } from "./mcp/sse.js";
export type { SseTransportOptions } from "./mcp/sse.js";
export {
DEFAULT_MAX_RESULT_CHARS,
bridgeMcpTools,
flattenMcpResult,
truncateForModel,
} from "./mcp/registry.js";
export type { BridgeOptions, BridgeResult, FlattenOptions } from "./mcp/registry.js";
export { parseMcpSpec } from "./mcp/spec.js";
export type { McpSpec, StdioMcpSpec, SseMcpSpec } from "./mcp/spec.js";
// ---------- code mode (v0.3 — `reasonix code`) ----------
export {
parseEditBlocks,
applyEditBlock,
applyEditBlocks,
snapshotBeforeEdits,
restoreSnapshots,
} from "./code/edit-blocks.js";
export type {
EditBlock,
ApplyResult,
ApplyStatus,
EditSnapshot,
} from "./code/edit-blocks.js";
export { CODE_SYSTEM_PROMPT, codeSystemPrompt } from "./code/prompt.js";
export {
MCP_PROTOCOL_VERSION,
isJsonRpcError,
} from "./mcp/types.js";
export type {
McpTool,
McpToolSchema,
CallToolResult,
ListToolsResult,
McpContentBlock,
InitializeResult,
JsonRpcRequest,
JsonRpcResponse,
JsonRpcMessage,
} from "./mcp/types.js";
export { fetchWithRetry } from "./retry.js";
export type { RetryOptions, RetryInfo } from "./retry.js";
export {
defaultConfigPath,
isPlausibleKey,
loadApiKey,
readConfig,
redactKey,
saveApiKey,
writeConfig,
} from "./config.js";
export type { ReasonixConfig } from "./config.js";
export type {
ChatMessage,
ToolCall,
ToolSpec,
ToolFunctionSpec,
Role,
JSONSchema,
} from "./types.js";
export const VERSION = "0.4.3";
+834
View File
@@ -0,0 +1,834 @@
import { type DeepSeekClient, Usage } from "./client.js";
import {
type BranchOptions,
type BranchSample,
aggregateBranchUsage,
runBranches,
} from "./consistency.js";
import { type HarvestOptions, type TypedPlanState, emptyPlanState, harvest } from "./harvest.js";
import { DEFAULT_MAX_RESULT_CHARS, truncateForModel } from "./mcp/registry.js";
import { AppendOnlyLog, type ImmutablePrefix, VolatileScratch } from "./memory.js";
import { type RepairReport, ToolCallRepair } from "./repair/index.js";
import { appendSessionMessage, loadSessionMessages, rewriteSession } from "./session.js";
import {
DEEPSEEK_CONTEXT_TOKENS,
DEFAULT_CONTEXT_TOKENS,
SessionStats,
type TurnStats,
} from "./telemetry.js";
import { ToolRegistry } from "./tools.js";
import type { ChatMessage, ToolCall } from "./types.js";
export type EventRole =
| "assistant_delta"
| "assistant_final"
/**
* Yielded immediately before a tool is dispatched. Lets the TUI put
* up a "▸ tool<X> running…" spinner while the tool's Promise is
* pending — otherwise the UI looks frozen whenever a tool call
* takes more than a few hundred ms (a big `filesystem_edit_file`
* is a typical trigger).
*/
| "tool_start"
| "tool"
| "done"
| "error"
| "warning"
| "branch_start"
| "branch_progress"
| "branch_done";
export interface BranchSummary {
budget: number;
chosenIndex: number;
uncertainties: number[]; // per-sample uncertainty counts
temperatures: number[];
}
export interface BranchProgress {
completed: number;
total: number;
latestIndex: number;
latestTemperature: number;
latestUncertainties: number;
}
export interface LoopEvent {
turn: number;
role: EventRole;
content: string;
reasoningDelta?: string;
toolName?: string;
/**
* Raw JSON-string arguments the model sent for a tool call (role === "tool").
* Populated so transcripts can persist *why* a tool was called, not just
* what it returned. Needed by `reasonix diff` to explain divergences.
*/
toolArgs?: string;
stats?: TurnStats;
planState?: TypedPlanState;
repair?: RepairReport;
branch?: BranchSummary;
branchProgress?: BranchProgress;
error?: string;
/**
* True on `assistant_final` events emitted by the no-tools fallback
* when the loop hit its budget, was aborted, or tripped the
* token-context guard. Consumers that act on assistant text (notably
* the code-mode edit applier) MUST treat these as display-only —
* the model is "wrapping up," not proposing new work. Applying
* SEARCH/REPLACE blocks found in a forced summary caused the
* "analysis became edits" bug in v0.4.1 and earlier.
*/
forcedSummary?: boolean;
}
export interface CacheFirstLoopOptions {
client: DeepSeekClient;
prefix: ImmutablePrefix;
tools?: ToolRegistry;
model?: string;
maxToolIters?: number;
stream?: boolean;
/**
* Pillar 2 — structured harvesting of R1 reasoning into a typed plan state.
* Pass `true` for defaults or an options object. Off by default (adds a
* cheap but non-zero V3 call per turn).
*/
harvest?: boolean | HarvestOptions;
/**
* Self-consistency branching. Pass a number for just a budget (e.g. 3) or
* a full `BranchOptions` object. Disables streaming for the branched turn
* because all samples must complete before selection. Auto-enables harvest
* since the default selector scores samples by plan-state uncertainty.
*/
branch?: number | BranchOptions;
/**
* Session name. When set, the loop pre-loads the session's prior messages
* into its log on construction, and appends every new log entry to
* `~/.reasonix/sessions/<name>.jsonl` so the next run can resume.
*/
session?: string;
}
/**
* Pillar 1 — Cache-First Loop.
*
* - prefix is immutable (cache target)
* - log is append-only (preserves prior-turn prefix)
* - scratch is per-turn volatile (never sent upstream)
*
* Yields a stream of events so a TUI can render progressively.
*/
export interface ReconfigurableOptions {
model?: string;
harvest?: boolean | HarvestOptions;
branch?: number | BranchOptions;
stream?: boolean;
}
export class CacheFirstLoop {
readonly client: DeepSeekClient;
readonly prefix: ImmutablePrefix;
readonly tools: ToolRegistry;
readonly maxToolIters: number;
readonly log = new AppendOnlyLog();
readonly scratch = new VolatileScratch();
readonly stats = new SessionStats();
readonly repair: ToolCallRepair;
// Mutable via configure() — slash commands in the TUI / library callers tweak
// these mid-session so users don't have to restart to try harvest or branch.
model: string;
stream: boolean;
harvestEnabled: boolean;
harvestOptions: HarvestOptions;
branchEnabled: boolean;
branchOptions: BranchOptions;
sessionName: string | null;
/** Number of messages that were pre-loaded from the session file. */
readonly resumedMessageCount: number;
private _turn = 0;
private _streamPreference: boolean;
/**
* Set by {@link abort} to short-circuit the tool-call loop after the
* current iteration. Reset at the start of each `step()` so an Esc
* during one turn doesn't poison the next.
*/
private _aborted = false;
constructor(opts: CacheFirstLoopOptions) {
this.client = opts.client;
this.prefix = opts.prefix;
this.tools = opts.tools ?? new ToolRegistry();
this.model = opts.model ?? "deepseek-chat";
// Iter cap is a safety net, not the primary stop condition. The
// primary stop is the token-context guard inside step(): after
// every model response we check whether the prompt is already past
// 80% of the model's context window, and if so divert to the
// forced-summary path. 64 is high enough that exploration almost
// never exhausts it before the token guard fires first — which
// is the point: let the real constraint (context window) drive
// the decision, keep the iter cap as a last-resort backstop for
// the case where something spins without growing the prompt.
this.maxToolIters = opts.maxToolIters ?? 64;
// Resolve branch config first (since it forces harvest on).
if (typeof opts.branch === "number") {
this.branchOptions = { budget: opts.branch };
} else if (opts.branch && typeof opts.branch === "object") {
this.branchOptions = opts.branch;
} else {
this.branchOptions = {};
}
this.branchEnabled = (this.branchOptions.budget ?? 1) > 1;
// Branching requires harvest for its default selector to work.
const harvestForced = this.branchEnabled;
this.harvestEnabled =
harvestForced ||
opts.harvest === true ||
(typeof opts.harvest === "object" && opts.harvest !== null);
this.harvestOptions =
typeof opts.harvest === "object" && opts.harvest !== null
? opts.harvest
: (this.branchOptions.harvestOptions ?? {});
// Streaming is incompatible with branching (need all samples to select).
this._streamPreference = opts.stream ?? true;
this.stream = this.branchEnabled ? false : this._streamPreference;
const allowedNames = new Set([...this.prefix.toolSpecs.map((s) => s.function.name)]);
this.repair = new ToolCallRepair({ allowedToolNames: allowedNames });
// Session resume: pre-load prior messages into the log if a session name
// is provided. New messages appended to the log are also persisted.
//
// Heal-on-load: if a previous run (or a pre-alpha.6 client) stored a
// tool result bigger than the cap, the next API call would blow past
// DeepSeek's 131k-token limit *before the user even types anything*.
// Truncating here lets the user pick up their session history without
// losing the conversational context around the oversized call.
this.sessionName = opts.session ?? null;
if (this.sessionName) {
const prior = loadSessionMessages(this.sessionName);
const { messages, healedCount, healedFrom } = healLoadedMessages(
prior,
DEFAULT_MAX_RESULT_CHARS,
);
for (const msg of messages) this.log.append(msg);
this.resumedMessageCount = messages.length;
if (healedCount > 0) {
process.stderr.write(
`▸ session "${this.sessionName}": healed ${healedCount} oversized tool result(s) (was ${healedFrom.toLocaleString()} chars total). Old payloads were truncated to fit DeepSeek's context window; the conversation is preserved.\n`,
);
}
} else {
this.resumedMessageCount = 0;
}
}
/**
* Shrink the log by re-truncating oversized tool results to a tighter
* cap, and persist the result back to disk so the next launch doesn't
* re-inherit a fat session file. Returns a summary the TUI can
* display.
*
* Only tool-role messages are touched (same rationale as
* {@link healLoadedMessages}). User and assistant messages carry
* authored intent we can't mechanically shrink without losing
* meaning.
*/
compact(tightCapChars = 4000): { healedCount: number; charsSaved: number } {
const before = this.log.toMessages();
const { messages, healedCount, healedFrom } = healLoadedMessages(before, tightCapChars);
const afterBytes = messages
.filter((m) => m.role === "tool")
.reduce((s, m) => s + (typeof m.content === "string" ? m.content.length : 0), 0);
const charsSaved = healedFrom - afterBytes;
if (healedCount > 0) {
this.log.compactInPlace(messages);
if (this.sessionName) {
try {
rewriteSession(this.sessionName, messages);
} catch {
/* disk full or perms — compaction still applies in-memory */
}
}
}
return { healedCount, charsSaved };
}
private appendAndPersist(message: ChatMessage): void {
this.log.append(message);
if (this.sessionName) {
try {
appendSessionMessage(this.sessionName, message);
} catch {
/* disk full or permission denied shouldn't kill the chat */
}
}
}
/**
* Reconfigure model/harvest/branch/stream mid-session. The loop's log,
* scratch, and stats are preserved — only the per-turn behavior changes.
* Used by the TUI's slash commands and by library callers who want to
* flip a knob between turns.
*/
configure(opts: ReconfigurableOptions): void {
if (opts.model !== undefined) this.model = opts.model;
if (opts.stream !== undefined) this._streamPreference = opts.stream;
if (opts.branch !== undefined) {
if (typeof opts.branch === "number") {
this.branchOptions = { budget: opts.branch };
} else if (opts.branch && typeof opts.branch === "object") {
this.branchOptions = opts.branch;
} else {
this.branchOptions = {};
}
this.branchEnabled = (this.branchOptions.budget ?? 1) > 1;
}
if (opts.harvest !== undefined) {
const want =
opts.harvest === true || (typeof opts.harvest === "object" && opts.harvest !== null);
this.harvestEnabled = want || this.branchEnabled;
if (typeof opts.harvest === "object" && opts.harvest !== null) {
this.harvestOptions = opts.harvest;
}
} else if (this.branchEnabled) {
// branch turned on without explicit harvest → force it on
this.harvestEnabled = true;
}
// Branching always forces non-streaming; otherwise honor preference.
this.stream = this.branchEnabled ? false : this._streamPreference;
}
private buildMessages(pendingUser: string | null): ChatMessage[] {
const msgs: ChatMessage[] = [...this.prefix.toMessages(), ...this.log.toMessages()];
if (pendingUser !== null) msgs.push({ role: "user", content: pendingUser });
return msgs;
}
/**
* Signal the currently-running {@link step} that the user wants to
* stop exploring. Takes effect at the next iteration boundary — if a
* tool call is mid-flight it will be allowed to finish, then the
* loop diverts to the forced-summary path so the user gets an
* answer instead of a cliff. Called by the TUI on Esc.
*/
abort(): void {
this._aborted = true;
}
/**
* Drop everything in the log after (and including) the most recent
* user message. Used by `/retry` so the caller can re-send that
* message with a fresh turn instead of layering another response on
* top of the prior exchange. Returns the content of the dropped user
* message, or `null` if there isn't one yet.
*
* Persists by rewriting the session file — otherwise the next
* launch would rehydrate the old exchange and `/retry` would seem
* to have done nothing.
*/
retryLastUser(): string | null {
const entries = this.log.entries;
let lastUserIdx = -1;
for (let i = entries.length - 1; i >= 0; i--) {
if (entries[i]!.role === "user") {
lastUserIdx = i;
break;
}
}
if (lastUserIdx < 0) return null;
const raw = entries[lastUserIdx]!.content;
const userText = typeof raw === "string" ? raw : "";
// Keep everything strictly before the user message. The caller
// will submit the text again through the normal path, which
// re-appends the user turn on first successful API response.
const preserved = entries.slice(0, lastUserIdx).map((m) => ({ ...m }));
this.log.compactInPlace(preserved);
if (this.sessionName) {
try {
rewriteSession(this.sessionName, preserved);
} catch {
/* disk-full / perms — in-memory compaction still applies */
}
}
return userText;
}
async *step(userInput: string): AsyncGenerator<LoopEvent> {
this._turn++;
this.scratch.reset();
this._aborted = false;
let pendingUser: string | null = userInput;
const toolSpecs = this.prefix.tools();
// 70% of the iter budget is the "you're getting close" threshold. We
// only warn once per step so the user sees a single signal, not a
// string of identical yellow lines stacked up.
const warnAt = Math.max(1, Math.floor(this.maxToolIters * 0.7));
let warnedForIterBudget = false;
for (let iter = 0; iter < this.maxToolIters; iter++) {
if (this._aborted) {
// Esc means "stop now" — not "stop and force another 30-90s
// reasoner call to produce a summary I didn't ask for". The
// user's mental model of cancel is immediate. We emit a
// synthetic assistant_final (tagged forcedSummary so the
// code-mode applier ignores it) with a short stopped
// message, then done. The prior tool outputs are still in
// the log if the user wants to continue — asking again
// will hit a warm cache and be cheap.
//
// Budget / context-guard still call forceSummaryAfterIterLimit
// because there the USER didn't choose to stop — we did —
// and leaving them staring at nothing is worse than one extra
// call.
yield {
turn: this._turn,
role: "warning",
content: `aborted at iter ${iter}/${this.maxToolIters} — stopped without producing a summary (press ↑ + Enter or /retry to resume)`,
};
const stoppedMsg =
"[aborted by user (Esc) — no summary produced. Ask again or /retry when ready; prior tool output is still in the log.]";
this.appendAndPersist({ role: "assistant", content: stoppedMsg });
yield {
turn: this._turn,
role: "assistant_final",
content: stoppedMsg,
forcedSummary: true,
};
yield { turn: this._turn, role: "done", content: stoppedMsg };
return;
}
if (!warnedForIterBudget && iter >= warnAt) {
warnedForIterBudget = true;
yield {
turn: this._turn,
role: "warning",
content: `${iter}/${this.maxToolIters} tool calls used — approaching budget. Press Esc to force a summary now.`,
};
}
const messages = this.buildMessages(pendingUser);
let assistantContent = "";
let reasoningContent = "";
let toolCalls: ToolCall[] = [];
let usage: TurnStats["usage"] | null = null;
let branchSummary: BranchSummary | undefined;
let preHarvestedPlanState: TypedPlanState | undefined;
try {
if (this.branchEnabled) {
const budget = this.branchOptions.budget ?? 1;
yield {
turn: this._turn,
role: "branch_start",
content: "",
branchProgress: {
completed: 0,
total: budget,
latestIndex: -1,
latestTemperature: -1,
latestUncertainties: -1,
},
};
// Queue samples as they complete so we can yield progress events
// in resolution order (not launch order).
const queue: BranchSample[] = [];
let waiter: ((s: BranchSample) => void) | null = null;
const onSampleDone = (sample: BranchSample) => {
if (waiter) {
const w = waiter;
waiter = null;
w(sample);
} else {
queue.push(sample);
}
};
const branchPromise = runBranches(
this.client,
{
model: this.model,
messages,
tools: toolSpecs.length ? toolSpecs : undefined,
},
{
...this.branchOptions,
harvestOptions: this.harvestOptions,
onSampleDone,
},
);
for (let k = 0; k < budget; k++) {
const sample: BranchSample =
queue.shift() ??
(await new Promise<BranchSample>((resolve) => {
waiter = resolve;
}));
yield {
turn: this._turn,
role: "branch_progress",
content: "",
branchProgress: {
completed: k + 1,
total: budget,
latestIndex: sample.index,
latestTemperature: sample.temperature,
latestUncertainties: sample.planState.uncertainties.length,
},
};
}
const result = await branchPromise;
assistantContent = result.chosen.response.content;
reasoningContent = result.chosen.response.reasoningContent ?? "";
toolCalls = result.chosen.response.toolCalls;
// Cost accounting: sum usage across ALL samples, not just the winner.
// (We paid for all three.) Harvest-call tokens are not tracked; they
// amount to rounding error compared to the main R1 calls.
const agg = aggregateBranchUsage(result.samples);
usage = new Usage(
agg.promptTokens,
agg.completionTokens,
agg.totalTokens,
agg.promptCacheHitTokens,
agg.promptCacheMissTokens,
);
preHarvestedPlanState = result.chosen.planState;
branchSummary = summarizeBranch(result.chosen, result.samples);
yield {
turn: this._turn,
role: "branch_done",
content: "",
branch: branchSummary,
};
} else if (this.stream) {
const callBuf: Map<number, ToolCall> = new Map();
for await (const chunk of this.client.stream({
model: this.model,
messages,
tools: toolSpecs.length ? toolSpecs : undefined,
})) {
if (chunk.contentDelta) {
assistantContent += chunk.contentDelta;
yield {
turn: this._turn,
role: "assistant_delta",
content: chunk.contentDelta,
};
}
if (chunk.reasoningDelta) {
reasoningContent += chunk.reasoningDelta;
yield {
turn: this._turn,
role: "assistant_delta",
content: "",
reasoningDelta: chunk.reasoningDelta,
};
}
if (chunk.toolCallDelta) {
const d = chunk.toolCallDelta;
const cur = callBuf.get(d.index) ?? {
id: d.id,
type: "function" as const,
function: { name: "", arguments: "" },
};
if (d.id) cur.id = d.id;
if (d.name) cur.function.name = (cur.function.name ?? "") + d.name;
if (d.argumentsDelta)
cur.function.arguments = (cur.function.arguments ?? "") + d.argumentsDelta;
callBuf.set(d.index, cur);
}
if (chunk.usage) usage = chunk.usage;
}
toolCalls = [...callBuf.values()];
} else {
const resp = await this.client.chat({
model: this.model,
messages,
tools: toolSpecs.length ? toolSpecs : undefined,
});
assistantContent = resp.content;
reasoningContent = resp.reasoningContent ?? "";
toolCalls = resp.toolCalls;
usage = resp.usage;
}
} catch (err) {
yield {
turn: this._turn,
role: "error",
content: "",
error: formatLoopError(err as Error),
};
return;
}
const turnStats = this.stats.record(this._turn, this.model, usage ?? new Usage());
// Commit the user turn to the log only on success of the first round-trip.
if (pendingUser !== null) {
this.appendAndPersist({ role: "user", content: pendingUser });
pendingUser = null;
}
this.scratch.reasoning = reasoningContent || null;
const planState = preHarvestedPlanState
? preHarvestedPlanState
: this.harvestEnabled
? await harvest(reasoningContent || null, this.client, this.harvestOptions)
: emptyPlanState();
const { calls: repairedCalls, report } = this.repair.process(
toolCalls,
reasoningContent || null,
);
this.appendAndPersist(this.assistantMessage(assistantContent, repairedCalls));
yield {
turn: this._turn,
role: "assistant_final",
content: assistantContent,
stats: turnStats,
planState,
repair: report,
branch: branchSummary,
};
if (repairedCalls.length === 0) {
yield { turn: this._turn, role: "done", content: assistantContent };
return;
}
// Token-budget guard — the real stop condition. Iter count is a
// proxy that misses the actual constraint: how close the prompt
// already is to DeepSeek's 131k context. If we're over 80%, the
// NEXT call (with the just-executed tools' results stuffed into
// history) will be worse, and fairly soon it'll 400 with
// "maximum context length". Divert to summary instead — same
// path as iter-cap and Esc-abort, just triggered by a different
// signal so the user sees *why* it happened.
const ctxMax = DEEPSEEK_CONTEXT_TOKENS[this.model] ?? DEFAULT_CONTEXT_TOKENS;
if (usage && usage.promptTokens / ctxMax > 0.8) {
yield {
turn: this._turn,
role: "warning",
content: `context ${usage.promptTokens}/${ctxMax} (${Math.round(
(usage.promptTokens / ctxMax) * 100,
)}%) — more tools would overflow. Forcing summary from what was gathered.`,
};
yield* this.forceSummaryAfterIterLimit({ reason: "context-guard" });
return;
}
for (const call of repairedCalls) {
const name = call.function?.name ?? "";
const args = call.function?.arguments ?? "{}";
// Announce the tool BEFORE awaiting it so the TUI can render a
// "running…" indicator. Without this, the window between
// assistant_final and the tool-result yield is silent from the
// UI's perspective — which makes long tool calls feel like the
// app has hung.
yield {
turn: this._turn,
role: "tool_start",
content: "",
toolName: name,
toolArgs: args,
};
const result = await this.tools.dispatch(name, args);
this.appendAndPersist({
role: "tool",
tool_call_id: call.id ?? "",
name,
content: result,
});
yield {
turn: this._turn,
role: "tool",
content: result,
toolName: name,
toolArgs: args,
};
}
}
// We exhausted the tool-call budget while the model still wanted to
// call more tools. Rather than stopping silently (which leaves the
// user staring at a blank prompt), force one final no-tools call so
// the model must produce a text summary from everything it has
// already seen.
yield* this.forceSummaryAfterIterLimit({ reason: "budget" });
}
private async *forceSummaryAfterIterLimit(
opts: { reason: "budget" | "aborted" | "context-guard" } = { reason: "budget" },
): AsyncGenerator<LoopEvent> {
try {
const messages = this.buildMessages(null);
// Passing `tools: undefined` was supposed to force a text
// response, but R1 can still hallucinate tool-call markup
// (e.g. DSML `<|DSML|function_calls>…</|DSML|function_calls>`)
// in prose when it's been primed by prior tool use. An explicit
// user-role instruction plus post-hoc stripping of known
// hallucination shapes keeps the user from seeing raw markup.
messages.push({
role: "user",
content:
"I'm out of tool-call budget for this turn. Summarize in plain prose what you learned from the tool results above. Do NOT emit any tool calls, function-call markup, DSML invocations, or SEARCH/REPLACE edit blocks — they will be silently discarded. Just plain text.",
});
const resp = await this.client.chat({
model: this.model,
messages,
// no tools → model is forced to answer in text
});
const rawContent = resp.content?.trim() ?? "";
const cleaned = stripHallucinatedToolMarkup(rawContent);
const summary =
cleaned ||
"(model emitted fake tool-call markup instead of a prose summary — try /retry with a narrower question, or /think to inspect R1's reasoning)";
const reasonPrefix = reasonPrefixFor(opts.reason, this.maxToolIters);
const annotated = `${reasonPrefix}\n\n${summary}`;
const summaryStats = this.stats.record(this._turn, this.model, resp.usage ?? new Usage());
this.appendAndPersist({ role: "assistant", content: summary });
yield {
turn: this._turn,
role: "assistant_final",
content: annotated,
stats: summaryStats,
forcedSummary: true,
};
yield { turn: this._turn, role: "done", content: summary };
} catch (err) {
const label = errorLabelFor(opts.reason, this.maxToolIters);
yield {
turn: this._turn,
role: "error",
content: "",
error: `${label} and the fallback summary call failed: ${(err as Error).message}. Run /clear and retry with a narrower question, or raise --max-tool-iters.`,
};
yield { turn: this._turn, role: "done", content: "" };
}
}
async run(userInput: string, onEvent?: (ev: LoopEvent) => void): Promise<string> {
let final = "";
for await (const ev of this.step(userInput)) {
onEvent?.(ev);
if (ev.role === "assistant_final") final = ev.content;
if (ev.role === "done") break;
}
return final;
}
private assistantMessage(content: string, toolCalls: ToolCall[]): ChatMessage {
const msg: ChatMessage = { role: "assistant", content };
if (toolCalls.length > 0) msg.tool_calls = toolCalls;
return msg;
}
}
/**
* R1 occasionally hallucinates tool-call markup as plain text when the
* real tool channel has been closed — typically our forced-summary
* path, where `tools: undefined` is supposed to force prose but isn't
* always respected. The markup isn't parsed by our tool-call path
* (the API response's structured `tool_calls` field is empty), so
* it's just noise in the user's view. Strip known envelope shapes.
*
* Exported so tests can exercise it against concrete R1 outputs.
*/
export function stripHallucinatedToolMarkup(s: string): string {
let out = s;
// DeepSeek's DSML envelope (both the full-width "|" character and
// the ASCII-only fallback we've seen — the full-width form is the
// one R1 emits in practice)
out = out.replace(/<|DSML|function_calls>[\s\S]*?<\/?|DSML|function_calls>/g, "");
out = out.replace(/<\|DSML\|function_calls>[\s\S]*?<\/?\|DSML\|function_calls>/g, "");
// Anthropic / generic XML-ish envelope
out = out.replace(/<function_calls>[\s\S]*?<\/function_calls>/g, "");
// Lone unpaired DSML opener left over after the closer was on a
// different line (seen when R1 truncates mid-call).
out = out.replace(/<|DSML|[\s\S]*$/g, "");
return out.trim();
}
function reasonPrefixFor(reason: "budget" | "aborted" | "context-guard", iterCap: number): string {
if (reason === "aborted") return "[aborted by user (Esc) — summarizing what I found so far]";
if (reason === "context-guard") {
return "[context budget running low — summarizing before the next call would overflow]";
}
return `[tool-call budget (${iterCap}) reached — forcing summary from what I found]`;
}
function errorLabelFor(reason: "budget" | "aborted" | "context-guard", iterCap: number): string {
if (reason === "aborted") return "aborted by user";
if (reason === "context-guard") return "context-guard triggered (prompt > 80% of window)";
return `tool-call budget (${iterCap}) reached`;
}
function summarizeBranch(chosen: BranchSample, samples: BranchSample[]): BranchSummary {
return {
budget: samples.length,
chosenIndex: chosen.index,
uncertainties: samples.map((s) => s.planState.uncertainties.length),
temperatures: samples.map((s) => s.temperature),
};
}
/**
* Truncate any tool-role message whose content exceeds the cap. User
* and assistant messages are left alone because (a) they're almost
* always small, (b) truncating user prompts would corrupt conversational
* intent in a way the user didn't author. Exported for tests.
*/
export function healLoadedMessages(
messages: ChatMessage[],
maxChars: number,
): { messages: ChatMessage[]; healedCount: number; healedFrom: number } {
let healedCount = 0;
let healedFrom = 0;
const out = messages.map((msg) => {
if (msg.role !== "tool") return msg;
const content = typeof msg.content === "string" ? msg.content : "";
if (content.length <= maxChars) return msg;
healedCount += 1;
healedFrom += content.length;
return { ...msg, content: truncateForModel(content, maxChars) };
});
return { messages: out, healedCount, healedFrom };
}
/**
* Annotate the `DeepSeek 400: … maximum context length …` error the API
* returns when a session's history has grown past 131,072 tokens. The
* raw message is a JSON blob; we surface a short actionable hint on top
* so the user knows to `/forget` or `/clear` rather than parsing the
* JSON themselves. Other errors pass through unchanged — the loop's
* error channel already formats them well enough.
*/
export function formatLoopError(err: Error): string {
const msg = err.message ?? "";
if (msg.includes("maximum context length")) {
// Pull the "requested X tokens" figure out of the JSON for scale.
const reqMatch = msg.match(/requested\s+(\d+)\s+tokens/);
const requested = reqMatch
? `${Number(reqMatch[1]).toLocaleString()} tokens`
: "too many tokens";
return `Context overflow (DeepSeek 400): session history is ${requested}, past the 131,072-token limit. Usually this means a single tool call returned a huge payload. v0.3.0-alpha.6+ caps new tool results at 32k chars, AND auto-heals oversized history on session load — restart Reasonix and this session should come back trimmed. If it still overflows, run /forget (delete the session) or /clear (drop the displayed history) to start fresh.`;
}
return msg;
}
+147
View File
@@ -0,0 +1,147 @@
# MCP client (v0.3 foundation)
Minimal [Model Context Protocol](https://spec.modelcontextprotocol.io/)
client, hand-rolled in TypeScript. Lets Reasonix consume tools from any
MCP server (filesystem, github, slack, puppeteer, …) while applying the
Cache-First Loop and tool-call repair to the whole thing automatically.
## Design choice: roll-our-own, not @modelcontextprotocol/sdk
Same reasoning that drove `client.ts` (DeepSeek) rather than `openai`:
- **Zero runtime deps** for this module. Consistent with Reasonix's
policy of owning the wire format where it matters.
- **Surface tuning**: we only implement what Reasonix actually uses —
initialize + tools/list + tools/call. Resources, prompts, sampling,
and progress notifications are deferred.
- **Insulation** from SDK breaking changes. The spec is more stable
than any single SDK release.
Swappable if needed: `McpClient` depends on the `McpTransport` interface,
so the day we do want the official SDK's transport layer we can adapt
it and keep everything else.
## What's shipped here
```
src/mcp/
├── types.ts JSON-RPC 2.0 + MCP-specific message types
├── stdio.ts McpTransport interface + StdioTransport (spawn child)
├── sse.ts SseTransport (HTTP+SSE for remote/hosted servers)
├── spec.ts parseMcpSpec — parses --mcp CLI arg into transport-tagged spec
├── catalog.ts curated list of popular official MCP servers
├── client.ts McpClient: initialize / listTools / callTool
├── registry.ts bridgeMcpTools: MCP → ToolRegistry
└── README.md (this file)
tests/mcp.test.ts — in-process fake transport, no child processes
tests/mcp-sse.test.ts — in-process http.Server fake for SSE
```
## What's NOT here (yet)
| feature | status | note |
|---|---|---|
| CLI wiring (`reasonix chat --mcp <cmd>`) | ✅ shipped | see Usage below |
| Bundled demo server | ✅ shipped | `examples/mcp-server-demo.ts`, exposes echo/add/get_time |
| Real-subprocess integration test | ✅ shipped | `tests/mcp-integration.test.ts` |
| Resources / `resources/list` / `resources/read` | deferred | Reasonix doesn't surface resources today |
| Prompts / `prompts/list` | deferred | ditto |
| Progress notifications | deferred | long-running tool support comes with the CLI work |
| Streaming results | deferred | current shape returns one CallToolResult per call |
| SSE transport | ✅ shipped | `src/mcp/sse.ts` — pass `http(s)://…` to `--mcp` |
| Streamable HTTP (2025-03-26 spec) | deferred | waiting for a real server to validate against |
| MCP server that Reasonix exposes | never | out of scope — Reasonix is a client |
## Usage (CLI)
`--mcp` is repeatable — attach one or many MCP servers; their tools become
first-class citizens of the loop.
```bash
# Single server, anonymous (tools use native names):
reasonix chat --mcp "node --import tsx examples/mcp-server-demo.ts"
# Official filesystem server:
reasonix chat --mcp "npx -y @modelcontextprotocol/server-filesystem /tmp/safe-dir"
# Multiple servers, each namespaced. Syntax: "name=command args..."
# Tools land in a shared registry as fs_read_file, demo_add, etc.
reasonix chat \
--mcp "fs=npx -y @modelcontextprotocol/server-filesystem /tmp/safe" \
--mcp "demo=node --import tsx examples/mcp-server-demo.ts"
# Global prefix (only honored when there's ONE anonymous server):
reasonix chat \
--mcp "npx -y @modelcontextprotocol/server-filesystem /tmp" \
--mcp-prefix fs_
# Same flag works with one-shot run:
reasonix run "list files in /tmp/safe-dir" \
--mcp "npx -y @modelcontextprotocol/server-filesystem /tmp/safe-dir"
```
Each spec is shell-split (spaces separate args; use quotes for paths with
spaces). Windows-friendly: backslashes pass through literally outside
quotes, so `C:\path\to\dir` works. Tools get folded into the
`ImmutablePrefix` for the model, and every call goes through Reasonix's
Cache-First loop + tool-call repair (scavenge / flatten / storm)
automatically.
## Usage (library)
```ts
import {
McpClient,
StdioTransport,
bridgeMcpTools,
CacheFirstLoop,
DeepSeekClient,
ImmutablePrefix,
} from "reasonix";
// 1. Spawn + connect to an MCP server
const transport = new StdioTransport({
command: "npx",
args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp/safe-dir"],
});
const mcp = new McpClient({ transport });
await mcp.initialize();
// 2. Bridge its tools into a Reasonix ToolRegistry
const { registry } = await bridgeMcpTools(mcp, { namePrefix: "fs_" });
// 3. Use them with the Cache-First Loop — same as any native tool
const client = new DeepSeekClient();
const loop = new CacheFirstLoop({
client,
prefix: new ImmutablePrefix({
system: "You can use the filesystem tools to help the user.",
toolSpecs: registry.specs(),
}),
tools: registry,
});
for await (const ev of loop.step("List the files in /tmp/safe-dir.")) {
if (ev.role === "assistant_final") console.log(ev.content);
}
// 4. Clean up
await mcp.close();
```
The payoff: the filesystem server's tools now inherit Reasonix's
cache-first prefix stability + repair (schema flatten, tool-call
scavenge, call-storm break) without the MCP server knowing anything
about it.
## Wire protocol notes (stdio)
- **Framing**: newline-delimited JSON. One JSON-RPC message per line,
UTF-8, no Content-Length header (that's LSP, not MCP stdio).
- **Stderr**: forwarded to the parent's stderr. Servers often print
startup banners there; that's fine.
- **Shutdown**: `close()` calls `child.stdin.end()` then SIGTERM if the
process hasn't exited.
- **Malformed lines**: dropped silently. Some servers emit non-JSON
during startup; logging every dropped line would be noise.
+79
View File
@@ -0,0 +1,79 @@
/**
* Curated catalog of popular MCP servers.
*
* Hardcoded because (a) this list changes slowly — maybe monthly, (b)
* fetching it over the network would make `reasonix mcp list` flaky
* offline or behind a proxy. When it does change, update this file and
* ship a patch release.
*
* Inclusion criteria:
* - actively maintained (official Anthropic repo OR widely used)
* - stdio-compatible (Reasonix doesn't do SSE yet)
* - installable with one `npx -y ...` command — zero manual setup
* - has a clear value proposition in one short line
*
* Not included: servers that need API keys / OAuth / complex config.
* Those get their own docs once we have patterns for them.
*/
export interface CatalogEntry {
/** Short name, used as the namespace prefix when suggested. */
name: string;
/** One-line description shown in `reasonix mcp list`. */
summary: string;
/** npm package id (for `npx -y <pkg>`). */
package: string;
/** Extra args the user must supply (e.g. a directory path). */
userArgs?: string;
/** Notes the user needs to know — shown dimmed. */
note?: string;
}
// Every entry below is verified to exist on npm as of this release.
// `fetch` and `sqlite` are deliberately *absent* — their reference
// servers are Python-only (`pip install mcp-server-fetch`), so a Node
// user running `npx -y @modelcontextprotocol/server-fetch` hits a 404
// from the npm registry. We'd rather ship a smaller list that always
// works than a longer list where two options silently 404 on the user.
export const MCP_CATALOG: CatalogEntry[] = [
{
name: "filesystem",
summary: "read/write/search files inside a sandboxed directory",
package: "@modelcontextprotocol/server-filesystem",
userArgs: "<dir>",
note: "the directory is a hard sandbox — the server refuses access outside it",
},
{
name: "memory",
summary: "persistent key-value memory across sessions",
package: "@modelcontextprotocol/server-memory",
},
{
name: "github",
summary: "read issues, PRs, code search (needs GITHUB_PERSONAL_ACCESS_TOKEN)",
package: "@modelcontextprotocol/server-github",
note: "set GITHUB_PERSONAL_ACCESS_TOKEN in your env before spawning",
},
{
name: "puppeteer",
summary: "browser automation — take screenshots, click, type",
package: "@modelcontextprotocol/server-puppeteer",
note: "downloads Chromium on first run (~200 MB)",
},
{
name: "everything",
summary: "official test server — exercises every MCP feature",
package: "@modelcontextprotocol/server-everything",
note: "useful for debugging your Reasonix setup",
},
];
/**
* Build the `reasonix chat --mcp "..."` command line for a catalog entry.
* Returns a copy-pasteable fragment starting at `--mcp`.
*/
export function mcpCommandFor(entry: CatalogEntry): string {
const pkg = entry.package;
const tail = entry.userArgs ? ` ${entry.userArgs}` : "";
return `--mcp "${entry.name}=npx -y ${pkg}${tail}"`;
}
+170
View File
@@ -0,0 +1,170 @@
/**
* MCP client — request/response correlation, initialize handshake,
* tools/list, tools/call. Built on top of a McpTransport so the same
* logic works against a real stdio server or an in-process fake.
*/
import type { McpTransport } from "./stdio.js";
import {
type CallToolParams,
type CallToolResult,
type InitializeParams,
type InitializeResult,
type JsonRpcId,
type JsonRpcMessage,
type JsonRpcRequest,
type JsonRpcResponse,
type ListToolsResult,
MCP_PROTOCOL_VERSION,
type McpClientInfo,
isJsonRpcError,
} from "./types.js";
export interface McpClientOptions {
transport: McpTransport;
clientInfo?: McpClientInfo;
/** Per-request timeout. Default 60s. */
requestTimeoutMs?: number;
}
interface PendingRequest {
resolve: (value: unknown) => void;
reject: (err: Error) => void;
timeout: NodeJS.Timeout;
}
export class McpClient {
private readonly transport: McpTransport;
private readonly clientInfo: McpClientInfo;
private readonly requestTimeoutMs: number;
private readonly pending = new Map<JsonRpcId, PendingRequest>();
private nextId = 1;
private readerStarted = false;
private initialized = false;
private _serverCapabilities: InitializeResult["capabilities"] = {};
constructor(opts: McpClientOptions) {
this.transport = opts.transport;
this.clientInfo = opts.clientInfo ?? { name: "reasonix", version: "0.3.0-dev" };
this.requestTimeoutMs = opts.requestTimeoutMs ?? 60_000;
}
/** Server's advertised capabilities, available after initialize(). */
get serverCapabilities(): InitializeResult["capabilities"] {
return this._serverCapabilities;
}
/**
* Complete the initialize → initialized handshake. Must be called
* before any other method (otherwise compliant servers reject).
*/
async initialize(): Promise<InitializeResult> {
if (this.initialized) throw new Error("MCP client already initialized");
this.startReaderIfNeeded();
const result = await this.request<InitializeResult>("initialize", {
protocolVersion: MCP_PROTOCOL_VERSION,
capabilities: { tools: {} },
clientInfo: this.clientInfo,
} satisfies InitializeParams);
this._serverCapabilities = result.capabilities ?? {};
// Per spec: client sends notifications/initialized after receiving the
// initialize response. Only then is the connection live for other
// methods.
await this.transport.send({
jsonrpc: "2.0",
method: "notifications/initialized",
});
this.initialized = true;
return result;
}
/** List tools the server exposes. */
async listTools(): Promise<ListToolsResult> {
this.assertInitialized();
return this.request<ListToolsResult>("tools/list", {});
}
/** Invoke a tool by name. Returns the raw MCP result (caller unwraps content). */
async callTool(name: string, args?: Record<string, unknown>): Promise<CallToolResult> {
this.assertInitialized();
return this.request<CallToolResult>("tools/call", {
name,
arguments: args ?? {},
} satisfies CallToolParams);
}
/** Close the transport and reject any outstanding requests. */
async close(): Promise<void> {
for (const [, pending] of this.pending) {
clearTimeout(pending.timeout);
pending.reject(new Error("MCP client closed"));
}
this.pending.clear();
await this.transport.close();
}
// ---------- internals ----------
private assertInitialized(): void {
if (!this.initialized) throw new Error("MCP client not initialized — call initialize() first");
}
private async request<R>(method: string, params: unknown): Promise<R> {
const id = this.nextId++;
const frame: JsonRpcRequest = { jsonrpc: "2.0", id, method, params };
const promise = new Promise<R>((resolve, reject) => {
const timeout = setTimeout(() => {
this.pending.delete(id);
reject(
new Error(`MCP request ${method} (id=${id}) timed out after ${this.requestTimeoutMs}ms`),
);
}, this.requestTimeoutMs);
this.pending.set(id, {
resolve: resolve as (value: unknown) => void,
reject,
timeout,
});
});
await this.transport.send(frame);
return promise;
}
private startReaderIfNeeded(): void {
if (this.readerStarted) return;
this.readerStarted = true;
// Fire-and-forget: the reader runs for the lifetime of the client.
void this.readLoop();
}
private async readLoop(): Promise<void> {
try {
for await (const msg of this.transport.messages()) {
this.dispatch(msg);
}
} catch (err) {
// Surface as rejections on all pending requests so nobody hangs.
for (const [, pending] of this.pending) {
clearTimeout(pending.timeout);
pending.reject(err as Error);
}
this.pending.clear();
}
}
private dispatch(msg: JsonRpcMessage): void {
// We only care about responses (have an `id`) for now. Server-initiated
// notifications are dropped until we support resources/prompts.
if (!("id" in msg) || msg.id === null || msg.id === undefined) return;
if (!("result" in msg) && !("error" in msg)) return; // it's a request from server
const pending = this.pending.get(msg.id);
if (!pending) return; // late response after timeout; drop
this.pending.delete(msg.id);
clearTimeout(pending.timeout);
const resp = msg as JsonRpcResponse;
if (isJsonRpcError(resp)) {
pending.reject(new Error(`MCP ${resp.error.code}: ${resp.error.message}`));
} else {
pending.resolve(resp.result);
}
}
}
+140
View File
@@ -0,0 +1,140 @@
/**
* Bridge: register an MCP server's tools into a Reasonix ToolRegistry.
*
* This is the integration surface. Once done, `CacheFirstLoop` sees the
* MCP tools as if they were native — they inherit Cache-First + repair
* (scavenge / truncation / storm) automatically. That's the payoff: any
* MCP ecosystem tool, wrapped in Reasonix's Pillar 1 + Pillar 3 benefits.
*/
import { ToolRegistry } from "../tools.js";
import type { JSONSchema } from "../types.js";
import type { McpClient } from "./client.js";
import type { CallToolResult, McpContentBlock } from "./types.js";
export interface BridgeOptions {
/**
* Prefix prepended to every MCP tool name when registered. Defaults to
* empty (no prefix). Useful when bridging multiple servers into one
* registry and names collide — e.g. `fs` + `gh` both exposing `search`.
*/
namePrefix?: string;
/** Registry to populate. Creates a fresh one if omitted. */
registry?: ToolRegistry;
/** Auto-flatten deep schemas (Pillar 3). Defaults to the registry's own default (true). */
autoFlatten?: boolean;
/**
* Per-tool-call result cap, in characters. If a tool returns more than
* this, the result is truncated and a `[…truncated N chars…]` marker is
* appended before the last KB so the model still sees a useful tail.
* Defaults to {@link DEFAULT_MAX_RESULT_CHARS}.
*
* Why this exists: DeepSeek V3's context is 131,072 tokens. A single
* `read_file` against a big source file can return >3 MB of text
* (~900k tokens) and permanently poison the session — every subsequent
* turn rebuilds the history and 400s. This cap is a floor. Users who
* legitimately want bigger payloads can raise it explicitly.
*/
maxResultChars?: number;
}
/**
* 32,000 chars ≈ 8k English tokens, or ~16k CJK tokens. Small enough to
* fit comfortably in history even across 5–10 tool calls, large enough
* that most file reads and directory listings fit un-truncated.
*/
export const DEFAULT_MAX_RESULT_CHARS = 32_000;
export interface BridgeResult {
registry: ToolRegistry;
/** Names actually registered (may differ from MCP names when a prefix is applied). */
registeredNames: string[];
/** Names the server listed but the bridge skipped (e.g. invalid schemas). */
skipped: Array<{ name: string; reason: string }>;
}
/**
* Walk a connected `McpClient`'s tools/list result, register each into a
* Reasonix `ToolRegistry`. Each registered `fn` proxies through the
* client's tools/call. Tool results are flattened into a string (joining
* text blocks with newlines, prefixing image blocks as placeholders) so
* they fit Reasonix's existing tool-dispatch contract.
*/
export async function bridgeMcpTools(
client: McpClient,
opts: BridgeOptions = {},
): Promise<BridgeResult> {
const registry = opts.registry ?? new ToolRegistry({ autoFlatten: opts.autoFlatten });
const prefix = opts.namePrefix ?? "";
const maxResultChars = opts.maxResultChars ?? DEFAULT_MAX_RESULT_CHARS;
const result: BridgeResult = { registry, registeredNames: [], skipped: [] };
const listed = await client.listTools();
for (const mcpTool of listed.tools) {
if (!mcpTool.name) {
result.skipped.push({ name: "?", reason: "empty tool name" });
continue;
}
const registeredName = `${prefix}${mcpTool.name}`;
registry.register({
name: registeredName,
description: mcpTool.description ?? "",
parameters: mcpTool.inputSchema as JSONSchema,
fn: async (args: Record<string, unknown>) => {
const toolResult = await client.callTool(mcpTool.name, args);
return flattenMcpResult(toolResult, { maxChars: maxResultChars });
},
});
result.registeredNames.push(registeredName);
}
return result;
}
export interface FlattenOptions {
/** Cap the flattened string at this many characters. Default: no cap. */
maxChars?: number;
}
/**
* Turn an MCP CallToolResult into a string — the contract Reasonix's
* ToolRegistry.dispatch returns. We:
* - join text blocks with newlines (most common case)
* - stringify image blocks as placeholders (LLM can't use bytes anyway
* in Reasonix's current surface; image support comes with multimodal
* prompts later)
* - prefix error results with "ERROR: " so the calling model sees the
* failure clearly even through JSON mode
* - optionally truncate to `maxChars` so a single oversized tool result
* (e.g. a big `read_file`) can't poison the session by blowing past
* the model's context window
*/
export function flattenMcpResult(result: CallToolResult, opts: FlattenOptions = {}): string {
const parts = result.content.map(blockToString);
const joined = parts.join("\n").trim();
const prefixed = result.isError ? `ERROR: ${joined || "(no error message from server)"}` : joined;
return opts.maxChars ? truncateForModel(prefixed, opts.maxChars) : prefixed;
}
/**
* Keep the head AND a short tail so the model sees both "what the tool
* started returning" and "how it ended". Head-only loses file endings
* (e.g. an error message appended at the bottom of a stack trace); the
* 1KB tail window covers that while costing almost nothing. Exported for
* tests and reuse by non-MCP tool adapters that want the same policy.
*/
export function truncateForModel(s: string, maxChars: number): string {
if (s.length <= maxChars) return s;
const tailBudget = Math.min(1024, Math.floor(maxChars * 0.1));
const headBudget = Math.max(0, maxChars - tailBudget);
const head = s.slice(0, headBudget);
const tail = s.slice(-tailBudget);
const dropped = s.length - head.length - tail.length;
return `${head}\n\n[…truncated ${dropped} chars — raise BridgeOptions.maxResultChars, or call the tool with a narrower scope (filter, head, pagination)…]\n\n${tail}`;
}
function blockToString(block: McpContentBlock): string {
if (block.type === "text") return block.text;
if (block.type === "image") return `[image ${block.mimeType}, ${block.data.length} chars base64]`;
// Unknown block type — preserve for diagnostics.
return `[unknown block: ${JSON.stringify(block)}]`;
}
+72
View File
@@ -0,0 +1,72 @@
/**
* Split a shell-style command string into argv, respecting single and
* double quotes. Intended for parsing the user's `--mcp "cmd args..."`
* flag — NOT a full shell parser (no variable expansion, no subshells,
* no globs, no `&&` / pipes).
*
* The tradeoff: users with paths containing spaces need to quote them
* (e.g. `--mcp 'npx -y pkg "/my path/here"'`), which is how they'd
* already quote them at the shell level.
*
* Throws on unterminated quotes — better than silently dropping half
* the command.
*/
export function shellSplit(input: string): string[] {
const tokens: string[] = [];
let cur = "";
let quote: '"' | "'" | null = null;
let i = 0;
const s = input;
while (i < s.length) {
const ch = s[i]!;
if (quote) {
if (ch === quote) {
quote = null;
i++;
continue;
}
// backslash escapes inside double quotes only
if (ch === "\\" && quote === '"' && i + 1 < s.length) {
cur += s[i + 1];
i += 2;
continue;
}
cur += ch;
i++;
continue;
}
if (ch === '"' || ch === "'") {
quote = ch as '"' | "'";
i++;
continue;
}
// Backslash escape ONLY applies inside double quotes (handled above).
// Outside quotes, backslashes pass through literally — otherwise
// Windows paths like `C:\path\to\exe` get mangled. POSIX users who
// want to escape a space outside quotes can use single quotes instead.
if (ch === " " || ch === "\t") {
if (cur.length > 0) {
tokens.push(cur);
cur = "";
}
i++;
continue;
}
cur += ch;
i++;
}
if (quote) {
throw new Error(
`shellSplit: unterminated ${quote === '"' ? "double" : "single"} quote in input`,
);
}
if (cur.length > 0) tokens.push(cur);
return tokens;
}
+71
View File
@@ -0,0 +1,71 @@
/**
* Parse the `--mcp` CLI argument into a transport-tagged spec.
*
* Accepted forms:
* "name=command args..." → stdio, namespaced (tools prefixed with `name_`)
* "command args..." → stdio, anonymous
* "name=https://host/sse" → SSE, namespaced
* "https://host/sse" → SSE, anonymous
* ("http://" is also honored — useful for local dev servers.)
*
* The identifier regex before `=` is deliberately narrow
* (`[a-zA-Z_][a-zA-Z0-9_]*`) so Windows drive letters ("C:\\...") and
* other strings containing `=` or `:` don't accidentally trigger the
* namespace branch. If a user ever wants their command to literally start
* with `foo=...` as a bare command, they can wrap it in quotes inside the
* shell command string.
*
* Transport is selected solely by whether the body begins with `http://`
* or `https://`. Anything else is stdio — including ws:// (unsupported)
* which will surface later as a spawn error, keeping the rule local.
*/
import { shellSplit } from "./shell-split.js";
export interface StdioMcpSpec {
transport: "stdio";
/** Namespace prefix applied to each registered tool, or null if anonymous. */
name: string | null;
/** Argv[0]. */
command: string;
/** Remaining argv. */
args: string[];
}
export interface SseMcpSpec {
transport: "sse";
name: string | null;
/** Fully qualified SSE endpoint URL. */
url: string;
}
export type McpSpec = StdioMcpSpec | SseMcpSpec;
const NAME_PREFIX = /^([a-zA-Z_][a-zA-Z0-9_]*)=(.*)$/;
const HTTP_URL = /^https?:\/\//i;
export function parseMcpSpec(input: string): McpSpec {
const trimmed = input.trim();
if (!trimmed) {
throw new Error("empty MCP spec");
}
const nameMatch = NAME_PREFIX.exec(trimmed);
const name = nameMatch ? nameMatch[1]! : null;
const body = (nameMatch ? nameMatch[2]! : trimmed).trim();
if (!body) {
throw new Error(`MCP spec has name but no command: ${input}`);
}
if (HTTP_URL.test(body)) {
return { transport: "sse", name, url: body };
}
const argv = shellSplit(body);
if (argv.length === 0) {
throw new Error(`MCP spec has name but no command: ${input}`);
}
const [command, ...args] = argv;
return { transport: "stdio", name, command: command!, args };
}
+194
View File
@@ -0,0 +1,194 @@
/**
* HTTP+SSE transport for MCP (spec version 2024-11-05).
*
* Wire shape:
* 1. Client opens GET to the SSE URL with `Accept: text/event-stream`.
* 2. Server's first SSE event is `event: endpoint`, `data: <url>` — the
* URL (relative or absolute) the client must POST JSON-RPC requests
* to. All subsequent server → client messages arrive as `event: message`
* SSE frames carrying a JSON-RPC response or server-initiated frame.
* 3. Client POSTs each outgoing JSON-RPC frame to the endpoint URL.
* The POST response body is ignored — replies land on the SSE stream.
*
* This transport exists so Reasonix can talk to hosted/remote MCP servers
* (e.g. a company's internal knowledge server fronted by auth). Stdio
* covers local subprocesses; SSE covers everything else.
*
* Note: the newer "Streamable HTTP" transport (2025 spec) folds the POST
* and SSE streams onto a single endpoint. We stay on 2024-11-05 here —
* that's what `MCP_PROTOCOL_VERSION` advertises in the initialize handshake
* and what currently-published servers implement.
*/
import { createParser } from "eventsource-parser";
import type { McpTransport } from "./stdio.js";
import type { JsonRpcMessage } from "./types.js";
export interface SseTransportOptions {
/** SSE endpoint URL, e.g. `https://mcp.example.com/sse`. */
url: string;
/** Extra headers sent on both the SSE GET and the JSON-RPC POSTs (e.g. `Authorization`). */
headers?: Record<string, string>;
}
/**
* Open an SSE stream to `url`, parse incoming events into JsonRpcMessages,
* POST outgoing frames to the endpoint URL the server advertises.
*/
export class SseTransport implements McpTransport {
private readonly url: string;
private readonly headers: Record<string, string>;
private readonly queue: JsonRpcMessage[] = [];
private readonly waiters: Array<(m: JsonRpcMessage | null) => void> = [];
private readonly controller = new AbortController();
private closed = false;
private postUrl: string | null = null;
private readonly endpointReady: Promise<string>;
private resolveEndpoint!: (url: string) => void;
private rejectEndpoint!: (err: Error) => void;
constructor(opts: SseTransportOptions) {
this.url = opts.url;
this.headers = opts.headers ?? {};
this.endpointReady = new Promise<string>((resolve, reject) => {
this.resolveEndpoint = resolve;
this.rejectEndpoint = reject;
});
// Swallow unhandled-rejection noise if nobody ever calls send().
this.endpointReady.catch(() => undefined);
void this.runStream();
}
async send(message: JsonRpcMessage): Promise<void> {
if (this.closed) throw new Error("MCP SSE transport is closed");
const postUrl = await this.endpointReady;
const res = await fetch(postUrl, {
method: "POST",
headers: { "content-type": "application/json", ...this.headers },
body: JSON.stringify(message),
signal: this.controller.signal,
});
// Drain body so the socket returns to the pool even if the server
// elected to write one. We explicitly don't parse it — responses
// arrive on the SSE channel.
await res.arrayBuffer().catch(() => undefined);
if (!res.ok) {
throw new Error(`MCP SSE POST ${postUrl} failed: ${res.status} ${res.statusText}`);
}
}
async *messages(): AsyncIterableIterator<JsonRpcMessage> {
while (true) {
if (this.queue.length > 0) {
yield this.queue.shift()!;
continue;
}
if (this.closed) return;
const next = await new Promise<JsonRpcMessage | null>((resolve) => {
this.waiters.push(resolve);
});
if (next === null) return;
yield next;
}
}
async close(): Promise<void> {
if (this.closed) return;
this.closed = true;
while (this.waiters.length > 0) this.waiters.shift()!(null);
// Reject any still-pending send() that was waiting for the endpoint.
this.rejectEndpoint(new Error("MCP SSE transport closed before endpoint was ready"));
try {
this.controller.abort();
} catch {
/* already aborted */
}
}
// ---------- internals ----------
private async runStream(): Promise<void> {
let res: Response;
try {
res = await fetch(this.url, {
method: "GET",
headers: { accept: "text/event-stream", ...this.headers },
signal: this.controller.signal,
});
} catch (err) {
this.failHandshake(`SSE connect to ${this.url} failed: ${(err as Error).message}`);
return;
}
if (!res.ok || !res.body) {
// Drain body to free the socket before giving up.
await res.body?.cancel().catch(() => undefined);
this.failHandshake(`SSE handshake ${this.url} → ${res.status} ${res.statusText}`);
return;
}
const parser = createParser({
onEvent: (ev) => this.handleEvent(ev.event ?? "message", ev.data),
});
const decoder = new TextDecoder();
try {
for await (const chunk of res.body as AsyncIterable<Uint8Array>) {
parser.feed(decoder.decode(chunk, { stream: true }));
}
} catch (err) {
if (!this.closed) {
this.pushError(`SSE stream error: ${(err as Error).message}`);
}
} finally {
this.markClosed();
}
}
private handleEvent(type: string, data: string): void {
if (type === "endpoint") {
if (this.postUrl) return; // ignore repeat announcements
try {
this.postUrl = new URL(data, this.url).toString();
this.resolveEndpoint(this.postUrl);
} catch (err) {
this.failHandshake(`SSE endpoint event had bad URL "${data}": ${(err as Error).message}`);
}
return;
}
if (type === "message") {
try {
const parsed = JSON.parse(data) as JsonRpcMessage;
this.pushMessage(parsed);
} catch {
// Malformed JSON-RPC on an SSE frame — drop it, same as stdio.
}
return;
}
// Unknown event types (server pings, custom extensions) — ignore.
}
private failHandshake(reason: string): void {
this.rejectEndpoint(new Error(reason));
this.pushError(reason);
this.markClosed();
}
private pushMessage(msg: JsonRpcMessage): void {
const waiter = this.waiters.shift();
if (waiter) waiter(msg);
else this.queue.push(msg);
}
private pushError(message: string): void {
this.pushMessage({
jsonrpc: "2.0",
id: null,
error: { code: -32000, message },
});
}
private markClosed(): void {
if (this.closed) return;
this.closed = true;
while (this.waiters.length > 0) this.waiters.shift()!(null);
}
}
+193
View File
@@ -0,0 +1,193 @@
/**
* Stdio transport for MCP.
*
* MCP's stdio wire format is **newline-delimited JSON** (one JSON-RPC
* message per line). We spawn the server as a child process, write
* frames to its stdin, parse its stdout line-by-line as they arrive.
*
* Transport is abstracted behind an interface so unit tests can fake it
* with an in-process duplex pair — spawning real servers in unit tests
* is flaky and slow.
*/
import { type ChildProcess, spawn } from "node:child_process";
import type { JsonRpcMessage } from "./types.js";
/**
* A transport sends JSON-RPC messages upstream and surfaces messages
* arriving downstream via an async iterator. One instance per server
* connection.
*/
export interface McpTransport {
/** Send one JSON-RPC message. Resolves when the bytes are accepted. */
send(message: JsonRpcMessage): Promise<void>;
/** Async iterator over incoming messages. Ends when the connection closes. */
messages(): AsyncIterableIterator<JsonRpcMessage>;
/** Close the underlying resource (kill child process, close streams). */
close(): Promise<void>;
}
export interface StdioTransportOptions {
/** Argv to spawn. First element is the command. */
command: string;
args?: string[];
/** Env overlay — merged over process.env unless replaceEnv=true. */
env?: Record<string, string>;
/** When true, only the env above is visible to the child. Default false. */
replaceEnv?: boolean;
/** CWD for the child. Default: process.cwd(). */
cwd?: string;
/**
* Spawn through a shell. Default: true on win32 (needed to resolve
* `.cmd` wrappers like `npx.cmd`, `pnpm.cmd`), false elsewhere.
* Explicitly pass `false` to opt out on Windows; pass `true` to force
* it on POSIX (rarely needed).
*/
shell?: boolean;
}
/**
* Spawn `command args...` as a child process and use its stdin/stdout as
* an MCP transport. Stderr is forwarded to the parent's stderr so server
* diagnostics are still visible.
*/
export class StdioTransport implements McpTransport {
private readonly child: ChildProcess;
private readonly queue: JsonRpcMessage[] = [];
private readonly waiters: Array<(m: JsonRpcMessage | null) => void> = [];
private closed = false;
private stdoutBuffer = "";
constructor(opts: StdioTransportOptions) {
const env = opts.replaceEnv ? { ...(opts.env ?? {}) } : { ...process.env, ...(opts.env ?? {}) };
// Windows wraps binaries as .cmd/.bat shims (npx.cmd, pnpm.cmd, …).
// child_process.spawn without shell:true can't resolve them, which
// breaks `--mcp "npx -y some-server"` — the most common MCP setup.
// Default shell:true on win32 and leave POSIX alone.
const shell = opts.shell ?? process.platform === "win32";
if (shell) {
// Node's shell:true + args[] triggers DEP0190 because it concatenates
// with spaces and doesn't quote args — unsafe if an arg contains
// shell metacharacters. We build a single command line ourselves,
// quoting ONLY the args (command stays bare so the shell's PATH /
// PATHEXT lookup finds `npx` → `npx.cmd` on Windows).
const line = [
opts.command,
...(opts.args ?? []).map((a) => quoteArg(a, process.platform === "win32")),
].join(" ");
this.child = spawn(line, [], {
env,
cwd: opts.cwd,
stdio: ["pipe", "pipe", "inherit"],
shell: true,
});
} else {
this.child = spawn(opts.command, opts.args ?? [], {
env,
cwd: opts.cwd,
stdio: ["pipe", "pipe", "inherit"],
});
}
this.child.stdout!.setEncoding("utf8");
this.child.stdout!.on("data", (chunk: string) => this.onStdout(chunk));
this.child.on("close", () => this.onClose());
this.child.on("error", (err) => {
// Surface spawn errors as a synthetic JsonRpcError so callers don't
// hang on a stream that never emits anything.
this.push({
jsonrpc: "2.0",
id: null,
error: { code: -32000, message: `transport error: ${err.message}` },
});
});
}
async send(message: JsonRpcMessage): Promise<void> {
if (this.closed) throw new Error("MCP transport is closed");
return new Promise((resolve, reject) => {
const line = `${JSON.stringify(message)}\n`;
this.child.stdin!.write(line, "utf8", (err) => {
if (err) reject(err);
else resolve();
});
});
}
async *messages(): AsyncIterableIterator<JsonRpcMessage> {
while (true) {
if (this.queue.length > 0) {
yield this.queue.shift()!;
continue;
}
if (this.closed) return;
const next = await new Promise<JsonRpcMessage | null>((resolve) => {
this.waiters.push(resolve);
});
if (next === null) return; // closed while we were waiting
yield next;
}
}
async close(): Promise<void> {
if (this.closed) return;
this.closed = true;
// Signal any pending waiters.
while (this.waiters.length > 0) this.waiters.shift()!(null);
try {
this.child.stdin!.end();
} catch {
/* already ended */
}
if (this.child.exitCode === null && !this.child.killed) {
this.child.kill("SIGTERM");
}
}
/** Parse incoming stdout chunks into NDJSON messages. */
private onStdout(chunk: string): void {
this.stdoutBuffer += chunk;
let newlineIdx: number;
// biome-ignore lint/suspicious/noAssignInExpressions: idiomatic loop shape
while ((newlineIdx = this.stdoutBuffer.indexOf("\n")) !== -1) {
const line = this.stdoutBuffer.slice(0, newlineIdx).trim();
this.stdoutBuffer = this.stdoutBuffer.slice(newlineIdx + 1);
if (!line) continue;
try {
const msg = JSON.parse(line) as JsonRpcMessage;
this.push(msg);
} catch {
// Malformed lines are dropped — some servers emit startup banners
// before the JSON-RPC loop begins. We surface the noise to stderr
// via the inherited stderr stream, not our event queue.
}
}
}
private onClose(): void {
this.closed = true;
while (this.waiters.length > 0) this.waiters.shift()!(null);
}
private push(msg: JsonRpcMessage): void {
const waiter = this.waiters.shift();
if (waiter) waiter(msg);
else this.queue.push(msg);
}
}
/**
* Quote a single argument for inclusion in a shell command line.
* On Windows (cmd.exe): wrap in double quotes, escape internal `"` as `""`,
* leave everything else alone. On POSIX: wrap in single quotes, escape
* internal `'` as `'\''`. Both handle spaces, wildcards, pipes, and all
* other metacharacters correctly.
*/
function quoteArg(s: string, windows: boolean): string {
if (!windows) {
// POSIX: single-quote, escape single quotes.
return `'${s.replace(/'/g, "'\\''")}'`;
}
// cmd.exe: double-quote, escape internal quotes by doubling.
return `"${s.replace(/"/g, '""')}"`;
}
+144
View File
@@ -0,0 +1,144 @@
/**
* MCP (Model Context Protocol) type definitions.
*
* Hand-rolled rather than importing @modelcontextprotocol/sdk because:
* - Reasonix's value-add isn't reimplementing the protocol, but *caching*
* it. Owning the types lets us tune them for our integration (strip
* fields we don't use, add the ones we do like Reasonix's prefixHash).
* - Zero dependencies — consistent with how we wrote the DeepSeek client.
* - If Anthropic bumps the SDK and introduces a breaking change, we're
* insulated as long as we keep up with the spec itself.
*
* Spec reference: https://spec.modelcontextprotocol.io/ (2024-11-05 draft
* at time of writing). Only the subset Reasonix consumes is modeled here —
* tools list/call + init handshake. Resources and prompts are deferred.
*
* Transport note: the wire format for stdio MCP is **newline-delimited
* JSON** (NDJSON), not the LSP-style Content-Length header framing that
* some readers might expect. One JSON-RPC message per line.
*/
// ---------- JSON-RPC 2.0 base ----------
export type JsonRpcId = string | number;
export interface JsonRpcRequest<P = unknown> {
jsonrpc: "2.0";
id: JsonRpcId;
method: string;
params?: P;
}
export interface JsonRpcNotification<P = unknown> {
jsonrpc: "2.0";
method: string;
params?: P;
}
export interface JsonRpcSuccess<R = unknown> {
jsonrpc: "2.0";
id: JsonRpcId;
result: R;
}
export interface JsonRpcError {
jsonrpc: "2.0";
id: JsonRpcId | null;
error: {
/** JSON-RPC standard codes: -32700 parse, -32600 invalid request, -32601 method not found, -32602 invalid params, -32603 internal. MCP also defines its own range. */
code: number;
message: string;
data?: unknown;
};
}
export type JsonRpcResponse<R = unknown> = JsonRpcSuccess<R> | JsonRpcError;
export type JsonRpcMessage = JsonRpcRequest | JsonRpcNotification | JsonRpcSuccess | JsonRpcError;
// ---------- MCP initialize ----------
export interface McpClientInfo {
name: string;
version: string;
}
export interface McpClientCapabilities {
/** Empty object advertises support without any optional sub-features. */
tools?: Record<string, never>;
// resources / prompts / sampling would go here — deferred.
}
export interface InitializeParams {
protocolVersion: string;
capabilities: McpClientCapabilities;
clientInfo: McpClientInfo;
}
export interface InitializeResult {
protocolVersion: string;
serverInfo: { name: string; version: string };
capabilities: {
tools?: { listChanged?: boolean };
resources?: unknown;
prompts?: unknown;
};
instructions?: string;
}
// ---------- MCP tools ----------
export interface McpToolSchema {
/** JSON Schema — compatible with Reasonix's tools.ts JSONSchema shape. */
type?: string;
properties?: Record<string, unknown>;
required?: string[];
[extra: string]: unknown;
}
export interface McpTool {
name: string;
description?: string;
/** MCP calls this `inputSchema`. Reasonix's `parameters` field is the same concept. */
inputSchema: McpToolSchema;
}
export interface ListToolsResult {
tools: McpTool[];
nextCursor?: string;
}
export interface CallToolParams {
name: string;
arguments?: Record<string, unknown>;
}
export interface McpContentBlockText {
type: "text";
text: string;
}
export interface McpContentBlockImage {
type: "image";
data: string;
mimeType: string;
}
/** MCP result content is an array of typed blocks. Reasonix consumes only text for now — image blocks get stringified with a placeholder. */
export type McpContentBlock = McpContentBlockText | McpContentBlockImage;
export interface CallToolResult {
content: McpContentBlock[];
/** True = tool raised an error; the content describes it. */
isError?: boolean;
}
// ---------- convenience ----------
/** Current MCP protocol version Reasonix is coded against. */
export const MCP_PROTOCOL_VERSION = "2024-11-05";
/** Type guard — success vs error response. */
export function isJsonRpcError(msg: JsonRpcResponse): msg is JsonRpcError {
return "error" in msg;
}
+87
View File
@@ -0,0 +1,87 @@
import { createHash } from "node:crypto";
import type { ChatMessage, ToolSpec } from "./types.js";
export interface ImmutablePrefixOptions {
system: string;
toolSpecs?: readonly ToolSpec[];
fewShots?: readonly ChatMessage[];
}
export class ImmutablePrefix {
readonly system: string;
readonly toolSpecs: readonly ToolSpec[];
readonly fewShots: readonly ChatMessage[];
constructor(opts: ImmutablePrefixOptions) {
this.system = opts.system;
this.toolSpecs = Object.freeze([...(opts.toolSpecs ?? [])]);
this.fewShots = Object.freeze([...(opts.fewShots ?? [])]);
}
toMessages(): ChatMessage[] {
return [{ role: "system", content: this.system }, ...this.fewShots.map((m) => ({ ...m }))];
}
tools(): ToolSpec[] {
return this.toolSpecs.map((t) => structuredClone(t) as ToolSpec);
}
get fingerprint(): string {
const blob = JSON.stringify({
system: this.system,
tools: this.toolSpecs,
shots: this.fewShots,
});
return createHash("sha256").update(blob).digest("hex").slice(0, 16);
}
}
export class AppendOnlyLog {
private _entries: ChatMessage[] = [];
append(message: ChatMessage): void {
if (!message || typeof message !== "object" || !("role" in message)) {
throw new Error(`invalid log entry: ${JSON.stringify(message)}`);
}
this._entries.push(message);
}
extend(messages: ChatMessage[]): void {
for (const m of messages) this.append(m);
}
/**
* Bulk-replace entries. Intentionally named to be hard to reach for —
* this is the one mutation path that breaks the log's append-only
* spirit, reserved for compaction flows (`/compact`) and recovery
* where the caller has consciously decided to drop old history. Any
* other use is almost certainly wrong; append() is what you want.
*/
compactInPlace(replacement: ChatMessage[]): void {
this._entries = [...replacement];
}
get entries(): readonly ChatMessage[] {
return this._entries;
}
toMessages(): ChatMessage[] {
return this._entries.map((e) => ({ ...e }));
}
get length(): number {
return this._entries.length;
}
}
export class VolatileScratch {
reasoning: string | null = null;
planState: Record<string, unknown> | null = null;
notes: string[] = [];
reset(): void {
this.reasoning = null;
this.planState = null;
this.notes = [];
}
}
+101
View File
@@ -0,0 +1,101 @@
/**
* Schema flattening for DeepSeek tool calls.
*
* DeepSeek loses arguments on schemas that are deep (>2 levels of nesting) or
* wide (>10 leaf parameters). This module transforms such schemas into a
* dot-notation flat schema and re-nests the model's arguments before dispatch.
*
* Example:
* { user: { profile: { name, age } } } ⇄ "user.profile.name", "user.profile.age"
*/
import type { JSONSchema } from "../types.js";
export interface FlattenDecision {
shouldFlatten: boolean;
leafCount: number;
maxDepth: number;
}
export function analyzeSchema(schema: JSONSchema | undefined): FlattenDecision {
if (!schema) return { shouldFlatten: false, leafCount: 0, maxDepth: 0 };
let leafCount = 0;
let maxDepth = 0;
walk(schema, 0, (depth, isLeaf) => {
if (isLeaf) leafCount++;
if (depth > maxDepth) maxDepth = depth;
});
return {
shouldFlatten: leafCount > 10 || maxDepth > 2,
leafCount,
maxDepth,
};
}
export function flattenSchema(schema: JSONSchema): JSONSchema {
const flatProps: Record<string, JSONSchema> = {};
const required: string[] = [];
collect("", schema, flatProps, required, true);
return {
type: "object",
properties: flatProps,
required,
};
}
export function nestArguments(flatArgs: Record<string, unknown>): Record<string, unknown> {
const out: Record<string, unknown> = {};
for (const [key, value] of Object.entries(flatArgs)) {
setByPath(out, key.split("."), value);
}
return out;
}
function walk(
schema: JSONSchema,
depth: number,
visit: (depth: number, isLeaf: boolean) => void,
): void {
if (schema.type === "object" && schema.properties) {
for (const child of Object.values(schema.properties)) {
walk(child, depth + 1, visit);
}
return;
}
if (schema.type === "array" && schema.items) {
walk(schema.items, depth + 1, visit);
return;
}
visit(depth, true);
}
function collect(
prefix: string,
schema: JSONSchema,
out: Record<string, JSONSchema>,
required: string[],
isRootRequired: boolean,
): void {
if (schema.type === "object" && schema.properties) {
const requiredSet = new Set(schema.required ?? []);
for (const [key, child] of Object.entries(schema.properties)) {
const nextPrefix = prefix ? `${prefix}.${key}` : key;
const childRequired = isRootRequired && requiredSet.has(key);
collect(nextPrefix, child, out, required, childRequired);
}
return;
}
// Treat anything non-object (including arrays) as a leaf for flattening purposes.
out[prefix] = schema;
if (isRootRequired) required.push(prefix);
}
function setByPath(target: Record<string, unknown>, path: string[], value: unknown): void {
let cur: any = target;
for (let i = 0; i < path.length - 1; i++) {
const key = path[i]!;
if (typeof cur[key] !== "object" || cur[key] === null) cur[key] = {};
cur = cur[key];
}
cur[path[path.length - 1]!] = value;
}
+105
View File
@@ -0,0 +1,105 @@
/**
* Pillar 3 — Tool-Call Repair pipeline.
*
* Order of passes per turn:
* 1. scavenge — recover tool calls leaked into <think>
* 2. truncation — close any half-emitted argument JSON
* 3. storm breaker — drop call-storm repeats
*
* Schema flattening is applied during loop construction (it changes what we
* advertise to the model), not per-turn.
*/
import type { ToolCall } from "../types.js";
import { scavengeToolCalls } from "./scavenge.js";
import { StormBreaker } from "./storm.js";
import { repairTruncatedJson } from "./truncation.js";
export { analyzeSchema, flattenSchema, nestArguments } from "./flatten.js";
export type { FlattenDecision } from "./flatten.js";
export { repairTruncatedJson } from "./truncation.js";
export type { TruncationRepairResult } from "./truncation.js";
export { scavengeToolCalls } from "./scavenge.js";
export type { ScavengeOptions, ScavengeResult } from "./scavenge.js";
export { StormBreaker } from "./storm.js";
export interface RepairReport {
scavenged: number;
truncationsFixed: number;
stormsBroken: number;
notes: string[];
}
export interface ToolCallRepairOptions {
allowedToolNames: ReadonlySet<string>;
stormWindow?: number;
stormThreshold?: number;
maxScavenge?: number;
}
export class ToolCallRepair {
private readonly storm: StormBreaker;
private readonly opts: ToolCallRepairOptions;
constructor(opts: ToolCallRepairOptions) {
this.opts = opts;
this.storm = new StormBreaker(opts.stormWindow ?? 6, opts.stormThreshold ?? 3);
}
process(
declaredCalls: ToolCall[],
reasoningContent: string | null,
): { calls: ToolCall[]; report: RepairReport } {
const report: RepairReport = {
scavenged: 0,
truncationsFixed: 0,
stormsBroken: 0,
notes: [],
};
// 1. Scavenge — only add calls whose (name,args) signature is novel.
const scavenged = scavengeToolCalls(reasoningContent, {
allowedNames: this.opts.allowedToolNames,
maxCalls: this.opts.maxScavenge ?? 4,
});
const seenSignatures = new Set(declaredCalls.map(signature));
const merged = [...declaredCalls];
for (const sc of scavenged.calls) {
if (!seenSignatures.has(signature(sc))) {
merged.push(sc);
report.scavenged++;
seenSignatures.add(signature(sc));
}
}
report.notes.push(...scavenged.notes);
// 2. Truncation repair on argument JSON.
for (const call of merged) {
const args = call.function?.arguments ?? "";
const r = repairTruncatedJson(args);
if (r.changed) {
call.function.arguments = r.repaired;
report.truncationsFixed++;
report.notes.push(...r.notes.map((n) => `[${call.function.name}] ${n}`));
}
}
// 3. Storm breaker.
const filtered: ToolCall[] = [];
for (const call of merged) {
const verdict = this.storm.inspect(call);
if (verdict.suppress) {
report.stormsBroken++;
if (verdict.reason) report.notes.push(verdict.reason);
continue;
}
filtered.push(call);
}
return { calls: filtered, report };
}
}
function signature(call: ToolCall): string {
return `${call.function?.name ?? ""}::${call.function?.arguments ?? ""}`;
}
+130
View File
@@ -0,0 +1,130 @@
/**
* Scavenge tool calls leaked into reasoning_content.
*
* R1 sometimes emits tool-call JSON inside <think>…</think> and then forgets
* to surface it in `tool_calls`. This pass extracts plausible calls and
* proposes them to the loop, which decides whether to merge them with the
* declared calls.
*/
import type { ToolCall } from "../types.js";
export interface ScavengeOptions {
/** Names of tools the model may legitimately call. Other names are ignored. */
allowedNames: ReadonlySet<string>;
/** Maximum number of calls to scavenge per pass (defence against runaway). */
maxCalls?: number;
}
export interface ScavengeResult {
calls: ToolCall[];
notes: string[];
}
export function scavengeToolCalls(
reasoningContent: string | null | undefined,
opts: ScavengeOptions,
): ScavengeResult {
if (!reasoningContent) return { calls: [], notes: [] };
const max = opts.maxCalls ?? 4;
const notes: string[] = [];
const out: ToolCall[] = [];
for (const candidate of iterateJsonObjects(reasoningContent)) {
if (out.length >= max) break;
const call = coerceToToolCall(candidate, opts.allowedNames);
if (call) {
out.push(call);
notes.push(`scavenged call: ${call.function.name}`);
}
}
return { calls: out, notes };
}
/** Yield every top-level JSON object substring in `text`. */
function* iterateJsonObjects(text: string): Generator<string> {
for (let i = 0; i < text.length; i++) {
if (text[i] !== "{") continue;
let depth = 0;
let inString = false;
let escaped = false;
for (let j = i; j < text.length; j++) {
const c = text[j]!;
if (escaped) {
escaped = false;
continue;
}
if (inString) {
if (c === "\\") {
escaped = true;
continue;
}
if (c === '"') inString = false;
continue;
}
if (c === '"') inString = true;
else if (c === "{") depth++;
else if (c === "}") {
depth--;
if (depth === 0) {
yield text.slice(i, j + 1);
i = j;
break;
}
}
}
}
}
function coerceToToolCall(
candidateJson: string,
allowedNames: ReadonlySet<string>,
): ToolCall | null {
let parsed: any;
try {
parsed = JSON.parse(candidateJson);
} catch {
return null;
}
if (!parsed || typeof parsed !== "object") return null;
// Pattern 1: { name, arguments }
if (typeof parsed.name === "string" && allowedNames.has(parsed.name)) {
const args = parsed.arguments;
return {
function: {
name: parsed.name,
arguments: typeof args === "string" ? args : JSON.stringify(args ?? {}),
},
};
}
// Pattern 2: OpenAI-style { type: "function", function: { name, arguments } }
if (
parsed.type === "function" &&
parsed.function &&
typeof parsed.function.name === "string" &&
allowedNames.has(parsed.function.name)
) {
const args = parsed.function.arguments;
return {
type: "function",
function: {
name: parsed.function.name,
arguments: typeof args === "string" ? args : JSON.stringify(args ?? {}),
},
};
}
// Pattern 3: { tool_name, tool_args } (R1 free-form variant)
if (typeof parsed.tool_name === "string" && allowedNames.has(parsed.tool_name)) {
return {
function: {
name: parsed.tool_name,
arguments: JSON.stringify(parsed.tool_args ?? {}),
},
};
}
return null;
}
+47
View File
@@ -0,0 +1,47 @@
import type { ToolCall } from "../types.js";
/**
* Call-storm breaker.
*
* Detects (tool, args) tuples repeating within a sliding window and suppresses
* the offending call. Surfaces a synthetic tool_result advising the model to
* change strategy on its next turn.
*/
export class StormBreaker {
private readonly windowSize: number;
private readonly threshold: number;
private readonly recent: Array<readonly [string, string]> = [];
constructor(windowSize = 6, threshold = 3) {
this.windowSize = windowSize;
this.threshold = threshold;
}
inspect(call: ToolCall): { suppress: boolean; reason?: string } {
const sig = signature(call);
if (!sig) return { suppress: false };
const count = this.recent.reduce(
(n, [name, args]) => (name === sig[0] && args === sig[1] ? n + 1 : n),
0,
);
if (count >= this.threshold - 1) {
return {
suppress: true,
reason: `call-storm suppressed: ${sig[0]} called with identical args ${count + 1} times within window=${this.windowSize}`,
};
}
this.recent.push(sig);
while (this.recent.length > this.windowSize) this.recent.shift();
return { suppress: false };
}
reset(): void {
this.recent.length = 0;
}
}
function signature(call: ToolCall): readonly [string, string] | null {
const name = call.function?.name;
if (!name) return null;
return [name, call.function?.arguments ?? ""] as const;
}
+97
View File
@@ -0,0 +1,97 @@
/**
* Truncation recovery for tool-call argument JSON cut off mid-structure
* (typically when the model hits max_tokens before finishing the JSON object).
*
* Strategy is purely local: balance braces, close strings, fill missing values
* with `null`. We deliberately do NOT make a continuation API call here — that
* decision belongs to the loop, which knows about budgets.
*/
export interface TruncationRepairResult {
repaired: string;
changed: boolean;
notes: string[];
}
export function repairTruncatedJson(input: string): TruncationRepairResult {
const notes: string[] = [];
if (!input || !input.trim()) {
return { repaired: "{}", changed: input !== "{}", notes: ["empty input → {}"] };
}
// Fast path: already parseable.
try {
JSON.parse(input);
return { repaired: input, changed: false, notes: [] };
} catch {
/* fall through */
}
const stack: ("{" | "[" | '"')[] = [];
let escaped = false;
let inString = false;
let lastSignificant = -1;
for (let i = 0; i < input.length; i++) {
const c = input[i]!;
if (!/\s/.test(c)) lastSignificant = i;
if (escaped) {
escaped = false;
continue;
}
if (inString) {
if (c === "\\") {
escaped = true;
continue;
}
if (c === '"') {
inString = false;
stack.pop();
}
continue;
}
if (c === '"') {
inString = true;
stack.push('"');
continue;
}
if (c === "{" || c === "[") stack.push(c);
else if (c === "}" || c === "]") stack.pop();
}
let s = input.slice(0, lastSignificant + 1);
// Trim a trailing comma which would block re-parse.
if (/,$/.test(s)) {
s = s.replace(/,$/, "");
notes.push("trimmed trailing comma");
}
// If we ended on a key without a value: "foo": → "foo": null
if (/"\s*:\s*$/.test(s)) {
s += " null";
notes.push("filled dangling key with null");
}
// If we ended inside a string, close it.
if (inString) {
s += '"';
stack.pop();
notes.push("closed unterminated string");
}
// Pop remaining open structures in reverse order.
while (stack.length > 0) {
const top = stack.pop();
if (top === "{") s += "}";
else if (top === "[") s += "]";
else if (top === '"') s += '"';
}
try {
JSON.parse(s);
return { repaired: s, changed: true, notes };
} catch (err) {
notes.push(`fallback to {}: ${(err as Error).message}`);
return { repaired: "{}", changed: true, notes };
}
}
Loaded 100 of 481 files, more files were not shown because too many files have changed in this diff. Show more