mirror of
https://github.com/esengine/DeepSeek-Reasonix.git
synced 2026-11-07 10:45:59 -10:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
27a32b7b94 | ||
|
|
1391534f6c | ||
|
|
169d66d46a | ||
|
|
2836b5d406 | ||
|
|
d0bd6852c1 | ||
|
|
cb2f2a550b | ||
|
|
e472f4cb79 | ||
|
|
14887264bb | ||
|
|
63ced2fa5b | ||
|
|
461e736cbc | ||
|
|
d932c50ab3 | ||
|
|
b4c461c6ad | ||
|
|
2f4e28b7cb | ||
|
|
5f715d05ee | ||
|
|
b662c63894 | ||
|
|
e793c7d59d | ||
|
|
7cd862194a | ||
|
|
407793c99e | ||
|
|
5e124cec36 | ||
|
|
7fce6e650c | ||
|
|
5941043489 | ||
|
|
f142f8874d | ||
|
|
fb5977743c | ||
|
|
26b50e9242 | ||
|
|
7233eddced | ||
|
|
3a92ba2787 | ||
|
|
6307b92af2 | ||
|
|
2e42ae53aa | ||
|
|
5ff0ef4535 | ||
|
|
ce963be2e2 | ||
|
|
e5357f0df5 | ||
|
|
c148b342fc | ||
|
|
74e796b1c6 | ||
|
|
c1965b1d1e | ||
|
|
725cb16ff8 | ||
|
|
ceb4840a63 | ||
|
|
4bca4cedd8 |
No files matched your search
@@ -37,3 +37,13 @@ jobs:
|
||||
|
||||
- name: Build (tsup)
|
||||
run: npm run build
|
||||
|
||||
# Smoke-test the bench harnesses themselves. --dry skips all LLM
|
||||
# calls, so this catches wiring regressions (task factories, CLI
|
||||
# parsing, file IO, checker determinism) without needing a
|
||||
# DEEPSEEK_API_KEY in CI.
|
||||
- name: τ-bench harness dry-run
|
||||
run: npx tsx benchmarks/tau-bench/runner.ts --dry --out /tmp/tau-dry.json
|
||||
|
||||
- name: Harvest-bench harness dry-run
|
||||
run: npx tsx benchmarks/harvest/runner.ts --dry --out /tmp/harvest-dry.json
|
||||
@@ -22,3 +22,5 @@ sessions/
|
||||
!benchmarks/tau-bench/transcripts/
|
||||
!benchmarks/tau-bench/transcripts/*.jsonl
|
||||
.turbo/
|
||||
# Local-only Claude Code settings — personal overrides, never committed.
|
||||
.claude/settings.local.json
|
||||
+1123
File diff suppressed because it is too large.
Load diff
@@ -6,20 +6,25 @@
|
||||
[](https://www.npmjs.com/package/reasonix)
|
||||
[](./package.json)
|
||||
|
||||
**The DeepSeek-native agent framework.** TypeScript. Ink TUI. No LangChain.
|
||||
|
||||
Reasonix is not another generic agent wrapper. Every abstraction is justified
|
||||
by a DeepSeek-specific property — dirt-cheap tokens, R1 reasoning traces,
|
||||
automatic prefix caching, JSON mode. Generic frameworks treat DeepSeek as
|
||||
"OpenAI with a different base URL" and leave these advantages on the table.
|
||||
Reasonix leans into them.
|
||||
**A DeepSeek-native AI coding assistant in your terminal.** Ink TUI. MCP
|
||||
first-class. No LangChain.
|
||||
|
||||
```bash
|
||||
npx reasonix chat # first run prompts for your DeepSeek key
|
||||
# inside the TUI, type /help for everything else
|
||||
npx reasonix
|
||||
```
|
||||
|
||||
No flag soup. All feature toggles live behind slash commands in the TUI.
|
||||
One command. First run walks you through a 30-second wizard (API key →
|
||||
preset → pick MCP servers from a checklist); every run after that drops
|
||||
straight into chat with your tools wired up. Inside the chat, type `/help`.
|
||||
|
||||
Why bother with yet another agent framework? Because every abstraction
|
||||
here earns its weight against a DeepSeek-specific property — dirt-cheap
|
||||
tokens, R1 reasoning traces, automatic prefix caching, JSON mode.
|
||||
Generic wrappers treat DeepSeek as "OpenAI with a different base URL"
|
||||
and leave these advantages on the table. Reasonix leans into them:
|
||||
on the same τ-bench-lite workload,
|
||||
[**94.4% cache hit, ~40% cheaper tokens, 100% pass rate**](#validated-numbers)
|
||||
vs. a cache-hostile baseline.
|
||||
|
||||
---
|
||||
|
||||
@@ -27,12 +32,15 @@ No flag soup. All feature toggles live behind slash commands in the TUI.
|
||||
|
||||
| Feature | How it works | Opt in |
|
||||
|---|---|---|
|
||||
| **Setup wizard** | First run of `npx reasonix`: pick preset, multi-select MCP servers from a curated catalog, saved to config so the next run just launches chat | always on (first run) |
|
||||
| **MCP (stdio + SSE)** | Multi-server bridge — every MCP tool inherits Cache-First + repair + context-safety automatically. `reasonix mcp list` shows the catalog | always on |
|
||||
| **Cache-First Loop** | Immutable prefix + append-only log = prefix byte-stable across turns → DeepSeek's automatic prefix cache hits at 70–95% | always on |
|
||||
| **R1 Thought Harvesting** | Parses `reasoning_content` into typed `{ subgoals, hypotheses, uncertainties, rejectedPaths }` via a cheap V3 call | `--harvest` |
|
||||
| **Self-Consistency Branching** | Runs N parallel samples at spread temperatures; picks the one with the fewest flagged uncertainties | `--branch <N>` |
|
||||
| **Context safety net** | Tool results capped at 32k chars · oversized sessions auto-heal on load · `/compact` to shrink further · ctx gauge in the status bar · Esc to abort exploration and get a forced summary | always on |
|
||||
| **R1 Thought Harvesting** | Parses `reasoning_content` into typed `{ subgoals, hypotheses, uncertainties, rejectedPaths }` via a cheap V3 call | `/preset smart` |
|
||||
| **Self-Consistency Branching** | Runs N parallel samples at spread temperatures; picks the one with the fewest flagged uncertainties | `/preset max` / `/branch N` |
|
||||
| **Tool-Call Repair** | Auto-flattens deep/wide schemas, scavenges tool calls leaked into `<think>`, repairs truncated JSON, breaks call-storms | always on |
|
||||
| **Retry layer** | Exponential backoff + jitter on 408/429/500/502/503/504 and network errors. 4xx auth errors don't retry | always on |
|
||||
| **Ink TUI** | Live cache-hit / cost panel. Streams R1 thinking to a compact preview. Renders Markdown (bold / lists / code / stripped LaTeX) | always on |
|
||||
| **Ink TUI** | Live cache-hit / cost / context panel. Streams R1 thinking to a compact preview. Renders Markdown (bold / lists / code / stripped LaTeX) | always on |
|
||||
|
||||
---
|
||||
|
||||
@@ -58,17 +66,17 @@ actually stays byte-stable.
|
||||
|
||||
## Validated numbers
|
||||
|
||||
**τ-bench-lite** — 8 multi-turn tool-use tasks, same tools / same prompt
|
||||
on both sides, sole variable is prefix stability. Measured on live
|
||||
DeepSeek `deepseek-chat`:
|
||||
**τ-bench-lite** — 8 multi-turn tool-use tasks × 3 repeats = 48 runs per
|
||||
side. Same tools / same prompt / same client on both sides, sole variable
|
||||
is prefix stability. Measured on live DeepSeek `deepseek-chat`:
|
||||
|
||||
| metric | baseline (cache-hostile) | Reasonix | delta |
|
||||
|---|---:|---:|---:|
|
||||
| runs | 8 | 8 | — |
|
||||
| **cache hit** | 43.9% | **94.3%** | **+50.3pp** |
|
||||
| cost / task | $0.002783 | $0.001621 | **−42% (×0.58)** |
|
||||
| runs | 24 | 24 | — |
|
||||
| **cache hit** | 46.6% | **94.4%** | **+47.7pp** |
|
||||
| cost / task | $0.002599 | $0.001579 | **−39% (×0.61)** |
|
||||
| vs Claude Sonnet 4.6 (token-count estimate) | — | — | **~96% cheaper** |
|
||||
| pass rate | 100% (8/8) | 88% (7/8) | 1 refusal-task predicate too strict (see [report.md][r]) |
|
||||
| pass rate | 96% (23/24) | **100% (24/24)** | Reasonix held the guardrail on every run |
|
||||
|
||||
**Verify it yourself — no API key, zero cost:**
|
||||
|
||||
@@ -86,64 +94,139 @@ stays byte-stable across every model call; baseline's prefix churns on
|
||||
every turn. The cache delta is *mechanically* attributable to log
|
||||
stability, not to a different system prompt.
|
||||
|
||||
Full 16-run report: [`benchmarks/tau-bench/report.md`][r]. Reproduce
|
||||
with your own API key: `npx tsx benchmarks/tau-bench/runner.ts`.
|
||||
Full 48-run report: [`benchmarks/tau-bench/report.md`][r]. Reproduce
|
||||
with your own API key: `npx tsx benchmarks/tau-bench/runner.ts --repeats 3`.
|
||||
|
||||
[r]: ./benchmarks/tau-bench/report.md
|
||||
|
||||
### MCP — works out of the box
|
||||
|
||||
Any [MCP](https://spec.modelcontextprotocol.io/) server's tools inherit
|
||||
Cache-First + repair + context-safety automatically. The wizard (`npx
|
||||
reasonix`) lets you multi-select from a curated catalog — no flags, no
|
||||
JSON-by-hand. Three live reference runs:
|
||||
|
||||
| server | turns | tool calls | cache hit | cost | vs Claude |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| bundled demo (`add` / `echo` / `get_time`) | 2 | 1 | **96.6%** (turn 2) | $0.000254 | −94.0% |
|
||||
| official `@modelcontextprotocol/server-filesystem` | 5 | 4 | **96.7%** overall | $0.001235 | −97.0% |
|
||||
| **both concurrently** (`demo_add` + `fs_write_file`) | 5 | 4 | **81.1%** | $0.001852 | −95.9% |
|
||||
|
||||
The third row is the ecosystem proof: two MCP servers running as
|
||||
separate subprocesses, tools from both exercised in one conversation.
|
||||
**One single prefix hash across all 5 turns** — byte-stability survives
|
||||
concurrent MCP subprocesses.
|
||||
|
||||
Reproduce without an API key (replay the committed transcripts):
|
||||
|
||||
```bash
|
||||
npx reasonix replay benchmarks/tau-bench/transcripts/mcp-demo.add.jsonl
|
||||
npx reasonix replay benchmarks/tau-bench/transcripts/mcp-filesystem.jsonl
|
||||
```
|
||||
|
||||
Supported transports: **stdio** (local `npx` or binary) and **HTTP+SSE**
|
||||
(remote / hosted servers, MCP 2024-11-05 spec). Pass an `http(s)://`
|
||||
URL to `--mcp` and Reasonix opens the SSE stream and POSTs JSON-RPC
|
||||
to the endpoint the server advertises.
|
||||
|
||||
[mcp]: ./benchmarks/tau-bench/transcripts/mcp-demo.add.jsonl
|
||||
|
||||
---
|
||||
|
||||
## Usage
|
||||
|
||||
### CLI
|
||||
### One command
|
||||
|
||||
```bash
|
||||
npx reasonix chat # auto-saves to session 'default'; resumes next time
|
||||
npx reasonix chat --session work # use a different named session
|
||||
npx reasonix chat --no-session # ephemeral — nothing persisted
|
||||
npx reasonix run "ask anything" # one-shot, streams to stdout
|
||||
npx reasonix stats session.jsonl # quick summary of a transcript
|
||||
npx reasonix replay chat.jsonl # pretty-print a transcript + rebuild cost/cache offline
|
||||
npx reasonix diff a.jsonl b.jsonl --md diff.md # compare two transcripts: cache/cost delta + first divergence
|
||||
npx reasonix
|
||||
```
|
||||
|
||||
Sessions live as JSONL under `~/.reasonix/sessions/<name>.jsonl` — every
|
||||
turn's message log is appended atomically, so killing the CLI never loses
|
||||
context. Inside the TUI: `/sessions` to list, `/forget` to delete the
|
||||
current one.
|
||||
First run: a wizard asks for your API key, lets you pick a preset
|
||||
(fast / smart / max), then offers a multi-select checklist of MCP
|
||||
servers — filesystem, memory, github, puppeteer, everything. Everything
|
||||
is saved to `~/.reasonix/config.json`. Subsequent runs drop straight
|
||||
into chat.
|
||||
|
||||
### Inside the chat — slash commands
|
||||
### Inside the chat
|
||||
|
||||
A command strip runs under the input box so you don't have to memorize
|
||||
anything. Type `/help` for the full list. The biggest shortcut:
|
||||
A status bar at the top shows cache hit %, cost, Claude-equivalent, and
|
||||
the **context gauge** (`ctx 42k/131k (32%)` — yellow at 50%, red + a
|
||||
`/compact` nudge at 80%). A command strip under the input lists the
|
||||
slash commands:
|
||||
|
||||
```
|
||||
/preset fast deepseek-chat, no harvest, no branch (default)
|
||||
/preset smart reasoner + harvest (~10x cost)
|
||||
/preset max reasoner + harvest + branch 3 (~30x cost, slowest)
|
||||
/help full list + hints
|
||||
/preset <fast|smart|max> one-tap bundles (model + harvest + branch)
|
||||
/mcp list attached MCP servers and tools
|
||||
/compact [cap] shrink oversized tool results in history
|
||||
/sessions · /forget list / delete saved sessions
|
||||
/setup reconfigure (exits and tells you to run `reasonix setup`)
|
||||
/clear · /exit
|
||||
```
|
||||
|
||||
One-tap switch between fast daily driver, careful thinker, and max-quality
|
||||
self-consistency. Individual knobs are available too:
|
||||
**Esc while thinking** — abort the current exploration and force the
|
||||
model to summarize what it already found. No more "model ran 24 tool
|
||||
calls and gave up" — you get an answer every time.
|
||||
|
||||
Sessions live as JSONL under `~/.reasonix/sessions/<name>.jsonl` —
|
||||
every message appended atomically, so killing the CLI never loses
|
||||
context. Oversized tool results auto-heal on load, so poisoning a
|
||||
session with one giant `read_file` doesn't brick your history.
|
||||
|
||||
### Code mode — `npx reasonix code`
|
||||
|
||||
A thin opinionated layer on top of chat: filesystem MCP bridged at
|
||||
`cwd`, coding system prompt, reasoner preset, per-directory session.
|
||||
The model proposes edits as **SEARCH/REPLACE blocks**:
|
||||
|
||||
```
|
||||
/status show current model / harvest / branch / stream
|
||||
/model <id> deepseek-chat or deepseek-reasoner
|
||||
/harvest [on|off] Pillar 2 — parse R1 reasoning into typed plan state
|
||||
/branch <N|off> run N parallel samples per turn, pick most confident
|
||||
/clear clear displayed history (log is kept)
|
||||
/exit quit
|
||||
src/foo.ts
|
||||
<<<<<<< SEARCH
|
||||
const x = 1;
|
||||
=======
|
||||
const x = 2;
|
||||
>>>>>>> REPLACE
|
||||
```
|
||||
|
||||
The top panel shows active flags live: `· harvest · branch3` appear next to
|
||||
the model once enabled.
|
||||
|
||||
### Flags (for automation / CI)
|
||||
|
||||
The same knobs are also available as CLI flags if you're scripting:
|
||||
Reasonix parses them out of each turn, applies them to disk, reports
|
||||
`✓ applied src/foo.ts` in the TUI. SEARCH must match byte-for-byte;
|
||||
we never fuzzy-match (silent wrong edits are worse than loud
|
||||
rejections). Run `git diff` to review, `git checkout .` to undo.
|
||||
|
||||
```bash
|
||||
npx reasonix chat -m deepseek-reasoner --harvest --branch 3 --transcript session.jsonl
|
||||
cd my-project
|
||||
npx reasonix code # default: cwd, reasoner, per-dir session
|
||||
npx reasonix code src/ # scope the filesystem sandbox tighter
|
||||
npx reasonix code --no-session # ephemeral
|
||||
```
|
||||
|
||||
First-run sandbox: because code mode uses the filesystem MCP from
|
||||
`@modelcontextprotocol/server-filesystem`, the model can only read
|
||||
and write inside the directory you pointed at. It literally can't
|
||||
touch files above that root.
|
||||
|
||||
### Advanced — CLI subcommands and flags
|
||||
|
||||
```bash
|
||||
npx reasonix setup # reconfigure any time
|
||||
npx reasonix chat --session work # a different named session
|
||||
npx reasonix chat --no-session # ephemeral — nothing persisted
|
||||
npx reasonix run "ask anything" # one-shot, streams to stdout
|
||||
npx reasonix stats session.jsonl # summarize a transcript
|
||||
npx reasonix replay chat.jsonl # scrub a transcript + rebuild cost/cache
|
||||
npx reasonix diff a.jsonl b.jsonl --md # compare two transcripts
|
||||
npx reasonix mcp list # curated MCP server catalog
|
||||
```
|
||||
|
||||
Power users can still bypass config and drive Reasonix with flags:
|
||||
|
||||
```bash
|
||||
npx reasonix chat \
|
||||
--preset max \
|
||||
--mcp "filesystem=npx -y @modelcontextprotocol/server-filesystem /tmp/safe" \
|
||||
--mcp "kb=https://mcp.example.com/sse" \
|
||||
--transcript session.jsonl \
|
||||
--no-config # ignore ~/.reasonix/config.json (for CI / reproducing issues)
|
||||
```
|
||||
|
||||
### Library
|
||||
@@ -189,16 +272,19 @@ console.log(loop.stats.summary());
|
||||
|
||||
### Configuration
|
||||
|
||||
On first run the CLI prompts for your DeepSeek API key and saves it to
|
||||
`~/.reasonix/config.json`. Alternatives:
|
||||
The wizard handles everything on first run. If you'd rather use env vars
|
||||
(CI, shared boxes, etc.):
|
||||
|
||||
```bash
|
||||
export DEEPSEEK_API_KEY=sk-... # env var (wins over config file)
|
||||
export DEEPSEEK_API_KEY=sk-... # wins over ~/.reasonix/config.json
|
||||
export DEEPSEEK_BASE_URL=https://... # optional alternate endpoint
|
||||
```
|
||||
|
||||
Get a key (free credit on signup): <https://platform.deepseek.com/api_keys>
|
||||
|
||||
Re-run `npx reasonix setup` any time to add/remove MCP servers or switch
|
||||
preset — your existing selections are pre-checked.
|
||||
|
||||
---
|
||||
|
||||
## Non-goals
|
||||
@@ -220,7 +306,7 @@ cd reasonix
|
||||
npm install
|
||||
npm run dev chat # run CLI from source via tsx
|
||||
npm run build # tsup to dist/
|
||||
npm test # vitest (89 tests)
|
||||
npm test # vitest (279 tests)
|
||||
npm run lint # biome
|
||||
npm run typecheck # tsc --noEmit
|
||||
```
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
# Harvest eval harness (Pillar 2)
|
||||
|
||||
Measures Pillar 2 (R1 Thought Harvesting) the same way `tau-bench/`
|
||||
measures Pillar 1 (Cache-First Loop): isolate one variable, produce
|
||||
numbers anyone can reproduce.
|
||||
|
||||
## What's different from τ-bench-lite?
|
||||
|
||||
Deliberately different task shape:
|
||||
|
||||
| | τ-bench-lite | harvest-bench |
|
||||
|---|---|---|
|
||||
| task style | multi-turn tool-use | single-turn reasoning |
|
||||
| user simulator | yes (LLM) | no |
|
||||
| DB state | yes | no |
|
||||
| checker | DB-predicate | regex + set/value compare |
|
||||
| target | Pillar 1 | Pillar 2 |
|
||||
| model | deepseek-chat (default) | deepseek-reasoner (only R1 has `reasoning_content` for harvest to work on) |
|
||||
|
||||
The two harnesses answer different questions:
|
||||
|
||||
- **τ-bench-lite**: *"Does Cache-First actually cut cost on real tool-use
|
||||
workflows?"* (yes — 47.7pp cache, −39% cost on 48-run data.)
|
||||
- **harvest-bench**: *"Does the extra V3 harvest call add measurable
|
||||
value above plain R1?"* (TBD — run it to find out.)
|
||||
|
||||
## Modes
|
||||
|
||||
Three modes isolate one variable each:
|
||||
|
||||
| mode | model | harvest | what it measures |
|
||||
|---|---|---|---|
|
||||
| `baseline` | deepseek-chat | off | floor reference — V3 at its best |
|
||||
| `reasoner` | deepseek-reasoner | off | raw R1 gain over V3 on reasoning |
|
||||
| `reasoner-harvest` | deepseek-reasoner | on | R1 + the extra V3 harvest call |
|
||||
|
||||
Deltas:
|
||||
|
||||
- `baseline → reasoner` answers "is R1 worth the price on these
|
||||
problems?"
|
||||
- `reasoner → reasoner-harvest` answers "is the harvest call worth
|
||||
its incremental cost?"
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
# Dry-run — no API, smoke-test the wiring
|
||||
npx tsx benchmarks/harvest/runner.ts --dry
|
||||
|
||||
# Full run (live DeepSeek, costs ~$0.20-0.60 for 6 tasks × 3 modes × 1 repeat)
|
||||
export DEEPSEEK_API_KEY=sk-...
|
||||
npx tsx benchmarks/harvest/runner.ts
|
||||
|
||||
# Tighter run with 3 repeats (~$0.60-2.00)
|
||||
npx tsx benchmarks/harvest/runner.ts --repeats 3
|
||||
|
||||
# Only the hard tasks (skip the easy floor)
|
||||
npx tsx benchmarks/harvest/runner.ts --task pseudoprime_base2
|
||||
npx tsx benchmarks/harvest/runner.ts --task derangements_d7
|
||||
npx tsx benchmarks/harvest/runner.ts --task euler_quadratic_break
|
||||
|
||||
# Bump per-call timeout (reasoner + harvest occasionally blow past the 120s default)
|
||||
npx tsx benchmarks/harvest/runner.ts --timeout 600
|
||||
|
||||
# Per-run transcripts so you can reasonix replay / diff them
|
||||
npx tsx benchmarks/harvest/runner.ts --repeats 3 --transcripts-dir transcripts/
|
||||
|
||||
# Render report
|
||||
npx tsx benchmarks/harvest/report.ts benchmarks/harvest/results-*.json
|
||||
```
|
||||
|
||||
## Tasks (v0.3 seed)
|
||||
|
||||
Split into two bands:
|
||||
|
||||
**Easy band** — V3 chat solves these, so baseline has a real pass rate and the bench has a reference floor.
|
||||
|
||||
| id | shape | why |
|
||||
|---|---|---|
|
||||
| `mod7_list` | number theory, 29-element list | R1 often tries enumeration first, then reaches for modular arithmetic — clean rejectedPaths signal |
|
||||
| `flips_until_3heads` | probability, single integer | classic recurrence; R1 either derives or recalls, harvest should see hypotheses diverge |
|
||||
| `three_hats` | logic puzzle, one-word answer | pure deduction chain, tests harvest's ability to extract the nested reasoning |
|
||||
|
||||
**Hard band** — picked specifically for known V3 failure modes. If R1 systematically beats V3 on these, and harvest further beats R1-only, we have our Pillar 2 story.
|
||||
|
||||
| id | shape | why V3 tends to fail |
|
||||
|---|---|---|
|
||||
| `pseudoprime_base2` | smallest Fermat pseudoprime to base 2 = **341** | V3 often answers 561 (Carmichael) or some other composite it remembers — R1 actually checks 2^n mod n |
|
||||
| `derangements_d7` | D_7 = **1854** | V3 approximates via n!/e and rounds wrong, or recalls D_6/D_8; R1 uses the recurrence D_n=(n−1)(D_{n−1}+D_{n−2}) |
|
||||
| `euler_quadratic_break` | smallest n where n²+n+41 is composite = **40** | V3 frequently confuses with n²−n+41 (first fails at 41); R1 checks small n systematically |
|
||||
|
||||
Adding a new task: see `tasks.ts`. Any checker that's deterministic is
|
||||
fair game — extract numeric/list/text with regex, compare with set
|
||||
equality or string matching.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- **No LLM-as-judge.** Brittle and expensive; defeats the point of a
|
||||
reproducible bench. If a checker is too hard to write deterministically,
|
||||
the task doesn't belong here.
|
||||
- **No tool-use tasks.** Those live in `tau-bench/`. Different story.
|
||||
- **No multi-turn.** v0.3 harness is single-turn Q/A. Multi-turn reasoning
|
||||
eval is separate scope.
|
||||
- **No benchmark-data cherry-picking.** When results come in, we publish
|
||||
them whether they validate Pillar 2 or not. "harvest didn't help on
|
||||
these 3 tasks" is still useful information.
|
||||
@@ -0,0 +1,82 @@
|
||||
# Reasonix harvest eval (Pillar 2)
|
||||
|
||||
**Date:** 2026-04-21T16:44:21.011Z
|
||||
**Tasks:** 6 · repeats × 1 · modes: baseline, reasoner, reasoner-harvest
|
||||
**Reasonix version:** 0.2.2
|
||||
|
||||
## Summary by mode
|
||||
|
||||
| mode | runs | pass rate | cache hit | cost / run | harvest turns | subgoals | uncertainties |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|
|
||||
| baseline | 6 | 100% | 39.5% | $0.001371 | 0.0 | 0.0 | 0.0 |
|
||||
| reasoner | 6 | 100% | 74.2% | $0.004170 | 0.0 | 0.0 | 0.0 |
|
||||
| reasoner-harvest | 6 | 83% | 61.6% | $0.003304 | 0.8 | 2.8 | 1.3 |
|
||||
|
||||
## Deltas
|
||||
|
||||
- **baseline → reasoner**
|
||||
- pass rate: 0pp
|
||||
- cost: ×3.04 (each run costs more)
|
||||
- harvest signal / run: 0.0 subgoals, 0.0 uncertainties
|
||||
|
||||
- **baseline → reasoner-harvest**
|
||||
- pass rate: -17pp
|
||||
- cost: ×2.41 (each run costs more)
|
||||
- harvest signal / run: 2.8 subgoals, 1.3 uncertainties
|
||||
|
||||
## Per-task breakdown
|
||||
|
||||
| task | mode | rep | verdict | cache | cost | sg | un | note |
|
||||
|---|---|---:|:---:|---:|---:|---:|---:|---|
|
||||
| mod7_list | baseline | 1 | ✅ | 86.5% | $0.001361 | 0 | 0 | |
|
||||
| mod7_list | reasoner | 1 | ✅ | 86.5% | $0.004797 | 0 | 0 | |
|
||||
| mod7_list | reasoner-harvest | 1 | ✅ | 86.5% | $0.003045 | 4 | 2 | |
|
||||
| flips_until_3heads | baseline | 1 | ✅ | 92.8% | $0.001198 | 0 | 0 | |
|
||||
| flips_until_3heads | reasoner | 1 | ✅ | 92.8% | $0.002830 | 0 | 0 | |
|
||||
| flips_until_3heads | reasoner-harvest | 1 | ✅ | 92.8% | $0.001691 | 4 | 0 | |
|
||||
| three_hats | baseline | 1 | ✅ | 57.7% | $0.001514 | 0 | 0 | |
|
||||
| three_hats | reasoner | 1 | ✅ | 57.7% | $0.009340 | 0 | 0 | |
|
||||
| three_hats | reasoner-harvest | 1 | ✅ | 57.7% | $0.009651 | 3 | 2 | |
|
||||
| pseudoprime_base2 | baseline | 1 | ✅ | 0.0% | $0.001894 | 0 | 0 | |
|
||||
| pseudoprime_base2 | reasoner | 1 | ✅ | 75.3% | $0.003128 | 0 | 0 | |
|
||||
| pseudoprime_base2 | reasoner-harvest | 1 | ❌ | 0.0% | $0.000000 | 0 | 0 | This operation was aborted |
|
||||
| derangements_d7 | baseline | 1 | ✅ | 0.0% | $0.001273 | 0 | 0 | |
|
||||
| derangements_d7 | reasoner | 1 | ✅ | 71.9% | $0.003492 | 0 | 0 | |
|
||||
| derangements_d7 | reasoner-harvest | 1 | ✅ | 71.9% | $0.002843 | 3 | 2 | |
|
||||
| euler_quadratic_break | baseline | 1 | ✅ | 0.0% | $0.000985 | 0 | 0 | |
|
||||
| euler_quadratic_break | reasoner | 1 | ✅ | 61.0% | $0.001431 | 0 | 0 | |
|
||||
| euler_quadratic_break | reasoner-harvest | 1 | ✅ | 61.0% | $0.002594 | 3 | 2 | |
|
||||
|
||||
## Scope
|
||||
|
||||
Unlike τ-bench-lite, these tasks are single-turn reasoning problems (no user simulator, no DB, no tool calls). Checkers are deterministic — regex + set / value compare, never an LLM judge. The point is to isolate whether the Pillar 2 harvest step adds measurable value above plain reasoner usage.
|
||||
|
||||
Interpretation: `baseline` (chat / V3) is a floor. `reasoner` shows the raw R1 gain. `reasoner-harvest` isolates the cost + quality delta from the extra V3 harvest call.
|
||||
## Findings (v0.3, 18-run set)
|
||||
|
||||
Two bands of tasks tried (easy: mod7 / flips / hats; hard: pseudoprime341 / D_7 / Euler quadratic). The hard band was chosen for *known V3 failure modes*. Result:
|
||||
|
||||
1. **V3 chat passed all six tasks, including the three "hard" ones.** DeepSeek V3 knew 341 is the smallest base-2 pseudoprime, computed D_7=1854, and identified n=40 as the first Euler-polynomial failure. The tasks are well-known enough to be in training; the trap-answer variants (561, 265, 41) that the checkers specifically reject didn't fire.
|
||||
2. **Reasoner cost 3.04× baseline for identical pass rates (6/6 both sides).** On this task set, R1 adds cost and latency without measurable quality. The cache-hit story *does* extend to reasoner (74.2% mean), so Pillar 1 generalizes to R1 — but that's not a Pillar 2 story.
|
||||
3. **Harvest produced real signal** (mean 2.8 subgoals, 1.3 uncertainties per run where it fired) but didn't improve outcomes. One run hit the 300s timeout (bumped from 120s), suggesting reasoner-harvest latency is still unpredictable on some problems.
|
||||
|
||||
### What this tells us about Pillar 2 positioning
|
||||
|
||||
We've now given harvest two shots at showing answer-quality value on pure reasoning tasks. Both times V3 ate its lunch. This isn't a framework bug — it's a positioning signal:
|
||||
|
||||
- **On well-known math / logic / counting problems, DeepSeek V3 is strong enough that paying for R1 is not justified by accuracy.** Pillar 2's "smart preset" cost multiplier (10×) quoted in the README is real but for most single-question Q/A it buys nothing.
|
||||
- **Harvest's plausible value surfaces are not "better math answers".** More likely: (a) transparency / auditability for developers, (b) planning support when tools are in the mix, (c) driving branch-sampling on cases where uncertainty correlates with wrongness. None of those are tested here.
|
||||
|
||||
### Recommendation
|
||||
|
||||
Don't ship v0.3 on the "harvest makes answers more correct" claim — the data doesn't support it. Two honest paths forward:
|
||||
|
||||
- **Reframe Pillar 2 as a developer-facing introspection feature** (debug visibility into R1's planning). Keep harvest opt-in; stop implying it improves accuracy on end-user tasks.
|
||||
- **Move harvest eval to tool-use contexts** where uncertainty matters for *which tool to call next*, not *is this math right*. That's a new task shape — different harness, not a bigger task set here.
|
||||
|
||||
In either case, the next Reasonix release is better served by MCP client (Pillar 1 leverage across the ecosystem) than by a bigger harvest-bench data push.
|
||||
|
||||
### Notes
|
||||
|
||||
- Cache hit on baseline runs surprised me (39.5% mean). That's DeepSeek's *cross-session* prompt cache: the same system prompt ran many times across modes and runs warmed the server-side cache. Real cross-session evidence that Cache-First's value isn't only per-session.
|
||||
- 5-subgoals cap still hit uniformly. `HarvestOptions.maxItems` default is 5; harvest is truncating. Moot unless we find a setting where that matters.
|
||||
@@ -0,0 +1,232 @@
|
||||
/**
|
||||
* Render a harvest-bench results.json into a markdown report.
|
||||
*
|
||||
* npx tsx benchmarks/harvest/report.ts benchmarks/harvest/results-<date>.json
|
||||
* npx tsx benchmarks/harvest/report.ts <input.json> --out report.md
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync } from "node:fs";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { ALL_MODES, type HarvestBenchReport, type HarvestMode, type HarvestRunResult } from "./types.js";
|
||||
|
||||
interface CliArgs {
|
||||
input: string;
|
||||
outPath: string;
|
||||
}
|
||||
|
||||
function parseArgs(argv: string[]): CliArgs {
|
||||
const out: CliArgs = { input: "", outPath: "benchmarks/harvest/report.md" };
|
||||
const positional: string[] = [];
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === "--out") out.outPath = argv[++i] ?? out.outPath;
|
||||
else if (a && !a.startsWith("--")) positional.push(a);
|
||||
}
|
||||
out.input = positional[0] ?? "";
|
||||
if (!out.input) {
|
||||
throw new Error("usage: npx tsx benchmarks/harvest/report.ts <results.json> [--out report.md]");
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
interface Agg {
|
||||
runs: number;
|
||||
passes: number;
|
||||
avgCache: number;
|
||||
avgCost: number;
|
||||
avgClaudeCost: number;
|
||||
avgHarvestedTurns: number;
|
||||
avgSubgoals: number;
|
||||
avgUncertainties: number;
|
||||
}
|
||||
|
||||
function aggregate(results: HarvestRunResult[]): Agg {
|
||||
if (results.length === 0) {
|
||||
return {
|
||||
runs: 0,
|
||||
passes: 0,
|
||||
avgCache: 0,
|
||||
avgCost: 0,
|
||||
avgClaudeCost: 0,
|
||||
avgHarvestedTurns: 0,
|
||||
avgSubgoals: 0,
|
||||
avgUncertainties: 0,
|
||||
};
|
||||
}
|
||||
const passes = results.filter((r) => r.verdict === "pass").length;
|
||||
const mean = (fn: (r: HarvestRunResult) => number) =>
|
||||
results.reduce((s, r) => s + fn(r), 0) / results.length;
|
||||
return {
|
||||
runs: results.length,
|
||||
passes,
|
||||
avgCache: mean((r) => r.cacheHitRatio),
|
||||
avgCost: mean((r) => r.costUsd),
|
||||
avgClaudeCost: mean((r) => r.claudeEquivalentUsd),
|
||||
avgHarvestedTurns: mean((r) => r.harvestedTurns),
|
||||
avgSubgoals: mean((r) => r.totalSubgoals),
|
||||
avgUncertainties: mean((r) => r.totalUncertainties),
|
||||
};
|
||||
}
|
||||
|
||||
export function renderReport(report: HarvestBenchReport): string {
|
||||
const lines: string[] = [];
|
||||
lines.push("# Reasonix harvest eval (Pillar 2)");
|
||||
lines.push("");
|
||||
lines.push(`**Date:** ${report.meta.date}`);
|
||||
lines.push(
|
||||
`**Tasks:** ${report.meta.taskCount} · repeats × ${report.meta.repeatsPerTask} · modes: ${report.meta.modesRun.join(", ")}`,
|
||||
);
|
||||
lines.push(`**Reasonix version:** ${report.meta.reasonixVersion}`);
|
||||
lines.push("");
|
||||
|
||||
// Per-mode summary
|
||||
lines.push("## Summary by mode");
|
||||
lines.push("");
|
||||
lines.push("| mode | runs | pass rate | cache hit | cost / run | harvest turns | subgoals | uncertainties |");
|
||||
lines.push("|---|---:|---:|---:|---:|---:|---:|---:|");
|
||||
for (const mode of ALL_MODES) {
|
||||
const rs = report.results.filter((r) => r.mode === mode);
|
||||
const a = aggregate(rs);
|
||||
if (a.runs === 0) continue;
|
||||
lines.push(
|
||||
`| ${mode} | ${a.runs} | ${pct(a.passes, a.runs)} | ${pct1(a.avgCache)} | $${a.avgCost.toFixed(6)} | ${a.avgHarvestedTurns.toFixed(1)} | ${a.avgSubgoals.toFixed(1)} | ${a.avgUncertainties.toFixed(1)} |`,
|
||||
);
|
||||
}
|
||||
lines.push("");
|
||||
|
||||
// Pairwise deltas that actually answer the question
|
||||
lines.push("## Deltas");
|
||||
lines.push("");
|
||||
const modes = ALL_MODES.filter((m) =>
|
||||
report.results.some((r) => r.mode === m),
|
||||
);
|
||||
if (modes.length >= 2) {
|
||||
const baseline = aggregate(report.results.filter((r) => r.mode === modes[0]));
|
||||
for (let i = 1; i < modes.length; i++) {
|
||||
const mode = modes[i]!;
|
||||
const agg = aggregate(report.results.filter((r) => r.mode === mode));
|
||||
const passDelta = (agg.passes / Math.max(agg.runs, 1)) - (baseline.passes / Math.max(baseline.runs, 1));
|
||||
const costRatio = baseline.avgCost > 0 ? agg.avgCost / baseline.avgCost : 0;
|
||||
lines.push(`- **${modes[0]} → ${mode}**`);
|
||||
lines.push(` - pass rate: ${(passDelta * 100).toFixed(0)}pp`);
|
||||
lines.push(` - cost: ×${costRatio.toFixed(2)} (each run costs ${costRatio > 1 ? "more" : "less"})`);
|
||||
lines.push(` - harvest signal / run: ${agg.avgSubgoals.toFixed(1)} subgoals, ${agg.avgUncertainties.toFixed(1)} uncertainties`);
|
||||
lines.push("");
|
||||
}
|
||||
}
|
||||
|
||||
// Per-task breakdown
|
||||
lines.push("## Per-task breakdown");
|
||||
lines.push("");
|
||||
const byTask = new Map<string, HarvestRunResult[]>();
|
||||
for (const r of report.results) {
|
||||
const list = byTask.get(r.taskId) ?? [];
|
||||
list.push(r);
|
||||
byTask.set(r.taskId, list);
|
||||
}
|
||||
lines.push("| task | mode | rep | verdict | cache | cost | sg | un | note |");
|
||||
lines.push("|---|---|---:|:---:|---:|---:|---:|---:|---|");
|
||||
for (const [taskId, runs] of byTask) {
|
||||
for (const r of runs) {
|
||||
const mark = r.verdict === "pass" ? "✅" : r.verdict === "fail" ? "❌" : "•";
|
||||
lines.push(
|
||||
`| ${taskId} | ${r.mode} | ${r.repeat} | ${mark} | ${pct1(r.cacheHitRatio)} | $${r.costUsd.toFixed(6)} | ${r.totalSubgoals} | ${r.totalUncertainties} | ${truncate(r.checkNote ?? "", 40)} |`,
|
||||
);
|
||||
}
|
||||
}
|
||||
lines.push("");
|
||||
|
||||
lines.push("## Scope");
|
||||
lines.push("");
|
||||
lines.push(
|
||||
"Unlike τ-bench-lite, these tasks are single-turn reasoning problems (no user simulator, no DB, no tool calls). Checkers are deterministic — regex + set / value compare, never an LLM judge. The point is to isolate whether the Pillar 2 harvest step adds measurable value above plain reasoner usage.",
|
||||
);
|
||||
lines.push("");
|
||||
lines.push(
|
||||
"Interpretation: `baseline` (chat / V3) is a floor. `reasoner` shows the raw R1 gain. `reasoner-harvest` isolates the cost + quality delta from the extra V3 harvest call.",
|
||||
);
|
||||
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Findings section is hand-written into report.md after rendering — the
|
||||
* numbers don't interpret themselves, and boilerplate sections full of
|
||||
* "TBD — analyse the data" are worse than nothing. Append with append().
|
||||
*
|
||||
* This section has been updated twice: after the first 9-run set (which
|
||||
* bottomed out at V3 on easy tasks) and after this 18-run set (which
|
||||
* bottomed out at V3 on *hard* tasks too).
|
||||
*/
|
||||
export function renderFindings(_report: HarvestBenchReport): string {
|
||||
return [
|
||||
"",
|
||||
"## Findings (v0.3, 18-run set)",
|
||||
"",
|
||||
"Two bands of tasks tried (easy: mod7 / flips / hats; hard: pseudoprime341 / D_7 / Euler quadratic). The hard band was chosen for *known V3 failure modes*. Result:",
|
||||
"",
|
||||
"1. **V3 chat passed all six tasks, including the three \"hard\" ones.** DeepSeek V3 knew 341 is the smallest base-2 pseudoprime, computed D_7=1854, and identified n=40 as the first Euler-polynomial failure. The tasks are well-known enough to be in training; the trap-answer variants (561, 265, 41) that the checkers specifically reject didn't fire.",
|
||||
"2. **Reasoner cost 3.04× baseline for identical pass rates (6/6 both sides).** On this task set, R1 adds cost and latency without measurable quality. The cache-hit story *does* extend to reasoner (74.2% mean), so Pillar 1 generalizes to R1 — but that's not a Pillar 2 story.",
|
||||
"3. **Harvest produced real signal** (mean 2.8 subgoals, 1.3 uncertainties per run where it fired) but didn't improve outcomes. One run hit the 300s timeout (bumped from 120s), suggesting reasoner-harvest latency is still unpredictable on some problems.",
|
||||
"",
|
||||
"### What this tells us about Pillar 2 positioning",
|
||||
"",
|
||||
"We've now given harvest two shots at showing answer-quality value on pure reasoning tasks. Both times V3 ate its lunch. This isn't a framework bug — it's a positioning signal:",
|
||||
"",
|
||||
"- **On well-known math / logic / counting problems, DeepSeek V3 is strong enough that paying for R1 is not justified by accuracy.** Pillar 2's \"smart preset\" cost multiplier (10×) quoted in the README is real but for most single-question Q/A it buys nothing.",
|
||||
"- **Harvest's plausible value surfaces are not \"better math answers\".** More likely: (a) transparency / auditability for developers, (b) planning support when tools are in the mix, (c) driving branch-sampling on cases where uncertainty correlates with wrongness. None of those are tested here.",
|
||||
"",
|
||||
"### Recommendation",
|
||||
"",
|
||||
"Don't ship v0.3 on the \"harvest makes answers more correct\" claim — the data doesn't support it. Two honest paths forward:",
|
||||
"",
|
||||
"- **Reframe Pillar 2 as a developer-facing introspection feature** (debug visibility into R1's planning). Keep harvest opt-in; stop implying it improves accuracy on end-user tasks.",
|
||||
"- **Move harvest eval to tool-use contexts** where uncertainty matters for *which tool to call next*, not *is this math right*. That's a new task shape — different harness, not a bigger task set here.",
|
||||
"",
|
||||
"In either case, the next Reasonix release is better served by MCP client (Pillar 1 leverage across the ecosystem) than by a bigger harvest-bench data push.",
|
||||
"",
|
||||
"### Notes",
|
||||
"",
|
||||
"- Cache hit on baseline runs surprised me (39.5% mean). That's DeepSeek's *cross-session* prompt cache: the same system prompt ran many times across modes and runs warmed the server-side cache. Real cross-session evidence that Cache-First's value isn't only per-session.",
|
||||
"- 5-subgoals cap still hit uniformly. `HarvestOptions.maxItems` default is 5; harvest is truncating. Moot unless we find a setting where that matters.",
|
||||
"",
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
function pct(num: number, denom: number): string {
|
||||
if (denom === 0) return "—";
|
||||
return `${((num / denom) * 100).toFixed(0)}%`;
|
||||
}
|
||||
|
||||
function pct1(x: number): string {
|
||||
return `${(x * 100).toFixed(1)}%`;
|
||||
}
|
||||
|
||||
function truncate(s: string, n: number): string {
|
||||
return s.length > n ? `${s.slice(0, n)}…` : s;
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
const raw = readFileSync(args.input, "utf8");
|
||||
const report = JSON.parse(raw) as HarvestBenchReport;
|
||||
const md = renderReport(report) + renderFindings(report);
|
||||
writeFileSync(args.outPath, md, "utf8");
|
||||
console.log(`wrote ${args.outPath} (${report.results.length} runs)`);
|
||||
}
|
||||
|
||||
function isMain(): boolean {
|
||||
const entry = process.argv[1];
|
||||
if (!entry) return false;
|
||||
return import.meta.url === pathToFileURL(entry).href;
|
||||
}
|
||||
|
||||
if (isMain()) {
|
||||
main().catch((err) => {
|
||||
console.error(err);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
|
||||
// Suppress unused import warning — kept in scope so ALL_MODES order matches renderer iteration.
|
||||
export type { HarvestMode };
|
||||
@@ -0,0 +1,305 @@
|
||||
{
|
||||
"meta": {
|
||||
"date": "2026-04-21T16:44:21.011Z",
|
||||
"repeatsPerTask": 1,
|
||||
"taskCount": 6,
|
||||
"modesRun": [
|
||||
"baseline",
|
||||
"reasoner",
|
||||
"reasoner-harvest"
|
||||
],
|
||||
"reasonixVersion": "0.2.2"
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"taskId": "mod7_list",
|
||||
"mode": "baseline",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "Alright, let's go step-by-step.\n\n---\n\n## **Step 1: Problem restatement**\nWe want positive integers \\( n \\leq 100 \\) such that \n\\[\nn^2 + n + 1 \\equiv 0 \\pmod{7}.\n\\]\n\n---\n\n## **Step 2: Try residues modulo 7**\nLet \\( n \\equiv r \\pmod{7} \\), where \\( r \\in \\{0, 1, 2, 3, 4, 5, 6\\} \\).\n\nWe compute \\( r^2 + r + 1 \\mod 7 \\):\n\n- \\( r = 0 \\): \\( 0 + 0 + 1 = 1 \\) → not divisible by 7.\n- \\( r = 1 \\): \\( 1 + 1 + 1 = 3 \\) → no.\n- \\( r = 2 \\): \\( 4 + 2 + 1 = 7 \\equiv 0 \\) → yes.\n- \\( r = 3 \\): \\( 9 + 3 + 1 = 13 \\equiv 6 \\) → no.\n- \\( r = 4 \\): \\( 16 + 4 + 1 = 21 \\equiv 0 \\) → yes.\n- \\( r = 5 \\): \\( 25 + 5 + 1 = 31 \\equiv 3 \\) → no.\n- \\( r = 6 \\): \\( 36 + 6 + 1 = 43 \\equiv 1 \\) → no.\n\nSo \\( n \\equiv 2 \\pmod{7} \\) or \\( n \\equiv 4 \\pmod{7} \\).\n\n---\n\n## **Step 3: General form**\n\\[\nn \\equiv 2 \\pmod{7} \\quad \\text{or} \\quad n \\equiv 4 \\pmod{7}.\n\\]\n\nThat means \\( n = 7k + 2 \\) or \\( n = 7k + 4 \\) for some integer \\( k \\ge 0 \\).\n\n---\n\n## **Step 4: Positive integers \\( n \\leq 100 \\)**\n**Case 1:** \\( n = 7k + 2 \\), \\( n \\leq 100 \\), \\( k \\ge 0 \\).\n\n\\( 7k + 2 \\leq 100 \\) → \\( 7k \\leq 98 \\) → \\( k \\leq 14 \\).\n\nSo \\( k = 0, 1, 2, \\dots, 14 \\) gives \\( n = 2, 9, 16, \\dots, 100 \\).\n\nCheck \\( n=100 \\): \\( 7 \\times 14 + 2 = 98 + 2 = 100 \\) okay.\n\n---\n\n**Case 2:** \\( n = 7k + 4 \\), \\( n \\leq 100 \\), \\( k \\ge 0 \\).\n\n\\( 7k + 4 \\leq 100 \\) → \\( 7k \\leq 96 \\) → \\( k \\leq 13 \\) (since \\( 7 \\times 13 = 91 \\), \\( 91+4=95 \\), \\( 7\\times14=98 \\) → \\( 98+4=102>100 \\)).\n\nSo \\( k = 0, 1, 2, \\dots, 13 \\) gives \\( n = 4, 11, 18, \\dots, 95 \\).\n\n---\n\n## **Step 5: List all such \\( n \\leq 100 \\)**\nFrom case 1: \\( 2, 9, 16, 23, 30, 37, 44, 51, 58, 65, 72, 79, 86, 93, 100 \\).\n\nFrom case 2: \\( 4, 11, 18, 25, 32, 39, 46, 53, 60, 67, 74, 81, 88, 95 \\).\n\nCombine and sort:\n\n\\[\n2, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72, 74, 79, 81, 86, 88, 93, 95, 100.\n\\]\n\n---\n\n## **Step 6: Check count**\nEach sequence has 15 and 14 terms respectively, total \\( 15 + 14 = 29 \\) numbers.\n\nCheck: \\( n=2 \\) works? \\( 4+2+1=7 \\) yes. \\( n=4 \\) works? \\( 16+4+1=21 \\) yes. \\( n=100 \\) works? \\( 10000+100+1=10101 \\), divide by 7: \\( 10101/7 = 1443 \\) yes.\n\n---\n\n**Final answer:**\n\\[\n\\boxed{2, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72, 74, 79, 81, 86, 88, 93, 95, 100}\n\\]\n\n---\n\nAnswer: 2, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72, 74, 79, 81, 86, 88, 93, 95, 100",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.8648648648648649,
|
||||
"costUsd": 0.0013612800000000003,
|
||||
"claudeEquivalentUsd": 0.018687,
|
||||
"harvestedTurns": 0,
|
||||
"totalSubgoals": 0,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 0,
|
||||
"totalRejectedPaths": 0
|
||||
},
|
||||
{
|
||||
"taskId": "mod7_list",
|
||||
"mode": "reasoner",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "The condition \\(n^2 + n + 1\\) divisible by 7 is equivalent to \\(n^2 + n + 1 \\equiv 0 \\pmod{7}\\). Testing residues modulo 7:\n\n- \\(n \\equiv 0: 0+0+1 \\equiv 1\\) \n- \\(n \\equiv 1: 1+1+1 \\equiv 3\\) \n- \\(n \\equiv 2: 4+2+1 \\equiv 7 \\equiv 0\\) \n- \\(n \\equiv 3: 9+3+1 \\equiv 13 \\equiv 6\\) \n- \\(n \\equiv 4: 16+4+1 \\equiv 21 \\equiv 0\\) \n- \\(n \\equiv 5: 25+5+1 \\equiv 31 \\equiv 3\\) \n- \\(n \\equiv 6: 36+6+1 \\equiv 43 \\equiv 1\\)\n\nThus, \\(n \\equiv 2\\) or \\(n \\equiv 4 \\pmod{7}\\) are the solutions modulo 7. For positive integers \\(n \\leq 100\\), we list all numbers congruent to 2 or 4 modulo 7 in increasing order.\n\nAnswer: 2, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72, 74, 79, 81, 86, 88, 93, 95, 100",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.8648648648648649,
|
||||
"costUsd": 0.00479742,
|
||||
"claudeEquivalentUsd": 0.032982,
|
||||
"harvestedTurns": 0,
|
||||
"totalSubgoals": 0,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 0,
|
||||
"totalRejectedPaths": 0
|
||||
},
|
||||
{
|
||||
"taskId": "mod7_list",
|
||||
"mode": "reasoner-harvest",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "The congruence \\(n^2 + n + 1 \\equiv 0 \\pmod{7}\\) is solved by testing residues modulo 7. The solutions are \\(n \\equiv 2\\) and \\(n \\equiv 4 \\pmod{7}\\). For positive integers \\(n \\le 100\\), these yield \\(n = 7k + 2\\) with \\(k = 0, 1, \\dots, 14\\) and \\(n = 7k + 4\\) with \\(k = 0, 1, \\dots, 13\\).\n\nAnswer: 2, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72, 74, 79, 81, 86, 88, 93, 95, 100",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.8648648648648649,
|
||||
"costUsd": 0.00304542,
|
||||
"claudeEquivalentUsd": 0.020982,
|
||||
"harvestedTurns": 1,
|
||||
"totalSubgoals": 4,
|
||||
"totalUncertainties": 2,
|
||||
"totalHypotheses": 2,
|
||||
"totalRejectedPaths": 2
|
||||
},
|
||||
{
|
||||
"taskId": "flips_until_3heads",
|
||||
"mode": "baseline",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "Alright, let's reason through this step-by-step.\n\n---\n\n## **Step 1: Define states**\nLet’s define states based on how many consecutive heads we have at the moment.\n\n- \\( E_0 \\): Expected flips to reach 3 heads in a row starting from 0 consecutive heads (or just starting, or after a tail).\n- \\( E_1 \\): Expected flips to reach 3 heads in a row starting from 1 consecutive head.\n- \\( E_2 \\): Expected flips to reach 3 heads in a row starting from 2 consecutive heads.\n\nWe want \\( E_0 \\).\n\n---\n\n## **Step 2: Transition equations**\nFrom \\( E_0 \\):\n- Flip H (prob \\( \\frac12 \\)) → go to state \\( E_1 \\), 1 flip used.\n- Flip T (prob \\( \\frac12 \\)) → stay in \\( E_0 \\), 1 flip used.\n\nSo:\n\\[\nE_0 = \\frac12 (1 + E_1) + \\frac12 (1 + E_0)\n\\]\n\n---\n\nFrom \\( E_1 \\) (already have 1 head):\n- Flip H (prob \\( \\frac12 \\)) → go to state \\( E_2 \\), 1 flip used.\n- Flip T (prob \\( \\frac12 \\)) → go to state \\( E_0 \\), 1 flip used.\n\nSo:\n\\[\nE_1 = \\frac12 (1 + E_2) + \\frac12 (1 + E_0)\n\\]\n\n---\n\nFrom \\( E_2 \\) (already have 2 heads):\n- Flip H (prob \\( \\frac12 \\)) → done (3 heads in a row), 1 flip used.\n- Flip T (prob \\( \\frac12 \\)) → go to state \\( E_0 \\), 1 flip used.\n\nSo:\n\\[\nE_2 = \\frac12 (1 + 0) + \\frac12 (1 + E_0)\n\\]\n\\[\nE_2 = \\frac12 \\cdot 1 + \\frac12 (1 + E_0) \\quad\\text{(Wait, careful:)}\n\\]\nActually: \nIf H: \\( 1 \\) flip and done. \nIf T: \\( 1 \\) flip and then \\( E_0 \\) more.\n\nSo:\n\\[\nE_2 = \\frac12 \\cdot 1 + \\frac12 (1 + E_0)\n\\]\nCheck: \nH: 1 flip → total 1, done. Contribution: \\( \\frac12 \\times 1 \\). \nT: 1 flip + \\( E_0 \\) more → total \\( 1 + E_0 \\). Contribution: \\( \\frac12 \\times (1 + E_0) \\).\n\nYes:\n\\[\nE_2 = \\frac12 \\cdot 1 + \\frac12 (1 + E_0) = \\frac12 + \\frac12 + \\frac12 E_0 = 1 + \\frac{E_0}{2}.\n\\]\n\n---\n\n## **Step 3: Solve system**\nWe have:\n\n(1) \\( E_0 = \\frac12 (1 + E_1) + \\frac12 (1 + E_0) \\) \nSimplify: \\( E_0 = 1 + \\frac12 E_1 + \\frac12 E_0 \\) \nMultiply by 2: \\( 2E_0 = 2 + E_1 + E_0 \\) \nSo \\( E_0 - E_1 = 2 \\) … (Eq A)\n\n---\n\n(2) \\( E_1 = \\frac12 (1 + E_2) + \\frac12 (1 + E_0) \\) \nSimplify: \\( E_1 = 1 + \\frac12 E_2 + \\frac12 E_0 \\) … (Eq B)\n\n---\n\n(3) \\( E_2 = 1 + \\frac12 E_0 \\) … (Eq C)\n\n---\n\nFrom Eq C into Eq B:\n\\[\nE_1 = 1 + \\frac12 \\left( 1 + \\frac12 E_0 \\right) + \\frac12 E_0\n\\]\n\\[\nE_1 = 1 + \\frac12 + \\frac14 E_0 + \\frac12 E_0\n\\]\n\\[\nE_1 = \\frac32 + \\frac34 E_0\n\\]\n\n---\n\nFrom Eq A: \\( E_0 - E_1 = 2 \\) \nSubstitute \\( E_1 \\):\n\\[\nE_0 - \\left( \\frac32 + \\frac34 E_0 \\right) = 2\n\\]\n\\[\nE_0 - \\frac32 - \\frac34 E_0 = 2\n\\]\n\\[\n\\frac14 E_0 - \\frac32 = 2\n\\]\n\\[\n\\frac14 E_0 = 2 + \\frac32 = \\frac72\n\\]\n\\[\nE_0 = 14\n\\]\n\n---\n\n## **Step 4: Conclusion**\nThe expected number of flips until we get 3 heads in a row is \\( 14 \\).\n\n---\n\n**Answer:** 14",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.927536231884058,
|
||||
"costUsd": 0.00119823,
|
||||
"claudeEquivalentUsd": 0.016467,
|
||||
"harvestedTurns": 0,
|
||||
"totalSubgoals": 0,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 0,
|
||||
"totalRejectedPaths": 0
|
||||
},
|
||||
{
|
||||
"taskId": "flips_until_3heads",
|
||||
"mode": "reasoner",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "The expected number of flips required to get three consecutive heads is 14, derived from solving the system of linear equations for the Markov chain states representing the current streak of heads.\n\nAnswer: 14",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.927536231884058,
|
||||
"costUsd": 0.00283024,
|
||||
"claudeEquivalentUsd": 0.019512,
|
||||
"harvestedTurns": 0,
|
||||
"totalSubgoals": 0,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 0,
|
||||
"totalRejectedPaths": 0
|
||||
},
|
||||
{
|
||||
"taskId": "flips_until_3heads",
|
||||
"mode": "reasoner-harvest",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "The expected number of flips until three consecutive heads appear is 14.\n\nAnswer: 14",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.927536231884058,
|
||||
"costUsd": 0.00169144,
|
||||
"claudeEquivalentUsd": 0.011712,
|
||||
"harvestedTurns": 1,
|
||||
"totalSubgoals": 4,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 2,
|
||||
"totalRejectedPaths": 0
|
||||
},
|
||||
{
|
||||
"taskId": "three_hats",
|
||||
"mode": "baseline",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "Let's break this down step by step.\n\n---\n\n**Step 1: Understanding the setup** \nThree people: A (first), B (second), C (third). \nEach hat is red (R) or blue (B). \nThey know: at least one hat is red. \nEach sees the other two hats, not their own. \nThey are asked in order: \"Do you know your hat color?\" \nA answers no, B answers no, C answers yes. \nWe want C’s hat color.\n\n---\n\n**Step 2: From A’s perspective when asked** \nA sees B and C. \nIf A had seen two blue hats, then A would know: \"At least one red hat means my hat must be red.\" \nSo A says \"no\" → A did **not** see two blue hats. \nThus A sees at least one red hat on {B, C}.\n\nPossible cases for what A sees (B, C): \n(R, R), (R, B), (B, R) — all have at least one red.\n\n---\n\n**Step 3: B’s reasoning when asked** \nB hears A say \"no\". \nB sees A and C. \n\nLet’s consider B’s possible deductions from A’s \"no\": \nB knows: If A had seen two blues, A would have said yes. Since A said no, A saw at least one red among B and C. \nBut B sees A and C. \n\nFrom B’s viewpoint: \n- If B saw C with blue and A with blue, then B would know: \"A saw me and C both blue → impossible because A said no.\" But B doesn’t see that because if B saw A blue and C blue, then B would know his own hat must be red (since at least one red total). But then B would answer yes. \nSo B’s \"no\" means B did **not** see A and C both blue. \n\nAlso, B considers: Could my hat be blue? Let’s check: \nSuppose B sees A red, C blue. Then B thinks: \"If my hat were blue, then A would have seen (B blue, C blue) → A would have said yes. But A said no, so my hat cannot be blue.\" \nSo if B saw (A red, C blue), B would deduce: \"My hat must be red.\" Then B would answer yes. \nBut B said no → so B did **not** see (A red, C blue).\n\nSimilarly, if B saw (A blue, C red): \nThen B thinks: \"If my hat were blue, then A would have seen (B blue, C red) → that’s fine (A sees one red on C), so A would still say no. If my hat were red, then A would see (B red, C red) → A would still say no. So I don’t know.\" That’s consistent with B saying no. \n\nIf B saw (A red, C red): \nThen B thinks: \"If my hat were blue, A sees (B blue, C red) → A says no. If my hat were red, A sees (B red, C red) → A says no. So I don’t know.\" That’s also consistent with B saying no.\n\nSo B says no in cases: \nB sees (A blue, C red) or (A red, C red). \nB would say yes if B saw (A red, C blue) — but that didn’t happen. \n\n---\n\n**Step 4: C’s reasoning** \nC hears A says no, B says no. \nC sees A and B. \n\nFrom C’s viewpoint: \nLet’s list all possible (A, B, C) hat colorings with at least one red, consistent with A’s \"no\" (which we already incorporated: A sees at least one red on B/C) and B’s \"no\" (meaning B didn’t see A red, C blue — because that would make B know).\n\nPossible cases for (A, B) that C could see: \n\n1. C sees (A blue, B red): \n Then C thinks: \n - If C were blue: (A blue, B red, C blue) → A sees (B red, C blue) → A says no. B sees (A blue, C blue) → B sees two blues? Wait, B sees A blue, C blue → then B knows his own hat is red (since at least one red total) → B would say yes. But B said no → contradiction. So C cannot be blue. \n So C must be red. \n\n2. C sees (A red, B blue): \n Then C thinks: \n - If C were blue: (A red, B blue, C blue) → A sees (B blue, C blue) → A would say yes (since at least one red total means A’s hat must be red). But A said no → contradiction. So C cannot be blue. \n So C must be red. \n\n3. C sees (A red, B red): \n Then C thinks: \n - If C were blue: (A red, B red, C blue) → A sees (B red, C blue) → A says no. B sees (A red, C blue) → B sees A red, C blue → earlier we said: If B sees A red, C blue, then B would deduce his own hat is red (reasoning: if B were blue, A would have seen two blues and said yes, but A said no, so B can’t be blue) → so B would say yes. But B actually said no → contradiction. So C cannot be blue. \n - If C were red: (A red, B red, C red) → A sees (B red, C red) → A says no. B sees (A red, C red) → B says no. Works fine. \n\nSo in all cases C sees for (A, B), C can deduce C’s hat is **red** because assuming C is blue leads to contradiction with B’s \"no\" (in cases 1 and 3) or with A’s \"no\" (in case 2).\n\n---\n\n**Step 5: Conclusion** \nC answers yes, and C’s hat must be red.\n\n---\n\nAnswer: red",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.5765765765765766,
|
||||
"costUsd": 0.0015142700000000003,
|
||||
"claudeEquivalentUsd": 0.020748,
|
||||
"harvestedTurns": 0,
|
||||
"totalSubgoals": 0,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 0,
|
||||
"totalRejectedPaths": 0
|
||||
},
|
||||
{
|
||||
"taskId": "three_hats",
|
||||
"mode": "reasoner",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "Answer: red",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.5765765765765766,
|
||||
"costUsd": 0.009340119999999999,
|
||||
"claudeEquivalentUsd": 0.064068,
|
||||
"harvestedTurns": 0,
|
||||
"totalSubgoals": 0,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 0,
|
||||
"totalRejectedPaths": 0
|
||||
},
|
||||
{
|
||||
"taskId": "three_hats",
|
||||
"mode": "reasoner-harvest",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "The third person deduces their hat color based on the previous responses. Since the first two answer \"no,\" it implies they each see at least one red hat and cannot determine their own color. Specifically, the second person's \"no\" indicates they see the third person's hat as red; otherwise, they would have known their own hat color. Therefore, the third person's hat must be red.\n\nAnswer: red",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.5765765765765766,
|
||||
"costUsd": 0.0096511,
|
||||
"claudeEquivalentUsd": 0.066198,
|
||||
"harvestedTurns": 1,
|
||||
"totalSubgoals": 3,
|
||||
"totalUncertainties": 2,
|
||||
"totalHypotheses": 3,
|
||||
"totalRejectedPaths": 3
|
||||
},
|
||||
{
|
||||
"taskId": "pseudoprime_base2",
|
||||
"mode": "baseline",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "Alright, let's go step-by-step.\n\n---\n\n**Step 1: Understanding the problem**\n\nWe want the smallest composite integer \\( n > 1 \\) such that\n\n\\[\n2^n \\equiv 2 \\pmod{n}.\n\\]\n\nEquivalently, \\( 2^{n-1} \\equiv 1 \\pmod{n} \\) for composite \\( n \\).\n\nThis means \\( n \\) is a **Fermat pseudoprime to base 2**.\n\nWe check composite numbers starting from the smallest.\n\n---\n\n**Step 2: Check small composites**\n\n- \\( n = 4 \\): composite. \n\\( 2^4 = 16 \\), \\( 16 \\mod 4 = 0 \\), not \\( 2 \\). So \\( 2^4 \\equiv 0 \\not\\equiv 2 \\pmod{4} \\). Fails.\n\n- \\( n = 6 \\): composite. \n\\( 2^6 = 64 \\), \\( 64 \\mod 6 = 4 \\), not \\( 2 \\). Fails.\n\n- \\( n = 8 \\): composite. \n\\( 2^8 = 256 \\), \\( 256 \\mod 8 = 0 \\), not \\( 2 \\). Fails.\n\n- \\( n = 9 \\): composite. \n\\( 2^9 = 512 \\), \\( 512 \\mod 9 \\): \n\\( 512 \\div 9 \\), \\( 9 \\times 56 = 504 \\), remainder \\( 8 \\), not \\( 2 \\). Fails.\n\n- \\( n = 10 \\): composite. \n\\( 2^{10} = 1024 \\), \\( 1024 \\mod 10 = 4 \\), not \\( 2 \\). Fails.\n\n---\n\n**Step 3: Try n = 15**\n\n\\( n = 15 \\): composite. \n\\( 2^{15} = 32768 \\). \nCheck mod 15: \nWe can use mod arithmetic: \n\\( 2^4 = 16 \\equiv 1 \\pmod{15} \\)?? Wait, check: \\( 16 \\mod 15 = 1 \\), yes. \nSo \\( 2^{15} = (2^4)^3 \\times 2^3 = 1^3 \\times 8 = 8 \\pmod{15} \\). \nSo \\( 2^{15} \\equiv 8 \\not\\equiv 2 \\pmod{15} \\). Fails.\n\n---\n\n**Step 4: Try n = 21**\n\n\\( n = 21 \\): composite. \nMod 21: \n\\( 2^6 = 64 \\equiv 1 \\pmod{21} \\)? Check: \\( 64 \\div 21 = 3 \\times 21 = 63 \\), remainder 1, yes. \nSo \\( 2^{21} = (2^6)^3 \\times 2^3 \\equiv 1^3 \\times 8 = 8 \\pmod{21} \\). Not 2. Fails.\n\n---\n\n**Step 5: Try n = 22**\n\n\\( n = 22 \\): composite. \nUse mod 22: \nEuler’s totient not needed, just check order: \n\\( 2^5 = 32 \\equiv 10 \\pmod{22} \\) \n\\( 2^{10} = (2^5)^2 \\equiv 10^2 = 100 \\equiv 12 \\pmod{22} \\) \n\\( 2^{11} \\equiv 24 \\equiv 2 \\pmod{22} \\) \nSo \\( 2^{22} = (2^{11})^2 \\equiv 2^2 = 4 \\pmod{22} \\). Not 2. Fails.\n\n---\n\n**Step 6: Try n = 25**\n\n\\( n = 25 \\): composite. \nCheck mod 25: \n\\( 2^{10} = 1024 \\equiv 24 \\pmod{25} \\) (since \\( 1000 \\equiv 0 \\), \\( 24 \\equiv -1 \\)) \nActually check: \\( 1024 - 1000 = 24 \\), yes \\( 24 \\equiv -1 \\). \nSo \\( 2^{20} \\equiv 1 \\pmod{25} \\). \nThus \\( 2^{25} = 2^{20} \\times 2^5 \\equiv 1 \\times 32 = 7 \\pmod{25} \\). Not 2. Fails.\n\n---\n\n**Step 7: Known smallest base-2 Fermat pseudoprime**\n\nFrom known number theory facts: \nThe smallest Fermat pseudoprime to base 2 is \\( n = 341 \\). \nBut let's check smaller ones just in case I’m skipping one.\n\n---\n\n**Step 8: Try n = 91**\n\n91 = 7 × 13. \nCheck mod 7: \\( 2^{91} \\equiv (2^6)^{15} \\times 2^1 \\equiv 1^{15} \\times 2 = 2 \\pmod{7} \\). \nCheck mod 13: \\( 2^{12} \\equiv 1 \\pmod{13} \\), \n\\( 91 = 12 \\times 7 + 7 \\), so \\( 2^{91} \\equiv 2^7 = 128 \\equiv 11 \\pmod{13} \\). \nFor \\( 2^{91} \\equiv 2 \\pmod{91} \\), we need \\( 2^{91} \\equiv 2 \\) mod 7 and mod 13. \nMod 13 we got 11, not 2. So fails.\n\n---\n\n**Step 9: Try n = 341**\n\n341 = 11 × 31. \nCheck mod 11: \\( 2^{10} \\equiv 1 \\pmod{11} \\), \n\\( 341 = 10 \\times 34 + 1 \\), so \\( 2^{341} \\equiv 2^1 = 2 \\pmod{11} \\). Good. \nCheck mod 31: \\( 2^5 = 32 \\equiv 1 \\pmod{31} \\), \n\\( 341 = 5 \\times 68 + 1 \\), so \\( 2^{341} \\equiv 2^1 = 2 \\pmod{31} \\). \nSo \\( 2^{341} \\equiv 2 \\) mod 11 and mod 31, hence mod 341. \nThus \\( 341 \\) works.\n\n---\n\n**Step 10: Check smaller than 341**\n\nKnown list: 341 is smallest. Let’s check a candidate like 121 = 11^2: \n\\( 2^{121} \\mod 121 \\): \nMod 11: \\( 2^{10} \\equiv 1 \\), \\( 121 = 10 \\times 12 + 1 \\), so \\( 2^{121} \\equiv 2 \\pmod{11} \\). \nBut for modulus 121, need more: \nActually, \\( 2^{110} \\equiv 1 \\pmod{121} \\) by Euler’s theorem? Not exactly, but known: 121 is not a pseudoprime base 2. Quick check: \n\\( 2^{121} - 2 \\) divisible by 121? Probably not. Let’s skip detailed check; known fact is 341 is smallest.\n\n---\n\n**Step 11: Verify no smaller composite works**\n\nFrom known sequence A001567: \nFirst few base-2 Fermat pseudoprimes: 341, 561, 645, 1105, … \nSo smallest is 341.\n\n---\n\n**Step 12: Conclusion**\n\nThe smallest composite \\( n > 1 \\) with \\( 2^n \\equiv 2 \\pmod{n} \\) is \\( n = 341 \\).\n\n---\n\n**Answer:** 341",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0,
|
||||
"costUsd": 0.0018940500000000002,
|
||||
"claudeEquivalentUsd": 0.02577,
|
||||
"harvestedTurns": 0,
|
||||
"totalSubgoals": 0,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 0,
|
||||
"totalRejectedPaths": 0
|
||||
},
|
||||
{
|
||||
"taskId": "pseudoprime_base2",
|
||||
"mode": "reasoner",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "The smallest composite integer \\(n > 1\\) satisfying \\(2^n \\equiv 2 \\pmod{n}\\) is \\(n = 341\\). This is composite (\\(341 = 11 \\times 31\\)), and it satisfies the congruence because \\(2^{341} \\equiv 2 \\pmod{11}\\) and \\(2^{341} \\equiv 2 \\pmod{31}\\) by Fermat's little theorem, implying \\(2^{341} \\equiv 2 \\pmod{341}\\). All smaller composite integers (e.g., 4, 6, 8, 9, 10, 15, 21) fail to satisfy the condition.\n\nAnswer: 341",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.7529411764705882,
|
||||
"costUsd": 0.0031281200000000003,
|
||||
"claudeEquivalentUsd": 0.02154,
|
||||
"harvestedTurns": 0,
|
||||
"totalSubgoals": 0,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 0,
|
||||
"totalRejectedPaths": 0
|
||||
},
|
||||
{
|
||||
"taskId": "pseudoprime_base2",
|
||||
"mode": "reasoner-harvest",
|
||||
"repeat": 1,
|
||||
"verdict": "fail",
|
||||
"checkNote": "This operation was aborted",
|
||||
"finalAgentMessage": "",
|
||||
"turns": 0,
|
||||
"cacheHitRatio": 0,
|
||||
"costUsd": 0,
|
||||
"claudeEquivalentUsd": 0,
|
||||
"harvestedTurns": 0,
|
||||
"totalSubgoals": 0,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 0,
|
||||
"totalRejectedPaths": 0,
|
||||
"errorMessage": "This operation was aborted"
|
||||
},
|
||||
{
|
||||
"taskId": "derangements_d7",
|
||||
"mode": "baseline",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "Alright, let's go step-by-step.\n\n---\n\n**Step 1: Understanding the problem**\n\nWe want the number of permutations of \\(\\{1, 2, 3, 4, 5, 6, 7\\}\\) with **no fixed points**. \nA fixed point means \\(\\pi(i) = i\\) for some \\(i\\). \nA permutation with no fixed points is called a **derangement**. \nThe number of derangements of \\(n\\) elements is denoted \\(D_n\\) or \\(!n\\).\n\n---\n\n**Step 2: Formula for derangements**\n\nThe formula for \\(D_n\\) is:\n\n\\[\nD_n = n! \\sum_{k=0}^{n} \\frac{(-1)^k}{k!}\n\\]\n\nFor \\(n=7\\):\n\n\\[\nD_7 = 7! \\left( \\frac{1}{0!} - \\frac{1}{1!} + \\frac{1}{2!} - \\frac{1}{3!} + \\frac{1}{4!} - \\frac{1}{5!} + \\frac{1}{6!} - \\frac{1}{7!} \\right)\n\\]\n\n---\n\n**Step 3: Compute step-by-step**\n\nFirst, \\(7! = 5040\\).\n\nNow compute the sum:\n\n\\[\n\\sum_{k=0}^{7} \\frac{(-1)^k}{k!} \n= 1 - 1 + \\frac{1}{2} - \\frac{1}{6} + \\frac{1}{24} - \\frac{1}{120} + \\frac{1}{720} - \\frac{1}{5040}\n\\]\n\nLet's compute term-by-term:\n\n- \\(1 - 1 = 0\\)\n- \\(0 + 0.5 = 0.5\\)\n- \\(0.5 - \\frac{1}{6} \\approx 0.5 - 0.1666667 = 0.3333333\\)\n- \\(0.3333333 + \\frac{1}{24} \\approx 0.3333333 + 0.0416667 = 0.375\\)\n- \\(0.375 - \\frac{1}{120} \\approx 0.375 - 0.0083333 = 0.3666667\\)\n- \\(0.3666667 + \\frac{1}{720} \\approx 0.3666667 + 0.0013889 = 0.3680556\\)\n- \\(0.3680556 - \\frac{1}{5040} \\approx 0.3680556 - 0.0001984 = 0.3678572\\)\n\nSo the sum is approximately \\(0.367857142857...\\)\n\n---\n\n**Step 4: Multiply by \\(7!\\)**\n\n\\[\nD_7 = 5040 \\times 0.367857142857\\ldots\n\\]\n\nLet's compute exactly using fractions:\n\n\\[\n\\sum_{k=0}^{7} \\frac{(-1)^k}{k!} \n= \\frac{1}{2!} - \\frac{1}{3!} + \\frac{1}{4!} - \\frac{1}{5!} + \\frac{1}{6!} - \\frac{1}{7!}\n\\]\n\nWrite with common denominator \\(7! = 5040\\):\n\n- \\(\\frac{1}{2!} = \\frac{2520}{5040}\\)\n- \\(\\frac{1}{3!} = \\frac{840}{5040}\\)\n- \\(\\frac{1}{4!} = \\frac{210}{5040}\\)\n- \\(\\frac{1}{5!} = \\frac{42}{5040}\\)\n- \\(\\frac{1}{6!} = \\frac{7}{5040}\\)\n- \\(\\frac{1}{7!} = \\frac{1}{5040}\\)\n\nNow alternate signs:\n\n\\[\n\\left( \\frac{2520}{5040} - \\frac{840}{5040} \\right) \n= \\frac{1680}{5040}\n\\]\n\\[\n\\frac{1680}{5040} + \\frac{210}{5040} = \\frac{1890}{5040}\n\\]\n\\[\n\\frac{1890}{5040} - \\frac{42}{5040} = \\frac{1848}{5040}\n\\]\n\\[\n\\frac{1848}{5040} + \\frac{7}{5040} = \\frac{1855}{5040}\n\\]\n\\[\n\\frac{1855}{5040} - \\frac{1}{5040} = \\frac{1854}{5040}\n\\]\n\nSo the sum is \\(\\frac{1854}{5040}\\).\n\n---\n\n**Step 5: Multiply by \\(7!\\)**\n\n\\[\nD_7 = 5040 \\times \\frac{1854}{5040} = 1854\n\\]\n\n---\n\n**Step 6: Quick check with known derangement numbers**\n\nKnown small derangements: \n\\(D_1 = 0, D_2 = 1, D_3 = 2, D_4 = 9, D_5 = 44, D_6 = 265, D_7 = 1854\\). \nYes, matches known sequence.\n\n---\n\n**Final answer:**\n\nAnswer: 1854",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0,
|
||||
"costUsd": 0.00127253,
|
||||
"claudeEquivalentUsd": 0.017292,
|
||||
"harvestedTurns": 0,
|
||||
"totalSubgoals": 0,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 0,
|
||||
"totalRejectedPaths": 0
|
||||
},
|
||||
{
|
||||
"taskId": "derangements_d7",
|
||||
"mode": "reasoner",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "The number of derangements of \\(n\\) distinct objects, denoted \\(D_n\\), counts permutations with no fixed points. For \\(n = 7\\), this can be computed using the recurrence \\(D_n = (n-1)(D_{n-1} + D_{n-2})\\) with initial values \\(D_1 = 0\\) and \\(D_2 = 1\\). \nThus: \n\\(D_3 = 2(1+0) = 2\\), \n\\(D_4 = 3(2+1) = 9\\), \n\\(D_5 = 4(9+2) = 44\\), \n\\(D_6 = 5(44+9) = 265\\), \n\\(D_7 = 6(265+44) = 1854\\).\n\nAnswer: 1854",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.7191011235955056,
|
||||
"costUsd": 0.00349167,
|
||||
"claudeEquivalentUsd": 0.024027,
|
||||
"harvestedTurns": 0,
|
||||
"totalSubgoals": 0,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 0,
|
||||
"totalRejectedPaths": 0
|
||||
},
|
||||
{
|
||||
"taskId": "derangements_d7",
|
||||
"mode": "reasoner-harvest",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "The number of derangements of \\(n\\) elements, denoted \\(D_n\\), counts permutations with no fixed points. For \\(n=7\\), this can be computed using the recurrence \\(D_n = (n-1)(D_{n-1} + D_{n-2})\\) with initial values \\(D_0 = 1\\) and \\(D_1 = 0\\). Applying the recurrence yields \\(D_2 = 1\\), \\(D_3 = 2\\), \\(D_4 = 9\\), \\(D_5 = 44\\), \\(D_6 = 265\\), and finally \\(D_7 = 6 \\times (265 + 44) = 1854\\).\n\nAnswer: 1854",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.7191011235955056,
|
||||
"costUsd": 0.0028434299999999997,
|
||||
"claudeEquivalentUsd": 0.019587,
|
||||
"harvestedTurns": 1,
|
||||
"totalSubgoals": 3,
|
||||
"totalUncertainties": 2,
|
||||
"totalHypotheses": 2,
|
||||
"totalRejectedPaths": 2
|
||||
},
|
||||
{
|
||||
"taskId": "euler_quadratic_break",
|
||||
"mode": "baseline",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "Let’s check \\( f(n) = n^2 + n + 41 \\) for small \\( n \\):\n\n- \\( n = 0 \\): \\( f(0) = 41 \\) (prime) \n- \\( n = 1 \\): \\( f(1) = 1 + 1 + 41 = 43 \\) (prime) \n- \\( n = 2 \\): \\( f(2) = 4 + 2 + 41 = 47 \\) (prime) \n- \\( n = 3 \\): \\( f(3) = 9 + 3 + 41 = 53 \\) (prime) \n- \\( n = 4 \\): \\( f(4) = 16 + 4 + 41 = 61 \\) (prime) \n- \\( n = 5 \\): \\( f(5) = 25 + 5 + 41 = 71 \\) (prime) \n- \\( n = 6 \\): \\( f(6) = 36 + 6 + 41 = 83 \\) (prime) \n- \\( n = 7 \\): \\( f(7) = 49 + 7 + 41 = 97 \\) (prime) \n- \\( n = 8 \\): \\( f(8) = 64 + 8 + 41 = 113 \\) (prime) \n- \\( n = 9 \\): \\( f(9) = 81 + 9 + 41 = 131 \\) (prime) \n- \\( n = 10 \\): \\( f(10) = 100 + 10 + 41 = 151 \\) (prime) \n\nWe can keep going, but the first known counterexample for this famous Euler polynomial is \\( n = 40 \\), because \n\\[\nf(40) = 40^2 + 40 + 41 = 1600 + 40 + 41 = 1681.\n\\] \nCheck if \\( 1681 \\) is prime: \n\\( 1681 = 41^2 \\) (since \\( 41 \\times 41 = 1681 \\)), so it’s composite.\n\nBut wait — let’s check if there’s a smaller \\( n \\) where \\( f(n) \\) is composite. \nWe can test \\( n = 41 \\): \n\\( f(41) = 41^2 + 41 + 41 = 1681 + 82 = 1763 \\), which is \\( 41 \\times 43 \\) (since \\( 1763/41 = 43 \\)), so composite. But \\( n = 41 \\) is larger than \\( n = 40 \\).\n\nCheck \\( n = 39 \\): \\( f(39) = 1521 + 39 + 41 = 1601 \\). Test divisibility: not divisible by primes up to \\( \\sqrt{1601} \\approx 40 \\), seems prime. \n\nCheck \\( n = 40 \\) is indeed the first composite? Let’s check \\( n = 1 \\) to \\( 39 \\) quickly for a smaller counterexample: \nWe can note \\( f(n) = n(n+1) + 41 \\). If \\( 41 \\mid f(n) \\), then \\( 41 \\mid n(n+1) \\). \nThat happens when \\( n \\equiv 0 \\pmod{41} \\) or \\( n \\equiv 40 \\pmod{41} \\). \nThe smallest positive such \\( n \\) is \\( n = 40 \\) (since \\( n=41 \\) is larger). \nSo for \\( n = 40 \\), \\( f(40) = 40 \\times 41 + 41 = 41 \\times 41 \\) is composite. \n\nCheck \\( n = 0 \\) to \\( 39 \\): no multiple of 41 except \\( n=0 \\) (which gives 41, prime). So \\( n=40 \\) is indeed the first nonnegative integer where \\( f(n) \\) is composite.\n\nThus smallest \\( n \\) is \\( 40 \\), \\( f(40) = 1681 = 41^2 \\).\n\n**Answer:** 40",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0,
|
||||
"costUsd": 0.00098535,
|
||||
"claudeEquivalentUsd": 0.013365,
|
||||
"harvestedTurns": 0,
|
||||
"totalSubgoals": 0,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 0,
|
||||
"totalRejectedPaths": 0
|
||||
},
|
||||
{
|
||||
"taskId": "euler_quadratic_break",
|
||||
"mode": "reasoner",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "The smallest nonnegative integer \\(n\\) for which \\(f(n) = n^2 + n + 41\\) is composite is \\(n = 40\\). At this value, \\(f(40) = 40^2 + 40 + 41 = 1681 = 41^2\\), which is not prime.\n\nAnswer: 40",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.6095238095238096,
|
||||
"costUsd": 0.0014309199999999998,
|
||||
"claudeEquivalentUsd": 0.0099,
|
||||
"harvestedTurns": 0,
|
||||
"totalSubgoals": 0,
|
||||
"totalUncertainties": 0,
|
||||
"totalHypotheses": 0,
|
||||
"totalRejectedPaths": 0
|
||||
},
|
||||
{
|
||||
"taskId": "euler_quadratic_break",
|
||||
"mode": "reasoner-harvest",
|
||||
"repeat": 1,
|
||||
"verdict": "pass",
|
||||
"finalAgentMessage": "The polynomial \\(f(n) = n^2 + n + 41\\) is known to produce primes for \\(n = 0, 1, 2, \\dots, 39\\). At \\(n = 40\\), we have \\(f(40) = 40^2 + 40 + 41 = 1681\\), and \\(1681 = 41^2\\), which is composite. Therefore, \\(n = 40\\) is the smallest nonnegative integer where \\(f(n)\\) fails to be prime.\n\nAnswer: 40",
|
||||
"turns": 1,
|
||||
"cacheHitRatio": 0.6095238095238096,
|
||||
"costUsd": 0.0025938099999999998,
|
||||
"claudeEquivalentUsd": 0.017865,
|
||||
"harvestedTurns": 1,
|
||||
"totalSubgoals": 3,
|
||||
"totalUncertainties": 2,
|
||||
"totalHypotheses": 3,
|
||||
"totalRejectedPaths": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,312 @@
|
||||
/**
|
||||
* Harvest-bench runner.
|
||||
*
|
||||
* Usage:
|
||||
* npx tsx benchmarks/harvest/runner.ts # all tasks, all modes, 1 repeat
|
||||
* npx tsx benchmarks/harvest/runner.ts --task mod7_list
|
||||
* npx tsx benchmarks/harvest/runner.ts --mode reasoner-harvest --repeats 3
|
||||
* npx tsx benchmarks/harvest/runner.ts --transcripts-dir transcripts/
|
||||
* npx tsx benchmarks/harvest/runner.ts --dry # smoke test wiring, no API
|
||||
*
|
||||
* Writes results.json. Render report.md with `report.ts`.
|
||||
*/
|
||||
|
||||
import { type WriteStream, mkdirSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import {
|
||||
CacheFirstLoop,
|
||||
DeepSeekClient,
|
||||
ImmutablePrefix,
|
||||
claudeEquivalentCost,
|
||||
loadDotenv,
|
||||
} from "../../src/index.js";
|
||||
import { openTranscriptFile, recordFromLoopEvent, writeRecord } from "../../src/transcript.js";
|
||||
import { TASKS } from "./tasks.js";
|
||||
import type {
|
||||
HarvestBenchMeta,
|
||||
HarvestBenchReport,
|
||||
HarvestMode,
|
||||
HarvestRunResult,
|
||||
HarvestTask,
|
||||
} from "./types.js";
|
||||
import { ALL_MODES } from "./types.js";
|
||||
|
||||
loadDotenv();
|
||||
|
||||
interface CliArgs {
|
||||
taskFilter: string | null;
|
||||
modes: HarvestMode[];
|
||||
repeats: number;
|
||||
outPath: string | null;
|
||||
transcriptsDir: string | null;
|
||||
/** Per-HTTP-call timeout in ms. Default 300_000 (5 min) — reasoner + harvest runs legitimately need this. */
|
||||
timeoutMs: number;
|
||||
dry: boolean;
|
||||
verbose: boolean;
|
||||
}
|
||||
|
||||
function parseArgs(argv: string[]): CliArgs {
|
||||
const out: CliArgs = {
|
||||
taskFilter: null,
|
||||
modes: [...ALL_MODES],
|
||||
repeats: 1,
|
||||
outPath: null,
|
||||
transcriptsDir: null,
|
||||
timeoutMs: 300_000,
|
||||
dry: false,
|
||||
verbose: false,
|
||||
};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === "--task") out.taskFilter = argv[++i] ?? null;
|
||||
else if (a === "--mode") {
|
||||
const v = (argv[++i] ?? "").toLowerCase() as HarvestMode;
|
||||
if ((ALL_MODES as string[]).includes(v)) out.modes = [v];
|
||||
} else if (a === "--repeats") out.repeats = Number.parseInt(argv[++i] ?? "1", 10);
|
||||
else if (a === "--out") out.outPath = argv[++i] ?? null;
|
||||
else if (a === "--transcripts-dir") out.transcriptsDir = argv[++i] ?? null;
|
||||
else if (a === "--timeout") {
|
||||
// Accept seconds for humans, convert to ms internally.
|
||||
const secs = Number.parseInt(argv[++i] ?? "300", 10);
|
||||
if (Number.isFinite(secs) && secs > 0) out.timeoutMs = secs * 1000;
|
||||
} else if (a === "--dry") out.dry = true;
|
||||
else if (a === "--verbose" || a === "-v") out.verbose = true;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// ---------- model resolution ----------
|
||||
|
||||
function modelForMode(mode: HarvestMode): string {
|
||||
if (mode === "baseline") return "deepseek-chat";
|
||||
return "deepseek-reasoner";
|
||||
}
|
||||
|
||||
function harvestForMode(mode: HarvestMode): boolean {
|
||||
return mode === "reasoner-harvest";
|
||||
}
|
||||
|
||||
// ---------- run one task in one mode ----------
|
||||
|
||||
async function runOnce(
|
||||
task: HarvestTask,
|
||||
mode: HarvestMode,
|
||||
rep: number,
|
||||
client: DeepSeekClient,
|
||||
transcriptStream: WriteStream | null,
|
||||
): Promise<HarvestRunResult> {
|
||||
const model = modelForMode(mode);
|
||||
const harvest = harvestForMode(mode);
|
||||
|
||||
const prefix = new ImmutablePrefix({ system: task.systemPrompt });
|
||||
const loop = new CacheFirstLoop({
|
||||
client,
|
||||
prefix,
|
||||
model,
|
||||
harvest,
|
||||
stream: false,
|
||||
});
|
||||
const prefixHash = prefix.fingerprint;
|
||||
|
||||
let harvestedTurns = 0;
|
||||
let totalSubgoals = 0;
|
||||
let totalUncertainties = 0;
|
||||
let totalHypotheses = 0;
|
||||
let totalRejectedPaths = 0;
|
||||
let finalText = "";
|
||||
let errorMessage: string | undefined;
|
||||
|
||||
if (transcriptStream) {
|
||||
writeRecord(transcriptStream, {
|
||||
ts: new Date().toISOString(),
|
||||
turn: 1,
|
||||
role: "user",
|
||||
content: task.prompt,
|
||||
});
|
||||
}
|
||||
|
||||
try {
|
||||
for await (const ev of loop.step(task.prompt)) {
|
||||
if (transcriptStream && ev.role !== "assistant_delta") {
|
||||
writeRecord(transcriptStream, recordFromLoopEvent(ev, { model, prefixHash }));
|
||||
}
|
||||
if (ev.role === "assistant_final") {
|
||||
finalText = ev.content;
|
||||
if (ev.planState) {
|
||||
if (
|
||||
ev.planState.subgoals.length +
|
||||
ev.planState.hypotheses.length +
|
||||
ev.planState.uncertainties.length +
|
||||
ev.planState.rejectedPaths.length >
|
||||
0
|
||||
) {
|
||||
harvestedTurns++;
|
||||
totalSubgoals += ev.planState.subgoals.length;
|
||||
totalHypotheses += ev.planState.hypotheses.length;
|
||||
totalUncertainties += ev.planState.uncertainties.length;
|
||||
totalRejectedPaths += ev.planState.rejectedPaths.length;
|
||||
}
|
||||
}
|
||||
} else if (ev.role === "done") {
|
||||
finalText = ev.content || finalText;
|
||||
break;
|
||||
} else if (ev.role === "error") {
|
||||
throw new Error(ev.error ?? "loop error");
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
errorMessage = (err as Error).message;
|
||||
}
|
||||
|
||||
const verdict = errorMessage ? { verdict: "fail" as const, note: errorMessage } : task.check(finalText);
|
||||
|
||||
return {
|
||||
taskId: task.id,
|
||||
mode,
|
||||
repeat: rep + 1,
|
||||
verdict: verdict.verdict,
|
||||
checkNote: verdict.note,
|
||||
finalAgentMessage: finalText,
|
||||
turns: loop.stats.turns.length,
|
||||
cacheHitRatio: loop.stats.aggregateCacheHitRatio,
|
||||
costUsd: loop.stats.totalCost,
|
||||
claudeEquivalentUsd: loop.stats.turns.reduce((s, t) => s + claudeEquivalentCost(t.usage), 0),
|
||||
harvestedTurns,
|
||||
totalSubgoals,
|
||||
totalUncertainties,
|
||||
totalHypotheses,
|
||||
totalRejectedPaths,
|
||||
errorMessage,
|
||||
};
|
||||
}
|
||||
|
||||
// ---------- dry mode ----------
|
||||
|
||||
function runDry(args: CliArgs): HarvestBenchReport {
|
||||
const tasks = filterTasks(args.taskFilter);
|
||||
const results: HarvestRunResult[] = [];
|
||||
for (const task of tasks) {
|
||||
// Exercise each checker with a deliberately wrong reply so we know
|
||||
// the wiring is live — not a pass on empty.
|
||||
const sanity = task.check("Answer: this is not the right answer at all");
|
||||
console.log(`[${task.id}] checker exercises ok (sanity verdict=${sanity.verdict})`);
|
||||
for (const mode of args.modes) {
|
||||
results.push({
|
||||
taskId: task.id,
|
||||
mode,
|
||||
repeat: 1,
|
||||
verdict: "inconclusive",
|
||||
checkNote: "dry-run (no model call)",
|
||||
finalAgentMessage: "[dry-run]",
|
||||
turns: 0,
|
||||
cacheHitRatio: 0,
|
||||
costUsd: 0,
|
||||
claudeEquivalentUsd: 0,
|
||||
harvestedTurns: 0,
|
||||
totalSubgoals: 0,
|
||||
totalUncertainties: 0,
|
||||
totalHypotheses: 0,
|
||||
totalRejectedPaths: 0,
|
||||
});
|
||||
}
|
||||
}
|
||||
return { meta: buildMeta(args, tasks.length), results };
|
||||
}
|
||||
|
||||
// ---------- main ----------
|
||||
|
||||
function filterTasks(filter: string | null): HarvestTask[] {
|
||||
if (!filter) return TASKS;
|
||||
const t = TASKS.find((x) => x.id === filter);
|
||||
if (!t) throw new Error(`unknown task: ${filter}`);
|
||||
return [t];
|
||||
}
|
||||
|
||||
function buildMeta(args: CliArgs, taskCount: number): HarvestBenchMeta {
|
||||
return {
|
||||
date: new Date().toISOString(),
|
||||
repeatsPerTask: args.repeats,
|
||||
taskCount,
|
||||
modesRun: args.modes,
|
||||
reasonixVersion: "0.2.2",
|
||||
};
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const args = parseArgs(process.argv.slice(2));
|
||||
|
||||
if (args.dry) {
|
||||
const report = runDry(args);
|
||||
writeReport(report, args.outPath);
|
||||
return;
|
||||
}
|
||||
|
||||
const client = new DeepSeekClient({ timeoutMs: args.timeoutMs });
|
||||
const tasks = filterTasks(args.taskFilter);
|
||||
|
||||
if (args.transcriptsDir) mkdirSync(args.transcriptsDir, { recursive: true });
|
||||
|
||||
const results: HarvestRunResult[] = [];
|
||||
for (const task of tasks) {
|
||||
for (let rep = 0; rep < args.repeats; rep++) {
|
||||
for (const mode of args.modes) {
|
||||
let stream: WriteStream | null = null;
|
||||
if (args.transcriptsDir) {
|
||||
const fname = `${task.id}.${mode}.r${rep + 1}.jsonl`;
|
||||
stream = openTranscriptFile(join(args.transcriptsDir, fname), {
|
||||
version: 1,
|
||||
source: `harvest-bench/${mode}`,
|
||||
model: modelForMode(mode),
|
||||
task: task.id,
|
||||
mode,
|
||||
repeat: rep + 1,
|
||||
startedAt: new Date().toISOString(),
|
||||
});
|
||||
}
|
||||
const started = Date.now();
|
||||
let result: HarvestRunResult;
|
||||
try {
|
||||
result = await runOnce(task, mode, rep, client, stream);
|
||||
} finally {
|
||||
stream?.end();
|
||||
}
|
||||
const elapsed = ((Date.now() - started) / 1000).toFixed(1);
|
||||
console.log(
|
||||
`[${task.id}/${mode}/r${rep + 1}] verdict=${result.verdict}` +
|
||||
` cache=${(result.cacheHitRatio * 100).toFixed(1)}%` +
|
||||
` cost=$${result.costUsd.toFixed(6)}` +
|
||||
` harvest=${result.harvestedTurns}:${result.totalSubgoals}s/${result.totalUncertainties}u` +
|
||||
` (${elapsed}s)` +
|
||||
(result.checkNote ? ` ${result.checkNote}` : ""),
|
||||
);
|
||||
results.push(result);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const report: HarvestBenchReport = { meta: buildMeta(args, tasks.length), results };
|
||||
writeReport(report, args.outPath);
|
||||
}
|
||||
|
||||
function writeReport(report: HarvestBenchReport, outPath: string | null): void {
|
||||
const path =
|
||||
outPath ?? `benchmarks/harvest/results-${new Date().toISOString().replace(/[:.]/g, "-")}.json`;
|
||||
writeFileSync(path, `${JSON.stringify(report, null, 2)}\n`, "utf8");
|
||||
console.log(`wrote ${path}`);
|
||||
}
|
||||
|
||||
function isMain(): boolean {
|
||||
const entry = process.argv[1];
|
||||
if (!entry) return false;
|
||||
return import.meta.url === pathToFileURL(entry).href;
|
||||
}
|
||||
|
||||
if (isMain()) {
|
||||
main().catch((err) => {
|
||||
console.error(err);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
|
||||
export { main as runHarvestBench };
|
||||
@@ -0,0 +1,171 @@
|
||||
/**
|
||||
* Seed reasoning tasks for the harvest eval harness.
|
||||
*
|
||||
* Each task has:
|
||||
* - a minimal system prompt (so the comparison is about the model, not
|
||||
* the prompt)
|
||||
* - a single user question
|
||||
* - a deterministic checker (regex + set/value compare)
|
||||
*
|
||||
* When adding a new task, prefer problems where:
|
||||
* - the answer has a clean shape (a number, a small set, a yes/no +
|
||||
* explanation) — easier to check
|
||||
* - reasoning actually matters (so harvest has signal to extract);
|
||||
* trivia questions are out of scope
|
||||
* - there's a known-good answer; judgment calls are out of scope
|
||||
*/
|
||||
|
||||
import type { HarvestTask } from "./types.js";
|
||||
|
||||
const CONCISE_SYSTEM = `You are a precise reasoner. Think carefully, then state your final answer on a clearly-labeled line starting with "Answer:".`;
|
||||
|
||||
/**
|
||||
* Parse all non-negative integers out of a string. Handles commas,
|
||||
* ranges written as "2-4" (expands to 2,3,4), and ignores numbers that
|
||||
* look like formula references (e.g. "n^2", "mod 7") by only accepting
|
||||
* integers after we strip common LaTeX decorations.
|
||||
*/
|
||||
function extractIntegers(text: string): number[] {
|
||||
// Strip LaTeX delimiters so n_1 / n^2 / \{ ... \} don't confuse us.
|
||||
const cleaned = text
|
||||
.replace(/\\[a-zA-Z]+/g, " ") // \frac, \mod etc.
|
||||
.replace(/[{}\\]/g, " ")
|
||||
.replace(/[_^]\d+/g, " "); // subscript/superscript numbers
|
||||
// Expand ranges: "2-4" → "2,3,4" (only for small gaps)
|
||||
const rangeExpanded = cleaned.replace(/(\d+)\s*[-–]\s*(\d+)/g, (_, a, b) => {
|
||||
const lo = Number.parseInt(a, 10);
|
||||
const hi = Number.parseInt(b, 10);
|
||||
if (!Number.isFinite(lo) || !Number.isFinite(hi) || hi - lo > 50) return `${a} ${b}`;
|
||||
const out: number[] = [];
|
||||
for (let i = Math.min(lo, hi); i <= Math.max(lo, hi); i++) out.push(i);
|
||||
return out.join(" ");
|
||||
});
|
||||
const matches = rangeExpanded.match(/-?\d+/g) ?? [];
|
||||
return matches.map((m) => Number.parseInt(m, 10)).filter((n) => Number.isFinite(n));
|
||||
}
|
||||
|
||||
/** Extract content after the last "Answer:" label (case-insensitive). */
|
||||
function extractAnswerLine(text: string): string {
|
||||
const matches = [...text.matchAll(/answer\s*[::]\s*([^\n]+)/gi)];
|
||||
if (matches.length === 0) return text;
|
||||
return matches[matches.length - 1]![1]!.trim();
|
||||
}
|
||||
|
||||
// ---------- expected answers ----------
|
||||
|
||||
/** Positive n ≤ 100 such that n^2 + n + 1 ≡ 0 (mod 7). */
|
||||
const MOD7_EXPECTED: number[] = (() => {
|
||||
const out: number[] = [];
|
||||
for (let n = 1; n <= 100; n++) {
|
||||
if ((n * n + n + 1) % 7 === 0) out.push(n);
|
||||
}
|
||||
return out;
|
||||
})();
|
||||
|
||||
/**
|
||||
* Single-integer checker. Pulls the first plausible integer from the last
|
||||
* "Answer:" line; compares for exact equality.
|
||||
*/
|
||||
function checkSingleInt(expected: number) {
|
||||
return (reply: string) => {
|
||||
const answerLine = extractAnswerLine(reply);
|
||||
const nums = extractIntegers(answerLine);
|
||||
if (nums.length === 0) return { verdict: "fail" as const, note: "no number in Answer line" };
|
||||
const n = nums[0]!;
|
||||
if (n === expected) return { verdict: "pass" as const };
|
||||
return { verdict: "fail" as const, note: `got ${n}, expected ${expected}` };
|
||||
};
|
||||
}
|
||||
|
||||
// ---------- tasks ----------
|
||||
|
||||
export const TASKS: HarvestTask[] = [
|
||||
{
|
||||
id: "mod7_list",
|
||||
description:
|
||||
"Number theory — find all positive n ≤ 100 with n^2+n+1 ≡ 0 (mod 7). 29-element set. Classic R1 scratch-work territory.",
|
||||
systemPrompt: CONCISE_SYSTEM,
|
||||
prompt:
|
||||
"Find all positive integers n with n ≤ 100 such that n^2 + n + 1 is divisible by 7. Briefly justify, then end with the full list on a line starting with \"Answer:\".",
|
||||
check: (reply) => {
|
||||
const answerLine = extractAnswerLine(reply);
|
||||
const found = new Set(extractIntegers(answerLine).filter((n) => n >= 1 && n <= 100));
|
||||
const expected = new Set(MOD7_EXPECTED);
|
||||
if (found.size === 0) {
|
||||
return { verdict: "fail", note: "no integers extracted from Answer line" };
|
||||
}
|
||||
// Exact set equality — both containment directions.
|
||||
for (const n of expected) if (!found.has(n)) return { verdict: "fail", note: `missing n=${n}` };
|
||||
for (const n of found) if (!expected.has(n)) return { verdict: "fail", note: `extra n=${n}` };
|
||||
return { verdict: "pass" };
|
||||
},
|
||||
},
|
||||
|
||||
{
|
||||
id: "flips_until_3heads",
|
||||
description:
|
||||
"Probability — expected coin flips until 3 consecutive heads. Answer: 14. Tests whether R1 derives via recurrence or via memorized result.",
|
||||
systemPrompt: CONCISE_SYSTEM,
|
||||
prompt:
|
||||
"A fair coin is flipped repeatedly until you see 3 heads in a row. What is the expected number of total flips? Give a single integer on a line starting with \"Answer:\".",
|
||||
check: (reply) => {
|
||||
const answerLine = extractAnswerLine(reply);
|
||||
const nums = extractIntegers(answerLine);
|
||||
if (nums.length === 0) return { verdict: "fail", note: "no number in Answer line" };
|
||||
// Accept the first sensible number — sometimes agents write "Answer: 14 flips".
|
||||
const n = nums[0]!;
|
||||
if (n === 14) return { verdict: "pass" };
|
||||
return { verdict: "fail", note: `got ${n}, expected 14` };
|
||||
},
|
||||
},
|
||||
|
||||
{
|
||||
id: "three_hats",
|
||||
description:
|
||||
"Logic — three people wear red/blue hats, at least one red, they're asked in order. First two say no, third says yes. What color? Answer: red (deducible because if third were blue, the first two could see one red + one unknown and wouldn't have said no on their full first round, for most configurations the third can only be certain if red).",
|
||||
systemPrompt: CONCISE_SYSTEM,
|
||||
prompt:
|
||||
'Three people wear hats that are each red or blue. They are told at least one hat is red. Each can see the other two hats but not their own. They are asked in order: "Do you know your hat color?" The first two answer no. The third answers yes. What color is the third person\'s hat? Give a one-word answer on a line starting with "Answer:".',
|
||||
check: (reply) => {
|
||||
const answerLine = extractAnswerLine(reply).toLowerCase();
|
||||
// Accept "red" as a standalone word (not "redirected" etc).
|
||||
if (/\bred\b/.test(answerLine) && !/\bblue\b/.test(answerLine)) {
|
||||
return { verdict: "pass" };
|
||||
}
|
||||
if (/\bblue\b/.test(answerLine)) return { verdict: "fail", note: "answered blue" };
|
||||
return { verdict: "fail", note: "no clear color in Answer line" };
|
||||
},
|
||||
},
|
||||
|
||||
// ---------- hard tasks (picked for known V3 failure modes) ----------
|
||||
|
||||
{
|
||||
id: "pseudoprime_base2",
|
||||
description:
|
||||
"Number theory — smallest composite n > 1 such that 2^n ≡ 2 (mod n). This is the smallest Fermat pseudoprime to base 2. Answer: 341 (= 11·31). V3 frequently answers 561 (Carmichael / wrong base-2 property) or 9 (not actually a pseudoprime to base 2, verifying the model isn't computing). R1 usually gets this right.",
|
||||
systemPrompt: CONCISE_SYSTEM,
|
||||
prompt:
|
||||
'Find the smallest composite integer n > 1 such that 2^n is congruent to 2 modulo n (i.e., n is a Fermat pseudoprime to base 2). Briefly justify, then give a single integer on a line starting with "Answer:".',
|
||||
check: checkSingleInt(341),
|
||||
},
|
||||
|
||||
{
|
||||
id: "derangements_d7",
|
||||
description:
|
||||
"Combinatorics — number of derangements of 7 elements. Answer: 1854 (D_7 = 7! · (1 − 1/1! + 1/2! − 1/3! + … ± 1/7!) = 5040 · 0.367857… = 1854). V3 sometimes approximates via n!/e and rounds wrong, or confuses with D_6=265 or D_8=14833. R1's explicit recurrence D_n = (n-1)(D_{n-1}+D_{n-2}) avoids the mistake.",
|
||||
systemPrompt: CONCISE_SYSTEM,
|
||||
prompt:
|
||||
'How many permutations of {1, 2, 3, 4, 5, 6, 7} have no fixed points? (This is D_7, the derangement number.) Briefly justify, then give a single integer on a line starting with "Answer:".',
|
||||
check: checkSingleInt(1854),
|
||||
},
|
||||
|
||||
{
|
||||
id: "euler_quadratic_break",
|
||||
description:
|
||||
"Number theory — smallest nonneg integer n where n^2 + n + 41 is NOT prime. Euler's prime-generating polynomial famously produces primes for n = 0..39 and first fails at n = 40 (40²+40+41 = 1681 = 41²). V3 often confuses this with the related n²-n+41 (first fails at 41) or cites the wrong threshold. R1 checks small n systematically.",
|
||||
systemPrompt: CONCISE_SYSTEM,
|
||||
prompt:
|
||||
'Consider the polynomial f(n) = n^2 + n + 41 for n = 0, 1, 2, …. What is the smallest nonneg integer n at which f(n) fails to be prime? Briefly justify (including the value of f(n) at that point), then give just the integer n on a line starting with "Answer:".',
|
||||
check: checkSingleInt(40),
|
||||
},
|
||||
];
|
||||
@@ -0,0 +1,82 @@
|
||||
/**
|
||||
* Types for the harvest eval harness.
|
||||
*
|
||||
* Scope: this is deliberately NOT τ-bench-lite. These tasks:
|
||||
* - are single-turn Q/A (no user simulator, no DB)
|
||||
* - target reasoning-heavy problems (math, logic, planning)
|
||||
* - use deterministic checkers (regex extraction + set/value compare),
|
||||
* never an LLM judge
|
||||
*
|
||||
* The point is to measure Pillar 2 (harvest) the same way τ-bench-lite
|
||||
* measures Pillar 1 (cache-first): isolate one variable, produce numbers
|
||||
* anyone can reproduce.
|
||||
*/
|
||||
|
||||
export type HarvestMode =
|
||||
/** deepseek-chat, no Reasonix tricks. Floor reference for answer quality on reasoning tasks (V3 at its best). */
|
||||
| "baseline"
|
||||
/** deepseek-reasoner, no harvest. Isolates "does reasoner help over chat on these problems?" */
|
||||
| "reasoner"
|
||||
/** deepseek-reasoner + harvest. Isolates "does the extra V3 harvest call add quality / change behavior?" */
|
||||
| "reasoner-harvest";
|
||||
|
||||
export const ALL_MODES: HarvestMode[] = ["baseline", "reasoner", "reasoner-harvest"];
|
||||
|
||||
/**
|
||||
* Checker verdict. We don't use boolean directly because "maybe" is real
|
||||
* — e.g. "answer in the right shape but can't verify without running the
|
||||
* program". For v0.3 we only use {pass, fail}, but the shape leaves room.
|
||||
*/
|
||||
export type CheckVerdict = "pass" | "fail" | "inconclusive";
|
||||
|
||||
export interface HarvestTask {
|
||||
id: string;
|
||||
/** One-line human description, not shown to the model. */
|
||||
description: string;
|
||||
/** System prompt. Kept minimal so harvest signal can be attributed to the model, not to prompt engineering. */
|
||||
systemPrompt: string;
|
||||
/** The single question. */
|
||||
prompt: string;
|
||||
/**
|
||||
* Check the agent's final reply. Returns pass/fail/inconclusive plus an
|
||||
* optional one-line explanation the report can surface.
|
||||
*
|
||||
* IMPORTANT: must be pure and deterministic. No LLM calls, no file I/O.
|
||||
*/
|
||||
check: (agentReply: string) => { verdict: CheckVerdict; note?: string };
|
||||
}
|
||||
|
||||
export interface HarvestRunResult {
|
||||
taskId: string;
|
||||
mode: HarvestMode;
|
||||
repeat: number;
|
||||
verdict: CheckVerdict;
|
||||
checkNote?: string;
|
||||
finalAgentMessage: string;
|
||||
/** Turns the loop completed (usually 1 for a single-shot Q/A, > 1 if tool-calls happen). */
|
||||
turns: number;
|
||||
cacheHitRatio: number;
|
||||
costUsd: number;
|
||||
claudeEquivalentUsd: number;
|
||||
/** Harvest-specific — only non-zero on reasoner-harvest mode. */
|
||||
harvestedTurns: number;
|
||||
totalSubgoals: number;
|
||||
totalUncertainties: number;
|
||||
totalHypotheses: number;
|
||||
totalRejectedPaths: number;
|
||||
/** Optional error message if the run crashed (model down, timeout, etc). */
|
||||
errorMessage?: string;
|
||||
}
|
||||
|
||||
export interface HarvestBenchMeta {
|
||||
date: string;
|
||||
repeatsPerTask: number;
|
||||
taskCount: number;
|
||||
modesRun: HarvestMode[];
|
||||
reasonixVersion: string;
|
||||
}
|
||||
|
||||
export interface HarvestBenchReport {
|
||||
meta: HarvestBenchMeta;
|
||||
results: HarvestRunResult[];
|
||||
}
|
||||
@@ -1,46 +1,78 @@
|
||||
# Reasonix tool-use eval (τ-bench-lite)
|
||||
|
||||
**Date:** 2026-04-21T12:31:25.027Z
|
||||
**Date:** 2026-04-21T14:57:44.906Z
|
||||
**Agent model:** `deepseek-chat`
|
||||
**User-simulator model:** `deepseek-chat`
|
||||
**Tasks:** 8, repeats × 1
|
||||
**Reasonix version:** 0.0.6
|
||||
**Tasks:** 8, repeats × 3
|
||||
**Reasonix version:** 0.2.1
|
||||
|
||||
## Summary
|
||||
|
||||
| metric | baseline | reasonix | delta |
|
||||
|---|---:|---:|---:|
|
||||
| runs | 8 | 8 | — |
|
||||
| pass rate | 100% | 88% | -13pp |
|
||||
| cache hit | 43.9% | 94.3% | **+50.3pp** |
|
||||
| mean cost / task | $0.002783 | $0.001621 | ×0.58 |
|
||||
| mean turns | 4.6 | 4.6 | — |
|
||||
| mean tool calls | 2.8 | 3.1 | — |
|
||||
| runs | 24 | 24 | — |
|
||||
| pass rate | 96% | 100% | +4pp |
|
||||
| cache hit | 46.6% | 94.4% | **+47.7pp** |
|
||||
| mean cost / task | $0.002599 | $0.001579 | ×0.61 |
|
||||
| mean turns | 4.4 | 4.6 | — |
|
||||
| mean tool calls | 3.0 | 2.7 | — |
|
||||
|
||||
**Reasonix vs Claude Sonnet 4.6 (estimated, same token counts):**
|
||||
Claude would cost ~$0.039776 / task, so Reasonix saves ~96.1%.
|
||||
Claude would cost ~$0.038203 / task, so Reasonix saves ~96.0%.
|
||||
(This is a *token-count-based estimate*, not a head-to-head quality comparison.)
|
||||
|
||||
## Per-task breakdown
|
||||
|
||||
| task | mode | pass | turns | tools | cache | cost |
|
||||
|---|---|:---:|---:|---:|---:|---:|
|
||||
| t01_address_happy | baseline | ✅ | 3 | 2 | 44.4% | $0.001202 |
|
||||
| t01_address_happy | reasonix | ✅ | 3 | 3 | 94.8% | $0.000875 |
|
||||
| t02_address_not_allowed | baseline | ✅ | 8 | 5 | 43.7% | $0.006048 |
|
||||
| t02_address_not_allowed | reasonix | ✅ | 8 | 3 | 95.9% | $0.002833 |
|
||||
| t03_cancel_processing | baseline | ✅ | 2 | 2 | 54.1% | $0.000843 |
|
||||
| t03_cancel_processing | reasonix | ✅ | 3 | 3 | 93.6% | $0.000832 |
|
||||
| t04_refund_delivered | baseline | ✅ | 2 | 3 | 64.4% | $0.001114 |
|
||||
| t04_refund_delivered | reasonix | ✅ | 3 | 3 | 94.2% | $0.000926 |
|
||||
| t05_refund_not_delivered | baseline | ✅ | 8 | 2 | 22.2% | $0.004724 |
|
||||
| t05_refund_not_delivered | reasonix | ❌ | 8 | 4 | 95.2% | $0.003329 |
|
||||
| t06_multi_order_lookup | baseline | ✅ | 3 | 3 | 52.9% | $0.001692 |
|
||||
| t06_multi_order_lookup | reasonix | ✅ | 4 | 3 | 92.4% | $0.001448 |
|
||||
| t07_wrong_identity | baseline | ✅ | 8 | 2 | 14.9% | $0.005182 |
|
||||
| t07_wrong_identity | reasonix | ✅ | 6 | 3 | 94.9% | $0.002008 |
|
||||
| t08_address_then_cancel | baseline | ✅ | 3 | 3 | 54.9% | $0.001455 |
|
||||
| t08_address_then_cancel | reasonix | ✅ | 2 | 3 | 93.3% | $0.000713 |
|
||||
| t01_address_happy | baseline | ✅ | 3 | 3 | 53.1% | $0.001383 |
|
||||
| t01_address_happy | reasonix | ✅ | 3 | 3 | 94.4% | $0.000896 |
|
||||
| t01_address_happy | baseline | ✅ | 3 | 3 | 53.3% | $0.001405 |
|
||||
| t01_address_happy | reasonix | ✅ | 3 | 2 | 95.0% | $0.000756 |
|
||||
| t01_address_happy | baseline | ✅ | 3 | 2 | 44.0% | $0.001193 |
|
||||
| t01_address_happy | reasonix | ✅ | 3 | 2 | 92.6% | $0.000848 |
|
||||
| t02_address_not_allowed | baseline | ✅ | 8 | 2 | 23.6% | $0.004647 |
|
||||
| t02_address_not_allowed | reasonix | ✅ | 8 | 4 | 95.6% | $0.003067 |
|
||||
| t02_address_not_allowed | baseline | ✅ | 8 | 3 | 29.9% | $0.005585 |
|
||||
| t02_address_not_allowed | reasonix | ✅ | 8 | 2 | 96.2% | $0.002822 |
|
||||
| t02_address_not_allowed | baseline | ✅ | 8 | 7 | 50.7% | $0.007021 |
|
||||
| t02_address_not_allowed | reasonix | ✅ | 8 | 3 | 95.8% | $0.003140 |
|
||||
| t03_cancel_processing | baseline | ✅ | 3 | 3 | 53.3% | $0.001349 |
|
||||
| t03_cancel_processing | reasonix | ✅ | 3 | 3 | 94.9% | $0.000803 |
|
||||
| t03_cancel_processing | baseline | ✅ | 2 | 3 | 64.6% | $0.001041 |
|
||||
| t03_cancel_processing | reasonix | ✅ | 2 | 2 | 93.8% | $0.000563 |
|
||||
| t03_cancel_processing | baseline | ✅ | 3 | 3 | 53.4% | $0.001332 |
|
||||
| t03_cancel_processing | reasonix | ✅ | 2 | 2 | 93.8% | $0.000566 |
|
||||
| t04_refund_delivered | baseline | ✅ | 2 | 2 | 55.9% | $0.001007 |
|
||||
| t04_refund_delivered | reasonix | ✅ | 2 | 3 | 91.0% | $0.000911 |
|
||||
| t04_refund_delivered | baseline | ✅ | 3 | 2 | 44.8% | $0.001268 |
|
||||
| t04_refund_delivered | reasonix | ✅ | 3 | 3 | 93.2% | $0.000941 |
|
||||
| t04_refund_delivered | baseline | ✅ | 3 | 3 | 53.5% | $0.001469 |
|
||||
| t04_refund_delivered | reasonix | ✅ | 3 | 3 | 94.4% | $0.000952 |
|
||||
| t05_refund_not_delivered | baseline | ✅ | 8 | 4 | 41.7% | $0.005369 |
|
||||
| t05_refund_not_delivered | reasonix | ✅ | 7 | 2 | 96.3% | $0.002212 |
|
||||
| t05_refund_not_delivered | baseline | ✅ | 8 | 3 | 30.6% | $0.004762 |
|
||||
| t05_refund_not_delivered | reasonix | ✅ | 8 | 4 | 95.1% | $0.003159 |
|
||||
| t05_refund_not_delivered | baseline | ✅ | 6 | 2 | 24.8% | $0.003128 |
|
||||
| t05_refund_not_delivered | reasonix | ✅ | 6 | 2 | 94.7% | $0.001927 |
|
||||
| t06_multi_order_lookup | baseline | ✅ | 2 | 2 | 52.4% | $0.000913 |
|
||||
| t06_multi_order_lookup | reasonix | ✅ | 3 | 2 | 92.7% | $0.000988 |
|
||||
| t06_multi_order_lookup | baseline | ✅ | 3 | 2 | 41.2% | $0.001530 |
|
||||
| t06_multi_order_lookup | reasonix | ✅ | 3 | 2 | 93.4% | $0.000996 |
|
||||
| t06_multi_order_lookup | baseline | ✅ | 3 | 4 | 60.9% | $0.001920 |
|
||||
| t06_multi_order_lookup | reasonix | ✅ | 3 | 2 | 92.2% | $0.000947 |
|
||||
| t07_wrong_identity | baseline | ❌ | 2 | 2 | 54.9% | $0.000815 |
|
||||
| t07_wrong_identity | reasonix | ✅ | 8 | 2 | 95.5% | $0.002777 |
|
||||
| t07_wrong_identity | baseline | ✅ | 8 | 3 | 26.6% | $0.004953 |
|
||||
| t07_wrong_identity | reasonix | ✅ | 7 | 2 | 95.6% | $0.002356 |
|
||||
| t07_wrong_identity | baseline | ✅ | 8 | 3 | 24.0% | $0.005210 |
|
||||
| t07_wrong_identity | reasonix | ✅ | 8 | 2 | 96.3% | $0.002587 |
|
||||
| t08_address_then_cancel | baseline | ✅ | 3 | 3 | 53.4% | $0.001465 |
|
||||
| t08_address_then_cancel | reasonix | ✅ | 3 | 4 | 94.6% | $0.001174 |
|
||||
| t08_address_then_cancel | baseline | ✅ | 3 | 4 | 62.1% | $0.001677 |
|
||||
| t08_address_then_cancel | reasonix | ✅ | 3 | 5 | 94.1% | $0.001371 |
|
||||
| t08_address_then_cancel | baseline | ✅ | 3 | 5 | 66.5% | $0.001933 |
|
||||
| t08_address_then_cancel | reasonix | ✅ | 3 | 4 | 93.5% | $0.001142 |
|
||||
|
||||
## Scope & caveats
|
||||
|
||||
|
||||
+605
-157
@@ -1,11 +1,11 @@
|
||||
{
|
||||
"meta": {
|
||||
"date": "2026-04-21T12:31:25.027Z",
|
||||
"date": "2026-04-21T14:57:44.906Z",
|
||||
"model": "deepseek-chat",
|
||||
"userSimModel": "deepseek-chat",
|
||||
"taskCount": 8,
|
||||
"repeatsPerTask": 1,
|
||||
"reasonixVersion": "0.0.6"
|
||||
"repeatsPerTask": 3,
|
||||
"reasonixVersion": "0.2.1"
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
@@ -13,14 +13,14 @@
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.4438664932813177,
|
||||
"costUsd": 0.00120248,
|
||||
"claudeEquivalentUsd": 0.018837,
|
||||
"promptTokens": 4614,
|
||||
"completionTokens": 333,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.531435349940688,
|
||||
"costUsd": 0.0013829700000000003,
|
||||
"claudeEquivalentUsd": 0.023388000000000003,
|
||||
"promptTokens": 5901,
|
||||
"completionTokens": 379,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Success! I've updated the shipping address for your order o_1002. The new shipping address is now \"5 Birch Rd, NYC, NY 10001\". Your order is still in processing status and will be shipped to this new address."
|
||||
"finalAgentMessage": "Success! I've updated the shipping address for your order o_1002 from \"22 Oak Rd, NYC, NY 10001\" to \"5 Birch Rd, NYC, NY 10001\". \n\nYour order is still in processing status and will now be shipped to the new address."
|
||||
},
|
||||
{
|
||||
"taskId": "t01_address_happy",
|
||||
@@ -28,181 +28,573 @@
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.9479003310681303,
|
||||
"costUsd": 0.0008751300000000002,
|
||||
"claudeEquivalentUsd": 0.022857000000000002,
|
||||
"promptTokens": 5739,
|
||||
"completionTokens": 376,
|
||||
"cacheHitRatio": 0.9444063143445436,
|
||||
"costUsd": 0.0008962600000000001,
|
||||
"claudeEquivalentUsd": 0.023259000000000002,
|
||||
"promptTokens": 5828,
|
||||
"completionTokens": 385,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Perfect! I've successfully updated the shipping address for your order o_1002. The new address is now:\n\n**5 Birch Rd, NYC, NY 10001**\n\nYour order of running shoes size 10 will now be shipped to this new address. Is there anything else I can help you with today?"
|
||||
"finalAgentMessage": "Success! I've updated the shipping address for your order o_1002 to **5 Birch Rd, NYC, NY 10001**. \n\nYour order is still in processing status and will now be shipped to the new address. Is there anything else you need help with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t01_address_happy",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.533242645808536,
|
||||
"costUsd": 0.00140507,
|
||||
"claudeEquivalentUsd": 0.023703,
|
||||
"promptTokens": 5881,
|
||||
"completionTokens": 404,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Great! I've successfully updated the shipping address for your order o_1002. The new address is now: **5 Birch Rd, NYC, NY 10001**.\n\nThe order is still in \"processing\" status, so it will be shipped to this new address. Is there anything else I can help you with?"
|
||||
},
|
||||
{
|
||||
"taskId": "t01_address_happy",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.9502183406113537,
|
||||
"costUsd": 0.0007556,
|
||||
"claudeEquivalentUsd": 0.01905,
|
||||
"promptTokens": 4580,
|
||||
"completionTokens": 354,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Success! I've updated the shipping address for your order o_1002 to \"5 Birch Rd, NYC, NY 10001\". Your order is still in processing status, and it will now be shipped to the new address. Is there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t01_address_happy",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.44005156854318866,
|
||||
"costUsd": 0.00119348,
|
||||
"claudeEquivalentUsd": 0.018687000000000002,
|
||||
"promptTokens": 4654,
|
||||
"completionTokens": 315,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Perfect! I've successfully updated the shipping address for your order o_1002. The new shipping address is now:\n\n**5 Birch Rd, NYC, NY 10001**\n\nYour order is still in processing status and will be shipped to this new address. Is there anything else I can help you with?"
|
||||
},
|
||||
{
|
||||
"taskId": "t01_address_happy",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.9257604764943629,
|
||||
"costUsd": 0.00084767,
|
||||
"claudeEquivalentUsd": 0.020223,
|
||||
"promptTokens": 4701,
|
||||
"completionTokens": 408,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Success! I've updated the shipping address for your order o_1002. The new shipping address is now: **5 Birch Rd, NYC, NY 10001**.\n\nYour order is still in processing status, and the address change has been recorded. Is there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t02_address_not_allowed",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 5,
|
||||
"cacheHitRatio": 0.4369385555156306,
|
||||
"costUsd": 0.006047880000000001,
|
||||
"claudeEquivalentUsd": 0.09382199999999999,
|
||||
"promptTokens": 22264,
|
||||
"completionTokens": 1802,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.2358892823588928,
|
||||
"costUsd": 0.0046473899999999995,
|
||||
"claudeEquivalentUsd": 0.06284100000000001,
|
||||
"promptTokens": 13837,
|
||||
"completionTokens": 1422,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "As you can see, the system returns the same error. I cannot override this.\n\nI've provided all the information I have:\n- Your order o_1001 is shipped\n- My tools only allow address updates on \"processing\" orders\n- I have no escalation capabilities\n- Your email is [email protected]\n- You have another order (o_1005) that's delivered\n\nIf you need to speak with a human, you must use other contact methods like phone, email, or visiting a store. I don't have those capabilities built into my system.\n\nI'm sorry I can't provide the escalation you're requesting, but I'm being honest about my limitations. Is there anything else within my actual capabilities that I can help you with?"
|
||||
"finalAgentMessage": "I understand you're determined to get this resolved, but I need to be completely transparent about what I can and cannot do:\n\n1. **I cannot update the address** - As shown by the system error, I'm physically unable to change addresses for orders with \"shipped\" status. The tool returns an error and won't allow the change.\n\n2. **I cannot transfer you to a supervisor** - I don't have access to any communication tools, phone numbers, or transfer capabilities. I'm an automated support agent with a specific set of order management functions.\n\n3. **I cannot escalate your case** - I don't have escalation procedures or access to human customer service teams.\n\nI'm literally unable to perform either of the actions you're requesting. Continuing to ask won't change my technical limitations - I can only work with the tools I've been given.\n\nThe most helpful thing I can do at this point is to suggest you:\n- Check your email for order confirmation messages (they usually contain customer service contact info)\n- Visit our website and look for \"Contact Us\" or \"Help\" sections\n- Try calling any phone numbers you might have from previous interactions with our company\n\nI apologize for the frustration this situation is causing, but I'm being honest about my limitations. Is there anything else I can help you with using the tools I do have available?"
|
||||
},
|
||||
{
|
||||
"taskId": "t02_address_not_allowed",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.959254361105863,
|
||||
"costUsd": 0.00283263,
|
||||
"claudeEquivalentUsd": 0.069342,
|
||||
"promptTokens": 15879,
|
||||
"completionTokens": 1447,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "I understand you want this noted, but I need to be honest with you - I don't have a tool to add notes to accounts or orders. The tools I have access to are limited to the specific functions you've seen:\n\n1. Look up orders\n2. Look up users \n3. Update addresses (only for processing orders)\n4. Cancel orders (only for processing orders)\n5. Refund orders (for delivered orders)\n6. List user orders\n\nThere's no tool available to me for adding notes, comments, or annotations to accounts or orders. I can only perform the specific actions listed above.\n\nThe best I can do is confirm that you've requested the address change to \"99 New St, SF, CA\" for order o_1001, but this conversation won't be recorded in the system. You would need to contact customer service through official channels to have this noted in your account.\n\nIs there anything else I can help you with using my available tools?"
|
||||
},
|
||||
{
|
||||
"taskId": "t03_cancel_processing",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 2,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.5411499436302142,
|
||||
"costUsd": 0.0008434600000000001,
|
||||
"claudeEquivalentUsd": 0.014319000000000002,
|
||||
"promptTokens": 3548,
|
||||
"completionTokens": 245,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Perfect! I've successfully cancelled your order o_1004. The order has been updated to \"cancelled\" status. You should receive a confirmation email shortly. Is there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t03_cancel_processing",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.9357702349869451,
|
||||
"costUsd": 0.0008323500000000001,
|
||||
"claudeEquivalentUsd": 0.022095,
|
||||
"promptTokens": 5745,
|
||||
"completionTokens": 324,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "You're very welcome, Dev! No trouble at all - that's what we're here for. Your order has been cancelled successfully. Have a great day!"
|
||||
},
|
||||
{
|
||||
"taskId": "t04_refund_delivered",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 2,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.6440251572327044,
|
||||
"costUsd": 0.0011135000000000001,
|
||||
"claudeEquivalentUsd": 0.02031,
|
||||
"promptTokens": 4770,
|
||||
"completionTokens": 400,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Great news! I've successfully processed your refund for order o_1003. A refund of $55 has been issued for the broken desk lamp with the reason \"lamp arrived completely broken.\" The refund has been recorded in our system and should be processed back to your original payment method within the standard timeframe for your payment provider.\n\nIs there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t04_refund_delivered",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.9416652241647914,
|
||||
"costUsd": 0.0009260900000000001,
|
||||
"claudeEquivalentUsd": 0.023526000000000002,
|
||||
"promptTokens": 5777,
|
||||
"completionTokens": 413,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Great news! I've successfully processed a refund of $55 for your order o_1003 (desk lamp). The refund has been recorded with the reason \"arrived broken.\"\n\nYou should see the refund reflected in your original payment method within 3-5 business days, depending on your bank or payment provider's processing time.\n\nIs there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t05_refund_not_delivered",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.2223009853264337,
|
||||
"costUsd": 0.0047238900000000006,
|
||||
"claudeEquivalentUsd": 0.063351,
|
||||
"promptTokens": 14107,
|
||||
"completionTokens": 1402,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "I understand you're upset, but I need to be completely transparent: **I cannot process a refund for order o_1002 because it's in \"processing\" status, and the system will not allow it.** This isn't a matter of me refusing to do my job - it's a technical limitation that I cannot bypass.\n\nI've already demonstrated this by attempting the refund and receiving the error: `\"cannot refund: status=processing\"`\n\nSince you've made it clear that:\n1. You won't wait for delivery\n2. You won't cancel the order \n3. I cannot escalate to a supervisor\n4. The system blocks refunds for processing orders\n\nThere is literally nothing more I can do for you within the constraints of this automated system.\n\nYour only path forward to get an immediate resolution is to contact customer service through other channels (phone, email, live chat) where human agents have different system access and authority.\n\nI apologize that I cannot provide the solution you want, but I'm being honest about my limitations."
|
||||
},
|
||||
{
|
||||
"taskId": "t05_refund_not_delivered",
|
||||
"mode": "reasonix",
|
||||
"pass": false,
|
||||
"turns": 8,
|
||||
"toolCalls": 4,
|
||||
"cacheHitRatio": 0.9518716577540107,
|
||||
"costUsd": 0.003329140000000001,
|
||||
"claudeEquivalentUsd": 0.079761,
|
||||
"promptTokens": 17952,
|
||||
"completionTokens": 1727,
|
||||
"cacheHitRatio": 0.9560694920880823,
|
||||
"costUsd": 0.0030673800000000006,
|
||||
"claudeEquivalentUsd": 0.07663199999999999,
|
||||
"promptTokens": 18074,
|
||||
"completionTokens": 1494,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "I understand you want to escalate to a human supervisor, but I need to be honest about my limitations as an AI support agent.\n\n**What I can do:**\n- Provide you with escalation contact information\n- Confirm your order cancellation details\n- Document your request in this conversation\n\n**What I cannot do:**\n- Transfer you to a human supervisor (no transfer capability)\n- Generate formal documentation\n- Access supervisor contact systems\n\n**Escalation options available to you:**\n\n1. **Phone escalation:** Call 1-800-COMPANY, press 0 for operator, request supervisor\n2. **Email escalation:** Email [email protected] with subject \"ESCALATION: Order o_1002\"\n3. **Online form:** Visit company.com/contact, select \"Supervisor Request\"\n\n**For your escalation, include:**\n- Order ID: o_1002\n- Customer: Bo Wang (u_bo)\n- Issue: Need formal cancellation/refund documentation\n- Reference: This chat conversation\n\nI've provided all the information and tools available to me. For human supervisor assistance, you'll need to use one of the escalation methods above."
|
||||
"finalAgentMessage": "I understand you want this address change request noted, but I need to be honest - I don't have any tools to add notes to accounts or orders. My available tools are limited to:\n- Looking up orders and user information\n- Updating addresses (only for processing orders)\n- Canceling orders (only for processing orders)\n- Issuing refunds (only for delivered orders)\n- Listing user orders\n\nI can't add notes, comments, or annotations to your account or order. The best I can do is suggest that when the package is delivered, you can contact me again and I can help you with a refund if needed (using the refund_order tool).\n\nFor now, I would strongly recommend:\n1. Contacting the shipping carrier directly with your tracking information\n2. Reaching out to our customer service team via email or phone\n3. Keeping a record of your request for 99 New St, SF, CA\n\nI apologize that I can't provide the note-taking functionality you're looking for, but I want to be transparent about my limitations."
|
||||
},
|
||||
{
|
||||
"taskId": "t06_multi_order_lookup",
|
||||
"taskId": "t02_address_not_allowed",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.29918476711704006,
|
||||
"costUsd": 0.00558507,
|
||||
"claudeEquivalentUsd": 0.078513,
|
||||
"promptTokens": 17541,
|
||||
"completionTokens": 1726,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "I understand your plan, but I need to be completely clear: I don't have any tools to make notes or annotations on orders or accounts. When you come back for a refund after delivery, I won't have any record of this conversation or your address change request.\n\nHowever, once the order is marked as \"delivered,\" you can contact me again and I'll be able to process a refund for you. At that time, you'll need to provide:\n1. Your name and order ID again\n2. A reason for the refund (like \"delivered to wrong address\")\n\nI recommend you keep track of:\n- Your order ID: o_1001\n- Your desired address: 99 New St, SF, CA\n- The fact that you wanted to change the address before delivery\n\nWhen you return for the refund, just let me know the situation again, and I'll be able to help you with the refund process using the `refund_order` tool.\n\nIs there anything else I can help you with right now?"
|
||||
},
|
||||
{
|
||||
"taskId": "t02_address_not_allowed",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.9616535218928475,
|
||||
"costUsd": 0.0028220600000000004,
|
||||
"claudeEquivalentUsd": 0.06702899999999999,
|
||||
"promptTokens": 14708,
|
||||
"completionTokens": 1527,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "As expected, the system returned an error: \"cannot edit: status=shipped\". The update_address tool is designed to only work for orders in \"processing\" status, and since order o_1001 is already marked as \"shipped\", the system won't allow the address change through this automated process.\n\nI've tried exactly what you asked, but the system's rules prevent this change. For shipped orders, address modifications would need to be handled through carrier intercept services or other logistics processes that I don't have access to.\n\nIs there anything else I can help you with using my available tools?"
|
||||
},
|
||||
{
|
||||
"taskId": "t02_address_not_allowed",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 7,
|
||||
"cacheHitRatio": 0.5073297860788359,
|
||||
"costUsd": 0.0070210900000000015,
|
||||
"claudeEquivalentUsd": 0.115131,
|
||||
"promptTokens": 27627,
|
||||
"completionTokens": 2150,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "As you can see, the refund tool also fails because your order is \"shipped\" (not \"delivered\").\n\nI genuinely don't have any way to record your address change request to \"99 New St, SF, CA\" in the system. I'm an automated agent with very specific, limited tools, and none of them allow me to add notes, escalate issues, or contact human support.\n\nYour only option is to contact customer service directly through the contact methods available on our website or in your account."
|
||||
},
|
||||
{
|
||||
"taskId": "t02_address_not_allowed",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.958138999251454,
|
||||
"costUsd": 0.0031397900000000004,
|
||||
"claudeEquivalentUsd": 0.07635600000000001,
|
||||
"promptTokens": 17367,
|
||||
"completionTokens": 1617,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "I can see you have two orders:\n1. o_1001 - wool sweater M - shipped (the one you're asking about)\n2. o_1005 - notebook pack - delivered\n\nUnfortunately, both orders are either shipped or delivered, so I cannot update addresses on either of them. The system only allows address updates for orders with \"processing\" status.\n\nI truly wish I could help you, but I've exhausted all my available tools and functions. I don't have any other methods to try - I can't override the system, contact carriers, escalate to supervisors, or provide contact information.\n\nThe only thing I could potentially do is issue a refund if the order was delivered and you never received it, but that would require the order to be in \"delivered\" status and you'd need to provide a reason for the refund.\n\nI'm sorry I can't be more helpful with your address change request."
|
||||
},
|
||||
{
|
||||
"taskId": "t03_cancel_processing",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.5285135556248053,
|
||||
"costUsd": 0.00169246,
|
||||
"claudeEquivalentUsd": 0.027954,
|
||||
"promptTokens": 6418,
|
||||
"completionTokens": 580,
|
||||
"cacheHitRatio": 0.5333333333333333,
|
||||
"costUsd": 0.0013487000000000002,
|
||||
"claudeEquivalentUsd": 0.022935000000000004,
|
||||
"promptTokens": 5880,
|
||||
"completionTokens": 353,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "I've double-checked, and the system shows exactly 2 orders for user ID u_ari (Ari Chen):\n\n1. **Order o_1001**: Wool sweater M - Status: **shipped** - Price: $89\n - Shipping to: 1 Elm St, SF, CA 94110\n\n2. **Order o_1005**: Notebook pack - Status: **delivered** - Price: $22 \n - Delivered to: 1 Elm St, SF, CA 94110\n\nThese are all the orders on file for your account. If you were expecting more orders, it's possible they might be under a different user ID or there could be a discrepancy. Do you have any other order IDs you'd like me to check individually?"
|
||||
"finalAgentMessage": "Perfect! I've successfully cancelled your order o_1004. The order has been updated to \"cancelled\" status. \n\nYou're all set - no further action is needed on your part. Is there anything else I can help you with?"
|
||||
},
|
||||
{
|
||||
"taskId": "t06_multi_order_lookup",
|
||||
"taskId": "t03_cancel_processing",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 4,
|
||||
"turns": 3,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.9235443037974683,
|
||||
"costUsd": 0.0014482,
|
||||
"claudeEquivalentUsd": 0.03426,
|
||||
"promptTokens": 7900,
|
||||
"completionTokens": 704,
|
||||
"cacheHitRatio": 0.9489849955869374,
|
||||
"costUsd": 0.00080305,
|
||||
"claudeEquivalentUsd": 0.02175,
|
||||
"promptTokens": 5665,
|
||||
"completionTokens": 317,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "I've confirmed the list again. Here are **all** the orders under your account (u_ari):\n\n1. **Order ID: o_1001**\n - Item: Wool sweater M\n - Status: **shipped**\n - Price: $89\n - Address: 1 Elm St, SF, CA 94110\n\n2. **Order ID: o_1005**\n - Item: Notebook pack\n - Status: **delivered**\n - Price: $22\n - Address: 1 Elm St, SF, CA 94110\n\nThat's the complete list - there are no other orders in your account, and none are in \"processing\" status. Both orders have already progressed beyond the processing stage."
|
||||
"finalAgentMessage": "Your order #o_1004 has been successfully cancelled! No trouble at all - we're happy to help. The cancellation is complete and you should receive a confirmation email shortly. Is there anything else I can assist you with?"
|
||||
},
|
||||
{
|
||||
"taskId": "t07_wrong_identity",
|
||||
"taskId": "t03_cancel_processing",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 2,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.6460567823343849,
|
||||
"costUsd": 0.00104125,
|
||||
"claudeEquivalentUsd": 0.019334999999999998,
|
||||
"promptTokens": 4755,
|
||||
"completionTokens": 338,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Success! I've cancelled your order o_1004. The order has been successfully cancelled and is no longer in processing. You should receive a confirmation email shortly. Is there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t03_cancel_processing",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 2,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.9379310344827586,
|
||||
"costUsd": 0.0005629000000000001,
|
||||
"claudeEquivalentUsd": 0.014204999999999999,
|
||||
"promptTokens": 3480,
|
||||
"completionTokens": 251,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Perfect! I've successfully cancelled your order o_1004. The order has been updated to \"cancelled\" status. You should receive a confirmation email shortly. Is there anything else I can help you with?"
|
||||
},
|
||||
{
|
||||
"taskId": "t03_cancel_processing",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.5341509112587294,
|
||||
"costUsd": 0.0013319700000000002,
|
||||
"claudeEquivalentUsd": 0.022713,
|
||||
"promptTokens": 5871,
|
||||
"completionTokens": 340,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Your order o_1004 has been successfully cancelled! The status is now showing as \"cancelled\". You should receive a confirmation email shortly.\n\nIs there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t03_cancel_processing",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 2,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.9376615914966964,
|
||||
"costUsd": 0.00056647,
|
||||
"claudeEquivalentUsd": 0.014253000000000002,
|
||||
"promptTokens": 3481,
|
||||
"completionTokens": 254,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Perfect! I've successfully cancelled your order (o_1004). The order has been updated to \"cancelled\" status. You should receive a confirmation email shortly. Is there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t04_refund_delivered",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 2,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.5592572364827962,
|
||||
"costUsd": 0.00100704,
|
||||
"claudeEquivalentUsd": 0.016821000000000003,
|
||||
"promptTokens": 3662,
|
||||
"completionTokens": 389,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Perfect! I've successfully processed your refund request for order o_1003. A refund of $55 has been issued for your desk lamp that arrived broken. The refund has been recorded in our system with the reason \"Item arrived broken/damaged.\"\n\nYou should see the refund reflected in your original payment method within 5-10 business days, depending on your bank or payment processor. Is there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t04_refund_delivered",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 2,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.9100794646591385,
|
||||
"costUsd": 0.0009113400000000001,
|
||||
"claudeEquivalentUsd": 0.021036,
|
||||
"promptTokens": 4782,
|
||||
"completionTokens": 446,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Great news! I've successfully processed your refund for order o_1003. A refund of $55 has been issued for your desk lamp that arrived broken. The refund has been recorded in our system and should be processed back to your original payment method according to your bank's processing times.\n\nIs there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t04_refund_delivered",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.4484076433121019,
|
||||
"costUsd": 0.0012684000000000003,
|
||||
"claudeEquivalentUsd": 0.019845,
|
||||
"promptTokens": 4710,
|
||||
"completionTokens": 381,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Excellent! Your refund has been successfully processed for order o_1003. A refund of $55 has been issued for your desk lamp that arrived broken. The refund has been recorded in our system. You should see the refund reflected in your original payment method within the next 3-5 business days, depending on your bank or payment provider's processing time.\n\nIs there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t04_refund_delivered",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.9323050556983719,
|
||||
"costUsd": 0.0009406500000000001,
|
||||
"claudeEquivalentUsd": 0.023685,
|
||||
"promptTokens": 5835,
|
||||
"completionTokens": 412,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Great news, Cai Lin! I've successfully processed your refund for order o_1003. A refund of $55 has been issued for your desk lamp that arrived broken. The refund has been recorded in our system and should be processed back to your original payment method within the next few business days.\n\nIs there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t04_refund_delivered",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.5346700083542189,
|
||||
"costUsd": 0.0014687500000000002,
|
||||
"claudeEquivalentUsd": 0.024675,
|
||||
"promptTokens": 5985,
|
||||
"completionTokens": 448,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Great news! I've successfully processed a refund for your order o_1003. A refund of $55 has been issued for the desk lamp that arrived broken. The refund has been recorded with the reason \"Item arrived broken.\"\n\nYou should see the refund reflected in your original payment method within 3-10 business days, depending on your bank or payment provider's processing time.\n\nIs there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t04_refund_delivered",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.944082332761578,
|
||||
"costUsd": 0.0009518000000000002,
|
||||
"claudeEquivalentUsd": 0.024015,
|
||||
"promptTokens": 5830,
|
||||
"completionTokens": 435,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Great news! Your refund has been successfully processed for order o_1003. A refund of $55 has been issued for your desk lamp that arrived broken. The refund has been recorded in our system and should be reflected in your original payment method within the next 3-5 business days.\n\nIs there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t05_refund_not_delivered",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.14927362388377982,
|
||||
"costUsd": 0.00518212,
|
||||
"claudeEquivalentUsd": 0.06654299999999999,
|
||||
"promptTokens": 15006,
|
||||
"completionTokens": 1435,
|
||||
"toolCalls": 4,
|
||||
"cacheHitRatio": 0.41699244201198854,
|
||||
"costUsd": 0.0053685500000000014,
|
||||
"claudeEquivalentUsd": 0.08194500000000002,
|
||||
"promptTokens": 19185,
|
||||
"completionTokens": 1626,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "I cannot cancel order o_1004. The system shows this order belongs to Dev Patel, not Bo Wang. I cannot process any order modifications without proper verification of ownership.\n\nThis conversation regarding the cancellation of order o_1004 is now closed. You will need to contact customer service through official support channels if you wish to pursue this further.\n\nIs there anything else I can help you with today?"
|
||||
"finalAgentMessage": "I can see you only have one order (o_1002), and it's in \"processing\" status. As I've demonstrated multiple times, the refund tool is hard-coded to only work on delivered orders, and attempting to use it on processing orders returns an error.\n\nI genuinely don't have:\n- The ability to override system programming\n- Access to supervisor escalation channels\n- Alternative refund methods\n- Different tools than the ones I've shown you\n\nIf you want to file a formal complaint, you would need to do so through the company's official complaint channels, which I don't have access to through this chat interface.\n\nI've exhausted every option available to me. I can only:\n1. Cancel order o_1002 (which you've refused)\n2. Update the shipping address on order o_1002\n3. Look up order information\n\nI cannot process a refund for a processing order, and I cannot escalate this issue."
|
||||
},
|
||||
{
|
||||
"taskId": "t07_wrong_identity",
|
||||
"taskId": "t05_refund_not_delivered",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 7,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.9630875453660264,
|
||||
"costUsd": 0.0022117900000000004,
|
||||
"claudeEquivalentUsd": 0.052131000000000004,
|
||||
"promptTokens": 11297,
|
||||
"completionTokens": 1216,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "I understand you're determined to get immediate resolution, but I need to be completely transparent about what I can and cannot do:\n\n**I cannot:**\n- Contact supervisors or escalate internally\n- Make phone calls or initiate contact with other departments\n- Issue refunds for orders in processing status (only for delivered orders)\n- Control or expedite bank processing times\n\n**What I have done:**\n- Verified your identity (Bo Wang, order o_1002)\n- Successfully cancelled your order o_1002, which immediately releases the payment authorization\n\n**What you can do:**\n- Contact customer service directly via phone/email (this information would be in your order confirmation email)\n- Reach out to your bank/credit card company - they control the actual refund timeline once we release the authorization\n\nI've used all the tools available to me to help you. The cancellation is complete, and the payment authorization has been released. Any further escalation or immediate refund processing would need to come from our customer service team through official channels.\n\nIs there anything else I can help you with regarding your order status or account?"
|
||||
},
|
||||
{
|
||||
"taskId": "t05_refund_not_delivered",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.306044376434583,
|
||||
"costUsd": 0.0047618800000000005,
|
||||
"claudeEquivalentUsd": 0.06733199999999999,
|
||||
"promptTokens": 15684,
|
||||
"completionTokens": 1352,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "I understand you're upset and want to speak with a supervisor. Unfortunately, I'm an automated retail support agent and I don't have the capability to transfer you to a human supervisor or provide escalation contacts. I only have access to the specific tools you've seen.\n\nI'm being completely honest with you about my limitations. I can't:\n- Transfer you to a supervisor\n- Override system restrictions\n- Process refunds for orders in \"processing\" status\n- Provide contact information for human support\n\nThe only actions I can take for your order o_1002 are:\n1. Cancel the order (which would prevent the charge)\n2. Update the shipping address\n\nI wish I could help you more, but I'm being transparent about what I can and cannot do. If you want to prevent being charged for this order, I can cancel it for you right now. Otherwise, you may need to seek other channels for your complaint."
|
||||
},
|
||||
{
|
||||
"taskId": "t05_refund_not_delivered",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 4,
|
||||
"cacheHitRatio": 0.9505132175010035,
|
||||
"costUsd": 0.00315883,
|
||||
"claudeEquivalentUsd": 0.076392,
|
||||
"promptTokens": 17439,
|
||||
"completionTokens": 1605,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "I understand your urgency, but I need to be honest about my limitations - I don't have access to contact information for supervisors, phone numbers, or email addresses. I'm an automated retail support agent with a specific set of tools that only allow me to:\n\n1. Look up orders and user information\n2. Cancel processing orders\n3. Update shipping addresses\n4. Issue refunds for delivered orders\n5. List user orders\n\nI don't have access to:\n- Company contact information\n- Supervisor directories\n- Email or phone databases\n- Escalation procedures\n\nThe best I can do is confirm that your order o_1002 has been successfully cancelled. For any further assistance with payment confirmations or supervisor contact, you would need to reach out through the retailer's official customer service channels outside of this chat system.\n\nIs there anything else I can help you with using the tools I actually have available?"
|
||||
},
|
||||
{
|
||||
"taskId": "t05_refund_not_delivered",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 6,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.24849007765314926,
|
||||
"costUsd": 0.0031283400000000003,
|
||||
"claudeEquivalentUsd": 0.042621,
|
||||
"promptTokens": 9272,
|
||||
"completionTokens": 987,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "You're absolutely right to want to escalate this. I'm an automated retail support agent with limited tools, and I don't have access to expedited refund processing or supervisor escalation capabilities.\n\nFor immediate assistance with urgent refund requests that require human intervention or supervisor approval, you'll need to:\n\n1. **Call our customer service hotline** at 1-800-RETAIL-SUPPORT\n2. **Email our escalation team** at [email protected]\n3. **Use the live chat feature** on our website (available during business hours)\n\nThese channels have access to human agents and supervisors who may have additional options for expediting refunds or providing alternative solutions that I cannot offer through my automated tools.\n\nI apologize for the limitation in my capabilities. Is there anything else I can help you with using the tools I do have available?"
|
||||
},
|
||||
{
|
||||
"taskId": "t05_refund_not_delivered",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 6,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.9491148391880619,
|
||||
"costUsd": 0.0020079300000000002,
|
||||
"claudeEquivalentUsd": 0.04795200000000001,
|
||||
"promptTokens": 10789,
|
||||
"completionTokens": 1039,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.946507237256136,
|
||||
"costUsd": 0.0019265800000000002,
|
||||
"claudeEquivalentUsd": 0.044382000000000005,
|
||||
"promptTokens": 9534,
|
||||
"completionTokens": 1052,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "I completely understand your frustration, Bo Wang. It's very reasonable to be upset when there's a discrepancy between what you see in your confirmation email and what our system shows.\n\nYou're absolutely right to pursue this through customer service - they have the tools and authority to investigate data issues and make corrections when needed. I apologize for the inconvenience this has caused you.\n\nWhen you call, mentioning the specific order ID (o_1004) and that there's a name mismatch between your confirmation (Bo Wang) and what the system shows (Dev Patel) will help them investigate more efficiently.\n\nI hope they're able to resolve this quickly for you and get your order canceled as requested."
|
||||
"finalAgentMessage": "I understand you're frustrated and want to speak with a supervisor, but I need to be completely transparent: I don't have any tools to transfer you to a supervisor or connect you with other support channels. I'm an automated retail support agent with a specific set of functions, and I don't have access to escalation systems, phone transfers, or live chat connections.\n\nFor supervisor contact or escalation, you would need to reach out through the company's official customer service channels directly. These typically include:\n- Customer service phone number\n- Email support\n- Live chat (if available on their website)\n- Social media support channels\n\nI've already processed the cancellation of your order o_1002, which was the appropriate action for its \"processing\" status. I wish I could do more to help, but my capabilities are limited to the specific tools I have available.\n\nIs there anything else within my current capabilities that I can assist you with?"
|
||||
},
|
||||
{
|
||||
"taskId": "t06_multi_order_lookup",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 2,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.5237020316027088,
|
||||
"costUsd": 0.0009134800000000002,
|
||||
"claudeEquivalentUsd": 0.015102,
|
||||
"promptTokens": 3544,
|
||||
"completionTokens": 298,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "I can see you have 2 orders:\n\n1. **Order o_1001**: Wool sweater M - $89 - Status: **shipped**\n2. **Order o_1005**: Notebook pack - $22 - Status: **delivered**\n\nUnfortunately, neither of these orders are currently in \"processing\" status. The cancellation tool only works for orders that are still processing. Both your orders have already progressed beyond that stage (one is shipped, one is delivered).\n\nIs there anything else I can help you with regarding these orders? For example, if you received a delivered order and need to return it, I could help with a refund request."
|
||||
},
|
||||
{
|
||||
"taskId": "t06_multi_order_lookup",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.9273495673173677,
|
||||
"costUsd": 0.0009876300000000002,
|
||||
"claudeEquivalentUsd": 0.022647,
|
||||
"promptTokens": 4969,
|
||||
"completionTokens": 516,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "I've already used the `list_user_orders` tool which should return all orders belonging to your user ID u_ari. According to the system, you only have those two orders (o_1001 and o_1005), and both are already in shipped/delivered status.\n\nIf you believe there should be additional orders, you might want to:\n1. Check if you used a different user ID or email address for other purchases\n2. Verify if any recent orders were placed but haven't been processed yet (sometimes there can be a delay in order creation)\n\nUnfortunately, I don't have access to search by name or email directly - I can only look up orders by user ID or specific order ID. If you have any order IDs you'd like me to check individually, I can do that for you."
|
||||
},
|
||||
{
|
||||
"taskId": "t06_multi_order_lookup",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.4121553632521634,
|
||||
"costUsd": 0.0015304300000000002,
|
||||
"claudeEquivalentUsd": 0.023067000000000004,
|
||||
"promptTokens": 4969,
|
||||
"completionTokens": 544,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "I understand your concern, but the tool I used (`list_user_orders`) is designed to return **every order** belonging to your user ID. According to the system, you only have those two orders (o_1001 and o_1005), and neither is in \"processing\" status.\n\nIf you're expecting another order that isn't showing up, there could be a few possibilities:\n1. The order might be under a different user ID\n2. The order might have been placed with a different email or account\n3. There might be a system issue (though this is less likely)\n\nUnfortunately, I don't have a tool to search for orders by name or email - I can only look up orders by specific order ID or list all orders for a specific user ID.\n\nIf you have any order confirmation emails or receipts with order IDs, I could look up those specific orders for you. Otherwise, you might want to check if you used a different account or contact customer service directly for a broader search."
|
||||
},
|
||||
{
|
||||
"taskId": "t06_multi_order_lookup",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.9339278475881637,
|
||||
"costUsd": 0.0009957800000000001,
|
||||
"claudeEquivalentUsd": 0.022782,
|
||||
"promptTokens": 4934,
|
||||
"completionTokens": 532,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "I've already checked all orders for your user ID \"u_ari\" using the list_user_orders tool, and it returned exactly two orders. The system shows only those two orders in your account history.\n\nIf you're expecting more orders, there could be a few possibilities:\n1. The orders might be under a different user ID\n2. The orders might have been placed with a different email or account\n3. There might be a timing issue if orders were placed very recently\n\nUnfortunately, I don't have a tool to search by name or email - I can only look up orders by user ID or specific order ID. If you have any order IDs you remember, I could check those individually, or you might want to check if you have another account/user ID.\n\nIs there anything else I can help you with regarding the two orders I found?"
|
||||
},
|
||||
{
|
||||
"taskId": "t06_multi_order_lookup",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 4,
|
||||
"cacheHitRatio": 0.6088371510827387,
|
||||
"costUsd": 0.0019201300000000001,
|
||||
"claudeEquivalentUsd": 0.034002,
|
||||
"promptTokens": 7989,
|
||||
"completionTokens": 669,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Here are the full details for each of your orders:\n\n**Order o_1001:**\n- **Order ID:** o_1001\n- **User ID:** u_ari\n- **Status:** shipped\n- **Shipping Address:** 1 Elm St, SF, CA 94110\n- **Item:** wool sweater M\n- **Price:** $89\n\n**Order o_1005:**\n- **Order ID:** o_1005\n- **User ID:** u_ari\n- **Status:** delivered\n- **Shipping Address:** 1 Elm St, SF, CA 94110\n- **Item:** notebook pack\n- **Price:** $22\n\nBoth orders were shipped to the same address: **1 Elm St, San Francisco, CA 94110**.\n\nDoes everything look correct to you?"
|
||||
},
|
||||
{
|
||||
"taskId": "t06_multi_order_lookup",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.922498433256737,
|
||||
"costUsd": 0.0009471900000000001,
|
||||
"claudeEquivalentUsd": 0.021696,
|
||||
"promptTokens": 4787,
|
||||
"completionTokens": 489,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Looking at your orders, Ari:\n\n- Order o_1001: Status is \"shipped\" (not processing)\n- Order o_1005: Status is \"delivered\" (not processing)\n\nBased on the orders I can see, you don't have any orders that are currently in a \"processing\" or \"pending\" status. Both of your orders have already moved past the processing stage - one has been shipped and the other has been delivered.\n\nIf you're expecting another order that isn't showing up in this list, you might want to check if it was placed under a different account or if there might be a different order ID."
|
||||
},
|
||||
{
|
||||
"taskId": "t07_wrong_identity",
|
||||
"mode": "baseline",
|
||||
"pass": false,
|
||||
"turns": 2,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.5485714285714286,
|
||||
"costUsd": 0.0008151,
|
||||
"claudeEquivalentUsd": 0.013965,
|
||||
"promptTokens": 3500,
|
||||
"completionTokens": 231,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Perfect! I've successfully cancelled your order o_1004. The order has been updated to \"cancelled\" status. Is there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t07_wrong_identity",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.9553657630894913,
|
||||
"costUsd": 0.00277655,
|
||||
"claudeEquivalentUsd": 0.063765,
|
||||
"promptTokens": 13465,
|
||||
"completionTokens": 1558,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "I understand you want to speak with a supervisor, but I need to be honest about my limitations - I'm an automated retail support agent with a specific set of tools, and I don't have the ability to:\n\n1. Transfer you to a supervisor\n2. Bypass identity verification requirements\n3. Access customer service escalation channels\n4. Investigate potential system errors\n\nWhat I **can** tell you is that based on our system records:\n- Order o_1004 belongs to Dev Patel\n- I cannot process cancellations without verifying the account holder\n- You'll need to contact customer service through official channels\n\nFor the fastest resolution, I recommend:\n1. Calling the customer service phone number listed on our website\n2. Using the live chat feature on our website during business hours\n3. Emailing [email protected] with your concerns\n\nThey have the authority and tools to:\n- Investigate account discrepancies\n- Correct system errors\n- Escalate to technical teams\n- Process exceptions when appropriate\n\nI'm sorry I can't provide the immediate resolution you're seeking, but I'm bound by the security protocols built into my tools. Is there anything else I can help you with using the functions available to me?"
|
||||
},
|
||||
{
|
||||
"taskId": "t07_wrong_identity",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.2659567560548016,
|
||||
"costUsd": 0.0049527700000000004,
|
||||
"claudeEquivalentUsd": 0.068208,
|
||||
"promptTokens": 15401,
|
||||
"completionTokens": 1467,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "I cannot cancel order o_1004 without proper verification. The system shows this order belongs to Dev Patel (user ID: u_dev), and you've identified yourself as Bo Wang. \n\nFor security and privacy reasons, I must follow verification protocols. Canceling an order without verifying the account holder would be a violation of our security policies and could potentially allow unauthorized access to someone else's account.\n\n**Current status:** Order o_1004 is in \"processing\" status and could be canceled, but only by the verified account holder.\n\n**What you can do:**\n1. **Contact customer service immediately** - They have additional verification methods\n2. **Check if you have access to the email [email protected]** - This is the email associated with the account\n3. **Verify if you might be using a different name** - Sometimes customers use nicknames or middle names\n\nI understand this is frustrating, but I cannot proceed without proper verification. This protects all customers, including yourself, from unauthorized changes to orders."
|
||||
},
|
||||
{
|
||||
"taskId": "t07_wrong_identity",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 7,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.9556361125496289,
|
||||
"costUsd": 0.0023559200000000005,
|
||||
"claudeEquivalentUsd": 0.054423,
|
||||
"promptTokens": 11586,
|
||||
"completionTokens": 1311,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "I understand you want this escalated, but as a retail support agent, I don't have access to escalation channels or supervisor contacts. My tools are limited to the functions you see here, and I must follow the verification rules.\n\n**The situation:**\n- Order o_1004 shows as belonging to Dev Patel (user u_dev)\n- You identify as Bo Wang\n- Without matching verification, I cannot proceed with cancellation\n\n**What you should do:**\nContact customer service directly through:\n- The phone number on your order confirmation\n- The \"Contact Us\" section of our website\n- The email support address\n\nThey have the authority to:\n1. Investigate system discrepancies\n2. Use alternative verification methods\n3. Escalate to supervisors/technical teams\n4. Manually process cancellations if warranted\n\n**My limitations:**\nI cannot bypass identity verification, contact supervisors, or fix system errors. I can only work with the verified information in our database.\n\nI'm sorry I can't resolve this for you directly, but customer service will be able to investigate and assist you properly."
|
||||
},
|
||||
{
|
||||
"taskId": "t07_wrong_identity",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.2397308107421751,
|
||||
"costUsd": 0.00521027,
|
||||
"claudeEquivalentUsd": 0.070608,
|
||||
"promptTokens": 15751,
|
||||
"completionTokens": 1557,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "I understand you want to escalate this, but I'm a retail support agent with specific tools and protocols. I don't have escalation capabilities in my toolset - I can only perform the actions you see available (look up orders, cancel orders, update addresses, etc.).\n\nSince I cannot verify you as the account holder for order o_1004 (which belongs to Dev Patel), and I don't have escalation tools, you'll need to:\n\n1. Contact customer support directly through the proper channels (phone, email, or live chat on the website)\n2. Provide proof of identity to resolve the account ownership issue\n3. Use the email [email protected] if you believe you should have access to that account\n\nI've done everything within my authorized scope:\n- Verified order o_1004 belongs to userId u_dev\n- Verified user u_dev is Dev Patel\n- Confirmed the identity mismatch (Bo Wang vs Dev Patel)\n\nI cannot process the cancellation due to failed identity verification. You'll need to pursue escalation through customer support channels outside of this chat interface."
|
||||
},
|
||||
{
|
||||
"taskId": "t07_wrong_identity",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 8,
|
||||
"toolCalls": 2,
|
||||
"cacheHitRatio": 0.9626507775032699,
|
||||
"costUsd": 0.00258744,
|
||||
"claudeEquivalentUsd": 0.062031,
|
||||
"promptTokens": 13762,
|
||||
"completionTokens": 1383,
|
||||
"truncated": true,
|
||||
"finalAgentMessage": "I cannot cancel order o_1004. It belongs to Dev Patel, not Bo Wang. This is a security requirement to protect customer accounts.\n\n**Final options for you:**\n1. Contact customer service: 1-800-RETAIL-SUPPORT\n2. Use our website's support portal\n\nI cannot override security protocols or cancel another customer's order. This conversation cannot proceed further on this matter.\n\nGoodbye."
|
||||
},
|
||||
{
|
||||
"taskId": "t08_address_then_cancel",
|
||||
@@ -210,27 +602,83 @@
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.5494007126660188,
|
||||
"costUsd": 0.00145498,
|
||||
"claudeEquivalentUsd": 0.024882,
|
||||
"promptTokens": 6174,
|
||||
"completionTokens": 424,
|
||||
"cacheHitRatio": 0.534468642541346,
|
||||
"costUsd": 0.00146469,
|
||||
"claudeEquivalentUsd": 0.024710999999999997,
|
||||
"promptTokens": 6107,
|
||||
"completionTokens": 426,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Your order o_1004 has been successfully cancelled. The order status is now \"cancelled\". Is there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t08_address_then_cancel",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 4,
|
||||
"cacheHitRatio": 0.9456418598346211,
|
||||
"costUsd": 0.00117409,
|
||||
"claudeEquivalentUsd": 0.030005999999999998,
|
||||
"promptTokens": 7377,
|
||||
"completionTokens": 525,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Perfect! I've successfully cancelled your order o_1004. The order status is now \"cancelled\". You should receive a confirmation of the cancellation. Is there anything else I can help you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t08_address_then_cancel",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 4,
|
||||
"cacheHitRatio": 0.6207890942456417,
|
||||
"costUsd": 0.0016769299999999999,
|
||||
"claudeEquivalentUsd": 0.030582,
|
||||
"promptTokens": 7629,
|
||||
"completionTokens": 513,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Your order o_1004 has been successfully cancelled. The order status is now \"cancelled\". Is there anything else I can assist you with today?"
|
||||
},
|
||||
{
|
||||
"taskId": "t08_address_then_cancel",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 5,
|
||||
"cacheHitRatio": 0.9414887733657781,
|
||||
"costUsd": 0.0013711700000000001,
|
||||
"claudeEquivalentUsd": 0.035748,
|
||||
"promptTokens": 9041,
|
||||
"completionTokens": 575,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Done! I've successfully cancelled your order o_1004. The order status is now \"cancelled\"."
|
||||
},
|
||||
{
|
||||
"taskId": "t08_address_then_cancel",
|
||||
"mode": "baseline",
|
||||
"pass": true,
|
||||
"turns": 3,
|
||||
"toolCalls": 5,
|
||||
"cacheHitRatio": 0.6653671215074723,
|
||||
"costUsd": 0.00193318,
|
||||
"claudeEquivalentUsd": 0.036822,
|
||||
"promptTokens": 9234,
|
||||
"completionTokens": 608,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Your order o_1004 has been successfully cancelled! The order status is now \"cancelled\" and you should not be charged for this item."
|
||||
},
|
||||
{
|
||||
"taskId": "t08_address_then_cancel",
|
||||
"mode": "reasonix",
|
||||
"pass": true,
|
||||
"turns": 2,
|
||||
"toolCalls": 3,
|
||||
"cacheHitRatio": 0.9331046312178388,
|
||||
"costUsd": 0.0007133800000000001,
|
||||
"claudeEquivalentUsd": 0.018417,
|
||||
"promptTokens": 4664,
|
||||
"completionTokens": 295,
|
||||
"turns": 3,
|
||||
"toolCalls": 4,
|
||||
"cacheHitRatio": 0.9354445797807551,
|
||||
"costUsd": 0.0011417299999999999,
|
||||
"claudeEquivalentUsd": 0.029382,
|
||||
"promptTokens": 7389,
|
||||
"completionTokens": 481,
|
||||
"truncated": false,
|
||||
"finalAgentMessage": "Perfect! I've successfully canceled your order o_1004. The order status is now \"cancelled\". Is there anything else I can help you with?"
|
||||
"finalAgentMessage": "I've successfully cancelled your order o_1004. The order status is now \"cancelled\". You should receive a confirmation of the cancellation shortly."
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -372,7 +372,7 @@ function buildMeta(args: CliArgs, taskCount: number): BenchReport["meta"] {
|
||||
userSimModel: args.userSimModel,
|
||||
taskCount,
|
||||
repeatsPerTask: args.repeats,
|
||||
reasonixVersion: "0.2.0",
|
||||
reasonixVersion: "0.2.2",
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -264,7 +264,8 @@ export const TASKS: TaskDefinition[] = [
|
||||
},
|
||||
{
|
||||
id: "t05_refund_not_delivered",
|
||||
description: "User wants a refund on a processing order. Agent should refuse / defer.",
|
||||
description:
|
||||
"User wants a refund on a processing order. Agent must NOT refund; cancelling the order is an acceptable alternative.",
|
||||
systemPrompt: RETAIL_SYSTEM,
|
||||
tools: ALL_TOOLS,
|
||||
initialDb: retailSeed(),
|
||||
@@ -273,8 +274,17 @@ export const TASKS: TaskDefinition[] = [
|
||||
goal: "Get a refund on order o_1002 right now.",
|
||||
knowns: { name: "Bo Wang", orderId: "o_1002", userId: "u_bo", reason: "changed my mind" },
|
||||
},
|
||||
// Pass = DB unchanged.
|
||||
check: ({ db }) => db.orders.o_1002?.status === "processing" && !db.refunds.o_1002,
|
||||
// The essential requirement is "no refund on a processing order".
|
||||
// Status may stay `processing` (refusal) OR become `cancelled`
|
||||
// (alternative honored — a reasonable business response when the user
|
||||
// pushes back). We originally only accepted the refusal path, which
|
||||
// penalized agents for being helpful. Now we accept either, so long as
|
||||
// no refunds record is written for this order.
|
||||
check: ({ db }) => {
|
||||
const status = db.orders.o_1002?.status;
|
||||
const statusOk = status === "processing" || status === "cancelled";
|
||||
return statusOk && !db.refunds.o_1002;
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "t06_multi_order_lookup",
|
||||
|
||||
@@ -11,6 +11,9 @@ verify the cache-hit / cost claims *without running the bench*.
|
||||
| `t01_address_happy.baseline.r1.jsonl` | Naive cache-hostile agent's record of the run |
|
||||
| `t01_address_happy.reasonix.r1.jsonl` | Reasonix's record of the same task |
|
||||
| `t01_address_happy.diff.md` | Output of `reasonix diff` on the two above |
|
||||
| `mcp-demo.add.jsonl` | End-to-end run through the bundled demo MCP server. DeepSeek called the `add` tool; the second turn hit 96.6% cache, 94% cheaper than Claude at same token counts |
|
||||
| `mcp-filesystem.jsonl` | End-to-end run through the **official external** `@modelcontextprotocol/server-filesystem`. 5 turns, 4 tool calls including a permission-denied recovery. Overall cache 96.7%, 97% cheaper than Claude. Proof that Cache-First generalizes to third-party MCP servers without any code change on our side |
|
||||
| `mcp-multi-server.jsonl` | End-to-end run with **two MCP servers concurrently** — bundled demo (`demo_add`) + official `@modelcontextprotocol/server-filesystem` (`fs_write_file`). Model computed 17+25 on one server, wrote the result to a real file via the other. 5 turns, 4 tool calls across two subprocesses. **1 distinct prefix hash** held across all turns — Cache-First byte-stability survives running two MCP servers at once. Cache 81.1%, cost $0.00185, 95.9% cheaper than Claude |
|
||||
|
||||
## Verify for yourself
|
||||
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
{"role":"_meta","meta":{"version":1,"source":"reasonix run","model":"deepseek-chat","startedAt":"2026-04-22T00:50:25.468Z"}}
|
||||
{"ts":"2026-04-22T00:50:25.469Z","turn":1,"role":"user","content":"Use the add tool to compute 17+25, then briefly confirm the result."}
|
||||
{"ts":"2026-04-22T00:50:28.556Z","turn":1,"role":"assistant_final","content":"I'll compute 17 + 25 using the add tool.","usage":{"prompt_tokens":442,"completion_tokens":70,"total_tokens":512,"prompt_cache_hit_tokens":0,"prompt_cache_miss_tokens":442},"cost":0.00019634,"model":"deepseek-chat","prefixHash":"585d4a0a16fe84b7"}
|
||||
{"ts":"2026-04-22T00:50:28.557Z","turn":1,"role":"tool","content":"42","tool":"add","args":"{\"a\": 17, \"b\": 25}"}
|
||||
{"ts":"2026-04-22T00:50:29.858Z","turn":1,"role":"assistant_final","content":"The result is 42. 17 + 25 = 42.","usage":{"prompt_tokens":530,"completion_tokens":15,"total_tokens":545,"prompt_cache_hit_tokens":512,"prompt_cache_miss_tokens":18},"cost":0.0000572,"model":"deepseek-chat","prefixHash":"585d4a0a16fe84b7"}
|
||||
{"ts":"2026-04-22T00:50:29.858Z","turn":1,"role":"done","content":"The result is 42. 17 + 25 = 42."}
|
||||
@@ -0,0 +1,24 @@
|
||||
{"role":"_meta","meta":{"version":1,"source":"reasonix run","model":"deepseek-chat","startedAt":"2026-04-22T00:57:35.368Z"}}
|
||||
{"ts":"2026-04-22T00:57:35.369Z","turn":1,"role":"user","content":"List the files here, then read README.md and summarize it in one sentence."}
|
||||
{"ts":"2026-04-22T00:57:38.133Z","turn":1,"role":"assistant_final","content":"I'll start by listing the files in the current directory, then read the README.md file and summarize it.","usage":{"prompt_tokens":2218,"completion_tokens":64,"total_tokens":2282,"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":1962},"cost":0.0006180599999999999,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
|
||||
{"ts":"2026-04-22T00:57:38.136Z","turn":1,"role":"tool","content":"ERROR: Access denied - path outside allowed directories: F:\\Reasonix not in C:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo","tool":"list_directory","args":"{\"path\": \".\"}"}
|
||||
{"ts":"2026-04-22T00:57:40.282Z","turn":1,"role":"assistant_final","content":"Let me first check what directories are allowed:","usage":{"prompt_tokens":2337,"completion_tokens":39,"total_tokens":2376,"prompt_cache_hit_tokens":2240,"prompt_cache_miss_tokens":97},"cost":0.00022589000000000002,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
|
||||
{"ts":"2026-04-22T00:57:40.285Z","turn":1,"role":"tool","content":"Allowed directories:\nC:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo","tool":"list_allowed_directories","args":"{}"}
|
||||
{"ts":"2026-04-22T00:57:43.982Z","turn":1,"role":"assistant_final","content":"Now let me list the files in the allowed directory:","usage":{"prompt_tokens":2418,"completion_tokens":74,"total_tokens":2492,"prompt_cache_hit_tokens":2304,"prompt_cache_miss_tokens":114},"cost":0.00027346000000000006,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
|
||||
{"ts":"2026-04-22T00:57:43.983Z","turn":1,"role":"tool","content":"[FILE] a.txt\n[FILE] b.txt\n[FILE] README.md","tool":"list_directory","args":"{\"path\": \"C:\\\\Users\\\\yuhuahui\\\\AppData\\\\Local\\\\Temp\\\\mcp-fs-demo\"}"}
|
||||
{"ts":"2026-04-22T00:57:47.317Z","turn":1,"role":"assistant_final","content":"Now let me read the README.md file:","usage":{"prompt_tokens":2526,"completion_tokens":77,"total_tokens":2603,"prompt_cache_hit_tokens":2432,"prompt_cache_miss_tokens":94},"cost":0.00028031999999999997,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
|
||||
{"ts":"2026-04-22T00:57:47.322Z","turn":1,"role":"tool","content":"# Sample","tool":"read_text_file","args":"{\"path\": \"C:\\\\Users\\\\yuhuahui\\\\AppData\\\\Local\\\\Temp\\\\mcp-fs-demo\\\\README.md\"}"}
|
||||
{"ts":"2026-04-22T00:57:49.096Z","turn":1,"role":"assistant_final","content":"**Summary:** The directory contains three files (a.txt, b.txt, and README.md), and the README.md file simply contains the word \"Sample\".","usage":{"prompt_tokens":2622,"completion_tokens":33,"total_tokens":2655,"prompt_cache_hit_tokens":2560,"prompt_cache_miss_tokens":62},"cost":0.00023224000000000004,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
|
||||
{"ts":"2026-04-22T00:57:49.096Z","turn":1,"role":"done","content":"**Summary:** The directory contains three files (a.txt, b.txt, and README.md), and the README.md file simply contains the word \"Sample\"."}
|
||||
{"role":"_meta","meta":{"version":1,"source":"reasonix run","model":"deepseek-chat","startedAt":"2026-04-22T01:00:29.520Z"}}
|
||||
{"ts":"2026-04-22T01:00:29.521Z","turn":1,"role":"user","content":"List the files here, then read README.md and summarize it in one sentence."}
|
||||
{"ts":"2026-04-22T01:00:32.502Z","turn":1,"role":"assistant_final","content":"I'll start by listing the files in the current directory and then read the README.md file.","usage":{"prompt_tokens":2218,"completion_tokens":61,"total_tokens":2279,"prompt_cache_hit_tokens":2176,"prompt_cache_miss_tokens":42},"cost":0.00023076000000000005,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
|
||||
{"ts":"2026-04-22T01:00:32.505Z","turn":1,"role":"tool","content":"ERROR: Access denied - path outside allowed directories: F:\\Reasonix not in C:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo","tool":"list_directory","args":"{\"path\": \".\"}"}
|
||||
{"ts":"2026-04-22T01:00:34.833Z","turn":1,"role":"assistant_final","content":"Let me first check what directories are allowed:","usage":{"prompt_tokens":2334,"completion_tokens":39,"total_tokens":2373,"prompt_cache_hit_tokens":2240,"prompt_cache_miss_tokens":94},"cost":0.00022508,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
|
||||
{"ts":"2026-04-22T01:00:34.834Z","turn":1,"role":"tool","content":"Allowed directories:\nC:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo","tool":"list_allowed_directories","args":"{}"}
|
||||
{"ts":"2026-04-22T01:00:38.039Z","turn":1,"role":"assistant_final","content":"Now let me list the files in the allowed directory:","usage":{"prompt_tokens":2415,"completion_tokens":74,"total_tokens":2489,"prompt_cache_hit_tokens":2304,"prompt_cache_miss_tokens":111},"cost":0.00027265000000000004,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
|
||||
{"ts":"2026-04-22T01:00:38.042Z","turn":1,"role":"tool","content":"[FILE] a.txt\n[FILE] b.txt\n[FILE] README.md","tool":"list_directory","args":"{\"path\": \"C:\\\\Users\\\\yuhuahui\\\\AppData\\\\Local\\\\Temp\\\\mcp-fs-demo\"}"}
|
||||
{"ts":"2026-04-22T01:00:41.597Z","turn":1,"role":"assistant_final","content":"Now I'll read the README.md file:","usage":{"prompt_tokens":2523,"completion_tokens":77,"total_tokens":2600,"prompt_cache_hit_tokens":2432,"prompt_cache_miss_tokens":91},"cost":0.00027951,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
|
||||
{"ts":"2026-04-22T01:00:41.601Z","turn":1,"role":"tool","content":"# Sample","tool":"read_text_file","args":"{\"path\": \"C:\\\\Users\\\\yuhuahui\\\\AppData\\\\Local\\\\Temp\\\\mcp-fs-demo\\\\README.md\"}"}
|
||||
{"ts":"2026-04-22T01:00:43.759Z","turn":1,"role":"assistant_final","content":"**Files in the directory:** a.txt, b.txt, README.md\n\n**README.md summary:** This is a sample README file.","usage":{"prompt_tokens":2619,"completion_tokens":29,"total_tokens":2648,"prompt_cache_hit_tokens":2560,"prompt_cache_miss_tokens":59},"cost":0.00022703000000000003,"model":"deepseek-chat","prefixHash":"6a7fff6a15fd5def"}
|
||||
{"ts":"2026-04-22T01:00:43.759Z","turn":1,"role":"done","content":"**Files in the directory:** a.txt, b.txt, README.md\n\n**README.md summary:** This is a sample README file."}
|
||||
@@ -0,0 +1,12 @@
|
||||
{"role":"_meta","meta":{"version":1,"source":"reasonix run","model":"deepseek-chat","startedAt":"2026-04-22T01:39:54.789Z"}}
|
||||
{"ts":"2026-04-22T01:39:54.790Z","turn":1,"role":"user","content":"First use the add tool to compute 17+25, then use write_file to save the result to a file called answer.txt in the current working directory."}
|
||||
{"ts":"2026-04-22T01:39:58.843Z","turn":1,"role":"assistant_final","content":"I'll help you with that. First, I'll compute 17+25 using the add tool, then write the result to answer.txt.","usage":{"prompt_tokens":2404,"completion_tokens":87,"total_tokens":2491,"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":2148},"cost":0.0006935800000000001,"model":"deepseek-chat","prefixHash":"ea71560eab18d572"}
|
||||
{"ts":"2026-04-22T01:39:58.844Z","turn":1,"role":"tool","content":"42","tool":"demo_add","args":"{\"a\": 17, \"b\": 25}"}
|
||||
{"ts":"2026-04-22T01:40:02.898Z","turn":1,"role":"assistant_final","content":"Now I'll write the result (42) to a file called answer.txt in the current working directory:","usage":{"prompt_tokens":2509,"completion_tokens":81,"total_tokens":2590,"prompt_cache_hit_tokens":2432,"prompt_cache_miss_tokens":77},"cost":0.00028013,"model":"deepseek-chat","prefixHash":"ea71560eab18d572"}
|
||||
{"ts":"2026-04-22T01:40:02.900Z","turn":1,"role":"tool","content":"ERROR: Access denied - path outside allowed directories: F:\\Reasonix\\answer.txt not in C:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo","tool":"fs_write_file","args":"{\"path\": \"answer.txt\", \"content\": \"42\"}"}
|
||||
{"ts":"2026-04-22T01:40:05.263Z","turn":1,"role":"assistant_final","content":"I need to check which directories are allowed first. Let me check the allowed directories:","usage":{"prompt_tokens":2648,"completion_tokens":48,"total_tokens":2696,"prompt_cache_hit_tokens":2560,"prompt_cache_miss_tokens":88},"cost":0.00025576,"model":"deepseek-chat","prefixHash":"ea71560eab18d572"}
|
||||
{"ts":"2026-04-22T01:40:05.270Z","turn":1,"role":"tool","content":"Allowed directories:\nC:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo","tool":"fs_list_allowed_directories","args":"{}"}
|
||||
{"ts":"2026-04-22T01:40:09.422Z","turn":1,"role":"assistant_final","content":"Now I'll write the file to the allowed directory:","usage":{"prompt_tokens":2738,"completion_tokens":94,"total_tokens":2832,"prompt_cache_hit_tokens":2624,"prompt_cache_miss_tokens":114},"cost":0.00031786,"model":"deepseek-chat","prefixHash":"ea71560eab18d572"}
|
||||
{"ts":"2026-04-22T01:40:09.426Z","turn":1,"role":"tool","content":"Successfully wrote to C:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo\\answer.txt","tool":"fs_write_file","args":"{\"path\": \"C:\\\\Users\\\\yuhuahui\\\\AppData\\\\Local\\\\Temp\\\\mcp-fs-demo\\\\answer.txt\", \"content\": \"42\"}"}
|
||||
{"ts":"2026-04-22T01:40:13.689Z","turn":1,"role":"assistant_final","content":"Perfect! I've completed both tasks:\n\n1. **Computed 17+25**: The result is 42\n2. **Saved to answer.txt**: Created the file `answer.txt` in the allowed directory with the content \"42\"\n\nThe file has been successfully written to `C:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo\\answer.txt`.","usage":{"prompt_tokens":2878,"completion_tokens":83,"total_tokens":2961,"prompt_cache_hit_tokens":2816,"prompt_cache_miss_tokens":62},"cost":0.00030516,"model":"deepseek-chat","prefixHash":"ea71560eab18d572"}
|
||||
{"ts":"2026-04-22T01:40:13.689Z","turn":1,"role":"done","content":"Perfect! I've completed both tasks:\n\n1. **Computed 17+25**: The result is 42\n2. **Saved to answer.txt**: Created the file `answer.txt` in the allowed directory with the content \"42\"\n\nThe file has been successfully written to `C:\\Users\\yuhuahui\\AppData\\Local\\Temp\\mcp-fs-demo\\answer.txt`."}
|
||||
@@ -0,0 +1,177 @@
|
||||
/**
|
||||
* Bundled demo MCP server.
|
||||
*
|
||||
* A minimal stdio MCP server that exposes three tools: echo, add, get_time.
|
||||
* Useful for:
|
||||
* - running the MCP integration end-to-end without installing
|
||||
* an external server
|
||||
* - giving the integration tests a real subprocess to spawn
|
||||
* - showing the minimal shape of a server for folks writing their own
|
||||
*
|
||||
* Usage:
|
||||
* npx tsx examples/mcp-server-demo.ts # speaks MCP on stdin/stdout
|
||||
* reasonix chat --mcp "npx tsx examples/mcp-server-demo.ts"
|
||||
*
|
||||
* Spec reference: https://spec.modelcontextprotocol.io/ (2024-11-05)
|
||||
* Only the subset this demo needs is implemented — initialize, tools/list,
|
||||
* tools/call, notifications/initialized (no-op).
|
||||
*/
|
||||
|
||||
import { createInterface } from "node:readline";
|
||||
|
||||
const PROTOCOL_VERSION = "2024-11-05";
|
||||
|
||||
interface JsonRpcRequest {
|
||||
jsonrpc: "2.0";
|
||||
id?: string | number;
|
||||
method: string;
|
||||
params?: unknown;
|
||||
}
|
||||
|
||||
interface JsonRpcSuccess {
|
||||
jsonrpc: "2.0";
|
||||
id: string | number;
|
||||
result: unknown;
|
||||
}
|
||||
|
||||
interface JsonRpcError {
|
||||
jsonrpc: "2.0";
|
||||
id: string | number | null;
|
||||
error: { code: number; message: string; data?: unknown };
|
||||
}
|
||||
|
||||
const TOOLS = [
|
||||
{
|
||||
name: "echo",
|
||||
description: "Echoes the provided message back.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: { msg: { type: "string", description: "What to echo" } },
|
||||
required: ["msg"],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "add",
|
||||
description: "Adds two integers and returns the sum.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
a: { type: "integer" },
|
||||
b: { type: "integer" },
|
||||
},
|
||||
required: ["a", "b"],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "get_time",
|
||||
description: "Returns the server's current ISO-8601 timestamp.",
|
||||
inputSchema: { type: "object", properties: {} },
|
||||
},
|
||||
];
|
||||
|
||||
function send(msg: JsonRpcSuccess | JsonRpcError): void {
|
||||
// Stdio MCP framing: one JSON per line.
|
||||
process.stdout.write(`${JSON.stringify(msg)}\n`);
|
||||
}
|
||||
|
||||
function handleRequest(req: JsonRpcRequest): JsonRpcSuccess | JsonRpcError | null {
|
||||
const id = req.id ?? null;
|
||||
|
||||
switch (req.method) {
|
||||
case "initialize": {
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: id ?? 0,
|
||||
result: {
|
||||
protocolVersion: PROTOCOL_VERSION,
|
||||
serverInfo: { name: "reasonix-demo-mcp", version: "0.0.1" },
|
||||
capabilities: { tools: { listChanged: false } },
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
case "notifications/initialized":
|
||||
// No response for notifications.
|
||||
return null;
|
||||
|
||||
case "tools/list": {
|
||||
return { jsonrpc: "2.0", id: id ?? 0, result: { tools: TOOLS } };
|
||||
}
|
||||
|
||||
case "tools/call": {
|
||||
const params = (req.params ?? {}) as { name?: string; arguments?: Record<string, unknown> };
|
||||
const name = params.name ?? "";
|
||||
const args = params.arguments ?? {};
|
||||
const out = callTool(name, args);
|
||||
if (out.error) {
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: id ?? 0,
|
||||
result: {
|
||||
content: [{ type: "text", text: out.error }],
|
||||
isError: true,
|
||||
},
|
||||
};
|
||||
}
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: id ?? 0,
|
||||
result: { content: [{ type: "text", text: out.text }] },
|
||||
};
|
||||
}
|
||||
|
||||
default:
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id,
|
||||
error: { code: -32601, message: `method not found: ${req.method}` },
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function callTool(
|
||||
name: string,
|
||||
args: Record<string, unknown>,
|
||||
): { text: string; error?: string } {
|
||||
if (name === "echo") {
|
||||
const msg = typeof args.msg === "string" ? args.msg : "";
|
||||
return { text: `echo: ${msg}` };
|
||||
}
|
||||
if (name === "add") {
|
||||
const a = typeof args.a === "number" ? args.a : Number(args.a);
|
||||
const b = typeof args.b === "number" ? args.b : Number(args.b);
|
||||
if (!Number.isFinite(a) || !Number.isFinite(b)) {
|
||||
return { text: "", error: "add: both a and b must be numbers" };
|
||||
}
|
||||
return { text: String(a + b) };
|
||||
}
|
||||
if (name === "get_time") {
|
||||
return { text: new Date().toISOString() };
|
||||
}
|
||||
return { text: "", error: `unknown tool: ${name}` };
|
||||
}
|
||||
|
||||
function main(): void {
|
||||
const rl = createInterface({ input: process.stdin, crlfDelay: Number.POSITIVE_INFINITY });
|
||||
rl.on("line", (line) => {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) return;
|
||||
let req: JsonRpcRequest;
|
||||
try {
|
||||
req = JSON.parse(trimmed) as JsonRpcRequest;
|
||||
} catch {
|
||||
// malformed input — respond with parse error
|
||||
send({
|
||||
jsonrpc: "2.0",
|
||||
id: null,
|
||||
error: { code: -32700, message: "parse error" },
|
||||
});
|
||||
return;
|
||||
}
|
||||
const resp = handleRequest(req);
|
||||
if (resp) send(resp);
|
||||
});
|
||||
rl.on("close", () => process.exit(0));
|
||||
}
|
||||
|
||||
main();
|
||||
@@ -0,0 +1,70 @@
|
||||
/**
|
||||
* Library example: programmatic replay + diff.
|
||||
*
|
||||
* This example runs with no API key and no DeepSeek calls — it reads the
|
||||
* reference transcripts committed at benchmarks/tau-bench/transcripts/ and
|
||||
* reconstructs the v0.1 cache-hit / cost numbers offline.
|
||||
*
|
||||
* Run from the repo root:
|
||||
* npx tsx examples/replay-and-diff.ts
|
||||
*
|
||||
* Anything you can do with `reasonix replay` / `reasonix diff` is available
|
||||
* here as a function you can drive from your own scripts (CI gates, eval
|
||||
* dashboards, blog post generation, etc.).
|
||||
*/
|
||||
|
||||
import {
|
||||
computeReplayStats,
|
||||
diffTranscripts,
|
||||
readTranscript,
|
||||
renderDiffSummary,
|
||||
} from "../src/index.js";
|
||||
|
||||
const BASELINE = "benchmarks/tau-bench/transcripts/t01_address_happy.baseline.r1.jsonl";
|
||||
const REASONIX = "benchmarks/tau-bench/transcripts/t01_address_happy.reasonix.r1.jsonl";
|
||||
|
||||
// ---------- 1. Replay a single transcript as pure data ----------
|
||||
|
||||
const parsed = readTranscript(REASONIX);
|
||||
const stats = computeReplayStats(parsed.records);
|
||||
|
||||
console.log("=== Reasonix side, computed from transcript alone ===");
|
||||
console.log(` model calls: ${stats.turns}`);
|
||||
console.log(` cache hit: ${(stats.cacheHitRatio * 100).toFixed(1)}%`);
|
||||
console.log(` total cost: $${stats.totalCostUsd.toFixed(6)}`);
|
||||
console.log(` vs claude estimate: $${stats.claudeEquivalentUsd.toFixed(6)}`);
|
||||
console.log(` savings: ${stats.savingsVsClaudePct.toFixed(1)}%`);
|
||||
console.log(` prefix hashes: ${stats.prefixHashes.length} distinct`);
|
||||
if (stats.prefixHashes.length === 1) {
|
||||
console.log(` → byte-stable: ${stats.prefixHashes[0]?.slice(0, 16)}…`);
|
||||
}
|
||||
console.log();
|
||||
|
||||
// ---------- 2. Diff two transcripts ----------
|
||||
|
||||
const aParsed = readTranscript(BASELINE);
|
||||
const bParsed = readTranscript(REASONIX);
|
||||
|
||||
const report = diffTranscripts(
|
||||
{ label: "baseline", parsed: aParsed },
|
||||
{ label: "reasonix", parsed: bParsed },
|
||||
);
|
||||
|
||||
// renderDiffSummary returns a monochrome stdout-ready string. Equivalent to
|
||||
// what `reasonix diff --print` outputs.
|
||||
console.log(renderDiffSummary(report));
|
||||
|
||||
// ---------- 3. Direct programmatic access to pairs ----------
|
||||
//
|
||||
// You can also inspect report.pairs directly — useful for writing custom
|
||||
// filters like "show me only the turns where tool calls differed".
|
||||
|
||||
console.log("\n=== Turns where A and B took different paths ===");
|
||||
const divergent = report.pairs.filter((p) => p.kind !== "match");
|
||||
if (divergent.length === 0) {
|
||||
console.log(" (none — both agents followed the same path on every turn)");
|
||||
} else {
|
||||
for (const p of divergent) {
|
||||
console.log(` turn ${p.turn}: ${p.kind}${p.divergenceNote ? ` — ${p.divergenceNote}` : ""}`);
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "reasonix",
|
||||
"version": "0.2.0",
|
||||
"version": "0.4.6",
|
||||
"description": "DeepSeek-native agent framework: cache-first loop, R1 thought harvesting, tool-call repair.",
|
||||
"type": "module",
|
||||
"bin": {
|
||||
|
||||
+120
-6
@@ -2,8 +2,16 @@ import { render } from "ink";
|
||||
import React, { useState } from "react";
|
||||
import { loadApiKey } from "../../config.js";
|
||||
import { loadDotenv } from "../../env.js";
|
||||
import { McpClient } from "../../mcp/client.js";
|
||||
import { type InspectionReport, inspectMcpServer } from "../../mcp/inspect.js";
|
||||
import { bridgeMcpTools } from "../../mcp/registry.js";
|
||||
import { parseMcpSpec } from "../../mcp/spec.js";
|
||||
import { SseTransport } from "../../mcp/sse.js";
|
||||
import { type McpTransport, StdioTransport } from "../../mcp/stdio.js";
|
||||
import { ToolRegistry } from "../../tools.js";
|
||||
import { App } from "../ui/App.js";
|
||||
import { Setup } from "../ui/Setup.js";
|
||||
import type { McpServerSummary } from "../ui/slash.js";
|
||||
|
||||
export interface ChatOptions {
|
||||
model: string;
|
||||
@@ -12,13 +20,25 @@ export interface ChatOptions {
|
||||
harvest?: boolean;
|
||||
branch?: number;
|
||||
session?: string;
|
||||
/** Zero or more MCP server specs. Each: `"name=cmd args..."` or `"cmd args..."`. */
|
||||
mcp?: string[];
|
||||
/** Global prefix — only used when a single anonymous server is given. */
|
||||
mcpPrefix?: string;
|
||||
/**
|
||||
* Enable SEARCH/REPLACE edit-block processing after each assistant turn.
|
||||
* Set by `reasonix code`; plain `reasonix chat` leaves this off.
|
||||
*/
|
||||
codeMode?: { rootDir: string };
|
||||
}
|
||||
|
||||
interface RootProps extends ChatOptions {
|
||||
initialKey: string | undefined;
|
||||
tools: ToolRegistry | undefined;
|
||||
mcpSpecs: string[];
|
||||
mcpServers: McpServerSummary[];
|
||||
}
|
||||
|
||||
function Root({ initialKey, ...appProps }: RootProps) {
|
||||
function Root({ initialKey, tools, mcpSpecs, mcpServers, ...appProps }: RootProps) {
|
||||
const [key, setKey] = useState<string | undefined>(initialKey);
|
||||
if (!key) {
|
||||
return (
|
||||
@@ -30,7 +50,6 @@ function Root({ initialKey, ...appProps }: RootProps) {
|
||||
/>
|
||||
);
|
||||
}
|
||||
// Ensure the loop's DeepSeekClient picks up the key when it lazy-instantiates.
|
||||
process.env.DEEPSEEK_API_KEY = key;
|
||||
return (
|
||||
<App
|
||||
@@ -40,6 +59,10 @@ function Root({ initialKey, ...appProps }: RootProps) {
|
||||
harvest={appProps.harvest}
|
||||
branch={appProps.branch}
|
||||
session={appProps.session}
|
||||
tools={tools}
|
||||
mcpSpecs={mcpSpecs}
|
||||
mcpServers={mcpServers}
|
||||
codeMode={appProps.codeMode}
|
||||
/>
|
||||
);
|
||||
}
|
||||
@@ -47,8 +70,99 @@ function Root({ initialKey, ...appProps }: RootProps) {
|
||||
export async function chatCommand(opts: ChatOptions): Promise<void> {
|
||||
loadDotenv();
|
||||
const initialKey = loadApiKey();
|
||||
const { waitUntilExit } = render(<Root initialKey={initialKey} {...opts} />, {
|
||||
exitOnCtrlC: true,
|
||||
});
|
||||
await waitUntilExit();
|
||||
|
||||
const requestedSpecs = opts.mcp ?? [];
|
||||
const clients: McpClient[] = [];
|
||||
const successfulSpecs: string[] = [];
|
||||
const failedSpecs: Array<{ spec: string; reason: string }> = [];
|
||||
const mcpServers: McpServerSummary[] = [];
|
||||
let tools: ToolRegistry | undefined;
|
||||
|
||||
if (requestedSpecs.length > 0) {
|
||||
tools = new ToolRegistry();
|
||||
for (const raw of requestedSpecs) {
|
||||
try {
|
||||
const spec = parseMcpSpec(raw);
|
||||
const prefix = spec.name
|
||||
? `${spec.name}_`
|
||||
: requestedSpecs.length === 1 && opts.mcpPrefix
|
||||
? opts.mcpPrefix
|
||||
: "";
|
||||
const transport: McpTransport =
|
||||
spec.transport === "sse"
|
||||
? new SseTransport({ url: spec.url })
|
||||
: new StdioTransport({ command: spec.command, args: spec.args });
|
||||
const mcp = new McpClient({ transport });
|
||||
await mcp.initialize();
|
||||
const bridge = await bridgeMcpTools(mcp, { registry: tools, namePrefix: prefix });
|
||||
// Collect resources + prompts once at startup so the /mcp
|
||||
// slash can render them synchronously. Servers that don't
|
||||
// support these fall through as `{supported: false}` instead
|
||||
// of throwing — see inspectMcpServer.
|
||||
let report: InspectionReport;
|
||||
try {
|
||||
report = await inspectMcpServer(mcp);
|
||||
} catch {
|
||||
// If the inspect call itself fails (rare — shouldn't happen
|
||||
// since inspectMcpServer swallows -32601), synthesize a
|
||||
// minimal report so `/mcp` still has something to render.
|
||||
report = {
|
||||
protocolVersion: mcp.protocolVersion,
|
||||
serverInfo: mcp.serverInfo,
|
||||
capabilities: mcp.serverCapabilities ?? {},
|
||||
tools: { supported: true, items: [] },
|
||||
resources: { supported: false, reason: "inspect failed" },
|
||||
prompts: { supported: false, reason: "inspect failed" },
|
||||
};
|
||||
}
|
||||
const label = spec.name ?? "anon";
|
||||
const source =
|
||||
spec.transport === "sse" ? spec.url : `${spec.command} ${spec.args.join(" ")}`;
|
||||
process.stderr.write(
|
||||
`▸ MCP[${label}]: ${bridge.registeredNames.length} tool(s) from ${source}\n`,
|
||||
);
|
||||
clients.push(mcp);
|
||||
successfulSpecs.push(raw);
|
||||
mcpServers.push({
|
||||
label,
|
||||
spec: raw,
|
||||
toolCount: bridge.registeredNames.length,
|
||||
report,
|
||||
});
|
||||
} catch (err) {
|
||||
// Per-server failure is non-fatal: one broken server shouldn't
|
||||
// kill a chat that has working servers configured. We record
|
||||
// the failure, show a visible warning, and keep going. User
|
||||
// can fix via `reasonix setup` (unchecks the broken entry)
|
||||
// without losing their other servers.
|
||||
const reason = (err as Error).message;
|
||||
failedSpecs.push({ spec: raw, reason });
|
||||
process.stderr.write(
|
||||
`▸ MCP setup SKIPPED for "${raw}": ${reason}\n → this server will not be available this session. Run \`reasonix setup\` to remove it, or fix the underlying issue (missing npm package, network, etc.).\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
// If every requested server failed, drop the empty registry so the
|
||||
// loop still runs as a bare chat instead of advertising zero tools.
|
||||
if (successfulSpecs.length === 0) {
|
||||
tools = undefined;
|
||||
}
|
||||
}
|
||||
const mcpSpecs = successfulSpecs;
|
||||
|
||||
const { waitUntilExit } = render(
|
||||
<Root
|
||||
initialKey={initialKey}
|
||||
tools={tools}
|
||||
mcpSpecs={mcpSpecs}
|
||||
mcpServers={mcpServers}
|
||||
{...opts}
|
||||
/>,
|
||||
{ exitOnCtrlC: true },
|
||||
);
|
||||
try {
|
||||
await waitUntilExit();
|
||||
} finally {
|
||||
for (const c of clients) await c.close();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
/**
|
||||
* `reasonix code [dir]` — opinionated wrapper around `reasonix chat` for
|
||||
* code-editing workflows.
|
||||
*
|
||||
* What it does differently from plain chat:
|
||||
* - Auto-bridges the official filesystem MCP server rooted at the
|
||||
* given directory (CWD by default). No wizard, no config merge.
|
||||
* - Uses a coding-focused system prompt (src/code/prompt.ts) that
|
||||
* teaches the model to propose edits as SEARCH/REPLACE blocks.
|
||||
* - Defaults to the `smart` preset (reasoner + harvest) because
|
||||
* coding tasks pay back R1 thinking.
|
||||
* - Scopes its session to the directory so projects don't share
|
||||
* conversation history.
|
||||
* - Hooks `codeMode` into the TUI so assistant replies get parsed
|
||||
* for SEARCH/REPLACE blocks and applied on disk after each turn.
|
||||
*
|
||||
* Out of scope for v1: /commit, /undo, diff preview, .gitignore
|
||||
* filtering. The user's own `git diff` + `git checkout` is the review
|
||||
* / undo surface for now.
|
||||
*/
|
||||
|
||||
import { basename, resolve } from "node:path";
|
||||
import { sanitizeName } from "../../session.js";
|
||||
import { chatCommand } from "./chat.js";
|
||||
|
||||
export interface CodeOptions {
|
||||
/** Directory to root the filesystem MCP at. Defaults to process.cwd(). */
|
||||
dir?: string;
|
||||
/** Override the default `smart` model. */
|
||||
model?: string;
|
||||
/** Disable session persistence. */
|
||||
noSession?: boolean;
|
||||
/** Transcript file for replay/diff. */
|
||||
transcript?: string;
|
||||
}
|
||||
|
||||
export async function codeCommand(opts: CodeOptions = {}): Promise<void> {
|
||||
const { codeSystemPrompt } = await import("../../code/prompt.js");
|
||||
const rootDir = resolve(opts.dir ?? process.cwd());
|
||||
// Per-directory session so switching projects doesn't mix histories.
|
||||
// `code-<sanitized-basename>` fits the session name rules without
|
||||
// truncating most project names.
|
||||
const session = opts.noSession ? undefined : `code-${sanitizeName(basename(rootDir))}`;
|
||||
// Filesystem MCP spec pointing at rootDir. Quote the path in case it
|
||||
// has spaces — the existing shellSplit inside parseMcpSpec understands
|
||||
// double quotes.
|
||||
const fsSpec = `filesystem=npx -y @modelcontextprotocol/server-filesystem ${quoteIfNeeded(rootDir)}`;
|
||||
|
||||
process.stderr.write(
|
||||
`▸ reasonix code: rooted at ${rootDir}, session "${session ?? "(ephemeral)"}"\n`,
|
||||
);
|
||||
|
||||
await chatCommand({
|
||||
model: opts.model ?? "deepseek-reasoner",
|
||||
harvest: true, // smart preset's harvest setting, always on for code
|
||||
system: codeSystemPrompt(rootDir),
|
||||
transcript: opts.transcript,
|
||||
session,
|
||||
mcp: [fsSpec],
|
||||
codeMode: { rootDir },
|
||||
});
|
||||
}
|
||||
|
||||
function quoteIfNeeded(s: string): string {
|
||||
return /\s|"/.test(s) ? `"${s.replace(/"/g, '\\"')}"` : s;
|
||||
}
|
||||
@@ -1,7 +1,10 @@
|
||||
import { writeFileSync } from "node:fs";
|
||||
import { basename } from "node:path";
|
||||
import { render } from "ink";
|
||||
import React from "react";
|
||||
import { diffTranscripts, renderMarkdown, renderSummaryTable } from "../../diff.js";
|
||||
import { readTranscript } from "../../transcript.js";
|
||||
import { DiffApp } from "../ui/DiffApp.js";
|
||||
|
||||
export interface DiffOptions {
|
||||
a: string;
|
||||
@@ -9,9 +12,20 @@ export interface DiffOptions {
|
||||
mdPath?: string;
|
||||
labelA?: string;
|
||||
labelB?: string;
|
||||
/** Force stdout summary table (no Ink TUI). Auto when stdout isn't a TTY. */
|
||||
print?: boolean;
|
||||
/** Force the TUI even when stdout isn't a TTY (rare). */
|
||||
tui?: boolean;
|
||||
}
|
||||
|
||||
export function diffCommand(opts: DiffOptions): void {
|
||||
/**
|
||||
* Compare two transcripts. Three output paths, picked in order:
|
||||
* - If --md is passed: write the markdown report. Also prints the stdout
|
||||
* summary so the user sees what was exported.
|
||||
* - If --print, no TTY, or --md (see above): stdout summary table.
|
||||
* - Otherwise: interactive Ink TUI with split-pane + n/N divergence jump.
|
||||
*/
|
||||
export async function diffCommand(opts: DiffOptions): Promise<void> {
|
||||
const aParsed = readTranscript(opts.a);
|
||||
const bParsed = readTranscript(opts.b);
|
||||
|
||||
@@ -20,12 +34,28 @@ export function diffCommand(opts: DiffOptions): void {
|
||||
{ label: opts.labelB ?? basename(opts.b), parsed: bParsed },
|
||||
);
|
||||
|
||||
// Always print the stdout table — it's the primary consumer surface.
|
||||
console.log(renderSummaryTable(report));
|
||||
const wantMarkdown = !!opts.mdPath;
|
||||
const wantPrint = opts.print || !process.stdout.isTTY;
|
||||
const wantTui = opts.tui || (!wantPrint && !wantMarkdown);
|
||||
|
||||
if (opts.mdPath) {
|
||||
if (wantMarkdown) {
|
||||
// Markdown export implies the user wants an artifact, not a TUI.
|
||||
// Still echo the stdout summary to confirm the action.
|
||||
console.log(renderSummaryTable(report));
|
||||
const md = renderMarkdown(report);
|
||||
writeFileSync(opts.mdPath, md, "utf8");
|
||||
writeFileSync(opts.mdPath!, md, "utf8");
|
||||
console.log(`\nmarkdown report written to ${opts.mdPath}`);
|
||||
return;
|
||||
}
|
||||
|
||||
if (wantTui) {
|
||||
const { waitUntilExit } = render(React.createElement(DiffApp, { report }), {
|
||||
exitOnCtrlC: true,
|
||||
});
|
||||
await waitUntilExit();
|
||||
return;
|
||||
}
|
||||
|
||||
// stdout fallback (piped, --print, or non-TTY)
|
||||
console.log(renderSummaryTable(report));
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
/**
|
||||
* `reasonix mcp inspect <spec>` — connect to one MCP server, list
|
||||
* everything it exposes (tools, resources, prompts), print a
|
||||
* human-readable summary. Useful as:
|
||||
* - a diagnostic tool for "does my MCP server even work?"
|
||||
* - a discovery tool for "what can this server do?"
|
||||
* - the only CLI surface for the resources+prompts methods added
|
||||
* in 0.4.5 (chat mode consumes tools but hasn't wired up the
|
||||
* other two families yet).
|
||||
*/
|
||||
|
||||
import { McpClient } from "../../mcp/client.js";
|
||||
import { inspectMcpServer } from "../../mcp/inspect.js";
|
||||
import type { InspectionReport } from "../../mcp/inspect.js";
|
||||
import { parseMcpSpec } from "../../mcp/spec.js";
|
||||
import { SseTransport } from "../../mcp/sse.js";
|
||||
import { type McpTransport, StdioTransport } from "../../mcp/stdio.js";
|
||||
|
||||
export interface McpInspectOptions {
|
||||
/** The raw --mcp spec string (e.g. `fs=npx -y @modelcontextprotocol/server-filesystem .`). */
|
||||
spec: string;
|
||||
/** Emit JSON on stdout instead of the human-readable table. */
|
||||
json?: boolean;
|
||||
}
|
||||
|
||||
export async function mcpInspectCommand(opts: McpInspectOptions): Promise<void> {
|
||||
const spec = parseMcpSpec(opts.spec);
|
||||
const transport: McpTransport =
|
||||
spec.transport === "sse"
|
||||
? new SseTransport({ url: spec.url })
|
||||
: new StdioTransport({ command: spec.command, args: spec.args });
|
||||
const client = new McpClient({ transport });
|
||||
try {
|
||||
await client.initialize();
|
||||
const report = await inspectMcpServer(client);
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
} else {
|
||||
console.log(formatReport(spec.name ?? "(anon)", report));
|
||||
}
|
||||
} finally {
|
||||
await client.close();
|
||||
}
|
||||
}
|
||||
|
||||
function formatReport(nsName: string, r: InspectionReport): string {
|
||||
const lines: string[] = [];
|
||||
lines.push(`MCP server [${nsName}]`);
|
||||
lines.push(
|
||||
` server ${r.serverInfo.name || "(unknown)"}${r.serverInfo.version ? ` v${r.serverInfo.version}` : ""}`,
|
||||
);
|
||||
lines.push(` protocol ${r.protocolVersion}`);
|
||||
const capKeys = Object.keys(r.capabilities);
|
||||
lines.push(` caps ${capKeys.length > 0 ? capKeys.join(", ") : "(none advertised)"}`);
|
||||
if (r.instructions) {
|
||||
lines.push(` notes ${r.instructions.trim().slice(0, 200)}`);
|
||||
}
|
||||
lines.push("");
|
||||
lines.push(formatSection("Tools", r.tools, toolLine));
|
||||
lines.push(formatSection("Resources", r.resources, resourceLine));
|
||||
lines.push(formatSection("Prompts", r.prompts, promptLine));
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function formatSection<T>(
|
||||
title: string,
|
||||
section: { supported: true; items: T[] } | { supported: false; reason: string },
|
||||
render: (item: T) => string,
|
||||
): string {
|
||||
if (!section.supported) {
|
||||
return `${title}: (not supported — ${section.reason})`;
|
||||
}
|
||||
if (section.items.length === 0) {
|
||||
return `${title}: (none)`;
|
||||
}
|
||||
const lines = [`${title} (${section.items.length}):`];
|
||||
for (const item of section.items) lines.push(` ${render(item)}`);
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function toolLine(t: { name: string; description?: string }): string {
|
||||
const desc = t.description ? ` — ${oneLine(t.description, 80)}` : "";
|
||||
return `· ${t.name}${desc}`;
|
||||
}
|
||||
|
||||
function resourceLine(r: { uri: string; name: string; mimeType?: string }): string {
|
||||
const mime = r.mimeType ? ` [${r.mimeType}]` : "";
|
||||
return `· ${r.name}${mime} ${r.uri}`;
|
||||
}
|
||||
|
||||
function promptLine(p: {
|
||||
name: string;
|
||||
description?: string;
|
||||
arguments?: Array<{ name: string; required?: boolean }>;
|
||||
}): string {
|
||||
const argPart =
|
||||
p.arguments && p.arguments.length > 0
|
||||
? ` (${p.arguments.map((a) => (a.required ? a.name : `${a.name}?`)).join(", ")})`
|
||||
: "";
|
||||
const desc = p.description ? ` — ${oneLine(p.description, 80)}` : "";
|
||||
return `· ${p.name}${argPart}${desc}`;
|
||||
}
|
||||
|
||||
function oneLine(s: string, max: number): string {
|
||||
const flat = s.replace(/\s+/g, " ").trim();
|
||||
return flat.length <= max ? flat : `${flat.slice(0, max - 1)}…`;
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
/**
|
||||
* `reasonix mcp <subcmd>` — the MCP discovery / helper surface.
|
||||
*
|
||||
* Currently one subcommand: `list`. Prints a curated catalog of popular
|
||||
* MCP servers with their `--mcp` commands ready to copy-paste. More
|
||||
* subcommands (`init`, `try`, etc.) may land in later alphas.
|
||||
*/
|
||||
|
||||
import { MCP_CATALOG, mcpCommandFor } from "../../mcp/catalog.js";
|
||||
|
||||
export interface McpListOptions {
|
||||
/** Emit JSON on stdout instead of the human-readable table. */
|
||||
json?: boolean;
|
||||
}
|
||||
|
||||
export function mcpListCommand(opts: McpListOptions): void {
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(MCP_CATALOG, null, 2));
|
||||
return;
|
||||
}
|
||||
|
||||
console.log("Popular MCP servers you can bridge into Reasonix:");
|
||||
console.log("");
|
||||
for (const entry of MCP_CATALOG) {
|
||||
console.log(` ${pad(entry.name, 12)} ${entry.summary}`);
|
||||
console.log(` ${mcpCommandFor(entry)}`);
|
||||
if (entry.note) console.log(` · ${entry.note}`);
|
||||
console.log("");
|
||||
}
|
||||
console.log("Usage: reasonix chat <one-of-the---mcp-lines-above>");
|
||||
console.log(
|
||||
"Docs: https://github.com/modelcontextprotocol/servers — Anthropic's official server repo",
|
||||
);
|
||||
console.log(
|
||||
" https://mcp.so — community-maintained catalog",
|
||||
);
|
||||
}
|
||||
|
||||
function pad(s: string, width: number): string {
|
||||
return s.length >= width ? s : s + " ".repeat(width - s.length);
|
||||
}
|
||||
@@ -1,21 +1,46 @@
|
||||
import { replayFromFile } from "../../replay.js";
|
||||
import { render } from "ink";
|
||||
import React from "react";
|
||||
import { groupRecordsByTurn, replayFromFile } from "../../replay.js";
|
||||
import type { TranscriptRecord } from "../../transcript.js";
|
||||
import { ReplayApp } from "../ui/ReplayApp.js";
|
||||
|
||||
export interface ReplayOptions {
|
||||
path: string;
|
||||
head?: number;
|
||||
tail?: number;
|
||||
/** Force stdout pretty-print mode (no Ink TUI). Also auto-enabled when stdout is not a TTY. */
|
||||
print?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Non-TUI transcript replay — pretty-prints the records + a summary to stdout.
|
||||
* The full Ink TUI replay ships later; this command is enough for `less` +
|
||||
* scroll workflows and is what CI / blog tooling will use.
|
||||
* Replay a transcript. Two modes:
|
||||
* - Interactive TUI (default when stdout is a TTY): Ink-based j/k navigation.
|
||||
* - Stdout pretty-print (--print, or when stdout is piped): one-shot text
|
||||
* dump + summary. Kept working so shell pipes, CI logs, and `less`
|
||||
* workflows still behave sensibly.
|
||||
*/
|
||||
export function replayCommand(opts: ReplayOptions): void {
|
||||
export async function replayCommand(opts: ReplayOptions): Promise<void> {
|
||||
const wantPrint =
|
||||
opts.print || !process.stdout.isTTY || opts.head !== undefined || opts.tail !== undefined;
|
||||
if (wantPrint) {
|
||||
printReplay(opts);
|
||||
return;
|
||||
}
|
||||
|
||||
const { parsed } = replayFromFile(opts.path);
|
||||
const pages = groupRecordsByTurn(parsed.records);
|
||||
const { waitUntilExit } = render(React.createElement(ReplayApp, { meta: parsed.meta, pages }), {
|
||||
exitOnCtrlC: true,
|
||||
});
|
||||
await waitUntilExit();
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
// stdout pretty-print path (original behavior, preserved for piping / CI)
|
||||
|
||||
function printReplay(opts: ReplayOptions): void {
|
||||
const { parsed, stats } = replayFromFile(opts.path);
|
||||
|
||||
// Header: meta line if present
|
||||
if (parsed.meta) {
|
||||
const m = parsed.meta;
|
||||
const bits: string[] = [`source=${m.source}`];
|
||||
@@ -49,6 +74,11 @@ export function replayCommand(opts: ReplayOptions): void {
|
||||
} else if (stats.prefixHashes.length > 1) {
|
||||
console.log(" (prefix churned — cache-hostile session)");
|
||||
}
|
||||
if (stats.harvestedTurns > 0) {
|
||||
console.log(`harvest (Pillar 2): ${stats.harvestedTurns} turn(s) produced plan state`);
|
||||
console.log(` subgoals: ${stats.totalSubgoals}`);
|
||||
console.log(` uncertainties: ${stats.totalUncertainties}`);
|
||||
}
|
||||
}
|
||||
|
||||
function sliceRecords(records: TranscriptRecord[], opts: ReplayOptions): TranscriptRecord[] {
|
||||
@@ -75,6 +105,21 @@ function renderRecord(rec: TranscriptRecord): void {
|
||||
})()
|
||||
: "";
|
||||
console.log(`${turn} AGENT:${cost}${cache} ${oneLine(rec.content)}`);
|
||||
if (rec.planState) {
|
||||
const ps = rec.planState;
|
||||
if (ps.subgoals.length)
|
||||
console.log(` ‹ subgoals (${ps.subgoals.length}): ${ps.subgoals.join(" · ")}`);
|
||||
if (ps.hypotheses.length)
|
||||
console.log(` ‹ hypotheses (${ps.hypotheses.length}): ${ps.hypotheses.join(" · ")}`);
|
||||
if (ps.uncertainties.length)
|
||||
console.log(
|
||||
` ‹ uncertainties(${ps.uncertainties.length}): ${ps.uncertainties.join(" · ")}`,
|
||||
);
|
||||
if (ps.rejectedPaths.length)
|
||||
console.log(
|
||||
` ‹ rejected (${ps.rejectedPaths.length}): ${ps.rejectedPaths.join(" · ")}`,
|
||||
);
|
||||
}
|
||||
} else if (rec.role === "tool") {
|
||||
const args = rec.args ? ` args=${oneLine(rec.args, 80)}` : "";
|
||||
console.log(`${turn} TOOL ${rec.tool ?? "?"}:${args} → ${oneLine(rec.content, 120)}`);
|
||||
|
||||
+115
-9
@@ -1,13 +1,31 @@
|
||||
import type { WriteStream } from "node:fs";
|
||||
import { stdin, stdout } from "node:process";
|
||||
import { createInterface } from "node:readline/promises";
|
||||
import { defaultConfigPath, isPlausibleKey, loadApiKey, saveApiKey } from "../../config.js";
|
||||
import { loadDotenv } from "../../env.js";
|
||||
import { CacheFirstLoop, DeepSeekClient, ImmutablePrefix } from "../../index.js";
|
||||
import { McpClient } from "../../mcp/client.js";
|
||||
import { bridgeMcpTools } from "../../mcp/registry.js";
|
||||
import { parseMcpSpec } from "../../mcp/spec.js";
|
||||
import { SseTransport } from "../../mcp/sse.js";
|
||||
import { type McpTransport, StdioTransport } from "../../mcp/stdio.js";
|
||||
import { ToolRegistry } from "../../tools.js";
|
||||
import { openTranscriptFile, recordFromLoopEvent, writeRecord } from "../../transcript.js";
|
||||
|
||||
export interface RunOptions {
|
||||
task: string;
|
||||
model: string;
|
||||
system: string;
|
||||
/** Opt into Pillar 2 plan-state harvesting. Adds a cheap V3 call per turn. */
|
||||
harvest?: boolean;
|
||||
/** Self-consistency budget. > 1 runs N parallel samples and picks the best. */
|
||||
branch?: number;
|
||||
/** JSONL transcript path — lets `reasonix replay` / `diff` audit this run. */
|
||||
transcript?: string;
|
||||
/** Zero or more MCP server specs. Each: `"name=cmd args..."` or `"cmd args..."`. */
|
||||
mcp?: string[];
|
||||
/** Global prefix — only honored when a single anonymous server is given. */
|
||||
mcpPrefix?: string;
|
||||
}
|
||||
|
||||
async function ensureApiKey(): Promise<string> {
|
||||
@@ -48,19 +66,107 @@ export async function runCommand(opts: RunOptions): Promise<void> {
|
||||
const apiKey = await ensureApiKey();
|
||||
process.env.DEEPSEEK_API_KEY = apiKey;
|
||||
|
||||
const client = new DeepSeekClient();
|
||||
const prefix = new ImmutablePrefix({ system: opts.system });
|
||||
const loop = new CacheFirstLoop({ client, prefix, model: opts.model });
|
||||
|
||||
for await (const ev of loop.step(opts.task)) {
|
||||
if (ev.role === "assistant_delta" && ev.content) process.stdout.write(ev.content);
|
||||
if (ev.role === "tool") process.stdout.write(`\n[tool ${ev.toolName}] ${ev.content}\n`);
|
||||
if (ev.role === "error") process.stderr.write(`\n[error] ${ev.error}\n`);
|
||||
if (ev.role === "done") process.stdout.write("\n");
|
||||
// Optional MCP setup — mirrors chat's flow. Must happen before loop
|
||||
// construction so the tools make it into the prefix.
|
||||
const requestedSpecs = opts.mcp ?? [];
|
||||
const clients: McpClient[] = [];
|
||||
let tools: ToolRegistry | undefined;
|
||||
let successCount = 0;
|
||||
if (requestedSpecs.length > 0) {
|
||||
tools = new ToolRegistry();
|
||||
for (const raw of requestedSpecs) {
|
||||
try {
|
||||
const spec = parseMcpSpec(raw);
|
||||
const prefix = spec.name
|
||||
? `${spec.name}_`
|
||||
: requestedSpecs.length === 1 && opts.mcpPrefix
|
||||
? opts.mcpPrefix
|
||||
: "";
|
||||
const transport: McpTransport =
|
||||
spec.transport === "sse"
|
||||
? new SseTransport({ url: spec.url })
|
||||
: new StdioTransport({ command: spec.command, args: spec.args });
|
||||
const mcp = new McpClient({ transport });
|
||||
await mcp.initialize();
|
||||
const bridge = await bridgeMcpTools(mcp, { registry: tools, namePrefix: prefix });
|
||||
const source =
|
||||
spec.transport === "sse" ? spec.url : `${spec.command} ${spec.args.join(" ")}`;
|
||||
process.stderr.write(
|
||||
`▸ MCP[${spec.name ?? "anon"}]: ${bridge.registeredNames.length} tool(s) from ${source}\n`,
|
||||
);
|
||||
clients.push(mcp);
|
||||
successCount++;
|
||||
} catch (err) {
|
||||
// Non-fatal — skip and continue, same as `reasonix chat`. A
|
||||
// one-shot `run` invocation with a broken MCP server otherwise
|
||||
// fails the whole run over a side-concern tool the task might
|
||||
// not even touch.
|
||||
process.stderr.write(
|
||||
`▸ MCP setup SKIPPED for "${raw}": ${(err as Error).message}\n → run \`reasonix setup\` to remove broken entries from your saved config.\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
if (successCount === 0) tools = undefined;
|
||||
}
|
||||
|
||||
const client = new DeepSeekClient();
|
||||
const prefix = new ImmutablePrefix({
|
||||
system: opts.system,
|
||||
toolSpecs: tools?.specs(),
|
||||
});
|
||||
const loop = new CacheFirstLoop({
|
||||
client,
|
||||
prefix,
|
||||
tools,
|
||||
model: opts.model,
|
||||
harvest: opts.harvest,
|
||||
branch: opts.branch,
|
||||
});
|
||||
const prefixHash = prefix.fingerprint;
|
||||
|
||||
let transcriptStream: WriteStream | null = null;
|
||||
if (opts.transcript) {
|
||||
transcriptStream = openTranscriptFile(opts.transcript, {
|
||||
version: 1,
|
||||
source: "reasonix run",
|
||||
model: opts.model,
|
||||
startedAt: new Date().toISOString(),
|
||||
});
|
||||
// Also persist the user turn itself (the loop's event stream starts with
|
||||
// assistant output, not the prompt we're about to send).
|
||||
writeRecord(transcriptStream, {
|
||||
ts: new Date().toISOString(),
|
||||
turn: 1,
|
||||
role: "user",
|
||||
content: opts.task,
|
||||
});
|
||||
}
|
||||
|
||||
try {
|
||||
for await (const ev of loop.step(opts.task)) {
|
||||
if (ev.role === "assistant_delta" && ev.content) process.stdout.write(ev.content);
|
||||
if (ev.role === "tool") process.stdout.write(`\n[tool ${ev.toolName}] ${ev.content}\n`);
|
||||
if (ev.role === "error") process.stderr.write(`\n[error] ${ev.error}\n`);
|
||||
if (ev.role === "done") process.stdout.write("\n");
|
||||
// Persist every non-streaming event — deltas would flood the file and
|
||||
// aren't useful for replay (replay renders final content, not keystrokes).
|
||||
if (transcriptStream && ev.role !== "assistant_delta") {
|
||||
writeRecord(transcriptStream, recordFromLoopEvent(ev, { model: opts.model, prefixHash }));
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
transcriptStream?.end();
|
||||
}
|
||||
|
||||
const s = loop.stats.summary();
|
||||
process.stdout.write(
|
||||
`\n— turns:${s.turns} cache:${(s.cacheHitRatio * 100).toFixed(1)}% ` +
|
||||
`cost:$${s.totalCostUsd.toFixed(6)} save-vs-claude:${s.savingsVsClaudePct.toFixed(1)}%\n`,
|
||||
);
|
||||
if (opts.transcript) {
|
||||
process.stdout.write(`\ntranscript: ${opts.transcript}\n`);
|
||||
process.stdout.write(` → npx reasonix replay ${opts.transcript}\n`);
|
||||
}
|
||||
|
||||
for (const c of clients) await c.close();
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
/**
|
||||
* `reasonix sessions` — CLI equivalent of the `/sessions` slash command.
|
||||
*
|
||||
* Two modes:
|
||||
* - `reasonix sessions` list every session under ~/.reasonix/sessions/
|
||||
* - `reasonix sessions <name>` dump one session's messages in readable form
|
||||
*
|
||||
* Neither needs an API key — pure filesystem reads of JSONL files written
|
||||
* by previous chat runs.
|
||||
*/
|
||||
|
||||
import { listSessions, loadSessionMessages, sessionPath } from "../../index.js";
|
||||
import type { ChatMessage } from "../../index.js";
|
||||
|
||||
export interface SessionsOptions {
|
||||
/** When present, inspect that session instead of listing. */
|
||||
name?: string;
|
||||
/** Include assistant tool-call metadata in the inspect output. */
|
||||
verbose?: boolean;
|
||||
}
|
||||
|
||||
export function sessionsCommand(opts: SessionsOptions): void {
|
||||
if (opts.name) {
|
||||
inspectSession(opts.name, !!opts.verbose);
|
||||
} else {
|
||||
listAll();
|
||||
}
|
||||
}
|
||||
|
||||
function listAll(): void {
|
||||
const items = listSessions();
|
||||
if (items.length === 0) {
|
||||
console.log(
|
||||
"no saved sessions yet — run `reasonix chat` (sessions are auto-saved unless --no-session).",
|
||||
);
|
||||
return;
|
||||
}
|
||||
console.log("Saved sessions (~/.reasonix/sessions/):");
|
||||
console.log("");
|
||||
console.log(` ${"name".padEnd(22)} ${"msgs".padStart(6)} ${"size".padStart(8)} modified`);
|
||||
console.log(` ${"─".repeat(60)}`);
|
||||
for (const s of items) {
|
||||
const sizeKb = `${(s.size / 1024).toFixed(1)} KB`;
|
||||
const when = s.mtime.toISOString().replace("T", " ").slice(0, 16);
|
||||
console.log(
|
||||
` ${s.name.padEnd(22)} ${String(s.messageCount).padStart(6)} ${sizeKb.padStart(8)} ${when}`,
|
||||
);
|
||||
}
|
||||
console.log("");
|
||||
console.log("Inspect: reasonix sessions <name>");
|
||||
console.log("Resume: reasonix chat --session <name>");
|
||||
}
|
||||
|
||||
function inspectSession(name: string, verbose: boolean): void {
|
||||
const path = sessionPath(name);
|
||||
const messages = loadSessionMessages(name);
|
||||
if (messages.length === 0) {
|
||||
console.error(`no session named "${name}" (or it's empty).`);
|
||||
console.error(`looked at: ${path}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(`[session] ${name} ${messages.length} messages ${path}`);
|
||||
console.log("");
|
||||
|
||||
let turnIndex = 0;
|
||||
for (const msg of messages) {
|
||||
renderMessage(msg, turnIndex, verbose);
|
||||
// Roughly bump "turn" after each user message so the reader can follow
|
||||
// the conversation shape without the transcript's richer turn numbering.
|
||||
if (msg.role === "user") turnIndex++;
|
||||
}
|
||||
}
|
||||
|
||||
function renderMessage(msg: ChatMessage, turnIdx: number, verbose: boolean): void {
|
||||
const turn = turnIdx > 0 ? `[t${turnIdx}]` : "[start]";
|
||||
const content = typeof msg.content === "string" ? msg.content : "";
|
||||
const flat = oneLine(content);
|
||||
|
||||
if (msg.role === "user") {
|
||||
console.log(`${turn} USER: ${flat}`);
|
||||
} else if (msg.role === "assistant") {
|
||||
console.log(`${turn} AGENT: ${flat || "(tool call only)"}`);
|
||||
if (verbose && msg.tool_calls?.length) {
|
||||
for (const tc of msg.tool_calls) {
|
||||
console.log(
|
||||
` → call ${tc.function?.name} ${truncate(tc.function?.arguments ?? "", 80)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
} else if (msg.role === "tool") {
|
||||
console.log(`${turn} TOOL ${msg.name ?? "?"}: ${truncate(flat, 160)}`);
|
||||
} else if (msg.role === "system") {
|
||||
if (verbose) console.log(`${turn} SYSTEM: ${truncate(flat, 160)}`);
|
||||
// otherwise suppress — session's system prompt is usually session-wide
|
||||
// boilerplate.
|
||||
}
|
||||
}
|
||||
|
||||
function oneLine(s: string, max = 200): string {
|
||||
const collapsed = s.replace(/\s+/g, " ").trim();
|
||||
return collapsed.length > max ? `${collapsed.slice(0, max)}…` : collapsed;
|
||||
}
|
||||
|
||||
function truncate(s: string, max: number): string {
|
||||
return s.length <= max ? s : `${s.slice(0, max)}…`;
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
/**
|
||||
* `reasonix setup` — re-mount the first-run wizard on demand so users
|
||||
* can reconfigure (add/remove MCP servers, switch preset) without
|
||||
* editing JSON by hand.
|
||||
*
|
||||
* Invoked both explicitly (`reasonix setup`) and implicitly (the no-args
|
||||
* entry point when `setupCompleted` is false).
|
||||
*/
|
||||
|
||||
import { render } from "ink";
|
||||
import React from "react";
|
||||
import { loadApiKey, readConfig } from "../../config.js";
|
||||
import { loadDotenv } from "../../env.js";
|
||||
import { Wizard } from "../ui/Wizard.js";
|
||||
|
||||
export interface SetupOptions {
|
||||
/**
|
||||
* When true, bypass the API-key step even if no key is saved — useful
|
||||
* from test harnesses. Normal CLI use always pushes through the key
|
||||
* step when missing.
|
||||
*/
|
||||
skipKeyStep?: boolean;
|
||||
}
|
||||
|
||||
export async function setupCommand(_opts: SetupOptions = {}): Promise<void> {
|
||||
loadDotenv();
|
||||
const existingKey = loadApiKey();
|
||||
const existing = readConfig();
|
||||
|
||||
const { waitUntilExit, unmount } = render(
|
||||
<Wizard
|
||||
existingApiKey={existingKey}
|
||||
initial={{ preset: existing.preset, mcp: existing.mcp }}
|
||||
onComplete={() => {
|
||||
// Ink handles its own enter-to-exit inside the "saved" step; we
|
||||
// just wait for the app to exit naturally.
|
||||
}}
|
||||
onCancel={() => {
|
||||
unmount();
|
||||
}}
|
||||
/>,
|
||||
{ exitOnCtrlC: true },
|
||||
);
|
||||
await waitUntilExit();
|
||||
}
|
||||
+182
-32
@@ -1,11 +1,18 @@
|
||||
import { Command } from "commander";
|
||||
import { readConfig } from "../config.js";
|
||||
import { VERSION } from "../index.js";
|
||||
import { chatCommand } from "./commands/chat.js";
|
||||
import { codeCommand } from "./commands/code.js";
|
||||
import { diffCommand } from "./commands/diff.js";
|
||||
import { mcpInspectCommand } from "./commands/mcp-inspect.js";
|
||||
import { mcpListCommand } from "./commands/mcp.js";
|
||||
import { replayCommand } from "./commands/replay.js";
|
||||
import { runCommand } from "./commands/run.js";
|
||||
import { sessionsCommand } from "./commands/sessions.js";
|
||||
import { setupCommand } from "./commands/setup.js";
|
||||
import { statsCommand } from "./commands/stats.js";
|
||||
import { versionCommand } from "./commands/version.js";
|
||||
import { resolveDefaults } from "./resolve.js";
|
||||
|
||||
const DEFAULT_SYSTEM =
|
||||
"You are Reasonix, a helpful DeepSeek-powered assistant. Be concise and accurate. Use tools when available.";
|
||||
@@ -16,55 +23,149 @@ program
|
||||
.description("DeepSeek-native agent framework — built for cache hits and cheap tokens.")
|
||||
.version(VERSION);
|
||||
|
||||
// `reasonix` with no subcommand → launch the friendliest flow.
|
||||
// First run (no config yet) → interactive setup wizard.
|
||||
// Otherwise → chat with saved defaults. This is the "one command to
|
||||
// rule them all" entry for non-power-users: they don't need to learn
|
||||
// `chat` / `setup` / `--mcp` — just type `reasonix`.
|
||||
program.action(async () => {
|
||||
const cfg = readConfig();
|
||||
if (!cfg.setupCompleted) {
|
||||
await setupCommand({});
|
||||
return;
|
||||
}
|
||||
const defaults = resolveDefaults({});
|
||||
await chatCommand({
|
||||
model: defaults.model,
|
||||
system: DEFAULT_SYSTEM,
|
||||
harvest: defaults.harvest,
|
||||
branch: defaults.branch,
|
||||
session: defaults.session,
|
||||
mcp: defaults.mcp,
|
||||
});
|
||||
});
|
||||
|
||||
program
|
||||
.command("setup")
|
||||
.description("Interactive wizard — API key, preset, MCP servers. Re-run any time to reconfigure.")
|
||||
.action(async () => {
|
||||
await setupCommand({});
|
||||
});
|
||||
|
||||
program
|
||||
.command("code [dir]")
|
||||
.description(
|
||||
"Code-editing chat — filesystem MCP auto-bridged at <dir> (default: cwd), coding system prompt, smart preset. Model proposes SEARCH/REPLACE blocks; Reasonix applies them to disk.",
|
||||
)
|
||||
.option("-m, --model <id>", "Override default reasoner model")
|
||||
.option("--no-session", "Disable session persistence for this run")
|
||||
.option("--transcript <path>", "Write a JSONL transcript to this path")
|
||||
.action(async (dir: string | undefined, opts) => {
|
||||
await codeCommand({
|
||||
dir,
|
||||
model: opts.model,
|
||||
noSession: opts.session === false,
|
||||
transcript: opts.transcript,
|
||||
});
|
||||
});
|
||||
|
||||
program
|
||||
.command("chat")
|
||||
.description("Interactive Ink TUI with live cache/cost panel.")
|
||||
.option("-m, --model <id>", "DeepSeek model id", "deepseek-chat")
|
||||
.option("-m, --model <id>", "DeepSeek model id (overrides preset)")
|
||||
.option("-s, --system <prompt>", "System prompt (pinned in the immutable prefix)", DEFAULT_SYSTEM)
|
||||
.option("--transcript <path>", "Write a JSONL transcript to this path")
|
||||
.option(
|
||||
"--preset <name>",
|
||||
"Bundle of model + harvest + branch. One of: fast, smart, max. Overrides config.preset.",
|
||||
)
|
||||
.option(
|
||||
"--harvest",
|
||||
"Extract typed plan state from R1 reasoning (Pillar 2, adds a cheap V3 call per turn)",
|
||||
"Extract typed plan state from R1 reasoning (Pillar 2). Overrides preset's harvest setting.",
|
||||
)
|
||||
.option(
|
||||
"--branch <n>",
|
||||
"Self-consistency: run N parallel samples per turn and pick the most confident (disables streaming; enables harvest)",
|
||||
(v) => Number.parseInt(v, 10),
|
||||
)
|
||||
.option(
|
||||
"--session <name>",
|
||||
"Use a named session (default: 'default'). Resume the same session next time.",
|
||||
)
|
||||
.option("--session <name>", "Use a named session (default: from config, usually 'default').")
|
||||
.option("--no-session", "Disable session persistence for this run (ephemeral chat)")
|
||||
.option(
|
||||
"--mcp <spec>",
|
||||
'MCP server spec; repeatable. "name=cmd args...", "cmd args...", or a URL (http/https → SSE transport). Overrides config.mcp when provided.',
|
||||
(value: string, previous: string[] = []) => [...previous, value],
|
||||
[] as string[],
|
||||
)
|
||||
.option(
|
||||
"--mcp-prefix <str>",
|
||||
"Global prefix applied to every MCP tool (only honored when no per-spec name is set; avoids collisions with a single anonymous server)",
|
||||
)
|
||||
.option("--no-config", "Ignore `~/.reasonix/config.json` — useful for CI or reproducing issues")
|
||||
.action(async (opts) => {
|
||||
// Default behavior: every chat is auto-saved to a session named 'default'
|
||||
// and auto-resumed next launch. Pass --no-session to opt out, or
|
||||
// --session <name> to use a different session.
|
||||
let session: string | undefined;
|
||||
if (opts.session === false) {
|
||||
session = undefined; // --no-session
|
||||
} else if (typeof opts.session === "string" && opts.session.length > 0) {
|
||||
session = opts.session;
|
||||
} else {
|
||||
session = "default";
|
||||
}
|
||||
await chatCommand({
|
||||
const defaults = resolveDefaults({
|
||||
model: opts.model,
|
||||
harvest: opts.harvest,
|
||||
branch: opts.branch,
|
||||
mcp: opts.mcp as string[],
|
||||
session: opts.session,
|
||||
preset: opts.preset,
|
||||
noConfig: opts.config === false,
|
||||
});
|
||||
await chatCommand({
|
||||
model: defaults.model,
|
||||
system: opts.system,
|
||||
transcript: opts.transcript,
|
||||
harvest: !!opts.harvest,
|
||||
branch: Number.isFinite(opts.branch) && opts.branch > 1 ? opts.branch : undefined,
|
||||
session,
|
||||
harvest: defaults.harvest,
|
||||
branch: defaults.branch,
|
||||
session: defaults.session,
|
||||
mcp: defaults.mcp,
|
||||
mcpPrefix: opts.mcpPrefix,
|
||||
});
|
||||
});
|
||||
|
||||
program
|
||||
.command("run <task>")
|
||||
.description("Run a single task non-interactively, streaming output.")
|
||||
.option("-m, --model <id>", "DeepSeek model id", "deepseek-chat")
|
||||
.option("-m, --model <id>", "DeepSeek model id (overrides preset)")
|
||||
.option("-s, --system <prompt>", "System prompt", DEFAULT_SYSTEM)
|
||||
.option("--preset <name>", "Bundle of model + harvest + branch: fast | smart | max")
|
||||
.option("--harvest", "Extract typed plan state from R1 reasoning (Pillar 2)")
|
||||
.option(
|
||||
"--branch <n>",
|
||||
"Self-consistency: run N parallel samples per turn and pick the most confident",
|
||||
(v) => Number.parseInt(v, 10),
|
||||
)
|
||||
.option("--transcript <path>", "Write a JSONL transcript to this path for replay/diff")
|
||||
.option(
|
||||
"--mcp <spec>",
|
||||
'MCP server spec; repeatable. "name=cmd args...", "cmd args...", or a URL (http/https → SSE).',
|
||||
(value: string, previous: string[] = []) => [...previous, value],
|
||||
[] as string[],
|
||||
)
|
||||
.option(
|
||||
"--mcp-prefix <str>",
|
||||
"Global prefix (only honored when no per-spec name is set; for a single anonymous server)",
|
||||
)
|
||||
.option("--no-config", "Ignore `~/.reasonix/config.json` — useful for CI or reproducing issues")
|
||||
.action(async (task: string, opts) => {
|
||||
await runCommand({ task, model: opts.model, system: opts.system });
|
||||
const defaults = resolveDefaults({
|
||||
model: opts.model,
|
||||
harvest: opts.harvest,
|
||||
branch: opts.branch,
|
||||
mcp: opts.mcp as string[],
|
||||
preset: opts.preset,
|
||||
noConfig: opts.config === false,
|
||||
});
|
||||
await runCommand({
|
||||
task,
|
||||
model: defaults.model,
|
||||
system: opts.system,
|
||||
harvest: defaults.harvest,
|
||||
branch: defaults.branch,
|
||||
transcript: opts.transcript,
|
||||
mcp: defaults.mcp,
|
||||
mcpPrefix: opts.mcpPrefix,
|
||||
});
|
||||
});
|
||||
|
||||
program
|
||||
@@ -74,16 +175,26 @@ program
|
||||
statsCommand({ transcript });
|
||||
});
|
||||
|
||||
program
|
||||
.command("sessions [name]")
|
||||
.description("List saved chat sessions, or inspect one by name.")
|
||||
.option("-v, --verbose", "Include system prompts + tool-call metadata when inspecting")
|
||||
.action((name: string | undefined, opts) => {
|
||||
sessionsCommand({ name, verbose: !!opts.verbose });
|
||||
});
|
||||
|
||||
program
|
||||
.command("replay <transcript>")
|
||||
.description(
|
||||
"Pretty-print a transcript + rebuild its session summary (cost, cache, prefix stability). No API calls.",
|
||||
"Interactive Ink TUI to scrub through a transcript + rebuild its session summary (cost, cache, prefix stability). No API calls.",
|
||||
)
|
||||
.option("--head <n>", "Show only the first N records", (v) => Number.parseInt(v, 10))
|
||||
.option("--tail <n>", "Show only the last N records", (v) => Number.parseInt(v, 10))
|
||||
.action((transcript: string, opts) => {
|
||||
replayCommand({
|
||||
.option("--print", "Dump to stdout instead of mounting the TUI (auto when piped)")
|
||||
.option("--head <n>", "stdout mode only — show first N records", (v) => Number.parseInt(v, 10))
|
||||
.option("--tail <n>", "stdout mode only — show last N records", (v) => Number.parseInt(v, 10))
|
||||
.action(async (transcript: string, opts) => {
|
||||
await replayCommand({
|
||||
path: transcript,
|
||||
print: !!opts.print,
|
||||
head: Number.isFinite(opts.head) ? opts.head : undefined,
|
||||
tail: Number.isFinite(opts.tail) ? opts.tail : undefined,
|
||||
});
|
||||
@@ -91,12 +202,51 @@ program
|
||||
|
||||
program
|
||||
.command("diff <a> <b>")
|
||||
.description("Compare two transcripts: aggregate deltas + first divergence.")
|
||||
.option("--md <path>", "Also write a markdown report (blog-ready) to this path")
|
||||
.description(
|
||||
"Compare two transcripts in a split-pane Ink TUI (default) or stdout table. Use n/N to jump across divergences.",
|
||||
)
|
||||
.option("--md <path>", "Write a markdown report (blog-ready) to this path")
|
||||
.option("--print", "Force stdout table instead of the TUI (auto when piped)")
|
||||
.option("--tui", "Force the TUI even when piped (rare)")
|
||||
.option("--label-a <label>", "Display label for transcript A (default: filename)")
|
||||
.option("--label-b <label>", "Display label for transcript B (default: filename)")
|
||||
.action((a: string, b: string, opts) => {
|
||||
diffCommand({ a, b, mdPath: opts.md, labelA: opts.labelA, labelB: opts.labelB });
|
||||
.action(async (a: string, b: string, opts) => {
|
||||
await diffCommand({
|
||||
a,
|
||||
b,
|
||||
mdPath: opts.md,
|
||||
labelA: opts.labelA,
|
||||
labelB: opts.labelB,
|
||||
print: !!opts.print,
|
||||
tui: !!opts.tui,
|
||||
});
|
||||
});
|
||||
|
||||
const mcp = program
|
||||
.command("mcp")
|
||||
.description("Model Context Protocol helpers — discover servers, test your setup.");
|
||||
|
||||
mcp
|
||||
.command("list")
|
||||
.description("Show a curated catalog of popular MCP servers with ready-to-use --mcp commands.")
|
||||
.option("--json", "Emit the catalog as JSON instead of the human-readable table")
|
||||
.action((opts) => {
|
||||
mcpListCommand({ json: !!opts.json });
|
||||
});
|
||||
|
||||
mcp
|
||||
.command("inspect <spec>")
|
||||
.description(
|
||||
'Connect to one MCP server and print its server info + tools/resources/prompts. <spec> takes the same forms as --mcp ("name=cmd args", "cmd args", or an SSE URL).',
|
||||
)
|
||||
.option("--json", "Emit the full inspection report as JSON instead of the human-readable summary")
|
||||
.action(async (spec: string, opts) => {
|
||||
try {
|
||||
await mcpInspectCommand({ spec, json: !!opts.json });
|
||||
} catch (err) {
|
||||
process.stderr.write(`mcp inspect failed: ${(err as Error).message}\n`);
|
||||
process.exit(1);
|
||||
}
|
||||
});
|
||||
|
||||
program.command("version").description("Print Reasonix version.").action(versionCommand);
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
/**
|
||||
* Merge config defaults with CLI flags into the concrete options that
|
||||
* `chatCommand` / `runCommand` need.
|
||||
*
|
||||
* Precedence (highest wins):
|
||||
* 1. Explicit per-setting CLI flag (`--model`, `--harvest`, `--branch`, `--mcp`)
|
||||
* 2. Explicit `--preset <name>` CLI flag
|
||||
* 3. `config.preset` from `~/.reasonix/config.json` (set by `reasonix setup`)
|
||||
* 4. Hardcoded "fast" preset defaults
|
||||
*
|
||||
* Keeping this logic in one place — rather than duplicating across
|
||||
* `chat` and `run` — means the precedence rule only lives in one unit
|
||||
* test and the shape of the merge is identical for both commands.
|
||||
*/
|
||||
|
||||
import { type PresetName, type ReasonixConfig, readConfig } from "../config.js";
|
||||
import { PRESETS } from "./ui/presets.js";
|
||||
|
||||
export interface ResolvedDefaults {
|
||||
model: string;
|
||||
harvest: boolean;
|
||||
branch: number | undefined;
|
||||
mcp: string[];
|
||||
session: string | undefined;
|
||||
}
|
||||
|
||||
export interface RawCliFlags {
|
||||
model?: string;
|
||||
harvest?: boolean;
|
||||
/** From `commander`; already parseInt'd. */
|
||||
branch?: number;
|
||||
mcp?: string[];
|
||||
/** Commander's `--no-session` surfaces as `false`; `--session X` as a string. */
|
||||
session?: string | false;
|
||||
/** `--preset <name>`. */
|
||||
preset?: string;
|
||||
/** When true, ignore config entirely (power-user escape hatch). */
|
||||
noConfig?: boolean;
|
||||
}
|
||||
|
||||
export function resolveDefaults(flags: RawCliFlags): ResolvedDefaults {
|
||||
const cfg: ReasonixConfig = flags.noConfig ? {} : readConfig();
|
||||
const preset = pickPreset(flags.preset, cfg.preset);
|
||||
const presetSettings = PRESETS[preset];
|
||||
|
||||
const model = flags.model ?? presetSettings.model;
|
||||
const harvest = flags.harvest === true ? true : presetSettings.harvest;
|
||||
const branchFromFlag = normalizeBranch(flags.branch);
|
||||
const branch = branchFromFlag ?? (presetSettings.branch > 1 ? presetSettings.branch : undefined);
|
||||
|
||||
// `--mcp` accumulator is [] when absent. Treat empty from flags as
|
||||
// "user didn't pass" → fall through to config. Users who explicitly
|
||||
// want zero MCP servers can pass `--no-config` or edit the file.
|
||||
const mcp = flags.mcp && flags.mcp.length > 0 ? flags.mcp : (cfg.mcp ?? []);
|
||||
|
||||
const session = resolveSession(flags.session, cfg.session);
|
||||
|
||||
return { model, harvest, branch, mcp, session };
|
||||
}
|
||||
|
||||
function pickPreset(
|
||||
flagPreset: string | undefined,
|
||||
configPreset: PresetName | undefined,
|
||||
): PresetName {
|
||||
if (flagPreset && isPresetName(flagPreset)) return flagPreset;
|
||||
if (configPreset) return configPreset;
|
||||
return "fast";
|
||||
}
|
||||
|
||||
function isPresetName(s: string): s is PresetName {
|
||||
return s === "fast" || s === "smart" || s === "max";
|
||||
}
|
||||
|
||||
function normalizeBranch(raw: number | undefined): number | undefined {
|
||||
if (raw === undefined) return undefined;
|
||||
if (!Number.isFinite(raw) || raw <= 1) return undefined;
|
||||
return Math.min(raw, 8);
|
||||
}
|
||||
|
||||
function resolveSession(
|
||||
flag: string | false | undefined,
|
||||
configSession: string | null | undefined,
|
||||
): string | undefined {
|
||||
if (flag === false) return undefined; // --no-session
|
||||
if (typeof flag === "string" && flag.length > 0) return flag;
|
||||
if (configSession === null) return undefined; // config opted out
|
||||
if (typeof configSession === "string" && configSession.length > 0) return configSession;
|
||||
return "default";
|
||||
}
|
||||
+465
-18
@@ -1,14 +1,25 @@
|
||||
import type { WriteStream } from "node:fs";
|
||||
import { Box, Static, Text, useApp } from "ink";
|
||||
import { Box, Static, Text, useApp, useInput } from "ink";
|
||||
import React, { useCallback, useEffect, useMemo, useRef, useState } from "react";
|
||||
import {
|
||||
type ApplyResult,
|
||||
type EditBlock,
|
||||
type EditSnapshot,
|
||||
applyEditBlocks,
|
||||
parseEditBlocks,
|
||||
restoreSnapshots,
|
||||
snapshotBeforeEdits,
|
||||
} from "../../code/edit-blocks.js";
|
||||
import { CacheFirstLoop, DeepSeekClient, ImmutablePrefix } from "../../index.js";
|
||||
import type { LoopEvent } from "../../loop.js";
|
||||
import type { SessionSummary } from "../../telemetry.js";
|
||||
import type { ToolRegistry } from "../../tools.js";
|
||||
import { openTranscriptFile, recordFromLoopEvent, writeRecord } from "../../transcript.js";
|
||||
import { type DisplayEvent, EventRow } from "./EventLog.js";
|
||||
import { PromptInput } from "./PromptInput.js";
|
||||
import { SlashSuggestions } from "./SlashSuggestions.js";
|
||||
import { StatsPanel } from "./StatsPanel.js";
|
||||
import { handleSlash, parseSlash } from "./slash.js";
|
||||
import { type McpServerSummary, handleSlash, parseSlash, suggestSlashCommands } from "./slash.js";
|
||||
|
||||
export interface AppProps {
|
||||
model: string;
|
||||
@@ -17,6 +28,25 @@ export interface AppProps {
|
||||
harvest?: boolean;
|
||||
branch?: number;
|
||||
session?: string;
|
||||
/**
|
||||
* Pre-populated tool registry (e.g. from bridgeMcpTools()). When present,
|
||||
* its specs are folded into the ImmutablePrefix so the model sees them,
|
||||
* and its dispatch is used for tool calls — MCP tools become first-class.
|
||||
*/
|
||||
tools?: ToolRegistry;
|
||||
/** Raw `--mcp` / config-derived spec strings, for `/mcp` slash display. */
|
||||
mcpSpecs?: string[];
|
||||
/**
|
||||
* Pre-captured inspection reports for each connected MCP server,
|
||||
* collected once at chat startup. Drives the rich `/mcp` slash view
|
||||
* (tools + resources + prompts per server).
|
||||
*/
|
||||
mcpServers?: McpServerSummary[];
|
||||
/**
|
||||
* When set, parse SEARCH/REPLACE blocks from assistant responses and
|
||||
* apply them to disk under `rootDir`. Set by `reasonix code`.
|
||||
*/
|
||||
codeMode?: { rootDir: string };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -32,18 +62,67 @@ interface StreamingState {
|
||||
reasoning: string;
|
||||
}
|
||||
|
||||
export function App({ model, system, transcript, harvest, branch, session }: AppProps) {
|
||||
export function App({
|
||||
model,
|
||||
system,
|
||||
transcript,
|
||||
harvest,
|
||||
branch,
|
||||
session,
|
||||
tools,
|
||||
mcpSpecs,
|
||||
mcpServers,
|
||||
codeMode,
|
||||
}: AppProps) {
|
||||
const { exit } = useApp();
|
||||
const [historical, setHistorical] = useState<DisplayEvent[]>([]);
|
||||
const [streaming, setStreaming] = useState<DisplayEvent | null>(null);
|
||||
const [input, setInput] = useState("");
|
||||
const [busy, setBusy] = useState(false);
|
||||
// Tracks whether the current turn has been aborted via Esc, so the
|
||||
// Esc handler only fires once per turn (repeated presses would yield
|
||||
// stacked warning events).
|
||||
const abortedThisTurn = useRef(false);
|
||||
// Name + truncated args of the tool currently dispatching. Populated
|
||||
// on `tool_start`, cleared on `tool` (or error). Drives the
|
||||
// "▸ tool<X> running…" pulse-spinner row so long tool calls don't
|
||||
// look like the app hung.
|
||||
const [ongoingTool, setOngoingTool] = useState<{ name: string; args?: string } | null>(null);
|
||||
// Snapshots of every file the *last* edit batch touched, keyed by
|
||||
// nothing more than "most recent". `/undo` restores from this ref
|
||||
// and nulls it out — one level of undo, Aider-style. Multi-step
|
||||
// undo would need a proper history stack and a clear policy for
|
||||
// when the stack clears; v1 keeps it simple.
|
||||
const lastEditSnapshots = useRef<EditSnapshot[] | null>(null);
|
||||
// Pending edit blocks awaiting `/apply` or `/discard`. We do NOT
|
||||
// auto-apply — v0.4.1 showed that "model proposed, so apply" turns
|
||||
// analysis into unintended edits. The user explicitly confirms now.
|
||||
const pendingEdits = useRef<EditBlock[]>([]);
|
||||
// Shell-style history of user prompts. ↑/↓ while idle walks it;
|
||||
// submit pushes to the end. Cursor -1 = "live input", 0+ = "N turns
|
||||
// back from newest". We don't persist history to disk — sessions
|
||||
// already keep the message log, and cross-session bash-style recall
|
||||
// would need per-project scoping we haven't designed.
|
||||
const promptHistory = useRef<string[]>([]);
|
||||
const historyCursor = useRef<number>(-1);
|
||||
// Full untruncated tool results, in arrival order. The EventLog
|
||||
// renderer clips tool output at 400 chars for display; `/tool N`
|
||||
// reads from this ref to show the real thing. Not persisted — a
|
||||
// resumed session replays the log (which has the same content in
|
||||
// `tool` messages) but we don't repopulate this ref on resume
|
||||
// because the user wouldn't expect `/tool` to reach back across
|
||||
// process boundaries.
|
||||
const toolHistoryRef = useRef<Array<{ toolName: string; text: string }>>([]);
|
||||
// Highlighted suggestion index. ↑/↓ move within the current match
|
||||
// set while the user is typing a `/…` prefix; Enter or Tab pick.
|
||||
const [slashSelected, setSlashSelected] = useState(0);
|
||||
const [summary, setSummary] = useState<SessionSummary>({
|
||||
turns: 0,
|
||||
totalCostUsd: 0,
|
||||
claudeEquivalentUsd: 0,
|
||||
savingsVsClaudePct: 0,
|
||||
cacheHitRatio: 0,
|
||||
lastPromptTokens: 0,
|
||||
});
|
||||
|
||||
const transcriptRef = useRef<WriteStream | null>(null);
|
||||
@@ -61,15 +140,37 @@ export function App({ model, system, transcript, harvest, branch, session }: App
|
||||
};
|
||||
}, []);
|
||||
|
||||
// The currently matching slash suggestions, or `null` when the
|
||||
// user isn't in slash-prefix mode. Shared between the useInput
|
||||
// handler (navigation + Tab-complete) and SlashSuggestions
|
||||
// (rendering + highlight) so both stay in sync.
|
||||
const slashMatches = useMemo(() => {
|
||||
if (!input.startsWith("/") || input.includes(" ")) return null;
|
||||
return suggestSlashCommands(input.slice(1), !!codeMode);
|
||||
}, [input, codeMode]);
|
||||
useEffect(() => {
|
||||
// Keep selection in range whenever the match set shrinks. Reset
|
||||
// to 0 whenever we re-enter slash mode (matches goes null → array
|
||||
// triggers the useEffect too).
|
||||
setSlashSelected((prev) => {
|
||||
if (!slashMatches || slashMatches.length === 0) return 0;
|
||||
if (prev >= slashMatches.length) return slashMatches.length - 1;
|
||||
return prev;
|
||||
});
|
||||
}, [slashMatches]);
|
||||
|
||||
const loopRef = useRef<CacheFirstLoop | null>(null);
|
||||
const loop = useMemo(() => {
|
||||
if (loopRef.current) return loopRef.current;
|
||||
const client = new DeepSeekClient();
|
||||
const prefix = new ImmutablePrefix({ system });
|
||||
const l = new CacheFirstLoop({ client, prefix, model, harvest, branch, session });
|
||||
const prefix = new ImmutablePrefix({
|
||||
system,
|
||||
toolSpecs: tools?.specs(),
|
||||
});
|
||||
const l = new CacheFirstLoop({ client, prefix, tools, model, harvest, branch, session });
|
||||
loopRef.current = l;
|
||||
return l;
|
||||
}, [model, system, harvest, branch, session]);
|
||||
}, [model, system, harvest, branch, session, tools]);
|
||||
|
||||
// Surface a one-time banner about session state on first mount.
|
||||
const sessionBannerShown = useRef(false);
|
||||
@@ -106,6 +207,110 @@ export function App({ model, system, transcript, harvest, branch, session }: App
|
||||
}
|
||||
}, [session, loop]);
|
||||
|
||||
// Esc during busy → forward to the loop as an abort signal. The loop
|
||||
// finishes the tool call in flight (we can't kill subprocess stdio
|
||||
// mid-write), then diverts to its no-tools summary path so the user
|
||||
// gets an answer instead of a hard stop. Only listens while busy so
|
||||
// we don't accidentally hijack Esc in other contexts.
|
||||
//
|
||||
// Also handles ↑/↓ shell-style history while idle. We don't use
|
||||
// ink-text-input's (absent) history support; parent-level useInput
|
||||
// is simpler and lets us own the cursor semantics.
|
||||
useInput((_input, key) => {
|
||||
if (key.escape && busy) {
|
||||
if (abortedThisTurn.current) return;
|
||||
abortedThisTurn.current = true;
|
||||
loop.abort();
|
||||
return;
|
||||
}
|
||||
if (busy) return;
|
||||
|
||||
// Slash-suggestion mode takes priority over history recall.
|
||||
// When the user is typing a `/…` prefix and there are matches,
|
||||
// ↑/↓ walk the suggestion list and Tab snaps the input to the
|
||||
// highlighted command. Enter is handled in `handleSubmit` so
|
||||
// TextInput's onSubmit still fires cleanly.
|
||||
if (slashMatches && slashMatches.length > 0) {
|
||||
if (key.upArrow) {
|
||||
setSlashSelected((i) => Math.max(0, i - 1));
|
||||
return;
|
||||
}
|
||||
if (key.downArrow) {
|
||||
setSlashSelected((i) => Math.min(slashMatches.length - 1, i + 1));
|
||||
return;
|
||||
}
|
||||
if (key.tab) {
|
||||
const sel = slashMatches[slashSelected] ?? slashMatches[0];
|
||||
if (sel) setInput(`/${sel.cmd}`);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
// Outside slash mode: ↑/↓ recall prior prompts.
|
||||
const hist = promptHistory.current;
|
||||
if (key.upArrow) {
|
||||
if (hist.length === 0) return;
|
||||
const nextCursor = Math.min(historyCursor.current + 1, hist.length - 1);
|
||||
historyCursor.current = nextCursor;
|
||||
setInput(hist[hist.length - 1 - nextCursor] ?? "");
|
||||
return;
|
||||
}
|
||||
if (key.downArrow) {
|
||||
if (historyCursor.current < 0) return;
|
||||
const nextCursor = historyCursor.current - 1;
|
||||
historyCursor.current = nextCursor;
|
||||
setInput(nextCursor < 0 ? "" : (hist[hist.length - 1 - nextCursor] ?? ""));
|
||||
return;
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* Callback wired into the `/undo` slash command. Restores the files
|
||||
* last edit batch to their pre-edit state and reports per-file
|
||||
* results. Only available when running in code mode — the slash
|
||||
* handler gates on this callback's presence.
|
||||
*/
|
||||
const codeUndo = useCallback((): string => {
|
||||
if (!codeMode) return "not in code mode";
|
||||
const snaps = lastEditSnapshots.current;
|
||||
if (!snaps || snaps.length === 0) {
|
||||
return "nothing to undo — no recent edit batch to restore";
|
||||
}
|
||||
const results = restoreSnapshots(snaps, codeMode.rootDir);
|
||||
lastEditSnapshots.current = null;
|
||||
return formatUndoResults(results);
|
||||
}, [codeMode]);
|
||||
|
||||
/**
|
||||
* /apply callback — write pending edit blocks to disk, snapshot
|
||||
* beforehand so /undo still works, report per-file results.
|
||||
*/
|
||||
const codeApply = useCallback((): string => {
|
||||
if (!codeMode) return "not in code mode";
|
||||
const blocks = pendingEdits.current;
|
||||
if (blocks.length === 0) {
|
||||
return "nothing pending — the assistant hasn't proposed edits since the last /apply or /discard.";
|
||||
}
|
||||
const snaps = snapshotBeforeEdits(blocks, codeMode.rootDir);
|
||||
const results = applyEditBlocks(blocks, codeMode.rootDir);
|
||||
const anyApplied = results.some((r) => r.status === "applied" || r.status === "created");
|
||||
if (anyApplied) lastEditSnapshots.current = snaps;
|
||||
pendingEdits.current = [];
|
||||
return formatEditResults(results);
|
||||
}, [codeMode]);
|
||||
|
||||
/**
|
||||
* /discard callback — forget the pending edits without touching
|
||||
* disk. Keeps the conversation going without the user having to
|
||||
* argue the model out of its proposal.
|
||||
*/
|
||||
const codeDiscard = useCallback((): string => {
|
||||
const count = pendingEdits.current.length;
|
||||
if (count === 0) return "nothing pending to discard.";
|
||||
pendingEdits.current = [];
|
||||
return `▸ discarded ${count} pending edit block(s). Nothing was written to disk.`;
|
||||
}, []);
|
||||
|
||||
const prefixHash = loop.prefix.fingerprint;
|
||||
|
||||
const writeTranscript = useCallback(
|
||||
@@ -119,12 +324,52 @@ export function App({ model, system, transcript, harvest, branch, session }: App
|
||||
|
||||
const handleSubmit = useCallback(
|
||||
async (raw: string) => {
|
||||
const text = raw.trim();
|
||||
let text = raw.trim();
|
||||
if (!text || busy) return;
|
||||
|
||||
// Slash auto-complete on Enter. When the user typed a prefix
|
||||
// (e.g. "/he") and the suggestion list is visible, substitute
|
||||
// the highlighted match so Enter runs it — same effect as Tab
|
||||
// + Enter, one keystroke less. Skip substitution if the user
|
||||
// already typed a full, exact command name (respect verbatim
|
||||
// input when they know what they want).
|
||||
if (text.startsWith("/") && !text.includes(" ")) {
|
||||
const typed = text.slice(1).toLowerCase();
|
||||
const matches = suggestSlashCommands(typed, !!codeMode);
|
||||
const exact = matches.find((m) => m.cmd === typed);
|
||||
if (!exact && matches.length > 0) {
|
||||
const chosen = matches[slashSelected] ?? matches[0];
|
||||
if (chosen) text = `/${chosen.cmd}`;
|
||||
}
|
||||
}
|
||||
|
||||
setInput("");
|
||||
historyCursor.current = -1;
|
||||
|
||||
// Y/N fast-path when edits are pending. One keystroke is all it
|
||||
// takes to commit or drop — matches the muscle memory of `git
|
||||
// add -p` / most prompts. Deliberately scoped: only when there
|
||||
// ARE pending edits, so "y" as a normal message still works
|
||||
// when nothing's waiting.
|
||||
if (codeMode && pendingEdits.current.length > 0 && (text === "y" || text === "n")) {
|
||||
const out = text === "y" ? codeApply() : codeDiscard();
|
||||
setHistorical((prev) => [...prev, { id: `sys-${Date.now()}`, role: "info", text: out }]);
|
||||
promptHistory.current.push(text);
|
||||
return;
|
||||
}
|
||||
|
||||
const slash = parseSlash(text);
|
||||
if (slash) {
|
||||
const result = handleSlash(slash.cmd, slash.args, loop);
|
||||
const result = handleSlash(slash.cmd, slash.args, loop, {
|
||||
mcpSpecs,
|
||||
mcpServers,
|
||||
codeUndo: codeMode ? codeUndo : undefined,
|
||||
codeApply: codeMode ? codeApply : undefined,
|
||||
codeDiscard: codeMode ? codeDiscard : undefined,
|
||||
codeRoot: codeMode?.rootDir,
|
||||
pendingEditCount: codeMode ? pendingEdits.current.length : undefined,
|
||||
toolHistory: () => toolHistoryRef.current,
|
||||
});
|
||||
if (result.exit) {
|
||||
transcriptRef.current?.end();
|
||||
exit();
|
||||
@@ -144,10 +389,19 @@ export function App({ model, system, transcript, harvest, branch, session }: App
|
||||
},
|
||||
]);
|
||||
}
|
||||
return;
|
||||
// `/retry` (and anything else that requests a resubmit) falls
|
||||
// through to the normal user-message flow with the provided
|
||||
// text instead of returning.
|
||||
if (result.resubmit) {
|
||||
text = result.resubmit;
|
||||
} else {
|
||||
promptHistory.current.push(text);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
// User message is immutable — push to Static immediately.
|
||||
promptHistory.current.push(text);
|
||||
setHistorical((prev) => [...prev, { id: `u-${Date.now()}`, role: "user", text }]);
|
||||
|
||||
const assistantId = `a-${Date.now()}`;
|
||||
@@ -159,6 +413,7 @@ export function App({ model, system, transcript, harvest, branch, session }: App
|
||||
|
||||
setStreaming({ id: assistantId, role: "assistant", text: "", streaming: true });
|
||||
setBusy(true);
|
||||
abortedThisTurn.current = false;
|
||||
|
||||
const flush = () => {
|
||||
if (!contentBuf.current && !reasoningBuf.current) return;
|
||||
@@ -206,12 +461,13 @@ export function App({ model, system, transcript, harvest, branch, session }: App
|
||||
flush();
|
||||
const repairNote = ev.repair ? describeRepair(ev.repair) : "";
|
||||
setStreaming(null);
|
||||
const finalText = ev.content || streamRef.text;
|
||||
setHistorical((prev) => [
|
||||
...prev,
|
||||
{
|
||||
id: assistantId,
|
||||
role: "assistant",
|
||||
text: ev.content || streamRef.text,
|
||||
text: finalText,
|
||||
reasoning: streamRef.reasoning || undefined,
|
||||
planState: ev.planState,
|
||||
branch: ev.branch,
|
||||
@@ -220,8 +476,40 @@ export function App({ model, system, transcript, harvest, branch, session }: App
|
||||
streaming: false,
|
||||
},
|
||||
]);
|
||||
if (codeMode && finalText && !ev.forcedSummary) {
|
||||
// Parse SEARCH/REPLACE blocks but DO NOT write them to
|
||||
// disk. Store as pending — the user has to say /apply
|
||||
// explicitly. This prevents "analyze the project" from
|
||||
// silently drifting into "edit the project".
|
||||
//
|
||||
// `ev.forcedSummary` gates us out entirely: if the loop
|
||||
// had to force a summary (budget / aborted / context-
|
||||
// guard), its text is a wrap-up, not a plan to execute.
|
||||
// Blocks dropped in a forced summary are display-only.
|
||||
const blocks = parseEditBlocks(finalText);
|
||||
if (blocks.length > 0) {
|
||||
pendingEdits.current = blocks;
|
||||
setHistorical((prev) => [
|
||||
...prev,
|
||||
{
|
||||
id: `pending-${Date.now()}`,
|
||||
role: "info",
|
||||
text: formatPendingPreview(blocks),
|
||||
},
|
||||
]);
|
||||
}
|
||||
}
|
||||
} else if (ev.role === "tool_start") {
|
||||
// Kick off the visual indicator. Cleared when `tool`
|
||||
// (result) or `error` arrives, or on the finally below.
|
||||
setOngoingTool({ name: ev.toolName ?? "?", args: ev.toolArgs });
|
||||
} else if (ev.role === "tool") {
|
||||
flush();
|
||||
setOngoingTool(null);
|
||||
toolHistoryRef.current.push({
|
||||
toolName: ev.toolName ?? "?",
|
||||
text: ev.content,
|
||||
});
|
||||
setHistorical((prev) => [
|
||||
...prev,
|
||||
{
|
||||
@@ -236,17 +524,35 @@ export function App({ model, system, transcript, harvest, branch, session }: App
|
||||
...prev,
|
||||
{ id: `e-${Date.now()}`, role: "error", text: ev.error ?? ev.content },
|
||||
]);
|
||||
} else if (ev.role === "warning") {
|
||||
setHistorical((prev) => [
|
||||
...prev,
|
||||
{ id: `w-${Date.now()}-${Math.random()}`, role: "warning", text: ev.content },
|
||||
]);
|
||||
}
|
||||
}
|
||||
flush();
|
||||
} finally {
|
||||
clearInterval(timer);
|
||||
setStreaming(null);
|
||||
setOngoingTool(null);
|
||||
setSummary(loop.stats.summary());
|
||||
setBusy(false);
|
||||
}
|
||||
},
|
||||
[busy, exit, loop, writeTranscript],
|
||||
[
|
||||
busy,
|
||||
codeApply,
|
||||
codeDiscard,
|
||||
codeMode,
|
||||
codeUndo,
|
||||
exit,
|
||||
loop,
|
||||
mcpSpecs,
|
||||
mcpServers,
|
||||
slashSelected,
|
||||
writeTranscript,
|
||||
],
|
||||
);
|
||||
|
||||
return (
|
||||
@@ -264,23 +570,164 @@ export function App({ model, system, transcript, harvest, branch, session }: App
|
||||
<EventRow event={streaming} />
|
||||
</Box>
|
||||
) : null}
|
||||
{ongoingTool ? <OngoingToolRow tool={ongoingTool} /> : null}
|
||||
<PromptInput value={input} onChange={setInput} onSubmit={handleSubmit} disabled={busy} />
|
||||
<CommandStrip />
|
||||
<SlashSuggestions matches={slashMatches} selectedIndex={slashSelected} />
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
function CommandStrip() {
|
||||
/**
|
||||
* Live spinner row while a tool call is in flight. Without this the
|
||||
* window between the model's `tool_calls` decision and the tool's
|
||||
* result (often seconds for a multi-KB `filesystem_edit_file`) looks
|
||||
* like the app has frozen — the streaming assistant display is
|
||||
* already cleared and the input is disabled, so there's nothing to
|
||||
* look at.
|
||||
*
|
||||
* We show three signals: a braille spinner (liveness), an elapsed
|
||||
* timer in seconds (so "long" has a number attached), and a
|
||||
* per-tool summary of the most informative argument fields (path,
|
||||
* edits count, pattern, etc.). MCP doesn't stream progress today —
|
||||
* when it does, this component is where the progress notifications
|
||||
* would land.
|
||||
*/
|
||||
function OngoingToolRow({ tool }: { tool: { name: string; args?: string } }) {
|
||||
const [tick, setTick] = useState(0);
|
||||
const [elapsed, setElapsed] = useState(0);
|
||||
useEffect(() => {
|
||||
const start = Date.now();
|
||||
const frameId = setInterval(() => setTick((t) => t + 1), 120);
|
||||
const secId = setInterval(() => setElapsed(Math.floor((Date.now() - start) / 1000)), 1000);
|
||||
return () => {
|
||||
clearInterval(frameId);
|
||||
clearInterval(secId);
|
||||
};
|
||||
}, []);
|
||||
const frames = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"];
|
||||
const summary = summarizeToolArgs(tool.name, tool.args);
|
||||
return (
|
||||
<Box paddingX={2}>
|
||||
<Text dimColor>
|
||||
/help · /preset {"<fast|smart|max>"} · /sessions · /model · /harvest · /branch · /clear ·
|
||||
/exit
|
||||
</Text>
|
||||
<Box marginY={1} flexDirection="column">
|
||||
<Box>
|
||||
<Text color="cyan">{frames[tick % frames.length]}</Text>
|
||||
<Text color="yellow">{` tool<${tool.name}> running…`}</Text>
|
||||
<Text dimColor>{` ${elapsed}s`}</Text>
|
||||
</Box>
|
||||
{summary ? (
|
||||
<Box paddingLeft={2}>
|
||||
<Text dimColor>{summary}</Text>
|
||||
</Box>
|
||||
) : null}
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Turn raw JSON tool arguments into a one-line human summary. For
|
||||
* common filesystem MCP tools we pull the actually-useful fields; for
|
||||
* anything else we fall back to a truncated raw string so the user
|
||||
* still sees *something* beyond the tool name.
|
||||
*
|
||||
* Match on suffix (e.g. `_read_file`) rather than exact name because
|
||||
* `bridgeMcpTools({ namePrefix: "filesystem_" })` prepends the server
|
||||
* namespace — tools arrive as `filesystem_read_file` in practice but
|
||||
* callers might wire up anonymous too.
|
||||
*/
|
||||
function summarizeToolArgs(name: string, args?: string): string {
|
||||
if (!args || args === "{}") return "";
|
||||
let parsed: Record<string, unknown>;
|
||||
try {
|
||||
parsed = JSON.parse(args) as Record<string, unknown>;
|
||||
} catch {
|
||||
// Unparseable JSON — show a head slice so user at least sees
|
||||
// what the model tried.
|
||||
return args.length > 80 ? `${args.slice(0, 80)}…` : args;
|
||||
}
|
||||
const hasSuffix = (s: string) => name === s || name.endsWith(`_${s}`);
|
||||
const path = typeof parsed.path === "string" ? parsed.path : undefined;
|
||||
if (hasSuffix("read_file")) {
|
||||
const head = typeof parsed.head === "number" ? `, head=${parsed.head}` : "";
|
||||
const tail = typeof parsed.tail === "number" ? `, tail=${parsed.tail}` : "";
|
||||
return `path: ${path ?? "?"}${head}${tail}`;
|
||||
}
|
||||
if (hasSuffix("write_file")) {
|
||||
const content = typeof parsed.content === "string" ? parsed.content : "";
|
||||
return `path: ${path ?? "?"} (${content.length} chars)`;
|
||||
}
|
||||
if (hasSuffix("edit_file")) {
|
||||
const edits = Array.isArray(parsed.edits) ? parsed.edits.length : 0;
|
||||
return `path: ${path ?? "?"} (${edits} edit${edits === 1 ? "" : "s"})`;
|
||||
}
|
||||
if (hasSuffix("list_directory") || hasSuffix("directory_tree")) {
|
||||
return `path: ${path ?? "?"}`;
|
||||
}
|
||||
if (hasSuffix("search_files")) {
|
||||
const pattern = typeof parsed.pattern === "string" ? parsed.pattern : "?";
|
||||
return `path: ${path ?? "?"} · pattern: ${pattern}`;
|
||||
}
|
||||
if (hasSuffix("move_file")) {
|
||||
const src = typeof parsed.source === "string" ? parsed.source : "?";
|
||||
const dst = typeof parsed.destination === "string" ? parsed.destination : "?";
|
||||
return `${src} → ${dst}`;
|
||||
}
|
||||
if (hasSuffix("get_file_info")) {
|
||||
return `path: ${path ?? "?"}`;
|
||||
}
|
||||
return args.length > 80 ? `${args.slice(0, 80)}…` : args;
|
||||
}
|
||||
|
||||
/**
|
||||
* Render a batch of SEARCH/REPLACE application results as one
|
||||
* human-scannable info line per edit. Prefixes denote status so the
|
||||
* line reads well even without color (e.g. when piped to a log file
|
||||
* or stripped for screenshots):
|
||||
* ✓ applied src/foo.ts
|
||||
* ✓ created src/new.ts
|
||||
* ✗ not-found src/bar.ts (SEARCH text does not match…)
|
||||
*/
|
||||
function formatEditResults(results: ApplyResult[]): string {
|
||||
const lines = results.map((r) => {
|
||||
const mark = r.status === "applied" || r.status === "created" ? "✓" : "✗";
|
||||
const detail = r.message ? ` (${r.message})` : "";
|
||||
return ` ${mark} ${r.status.padEnd(11)} ${r.path}${detail}`;
|
||||
});
|
||||
const ok = results.filter((r) => r.status === "applied" || r.status === "created").length;
|
||||
const total = results.length;
|
||||
const header = `▸ edit blocks: ${ok}/${total} applied — /undo to roll back, or \`git diff\` to review`;
|
||||
return [header, ...lines].join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Pending-edits preview shown after each assistant turn that proposed
|
||||
* changes. We keep it deliberately thin — path + approximate
|
||||
* ±line-count — because a full diff would crowd the TUI and the user
|
||||
* can always run `git diff` after /apply.
|
||||
*/
|
||||
function formatPendingPreview(blocks: EditBlock[]): string {
|
||||
const lines = blocks.map((b) => {
|
||||
const removed = b.search === "" ? 0 : countLines(b.search);
|
||||
const added = countLines(b.replace);
|
||||
const tag = b.search === "" ? "NEW " : " ";
|
||||
return ` ${tag}${b.path} (-${removed} +${added} lines)`;
|
||||
});
|
||||
const header = `▸ ${blocks.length} pending edit block(s) — /apply (or y) to commit · /discard (or n) to drop`;
|
||||
return [header, ...lines].join("\n");
|
||||
}
|
||||
|
||||
function countLines(s: string): number {
|
||||
if (s.length === 0) return 0;
|
||||
return (s.match(/\n/g)?.length ?? 0) + 1;
|
||||
}
|
||||
|
||||
function formatUndoResults(results: ApplyResult[]): string {
|
||||
const lines = results.map((r) => {
|
||||
const mark = r.status === "applied" ? "✓" : "✗";
|
||||
const detail = r.message ? ` (${r.message})` : "";
|
||||
return ` ${mark} ${r.path}${detail}`;
|
||||
});
|
||||
return [`▸ undo: restored ${results.length} file(s) to pre-edit state`, ...lines].join("\n");
|
||||
}
|
||||
|
||||
function describeRepair(repair: {
|
||||
scavenged: number;
|
||||
truncationsFixed: number;
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
/**
|
||||
* Ink TUI for `reasonix diff`. Split-pane: A on the left, B on the right,
|
||||
* shared cursor. Header shows aggregate deltas; footer shows the current
|
||||
* pair's divergence note (if any) + key cheat sheet.
|
||||
*
|
||||
* j/k moves the cursor by one turn; n/N jumps to the next/prev divergent
|
||||
* turn — which is the whole point of a diff tool. Quit with q.
|
||||
*
|
||||
* Pure navigation lives in src/diff.ts (findNextDivergence / findPrevDivergence).
|
||||
*/
|
||||
|
||||
import { Box, Static, Text, useApp, useInput } from "ink";
|
||||
import React, { useState } from "react";
|
||||
import {
|
||||
type DiffReport,
|
||||
type TurnPair,
|
||||
findNextDivergence,
|
||||
findPrevDivergence,
|
||||
} from "../../diff.js";
|
||||
import { RecordView } from "./RecordView.js";
|
||||
|
||||
export interface DiffAppProps {
|
||||
report: DiffReport;
|
||||
}
|
||||
|
||||
export function DiffApp({ report }: DiffAppProps) {
|
||||
const { exit } = useApp();
|
||||
const maxIdx = Math.max(0, report.pairs.length - 1);
|
||||
// Start at the first divergence when one exists — that's the user's most
|
||||
// likely destination. Falls back to idx 0 for fully-matching diffs.
|
||||
const initialIdx = report.firstDivergenceTurn
|
||||
? report.pairs.findIndex((p) => p.turn === report.firstDivergenceTurn)
|
||||
: 0;
|
||||
const [idx, setIdx] = useState(Math.max(0, initialIdx));
|
||||
|
||||
useInput((input, key) => {
|
||||
if (input === "q" || (key.ctrl && input === "c")) {
|
||||
exit();
|
||||
return;
|
||||
}
|
||||
if (input === "j" || key.downArrow || input === " " || key.return) {
|
||||
setIdx((i) => Math.min(maxIdx, i + 1));
|
||||
} else if (input === "k" || key.upArrow) {
|
||||
setIdx((i) => Math.max(0, i - 1));
|
||||
} else if (input === "g") {
|
||||
setIdx(0);
|
||||
} else if (input === "G") {
|
||||
setIdx(maxIdx);
|
||||
} else if (input === "n") {
|
||||
const next = findNextDivergence(report.pairs, idx);
|
||||
if (next !== -1) setIdx(next);
|
||||
} else if (input === "N" || input === "p") {
|
||||
const prev = findPrevDivergence(report.pairs, idx);
|
||||
if (prev !== -1) setIdx(prev);
|
||||
}
|
||||
});
|
||||
|
||||
const pair = report.pairs[idx];
|
||||
|
||||
return (
|
||||
<Box flexDirection="column">
|
||||
<DiffHeader report={report} />
|
||||
|
||||
<Box marginTop={1} paddingX={1} justifyContent="space-between">
|
||||
<Text color="cyan" bold>
|
||||
turn {pair?.turn ?? "?"} ({idx + 1} / {report.pairs.length})
|
||||
</Text>
|
||||
<Text>{pair ? <KindBadge kind={pair.kind} /> : null}</Text>
|
||||
</Box>
|
||||
|
||||
<Box flexDirection="row" marginTop={1}>
|
||||
<Pane label={report.a.label} headerColor="blue" records={paneRecords(pair, "a")} />
|
||||
<Pane label={report.b.label} headerColor="magenta" records={paneRecords(pair, "b")} />
|
||||
</Box>
|
||||
|
||||
{pair?.divergenceNote ? (
|
||||
<Box marginTop={1} paddingX={1}>
|
||||
<Text color="yellow">★ </Text>
|
||||
<Text>{pair.divergenceNote}</Text>
|
||||
</Box>
|
||||
) : null}
|
||||
|
||||
<Box marginTop={1} paddingX={1} borderStyle="single" borderColor="gray">
|
||||
<Text dimColor>
|
||||
<Text bold>j</Text>/<Text bold>↓</Text> next · <Text bold>k</Text>/<Text bold>↑</Text>{" "}
|
||||
prev · <Text bold>n</Text> next-diverge · <Text bold>N</Text>/<Text bold>p</Text>{" "}
|
||||
prev-diverge · <Text bold>g</Text>/<Text bold>G</Text> first/last · <Text bold>q</Text>{" "}
|
||||
quit
|
||||
</Text>
|
||||
</Box>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
function DiffHeader({ report }: { report: DiffReport }) {
|
||||
const a = report.a;
|
||||
const b = report.b;
|
||||
|
||||
const cacheDelta = b.stats.cacheHitRatio - a.stats.cacheHitRatio;
|
||||
const costDelta =
|
||||
a.stats.totalCostUsd > 0
|
||||
? ((b.stats.totalCostUsd - a.stats.totalCostUsd) / a.stats.totalCostUsd) * 100
|
||||
: 0;
|
||||
|
||||
// Prefix stability one-liner (same logic as the stdout summary).
|
||||
const aStable = a.stats.prefixHashes.length <= 1;
|
||||
const bStable = b.stats.prefixHashes.length <= 1;
|
||||
let prefixLine: string | null = null;
|
||||
if (aStable !== bStable) {
|
||||
const stableLabel = aStable ? report.a.label : report.b.label;
|
||||
const churnLabel = aStable ? report.b.label : report.a.label;
|
||||
const churnCount = aStable ? b.stats.prefixHashes.length : a.stats.prefixHashes.length;
|
||||
prefixLine = `${stableLabel} stayed byte-stable; ${churnLabel} churned ${churnCount} distinct prefixes.`;
|
||||
} else if (a.stats.prefixHashes[0] && a.stats.prefixHashes[0] === b.stats.prefixHashes[0]) {
|
||||
prefixLine = `shared prefix hash ${a.stats.prefixHashes[0].slice(0, 12)}… — cache delta attributable to log stability, not prompt change.`;
|
||||
}
|
||||
|
||||
return (
|
||||
<Box flexDirection="column" borderStyle="round" borderColor="cyan" paddingX={1}>
|
||||
<Box justifyContent="space-between">
|
||||
<Text>
|
||||
<Text color="cyan" bold>
|
||||
reasonix diff
|
||||
</Text>
|
||||
<Text dimColor> · A=</Text>
|
||||
<Text color="blue">{a.label}</Text>
|
||||
<Text dimColor> vs B=</Text>
|
||||
<Text color="magenta">{b.label}</Text>
|
||||
</Text>
|
||||
<Text dimColor>{report.pairs.length} turns aligned</Text>
|
||||
</Box>
|
||||
|
||||
<Box marginTop={1} gap={3}>
|
||||
<Text>
|
||||
<Text dimColor>cache </Text>
|
||||
<Text>{(a.stats.cacheHitRatio * 100).toFixed(1)}%</Text>
|
||||
<Text dimColor> → </Text>
|
||||
<Text>{(b.stats.cacheHitRatio * 100).toFixed(1)}%</Text>
|
||||
<Text color={cacheDelta >= 0 ? "green" : "red"} bold>
|
||||
{" "}
|
||||
{cacheDelta >= 0 ? "+" : ""}
|
||||
{(cacheDelta * 100).toFixed(1)}pp
|
||||
</Text>
|
||||
</Text>
|
||||
<Text>
|
||||
<Text dimColor>cost </Text>
|
||||
<Text>${a.stats.totalCostUsd.toFixed(6)}</Text>
|
||||
<Text dimColor> → </Text>
|
||||
<Text>${b.stats.totalCostUsd.toFixed(6)}</Text>
|
||||
<Text color={costDelta <= 0 ? "green" : "red"} bold>
|
||||
{" "}
|
||||
{costDelta >= 0 ? "+" : ""}
|
||||
{costDelta.toFixed(1)}%
|
||||
</Text>
|
||||
</Text>
|
||||
<Text>
|
||||
<Text dimColor>model calls </Text>
|
||||
<Text>
|
||||
{a.stats.turns} → {b.stats.turns}
|
||||
</Text>
|
||||
</Text>
|
||||
</Box>
|
||||
|
||||
{prefixLine ? (
|
||||
<Box marginTop={1}>
|
||||
<Text dimColor italic>
|
||||
{prefixLine}
|
||||
</Text>
|
||||
</Box>
|
||||
) : null}
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
function Pane({
|
||||
label,
|
||||
headerColor,
|
||||
records,
|
||||
}: {
|
||||
label: string;
|
||||
headerColor: "blue" | "magenta";
|
||||
records: TurnPair["aTools"];
|
||||
}) {
|
||||
return (
|
||||
<Box
|
||||
flexDirection="column"
|
||||
flexGrow={1}
|
||||
paddingX={1}
|
||||
borderStyle="single"
|
||||
borderColor={headerColor}
|
||||
>
|
||||
<Text color={headerColor} bold>
|
||||
{label}
|
||||
</Text>
|
||||
{records.length === 0 ? (
|
||||
<Box marginTop={1}>
|
||||
<Text dimColor italic>
|
||||
(no records on this side for this turn)
|
||||
</Text>
|
||||
</Box>
|
||||
) : (
|
||||
<Static items={records.map((rec, i) => ({ key: `${label}-${i}`, rec }))}>
|
||||
{({ key, rec }) => <RecordView key={key} rec={rec} compact />}
|
||||
</Static>
|
||||
)}
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
function KindBadge({ kind }: { kind: TurnPair["kind"] }) {
|
||||
if (kind === "match") {
|
||||
return <Text color="green">✓ match</Text>;
|
||||
}
|
||||
if (kind === "diverge") {
|
||||
return <Text color="yellow">★ diverge</Text>;
|
||||
}
|
||||
if (kind === "only_in_a") {
|
||||
return <Text color="blue">← only in A</Text>;
|
||||
}
|
||||
return <Text color="magenta">→ only in B</Text>;
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------
|
||||
|
||||
function paneRecords(pair: TurnPair | undefined, side: "a" | "b"): TurnPair["aTools"] {
|
||||
if (!pair) return [];
|
||||
const tools = side === "a" ? pair.aTools : pair.bTools;
|
||||
const assistant = side === "a" ? pair.aAssistant : pair.bAssistant;
|
||||
const out: TurnPair["aTools"] = [...tools];
|
||||
if (assistant) out.push(assistant);
|
||||
return out;
|
||||
}
|
||||
+61
-26
@@ -3,9 +3,10 @@ import React, { useEffect, useState } from "react";
|
||||
import { type TypedPlanState, isPlanStateEmpty } from "../../harvest.js";
|
||||
import type { BranchProgress, BranchSummary } from "../../loop.js";
|
||||
import type { TurnStats } from "../../telemetry.js";
|
||||
import { PlanStateBlock } from "./PlanStateBlock.js";
|
||||
import { Markdown } from "./markdown.js";
|
||||
|
||||
export type DisplayRole = "user" | "assistant" | "tool" | "system" | "error" | "info";
|
||||
export type DisplayRole = "user" | "assistant" | "tool" | "system" | "error" | "info" | "warning";
|
||||
|
||||
export interface DisplayEvent {
|
||||
id: string;
|
||||
@@ -53,10 +54,21 @@ export const EventRow = React.memo(function EventRow({ event }: { event: Display
|
||||
);
|
||||
}
|
||||
if (event.role === "tool") {
|
||||
// `flattenMcpResult` prefixes server-side errors with "ERROR: ".
|
||||
// Render those in red with a ✗ marker so they don't blend into
|
||||
// successful tool output (yellow) — the failure mode is what the
|
||||
// model most likely needs to act on next, and the user needs to
|
||||
// see at a glance.
|
||||
const isError = event.text.startsWith("ERROR:");
|
||||
const color = isError ? "red" : "yellow";
|
||||
const marker = isError ? "✗" : "→";
|
||||
return (
|
||||
<Box flexDirection="column" marginTop={1}>
|
||||
<Text color="yellow">{`tool<${event.toolName ?? "?"}> →`}</Text>
|
||||
<Text dimColor> {truncate(event.text, 400)}</Text>
|
||||
<Text color={color}>{`tool<${event.toolName ?? "?"}> ${marker}`}</Text>
|
||||
<Text color={isError ? "red" : undefined} dimColor={!isError}>
|
||||
{" "}
|
||||
{truncate(event.text, 400)}
|
||||
</Text>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
@@ -77,6 +89,14 @@ export const EventRow = React.memo(function EventRow({ event }: { event: Display
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
if (event.role === "warning") {
|
||||
return (
|
||||
<Box>
|
||||
<Text color="yellow">▸ </Text>
|
||||
<Text color="yellow">{event.text}</Text>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
return (
|
||||
<Box>
|
||||
<Text>{event.text}</Text>
|
||||
@@ -104,30 +124,16 @@ function BranchBlock({ branch }: { branch: BranchSummary }) {
|
||||
);
|
||||
}
|
||||
|
||||
function PlanStateBlock({ planState }: { planState: TypedPlanState }) {
|
||||
const lines: Array<[string, string[]]> = [];
|
||||
if (planState.subgoals.length) lines.push(["subgoals", planState.subgoals]);
|
||||
if (planState.hypotheses.length) lines.push(["hypotheses", planState.hypotheses]);
|
||||
if (planState.uncertainties.length) lines.push(["uncertainties", planState.uncertainties]);
|
||||
if (planState.rejectedPaths.length) lines.push(["rejected", planState.rejectedPaths]);
|
||||
return (
|
||||
<Box flexDirection="column" marginBottom={1}>
|
||||
{lines.map(([label, items]) => (
|
||||
<Text key={label} color="magenta">
|
||||
{"‹ "}
|
||||
<Text bold>{label}</Text>
|
||||
{` (${items.length}): ${items.join(" · ")}`}
|
||||
</Text>
|
||||
))}
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
function ReasoningBlock({ reasoning }: { reasoning: string }) {
|
||||
const max = 220;
|
||||
const max = 260;
|
||||
const flat = reasoning.replace(/\s+/g, " ").trim();
|
||||
// Show the TAIL of the reasoning rather than the head. R1 opens
|
||||
// with generic scaffolding ("let me look at the structure...") that
|
||||
// repeats across turns and hides the part users actually want to
|
||||
// see — the decision right before the model commits to an action.
|
||||
// Users can dump the full reasoning with `/think` if needed.
|
||||
const preview =
|
||||
flat.length <= max ? flat : `${flat.slice(0, max)}… (+${flat.length - max} chars)`;
|
||||
flat.length <= max ? flat : `… (+${flat.length - max} earlier chars) ${flat.slice(-max)}`;
|
||||
return (
|
||||
<Box marginBottom={1}>
|
||||
<Text dimColor italic>
|
||||
@@ -202,14 +208,22 @@ function StreamingAssistant({ event }: { event: DisplayEvent }) {
|
||||
|
||||
const tail = lastLine(event.text, 140);
|
||||
const reasoningTail = event.reasoning ? lastLine(event.reasoning, 120) : "";
|
||||
// R1 ("deepseek-reasoner") generates reasoning_content first, then
|
||||
// content. While reasoning is streaming but content is still empty,
|
||||
// we were showing "(waiting for first token…)" — which looked like a
|
||||
// hang. The data is flowing, it's just landing in the thinking
|
||||
// channel. Reflect that honestly.
|
||||
const reasoningOnly = !event.text && !!event.reasoning;
|
||||
return (
|
||||
<Box flexDirection="column" marginTop={1}>
|
||||
<Box>
|
||||
<Text bold color="green">
|
||||
assistant{" "}
|
||||
</Text>
|
||||
<Pulse />
|
||||
<Text dimColor>
|
||||
(streaming · {event.text.length}
|
||||
{" "}
|
||||
({reasoningOnly ? "reasoning" : "streaming"} · {event.text.length}
|
||||
{event.reasoning ? ` + think ${event.reasoning.length}` : ""} chars){" "}
|
||||
</Text>
|
||||
<Elapsed />
|
||||
@@ -221,15 +235,36 @@ function StreamingAssistant({ event }: { event: DisplayEvent }) {
|
||||
) : null}
|
||||
{tail ? (
|
||||
<Text dimColor>▸ {tail}</Text>
|
||||
) : reasoningOnly ? (
|
||||
<Text color="yellow" dimColor>
|
||||
{
|
||||
" R1 is thinking before it speaks — body text starts when reasoning completes (typically 20-90s)."
|
||||
}
|
||||
</Text>
|
||||
) : (
|
||||
<Text dimColor italic>
|
||||
{" (waiting for first token…)"}
|
||||
{" (waiting for first byte — connection is open)"}
|
||||
</Text>
|
||||
)}
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Blinking indicator so the user can tell the stream is alive even
|
||||
* when the reasoner hasn't produced body text yet. Ticks every 500 ms
|
||||
* regardless of content flow — it's a heartbeat, not a progress bar.
|
||||
*/
|
||||
function Pulse() {
|
||||
const [tick, setTick] = useState(0);
|
||||
useEffect(() => {
|
||||
const id = setInterval(() => setTick((t) => t + 1), 500);
|
||||
return () => clearInterval(id);
|
||||
}, []);
|
||||
const frames = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"];
|
||||
return <Text color="cyan">{frames[tick % frames.length]}</Text>;
|
||||
}
|
||||
|
||||
function lastLine(s: string, maxChars: number): string {
|
||||
const flat = s.replace(/\s+/g, " ").trim();
|
||||
if (!flat) return "";
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
/**
|
||||
* Shared Ink block that renders a TypedPlanState. Used by the live chat
|
||||
* EventLog AND by the RecordView in replay/diff TUIs, so harvest output
|
||||
* looks identical live and on replay.
|
||||
*
|
||||
* Colors are semantic (not decorative):
|
||||
* - subgoals: cyan — structure / plan
|
||||
* - hypotheses: green — current beliefs (like assistant)
|
||||
* - uncertainties: yellow — attention required (like tool)
|
||||
* - rejected paths: red dim — ruled out (muted, like error-but-resolved)
|
||||
*
|
||||
* Only the label is colored + bold. The items themselves render in the
|
||||
* terminal's default foreground, so they stay readable on any background —
|
||||
* which is why the old single-magenta block was hard to see on dark themes.
|
||||
*/
|
||||
|
||||
import { Box, Text } from "ink";
|
||||
import React from "react";
|
||||
import type { TypedPlanState } from "../../harvest.js";
|
||||
|
||||
type FieldColor = "cyan" | "green" | "yellow" | "red";
|
||||
|
||||
export function PlanStateBlock({ planState }: { planState: TypedPlanState }) {
|
||||
const fields: Array<[string, string[], FieldColor, boolean]> = [];
|
||||
if (planState.subgoals.length) fields.push(["subgoals", planState.subgoals, "cyan", false]);
|
||||
if (planState.hypotheses.length)
|
||||
fields.push(["hypotheses", planState.hypotheses, "green", false]);
|
||||
if (planState.uncertainties.length)
|
||||
fields.push(["uncertainties", planState.uncertainties, "yellow", false]);
|
||||
if (planState.rejectedPaths.length)
|
||||
fields.push(["rejected", planState.rejectedPaths, "red", true]);
|
||||
if (fields.length === 0) return null;
|
||||
|
||||
return (
|
||||
<Box flexDirection="column" marginBottom={1}>
|
||||
{fields.map(([label, items, color, dim]) => (
|
||||
<Text key={label}>
|
||||
<Text color={color} bold dimColor={dim}>
|
||||
{"‹ "}
|
||||
{label}
|
||||
</Text>
|
||||
<Text dimColor>{` (${items.length})`}</Text>
|
||||
<Text>{`: ${items.join(" · ")}`}</Text>
|
||||
</Text>
|
||||
))}
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
/**
|
||||
* Shared renderer for a single TranscriptRecord. Used by ReplayApp and
|
||||
* DiffApp — both need the same visual grammar (user cyan, assistant green,
|
||||
* tool yellow, error red, cache badge colored by threshold) so transcripts
|
||||
* look consistent wherever they're displayed.
|
||||
*
|
||||
* Kept small on purpose: no streaming/branch/planState paths (those are
|
||||
* live-chat concerns and never appear in replayed transcripts).
|
||||
*/
|
||||
|
||||
import { Box, Text } from "ink";
|
||||
import React from "react";
|
||||
import type { TranscriptRecord } from "../../transcript.js";
|
||||
import { PlanStateBlock } from "./PlanStateBlock.js";
|
||||
|
||||
export interface RecordViewProps {
|
||||
rec: TranscriptRecord;
|
||||
/**
|
||||
* When rendering side-by-side in diff mode, shorter truncation limits
|
||||
* keep long tool results from dominating the pane. Passes through
|
||||
* untouched when undefined.
|
||||
*/
|
||||
compact?: boolean;
|
||||
}
|
||||
|
||||
export function RecordView({ rec, compact = false }: RecordViewProps) {
|
||||
const toolArgsMax = compact ? 120 : 200;
|
||||
const toolContentMax = compact ? 200 : 400;
|
||||
|
||||
if (rec.role === "user") {
|
||||
return (
|
||||
<Box marginTop={1}>
|
||||
<Text bold color="cyan">
|
||||
you ›{" "}
|
||||
</Text>
|
||||
<Text>{rec.content}</Text>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
if (rec.role === "assistant_final") {
|
||||
return (
|
||||
<Box flexDirection="column" marginTop={1}>
|
||||
<Box>
|
||||
<Text bold color="green">
|
||||
assistant
|
||||
</Text>
|
||||
{rec.cost !== undefined ? (
|
||||
<Text dimColor>
|
||||
{" $"}
|
||||
{rec.cost.toFixed(6)}
|
||||
</Text>
|
||||
) : null}
|
||||
{rec.usage ? <CacheBadge usage={rec.usage} /> : null}
|
||||
</Box>
|
||||
{rec.planState ? <PlanStateBlock planState={rec.planState} /> : null}
|
||||
{rec.content ? (
|
||||
<Text>{rec.content}</Text>
|
||||
) : (
|
||||
<Text dimColor italic>
|
||||
(tool-call response only)
|
||||
</Text>
|
||||
)}
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
if (rec.role === "tool") {
|
||||
return (
|
||||
<Box flexDirection="column" marginTop={1}>
|
||||
<Text color="yellow">
|
||||
{"tool<"}
|
||||
{rec.tool ?? "?"}
|
||||
{">"}
|
||||
</Text>
|
||||
{rec.args ? (
|
||||
<Text dimColor>
|
||||
{" args: "}
|
||||
{truncate(rec.args, toolArgsMax)}
|
||||
</Text>
|
||||
) : null}
|
||||
<Text dimColor>
|
||||
{" → "}
|
||||
{truncate(rec.content, toolContentMax)}
|
||||
</Text>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
if (rec.role === "error") {
|
||||
return (
|
||||
<Box marginTop={1}>
|
||||
<Text color="red" bold>
|
||||
error{" "}
|
||||
</Text>
|
||||
<Text color="red">{rec.error ?? rec.content}</Text>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
if (rec.role === "done" || rec.role === "assistant_delta") {
|
||||
// Noise in replay; skip.
|
||||
return null;
|
||||
}
|
||||
return (
|
||||
<Box>
|
||||
<Text dimColor>
|
||||
[{rec.role}] {rec.content}
|
||||
</Text>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
function CacheBadge({ usage }: { usage: NonNullable<TranscriptRecord["usage"]> }) {
|
||||
const hit = usage.prompt_cache_hit_tokens ?? 0;
|
||||
const miss = usage.prompt_cache_miss_tokens ?? 0;
|
||||
const total = hit + miss;
|
||||
if (total === 0) return null;
|
||||
const pct = (hit / total) * 100;
|
||||
const color = pct >= 70 ? "green" : pct >= 40 ? "yellow" : "red";
|
||||
return (
|
||||
<Text>
|
||||
<Text dimColor>{" · cache "}</Text>
|
||||
<Text color={color}>{pct.toFixed(1)}%</Text>
|
||||
</Text>
|
||||
);
|
||||
}
|
||||
|
||||
function truncate(s: string, max: number): string {
|
||||
return s.length <= max ? s : `${s.slice(0, max)}… (+${s.length - max} chars)`;
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
/**
|
||||
* Ink TUI for `reasonix replay`. Read-only: no input box, no loop.
|
||||
* j/k navigation across turn-pages, cumulative stats sidebar updates
|
||||
* as you move through time.
|
||||
*
|
||||
* The navigation logic (grouping records into pages, computing cumulative
|
||||
* stats) lives in src/replay.ts as pure functions; this file is just
|
||||
* presentation + key bindings.
|
||||
*/
|
||||
|
||||
import { Box, Static, Text, useApp, useInput } from "ink";
|
||||
import React, { useMemo, useState } from "react";
|
||||
import { type TurnPage, computeCumulativeStats } from "../../replay.js";
|
||||
import type { TranscriptMeta } from "../../transcript.js";
|
||||
import { RecordView } from "./RecordView.js";
|
||||
import { StatsPanel } from "./StatsPanel.js";
|
||||
|
||||
export interface ReplayAppProps {
|
||||
meta: TranscriptMeta | null;
|
||||
pages: TurnPage[];
|
||||
}
|
||||
|
||||
export function ReplayApp({ meta, pages }: ReplayAppProps) {
|
||||
const { exit } = useApp();
|
||||
const maxIdx = Math.max(0, pages.length - 1);
|
||||
// Start at the last page — more useful than "start from the beginning"
|
||||
// in practice: users mostly want to see the summary + last turn first.
|
||||
const [idx, setIdx] = useState(maxIdx);
|
||||
|
||||
useInput((input, key) => {
|
||||
if (input === "q" || (key.ctrl && input === "c")) {
|
||||
exit();
|
||||
return;
|
||||
}
|
||||
if (input === "j" || key.downArrow || input === " " || key.return) {
|
||||
setIdx((i) => Math.min(maxIdx, i + 1));
|
||||
} else if (input === "k" || key.upArrow) {
|
||||
setIdx((i) => Math.max(0, i - 1));
|
||||
} else if (input === "g") {
|
||||
setIdx(0);
|
||||
} else if (input === "G") {
|
||||
setIdx(maxIdx);
|
||||
} else if (input === "h" || key.leftArrow) {
|
||||
setIdx(0);
|
||||
} else if (input === "l" || key.rightArrow) {
|
||||
setIdx(maxIdx);
|
||||
}
|
||||
});
|
||||
|
||||
const cumStats = useMemo(() => computeCumulativeStats(pages, idx), [pages, idx]);
|
||||
|
||||
const summary = {
|
||||
turns: cumStats.turns,
|
||||
totalCostUsd: cumStats.totalCostUsd,
|
||||
claudeEquivalentUsd: cumStats.claudeEquivalentUsd,
|
||||
savingsVsClaudePct: cumStats.savingsVsClaudePct,
|
||||
cacheHitRatio: cumStats.cacheHitRatio,
|
||||
// Replay is read-only — no live last-turn prompt tokens to show.
|
||||
lastPromptTokens: 0,
|
||||
};
|
||||
|
||||
const prefixHash =
|
||||
cumStats.prefixHashes.length === 1
|
||||
? cumStats.prefixHashes[0]!.slice(0, 16)
|
||||
: cumStats.prefixHashes.length === 0
|
||||
? "(untracked)"
|
||||
: `(churned ×${cumStats.prefixHashes.length})`;
|
||||
|
||||
const currentPage = pages[idx];
|
||||
const progressLabel =
|
||||
pages.length === 0 ? "empty transcript" : `turn ${idx + 1} / ${pages.length}`;
|
||||
|
||||
return (
|
||||
<Box flexDirection="column">
|
||||
<StatsPanel
|
||||
summary={summary}
|
||||
model={cumStats.models[0] ?? meta?.model ?? "?"}
|
||||
prefixHash={prefixHash}
|
||||
/>
|
||||
|
||||
<Box flexDirection="column" marginTop={1} paddingX={1}>
|
||||
<Box justifyContent="space-between">
|
||||
<Text color="cyan" bold>
|
||||
{progressLabel}
|
||||
</Text>
|
||||
{meta ? (
|
||||
<Text dimColor>
|
||||
{meta.source}
|
||||
{meta.task ? ` · ${meta.task}` : ""}
|
||||
{meta.mode ? ` · ${meta.mode}` : ""}
|
||||
</Text>
|
||||
) : null}
|
||||
</Box>
|
||||
|
||||
{currentPage ? (
|
||||
<Static items={currentPage.records.map((rec, i) => ({ key: `${idx}-${i}`, rec }))}>
|
||||
{({ key, rec }) => <RecordView key={key} rec={rec} />}
|
||||
</Static>
|
||||
) : (
|
||||
<Text dimColor italic>
|
||||
no records
|
||||
</Text>
|
||||
)}
|
||||
</Box>
|
||||
|
||||
<Box marginTop={1} paddingX={1} borderStyle="single" borderColor="gray">
|
||||
<Text dimColor>
|
||||
<Text bold>j</Text>/<Text bold>↓</Text>/<Text bold>space</Text> next · <Text bold>k</Text>
|
||||
/<Text bold>↑</Text> prev · <Text bold>g</Text> first · <Text bold>G</Text> last ·{" "}
|
||||
<Text bold>q</Text> quit
|
||||
</Text>
|
||||
</Box>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,182 @@
|
||||
/**
|
||||
* Minimal arrow-key list components for Ink — single-select and
|
||||
* multi-select. No external deps beyond Ink's `useInput`.
|
||||
*
|
||||
* Why hand-roll: `ink-select-input` exists, but it defaults to
|
||||
* Enter-only interaction (no space-to-toggle for multi-select), doesn't
|
||||
* expose the "hint / footer" slot we want under each item, and would be
|
||||
* another dep for ~60 lines of UI code. Reasonix already has one React
|
||||
* rendering quirk bundled (`ink-text-input`); adding more is low value.
|
||||
*/
|
||||
|
||||
import { Box, Text, useInput } from "ink";
|
||||
import React, { useState } from "react";
|
||||
|
||||
export interface SelectItem<V extends string = string> {
|
||||
/** Stable identifier — returned to caller on submit. */
|
||||
value: V;
|
||||
/** First-row label. */
|
||||
label: string;
|
||||
/** Optional second row rendered dimmed. */
|
||||
hint?: string;
|
||||
/** If true, item is not selectable (rendered dimmed, skipped on nav). */
|
||||
disabled?: boolean;
|
||||
}
|
||||
|
||||
export interface SingleSelectProps<V extends string> {
|
||||
items: SelectItem<V>[];
|
||||
initialValue?: V;
|
||||
onSubmit: (value: V) => void;
|
||||
onCancel?: () => void;
|
||||
}
|
||||
|
||||
export function SingleSelect<V extends string>({
|
||||
items,
|
||||
initialValue,
|
||||
onSubmit,
|
||||
onCancel,
|
||||
}: SingleSelectProps<V>) {
|
||||
const initialIndex = Math.max(
|
||||
0,
|
||||
items.findIndex((i) => i.value === initialValue && !i.disabled),
|
||||
);
|
||||
const [index, setIndex] = useState(initialIndex === -1 ? 0 : initialIndex);
|
||||
|
||||
useInput((_input, key) => {
|
||||
if (key.upArrow) {
|
||||
setIndex((i) => findNextEnabled(items, i, -1));
|
||||
} else if (key.downArrow) {
|
||||
setIndex((i) => findNextEnabled(items, i, +1));
|
||||
} else if (key.return) {
|
||||
const chosen = items[index];
|
||||
if (chosen && !chosen.disabled) onSubmit(chosen.value);
|
||||
} else if (key.escape && onCancel) {
|
||||
onCancel();
|
||||
}
|
||||
});
|
||||
|
||||
return (
|
||||
<Box flexDirection="column">
|
||||
{items.map((item, i) => (
|
||||
<SelectRow
|
||||
key={item.value}
|
||||
item={item}
|
||||
active={i === index}
|
||||
marker={i === index ? "▸" : " "}
|
||||
/>
|
||||
))}
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
export interface MultiSelectProps<V extends string> {
|
||||
items: SelectItem<V>[];
|
||||
initialSelected?: V[];
|
||||
onSubmit: (values: V[]) => void;
|
||||
onCancel?: () => void;
|
||||
/** Footer hint under the list — e.g. "space toggle · enter confirm". */
|
||||
footer?: string;
|
||||
}
|
||||
|
||||
export function MultiSelect<V extends string>({
|
||||
items,
|
||||
initialSelected = [],
|
||||
onSubmit,
|
||||
onCancel,
|
||||
footer,
|
||||
}: MultiSelectProps<V>) {
|
||||
const [index, setIndex] = useState(() => {
|
||||
const first = items.findIndex((i) => !i.disabled);
|
||||
return first === -1 ? 0 : first;
|
||||
});
|
||||
const [selected, setSelected] = useState<Set<V>>(new Set(initialSelected));
|
||||
|
||||
useInput((input, key) => {
|
||||
if (key.upArrow) {
|
||||
setIndex((i) => findNextEnabled(items, i, -1));
|
||||
} else if (key.downArrow) {
|
||||
setIndex((i) => findNextEnabled(items, i, +1));
|
||||
} else if (input === " ") {
|
||||
const item = items[index];
|
||||
if (!item || item.disabled) return;
|
||||
setSelected((prev) => {
|
||||
const next = new Set(prev);
|
||||
if (next.has(item.value)) next.delete(item.value);
|
||||
else next.add(item.value);
|
||||
return next;
|
||||
});
|
||||
} else if (key.return) {
|
||||
// Preserve catalog order rather than insertion order, so reruns
|
||||
// produce the same spec list for the same checkbox set — makes the
|
||||
// `config.json` diff trivially stable.
|
||||
const ordered = items.filter((i) => selected.has(i.value)).map((i) => i.value);
|
||||
onSubmit(ordered);
|
||||
} else if (key.escape && onCancel) {
|
||||
onCancel();
|
||||
}
|
||||
});
|
||||
|
||||
return (
|
||||
<Box flexDirection="column">
|
||||
{items.map((item, i) => {
|
||||
const checked = selected.has(item.value);
|
||||
const marker = checked ? "[x]" : "[ ]";
|
||||
return (
|
||||
<SelectRow
|
||||
key={item.value}
|
||||
item={item}
|
||||
active={i === index}
|
||||
marker={`${i === index ? "▸" : " "} ${marker}`}
|
||||
/>
|
||||
);
|
||||
})}
|
||||
{footer ? (
|
||||
<Box marginTop={1}>
|
||||
<Text dimColor>{footer}</Text>
|
||||
</Box>
|
||||
) : null}
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
// ---------- internals ----------
|
||||
|
||||
function SelectRow<V extends string>({
|
||||
item,
|
||||
active,
|
||||
marker,
|
||||
}: {
|
||||
item: SelectItem<V>;
|
||||
active: boolean;
|
||||
marker: string;
|
||||
}) {
|
||||
const color = item.disabled ? "gray" : active ? "cyan" : undefined;
|
||||
return (
|
||||
<Box flexDirection="column">
|
||||
<Box>
|
||||
<Text color={color}>
|
||||
{marker} {item.label}
|
||||
</Text>
|
||||
</Box>
|
||||
{item.hint ? (
|
||||
<Box paddingLeft={marker.length + 1}>
|
||||
<Text dimColor>{item.hint}</Text>
|
||||
</Box>
|
||||
) : null}
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
function findNextEnabled<V extends string>(
|
||||
items: SelectItem<V>[],
|
||||
from: number,
|
||||
step: -1 | 1,
|
||||
): number {
|
||||
if (items.length === 0) return 0;
|
||||
let i = from;
|
||||
for (let tries = 0; tries < items.length; tries++) {
|
||||
i = (i + step + items.length) % items.length;
|
||||
if (!items[i]?.disabled) return i;
|
||||
}
|
||||
return from;
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
import { Box, Text } from "ink";
|
||||
// biome-ignore lint/style/useImportType: tsconfig.jsx = "react" needs React in value scope for JSX compilation
|
||||
import React from "react";
|
||||
import type { SlashCommandSpec } from "./slash.js";
|
||||
|
||||
export interface SlashSuggestionsProps {
|
||||
/**
|
||||
* Current matching suggestions, computed by the parent. `null` means
|
||||
* "not in slash-prefix mode" — render nothing. Empty array means "in
|
||||
* slash mode but no matches" — render the "no matches" hint.
|
||||
*/
|
||||
matches: SlashCommandSpec[] | null;
|
||||
/** Index (within `matches`) of the currently highlighted row. */
|
||||
selectedIndex: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Floating slash-command panel. Rendered below the input box when
|
||||
* the user is typing a `/…` prefix. Navigation state lives in the
|
||||
* parent (App.tsx owns `slashSelected`) so ↑/↓/Tab/Enter stay
|
||||
* consistent with the useInput handler. This component is pure
|
||||
* display: `matches` and `selectedIndex` come in, rows go out.
|
||||
*/
|
||||
export function SlashSuggestions({
|
||||
matches,
|
||||
selectedIndex,
|
||||
}: SlashSuggestionsProps): React.ReactElement | null {
|
||||
if (matches === null) return null;
|
||||
if (matches.length === 0) {
|
||||
return (
|
||||
<Box paddingX={1}>
|
||||
<Text color="yellow">no slash command matches that prefix</Text>
|
||||
<Text dimColor> — Backspace to edit, or /help for the full list</Text>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
// Limit rows so the suggestion list never dwarfs the rest of the
|
||||
// UI. Keep the currently-selected row in view by sliding the
|
||||
// window when the selection is near either edge.
|
||||
const MAX = 8;
|
||||
const total = matches.length;
|
||||
const windowStart =
|
||||
total <= MAX ? 0 : Math.max(0, Math.min(selectedIndex - Math.floor(MAX / 2), total - MAX));
|
||||
const shown = matches.slice(windowStart, windowStart + MAX);
|
||||
const hiddenAbove = windowStart;
|
||||
const hiddenBelow = total - windowStart - shown.length;
|
||||
return (
|
||||
<Box flexDirection="column" paddingX={1}>
|
||||
{hiddenAbove > 0 ? <Text dimColor> ↑ {hiddenAbove} more above</Text> : null}
|
||||
{shown.map((spec, i) => (
|
||||
<SuggestionRow key={spec.cmd} spec={spec} isSelected={windowStart + i === selectedIndex} />
|
||||
))}
|
||||
{hiddenBelow > 0 ? <Text dimColor> ↓ {hiddenBelow} more below</Text> : null}
|
||||
<Text dimColor> ↑/↓ navigate · Tab or Enter to pick</Text>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
function SuggestionRow({ spec, isSelected }: { spec: SlashCommandSpec; isSelected: boolean }) {
|
||||
const marker = isSelected ? "▸" : " ";
|
||||
const name = `/${spec.cmd}`;
|
||||
const argsSuffix = spec.argsHint ? ` ${spec.argsHint}` : "";
|
||||
// Selected row gets full cyan coloring so it pops out against the
|
||||
// dimmed others — mirrors the style GitHub Issues / Discord use
|
||||
// for focused dropdown rows.
|
||||
if (isSelected) {
|
||||
return (
|
||||
<Box>
|
||||
<Text bold color="cyan">
|
||||
{marker} {name.padEnd(12)}
|
||||
{argsSuffix.padEnd(16)}
|
||||
</Text>
|
||||
<Text color="cyan"> {spec.summary}</Text>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
return (
|
||||
<Box>
|
||||
<Text dimColor>
|
||||
{marker} {name.padEnd(12)}
|
||||
{argsSuffix.padEnd(16)} {spec.summary}
|
||||
</Text>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
import { Box, Text } from "ink";
|
||||
import React from "react";
|
||||
import { DEEPSEEK_CONTEXT_TOKENS, DEFAULT_CONTEXT_TOKENS } from "../../telemetry.js";
|
||||
import type { SessionSummary } from "../../telemetry.js";
|
||||
|
||||
export interface StatsPanelProps {
|
||||
@@ -21,6 +22,11 @@ export function StatsPanel({
|
||||
const hitColor =
|
||||
summary.cacheHitRatio >= 0.7 ? "green" : summary.cacheHitRatio >= 0.4 ? "yellow" : "red";
|
||||
const branchOn = (branchBudget ?? 1) > 1;
|
||||
|
||||
const ctxMax = DEEPSEEK_CONTEXT_TOKENS[model] ?? DEFAULT_CONTEXT_TOKENS;
|
||||
const ctxRatio = summary.lastPromptTokens / ctxMax;
|
||||
const ctxColor = ctxRatio >= 0.8 ? "red" : ctxRatio >= 0.5 ? "yellow" : undefined;
|
||||
|
||||
return (
|
||||
<Box borderStyle="round" borderColor="cyan" flexDirection="column" paddingX={1}>
|
||||
<Box justifyContent="space-between">
|
||||
@@ -58,7 +64,33 @@ export function StatsPanel({
|
||||
{summary.savingsVsClaudePct.toFixed(1)}%
|
||||
</Text>
|
||||
</Text>
|
||||
{summary.lastPromptTokens > 0 ? (
|
||||
<Text>
|
||||
<Text dimColor>ctx </Text>
|
||||
<Text color={ctxColor} bold={ctxColor !== undefined}>
|
||||
{formatTokens(summary.lastPromptTokens)}/{formatTokens(ctxMax)}
|
||||
</Text>
|
||||
<Text dimColor> ({(ctxRatio * 100).toFixed(0)}%)</Text>
|
||||
{ctxRatio >= 0.8 ? (
|
||||
<Text color="red" bold>
|
||||
{" "}
|
||||
· /compact
|
||||
</Text>
|
||||
) : null}
|
||||
</Text>
|
||||
) : null}
|
||||
</Box>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Compact integer formatter: 1234 → "1.2k", 131072 → "131k". Keeps the
|
||||
* panel narrow enough to fit on 80-col terminals even when the context
|
||||
* is near full.
|
||||
*/
|
||||
function formatTokens(n: number): string {
|
||||
if (n < 1000) return String(n);
|
||||
const k = n / 1000;
|
||||
return k >= 100 ? `${k.toFixed(0)}k` : `${k.toFixed(1)}k`;
|
||||
}
|
||||
@@ -0,0 +1,447 @@
|
||||
/**
|
||||
* First-run / re-configure wizard.
|
||||
*
|
||||
* Walks a new user through: API key → preset pick → MCP server pick →
|
||||
* per-server args → save. Saved output lives in `~/.reasonix/config.json`
|
||||
* so the next `reasonix chat` starts with everything already wired.
|
||||
*
|
||||
* The wizard is the antidote to "too many CLI flags" — a new user should
|
||||
* never have to read `--help` to get MCP + a sensible model combo
|
||||
* working. Everything a user could set via `--mcp`, `--harvest`,
|
||||
* `--branch`, etc. can be picked here in a few arrow-key presses.
|
||||
*/
|
||||
|
||||
import { Box, Text, useApp, useInput } from "ink";
|
||||
import TextInput from "ink-text-input";
|
||||
// biome-ignore lint/style/useImportType: JSX (jsx: "react") needs React as a value at runtime
|
||||
import React, { useState } from "react";
|
||||
import {
|
||||
type PresetName,
|
||||
type ReasonixConfig,
|
||||
defaultConfigPath,
|
||||
isPlausibleKey,
|
||||
readConfig,
|
||||
redactKey,
|
||||
writeConfig,
|
||||
} from "../../config.js";
|
||||
import { type CatalogEntry, MCP_CATALOG } from "../../mcp/catalog.js";
|
||||
import { MultiSelect, type SelectItem, SingleSelect } from "./Select.js";
|
||||
import { PRESET_DESCRIPTIONS } from "./presets.js";
|
||||
|
||||
export interface WizardProps {
|
||||
/** Called once the config has been saved. */
|
||||
onComplete: (cfg: ReasonixConfig) => void;
|
||||
/** Called if the user presses Esc to abort. */
|
||||
onCancel?: () => void;
|
||||
/** Skip the API-key step if a key already exists (env or config). */
|
||||
existingApiKey?: string;
|
||||
/** Pre-fill selections when re-running (reconfigure flow). */
|
||||
initial?: {
|
||||
preset?: PresetName;
|
||||
mcp?: string[];
|
||||
};
|
||||
}
|
||||
|
||||
type Step = "apiKey" | "preset" | "mcp" | "mcpArgs" | "review" | "saved";
|
||||
|
||||
interface WizardData {
|
||||
apiKey: string;
|
||||
preset: PresetName;
|
||||
selectedCatalog: string[]; // entries from MCP_CATALOG by `name`
|
||||
/** Captured user inputs per catalog entry that has `userArgs` (e.g. fs dir). */
|
||||
catalogArgs: Record<string, string>;
|
||||
}
|
||||
|
||||
const CATALOG_BY_NAME = new Map(MCP_CATALOG.map((e) => [e.name, e]));
|
||||
|
||||
export function Wizard({ onComplete, onCancel, existingApiKey, initial }: WizardProps) {
|
||||
const { exit } = useApp();
|
||||
const [step, setStep] = useState<Step>(existingApiKey ? "preset" : "apiKey");
|
||||
const [data, setData] = useState<WizardData>({
|
||||
apiKey: existingApiKey ?? "",
|
||||
preset: initial?.preset ?? "fast",
|
||||
selectedCatalog: deriveInitialCatalog(initial?.mcp ?? []),
|
||||
catalogArgs: {},
|
||||
});
|
||||
const [error, setError] = useState<string | null>(null);
|
||||
|
||||
// Global Esc → cancel. Disabled once we've started saving to avoid
|
||||
// ejecting out of a half-written state.
|
||||
useInput((_input, key) => {
|
||||
if (key.escape && step !== "saved" && onCancel) onCancel();
|
||||
});
|
||||
|
||||
if (step === "apiKey") {
|
||||
return (
|
||||
<ApiKeyStep
|
||||
onSubmit={(key) => {
|
||||
setData((d) => ({ ...d, apiKey: key }));
|
||||
setError(null);
|
||||
setStep("preset");
|
||||
}}
|
||||
error={error}
|
||||
onError={setError}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
if (step === "preset") {
|
||||
return (
|
||||
<StepFrame title="Pick a preset" step={1} total={3}>
|
||||
<SingleSelect<PresetName>
|
||||
items={presetItems()}
|
||||
initialValue={data.preset}
|
||||
onSubmit={(preset) => {
|
||||
setData((d) => ({ ...d, preset }));
|
||||
setStep("mcp");
|
||||
}}
|
||||
/>
|
||||
<Box marginTop={1}>
|
||||
<Text dimColor>↑/↓ move · enter confirm · esc cancel</Text>
|
||||
</Box>
|
||||
</StepFrame>
|
||||
);
|
||||
}
|
||||
|
||||
if (step === "mcp") {
|
||||
return (
|
||||
<StepFrame title="Which MCP servers should Reasonix wire up for you?" step={2} total={3}>
|
||||
<MultiSelect
|
||||
items={mcpItems()}
|
||||
initialSelected={data.selectedCatalog}
|
||||
onSubmit={(selected) => {
|
||||
setData((d) => ({ ...d, selectedCatalog: selected }));
|
||||
// Only advance to the args step if any selected entry needs args.
|
||||
const needsArgs = selected.some((name) => CATALOG_BY_NAME.get(name)?.userArgs);
|
||||
setStep(needsArgs ? "mcpArgs" : "review");
|
||||
}}
|
||||
footer="↑/↓ move · space toggle · enter confirm · esc cancel · leave empty to skip"
|
||||
/>
|
||||
</StepFrame>
|
||||
);
|
||||
}
|
||||
|
||||
if (step === "mcpArgs") {
|
||||
const pending = data.selectedCatalog.filter((name) => {
|
||||
const entry = CATALOG_BY_NAME.get(name);
|
||||
return entry?.userArgs && !data.catalogArgs[name];
|
||||
});
|
||||
if (pending.length === 0) {
|
||||
setStep("review");
|
||||
return null;
|
||||
}
|
||||
const currentName = pending[0]!;
|
||||
const entry = CATALOG_BY_NAME.get(currentName)!;
|
||||
return (
|
||||
<McpArgsStep
|
||||
entry={entry}
|
||||
error={error}
|
||||
onSubmit={(value) => {
|
||||
setData((d) => ({
|
||||
...d,
|
||||
catalogArgs: { ...d.catalogArgs, [currentName]: value },
|
||||
}));
|
||||
setError(null);
|
||||
}}
|
||||
onError={setError}
|
||||
/>
|
||||
);
|
||||
}
|
||||
|
||||
if (step === "review") {
|
||||
const specs = data.selectedCatalog.map((name) => buildSpec(name, data.catalogArgs));
|
||||
return (
|
||||
<StepFrame title="Ready to save" step={3} total={3}>
|
||||
<Box flexDirection="column">
|
||||
<SummaryLine label="API key" value={redactKey(data.apiKey)} />
|
||||
<SummaryLine label="Preset" value={data.preset} />
|
||||
<SummaryLine
|
||||
label="MCP"
|
||||
value={specs.length === 0 ? "(none)" : `${specs.length} server(s)`}
|
||||
/>
|
||||
{specs.map((spec, i) => (
|
||||
// biome-ignore lint/suspicious/noArrayIndexKey: review-only render, order fixed
|
||||
<Box key={i} paddingLeft={14}>
|
||||
<Text dimColor>· {spec}</Text>
|
||||
</Box>
|
||||
))}
|
||||
<Box marginTop={1}>
|
||||
<Text>Saves to {defaultConfigPath()}</Text>
|
||||
</Box>
|
||||
{error ? (
|
||||
<Box marginTop={1}>
|
||||
<Text color="red">{error}</Text>
|
||||
</Box>
|
||||
) : null}
|
||||
<Box marginTop={1}>
|
||||
<Text dimColor>enter save · esc cancel</Text>
|
||||
</Box>
|
||||
</Box>
|
||||
<ReviewConfirm
|
||||
onConfirm={() => {
|
||||
try {
|
||||
const specsNow = data.selectedCatalog.map((name) =>
|
||||
buildSpec(name, data.catalogArgs),
|
||||
);
|
||||
const prev = readConfig();
|
||||
const next: ReasonixConfig = {
|
||||
...prev,
|
||||
apiKey: data.apiKey,
|
||||
preset: data.preset,
|
||||
mcp: specsNow,
|
||||
setupCompleted: true,
|
||||
};
|
||||
writeConfig(next);
|
||||
setStep("saved");
|
||||
onComplete(next);
|
||||
} catch (e) {
|
||||
setError(`Could not save config: ${(e as Error).message}`);
|
||||
}
|
||||
}}
|
||||
/>
|
||||
</StepFrame>
|
||||
);
|
||||
}
|
||||
|
||||
// saved
|
||||
return (
|
||||
<Box flexDirection="column" borderStyle="round" borderColor="green" paddingX={1}>
|
||||
<Text bold color="green">
|
||||
▸ Saved.
|
||||
</Text>
|
||||
<Box marginTop={1}>
|
||||
<Text>Run `reasonix` any time to start chatting — your settings are remembered.</Text>
|
||||
</Box>
|
||||
<Box marginTop={1}>
|
||||
<Text dimColor>Press enter to exit.</Text>
|
||||
</Box>
|
||||
<ExitOnEnter onExit={exit} />
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
// ---------- step components ----------
|
||||
|
||||
function ApiKeyStep({
|
||||
onSubmit,
|
||||
error,
|
||||
onError,
|
||||
}: {
|
||||
onSubmit: (key: string) => void;
|
||||
error: string | null;
|
||||
onError: (e: string | null) => void;
|
||||
}) {
|
||||
const [value, setValue] = useState("");
|
||||
return (
|
||||
<Box flexDirection="column" borderStyle="round" borderColor="cyan" paddingX={1}>
|
||||
<Text bold color="cyan">
|
||||
Welcome to Reasonix.
|
||||
</Text>
|
||||
<Box marginTop={1}>
|
||||
<Text>Paste your DeepSeek API key to get started.</Text>
|
||||
</Box>
|
||||
<Text dimColor>Get one (free credit on signup): https://platform.deepseek.com/api_keys</Text>
|
||||
<Text dimColor>Saved locally to {defaultConfigPath()}</Text>
|
||||
<Box marginTop={1}>
|
||||
<Text bold color="cyan">
|
||||
{"key › "}
|
||||
</Text>
|
||||
<TextInput
|
||||
value={value}
|
||||
onChange={setValue}
|
||||
onSubmit={(raw) => {
|
||||
const trimmed = raw.trim();
|
||||
if (!isPlausibleKey(trimmed)) {
|
||||
onError("Doesn't look like a DeepSeek key. They start with 'sk-' and are 30+ chars.");
|
||||
setValue("");
|
||||
return;
|
||||
}
|
||||
onSubmit(trimmed);
|
||||
}}
|
||||
mask="•"
|
||||
placeholder="sk-..."
|
||||
/>
|
||||
</Box>
|
||||
{error ? (
|
||||
<Box marginTop={1}>
|
||||
<Text color="red">{error}</Text>
|
||||
</Box>
|
||||
) : value ? (
|
||||
<Box marginTop={1}>
|
||||
<Text dimColor>preview: {redactKey(value)}</Text>
|
||||
</Box>
|
||||
) : null}
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
function McpArgsStep({
|
||||
entry,
|
||||
error,
|
||||
onSubmit,
|
||||
onError,
|
||||
}: {
|
||||
entry: CatalogEntry;
|
||||
error: string | null;
|
||||
onSubmit: (value: string) => void;
|
||||
onError: (e: string | null) => void;
|
||||
}) {
|
||||
const [value, setValue] = useState("");
|
||||
return (
|
||||
<StepFrame title={`Configure ${entry.name}`} step={2} total={3}>
|
||||
<Box flexDirection="column">
|
||||
<Text>{entry.summary}</Text>
|
||||
{entry.note ? (
|
||||
<Box marginTop={1}>
|
||||
<Text dimColor>{entry.note}</Text>
|
||||
</Box>
|
||||
) : null}
|
||||
<Box marginTop={1}>
|
||||
<Text>Required parameter: </Text>
|
||||
<Text bold>{entry.userArgs}</Text>
|
||||
</Box>
|
||||
<Box marginTop={1}>
|
||||
<Text bold color="cyan">
|
||||
{entry.userArgs}
|
||||
{" › "}
|
||||
</Text>
|
||||
<TextInput
|
||||
value={value}
|
||||
onChange={setValue}
|
||||
onSubmit={(raw) => {
|
||||
const trimmed = raw.trim();
|
||||
if (!trimmed) {
|
||||
onError(`${entry.name} needs a value — got an empty string.`);
|
||||
return;
|
||||
}
|
||||
onSubmit(trimmed);
|
||||
setValue("");
|
||||
}}
|
||||
placeholder={placeholderFor(entry)}
|
||||
/>
|
||||
</Box>
|
||||
{error ? (
|
||||
<Box marginTop={1}>
|
||||
<Text color="red">{error}</Text>
|
||||
</Box>
|
||||
) : null}
|
||||
</Box>
|
||||
</StepFrame>
|
||||
);
|
||||
}
|
||||
|
||||
function ReviewConfirm({ onConfirm }: { onConfirm: () => void }) {
|
||||
useInput((_i, key) => {
|
||||
if (key.return) onConfirm();
|
||||
});
|
||||
return null;
|
||||
}
|
||||
|
||||
function ExitOnEnter({ onExit }: { onExit: () => void }) {
|
||||
useInput((_i, key) => {
|
||||
if (key.return) onExit();
|
||||
});
|
||||
return null;
|
||||
}
|
||||
|
||||
// ---------- small bits ----------
|
||||
|
||||
function StepFrame({
|
||||
title,
|
||||
step,
|
||||
total,
|
||||
children,
|
||||
}: {
|
||||
title: string;
|
||||
step: number;
|
||||
total: number;
|
||||
children: React.ReactNode;
|
||||
}) {
|
||||
return (
|
||||
<Box flexDirection="column" borderStyle="round" borderColor="cyan" paddingX={1}>
|
||||
<Box>
|
||||
<Text dimColor>
|
||||
Step {step}/{total} ·{" "}
|
||||
</Text>
|
||||
<Text bold color="cyan">
|
||||
{title}
|
||||
</Text>
|
||||
</Box>
|
||||
<Box marginTop={1} flexDirection="column">
|
||||
{children}
|
||||
</Box>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
function SummaryLine({ label, value }: { label: string; value: string }) {
|
||||
return (
|
||||
<Box>
|
||||
<Text>{label.padEnd(12)}</Text>
|
||||
<Text bold>{value}</Text>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
// ---------- data helpers ----------
|
||||
|
||||
function presetItems(): SelectItem<PresetName>[] {
|
||||
return (["fast", "smart", "max"] as PresetName[]).map((name) => ({
|
||||
value: name,
|
||||
label: `${name} — ${PRESET_DESCRIPTIONS[name].headline}`,
|
||||
hint: PRESET_DESCRIPTIONS[name].cost,
|
||||
}));
|
||||
}
|
||||
|
||||
function mcpItems(): SelectItem<string>[] {
|
||||
return MCP_CATALOG.map((entry) => {
|
||||
const hintParts: string[] = [entry.summary];
|
||||
if (entry.userArgs) hintParts.push(`(you'll provide ${entry.userArgs})`);
|
||||
if (entry.note) hintParts.push(entry.note);
|
||||
return {
|
||||
value: entry.name,
|
||||
label: entry.name,
|
||||
hint: hintParts.join(" · "),
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
function placeholderFor(entry: CatalogEntry): string {
|
||||
if (entry.name === "filesystem") return "e.g. /tmp/reasonix-sandbox";
|
||||
if (entry.name === "sqlite") return "e.g. ./notes.sqlite";
|
||||
return entry.userArgs ?? "";
|
||||
}
|
||||
|
||||
function deriveInitialCatalog(existingSpecs: string[]): string[] {
|
||||
// Best-effort recovery: if the user previously picked catalog entries,
|
||||
// the spec strings look like `name=npx -y <pkg> ...`. Match by package
|
||||
// so reconfigure pre-checks the same boxes.
|
||||
const packageToName = new Map(MCP_CATALOG.map((e) => [e.package, e.name]));
|
||||
const out: string[] = [];
|
||||
for (const spec of existingSpecs) {
|
||||
for (const [pkg, name] of packageToName) {
|
||||
if (spec.includes(pkg)) {
|
||||
out.push(name);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the `--mcp` spec string for a catalog entry. Same format
|
||||
* `mcpCommandFor` produces for `reasonix mcp list`, minus the leading
|
||||
* `--mcp "..."` wrapper — we store the inner spec directly.
|
||||
*/
|
||||
export function buildSpec(name: string, argsByName: Record<string, string>): string {
|
||||
const entry = CATALOG_BY_NAME.get(name);
|
||||
if (!entry) return name; // shouldn't happen; fall back gracefully
|
||||
const userArg = entry.userArgs ? argsByName[name] : undefined;
|
||||
const tail = userArg ? ` ${quoteIfNeeded(userArg)}` : "";
|
||||
return `${entry.name}=npx -y ${entry.package}${tail}`;
|
||||
}
|
||||
|
||||
function quoteIfNeeded(s: string): string {
|
||||
return /\s|"/.test(s) ? `"${s.replace(/"/g, '\\"')}"` : s;
|
||||
}
|
||||
+103
-3
@@ -206,10 +206,22 @@ interface CodeBlock {
|
||||
interface HrBlock {
|
||||
kind: "hr";
|
||||
}
|
||||
// First-class Aider-style SEARCH/REPLACE block. We detect these at
|
||||
// parse time instead of routing them through the paragraph / inline
|
||||
// markdown path because the inline parser would otherwise eat `**`
|
||||
// inside JSDoc `/** ... *\/` comments and `para.join(" ")` would
|
||||
// collapse the block's newlines. Rendered as a diff so the user can
|
||||
// actually read what's about to change.
|
||||
interface EditBlockView {
|
||||
kind: "edit-block";
|
||||
filename: string;
|
||||
search: string;
|
||||
replace: string;
|
||||
}
|
||||
|
||||
type Block = ParagraphBlock | HeadingBlock | BulletBlock | CodeBlock | HrBlock;
|
||||
type Block = ParagraphBlock | HeadingBlock | BulletBlock | CodeBlock | HrBlock | EditBlockView;
|
||||
|
||||
function parseBlocks(raw: string): Block[] {
|
||||
export function parseBlocks(raw: string): Block[] {
|
||||
const lines = raw.split(/\r?\n/);
|
||||
const out: Block[] = [];
|
||||
let para: string[] = [];
|
||||
@@ -231,9 +243,50 @@ function parseBlocks(raw: string): Block[] {
|
||||
}
|
||||
};
|
||||
|
||||
for (const rawLine of lines) {
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const rawLine = lines[i]!;
|
||||
const line = rawLine.replace(/\s+$/g, "");
|
||||
|
||||
// Detect Aider-style SEARCH/REPLACE block. Matches the preceding
|
||||
// non-blank line as the filename, then `<<<<<<< SEARCH`, content,
|
||||
// `=======`, content, `>>>>>>> REPLACE`. We don't do markdown
|
||||
// inside — neither the paragraph nor inline parsers should touch
|
||||
// this content.
|
||||
if (!inCode && /^<{7} SEARCH\s*$/.test(line)) {
|
||||
// Filename is the previous non-blank line we just pushed to para.
|
||||
// Pull it back out; if there isn't one, treat as literal text.
|
||||
const filename = para.pop()?.trim();
|
||||
if (filename) {
|
||||
flushPara();
|
||||
flushList();
|
||||
let j = i + 1;
|
||||
const searchLines: string[] = [];
|
||||
while (j < lines.length && !/^={7}\s*$/.test(lines[j]!)) {
|
||||
searchLines.push(lines[j]!);
|
||||
j++;
|
||||
}
|
||||
const replaceLines: string[] = [];
|
||||
let k = j + 1;
|
||||
while (k < lines.length && !/^>{7} REPLACE\s*$/.test(lines[k]!)) {
|
||||
replaceLines.push(lines[k]!);
|
||||
k++;
|
||||
}
|
||||
if (j < lines.length && k < lines.length) {
|
||||
out.push({
|
||||
kind: "edit-block",
|
||||
filename,
|
||||
search: searchLines.join("\n"),
|
||||
replace: replaceLines.join("\n"),
|
||||
});
|
||||
i = k;
|
||||
continue;
|
||||
}
|
||||
// Malformed — no separator or no close. Fall through: put
|
||||
// the filename back in the paragraph so we don't lose it.
|
||||
para.push(filename);
|
||||
}
|
||||
}
|
||||
|
||||
const fence = line.match(/^```(\w*)/);
|
||||
if (fence) {
|
||||
if (inCode) {
|
||||
@@ -336,11 +389,58 @@ function BlockView({ block }: { block: Block }) {
|
||||
<Text color="yellow">{block.text}</Text>
|
||||
</Box>
|
||||
);
|
||||
case "edit-block":
|
||||
return <EditBlockRow block={block} />;
|
||||
case "hr":
|
||||
return <Text dimColor>{"────────────────────────"}</Text>;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* SEARCH/REPLACE rendered as a minimal diff: filename on top, red
|
||||
* `-` lines, a gutter, green `+` lines. No inner markdown / inline
|
||||
* parsing — content is shown verbatim so JSDoc `/**` and `*` won't
|
||||
* be eaten by bold/italic regex.
|
||||
*
|
||||
* A truly empty SEARCH means "new file" and we label the filename
|
||||
* accordingly instead of rendering an empty red half.
|
||||
*/
|
||||
function EditBlockRow({ block }: { block: EditBlockView }) {
|
||||
const isNewFile = block.search.length === 0;
|
||||
const searchLines = block.search.split("\n");
|
||||
const replaceLines = block.replace.split("\n");
|
||||
return (
|
||||
<Box flexDirection="column" borderStyle="round" borderColor="cyan" paddingX={1}>
|
||||
<Box>
|
||||
<Text bold color="cyan">
|
||||
{block.filename}
|
||||
</Text>
|
||||
{isNewFile ? (
|
||||
<Text color="green" bold>
|
||||
{" (new file)"}
|
||||
</Text>
|
||||
) : null}
|
||||
</Box>
|
||||
{isNewFile ? null : (
|
||||
<Box flexDirection="column" marginTop={1}>
|
||||
{searchLines.map((line, i) => (
|
||||
<Text key={`s-${i}-${line.length}`} color="red">
|
||||
{`- ${line}`}
|
||||
</Text>
|
||||
))}
|
||||
</Box>
|
||||
)}
|
||||
<Box flexDirection="column" marginTop={isNewFile ? 1 : 0}>
|
||||
{replaceLines.map((line, i) => (
|
||||
<Text key={`r-${i}-${line.length}`} color="green">
|
||||
{`+ ${line}`}
|
||||
</Text>
|
||||
))}
|
||||
</Box>
|
||||
</Box>
|
||||
);
|
||||
}
|
||||
|
||||
export function Markdown({ text }: { text: string }) {
|
||||
const cleaned = stripMath(text);
|
||||
const blocks = React.useMemo(() => parseBlocks(cleaned), [cleaned]);
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
/**
|
||||
* One place that defines what each preset means. Both `slash.ts`
|
||||
* (in-chat `/preset`) and the wizard (first-run setup) read from here.
|
||||
*
|
||||
* Presets are the single vocabulary we teach new users: they don't need
|
||||
* to know model IDs, Pillar 2, branch budgets, or cost tradeoffs
|
||||
* independently — they pick "fast / smart / max" and we translate.
|
||||
*/
|
||||
|
||||
import type { PresetName } from "../../config.js";
|
||||
|
||||
export interface PresetSettings {
|
||||
model: string;
|
||||
harvest: boolean;
|
||||
/** Branch budget. `1` means branching off. */
|
||||
branch: number;
|
||||
}
|
||||
|
||||
export const PRESETS: Record<PresetName, PresetSettings> = {
|
||||
fast: { model: "deepseek-chat", harvest: false, branch: 1 },
|
||||
smart: { model: "deepseek-reasoner", harvest: true, branch: 1 },
|
||||
max: { model: "deepseek-reasoner", harvest: true, branch: 3 },
|
||||
};
|
||||
|
||||
export const PRESET_DESCRIPTIONS: Record<PresetName, { headline: string; cost: string }> = {
|
||||
fast: {
|
||||
headline: "deepseek-chat, no reasoning harvest, no branching",
|
||||
cost: "~1¢ per 100 turns · default",
|
||||
},
|
||||
smart: {
|
||||
headline: "deepseek-reasoner + Pillar 2 harvest",
|
||||
cost: "~10× cost vs fast · slower · better on multi-step tasks",
|
||||
},
|
||||
max: {
|
||||
headline: "reasoner + harvest + self-consistency (3 branches)",
|
||||
cost: "~30× cost vs fast · slowest · for hard single-shots",
|
||||
},
|
||||
};
|
||||
|
||||
export function resolvePreset(name: PresetName | undefined): PresetSettings {
|
||||
return PRESETS[name ?? "fast"];
|
||||
}
|
||||
+476
-8
@@ -1,5 +1,8 @@
|
||||
import { spawnSync } from "node:child_process";
|
||||
import type { CacheFirstLoop } from "../../loop.js";
|
||||
import type { InspectionReport } from "../../mcp/inspect.js";
|
||||
import { deleteSession, listSessions } from "../../session.js";
|
||||
import { DEEPSEEK_CONTEXT_TOKENS, DEFAULT_CONTEXT_TOKENS } from "../../telemetry.js";
|
||||
|
||||
export interface SlashResult {
|
||||
/** Text to display back to the user as a system/info line. */
|
||||
@@ -10,6 +13,146 @@ export interface SlashResult {
|
||||
clear?: boolean;
|
||||
/** Unknown command — display usage hint. */
|
||||
unknown?: boolean;
|
||||
/**
|
||||
* Re-submit this text as a user message after displaying `info`.
|
||||
* Used by `/retry` — the slash command truncates the log, then
|
||||
* asks the TUI to push the original text back through the normal
|
||||
* submit flow so a fresh turn runs.
|
||||
*/
|
||||
resubmit?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extra runtime context a slash handler may care about but that isn't
|
||||
* already on the loop. Kept as an optional object so tests that only
|
||||
* need loop-scoped commands can skip it, and callers only populate the
|
||||
* slots that apply to their session.
|
||||
*/
|
||||
export interface SlashContext {
|
||||
/**
|
||||
* The exact `--mcp` / config-derived spec strings that were bridged
|
||||
* into this session (one entry per server). Used by `/mcp`. Empty or
|
||||
* omitted → no MCP servers attached.
|
||||
*/
|
||||
mcpSpecs?: string[];
|
||||
/**
|
||||
* Callback for `/undo` — provided by the TUI when it's running in
|
||||
* code mode. Returns a human-readable report of what was restored.
|
||||
* Absent outside code mode → `/undo` replies "not available here".
|
||||
*/
|
||||
codeUndo?: () => string;
|
||||
/**
|
||||
* Callback for `/apply` — commits pending edit blocks to disk. Returns
|
||||
* a report of what landed. Absent → `/apply` replies "nothing pending"
|
||||
* or "not available outside code mode".
|
||||
*/
|
||||
codeApply?: () => string;
|
||||
/**
|
||||
* Callback for `/discard` — drops the pending edit blocks without
|
||||
* touching disk.
|
||||
*/
|
||||
codeDiscard?: () => string;
|
||||
/**
|
||||
* Root directory passed by `reasonix code`. Enables `/commit`, which
|
||||
* runs `git add -A && git commit` in this directory. Missing → `/commit`
|
||||
* replies "only available in code mode".
|
||||
*/
|
||||
codeRoot?: string;
|
||||
/**
|
||||
* How many edit blocks are currently pending `/apply` or `/discard`.
|
||||
* Surfaced by `/status`. TUI populates it live from its pending ref;
|
||||
* omitted → treat as 0 (chat-only session).
|
||||
*/
|
||||
pendingEditCount?: number;
|
||||
/**
|
||||
* Callback returning every tool result seen this session in
|
||||
* chronological order (oldest first). Powers `/tool [N]` for
|
||||
* inspecting the full untruncated output that `EventLog` clips at
|
||||
* 400 chars for display. Absent → `/tool` replies "not available".
|
||||
*/
|
||||
toolHistory?: () => Array<{ toolName: string; text: string }>;
|
||||
/**
|
||||
* Pre-captured inspection reports for each connected MCP server.
|
||||
* Populated once at chat startup (chat.tsx) so `/mcp` can render
|
||||
* tools + resources + prompts synchronously without needing async
|
||||
* handler support.
|
||||
*/
|
||||
mcpServers?: McpServerSummary[];
|
||||
}
|
||||
|
||||
export interface McpServerSummary {
|
||||
/** Short label shown in the `/mcp` output (server namespace or "anon"). */
|
||||
label: string;
|
||||
/** Original --mcp spec string. */
|
||||
spec: string;
|
||||
/** Count of tools bridged into the Reasonix registry from this server. */
|
||||
toolCount: number;
|
||||
/** Full inspection snapshot — used for the resources + prompts sections. */
|
||||
report: InspectionReport;
|
||||
}
|
||||
|
||||
/**
|
||||
* Slash command registry. Drives `/help`, the on-type suggestion
|
||||
* popup (`SlashSuggestions`), and auto-complete. Kept as data rather
|
||||
* than derived from the `handleSlash` switch so summaries can be
|
||||
* user-facing rather than code comments.
|
||||
*
|
||||
* `contextual` gates commands that only make sense in certain modes:
|
||||
* - `"code"` — only show when the TUI is running `reasonix code`
|
||||
* - absent → always shown
|
||||
*/
|
||||
export interface SlashCommandSpec {
|
||||
cmd: string;
|
||||
summary: string;
|
||||
contextual?: "code";
|
||||
/** If the command takes args, hint text shown after the name. */
|
||||
argsHint?: string;
|
||||
}
|
||||
|
||||
export const SLASH_COMMANDS: readonly SlashCommandSpec[] = [
|
||||
{ cmd: "help", summary: "show the full command reference" },
|
||||
{ cmd: "status", summary: "current model, flags, context, session" },
|
||||
{
|
||||
cmd: "preset",
|
||||
argsHint: "<fast|smart|max>",
|
||||
summary: "one-tap model + harvest + branch bundle",
|
||||
},
|
||||
{ cmd: "model", argsHint: "<id>", summary: "switch DeepSeek model id" },
|
||||
{ cmd: "harvest", argsHint: "[on|off]", summary: "toggle Pillar-2 plan-state extraction" },
|
||||
{ cmd: "branch", argsHint: "<N|off>", summary: "run N parallel samples per turn (N>=2)" },
|
||||
{ cmd: "mcp", summary: "list MCP servers + tools attached to this session" },
|
||||
{ cmd: "tool", argsHint: "[N]", summary: "dump full output of the Nth tool call (1=latest)" },
|
||||
{ cmd: "think", summary: "dump the last turn's full R1 reasoning (reasoner only)" },
|
||||
{ cmd: "retry", summary: "truncate & resend your last message (fresh sample)" },
|
||||
{ cmd: "compact", argsHint: "[cap]", summary: "shrink oversized tool results in the log" },
|
||||
{ cmd: "sessions", summary: "list saved sessions (current marked with ▸)" },
|
||||
{ cmd: "forget", summary: "delete the current session from disk" },
|
||||
{ cmd: "setup", summary: "reminds you to exit and run `reasonix setup`" },
|
||||
{ cmd: "clear", summary: "clear the visible scrollback (log + session kept)" },
|
||||
{ cmd: "exit", summary: "quit the TUI" },
|
||||
// Code-mode only
|
||||
{ cmd: "apply", summary: "commit pending edit blocks to disk", contextual: "code" },
|
||||
{ cmd: "discard", summary: "drop pending edit blocks without writing", contextual: "code" },
|
||||
{ cmd: "undo", summary: "roll back the last applied edit batch", contextual: "code" },
|
||||
{
|
||||
cmd: "commit",
|
||||
argsHint: '"msg"',
|
||||
summary: "git add -A && git commit -m ...",
|
||||
contextual: "code",
|
||||
},
|
||||
];
|
||||
|
||||
/**
|
||||
* Filter the registry by a prefix string (without the leading `/`).
|
||||
* Empty prefix returns the full non-contextual list (plus code-mode
|
||||
* entries when `codeMode` is true). Case-insensitive.
|
||||
*/
|
||||
export function suggestSlashCommands(prefix: string, codeMode = false): SlashCommandSpec[] {
|
||||
const p = prefix.toLowerCase();
|
||||
return SLASH_COMMANDS.filter((c) => {
|
||||
if (c.contextual === "code" && !codeMode) return false;
|
||||
return c.cmd.startsWith(p);
|
||||
});
|
||||
}
|
||||
|
||||
export function parseSlash(text: string): { cmd: string; args: string[] } | null {
|
||||
@@ -20,7 +163,12 @@ export function parseSlash(text: string): { cmd: string; args: string[] } | null
|
||||
return { cmd, args: parts.slice(1) };
|
||||
}
|
||||
|
||||
export function handleSlash(cmd: string, args: string[], loop: CacheFirstLoop): SlashResult {
|
||||
export function handleSlash(
|
||||
cmd: string,
|
||||
args: string[],
|
||||
loop: CacheFirstLoop,
|
||||
ctx: SlashContext = {},
|
||||
): SlashResult {
|
||||
switch (cmd) {
|
||||
case "exit":
|
||||
case "quit":
|
||||
@@ -40,6 +188,16 @@ export function handleSlash(cmd: string, args: string[], loop: CacheFirstLoop):
|
||||
" /model <id> deepseek-chat or deepseek-reasoner",
|
||||
" /harvest [on|off] Pillar 2: structured plan-state extraction",
|
||||
" /branch <N|off> run N parallel samples (N>=2), pick most confident",
|
||||
" /mcp list MCP servers + tools attached to this session",
|
||||
" /setup (exit + reconfigure) → run `reasonix setup`",
|
||||
" /compact [cap] shrink large tool results in history (default 4k/result)",
|
||||
" /think dump the most recent turn's full R1 reasoning (reasoner only)",
|
||||
" /tool [N] list tool calls (or dump full output of #N, 1=most recent)",
|
||||
" /retry truncate & resend your last message (fresh sample from the model)",
|
||||
" /apply (code mode) commit the pending edit blocks to disk",
|
||||
" /discard (code mode) drop pending edits without writing",
|
||||
" /undo (code mode) roll back the last applied edit batch",
|
||||
' /commit "msg" (code mode) git add -A && git commit -m "msg"',
|
||||
" /sessions list saved sessions (current is marked with ▸)",
|
||||
" /forget delete the current session from disk",
|
||||
" /clear clear displayed history (log + session kept)",
|
||||
@@ -56,6 +214,192 @@ export function handleSlash(cmd: string, args: string[], loop: CacheFirstLoop):
|
||||
].join("\n"),
|
||||
};
|
||||
|
||||
case "mcp": {
|
||||
const servers = ctx.mcpServers ?? [];
|
||||
const specs = ctx.mcpSpecs ?? [];
|
||||
const toolSpecs = loop.prefix.toolSpecs ?? [];
|
||||
if (servers.length === 0 && specs.length === 0 && toolSpecs.length === 0) {
|
||||
return {
|
||||
info:
|
||||
"no MCP servers attached. Run `reasonix setup` to pick some, " +
|
||||
'or launch with --mcp "<spec>". `reasonix mcp list` shows the catalog.',
|
||||
};
|
||||
}
|
||||
// Rich path — we have full inspection reports, so show each
|
||||
// server with its tools / resources / prompts grouped together.
|
||||
if (servers.length > 0) {
|
||||
const lines: string[] = [];
|
||||
for (const s of servers) {
|
||||
const { report } = s;
|
||||
const serverName = report.serverInfo.name || "(unknown)";
|
||||
const serverVer = report.serverInfo.version ? ` v${report.serverInfo.version}` : "";
|
||||
lines.push(`[${s.label}] ${serverName}${serverVer} — ${s.spec}`);
|
||||
lines.push(` tools ${s.toolCount}`);
|
||||
appendSection(lines, "resources", report.resources);
|
||||
appendSection(lines, "prompts ", report.prompts);
|
||||
lines.push("");
|
||||
}
|
||||
lines.push(
|
||||
"Chat mode consumes tools today; resources+prompts are surfaced here for awareness.",
|
||||
);
|
||||
lines.push(
|
||||
"Full catalog: `reasonix mcp list` · deeper diagnosis: `reasonix mcp inspect <spec>`.",
|
||||
);
|
||||
return { info: lines.join("\n") };
|
||||
}
|
||||
// Fallback — older path when the TUI hasn't populated `mcpServers`.
|
||||
const lines: string[] = [];
|
||||
if (specs.length > 0) {
|
||||
lines.push(`MCP servers (${specs.length}):`);
|
||||
for (const spec of specs) lines.push(` · ${spec}`);
|
||||
lines.push("");
|
||||
}
|
||||
if (toolSpecs.length > 0) {
|
||||
lines.push(`Tools in registry (${toolSpecs.length}):`);
|
||||
for (const t of toolSpecs) lines.push(` · ${t.function.name}`);
|
||||
}
|
||||
lines.push("");
|
||||
lines.push("To change this set, exit and run `reasonix setup`.");
|
||||
return { info: lines.join("\n") };
|
||||
}
|
||||
|
||||
case "setup":
|
||||
return {
|
||||
info:
|
||||
"To reconfigure (preset, MCP servers, API key), exit this chat and run " +
|
||||
"`reasonix setup`. Changes take effect on next launch.",
|
||||
};
|
||||
|
||||
case "retry": {
|
||||
const prev = loop.retryLastUser();
|
||||
if (!prev) {
|
||||
return {
|
||||
info: "nothing to retry — no prior user message in this session's log.",
|
||||
};
|
||||
}
|
||||
const preview = prev.length > 80 ? `${prev.slice(0, 80)}…` : prev;
|
||||
return {
|
||||
info: `▸ retrying: "${preview}"`,
|
||||
resubmit: prev,
|
||||
};
|
||||
}
|
||||
|
||||
case "think":
|
||||
case "reasoning": {
|
||||
const raw = loop.scratch.reasoning;
|
||||
if (!raw || !raw.trim()) {
|
||||
return {
|
||||
info:
|
||||
"no reasoning cached. `/think` shows the full R1 thought for the most recent turn — " +
|
||||
"only `deepseek-reasoner` produces it, and only once the turn completes.",
|
||||
};
|
||||
}
|
||||
return { info: `↳ full thinking (${raw.length} chars):\n\n${raw.trim()}` };
|
||||
}
|
||||
|
||||
case "tool": {
|
||||
// EventLog truncates tool results at 400 chars for display. When
|
||||
// the user wants to check what the model actually read (e.g. to
|
||||
// verify it isn't hallucinating a file's contents), they need
|
||||
// the full text. `/tool` is the escape hatch.
|
||||
const history = ctx.toolHistory?.() ?? [];
|
||||
if (history.length === 0) {
|
||||
return {
|
||||
info:
|
||||
"no tool calls yet in this session. `/tool` lists them once the model has actually " +
|
||||
"used a tool; `/tool N` dumps the full (untruncated) output of the Nth-most-recent.",
|
||||
};
|
||||
}
|
||||
const raw = (args[0] ?? "").toLowerCase();
|
||||
if (raw === "" || raw === "list" || raw === "ls") {
|
||||
return { info: formatToolList(history) };
|
||||
}
|
||||
const n = Number.parseInt(raw, 10);
|
||||
if (!Number.isFinite(n) || n < 1) {
|
||||
return {
|
||||
info: "usage: /tool [N] (no arg → list; N=1 → most recent result in full, N=2 → previous, …)",
|
||||
};
|
||||
}
|
||||
if (n > history.length) {
|
||||
return {
|
||||
info: `only ${history.length} tool call(s) in history — asked for #${n}. Try /tool with no arg to see the list.`,
|
||||
};
|
||||
}
|
||||
const entry = history[history.length - n];
|
||||
if (!entry) {
|
||||
return { info: `could not read tool call #${n}` };
|
||||
}
|
||||
return {
|
||||
info: `↳ tool<${entry.toolName}> #${n} (${entry.text.length} chars):\n\n${entry.text}`,
|
||||
};
|
||||
}
|
||||
|
||||
case "undo": {
|
||||
if (!ctx.codeUndo) {
|
||||
return {
|
||||
info: "/undo is only available inside `reasonix code` — chat mode doesn't apply edits.",
|
||||
};
|
||||
}
|
||||
return { info: ctx.codeUndo() };
|
||||
}
|
||||
|
||||
case "apply": {
|
||||
if (!ctx.codeApply) {
|
||||
return {
|
||||
info: "/apply is only available inside `reasonix code` (nothing to apply here).",
|
||||
};
|
||||
}
|
||||
return { info: ctx.codeApply() };
|
||||
}
|
||||
|
||||
case "discard": {
|
||||
if (!ctx.codeDiscard) {
|
||||
return {
|
||||
info: "/discard is only available inside `reasonix code`.",
|
||||
};
|
||||
}
|
||||
return { info: ctx.codeDiscard() };
|
||||
}
|
||||
|
||||
case "commit": {
|
||||
if (!ctx.codeRoot) {
|
||||
return {
|
||||
info: "/commit is only available inside `reasonix code` (needs a rooted git repo).",
|
||||
};
|
||||
}
|
||||
// Reassemble the original argv. The parser lowercases cmd but
|
||||
// leaves args alone, and the TUI splits on whitespace which
|
||||
// mangles quoted messages — rejoin with spaces and strip a
|
||||
// surrounding pair of double quotes if the user wrote them.
|
||||
const raw = args.join(" ").trim();
|
||||
const message = stripOuterQuotes(raw);
|
||||
if (!message) {
|
||||
return {
|
||||
info: `usage: /commit "your commit message" — runs \`git add -A && git commit -m "…"\` in ${ctx.codeRoot}`,
|
||||
};
|
||||
}
|
||||
return runGitCommit(ctx.codeRoot, message);
|
||||
}
|
||||
|
||||
case "compact": {
|
||||
// Manual companion to the automatic heal-on-load. Re-applies
|
||||
// truncation with a tighter cap (4k chars per tool result) and
|
||||
// rewrites the session file so the shrink persists. Useful when
|
||||
// the ctx gauge in StatsPanel goes yellow/red mid-session and
|
||||
// the user wants to keep chatting without /forget'ing everything.
|
||||
const tight = Number.parseInt(args[0] ?? "", 10);
|
||||
const cap = Number.isFinite(tight) && tight >= 500 ? tight : 4000;
|
||||
const { healedCount, charsSaved } = loop.compact(cap);
|
||||
if (healedCount === 0) {
|
||||
return {
|
||||
info: `▸ nothing to compact — no tool result in history exceeds ${cap.toLocaleString()} chars.`,
|
||||
};
|
||||
}
|
||||
return {
|
||||
info: `▸ compacted ${healedCount} tool result(s), saved ${charsSaved.toLocaleString()} chars (~${Math.round(charsSaved / 4).toLocaleString()} tokens). Session file rewritten.`,
|
||||
};
|
||||
}
|
||||
|
||||
case "sessions": {
|
||||
const items = listSessions();
|
||||
if (items.length === 0) {
|
||||
@@ -92,13 +436,31 @@ export function handleSlash(cmd: string, args: string[], loop: CacheFirstLoop):
|
||||
|
||||
case "status": {
|
||||
const branchBudget = loop.branchOptions.budget ?? 1;
|
||||
return {
|
||||
info:
|
||||
`model=${loop.model} ` +
|
||||
`harvest=${loop.harvestEnabled ? "on" : "off"} ` +
|
||||
`branch=${branchBudget > 1 ? branchBudget : "off"} ` +
|
||||
`stream=${loop.stream ? "on" : "off"}`,
|
||||
};
|
||||
const ctxMax = DEEPSEEK_CONTEXT_TOKENS[loop.model] ?? DEFAULT_CONTEXT_TOKENS;
|
||||
const lastPromptTokens = loop.stats.summary().lastPromptTokens;
|
||||
const ctxPct = ctxMax > 0 ? Math.round((lastPromptTokens / ctxMax) * 100) : 0;
|
||||
const ctxLine =
|
||||
lastPromptTokens > 0
|
||||
? ` ctx ${compactNum(lastPromptTokens)}/${compactNum(ctxMax)} (${ctxPct}%)`
|
||||
: " ctx no turns yet";
|
||||
const pending = ctx.pendingEditCount ?? 0;
|
||||
const sessionLine = loop.sessionName
|
||||
? ` session "${loop.sessionName}" · ${loop.log.length} messages in log (resumed ${loop.resumedMessageCount})`
|
||||
: " session (ephemeral — no persistence)";
|
||||
const mcpCount = ctx.mcpSpecs?.length ?? 0;
|
||||
const toolCount = loop.prefix.toolSpecs.length;
|
||||
const mcpLine = ` mcp ${mcpCount} server(s), ${toolCount} tool(s) in registry`;
|
||||
const pendingLine =
|
||||
pending > 0 ? ` edits ${pending} pending (/apply to commit, /discard to drop)` : "";
|
||||
const lines = [
|
||||
` model ${loop.model}`,
|
||||
` flags harvest=${loop.harvestEnabled ? "on" : "off"} · branch=${branchBudget > 1 ? branchBudget : "off"} · stream=${loop.stream ? "on" : "off"}`,
|
||||
ctxLine,
|
||||
mcpLine,
|
||||
sessionLine,
|
||||
];
|
||||
if (pendingLine) lines.push(pendingLine);
|
||||
return { info: lines.join("\n") };
|
||||
}
|
||||
|
||||
case "model": {
|
||||
@@ -155,3 +517,109 @@ export function handleSlash(cmd: string, args: string[], loop: CacheFirstLoop):
|
||||
return { unknown: true, info: `unknown command: /${cmd} (try /help)` };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Render a section (resources / prompts) of an MCP inspection into a
|
||||
* compact "name count items" form, collapsing when unsupported.
|
||||
* Names-only — descriptions and full metadata live in
|
||||
* `reasonix mcp inspect`, which is purpose-built for the deep view.
|
||||
*/
|
||||
function appendSection(
|
||||
lines: string[],
|
||||
label: string,
|
||||
section:
|
||||
| { supported: true; items: Array<{ name: string }> }
|
||||
| { supported: false; reason: string }
|
||||
| undefined,
|
||||
): void {
|
||||
if (!section || !section.supported) {
|
||||
lines.push(
|
||||
` ${label.trim()} ${section?.supported === false ? "(not supported)" : "(none)"}`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
const names = section.items.map((i) => i.name);
|
||||
if (names.length === 0) {
|
||||
lines.push(` ${label.trim()} (none)`);
|
||||
return;
|
||||
}
|
||||
const head = names.slice(0, 5).join(", ");
|
||||
const more = names.length > 5 ? ` +${names.length - 5} more` : "";
|
||||
lines.push(` ${label.trim()} ${names.length} [${head}${more}]`);
|
||||
}
|
||||
|
||||
function formatToolList(history: Array<{ toolName: string; text: string }>): string {
|
||||
const total = history.length;
|
||||
const header = `Tool calls in this session (${total}, most recent first):`;
|
||||
// Show the 10 most recent. Older ones are rarely what the user
|
||||
// wants — and the help footer tells them how to reach any entry
|
||||
// by index if they do.
|
||||
const shown = Math.min(total, 10);
|
||||
const lines: string[] = [header];
|
||||
for (let i = 0; i < shown; i++) {
|
||||
const entry = history[total - 1 - i];
|
||||
if (!entry) continue;
|
||||
const idx = i + 1; // 1-based from most recent
|
||||
const flat = entry.text.replace(/\s+/g, " ").trim();
|
||||
const preview = flat.length > 80 ? `${flat.slice(0, 80)}…` : flat;
|
||||
const name = entry.toolName.length > 24 ? `${entry.toolName.slice(0, 23)}…` : entry.toolName;
|
||||
lines.push(
|
||||
` #${String(idx).padStart(2)} ${name.padEnd(24)} ${String(entry.text.length).padStart(6)} chars ${preview}`,
|
||||
);
|
||||
}
|
||||
if (total > shown) {
|
||||
lines.push(` … (${total - shown} earlier, reach with /tool N)`);
|
||||
}
|
||||
lines.push("");
|
||||
lines.push("View full output: /tool N (N=1 → most recent)");
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function compactNum(n: number): string {
|
||||
if (n < 1000) return String(n);
|
||||
const k = n / 1000;
|
||||
return k >= 100 ? `${Math.round(k)}k` : `${k.toFixed(1)}k`;
|
||||
}
|
||||
|
||||
function stripOuterQuotes(s: string): string {
|
||||
if (s.length >= 2 && s.startsWith('"') && s.endsWith('"')) {
|
||||
return s.slice(1, -1);
|
||||
}
|
||||
return s;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run `git add -A` then `git commit -m <message>` in `rootDir`. Returns
|
||||
* a SlashResult with a human-scannable info line. We surface stderr on
|
||||
* failure so the user sees exactly what git complained about (bad
|
||||
* config, pre-commit hook rejection, nothing staged, etc.).
|
||||
*/
|
||||
function runGitCommit(rootDir: string, message: string): SlashResult {
|
||||
const add = spawnSync("git", ["add", "-A"], { cwd: rootDir, encoding: "utf8" });
|
||||
if (add.error || add.status !== 0) {
|
||||
return { info: `git add failed (${add.status ?? "?"}):\n${gitTail(add)}` };
|
||||
}
|
||||
const commit = spawnSync("git", ["commit", "-m", message], {
|
||||
cwd: rootDir,
|
||||
encoding: "utf8",
|
||||
});
|
||||
if (commit.error || commit.status !== 0) {
|
||||
return { info: `git commit failed (${commit.status ?? "?"}):\n${gitTail(commit)}` };
|
||||
}
|
||||
const firstLine = (commit.stdout || "").split(/\r?\n/)[0] ?? "";
|
||||
return { info: `▸ committed: ${message}${firstLine ? `\n ${firstLine}` : ""}` };
|
||||
}
|
||||
|
||||
/**
|
||||
* Safely extract whatever diagnostic text is available from a spawnSync
|
||||
* result — on Windows or when cwd doesn't exist, `stderr`/`stdout` can
|
||||
* be `undefined` and the caller has only `error.message` to go on.
|
||||
*/
|
||||
function gitTail(res: ReturnType<typeof spawnSync>): string {
|
||||
const stderr = (res.stderr as string | undefined) ?? "";
|
||||
const stdout = (res.stdout as string | undefined) ?? "";
|
||||
const body = stderr.trim() || stdout.trim();
|
||||
if (body) return body;
|
||||
if (res.error) return (res.error as Error).message;
|
||||
return "(no output from git)";
|
||||
}
|
||||
@@ -0,0 +1,241 @@
|
||||
/**
|
||||
* Aider-style SEARCH/REPLACE edit blocks.
|
||||
*
|
||||
* The model emits blocks in this exact shape, one or more per response:
|
||||
*
|
||||
* path/to/file.ts
|
||||
* <<<<<<< SEARCH
|
||||
* exact existing lines (whitespace-sensitive)
|
||||
* =======
|
||||
* replacement lines
|
||||
* >>>>>>> REPLACE
|
||||
*
|
||||
* We chose this over unified diffs because:
|
||||
* - Models produce it reliably — no line-number drift.
|
||||
* - It tolerates multi-edit responses without ambiguity over which
|
||||
* hunk belongs to which file.
|
||||
* - Aider has years of evidence that this format works even against
|
||||
* weaker models than DeepSeek R1, so it's a conservative pick.
|
||||
*
|
||||
* The SEARCH text must match the file byte-for-byte. Empty SEARCH is a
|
||||
* sentinel for "create new file" — the REPLACE becomes the whole file.
|
||||
* If SEARCH doesn't match we refuse the edit and surface the failure;
|
||||
* we do NOT guess or fuzzy-match. A wrong silent edit is worse than a
|
||||
* missing one — the user can re-ask with the exact current content.
|
||||
*/
|
||||
|
||||
import { existsSync, mkdirSync, readFileSync, unlinkSync, writeFileSync } from "node:fs";
|
||||
import { dirname, resolve } from "node:path";
|
||||
|
||||
export interface EditBlock {
|
||||
/** Path as written by the model — relative to rootDir, or absolute. */
|
||||
path: string;
|
||||
/** Literal text to match in the target file. Empty → create new file. */
|
||||
search: string;
|
||||
/** Replacement text to write in place of `search`. */
|
||||
replace: string;
|
||||
/** Char offset in the source message where this block started. */
|
||||
offset: number;
|
||||
}
|
||||
|
||||
export type ApplyStatus =
|
||||
/** Edit landed on disk. */
|
||||
| "applied"
|
||||
/** New file created (SEARCH was empty and file didn't exist). */
|
||||
| "created"
|
||||
/** File exists but SEARCH block wasn't found in its content. */
|
||||
| "not-found"
|
||||
/** File doesn't exist and SEARCH was non-empty (can't create without content). */
|
||||
| "file-missing"
|
||||
/** Path escapes rootDir — refused on safety grounds. */
|
||||
| "path-escape"
|
||||
/** fs write / read threw. */
|
||||
| "error";
|
||||
|
||||
export interface ApplyResult {
|
||||
path: string;
|
||||
status: ApplyStatus;
|
||||
/** Extra detail (e.g. error message) for logs. */
|
||||
message?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* One edit block per match. The regex is anchored to the 7-char marker
|
||||
* lines because those are visually distinct and unlikely to appear in
|
||||
* normal code.
|
||||
*
|
||||
* Anchored with `^` + `m` flag so the filename has to live on its own
|
||||
* line. Keeps us from matching e.g. a JS-import string that happens to
|
||||
* contain `<<<<<<< SEARCH` in inner text.
|
||||
*/
|
||||
// `\n?` before the =======/REPLACE separators makes the body optional:
|
||||
// empty SEARCH (new-file sentinel) works without requiring a gratuitous
|
||||
// empty line, and the same holds for empty REPLACE (file-deletion
|
||||
// semantics, not yet supported but cheaply representable).
|
||||
const BLOCK_RE = /^(\S[^\n]*)\n<{7} SEARCH\n([\s\S]*?)\n?={7}\n([\s\S]*?)\n?>{7} REPLACE/gm;
|
||||
|
||||
export function parseEditBlocks(text: string): EditBlock[] {
|
||||
const out: EditBlock[] = [];
|
||||
BLOCK_RE.lastIndex = 0;
|
||||
let m: RegExpExecArray | null = BLOCK_RE.exec(text);
|
||||
while (m !== null) {
|
||||
out.push({
|
||||
path: m[1]!.trim(),
|
||||
search: m[2]!,
|
||||
replace: m[3]!,
|
||||
offset: m.index,
|
||||
});
|
||||
m = BLOCK_RE.exec(text);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
export function applyEditBlock(block: EditBlock, rootDir: string): ApplyResult {
|
||||
const absRoot = resolve(rootDir);
|
||||
const absTarget = resolve(absRoot, block.path);
|
||||
// Refuse paths that escape rootDir. `resolve` normalizes `..`, so
|
||||
// startsWith on the normalized pair is enough.
|
||||
if (absTarget !== absRoot && !absTarget.startsWith(`${absRoot}${sep()}`)) {
|
||||
return {
|
||||
path: block.path,
|
||||
status: "path-escape",
|
||||
message: `resolved path ${absTarget} is outside rootDir ${absRoot}`,
|
||||
};
|
||||
}
|
||||
|
||||
const searchEmpty = block.search.length === 0;
|
||||
const exists = existsSync(absTarget);
|
||||
|
||||
try {
|
||||
if (!exists) {
|
||||
if (!searchEmpty) {
|
||||
return {
|
||||
path: block.path,
|
||||
status: "file-missing",
|
||||
message: "file does not exist; to create it, use an empty SEARCH block",
|
||||
};
|
||||
}
|
||||
mkdirSync(dirname(absTarget), { recursive: true });
|
||||
writeFileSync(absTarget, block.replace, "utf8");
|
||||
return { path: block.path, status: "created" };
|
||||
}
|
||||
|
||||
const content = readFileSync(absTarget, "utf8");
|
||||
if (searchEmpty) {
|
||||
return {
|
||||
path: block.path,
|
||||
status: "not-found",
|
||||
message: "empty SEARCH only creates new files — this file already exists",
|
||||
};
|
||||
}
|
||||
const idx = content.indexOf(block.search);
|
||||
if (idx === -1) {
|
||||
return {
|
||||
path: block.path,
|
||||
status: "not-found",
|
||||
message: "SEARCH text does not match the current file content exactly",
|
||||
};
|
||||
}
|
||||
// Replace only the first occurrence — if the model needs multiple
|
||||
// identical edits it should emit multiple blocks (each anchored by
|
||||
// more surrounding context). Auto-expanding to replace-all is a
|
||||
// footgun when the same string legitimately appears in several
|
||||
// unrelated places.
|
||||
const replaced = `${content.slice(0, idx)}${block.replace}${content.slice(idx + block.search.length)}`;
|
||||
writeFileSync(absTarget, replaced, "utf8");
|
||||
return { path: block.path, status: "applied" };
|
||||
} catch (err) {
|
||||
return { path: block.path, status: "error", message: (err as Error).message };
|
||||
}
|
||||
}
|
||||
|
||||
export function applyEditBlocks(blocks: EditBlock[], rootDir: string): ApplyResult[] {
|
||||
return blocks.map((b) => applyEditBlock(b, rootDir));
|
||||
}
|
||||
|
||||
// ---------- snapshot / restore (for /undo) ----------
|
||||
|
||||
export interface EditSnapshot {
|
||||
/** Path relative to rootDir, as the block named it. */
|
||||
path: string;
|
||||
/**
|
||||
* File content before the edit batch was applied. `null` means the
|
||||
* file didn't exist yet — restoring that means deleting whatever the
|
||||
* edit created.
|
||||
*/
|
||||
prevContent: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Capture the current state of every file an edit batch is about to
|
||||
* touch, so `/undo` can roll back if the user doesn't like the result.
|
||||
* De-duplicates by path because one batch can contain multiple blocks
|
||||
* for the same file, and we only want one "before" snapshot per file.
|
||||
*/
|
||||
export function snapshotBeforeEdits(blocks: EditBlock[], rootDir: string): EditSnapshot[] {
|
||||
const absRoot = resolve(rootDir);
|
||||
const seen = new Set<string>();
|
||||
const snapshots: EditSnapshot[] = [];
|
||||
for (const b of blocks) {
|
||||
if (seen.has(b.path)) continue;
|
||||
seen.add(b.path);
|
||||
const abs = resolve(absRoot, b.path);
|
||||
if (!existsSync(abs)) {
|
||||
snapshots.push({ path: b.path, prevContent: null });
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
snapshots.push({ path: b.path, prevContent: readFileSync(abs, "utf8") });
|
||||
} catch {
|
||||
// Unreadable (permission / binary) — record null so we at least
|
||||
// don't pretend the snapshot is authoritative. The restore path
|
||||
// will treat null as "delete on undo", which is wrong in that
|
||||
// case but the file wasn't ours to begin with.
|
||||
snapshots.push({ path: b.path, prevContent: null });
|
||||
}
|
||||
}
|
||||
return snapshots;
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore files to their snapshotted state. Snapshots with
|
||||
* `prevContent === null` were created by the edit, so undo = delete.
|
||||
* Otherwise the prior content is written back, replacing whatever the
|
||||
* edit left behind.
|
||||
*/
|
||||
export function restoreSnapshots(snapshots: EditSnapshot[], rootDir: string): ApplyResult[] {
|
||||
const absRoot = resolve(rootDir);
|
||||
return snapshots.map((snap) => {
|
||||
const abs = resolve(absRoot, snap.path);
|
||||
if (abs !== absRoot && !abs.startsWith(`${absRoot}${sep()}`)) {
|
||||
return {
|
||||
path: snap.path,
|
||||
status: "path-escape",
|
||||
message: "snapshot path escapes rootDir — refusing to restore",
|
||||
};
|
||||
}
|
||||
try {
|
||||
if (snap.prevContent === null) {
|
||||
if (existsSync(abs)) unlinkSync(abs);
|
||||
return {
|
||||
path: snap.path,
|
||||
status: "applied",
|
||||
message: "removed (the edit had created it)",
|
||||
};
|
||||
}
|
||||
writeFileSync(abs, snap.prevContent, "utf8");
|
||||
return {
|
||||
path: snap.path,
|
||||
status: "applied",
|
||||
message: "restored to pre-edit content",
|
||||
};
|
||||
} catch (err) {
|
||||
return { path: snap.path, status: "error", message: (err as Error).message };
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/** Platform separator — `\` on Windows, `/` elsewhere. */
|
||||
function sep(): string {
|
||||
return process.platform === "win32" ? "\\" : "/";
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
/**
|
||||
* System prompt used by `reasonix code`. Teaches the model:
|
||||
*
|
||||
* 1. It has a filesystem MCP bridge rooted at the user's CWD.
|
||||
* 2. To modify files it emits SEARCH/REPLACE blocks (not
|
||||
* `write_file` — that would whole-file rewrite and kill diff
|
||||
* reviewability).
|
||||
* 3. Read first, edit second — SEARCH must match byte-for-byte.
|
||||
* 4. Be concise. The user can read a diff faster than prose.
|
||||
*
|
||||
* Kept short on purpose. Long system prompts eat context budget that
|
||||
* the Cache-First Loop is trying to conserve. The SEARCH/REPLACE spec
|
||||
* is the one unavoidable bloat; we trim everything else.
|
||||
*/
|
||||
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
export const CODE_SYSTEM_PROMPT = `You are Reasonix Code, a coding assistant. You have filesystem tools (read_file, write_file, list_directory, search_files, etc.) rooted at the user's working directory.
|
||||
|
||||
# When to edit vs. when to explore
|
||||
|
||||
Only propose edits when the user explicitly asks you to change, fix, add, remove, refactor, or write something. Do NOT propose edits when the user asks you to:
|
||||
- analyze, read, explore, describe, or summarize a project
|
||||
- explain how something works
|
||||
- answer a question about the code
|
||||
|
||||
In those cases, use tools to gather what you need, then reply in prose. No SEARCH/REPLACE blocks, no file changes. If you're unsure what the user wants, ask.
|
||||
|
||||
When you do propose edits, the user will review them and decide whether to \`/apply\` or \`/discard\`. Don't assume they'll accept — write as if each edit will be audited, because it will.
|
||||
|
||||
# Editing files
|
||||
|
||||
When you've been asked to change a file, output one or more SEARCH/REPLACE blocks in this exact format:
|
||||
|
||||
path/to/file.ext
|
||||
<<<<<<< SEARCH
|
||||
exact existing lines from the file, including whitespace
|
||||
=======
|
||||
the new lines
|
||||
>>>>>>> REPLACE
|
||||
|
||||
Rules:
|
||||
- Always read_file first so your SEARCH matches byte-for-byte. If it doesn't match, the edit is rejected and you'll have to retry with the exact current content.
|
||||
- One edit per block. Multiple blocks in one response are fine.
|
||||
- To create a new file, leave SEARCH empty:
|
||||
path/to/new.ts
|
||||
<<<<<<< SEARCH
|
||||
=======
|
||||
(whole file content here)
|
||||
>>>>>>> REPLACE
|
||||
- Do NOT use write_file to change existing files — the user reviews your edits as SEARCH/REPLACE. write_file is only for files you explicitly want to overwrite wholesale (rare).
|
||||
- Paths are relative to the working directory. Don't use absolute paths.
|
||||
|
||||
# Exploration
|
||||
|
||||
- Avoid listing or reading inside these common dependency / build directories unless the user explicitly asks about them: node_modules, dist, build, out, .next, .nuxt, .svelte-kit, .git, .venv, venv, __pycache__, target, coverage, .turbo, .cache. They're expensive and usually irrelevant.
|
||||
- Prefer search_files / grep over list_directory when you know roughly what you're looking for — it saves context and avoids enumerating huge trees.
|
||||
|
||||
# Style
|
||||
|
||||
- Show edits; don't narrate them in prose. "Here's the fix:" is enough.
|
||||
- One short paragraph explaining *why*, then the blocks.
|
||||
- If you need to explore first (list / grep / read), do it with tool calls before writing any prose — silence while exploring is fine.
|
||||
`;
|
||||
|
||||
/**
|
||||
* Inject the project's `.gitignore` content into the system prompt as a
|
||||
* "respect this on top of the built-in denylist" hint. We don't parse
|
||||
* the file — we hand it to the model as-is. Truncate long ones so we
|
||||
* don't eat context budget on huge generated ignore lists.
|
||||
*
|
||||
* Missing or unreadable .gitignore → returns the base prompt unchanged.
|
||||
*/
|
||||
export function codeSystemPrompt(rootDir: string): string {
|
||||
const gitignorePath = join(rootDir, ".gitignore");
|
||||
if (!existsSync(gitignorePath)) return CODE_SYSTEM_PROMPT;
|
||||
let content: string;
|
||||
try {
|
||||
content = readFileSync(gitignorePath, "utf8");
|
||||
} catch {
|
||||
return CODE_SYSTEM_PROMPT;
|
||||
}
|
||||
const MAX = 2000;
|
||||
const truncated =
|
||||
content.length > MAX
|
||||
? `${content.slice(0, MAX)}\n… (truncated ${content.length - MAX} chars)`
|
||||
: content;
|
||||
return `${CODE_SYSTEM_PROMPT}
|
||||
|
||||
# Project .gitignore
|
||||
|
||||
The user's repo ships this .gitignore — treat every pattern as "don't traverse or edit inside these paths unless explicitly asked":
|
||||
|
||||
\`\`\`
|
||||
${truncated}
|
||||
\`\`\`
|
||||
`;
|
||||
}
|
||||
@@ -8,15 +8,42 @@
|
||||
* The library itself never touches the config file — it only reads
|
||||
* `DEEPSEEK_API_KEY` from the environment. The CLI is responsible for
|
||||
* pulling from the config file and exposing it via env var to the loop.
|
||||
*
|
||||
* Beyond the API key, the config also remembers the user's *defaults*
|
||||
* from `reasonix setup`: preset, MCP servers, session. This is what
|
||||
* makes `reasonix chat` with no flags "just work" after first-run.
|
||||
*/
|
||||
|
||||
import { chmodSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { dirname, join } from "node:path";
|
||||
|
||||
/** One of the preset bundles (model + harvest + branch combo). */
|
||||
export type PresetName = "fast" | "smart" | "max";
|
||||
|
||||
export interface ReasonixConfig {
|
||||
apiKey?: string;
|
||||
baseUrl?: string;
|
||||
/**
|
||||
* Default preset for `reasonix chat` / `reasonix run` when no flags override.
|
||||
* Maps to model + harvest + branch combos (see presets.ts). Missing → "fast".
|
||||
*/
|
||||
preset?: PresetName;
|
||||
/**
|
||||
* Default MCP server specs to bridge on every `reasonix chat`, in the
|
||||
* same `"name=cmd args..."` format that `--mcp` takes. Stored as strings
|
||||
* so `reasonix setup` stays symmetrical with the flag — one parser, one
|
||||
* format in the config file, grep-friendly.
|
||||
*/
|
||||
mcp?: string[];
|
||||
/**
|
||||
* Default session name (null/missing → "default", which is what the
|
||||
* CLI has been doing anyway). `reasonix setup` lets users pick a name
|
||||
* or opt into ephemeral.
|
||||
*/
|
||||
session?: string | null;
|
||||
/** Marks that `reasonix setup` has completed at least once. */
|
||||
setupCompleted?: boolean;
|
||||
}
|
||||
|
||||
export function defaultConfigPath(): string {
|
||||
|
||||
+74
@@ -53,6 +53,32 @@ export interface DiffReport {
|
||||
firstDivergenceTurn: number | null;
|
||||
}
|
||||
|
||||
// ---------- navigation helpers (used by the Ink diff TUI) ----------
|
||||
|
||||
/**
|
||||
* Find the next pair (strictly after `fromIdx`) whose kind is not "match".
|
||||
* Returns -1 when no later divergence exists. Used by DiffApp's `n` key.
|
||||
*/
|
||||
export function findNextDivergence(pairs: TurnPair[], fromIdx: number): number {
|
||||
for (let i = fromIdx + 1; i < pairs.length; i++) {
|
||||
if (pairs[i]!.kind !== "match") return i;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the previous pair (strictly before `fromIdx`) whose kind is not
|
||||
* "match". Returns -1 when no earlier divergence exists. Used by
|
||||
* DiffApp's `N` / `p` key.
|
||||
*/
|
||||
export function findPrevDivergence(pairs: TurnPair[], fromIdx: number): number {
|
||||
const start = Math.min(fromIdx - 1, pairs.length - 1);
|
||||
for (let i = start; i >= 0; i--) {
|
||||
if (pairs[i]!.kind !== "match") return i;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
export function diffTranscripts(
|
||||
a: { label: string; parsed: ReadTranscriptResult },
|
||||
b: { label: string; parsed: ReadTranscriptResult },
|
||||
@@ -245,6 +271,43 @@ export function renderSummaryTable(report: DiffReport, _opts: RenderOptions = {}
|
||||
),
|
||||
);
|
||||
lines.push(statRow("prefix hashes", a.stats.prefixHashes.length, b.stats.prefixHashes.length));
|
||||
// Harvest signal row — only surface when at least one side carries plan
|
||||
// state. Keeps no-harvest diffs visually identical to pre-v0.3 output.
|
||||
if (a.stats.harvestedTurns > 0 || b.stats.harvestedTurns > 0) {
|
||||
lines.push(
|
||||
row(
|
||||
[
|
||||
"harvest turns",
|
||||
`${a.stats.harvestedTurns}`,
|
||||
`${b.stats.harvestedTurns}`,
|
||||
signed(b.stats.harvestedTurns - a.stats.harvestedTurns),
|
||||
],
|
||||
[20, 14, 14, 14],
|
||||
),
|
||||
);
|
||||
lines.push(
|
||||
row(
|
||||
[
|
||||
" subgoals",
|
||||
`${a.stats.totalSubgoals}`,
|
||||
`${b.stats.totalSubgoals}`,
|
||||
signed(b.stats.totalSubgoals - a.stats.totalSubgoals),
|
||||
],
|
||||
[20, 14, 14, 14],
|
||||
),
|
||||
);
|
||||
lines.push(
|
||||
row(
|
||||
[
|
||||
" uncertainties",
|
||||
`${a.stats.totalUncertainties}`,
|
||||
`${b.stats.totalUncertainties}`,
|
||||
signed(b.stats.totalUncertainties - a.stats.totalUncertainties),
|
||||
],
|
||||
[20, 14, 14, 14],
|
||||
),
|
||||
);
|
||||
}
|
||||
lines.push("");
|
||||
|
||||
// Prefix stability story — the headline finding when comparing bench modes.
|
||||
@@ -322,6 +385,17 @@ export function renderMarkdown(report: DiffReport): string {
|
||||
out.push(
|
||||
`| prefix hashes | ${a.stats.prefixHashes.length} | ${b.stats.prefixHashes.length} | — |`,
|
||||
);
|
||||
if (a.stats.harvestedTurns > 0 || b.stats.harvestedTurns > 0) {
|
||||
out.push(
|
||||
`| harvest turns | ${a.stats.harvestedTurns} | ${b.stats.harvestedTurns} | ${signed(b.stats.harvestedTurns - a.stats.harvestedTurns)} |`,
|
||||
);
|
||||
out.push(
|
||||
`| harvest subgoals | ${a.stats.totalSubgoals} | ${b.stats.totalSubgoals} | ${signed(b.stats.totalSubgoals - a.stats.totalSubgoals)} |`,
|
||||
);
|
||||
out.push(
|
||||
`| harvest uncertainties | ${a.stats.totalUncertainties} | ${b.stats.totalUncertainties} | ${signed(b.stats.totalUncertainties - a.stats.totalUncertainties)} |`,
|
||||
);
|
||||
}
|
||||
out.push("");
|
||||
|
||||
out.push("## Turn-by-turn");
|
||||
|
||||
+69
-2
@@ -3,7 +3,12 @@
|
||||
export { DeepSeekClient, Usage } from "./client.js";
|
||||
export type { ChatResponse, StreamChunk, DeepSeekClientOptions } from "./client.js";
|
||||
|
||||
export { CacheFirstLoop } from "./loop.js";
|
||||
export {
|
||||
CacheFirstLoop,
|
||||
formatLoopError,
|
||||
healLoadedMessages,
|
||||
stripHallucinatedToolMarkup,
|
||||
} from "./loop.js";
|
||||
export type {
|
||||
CacheFirstLoopOptions,
|
||||
LoopEvent,
|
||||
@@ -85,6 +90,68 @@ export {
|
||||
} from "./diff.js";
|
||||
export type { DiffReport, DiffSide, TurnPair, RenderOptions as DiffRenderOptions } from "./diff.js";
|
||||
|
||||
// ---------- MCP (v0.3 foundation) ----------
|
||||
export { McpClient } from "./mcp/client.js";
|
||||
export type { McpClientOptions } from "./mcp/client.js";
|
||||
export { StdioTransport } from "./mcp/stdio.js";
|
||||
export type { McpTransport, StdioTransportOptions } from "./mcp/stdio.js";
|
||||
export { SseTransport } from "./mcp/sse.js";
|
||||
export type { SseTransportOptions } from "./mcp/sse.js";
|
||||
export {
|
||||
DEFAULT_MAX_RESULT_CHARS,
|
||||
bridgeMcpTools,
|
||||
flattenMcpResult,
|
||||
truncateForModel,
|
||||
} from "./mcp/registry.js";
|
||||
export type { BridgeOptions, BridgeResult, FlattenOptions } from "./mcp/registry.js";
|
||||
export { parseMcpSpec } from "./mcp/spec.js";
|
||||
export type { McpSpec, StdioMcpSpec, SseMcpSpec } from "./mcp/spec.js";
|
||||
export { inspectMcpServer } from "./mcp/inspect.js";
|
||||
export type { InspectionReport, SectionResult } from "./mcp/inspect.js";
|
||||
|
||||
// ---------- code mode (v0.3 — `reasonix code`) ----------
|
||||
export {
|
||||
parseEditBlocks,
|
||||
applyEditBlock,
|
||||
applyEditBlocks,
|
||||
snapshotBeforeEdits,
|
||||
restoreSnapshots,
|
||||
} from "./code/edit-blocks.js";
|
||||
export type {
|
||||
EditBlock,
|
||||
ApplyResult,
|
||||
ApplyStatus,
|
||||
EditSnapshot,
|
||||
} from "./code/edit-blocks.js";
|
||||
export { CODE_SYSTEM_PROMPT, codeSystemPrompt } from "./code/prompt.js";
|
||||
export {
|
||||
MCP_PROTOCOL_VERSION,
|
||||
isJsonRpcError,
|
||||
} from "./mcp/types.js";
|
||||
export type {
|
||||
McpTool,
|
||||
McpToolSchema,
|
||||
CallToolResult,
|
||||
ListToolsResult,
|
||||
McpContentBlock,
|
||||
InitializeResult,
|
||||
JsonRpcRequest,
|
||||
JsonRpcResponse,
|
||||
JsonRpcMessage,
|
||||
McpResource,
|
||||
McpResourceContents,
|
||||
McpResourceContentsText,
|
||||
McpResourceContentsBlob,
|
||||
ListResourcesResult,
|
||||
ReadResourceResult,
|
||||
McpPrompt,
|
||||
McpPromptArgument,
|
||||
McpPromptMessage,
|
||||
McpPromptResourceBlock,
|
||||
ListPromptsResult,
|
||||
GetPromptResult,
|
||||
} from "./mcp/types.js";
|
||||
|
||||
export { fetchWithRetry } from "./retry.js";
|
||||
export type { RetryOptions, RetryInfo } from "./retry.js";
|
||||
|
||||
@@ -108,4 +175,4 @@ export type {
|
||||
JSONSchema,
|
||||
} from "./types.js";
|
||||
|
||||
export const VERSION = "0.2.0";
|
||||
export const VERSION = "0.4.3";
|
||||
+359
-7
@@ -6,19 +6,34 @@ import {
|
||||
runBranches,
|
||||
} from "./consistency.js";
|
||||
import { type HarvestOptions, type TypedPlanState, emptyPlanState, harvest } from "./harvest.js";
|
||||
import { DEFAULT_MAX_RESULT_CHARS, truncateForModel } from "./mcp/registry.js";
|
||||
import { AppendOnlyLog, type ImmutablePrefix, VolatileScratch } from "./memory.js";
|
||||
import { type RepairReport, ToolCallRepair } from "./repair/index.js";
|
||||
import { appendSessionMessage, loadSessionMessages } from "./session.js";
|
||||
import { SessionStats, type TurnStats } from "./telemetry.js";
|
||||
import { appendSessionMessage, loadSessionMessages, rewriteSession } from "./session.js";
|
||||
import {
|
||||
DEEPSEEK_CONTEXT_TOKENS,
|
||||
DEFAULT_CONTEXT_TOKENS,
|
||||
SessionStats,
|
||||
type TurnStats,
|
||||
} from "./telemetry.js";
|
||||
import { ToolRegistry } from "./tools.js";
|
||||
import type { ChatMessage, ToolCall } from "./types.js";
|
||||
|
||||
export type EventRole =
|
||||
| "assistant_delta"
|
||||
| "assistant_final"
|
||||
/**
|
||||
* Yielded immediately before a tool is dispatched. Lets the TUI put
|
||||
* up a "▸ tool<X> running…" spinner while the tool's Promise is
|
||||
* pending — otherwise the UI looks frozen whenever a tool call
|
||||
* takes more than a few hundred ms (a big `filesystem_edit_file`
|
||||
* is a typical trigger).
|
||||
*/
|
||||
| "tool_start"
|
||||
| "tool"
|
||||
| "done"
|
||||
| "error"
|
||||
| "warning"
|
||||
| "branch_start"
|
||||
| "branch_progress"
|
||||
| "branch_done";
|
||||
@@ -56,6 +71,16 @@ export interface LoopEvent {
|
||||
branch?: BranchSummary;
|
||||
branchProgress?: BranchProgress;
|
||||
error?: string;
|
||||
/**
|
||||
* True on `assistant_final` events emitted by the no-tools fallback
|
||||
* when the loop hit its budget, was aborted, or tripped the
|
||||
* token-context guard. Consumers that act on assistant text (notably
|
||||
* the code-mode edit applier) MUST treat these as display-only —
|
||||
* the model is "wrapping up," not proposing new work. Applying
|
||||
* SEARCH/REPLACE blocks found in a forced summary caused the
|
||||
* "analysis became edits" bug in v0.4.1 and earlier.
|
||||
*/
|
||||
forcedSummary?: boolean;
|
||||
}
|
||||
|
||||
export interface CacheFirstLoopOptions {
|
||||
@@ -127,13 +152,28 @@ export class CacheFirstLoop {
|
||||
|
||||
private _turn = 0;
|
||||
private _streamPreference: boolean;
|
||||
/**
|
||||
* Set by {@link abort} to short-circuit the tool-call loop after the
|
||||
* current iteration. Reset at the start of each `step()` so an Esc
|
||||
* during one turn doesn't poison the next.
|
||||
*/
|
||||
private _aborted = false;
|
||||
|
||||
constructor(opts: CacheFirstLoopOptions) {
|
||||
this.client = opts.client;
|
||||
this.prefix = opts.prefix;
|
||||
this.tools = opts.tools ?? new ToolRegistry();
|
||||
this.model = opts.model ?? "deepseek-chat";
|
||||
this.maxToolIters = opts.maxToolIters ?? 8;
|
||||
// Iter cap is a safety net, not the primary stop condition. The
|
||||
// primary stop is the token-context guard inside step(): after
|
||||
// every model response we check whether the prompt is already past
|
||||
// 80% of the model's context window, and if so divert to the
|
||||
// forced-summary path. 64 is high enough that exploration almost
|
||||
// never exhausts it before the token guard fires first — which
|
||||
// is the point: let the real constraint (context window) drive
|
||||
// the decision, keep the iter cap as a last-resort backstop for
|
||||
// the case where something spins without growing the prompt.
|
||||
this.maxToolIters = opts.maxToolIters ?? 64;
|
||||
|
||||
// Resolve branch config first (since it forces harvest on).
|
||||
if (typeof opts.branch === "number") {
|
||||
@@ -165,16 +205,62 @@ export class CacheFirstLoop {
|
||||
|
||||
// Session resume: pre-load prior messages into the log if a session name
|
||||
// is provided. New messages appended to the log are also persisted.
|
||||
//
|
||||
// Heal-on-load: if a previous run (or a pre-alpha.6 client) stored a
|
||||
// tool result bigger than the cap, the next API call would blow past
|
||||
// DeepSeek's 131k-token limit *before the user even types anything*.
|
||||
// Truncating here lets the user pick up their session history without
|
||||
// losing the conversational context around the oversized call.
|
||||
this.sessionName = opts.session ?? null;
|
||||
if (this.sessionName) {
|
||||
const prior = loadSessionMessages(this.sessionName);
|
||||
for (const msg of prior) this.log.append(msg);
|
||||
this.resumedMessageCount = prior.length;
|
||||
const { messages, healedCount, healedFrom } = healLoadedMessages(
|
||||
prior,
|
||||
DEFAULT_MAX_RESULT_CHARS,
|
||||
);
|
||||
for (const msg of messages) this.log.append(msg);
|
||||
this.resumedMessageCount = messages.length;
|
||||
if (healedCount > 0) {
|
||||
process.stderr.write(
|
||||
`▸ session "${this.sessionName}": healed ${healedCount} oversized tool result(s) (was ${healedFrom.toLocaleString()} chars total). Old payloads were truncated to fit DeepSeek's context window; the conversation is preserved.\n`,
|
||||
);
|
||||
}
|
||||
} else {
|
||||
this.resumedMessageCount = 0;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Shrink the log by re-truncating oversized tool results to a tighter
|
||||
* cap, and persist the result back to disk so the next launch doesn't
|
||||
* re-inherit a fat session file. Returns a summary the TUI can
|
||||
* display.
|
||||
*
|
||||
* Only tool-role messages are touched (same rationale as
|
||||
* {@link healLoadedMessages}). User and assistant messages carry
|
||||
* authored intent we can't mechanically shrink without losing
|
||||
* meaning.
|
||||
*/
|
||||
compact(tightCapChars = 4000): { healedCount: number; charsSaved: number } {
|
||||
const before = this.log.toMessages();
|
||||
const { messages, healedCount, healedFrom } = healLoadedMessages(before, tightCapChars);
|
||||
const afterBytes = messages
|
||||
.filter((m) => m.role === "tool")
|
||||
.reduce((s, m) => s + (typeof m.content === "string" ? m.content.length : 0), 0);
|
||||
const charsSaved = healedFrom - afterBytes;
|
||||
if (healedCount > 0) {
|
||||
this.log.compactInPlace(messages);
|
||||
if (this.sessionName) {
|
||||
try {
|
||||
rewriteSession(this.sessionName, messages);
|
||||
} catch {
|
||||
/* disk full or perms — compaction still applies in-memory */
|
||||
}
|
||||
}
|
||||
}
|
||||
return { healedCount, charsSaved };
|
||||
}
|
||||
|
||||
private appendAndPersist(message: ChatMessage): void {
|
||||
this.log.append(message);
|
||||
if (this.sessionName) {
|
||||
@@ -229,13 +315,107 @@ export class CacheFirstLoop {
|
||||
return msgs;
|
||||
}
|
||||
|
||||
/**
|
||||
* Signal the currently-running {@link step} that the user wants to
|
||||
* stop exploring. Takes effect at the next iteration boundary — if a
|
||||
* tool call is mid-flight it will be allowed to finish, then the
|
||||
* loop diverts to the forced-summary path so the user gets an
|
||||
* answer instead of a cliff. Called by the TUI on Esc.
|
||||
*/
|
||||
abort(): void {
|
||||
this._aborted = true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop everything in the log after (and including) the most recent
|
||||
* user message. Used by `/retry` so the caller can re-send that
|
||||
* message with a fresh turn instead of layering another response on
|
||||
* top of the prior exchange. Returns the content of the dropped user
|
||||
* message, or `null` if there isn't one yet.
|
||||
*
|
||||
* Persists by rewriting the session file — otherwise the next
|
||||
* launch would rehydrate the old exchange and `/retry` would seem
|
||||
* to have done nothing.
|
||||
*/
|
||||
retryLastUser(): string | null {
|
||||
const entries = this.log.entries;
|
||||
let lastUserIdx = -1;
|
||||
for (let i = entries.length - 1; i >= 0; i--) {
|
||||
if (entries[i]!.role === "user") {
|
||||
lastUserIdx = i;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (lastUserIdx < 0) return null;
|
||||
const raw = entries[lastUserIdx]!.content;
|
||||
const userText = typeof raw === "string" ? raw : "";
|
||||
// Keep everything strictly before the user message. The caller
|
||||
// will submit the text again through the normal path, which
|
||||
// re-appends the user turn on first successful API response.
|
||||
const preserved = entries.slice(0, lastUserIdx).map((m) => ({ ...m }));
|
||||
this.log.compactInPlace(preserved);
|
||||
if (this.sessionName) {
|
||||
try {
|
||||
rewriteSession(this.sessionName, preserved);
|
||||
} catch {
|
||||
/* disk-full / perms — in-memory compaction still applies */
|
||||
}
|
||||
}
|
||||
return userText;
|
||||
}
|
||||
|
||||
async *step(userInput: string): AsyncGenerator<LoopEvent> {
|
||||
this._turn++;
|
||||
this.scratch.reset();
|
||||
this._aborted = false;
|
||||
let pendingUser: string | null = userInput;
|
||||
const toolSpecs = this.prefix.tools();
|
||||
// 70% of the iter budget is the "you're getting close" threshold. We
|
||||
// only warn once per step so the user sees a single signal, not a
|
||||
// string of identical yellow lines stacked up.
|
||||
const warnAt = Math.max(1, Math.floor(this.maxToolIters * 0.7));
|
||||
let warnedForIterBudget = false;
|
||||
|
||||
for (let iter = 0; iter < this.maxToolIters; iter++) {
|
||||
if (this._aborted) {
|
||||
// Esc means "stop now" — not "stop and force another 30-90s
|
||||
// reasoner call to produce a summary I didn't ask for". The
|
||||
// user's mental model of cancel is immediate. We emit a
|
||||
// synthetic assistant_final (tagged forcedSummary so the
|
||||
// code-mode applier ignores it) with a short stopped
|
||||
// message, then done. The prior tool outputs are still in
|
||||
// the log if the user wants to continue — asking again
|
||||
// will hit a warm cache and be cheap.
|
||||
//
|
||||
// Budget / context-guard still call forceSummaryAfterIterLimit
|
||||
// because there the USER didn't choose to stop — we did —
|
||||
// and leaving them staring at nothing is worse than one extra
|
||||
// call.
|
||||
yield {
|
||||
turn: this._turn,
|
||||
role: "warning",
|
||||
content: `aborted at iter ${iter}/${this.maxToolIters} — stopped without producing a summary (press ↑ + Enter or /retry to resume)`,
|
||||
};
|
||||
const stoppedMsg =
|
||||
"[aborted by user (Esc) — no summary produced. Ask again or /retry when ready; prior tool output is still in the log.]";
|
||||
this.appendAndPersist({ role: "assistant", content: stoppedMsg });
|
||||
yield {
|
||||
turn: this._turn,
|
||||
role: "assistant_final",
|
||||
content: stoppedMsg,
|
||||
forcedSummary: true,
|
||||
};
|
||||
yield { turn: this._turn, role: "done", content: stoppedMsg };
|
||||
return;
|
||||
}
|
||||
if (!warnedForIterBudget && iter >= warnAt) {
|
||||
warnedForIterBudget = true;
|
||||
yield {
|
||||
turn: this._turn,
|
||||
role: "warning",
|
||||
content: `${iter}/${this.maxToolIters} tool calls used — approaching budget. Press Esc to force a summary now.`,
|
||||
};
|
||||
}
|
||||
const messages = this.buildMessages(pendingUser);
|
||||
|
||||
let assistantContent = "";
|
||||
@@ -391,7 +571,7 @@ export class CacheFirstLoop {
|
||||
turn: this._turn,
|
||||
role: "error",
|
||||
content: "",
|
||||
error: (err as Error).message,
|
||||
error: formatLoopError(err as Error),
|
||||
};
|
||||
return;
|
||||
}
|
||||
@@ -414,6 +594,7 @@ export class CacheFirstLoop {
|
||||
const { calls: repairedCalls, report } = this.repair.process(
|
||||
toolCalls,
|
||||
reasoningContent || null,
|
||||
assistantContent || null,
|
||||
);
|
||||
|
||||
this.appendAndPersist(this.assistantMessage(assistantContent, repairedCalls));
|
||||
@@ -433,9 +614,42 @@ export class CacheFirstLoop {
|
||||
return;
|
||||
}
|
||||
|
||||
// Token-budget guard — the real stop condition. Iter count is a
|
||||
// proxy that misses the actual constraint: how close the prompt
|
||||
// already is to DeepSeek's 131k context. If we're over 80%, the
|
||||
// NEXT call (with the just-executed tools' results stuffed into
|
||||
// history) will be worse, and fairly soon it'll 400 with
|
||||
// "maximum context length". Divert to summary instead — same
|
||||
// path as iter-cap and Esc-abort, just triggered by a different
|
||||
// signal so the user sees *why* it happened.
|
||||
const ctxMax = DEEPSEEK_CONTEXT_TOKENS[this.model] ?? DEFAULT_CONTEXT_TOKENS;
|
||||
if (usage && usage.promptTokens / ctxMax > 0.8) {
|
||||
yield {
|
||||
turn: this._turn,
|
||||
role: "warning",
|
||||
content: `context ${usage.promptTokens}/${ctxMax} (${Math.round(
|
||||
(usage.promptTokens / ctxMax) * 100,
|
||||
)}%) — more tools would overflow. Forcing summary from what was gathered.`,
|
||||
};
|
||||
yield* this.forceSummaryAfterIterLimit({ reason: "context-guard" });
|
||||
return;
|
||||
}
|
||||
|
||||
for (const call of repairedCalls) {
|
||||
const name = call.function?.name ?? "";
|
||||
const args = call.function?.arguments ?? "{}";
|
||||
// Announce the tool BEFORE awaiting it so the TUI can render a
|
||||
// "running…" indicator. Without this, the window between
|
||||
// assistant_final and the tool-result yield is silent from the
|
||||
// UI's perspective — which makes long tool calls feel like the
|
||||
// app has hung.
|
||||
yield {
|
||||
turn: this._turn,
|
||||
role: "tool_start",
|
||||
content: "",
|
||||
toolName: name,
|
||||
toolArgs: args,
|
||||
};
|
||||
const result = await this.tools.dispatch(name, args);
|
||||
this.appendAndPersist({
|
||||
role: "tool",
|
||||
@@ -453,7 +667,62 @@ export class CacheFirstLoop {
|
||||
}
|
||||
}
|
||||
|
||||
yield { turn: this._turn, role: "done", content: "[max_tool_iters reached]" };
|
||||
// We exhausted the tool-call budget while the model still wanted to
|
||||
// call more tools. Rather than stopping silently (which leaves the
|
||||
// user staring at a blank prompt), force one final no-tools call so
|
||||
// the model must produce a text summary from everything it has
|
||||
// already seen.
|
||||
yield* this.forceSummaryAfterIterLimit({ reason: "budget" });
|
||||
}
|
||||
|
||||
private async *forceSummaryAfterIterLimit(
|
||||
opts: { reason: "budget" | "aborted" | "context-guard" } = { reason: "budget" },
|
||||
): AsyncGenerator<LoopEvent> {
|
||||
try {
|
||||
const messages = this.buildMessages(null);
|
||||
// Passing `tools: undefined` was supposed to force a text
|
||||
// response, but R1 can still hallucinate tool-call markup
|
||||
// (e.g. DSML `<|DSML|function_calls>…</|DSML|function_calls>`)
|
||||
// in prose when it's been primed by prior tool use. An explicit
|
||||
// user-role instruction plus post-hoc stripping of known
|
||||
// hallucination shapes keeps the user from seeing raw markup.
|
||||
messages.push({
|
||||
role: "user",
|
||||
content:
|
||||
"I'm out of tool-call budget for this turn. Summarize in plain prose what you learned from the tool results above. Do NOT emit any tool calls, function-call markup, DSML invocations, or SEARCH/REPLACE edit blocks — they will be silently discarded. Just plain text.",
|
||||
});
|
||||
const resp = await this.client.chat({
|
||||
model: this.model,
|
||||
messages,
|
||||
// no tools → model is forced to answer in text
|
||||
});
|
||||
const rawContent = resp.content?.trim() ?? "";
|
||||
const cleaned = stripHallucinatedToolMarkup(rawContent);
|
||||
const summary =
|
||||
cleaned ||
|
||||
"(model emitted fake tool-call markup instead of a prose summary — try /retry with a narrower question, or /think to inspect R1's reasoning)";
|
||||
const reasonPrefix = reasonPrefixFor(opts.reason, this.maxToolIters);
|
||||
const annotated = `${reasonPrefix}\n\n${summary}`;
|
||||
const summaryStats = this.stats.record(this._turn, this.model, resp.usage ?? new Usage());
|
||||
this.appendAndPersist({ role: "assistant", content: summary });
|
||||
yield {
|
||||
turn: this._turn,
|
||||
role: "assistant_final",
|
||||
content: annotated,
|
||||
stats: summaryStats,
|
||||
forcedSummary: true,
|
||||
};
|
||||
yield { turn: this._turn, role: "done", content: summary };
|
||||
} catch (err) {
|
||||
const label = errorLabelFor(opts.reason, this.maxToolIters);
|
||||
yield {
|
||||
turn: this._turn,
|
||||
role: "error",
|
||||
content: "",
|
||||
error: `${label} and the fallback summary call failed: ${(err as Error).message}. Run /clear and retry with a narrower question, or raise --max-tool-iters.`,
|
||||
};
|
||||
yield { turn: this._turn, role: "done", content: "" };
|
||||
}
|
||||
}
|
||||
|
||||
async run(userInput: string, onEvent?: (ev: LoopEvent) => void): Promise<string> {
|
||||
@@ -473,6 +742,45 @@ export class CacheFirstLoop {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* R1 occasionally hallucinates tool-call markup as plain text when the
|
||||
* real tool channel has been closed — typically our forced-summary
|
||||
* path, where `tools: undefined` is supposed to force prose but isn't
|
||||
* always respected. The markup isn't parsed by our tool-call path
|
||||
* (the API response's structured `tool_calls` field is empty), so
|
||||
* it's just noise in the user's view. Strip known envelope shapes.
|
||||
*
|
||||
* Exported so tests can exercise it against concrete R1 outputs.
|
||||
*/
|
||||
export function stripHallucinatedToolMarkup(s: string): string {
|
||||
let out = s;
|
||||
// DeepSeek's DSML envelope (both the full-width "|" character and
|
||||
// the ASCII-only fallback we've seen — the full-width form is the
|
||||
// one R1 emits in practice)
|
||||
out = out.replace(/<|DSML|function_calls>[\s\S]*?<\/?|DSML|function_calls>/g, "");
|
||||
out = out.replace(/<\|DSML\|function_calls>[\s\S]*?<\/?\|DSML\|function_calls>/g, "");
|
||||
// Anthropic / generic XML-ish envelope
|
||||
out = out.replace(/<function_calls>[\s\S]*?<\/function_calls>/g, "");
|
||||
// Lone unpaired DSML opener left over after the closer was on a
|
||||
// different line (seen when R1 truncates mid-call).
|
||||
out = out.replace(/<|DSML|[\s\S]*$/g, "");
|
||||
return out.trim();
|
||||
}
|
||||
|
||||
function reasonPrefixFor(reason: "budget" | "aborted" | "context-guard", iterCap: number): string {
|
||||
if (reason === "aborted") return "[aborted by user (Esc) — summarizing what I found so far]";
|
||||
if (reason === "context-guard") {
|
||||
return "[context budget running low — summarizing before the next call would overflow]";
|
||||
}
|
||||
return `[tool-call budget (${iterCap}) reached — forcing summary from what I found]`;
|
||||
}
|
||||
|
||||
function errorLabelFor(reason: "budget" | "aborted" | "context-guard", iterCap: number): string {
|
||||
if (reason === "aborted") return "aborted by user";
|
||||
if (reason === "context-guard") return "context-guard triggered (prompt > 80% of window)";
|
||||
return `tool-call budget (${iterCap}) reached`;
|
||||
}
|
||||
|
||||
function summarizeBranch(chosen: BranchSample, samples: BranchSample[]): BranchSummary {
|
||||
return {
|
||||
budget: samples.length,
|
||||
@@ -481,3 +789,47 @@ function summarizeBranch(chosen: BranchSample, samples: BranchSample[]): BranchS
|
||||
temperatures: samples.map((s) => s.temperature),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Truncate any tool-role message whose content exceeds the cap. User
|
||||
* and assistant messages are left alone because (a) they're almost
|
||||
* always small, (b) truncating user prompts would corrupt conversational
|
||||
* intent in a way the user didn't author. Exported for tests.
|
||||
*/
|
||||
export function healLoadedMessages(
|
||||
messages: ChatMessage[],
|
||||
maxChars: number,
|
||||
): { messages: ChatMessage[]; healedCount: number; healedFrom: number } {
|
||||
let healedCount = 0;
|
||||
let healedFrom = 0;
|
||||
const out = messages.map((msg) => {
|
||||
if (msg.role !== "tool") return msg;
|
||||
const content = typeof msg.content === "string" ? msg.content : "";
|
||||
if (content.length <= maxChars) return msg;
|
||||
healedCount += 1;
|
||||
healedFrom += content.length;
|
||||
return { ...msg, content: truncateForModel(content, maxChars) };
|
||||
});
|
||||
return { messages: out, healedCount, healedFrom };
|
||||
}
|
||||
|
||||
/**
|
||||
* Annotate the `DeepSeek 400: … maximum context length …` error the API
|
||||
* returns when a session's history has grown past 131,072 tokens. The
|
||||
* raw message is a JSON blob; we surface a short actionable hint on top
|
||||
* so the user knows to `/forget` or `/clear` rather than parsing the
|
||||
* JSON themselves. Other errors pass through unchanged — the loop's
|
||||
* error channel already formats them well enough.
|
||||
*/
|
||||
export function formatLoopError(err: Error): string {
|
||||
const msg = err.message ?? "";
|
||||
if (msg.includes("maximum context length")) {
|
||||
// Pull the "requested X tokens" figure out of the JSON for scale.
|
||||
const reqMatch = msg.match(/requested\s+(\d+)\s+tokens/);
|
||||
const requested = reqMatch
|
||||
? `${Number(reqMatch[1]).toLocaleString()} tokens`
|
||||
: "too many tokens";
|
||||
return `Context overflow (DeepSeek 400): session history is ${requested}, past the 131,072-token limit. Usually this means a single tool call returned a huge payload. v0.3.0-alpha.6+ caps new tool results at 32k chars, AND auto-heals oversized history on session load — restart Reasonix and this session should come back trimmed. If it still overflows, run /forget (delete the session) or /clear (drop the displayed history) to start fresh.`;
|
||||
}
|
||||
return msg;
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
# MCP client (v0.3 foundation)
|
||||
|
||||
Minimal [Model Context Protocol](https://spec.modelcontextprotocol.io/)
|
||||
client, hand-rolled in TypeScript. Lets Reasonix consume tools from any
|
||||
MCP server (filesystem, github, slack, puppeteer, …) while applying the
|
||||
Cache-First Loop and tool-call repair to the whole thing automatically.
|
||||
|
||||
## Design choice: roll-our-own, not @modelcontextprotocol/sdk
|
||||
|
||||
Same reasoning that drove `client.ts` (DeepSeek) rather than `openai`:
|
||||
|
||||
- **Zero runtime deps** for this module. Consistent with Reasonix's
|
||||
policy of owning the wire format where it matters.
|
||||
- **Surface tuning**: we only implement what Reasonix actually uses —
|
||||
initialize + tools/list + tools/call. Resources, prompts, sampling,
|
||||
and progress notifications are deferred.
|
||||
- **Insulation** from SDK breaking changes. The spec is more stable
|
||||
than any single SDK release.
|
||||
|
||||
Swappable if needed: `McpClient` depends on the `McpTransport` interface,
|
||||
so the day we do want the official SDK's transport layer we can adapt
|
||||
it and keep everything else.
|
||||
|
||||
## What's shipped here
|
||||
|
||||
```
|
||||
src/mcp/
|
||||
├── types.ts JSON-RPC 2.0 + MCP-specific message types
|
||||
├── stdio.ts McpTransport interface + StdioTransport (spawn child)
|
||||
├── sse.ts SseTransport (HTTP+SSE for remote/hosted servers)
|
||||
├── spec.ts parseMcpSpec — parses --mcp CLI arg into transport-tagged spec
|
||||
├── catalog.ts curated list of popular official MCP servers
|
||||
├── client.ts McpClient: initialize / listTools / callTool
|
||||
├── registry.ts bridgeMcpTools: MCP → ToolRegistry
|
||||
└── README.md (this file)
|
||||
|
||||
tests/mcp.test.ts — in-process fake transport, no child processes
|
||||
tests/mcp-sse.test.ts — in-process http.Server fake for SSE
|
||||
```
|
||||
|
||||
## What's NOT here (yet)
|
||||
|
||||
| feature | status | note |
|
||||
|---|---|---|
|
||||
| CLI wiring (`reasonix chat --mcp <cmd>`) | ✅ shipped | see Usage below |
|
||||
| Bundled demo server | ✅ shipped | `examples/mcp-server-demo.ts`, exposes echo/add/get_time |
|
||||
| Real-subprocess integration test | ✅ shipped | `tests/mcp-integration.test.ts` |
|
||||
| Resources / `resources/list` / `resources/read` | deferred | Reasonix doesn't surface resources today |
|
||||
| Prompts / `prompts/list` | deferred | ditto |
|
||||
| Progress notifications | deferred | long-running tool support comes with the CLI work |
|
||||
| Streaming results | deferred | current shape returns one CallToolResult per call |
|
||||
| SSE transport | ✅ shipped | `src/mcp/sse.ts` — pass `http(s)://…` to `--mcp` |
|
||||
| Streamable HTTP (2025-03-26 spec) | deferred | waiting for a real server to validate against |
|
||||
| MCP server that Reasonix exposes | never | out of scope — Reasonix is a client |
|
||||
|
||||
## Usage (CLI)
|
||||
|
||||
`--mcp` is repeatable — attach one or many MCP servers; their tools become
|
||||
first-class citizens of the loop.
|
||||
|
||||
```bash
|
||||
# Single server, anonymous (tools use native names):
|
||||
reasonix chat --mcp "node --import tsx examples/mcp-server-demo.ts"
|
||||
|
||||
# Official filesystem server:
|
||||
reasonix chat --mcp "npx -y @modelcontextprotocol/server-filesystem /tmp/safe-dir"
|
||||
|
||||
# Multiple servers, each namespaced. Syntax: "name=command args..."
|
||||
# Tools land in a shared registry as fs_read_file, demo_add, etc.
|
||||
reasonix chat \
|
||||
--mcp "fs=npx -y @modelcontextprotocol/server-filesystem /tmp/safe" \
|
||||
--mcp "demo=node --import tsx examples/mcp-server-demo.ts"
|
||||
|
||||
# Global prefix (only honored when there's ONE anonymous server):
|
||||
reasonix chat \
|
||||
--mcp "npx -y @modelcontextprotocol/server-filesystem /tmp" \
|
||||
--mcp-prefix fs_
|
||||
|
||||
# Same flag works with one-shot run:
|
||||
reasonix run "list files in /tmp/safe-dir" \
|
||||
--mcp "npx -y @modelcontextprotocol/server-filesystem /tmp/safe-dir"
|
||||
```
|
||||
|
||||
Each spec is shell-split (spaces separate args; use quotes for paths with
|
||||
spaces). Windows-friendly: backslashes pass through literally outside
|
||||
quotes, so `C:\path\to\dir` works. Tools get folded into the
|
||||
`ImmutablePrefix` for the model, and every call goes through Reasonix's
|
||||
Cache-First loop + tool-call repair (scavenge / flatten / storm)
|
||||
automatically.
|
||||
|
||||
## Usage (library)
|
||||
|
||||
```ts
|
||||
import {
|
||||
McpClient,
|
||||
StdioTransport,
|
||||
bridgeMcpTools,
|
||||
CacheFirstLoop,
|
||||
DeepSeekClient,
|
||||
ImmutablePrefix,
|
||||
} from "reasonix";
|
||||
|
||||
// 1. Spawn + connect to an MCP server
|
||||
const transport = new StdioTransport({
|
||||
command: "npx",
|
||||
args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp/safe-dir"],
|
||||
});
|
||||
const mcp = new McpClient({ transport });
|
||||
await mcp.initialize();
|
||||
|
||||
// 2. Bridge its tools into a Reasonix ToolRegistry
|
||||
const { registry } = await bridgeMcpTools(mcp, { namePrefix: "fs_" });
|
||||
|
||||
// 3. Use them with the Cache-First Loop — same as any native tool
|
||||
const client = new DeepSeekClient();
|
||||
const loop = new CacheFirstLoop({
|
||||
client,
|
||||
prefix: new ImmutablePrefix({
|
||||
system: "You can use the filesystem tools to help the user.",
|
||||
toolSpecs: registry.specs(),
|
||||
}),
|
||||
tools: registry,
|
||||
});
|
||||
|
||||
for await (const ev of loop.step("List the files in /tmp/safe-dir.")) {
|
||||
if (ev.role === "assistant_final") console.log(ev.content);
|
||||
}
|
||||
|
||||
// 4. Clean up
|
||||
await mcp.close();
|
||||
```
|
||||
|
||||
The payoff: the filesystem server's tools now inherit Reasonix's
|
||||
cache-first prefix stability + repair (schema flatten, tool-call
|
||||
scavenge, call-storm break) without the MCP server knowing anything
|
||||
about it.
|
||||
|
||||
## Wire protocol notes (stdio)
|
||||
|
||||
- **Framing**: newline-delimited JSON. One JSON-RPC message per line,
|
||||
UTF-8, no Content-Length header (that's LSP, not MCP stdio).
|
||||
- **Stderr**: forwarded to the parent's stderr. Servers often print
|
||||
startup banners there; that's fine.
|
||||
- **Shutdown**: `close()` calls `child.stdin.end()` then SIGTERM if the
|
||||
process hasn't exited.
|
||||
- **Malformed lines**: dropped silently. Some servers emit non-JSON
|
||||
during startup; logging every dropped line would be noise.
|
||||
@@ -0,0 +1,79 @@
|
||||
/**
|
||||
* Curated catalog of popular MCP servers.
|
||||
*
|
||||
* Hardcoded because (a) this list changes slowly — maybe monthly, (b)
|
||||
* fetching it over the network would make `reasonix mcp list` flaky
|
||||
* offline or behind a proxy. When it does change, update this file and
|
||||
* ship a patch release.
|
||||
*
|
||||
* Inclusion criteria:
|
||||
* - actively maintained (official Anthropic repo OR widely used)
|
||||
* - stdio-compatible (Reasonix doesn't do SSE yet)
|
||||
* - installable with one `npx -y ...` command — zero manual setup
|
||||
* - has a clear value proposition in one short line
|
||||
*
|
||||
* Not included: servers that need API keys / OAuth / complex config.
|
||||
* Those get their own docs once we have patterns for them.
|
||||
*/
|
||||
|
||||
export interface CatalogEntry {
|
||||
/** Short name, used as the namespace prefix when suggested. */
|
||||
name: string;
|
||||
/** One-line description shown in `reasonix mcp list`. */
|
||||
summary: string;
|
||||
/** npm package id (for `npx -y <pkg>`). */
|
||||
package: string;
|
||||
/** Extra args the user must supply (e.g. a directory path). */
|
||||
userArgs?: string;
|
||||
/** Notes the user needs to know — shown dimmed. */
|
||||
note?: string;
|
||||
}
|
||||
|
||||
// Every entry below is verified to exist on npm as of this release.
|
||||
// `fetch` and `sqlite` are deliberately *absent* — their reference
|
||||
// servers are Python-only (`pip install mcp-server-fetch`), so a Node
|
||||
// user running `npx -y @modelcontextprotocol/server-fetch` hits a 404
|
||||
// from the npm registry. We'd rather ship a smaller list that always
|
||||
// works than a longer list where two options silently 404 on the user.
|
||||
export const MCP_CATALOG: CatalogEntry[] = [
|
||||
{
|
||||
name: "filesystem",
|
||||
summary: "read/write/search files inside a sandboxed directory",
|
||||
package: "@modelcontextprotocol/server-filesystem",
|
||||
userArgs: "<dir>",
|
||||
note: "the directory is a hard sandbox — the server refuses access outside it",
|
||||
},
|
||||
{
|
||||
name: "memory",
|
||||
summary: "persistent key-value memory across sessions",
|
||||
package: "@modelcontextprotocol/server-memory",
|
||||
},
|
||||
{
|
||||
name: "github",
|
||||
summary: "read issues, PRs, code search (needs GITHUB_PERSONAL_ACCESS_TOKEN)",
|
||||
package: "@modelcontextprotocol/server-github",
|
||||
note: "set GITHUB_PERSONAL_ACCESS_TOKEN in your env before spawning",
|
||||
},
|
||||
{
|
||||
name: "puppeteer",
|
||||
summary: "browser automation — take screenshots, click, type",
|
||||
package: "@modelcontextprotocol/server-puppeteer",
|
||||
note: "downloads Chromium on first run (~200 MB)",
|
||||
},
|
||||
{
|
||||
name: "everything",
|
||||
summary: "official test server — exercises every MCP feature",
|
||||
package: "@modelcontextprotocol/server-everything",
|
||||
note: "useful for debugging your Reasonix setup",
|
||||
},
|
||||
];
|
||||
|
||||
/**
|
||||
* Build the `reasonix chat --mcp "..."` command line for a catalog entry.
|
||||
* Returns a copy-pasteable fragment starting at `--mcp`.
|
||||
*/
|
||||
export function mcpCommandFor(entry: CatalogEntry): string {
|
||||
const pkg = entry.package;
|
||||
const tail = entry.userArgs ? ` ${entry.userArgs}` : "";
|
||||
return `--mcp "${entry.name}=npx -y ${pkg}${tail}"`;
|
||||
}
|
||||
@@ -0,0 +1,247 @@
|
||||
/**
|
||||
* MCP client — request/response correlation, initialize handshake,
|
||||
* tools/list, tools/call. Built on top of a McpTransport so the same
|
||||
* logic works against a real stdio server or an in-process fake.
|
||||
*/
|
||||
|
||||
import type { McpTransport } from "./stdio.js";
|
||||
import {
|
||||
type CallToolParams,
|
||||
type CallToolResult,
|
||||
type GetPromptParams,
|
||||
type GetPromptResult,
|
||||
type InitializeParams,
|
||||
type InitializeResult,
|
||||
type JsonRpcId,
|
||||
type JsonRpcMessage,
|
||||
type JsonRpcRequest,
|
||||
type JsonRpcResponse,
|
||||
type ListPromptsParams,
|
||||
type ListPromptsResult,
|
||||
type ListResourcesParams,
|
||||
type ListResourcesResult,
|
||||
type ListToolsResult,
|
||||
MCP_PROTOCOL_VERSION,
|
||||
type McpClientInfo,
|
||||
type ReadResourceParams,
|
||||
type ReadResourceResult,
|
||||
isJsonRpcError,
|
||||
} from "./types.js";
|
||||
|
||||
export interface McpClientOptions {
|
||||
transport: McpTransport;
|
||||
clientInfo?: McpClientInfo;
|
||||
/** Per-request timeout. Default 60s. */
|
||||
requestTimeoutMs?: number;
|
||||
}
|
||||
|
||||
interface PendingRequest {
|
||||
resolve: (value: unknown) => void;
|
||||
reject: (err: Error) => void;
|
||||
timeout: NodeJS.Timeout;
|
||||
}
|
||||
|
||||
export class McpClient {
|
||||
private readonly transport: McpTransport;
|
||||
private readonly clientInfo: McpClientInfo;
|
||||
private readonly requestTimeoutMs: number;
|
||||
private readonly pending = new Map<JsonRpcId, PendingRequest>();
|
||||
private nextId = 1;
|
||||
private readerStarted = false;
|
||||
private initialized = false;
|
||||
private _serverCapabilities: InitializeResult["capabilities"] = {};
|
||||
private _serverInfo: InitializeResult["serverInfo"] = { name: "", version: "" };
|
||||
private _protocolVersion = "";
|
||||
private _instructions: string | undefined;
|
||||
|
||||
constructor(opts: McpClientOptions) {
|
||||
this.transport = opts.transport;
|
||||
this.clientInfo = opts.clientInfo ?? { name: "reasonix", version: "0.3.0-dev" };
|
||||
this.requestTimeoutMs = opts.requestTimeoutMs ?? 60_000;
|
||||
}
|
||||
|
||||
/** Server's advertised capabilities, available after initialize(). */
|
||||
get serverCapabilities(): InitializeResult["capabilities"] {
|
||||
return this._serverCapabilities;
|
||||
}
|
||||
|
||||
/** Server's self-reported name + version, available after initialize(). */
|
||||
get serverInfo(): InitializeResult["serverInfo"] {
|
||||
return this._serverInfo;
|
||||
}
|
||||
|
||||
/** Protocol version the server agreed to during the handshake. */
|
||||
get protocolVersion(): string {
|
||||
return this._protocolVersion;
|
||||
}
|
||||
|
||||
/** Optional free-form instructions the server provides at handshake. */
|
||||
get serverInstructions(): string | undefined {
|
||||
return this._instructions;
|
||||
}
|
||||
|
||||
/**
|
||||
* Complete the initialize → initialized handshake. Must be called
|
||||
* before any other method (otherwise compliant servers reject).
|
||||
*/
|
||||
async initialize(): Promise<InitializeResult> {
|
||||
if (this.initialized) throw new Error("MCP client already initialized");
|
||||
this.startReaderIfNeeded();
|
||||
const result = await this.request<InitializeResult>("initialize", {
|
||||
protocolVersion: MCP_PROTOCOL_VERSION,
|
||||
// Advertise every method the client can consume so servers know
|
||||
// they can send listChanged notifications etc. Sub-feature flags
|
||||
// (e.g. `resources.subscribe`) are omitted — we don't implement
|
||||
// those yet and the empty object means "method-level support, no
|
||||
// sub-features."
|
||||
capabilities: { tools: {}, resources: {}, prompts: {} },
|
||||
clientInfo: this.clientInfo,
|
||||
} satisfies InitializeParams);
|
||||
this._serverCapabilities = result.capabilities ?? {};
|
||||
this._serverInfo = result.serverInfo ?? { name: "", version: "" };
|
||||
this._protocolVersion = result.protocolVersion ?? "";
|
||||
this._instructions = result.instructions;
|
||||
// Per spec: client sends notifications/initialized after receiving the
|
||||
// initialize response. Only then is the connection live for other
|
||||
// methods.
|
||||
await this.transport.send({
|
||||
jsonrpc: "2.0",
|
||||
method: "notifications/initialized",
|
||||
});
|
||||
this.initialized = true;
|
||||
return result;
|
||||
}
|
||||
|
||||
/** List tools the server exposes. */
|
||||
async listTools(): Promise<ListToolsResult> {
|
||||
this.assertInitialized();
|
||||
return this.request<ListToolsResult>("tools/list", {});
|
||||
}
|
||||
|
||||
/** Invoke a tool by name. Returns the raw MCP result (caller unwraps content). */
|
||||
async callTool(name: string, args?: Record<string, unknown>): Promise<CallToolResult> {
|
||||
this.assertInitialized();
|
||||
return this.request<CallToolResult>("tools/call", {
|
||||
name,
|
||||
arguments: args ?? {},
|
||||
} satisfies CallToolParams);
|
||||
}
|
||||
|
||||
/**
|
||||
* List resources the server exposes. Supports a pagination cursor;
|
||||
* callers interested in the full set should loop on `nextCursor`.
|
||||
* Servers that don't support resources respond with method-not-found
|
||||
* (−32601) — we surface that as a thrown Error so callers can gate
|
||||
* on the `serverCapabilities.resources` field first.
|
||||
*/
|
||||
async listResources(cursor?: string): Promise<ListResourcesResult> {
|
||||
this.assertInitialized();
|
||||
return this.request<ListResourcesResult>("resources/list", {
|
||||
...(cursor ? { cursor } : {}),
|
||||
} satisfies ListResourcesParams);
|
||||
}
|
||||
|
||||
/** Read the contents of a resource by URI. */
|
||||
async readResource(uri: string): Promise<ReadResourceResult> {
|
||||
this.assertInitialized();
|
||||
return this.request<ReadResourceResult>("resources/read", {
|
||||
uri,
|
||||
} satisfies ReadResourceParams);
|
||||
}
|
||||
|
||||
/** List prompt templates the server exposes. */
|
||||
async listPrompts(cursor?: string): Promise<ListPromptsResult> {
|
||||
this.assertInitialized();
|
||||
return this.request<ListPromptsResult>("prompts/list", {
|
||||
...(cursor ? { cursor } : {}),
|
||||
} satisfies ListPromptsParams);
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch a rendered prompt by name. `args` supplies values for any
|
||||
* required template arguments; the server validates. Returns messages
|
||||
* ready to prepend to the model's input.
|
||||
*/
|
||||
async getPrompt(name: string, args?: Record<string, string>): Promise<GetPromptResult> {
|
||||
this.assertInitialized();
|
||||
return this.request<GetPromptResult>("prompts/get", {
|
||||
name,
|
||||
...(args ? { arguments: args } : {}),
|
||||
} satisfies GetPromptParams);
|
||||
}
|
||||
|
||||
/** Close the transport and reject any outstanding requests. */
|
||||
async close(): Promise<void> {
|
||||
for (const [, pending] of this.pending) {
|
||||
clearTimeout(pending.timeout);
|
||||
pending.reject(new Error("MCP client closed"));
|
||||
}
|
||||
this.pending.clear();
|
||||
await this.transport.close();
|
||||
}
|
||||
|
||||
// ---------- internals ----------
|
||||
|
||||
private assertInitialized(): void {
|
||||
if (!this.initialized) throw new Error("MCP client not initialized — call initialize() first");
|
||||
}
|
||||
|
||||
private async request<R>(method: string, params: unknown): Promise<R> {
|
||||
const id = this.nextId++;
|
||||
const frame: JsonRpcRequest = { jsonrpc: "2.0", id, method, params };
|
||||
const promise = new Promise<R>((resolve, reject) => {
|
||||
const timeout = setTimeout(() => {
|
||||
this.pending.delete(id);
|
||||
reject(
|
||||
new Error(`MCP request ${method} (id=${id}) timed out after ${this.requestTimeoutMs}ms`),
|
||||
);
|
||||
}, this.requestTimeoutMs);
|
||||
this.pending.set(id, {
|
||||
resolve: resolve as (value: unknown) => void,
|
||||
reject,
|
||||
timeout,
|
||||
});
|
||||
});
|
||||
await this.transport.send(frame);
|
||||
return promise;
|
||||
}
|
||||
|
||||
private startReaderIfNeeded(): void {
|
||||
if (this.readerStarted) return;
|
||||
this.readerStarted = true;
|
||||
// Fire-and-forget: the reader runs for the lifetime of the client.
|
||||
void this.readLoop();
|
||||
}
|
||||
|
||||
private async readLoop(): Promise<void> {
|
||||
try {
|
||||
for await (const msg of this.transport.messages()) {
|
||||
this.dispatch(msg);
|
||||
}
|
||||
} catch (err) {
|
||||
// Surface as rejections on all pending requests so nobody hangs.
|
||||
for (const [, pending] of this.pending) {
|
||||
clearTimeout(pending.timeout);
|
||||
pending.reject(err as Error);
|
||||
}
|
||||
this.pending.clear();
|
||||
}
|
||||
}
|
||||
|
||||
private dispatch(msg: JsonRpcMessage): void {
|
||||
// We only care about responses (have an `id`) for now. Server-initiated
|
||||
// notifications are dropped until we support resources/prompts.
|
||||
if (!("id" in msg) || msg.id === null || msg.id === undefined) return;
|
||||
if (!("result" in msg) && !("error" in msg)) return; // it's a request from server
|
||||
const pending = this.pending.get(msg.id);
|
||||
if (!pending) return; // late response after timeout; drop
|
||||
this.pending.delete(msg.id);
|
||||
clearTimeout(pending.timeout);
|
||||
const resp = msg as JsonRpcResponse;
|
||||
if (isJsonRpcError(resp)) {
|
||||
pending.reject(new Error(`MCP ${resp.error.code}: ${resp.error.message}`));
|
||||
} else {
|
||||
pending.resolve(resp.result);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
/**
|
||||
* Gather a full inspection report from an initialized MCP client:
|
||||
* server info, capabilities, tools, resources, prompts. Methods the
|
||||
* server doesn't support come back as `{ supported: false }` instead
|
||||
* of throwing, so a CLI or UI can render a consistent "what this
|
||||
* server exposes" summary even against minimal implementations.
|
||||
*
|
||||
* Pure with respect to I/O beyond the passed-in client — the CLI
|
||||
* layer owns argument parsing, connection setup, and printing.
|
||||
*/
|
||||
|
||||
import type { McpClient } from "./client.js";
|
||||
import type { McpPrompt, McpResource, McpTool } from "./types.js";
|
||||
|
||||
export interface InspectionReport {
|
||||
protocolVersion: string;
|
||||
serverInfo: { name: string; version: string };
|
||||
capabilities: Record<string, unknown>;
|
||||
instructions?: string;
|
||||
tools: SectionResult<McpTool>;
|
||||
resources: SectionResult<McpResource>;
|
||||
prompts: SectionResult<McpPrompt>;
|
||||
}
|
||||
|
||||
export type SectionResult<T> =
|
||||
| { supported: true; items: T[] }
|
||||
| { supported: false; reason: string };
|
||||
|
||||
/**
|
||||
* Run an inspection against a **already-initialized** client. Caller
|
||||
* is responsible for `initialize()` before this and `close()` after.
|
||||
* We keep this pure so unit tests can feed in a FakeMcpTransport and
|
||||
* verify the aggregate shape without spinning up a real process.
|
||||
*/
|
||||
export async function inspectMcpServer(client: McpClient): Promise<InspectionReport> {
|
||||
// We always *try* the three listings so the client learns whether a
|
||||
// server without explicit capability flags still serves them —
|
||||
// some servers omit capabilities but still respond to the methods.
|
||||
const tools = await trySection<McpTool>(() => client.listTools().then((r) => r.tools));
|
||||
const resources = await trySection<McpResource>(() =>
|
||||
client.listResources().then((r) => r.resources),
|
||||
);
|
||||
const prompts = await trySection<McpPrompt>(() => client.listPrompts().then((r) => r.prompts));
|
||||
|
||||
return {
|
||||
protocolVersion: client.protocolVersion || "(unknown)",
|
||||
serverInfo: client.serverInfo,
|
||||
capabilities: client.serverCapabilities ?? {},
|
||||
instructions: client.serverInstructions,
|
||||
tools,
|
||||
resources,
|
||||
prompts,
|
||||
};
|
||||
}
|
||||
|
||||
async function trySection<T>(load: () => Promise<T[]>): Promise<SectionResult<T>> {
|
||||
try {
|
||||
const items = await load();
|
||||
return { supported: true, items };
|
||||
} catch (err) {
|
||||
const msg = (err as Error).message ?? String(err);
|
||||
// -32601 is JSON-RPC "method not found" — the canonical response
|
||||
// from a server that doesn't implement this family. Treat it as
|
||||
// "not supported" rather than a hard error, so the CLI can render
|
||||
// a clean summary instead of aborting on the first missing method.
|
||||
if (/-32601/.test(msg) || /method not found/i.test(msg)) {
|
||||
return { supported: false, reason: "method not found (-32601)" };
|
||||
}
|
||||
return { supported: false, reason: msg };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,140 @@
|
||||
/**
|
||||
* Bridge: register an MCP server's tools into a Reasonix ToolRegistry.
|
||||
*
|
||||
* This is the integration surface. Once done, `CacheFirstLoop` sees the
|
||||
* MCP tools as if they were native — they inherit Cache-First + repair
|
||||
* (scavenge / truncation / storm) automatically. That's the payoff: any
|
||||
* MCP ecosystem tool, wrapped in Reasonix's Pillar 1 + Pillar 3 benefits.
|
||||
*/
|
||||
|
||||
import { ToolRegistry } from "../tools.js";
|
||||
import type { JSONSchema } from "../types.js";
|
||||
import type { McpClient } from "./client.js";
|
||||
import type { CallToolResult, McpContentBlock } from "./types.js";
|
||||
|
||||
export interface BridgeOptions {
|
||||
/**
|
||||
* Prefix prepended to every MCP tool name when registered. Defaults to
|
||||
* empty (no prefix). Useful when bridging multiple servers into one
|
||||
* registry and names collide — e.g. `fs` + `gh` both exposing `search`.
|
||||
*/
|
||||
namePrefix?: string;
|
||||
/** Registry to populate. Creates a fresh one if omitted. */
|
||||
registry?: ToolRegistry;
|
||||
/** Auto-flatten deep schemas (Pillar 3). Defaults to the registry's own default (true). */
|
||||
autoFlatten?: boolean;
|
||||
/**
|
||||
* Per-tool-call result cap, in characters. If a tool returns more than
|
||||
* this, the result is truncated and a `[…truncated N chars…]` marker is
|
||||
* appended before the last KB so the model still sees a useful tail.
|
||||
* Defaults to {@link DEFAULT_MAX_RESULT_CHARS}.
|
||||
*
|
||||
* Why this exists: DeepSeek V3's context is 131,072 tokens. A single
|
||||
* `read_file` against a big source file can return >3 MB of text
|
||||
* (~900k tokens) and permanently poison the session — every subsequent
|
||||
* turn rebuilds the history and 400s. This cap is a floor. Users who
|
||||
* legitimately want bigger payloads can raise it explicitly.
|
||||
*/
|
||||
maxResultChars?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* 32,000 chars ≈ 8k English tokens, or ~16k CJK tokens. Small enough to
|
||||
* fit comfortably in history even across 5–10 tool calls, large enough
|
||||
* that most file reads and directory listings fit un-truncated.
|
||||
*/
|
||||
export const DEFAULT_MAX_RESULT_CHARS = 32_000;
|
||||
|
||||
export interface BridgeResult {
|
||||
registry: ToolRegistry;
|
||||
/** Names actually registered (may differ from MCP names when a prefix is applied). */
|
||||
registeredNames: string[];
|
||||
/** Names the server listed but the bridge skipped (e.g. invalid schemas). */
|
||||
skipped: Array<{ name: string; reason: string }>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Walk a connected `McpClient`'s tools/list result, register each into a
|
||||
* Reasonix `ToolRegistry`. Each registered `fn` proxies through the
|
||||
* client's tools/call. Tool results are flattened into a string (joining
|
||||
* text blocks with newlines, prefixing image blocks as placeholders) so
|
||||
* they fit Reasonix's existing tool-dispatch contract.
|
||||
*/
|
||||
export async function bridgeMcpTools(
|
||||
client: McpClient,
|
||||
opts: BridgeOptions = {},
|
||||
): Promise<BridgeResult> {
|
||||
const registry = opts.registry ?? new ToolRegistry({ autoFlatten: opts.autoFlatten });
|
||||
const prefix = opts.namePrefix ?? "";
|
||||
const maxResultChars = opts.maxResultChars ?? DEFAULT_MAX_RESULT_CHARS;
|
||||
const result: BridgeResult = { registry, registeredNames: [], skipped: [] };
|
||||
|
||||
const listed = await client.listTools();
|
||||
for (const mcpTool of listed.tools) {
|
||||
if (!mcpTool.name) {
|
||||
result.skipped.push({ name: "?", reason: "empty tool name" });
|
||||
continue;
|
||||
}
|
||||
const registeredName = `${prefix}${mcpTool.name}`;
|
||||
registry.register({
|
||||
name: registeredName,
|
||||
description: mcpTool.description ?? "",
|
||||
parameters: mcpTool.inputSchema as JSONSchema,
|
||||
fn: async (args: Record<string, unknown>) => {
|
||||
const toolResult = await client.callTool(mcpTool.name, args);
|
||||
return flattenMcpResult(toolResult, { maxChars: maxResultChars });
|
||||
},
|
||||
});
|
||||
result.registeredNames.push(registeredName);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
export interface FlattenOptions {
|
||||
/** Cap the flattened string at this many characters. Default: no cap. */
|
||||
maxChars?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Turn an MCP CallToolResult into a string — the contract Reasonix's
|
||||
* ToolRegistry.dispatch returns. We:
|
||||
* - join text blocks with newlines (most common case)
|
||||
* - stringify image blocks as placeholders (LLM can't use bytes anyway
|
||||
* in Reasonix's current surface; image support comes with multimodal
|
||||
* prompts later)
|
||||
* - prefix error results with "ERROR: " so the calling model sees the
|
||||
* failure clearly even through JSON mode
|
||||
* - optionally truncate to `maxChars` so a single oversized tool result
|
||||
* (e.g. a big `read_file`) can't poison the session by blowing past
|
||||
* the model's context window
|
||||
*/
|
||||
export function flattenMcpResult(result: CallToolResult, opts: FlattenOptions = {}): string {
|
||||
const parts = result.content.map(blockToString);
|
||||
const joined = parts.join("\n").trim();
|
||||
const prefixed = result.isError ? `ERROR: ${joined || "(no error message from server)"}` : joined;
|
||||
return opts.maxChars ? truncateForModel(prefixed, opts.maxChars) : prefixed;
|
||||
}
|
||||
|
||||
/**
|
||||
* Keep the head AND a short tail so the model sees both "what the tool
|
||||
* started returning" and "how it ended". Head-only loses file endings
|
||||
* (e.g. an error message appended at the bottom of a stack trace); the
|
||||
* 1KB tail window covers that while costing almost nothing. Exported for
|
||||
* tests and reuse by non-MCP tool adapters that want the same policy.
|
||||
*/
|
||||
export function truncateForModel(s: string, maxChars: number): string {
|
||||
if (s.length <= maxChars) return s;
|
||||
const tailBudget = Math.min(1024, Math.floor(maxChars * 0.1));
|
||||
const headBudget = Math.max(0, maxChars - tailBudget);
|
||||
const head = s.slice(0, headBudget);
|
||||
const tail = s.slice(-tailBudget);
|
||||
const dropped = s.length - head.length - tail.length;
|
||||
return `${head}\n\n[…truncated ${dropped} chars — raise BridgeOptions.maxResultChars, or call the tool with a narrower scope (filter, head, pagination)…]\n\n${tail}`;
|
||||
}
|
||||
|
||||
function blockToString(block: McpContentBlock): string {
|
||||
if (block.type === "text") return block.text;
|
||||
if (block.type === "image") return `[image ${block.mimeType}, ${block.data.length} chars base64]`;
|
||||
// Unknown block type — preserve for diagnostics.
|
||||
return `[unknown block: ${JSON.stringify(block)}]`;
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
/**
|
||||
* Split a shell-style command string into argv, respecting single and
|
||||
* double quotes. Intended for parsing the user's `--mcp "cmd args..."`
|
||||
* flag — NOT a full shell parser (no variable expansion, no subshells,
|
||||
* no globs, no `&&` / pipes).
|
||||
*
|
||||
* The tradeoff: users with paths containing spaces need to quote them
|
||||
* (e.g. `--mcp 'npx -y pkg "/my path/here"'`), which is how they'd
|
||||
* already quote them at the shell level.
|
||||
*
|
||||
* Throws on unterminated quotes — better than silently dropping half
|
||||
* the command.
|
||||
*/
|
||||
export function shellSplit(input: string): string[] {
|
||||
const tokens: string[] = [];
|
||||
let cur = "";
|
||||
let quote: '"' | "'" | null = null;
|
||||
let i = 0;
|
||||
const s = input;
|
||||
|
||||
while (i < s.length) {
|
||||
const ch = s[i]!;
|
||||
|
||||
if (quote) {
|
||||
if (ch === quote) {
|
||||
quote = null;
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
// backslash escapes inside double quotes only
|
||||
if (ch === "\\" && quote === '"' && i + 1 < s.length) {
|
||||
cur += s[i + 1];
|
||||
i += 2;
|
||||
continue;
|
||||
}
|
||||
cur += ch;
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (ch === '"' || ch === "'") {
|
||||
quote = ch as '"' | "'";
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Backslash escape ONLY applies inside double quotes (handled above).
|
||||
// Outside quotes, backslashes pass through literally — otherwise
|
||||
// Windows paths like `C:\path\to\exe` get mangled. POSIX users who
|
||||
// want to escape a space outside quotes can use single quotes instead.
|
||||
|
||||
if (ch === " " || ch === "\t") {
|
||||
if (cur.length > 0) {
|
||||
tokens.push(cur);
|
||||
cur = "";
|
||||
}
|
||||
i++;
|
||||
continue;
|
||||
}
|
||||
|
||||
cur += ch;
|
||||
i++;
|
||||
}
|
||||
|
||||
if (quote) {
|
||||
throw new Error(
|
||||
`shellSplit: unterminated ${quote === '"' ? "double" : "single"} quote in input`,
|
||||
);
|
||||
}
|
||||
if (cur.length > 0) tokens.push(cur);
|
||||
return tokens;
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
/**
|
||||
* Parse the `--mcp` CLI argument into a transport-tagged spec.
|
||||
*
|
||||
* Accepted forms:
|
||||
* "name=command args..." → stdio, namespaced (tools prefixed with `name_`)
|
||||
* "command args..." → stdio, anonymous
|
||||
* "name=https://host/sse" → SSE, namespaced
|
||||
* "https://host/sse" → SSE, anonymous
|
||||
* ("http://" is also honored — useful for local dev servers.)
|
||||
*
|
||||
* The identifier regex before `=` is deliberately narrow
|
||||
* (`[a-zA-Z_][a-zA-Z0-9_]*`) so Windows drive letters ("C:\\...") and
|
||||
* other strings containing `=` or `:` don't accidentally trigger the
|
||||
* namespace branch. If a user ever wants their command to literally start
|
||||
* with `foo=...` as a bare command, they can wrap it in quotes inside the
|
||||
* shell command string.
|
||||
*
|
||||
* Transport is selected solely by whether the body begins with `http://`
|
||||
* or `https://`. Anything else is stdio — including ws:// (unsupported)
|
||||
* which will surface later as a spawn error, keeping the rule local.
|
||||
*/
|
||||
|
||||
import { shellSplit } from "./shell-split.js";
|
||||
|
||||
export interface StdioMcpSpec {
|
||||
transport: "stdio";
|
||||
/** Namespace prefix applied to each registered tool, or null if anonymous. */
|
||||
name: string | null;
|
||||
/** Argv[0]. */
|
||||
command: string;
|
||||
/** Remaining argv. */
|
||||
args: string[];
|
||||
}
|
||||
|
||||
export interface SseMcpSpec {
|
||||
transport: "sse";
|
||||
name: string | null;
|
||||
/** Fully qualified SSE endpoint URL. */
|
||||
url: string;
|
||||
}
|
||||
|
||||
export type McpSpec = StdioMcpSpec | SseMcpSpec;
|
||||
|
||||
const NAME_PREFIX = /^([a-zA-Z_][a-zA-Z0-9_]*)=(.*)$/;
|
||||
const HTTP_URL = /^https?:\/\//i;
|
||||
|
||||
export function parseMcpSpec(input: string): McpSpec {
|
||||
const trimmed = input.trim();
|
||||
if (!trimmed) {
|
||||
throw new Error("empty MCP spec");
|
||||
}
|
||||
|
||||
const nameMatch = NAME_PREFIX.exec(trimmed);
|
||||
const name = nameMatch ? nameMatch[1]! : null;
|
||||
const body = (nameMatch ? nameMatch[2]! : trimmed).trim();
|
||||
|
||||
if (!body) {
|
||||
throw new Error(`MCP spec has name but no command: ${input}`);
|
||||
}
|
||||
|
||||
if (HTTP_URL.test(body)) {
|
||||
return { transport: "sse", name, url: body };
|
||||
}
|
||||
|
||||
const argv = shellSplit(body);
|
||||
if (argv.length === 0) {
|
||||
throw new Error(`MCP spec has name but no command: ${input}`);
|
||||
}
|
||||
const [command, ...args] = argv;
|
||||
return { transport: "stdio", name, command: command!, args };
|
||||
}
|
||||
+194
@@ -0,0 +1,194 @@
|
||||
/**
|
||||
* HTTP+SSE transport for MCP (spec version 2024-11-05).
|
||||
*
|
||||
* Wire shape:
|
||||
* 1. Client opens GET to the SSE URL with `Accept: text/event-stream`.
|
||||
* 2. Server's first SSE event is `event: endpoint`, `data: <url>` — the
|
||||
* URL (relative or absolute) the client must POST JSON-RPC requests
|
||||
* to. All subsequent server → client messages arrive as `event: message`
|
||||
* SSE frames carrying a JSON-RPC response or server-initiated frame.
|
||||
* 3. Client POSTs each outgoing JSON-RPC frame to the endpoint URL.
|
||||
* The POST response body is ignored — replies land on the SSE stream.
|
||||
*
|
||||
* This transport exists so Reasonix can talk to hosted/remote MCP servers
|
||||
* (e.g. a company's internal knowledge server fronted by auth). Stdio
|
||||
* covers local subprocesses; SSE covers everything else.
|
||||
*
|
||||
* Note: the newer "Streamable HTTP" transport (2025 spec) folds the POST
|
||||
* and SSE streams onto a single endpoint. We stay on 2024-11-05 here —
|
||||
* that's what `MCP_PROTOCOL_VERSION` advertises in the initialize handshake
|
||||
* and what currently-published servers implement.
|
||||
*/
|
||||
|
||||
import { createParser } from "eventsource-parser";
|
||||
import type { McpTransport } from "./stdio.js";
|
||||
import type { JsonRpcMessage } from "./types.js";
|
||||
|
||||
export interface SseTransportOptions {
|
||||
/** SSE endpoint URL, e.g. `https://mcp.example.com/sse`. */
|
||||
url: string;
|
||||
/** Extra headers sent on both the SSE GET and the JSON-RPC POSTs (e.g. `Authorization`). */
|
||||
headers?: Record<string, string>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Open an SSE stream to `url`, parse incoming events into JsonRpcMessages,
|
||||
* POST outgoing frames to the endpoint URL the server advertises.
|
||||
*/
|
||||
export class SseTransport implements McpTransport {
|
||||
private readonly url: string;
|
||||
private readonly headers: Record<string, string>;
|
||||
private readonly queue: JsonRpcMessage[] = [];
|
||||
private readonly waiters: Array<(m: JsonRpcMessage | null) => void> = [];
|
||||
private readonly controller = new AbortController();
|
||||
private closed = false;
|
||||
private postUrl: string | null = null;
|
||||
private readonly endpointReady: Promise<string>;
|
||||
private resolveEndpoint!: (url: string) => void;
|
||||
private rejectEndpoint!: (err: Error) => void;
|
||||
|
||||
constructor(opts: SseTransportOptions) {
|
||||
this.url = opts.url;
|
||||
this.headers = opts.headers ?? {};
|
||||
this.endpointReady = new Promise<string>((resolve, reject) => {
|
||||
this.resolveEndpoint = resolve;
|
||||
this.rejectEndpoint = reject;
|
||||
});
|
||||
// Swallow unhandled-rejection noise if nobody ever calls send().
|
||||
this.endpointReady.catch(() => undefined);
|
||||
void this.runStream();
|
||||
}
|
||||
|
||||
async send(message: JsonRpcMessage): Promise<void> {
|
||||
if (this.closed) throw new Error("MCP SSE transport is closed");
|
||||
const postUrl = await this.endpointReady;
|
||||
const res = await fetch(postUrl, {
|
||||
method: "POST",
|
||||
headers: { "content-type": "application/json", ...this.headers },
|
||||
body: JSON.stringify(message),
|
||||
signal: this.controller.signal,
|
||||
});
|
||||
// Drain body so the socket returns to the pool even if the server
|
||||
// elected to write one. We explicitly don't parse it — responses
|
||||
// arrive on the SSE channel.
|
||||
await res.arrayBuffer().catch(() => undefined);
|
||||
if (!res.ok) {
|
||||
throw new Error(`MCP SSE POST ${postUrl} failed: ${res.status} ${res.statusText}`);
|
||||
}
|
||||
}
|
||||
|
||||
async *messages(): AsyncIterableIterator<JsonRpcMessage> {
|
||||
while (true) {
|
||||
if (this.queue.length > 0) {
|
||||
yield this.queue.shift()!;
|
||||
continue;
|
||||
}
|
||||
if (this.closed) return;
|
||||
const next = await new Promise<JsonRpcMessage | null>((resolve) => {
|
||||
this.waiters.push(resolve);
|
||||
});
|
||||
if (next === null) return;
|
||||
yield next;
|
||||
}
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
if (this.closed) return;
|
||||
this.closed = true;
|
||||
while (this.waiters.length > 0) this.waiters.shift()!(null);
|
||||
// Reject any still-pending send() that was waiting for the endpoint.
|
||||
this.rejectEndpoint(new Error("MCP SSE transport closed before endpoint was ready"));
|
||||
try {
|
||||
this.controller.abort();
|
||||
} catch {
|
||||
/* already aborted */
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- internals ----------
|
||||
|
||||
private async runStream(): Promise<void> {
|
||||
let res: Response;
|
||||
try {
|
||||
res = await fetch(this.url, {
|
||||
method: "GET",
|
||||
headers: { accept: "text/event-stream", ...this.headers },
|
||||
signal: this.controller.signal,
|
||||
});
|
||||
} catch (err) {
|
||||
this.failHandshake(`SSE connect to ${this.url} failed: ${(err as Error).message}`);
|
||||
return;
|
||||
}
|
||||
if (!res.ok || !res.body) {
|
||||
// Drain body to free the socket before giving up.
|
||||
await res.body?.cancel().catch(() => undefined);
|
||||
this.failHandshake(`SSE handshake ${this.url} → ${res.status} ${res.statusText}`);
|
||||
return;
|
||||
}
|
||||
|
||||
const parser = createParser({
|
||||
onEvent: (ev) => this.handleEvent(ev.event ?? "message", ev.data),
|
||||
});
|
||||
const decoder = new TextDecoder();
|
||||
try {
|
||||
for await (const chunk of res.body as AsyncIterable<Uint8Array>) {
|
||||
parser.feed(decoder.decode(chunk, { stream: true }));
|
||||
}
|
||||
} catch (err) {
|
||||
if (!this.closed) {
|
||||
this.pushError(`SSE stream error: ${(err as Error).message}`);
|
||||
}
|
||||
} finally {
|
||||
this.markClosed();
|
||||
}
|
||||
}
|
||||
|
||||
private handleEvent(type: string, data: string): void {
|
||||
if (type === "endpoint") {
|
||||
if (this.postUrl) return; // ignore repeat announcements
|
||||
try {
|
||||
this.postUrl = new URL(data, this.url).toString();
|
||||
this.resolveEndpoint(this.postUrl);
|
||||
} catch (err) {
|
||||
this.failHandshake(`SSE endpoint event had bad URL "${data}": ${(err as Error).message}`);
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (type === "message") {
|
||||
try {
|
||||
const parsed = JSON.parse(data) as JsonRpcMessage;
|
||||
this.pushMessage(parsed);
|
||||
} catch {
|
||||
// Malformed JSON-RPC on an SSE frame — drop it, same as stdio.
|
||||
}
|
||||
return;
|
||||
}
|
||||
// Unknown event types (server pings, custom extensions) — ignore.
|
||||
}
|
||||
|
||||
private failHandshake(reason: string): void {
|
||||
this.rejectEndpoint(new Error(reason));
|
||||
this.pushError(reason);
|
||||
this.markClosed();
|
||||
}
|
||||
|
||||
private pushMessage(msg: JsonRpcMessage): void {
|
||||
const waiter = this.waiters.shift();
|
||||
if (waiter) waiter(msg);
|
||||
else this.queue.push(msg);
|
||||
}
|
||||
|
||||
private pushError(message: string): void {
|
||||
this.pushMessage({
|
||||
jsonrpc: "2.0",
|
||||
id: null,
|
||||
error: { code: -32000, message },
|
||||
});
|
||||
}
|
||||
|
||||
private markClosed(): void {
|
||||
if (this.closed) return;
|
||||
this.closed = true;
|
||||
while (this.waiters.length > 0) this.waiters.shift()!(null);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,193 @@
|
||||
/**
|
||||
* Stdio transport for MCP.
|
||||
*
|
||||
* MCP's stdio wire format is **newline-delimited JSON** (one JSON-RPC
|
||||
* message per line). We spawn the server as a child process, write
|
||||
* frames to its stdin, parse its stdout line-by-line as they arrive.
|
||||
*
|
||||
* Transport is abstracted behind an interface so unit tests can fake it
|
||||
* with an in-process duplex pair — spawning real servers in unit tests
|
||||
* is flaky and slow.
|
||||
*/
|
||||
|
||||
import { type ChildProcess, spawn } from "node:child_process";
|
||||
import type { JsonRpcMessage } from "./types.js";
|
||||
|
||||
/**
|
||||
* A transport sends JSON-RPC messages upstream and surfaces messages
|
||||
* arriving downstream via an async iterator. One instance per server
|
||||
* connection.
|
||||
*/
|
||||
export interface McpTransport {
|
||||
/** Send one JSON-RPC message. Resolves when the bytes are accepted. */
|
||||
send(message: JsonRpcMessage): Promise<void>;
|
||||
/** Async iterator over incoming messages. Ends when the connection closes. */
|
||||
messages(): AsyncIterableIterator<JsonRpcMessage>;
|
||||
/** Close the underlying resource (kill child process, close streams). */
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
export interface StdioTransportOptions {
|
||||
/** Argv to spawn. First element is the command. */
|
||||
command: string;
|
||||
args?: string[];
|
||||
/** Env overlay — merged over process.env unless replaceEnv=true. */
|
||||
env?: Record<string, string>;
|
||||
/** When true, only the env above is visible to the child. Default false. */
|
||||
replaceEnv?: boolean;
|
||||
/** CWD for the child. Default: process.cwd(). */
|
||||
cwd?: string;
|
||||
/**
|
||||
* Spawn through a shell. Default: true on win32 (needed to resolve
|
||||
* `.cmd` wrappers like `npx.cmd`, `pnpm.cmd`), false elsewhere.
|
||||
* Explicitly pass `false` to opt out on Windows; pass `true` to force
|
||||
* it on POSIX (rarely needed).
|
||||
*/
|
||||
shell?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Spawn `command args...` as a child process and use its stdin/stdout as
|
||||
* an MCP transport. Stderr is forwarded to the parent's stderr so server
|
||||
* diagnostics are still visible.
|
||||
*/
|
||||
export class StdioTransport implements McpTransport {
|
||||
private readonly child: ChildProcess;
|
||||
private readonly queue: JsonRpcMessage[] = [];
|
||||
private readonly waiters: Array<(m: JsonRpcMessage | null) => void> = [];
|
||||
private closed = false;
|
||||
private stdoutBuffer = "";
|
||||
|
||||
constructor(opts: StdioTransportOptions) {
|
||||
const env = opts.replaceEnv ? { ...(opts.env ?? {}) } : { ...process.env, ...(opts.env ?? {}) };
|
||||
// Windows wraps binaries as .cmd/.bat shims (npx.cmd, pnpm.cmd, …).
|
||||
// child_process.spawn without shell:true can't resolve them, which
|
||||
// breaks `--mcp "npx -y some-server"` — the most common MCP setup.
|
||||
// Default shell:true on win32 and leave POSIX alone.
|
||||
const shell = opts.shell ?? process.platform === "win32";
|
||||
|
||||
if (shell) {
|
||||
// Node's shell:true + args[] triggers DEP0190 because it concatenates
|
||||
// with spaces and doesn't quote args — unsafe if an arg contains
|
||||
// shell metacharacters. We build a single command line ourselves,
|
||||
// quoting ONLY the args (command stays bare so the shell's PATH /
|
||||
// PATHEXT lookup finds `npx` → `npx.cmd` on Windows).
|
||||
const line = [
|
||||
opts.command,
|
||||
...(opts.args ?? []).map((a) => quoteArg(a, process.platform === "win32")),
|
||||
].join(" ");
|
||||
this.child = spawn(line, [], {
|
||||
env,
|
||||
cwd: opts.cwd,
|
||||
stdio: ["pipe", "pipe", "inherit"],
|
||||
shell: true,
|
||||
});
|
||||
} else {
|
||||
this.child = spawn(opts.command, opts.args ?? [], {
|
||||
env,
|
||||
cwd: opts.cwd,
|
||||
stdio: ["pipe", "pipe", "inherit"],
|
||||
});
|
||||
}
|
||||
this.child.stdout!.setEncoding("utf8");
|
||||
this.child.stdout!.on("data", (chunk: string) => this.onStdout(chunk));
|
||||
this.child.on("close", () => this.onClose());
|
||||
this.child.on("error", (err) => {
|
||||
// Surface spawn errors as a synthetic JsonRpcError so callers don't
|
||||
// hang on a stream that never emits anything.
|
||||
this.push({
|
||||
jsonrpc: "2.0",
|
||||
id: null,
|
||||
error: { code: -32000, message: `transport error: ${err.message}` },
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async send(message: JsonRpcMessage): Promise<void> {
|
||||
if (this.closed) throw new Error("MCP transport is closed");
|
||||
return new Promise((resolve, reject) => {
|
||||
const line = `${JSON.stringify(message)}\n`;
|
||||
this.child.stdin!.write(line, "utf8", (err) => {
|
||||
if (err) reject(err);
|
||||
else resolve();
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
async *messages(): AsyncIterableIterator<JsonRpcMessage> {
|
||||
while (true) {
|
||||
if (this.queue.length > 0) {
|
||||
yield this.queue.shift()!;
|
||||
continue;
|
||||
}
|
||||
if (this.closed) return;
|
||||
const next = await new Promise<JsonRpcMessage | null>((resolve) => {
|
||||
this.waiters.push(resolve);
|
||||
});
|
||||
if (next === null) return; // closed while we were waiting
|
||||
yield next;
|
||||
}
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
if (this.closed) return;
|
||||
this.closed = true;
|
||||
// Signal any pending waiters.
|
||||
while (this.waiters.length > 0) this.waiters.shift()!(null);
|
||||
try {
|
||||
this.child.stdin!.end();
|
||||
} catch {
|
||||
/* already ended */
|
||||
}
|
||||
if (this.child.exitCode === null && !this.child.killed) {
|
||||
this.child.kill("SIGTERM");
|
||||
}
|
||||
}
|
||||
|
||||
/** Parse incoming stdout chunks into NDJSON messages. */
|
||||
private onStdout(chunk: string): void {
|
||||
this.stdoutBuffer += chunk;
|
||||
let newlineIdx: number;
|
||||
// biome-ignore lint/suspicious/noAssignInExpressions: idiomatic loop shape
|
||||
while ((newlineIdx = this.stdoutBuffer.indexOf("\n")) !== -1) {
|
||||
const line = this.stdoutBuffer.slice(0, newlineIdx).trim();
|
||||
this.stdoutBuffer = this.stdoutBuffer.slice(newlineIdx + 1);
|
||||
if (!line) continue;
|
||||
try {
|
||||
const msg = JSON.parse(line) as JsonRpcMessage;
|
||||
this.push(msg);
|
||||
} catch {
|
||||
// Malformed lines are dropped — some servers emit startup banners
|
||||
// before the JSON-RPC loop begins. We surface the noise to stderr
|
||||
// via the inherited stderr stream, not our event queue.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private onClose(): void {
|
||||
this.closed = true;
|
||||
while (this.waiters.length > 0) this.waiters.shift()!(null);
|
||||
}
|
||||
|
||||
private push(msg: JsonRpcMessage): void {
|
||||
const waiter = this.waiters.shift();
|
||||
if (waiter) waiter(msg);
|
||||
else this.queue.push(msg);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Quote a single argument for inclusion in a shell command line.
|
||||
* On Windows (cmd.exe): wrap in double quotes, escape internal `"` as `""`,
|
||||
* leave everything else alone. On POSIX: wrap in single quotes, escape
|
||||
* internal `'` as `'\''`. Both handle spaces, wildcards, pipes, and all
|
||||
* other metacharacters correctly.
|
||||
*/
|
||||
function quoteArg(s: string, windows: boolean): string {
|
||||
if (!windows) {
|
||||
// POSIX: single-quote, escape single quotes.
|
||||
return `'${s.replace(/'/g, "'\\''")}'`;
|
||||
}
|
||||
// cmd.exe: double-quote, escape internal quotes by doubling.
|
||||
return `"${s.replace(/"/g, '""')}"`;
|
||||
}
|
||||
@@ -0,0 +1,255 @@
|
||||
/**
|
||||
* MCP (Model Context Protocol) type definitions.
|
||||
*
|
||||
* Hand-rolled rather than importing @modelcontextprotocol/sdk because:
|
||||
* - Reasonix's value-add isn't reimplementing the protocol, but *caching*
|
||||
* it. Owning the types lets us tune them for our integration (strip
|
||||
* fields we don't use, add the ones we do like Reasonix's prefixHash).
|
||||
* - Zero dependencies — consistent with how we wrote the DeepSeek client.
|
||||
* - If Anthropic bumps the SDK and introduces a breaking change, we're
|
||||
* insulated as long as we keep up with the spec itself.
|
||||
*
|
||||
* Spec reference: https://spec.modelcontextprotocol.io/ (2024-11-05 draft
|
||||
* at time of writing). Reasonix models the subset it consumes: tools
|
||||
* list/call, resources list/read, prompts list/get, plus the init
|
||||
* handshake. Sampling and progress notifications remain deferred.
|
||||
*
|
||||
* Transport note: the wire format for stdio MCP is **newline-delimited
|
||||
* JSON** (NDJSON), not the LSP-style Content-Length header framing that
|
||||
* some readers might expect. One JSON-RPC message per line.
|
||||
*/
|
||||
|
||||
// ---------- JSON-RPC 2.0 base ----------
|
||||
|
||||
export type JsonRpcId = string | number;
|
||||
|
||||
export interface JsonRpcRequest<P = unknown> {
|
||||
jsonrpc: "2.0";
|
||||
id: JsonRpcId;
|
||||
method: string;
|
||||
params?: P;
|
||||
}
|
||||
|
||||
export interface JsonRpcNotification<P = unknown> {
|
||||
jsonrpc: "2.0";
|
||||
method: string;
|
||||
params?: P;
|
||||
}
|
||||
|
||||
export interface JsonRpcSuccess<R = unknown> {
|
||||
jsonrpc: "2.0";
|
||||
id: JsonRpcId;
|
||||
result: R;
|
||||
}
|
||||
|
||||
export interface JsonRpcError {
|
||||
jsonrpc: "2.0";
|
||||
id: JsonRpcId | null;
|
||||
error: {
|
||||
/** JSON-RPC standard codes: -32700 parse, -32600 invalid request, -32601 method not found, -32602 invalid params, -32603 internal. MCP also defines its own range. */
|
||||
code: number;
|
||||
message: string;
|
||||
data?: unknown;
|
||||
};
|
||||
}
|
||||
|
||||
export type JsonRpcResponse<R = unknown> = JsonRpcSuccess<R> | JsonRpcError;
|
||||
|
||||
export type JsonRpcMessage = JsonRpcRequest | JsonRpcNotification | JsonRpcSuccess | JsonRpcError;
|
||||
|
||||
// ---------- MCP initialize ----------
|
||||
|
||||
export interface McpClientInfo {
|
||||
name: string;
|
||||
version: string;
|
||||
}
|
||||
|
||||
export interface McpClientCapabilities {
|
||||
/** Empty object advertises support without any optional sub-features. */
|
||||
tools?: Record<string, never>;
|
||||
/** Advertised when the client can consume `resources/list` + `resources/read`. */
|
||||
resources?: Record<string, never>;
|
||||
/** Advertised when the client can consume `prompts/list` + `prompts/get`. */
|
||||
prompts?: Record<string, never>;
|
||||
// sampling would go here — deferred.
|
||||
}
|
||||
|
||||
export interface InitializeParams {
|
||||
protocolVersion: string;
|
||||
capabilities: McpClientCapabilities;
|
||||
clientInfo: McpClientInfo;
|
||||
}
|
||||
|
||||
export interface InitializeResult {
|
||||
protocolVersion: string;
|
||||
serverInfo: { name: string; version: string };
|
||||
capabilities: {
|
||||
tools?: { listChanged?: boolean };
|
||||
resources?: unknown;
|
||||
prompts?: unknown;
|
||||
};
|
||||
instructions?: string;
|
||||
}
|
||||
|
||||
// ---------- MCP tools ----------
|
||||
|
||||
export interface McpToolSchema {
|
||||
/** JSON Schema — compatible with Reasonix's tools.ts JSONSchema shape. */
|
||||
type?: string;
|
||||
properties?: Record<string, unknown>;
|
||||
required?: string[];
|
||||
[extra: string]: unknown;
|
||||
}
|
||||
|
||||
export interface McpTool {
|
||||
name: string;
|
||||
description?: string;
|
||||
/** MCP calls this `inputSchema`. Reasonix's `parameters` field is the same concept. */
|
||||
inputSchema: McpToolSchema;
|
||||
}
|
||||
|
||||
export interface ListToolsResult {
|
||||
tools: McpTool[];
|
||||
nextCursor?: string;
|
||||
}
|
||||
|
||||
export interface CallToolParams {
|
||||
name: string;
|
||||
arguments?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface McpContentBlockText {
|
||||
type: "text";
|
||||
text: string;
|
||||
}
|
||||
|
||||
export interface McpContentBlockImage {
|
||||
type: "image";
|
||||
data: string;
|
||||
mimeType: string;
|
||||
}
|
||||
|
||||
/** MCP result content is an array of typed blocks. Reasonix consumes only text for now — image blocks get stringified with a placeholder. */
|
||||
export type McpContentBlock = McpContentBlockText | McpContentBlockImage;
|
||||
|
||||
export interface CallToolResult {
|
||||
content: McpContentBlock[];
|
||||
/** True = tool raised an error; the content describes it. */
|
||||
isError?: boolean;
|
||||
}
|
||||
|
||||
// ---------- MCP resources ----------
|
||||
|
||||
/**
|
||||
* A resource the server can expose — think "file the model can read."
|
||||
* The URI is opaque to the client: servers may use `file://`, custom
|
||||
* schemes, or bare strings. Reasonix doesn't interpret them.
|
||||
*/
|
||||
export interface McpResource {
|
||||
uri: string;
|
||||
name: string;
|
||||
description?: string;
|
||||
/** Hint for the content type (e.g. "text/markdown"). Purely informational. */
|
||||
mimeType?: string;
|
||||
}
|
||||
|
||||
export interface ListResourcesParams {
|
||||
/** Pagination cursor from a previous listResources response. */
|
||||
cursor?: string;
|
||||
}
|
||||
|
||||
export interface ListResourcesResult {
|
||||
resources: McpResource[];
|
||||
nextCursor?: string;
|
||||
}
|
||||
|
||||
export interface ReadResourceParams {
|
||||
uri: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* One resource can return multiple content blobs (e.g. the file + a
|
||||
* side-car). `text` is the common case for UTF-8 content; `blob` is
|
||||
* base64-encoded bytes for binary content. Servers populate exactly
|
||||
* one of the two for each entry.
|
||||
*/
|
||||
export interface McpResourceContentsText {
|
||||
uri: string;
|
||||
mimeType?: string;
|
||||
text: string;
|
||||
}
|
||||
|
||||
export interface McpResourceContentsBlob {
|
||||
uri: string;
|
||||
mimeType?: string;
|
||||
blob: string;
|
||||
}
|
||||
|
||||
export type McpResourceContents = McpResourceContentsText | McpResourceContentsBlob;
|
||||
|
||||
export interface ReadResourceResult {
|
||||
contents: McpResourceContents[];
|
||||
}
|
||||
|
||||
// ---------- MCP prompts ----------
|
||||
|
||||
/**
|
||||
* A parameterizable prompt template the server exposes. Clients fetch
|
||||
* it with `prompts/get` and pass the result to the model as-is.
|
||||
*/
|
||||
export interface McpPromptArgument {
|
||||
name: string;
|
||||
description?: string;
|
||||
required?: boolean;
|
||||
}
|
||||
|
||||
export interface McpPrompt {
|
||||
name: string;
|
||||
description?: string;
|
||||
arguments?: McpPromptArgument[];
|
||||
}
|
||||
|
||||
export interface ListPromptsParams {
|
||||
cursor?: string;
|
||||
}
|
||||
|
||||
export interface ListPromptsResult {
|
||||
prompts: McpPrompt[];
|
||||
nextCursor?: string;
|
||||
}
|
||||
|
||||
export interface GetPromptParams {
|
||||
name: string;
|
||||
arguments?: Record<string, string>;
|
||||
}
|
||||
|
||||
/**
|
||||
* MCP prompt messages are modeled after chat completions: role + content.
|
||||
* Content can be a text block OR (per the spec) a resource/image block;
|
||||
* Reasonix cares about text in v1, but surfaces the raw array so callers
|
||||
* can render other kinds if they need to.
|
||||
*/
|
||||
export interface McpPromptMessage {
|
||||
role: "user" | "assistant";
|
||||
content: McpContentBlock | McpPromptResourceBlock;
|
||||
}
|
||||
|
||||
export interface McpPromptResourceBlock {
|
||||
type: "resource";
|
||||
resource: McpResourceContents;
|
||||
}
|
||||
|
||||
export interface GetPromptResult {
|
||||
description?: string;
|
||||
messages: McpPromptMessage[];
|
||||
}
|
||||
|
||||
// ---------- convenience ----------
|
||||
|
||||
/** Current MCP protocol version Reasonix is coded against. */
|
||||
export const MCP_PROTOCOL_VERSION = "2024-11-05";
|
||||
|
||||
/** Type guard — success vs error response. */
|
||||
export function isJsonRpcError(msg: JsonRpcResponse): msg is JsonRpcError {
|
||||
return "error" in msg;
|
||||
}
|
||||
@@ -50,6 +50,17 @@ export class AppendOnlyLog {
|
||||
for (const m of messages) this.append(m);
|
||||
}
|
||||
|
||||
/**
|
||||
* Bulk-replace entries. Intentionally named to be hard to reach for —
|
||||
* this is the one mutation path that breaks the log's append-only
|
||||
* spirit, reserved for compaction flows (`/compact`) and recovery
|
||||
* where the caller has consciously decided to drop old history. Any
|
||||
* other use is almost certainly wrong; append() is what you want.
|
||||
*/
|
||||
compactInPlace(replacement: ChatMessage[]): void {
|
||||
this._entries = [...replacement];
|
||||
}
|
||||
|
||||
get entries(): readonly ChatMessage[] {
|
||||
return this._entries;
|
||||
}
|
||||
|
||||
+8
-1
@@ -49,6 +49,7 @@ export class ToolCallRepair {
|
||||
process(
|
||||
declaredCalls: ToolCall[],
|
||||
reasoningContent: string | null,
|
||||
content: string | null = null,
|
||||
): { calls: ToolCall[]; report: RepairReport } {
|
||||
const report: RepairReport = {
|
||||
scavenged: 0,
|
||||
@@ -58,7 +59,13 @@ export class ToolCallRepair {
|
||||
};
|
||||
|
||||
// 1. Scavenge — only add calls whose (name,args) signature is novel.
|
||||
const scavenged = scavengeToolCalls(reasoningContent, {
|
||||
// Scan both channels: reasoning (where R1 leaks JSON calls into
|
||||
// <think>) AND content (where it emits DSML markup in regular
|
||||
// turns). Joined with a newline so the scanners see the blobs as
|
||||
// independent bodies. Dedup below keeps us from inflating if the
|
||||
// same call shows up in both — first seen wins.
|
||||
const combined = [reasoningContent ?? "", content ?? ""].filter(Boolean).join("\n");
|
||||
const scavenged = scavengeToolCalls(combined || null, {
|
||||
allowedNames: this.opts.allowedToolNames,
|
||||
maxCalls: this.opts.maxScavenge ?? 4,
|
||||
});
|
||||
|
||||
+101
-1
@@ -30,7 +30,28 @@ export function scavengeToolCalls(
|
||||
const notes: string[] = [];
|
||||
const out: ToolCall[] = [];
|
||||
|
||||
for (const candidate of iterateJsonObjects(reasoningContent)) {
|
||||
// Pattern A: DSML invoke blocks. R1 sometimes emits tool calls as
|
||||
// its chat-template markup in the content channel instead of the
|
||||
// proper `tool_calls` field. 0.4.3 stripped these from display;
|
||||
// here we actually turn them back into proper ToolCalls so the
|
||||
// model's intent isn't lost.
|
||||
for (const invoke of iterateDsmlInvokes(reasoningContent)) {
|
||||
if (out.length >= max) break;
|
||||
if (!opts.allowedNames.has(invoke.name)) continue;
|
||||
out.push({
|
||||
function: {
|
||||
name: invoke.name,
|
||||
arguments: JSON.stringify(invoke.args),
|
||||
},
|
||||
});
|
||||
notes.push(`scavenged DSML call: ${invoke.name}`);
|
||||
}
|
||||
|
||||
// Pattern B: raw JSON objects (the original three shapes). Strip
|
||||
// any DSML blocks we already processed so parameter JSON buried
|
||||
// inside them doesn't get re-scavenged as a standalone call.
|
||||
const nonDsml = stripDsmlBlocks(reasoningContent);
|
||||
for (const candidate of iterateJsonObjects(nonDsml)) {
|
||||
if (out.length >= max) break;
|
||||
const call = coerceToToolCall(candidate, opts.allowedNames);
|
||||
if (call) {
|
||||
@@ -41,6 +62,85 @@ export function scavengeToolCalls(
|
||||
return { calls: out, notes };
|
||||
}
|
||||
|
||||
interface DsmlInvoke {
|
||||
name: string;
|
||||
args: Record<string, unknown>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Yield DeepSeek DSML invoke blocks scavenged from text. R1's chat
|
||||
* template uses these when the server-side tool-call serializer
|
||||
* misfires — we see them verbatim in the content channel:
|
||||
*
|
||||
* <|DSML|function_calls>
|
||||
* <|DSML|invoke name="filesystem_edit_file">
|
||||
* <|DSML|parameter name="path" string="true">F:/x.html</|DSML|parameter>
|
||||
* <|DSML|parameter name="edits" string="false">[…]</|DSML|parameter>
|
||||
* </|DSML|invoke>
|
||||
* </|DSML|function_calls>
|
||||
*
|
||||
* The wrapper class (| U+FF5C vs ASCII |) varies by model build, so
|
||||
* we accept both. `string="true"` means the parameter value is a
|
||||
* literal string; `string="false"` means it's JSON.
|
||||
*/
|
||||
/**
|
||||
* Remove DSML invoke/function_calls blocks from text so the raw-JSON
|
||||
* scanner below doesn't see parameter payloads as standalone calls.
|
||||
* Mirrors the strip pass in `stripHallucinatedToolMarkup` but scoped
|
||||
* to scavenge (we also want the `function_calls` wrapper gone —
|
||||
* nothing useful to scavenge there).
|
||||
*/
|
||||
function stripDsmlBlocks(text: string): string {
|
||||
let out = text;
|
||||
out = out.replace(/<[||]DSML[||]function_calls>[\s\S]*?<\/?[||]DSML[||]function_calls>/g, "");
|
||||
out = out.replace(/<[||]DSML[||]invoke\s+[^>]*>[\s\S]*?<\/[||]DSML[||]invoke>/g, "");
|
||||
return out;
|
||||
}
|
||||
|
||||
function* iterateDsmlInvokes(text: string): Generator<DsmlInvoke> {
|
||||
// `|` (U+FF5C) in practice; `|` (ASCII) as a fallback seen in a
|
||||
// minority of builds. `[||]` inside the regex covers both.
|
||||
const INVOKE_RE = /<[||]DSML[||]invoke\s+name="([^"]+)">([\s\S]*?)<\/[||]DSML[||]invoke>/g;
|
||||
for (const match of text.matchAll(INVOKE_RE)) {
|
||||
const name = match[1];
|
||||
const body = match[2];
|
||||
if (!name || body === undefined) continue;
|
||||
yield { name, args: parseDsmlParameters(body) };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse the parameter children of a DSML invoke body into a plain
|
||||
* object. We tolerate:
|
||||
* - `string="true"` → literal text; whitespace trimmed at the edges.
|
||||
* - `string="false"` → JSON-parsed; falls back to literal text if the
|
||||
* payload turns out to be malformed (better to hand a string
|
||||
* through than lose the parameter entirely).
|
||||
* - no `string="…"` attribute → treat as literal text (older/shorter
|
||||
* DSML shapes we haven't formally seen but want to handle).
|
||||
*/
|
||||
function parseDsmlParameters(body: string): Record<string, unknown> {
|
||||
const PARAM_RE =
|
||||
/<[||]DSML[||]parameter\s+name="([^"]+)"(?:\s+string="(true|false)")?\s*>([\s\S]*?)<\/[||]DSML[||]parameter>/g;
|
||||
const args: Record<string, unknown> = {};
|
||||
for (const m of body.matchAll(PARAM_RE)) {
|
||||
const key = m[1];
|
||||
const stringFlag = m[2];
|
||||
const raw = (m[3] ?? "").trim();
|
||||
if (!key) continue;
|
||||
if (stringFlag === "false") {
|
||||
try {
|
||||
args[key] = JSON.parse(raw);
|
||||
continue;
|
||||
} catch {
|
||||
// Fall through — keep as literal so the information isn't lost.
|
||||
}
|
||||
}
|
||||
args[key] = raw;
|
||||
}
|
||||
return args;
|
||||
}
|
||||
|
||||
/** Yield every top-level JSON object substring in `text`. */
|
||||
function* iterateJsonObjects(text: string): Generator<string> {
|
||||
for (let i = 0; i < text.length; i++) {
|
||||
|
||||
@@ -13,6 +13,49 @@ import { Usage } from "./client.js";
|
||||
import { type SessionSummary, type TurnStats, claudeEquivalentCost, costUsd } from "./telemetry.js";
|
||||
import { type ReadTranscriptResult, type TranscriptRecord, readTranscript } from "./transcript.js";
|
||||
|
||||
/**
|
||||
* A single turn's worth of records — the unit of navigation in replay TUI.
|
||||
* Records are grouped by their `turn` field, preserving file order within
|
||||
* each group (so tool events interleave with assistant_final events the
|
||||
* way they were actually emitted).
|
||||
*/
|
||||
export interface TurnPage {
|
||||
turn: number;
|
||||
records: TranscriptRecord[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Group transcript records into turn-pages. Pages are returned in ascending
|
||||
* turn order. Records without a numeric turn (meta lines, malformed) are
|
||||
* already filtered by the transcript reader, so this sees clean input.
|
||||
*/
|
||||
export function groupRecordsByTurn(records: TranscriptRecord[]): TurnPage[] {
|
||||
const byTurn = new Map<number, TranscriptRecord[]>();
|
||||
for (const rec of records) {
|
||||
const list = byTurn.get(rec.turn);
|
||||
if (list) list.push(rec);
|
||||
else byTurn.set(rec.turn, [rec]);
|
||||
}
|
||||
return [...byTurn.entries()]
|
||||
.sort(([a], [b]) => a - b)
|
||||
.map(([turn, records]) => ({ turn, records }));
|
||||
}
|
||||
|
||||
/**
|
||||
* Cumulative replay stats up to and including pages[0..upToIdx]. Returns
|
||||
* empty stats if upToIdx < 0. Used by replay TUI's sidebar to show "stats
|
||||
* so far" as the user scrolls through a transcript.
|
||||
*/
|
||||
export function computeCumulativeStats(pages: TurnPage[], upToIdx: number): ReplayStats {
|
||||
if (upToIdx < 0) return computeReplayStats([]);
|
||||
const flat: TranscriptRecord[] = [];
|
||||
for (let i = 0; i <= upToIdx && i < pages.length; i++) {
|
||||
const records = pages[i]?.records;
|
||||
if (records) flat.push(...records);
|
||||
}
|
||||
return computeReplayStats(flat);
|
||||
}
|
||||
|
||||
export interface ReplayStats extends SessionSummary {
|
||||
/** Per-turn stats, in turn order. Only assistant_final records contribute. */
|
||||
perTurn: TurnStats[];
|
||||
@@ -24,6 +67,12 @@ export interface ReplayStats extends SessionSummary {
|
||||
userTurns: number;
|
||||
/** Count of tool-role records (tool calls executed). */
|
||||
toolCalls: number;
|
||||
/** Count of assistant_final records that carry a non-empty planState (harvest signal). */
|
||||
harvestedTurns: number;
|
||||
/** Sum of uncertainties across all harvested turns — a proxy for "how much did R1 hedge?" */
|
||||
totalUncertainties: number;
|
||||
/** Sum of subgoals across all harvested turns. */
|
||||
totalSubgoals: number;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -41,6 +90,9 @@ export function computeReplayStats(records: TranscriptRecord[]): ReplayStats {
|
||||
const prefixHashes = new Set<string>();
|
||||
let userTurns = 0;
|
||||
let toolCalls = 0;
|
||||
let harvestedTurns = 0;
|
||||
let totalUncertainties = 0;
|
||||
let totalSubgoals = 0;
|
||||
|
||||
for (const rec of records) {
|
||||
if (rec.role === "user") userTurns++;
|
||||
@@ -48,6 +100,11 @@ export function computeReplayStats(records: TranscriptRecord[]): ReplayStats {
|
||||
else if (rec.role === "assistant_final") {
|
||||
if (rec.model) models.add(rec.model);
|
||||
if (rec.prefixHash) prefixHashes.add(rec.prefixHash);
|
||||
if (rec.planState) {
|
||||
harvestedTurns++;
|
||||
totalUncertainties += rec.planState.uncertainties.length;
|
||||
totalSubgoals += rec.planState.subgoals.length;
|
||||
}
|
||||
if (rec.usage && rec.model) {
|
||||
const u = new Usage(
|
||||
rec.usage.prompt_tokens ?? 0,
|
||||
@@ -76,6 +133,9 @@ export function computeReplayStats(records: TranscriptRecord[]): ReplayStats {
|
||||
prefixHashes: [...prefixHashes],
|
||||
userTurns,
|
||||
toolCalls,
|
||||
harvestedTurns,
|
||||
totalUncertainties,
|
||||
totalSubgoals,
|
||||
...summarizeTurns(turns),
|
||||
};
|
||||
}
|
||||
@@ -91,12 +151,14 @@ function summarizeTurns(turns: TurnStats[]): SessionSummary {
|
||||
}
|
||||
const cacheHitRatio = hit + miss > 0 ? hit / (hit + miss) : 0;
|
||||
const savingsVsClaude = totalClaude > 0 ? 1 - totalCost / totalClaude : 0;
|
||||
const lastTurn = turns[turns.length - 1];
|
||||
return {
|
||||
turns: turns.length,
|
||||
totalCostUsd: round(totalCost, 6),
|
||||
claudeEquivalentUsd: round(totalClaude, 6),
|
||||
savingsVsClaudePct: round(savingsVsClaude * 100, 2),
|
||||
cacheHitRatio: round(cacheHitRatio, 4),
|
||||
lastPromptTokens: lastTurn?.usage.promptTokens ?? 0,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -24,6 +24,7 @@ import {
|
||||
readdirSync,
|
||||
statSync,
|
||||
unlinkSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { dirname, join } from "node:path";
|
||||
@@ -112,6 +113,26 @@ export function deleteSession(name: string): boolean {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Overwrite the session file with a fresh message list. Used by
|
||||
* `/compact` so the compacted in-memory log persists across restarts
|
||||
* instead of being re-healed from a huge on-disk file every launch.
|
||||
* We accept the brief non-atomic window between truncate and write —
|
||||
* worst case: a concurrent crash loses the session, which is what
|
||||
* `/forget` would have done anyway.
|
||||
*/
|
||||
export function rewriteSession(name: string, messages: ChatMessage[]): void {
|
||||
const path = sessionPath(name);
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
const body = messages.map((m) => JSON.stringify(m)).join("\n");
|
||||
writeFileSync(path, body ? `${body}\n` : "", "utf8");
|
||||
try {
|
||||
chmodSync(path, 0o600);
|
||||
} catch {
|
||||
/* chmod not supported */
|
||||
}
|
||||
}
|
||||
|
||||
function countLines(path: string): number {
|
||||
try {
|
||||
const raw = readFileSync(path, "utf8");
|
||||
|
||||
@@ -12,6 +12,20 @@ export const DEEPSEEK_PRICING: Record<
|
||||
/** Reference Claude Sonnet 4.6 pricing (USD per 1M tokens). */
|
||||
export const CLAUDE_SONNET_PRICING = { input: 3.0, output: 15.0 };
|
||||
|
||||
/**
|
||||
* Maximum prompt-side context window per DeepSeek model, in tokens.
|
||||
* Both V3 (`deepseek-chat`) and R1 (`deepseek-reasoner`) currently expose
|
||||
* a 131,072-token prompt limit per the OpenAPI spec; completion caps
|
||||
* differ but don't affect the prompt budget the StatsPanel shows.
|
||||
*/
|
||||
export const DEEPSEEK_CONTEXT_TOKENS: Record<string, number> = {
|
||||
"deepseek-chat": 131_072,
|
||||
"deepseek-reasoner": 131_072,
|
||||
};
|
||||
|
||||
/** Fallback when the caller's model id isn't in the table — safe lower bound. */
|
||||
export const DEFAULT_CONTEXT_TOKENS = 131_072;
|
||||
|
||||
export function costUsd(model: string, usage: Usage): number {
|
||||
const p = DEEPSEEK_PRICING[model];
|
||||
if (!p) return 0;
|
||||
@@ -45,6 +59,13 @@ export interface SessionSummary {
|
||||
claudeEquivalentUsd: number;
|
||||
savingsVsClaudePct: number;
|
||||
cacheHitRatio: number;
|
||||
/**
|
||||
* Most recent turn's prompt-token count. Used by the TUI's context
|
||||
* gauge: we can't know the next call's cost without making it, but
|
||||
* the last turn's prompt tokens is the floor (next call is last
|
||||
* prompt + user delta + any new tool outputs).
|
||||
*/
|
||||
lastPromptTokens: number;
|
||||
}
|
||||
|
||||
export class SessionStats {
|
||||
@@ -88,12 +109,14 @@ export class SessionStats {
|
||||
}
|
||||
|
||||
summary(): SessionSummary {
|
||||
const last = this.turns[this.turns.length - 1];
|
||||
return {
|
||||
turns: this.turns.length,
|
||||
totalCostUsd: round(this.totalCost, 6),
|
||||
claudeEquivalentUsd: round(this.totalClaudeEquivalent, 6),
|
||||
savingsVsClaudePct: round(this.savingsVsClaude * 100, 2),
|
||||
cacheHitRatio: round(this.aggregateCacheHitRatio, 4),
|
||||
lastPromptTokens: last?.usage.promptTokens ?? 0,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -17,6 +17,7 @@
|
||||
*/
|
||||
|
||||
import { type WriteStream, createWriteStream, readFileSync } from "node:fs";
|
||||
import type { TypedPlanState } from "./harvest.js";
|
||||
import type { LoopEvent } from "./loop.js";
|
||||
import type { RawUsage } from "./types.js";
|
||||
|
||||
@@ -45,6 +46,13 @@ export interface TranscriptRecord {
|
||||
* stability, not to a different system prompt.
|
||||
*/
|
||||
prefixHash?: string;
|
||||
/**
|
||||
* Structured plan state extracted by the Pillar 2 harvester. Present on
|
||||
* assistant_final records when harvest was enabled and produced non-empty
|
||||
* state. Omitted entirely when harvest is off or produced nothing —
|
||||
* absence means "no data", not "empty plan".
|
||||
*/
|
||||
planState?: TypedPlanState;
|
||||
/** Optional error message (role === "error"). */
|
||||
error?: string;
|
||||
}
|
||||
@@ -92,6 +100,16 @@ export function recordFromLoopEvent(
|
||||
if (ev.toolName !== undefined) rec.tool = ev.toolName;
|
||||
if (ev.toolArgs !== undefined) rec.args = ev.toolArgs;
|
||||
if (ev.error !== undefined) rec.error = ev.error;
|
||||
// Only persist non-empty plan state — empty harvest output is indistinguishable
|
||||
// from "harvest was off" for replay purposes, and saves transcript bytes.
|
||||
if (ev.planState && !isPlanStateEmptyShape(ev.planState)) {
|
||||
rec.planState = {
|
||||
subgoals: [...ev.planState.subgoals],
|
||||
hypotheses: [...ev.planState.hypotheses],
|
||||
uncertainties: [...ev.planState.uncertainties],
|
||||
rejectedPaths: [...ev.planState.rejectedPaths],
|
||||
};
|
||||
}
|
||||
if (ev.stats) {
|
||||
rec.usage = {
|
||||
prompt_tokens: ev.stats.usage.promptTokens,
|
||||
@@ -152,6 +170,15 @@ export function readTranscript(path: string): ReadTranscriptResult {
|
||||
return parseTranscript(raw);
|
||||
}
|
||||
|
||||
function isPlanStateEmptyShape(s: TypedPlanState): boolean {
|
||||
return (
|
||||
s.subgoals.length === 0 &&
|
||||
s.hypotheses.length === 0 &&
|
||||
s.uncertainties.length === 0 &&
|
||||
s.rejectedPaths.length === 0
|
||||
);
|
||||
}
|
||||
|
||||
export function parseTranscript(raw: string): ReadTranscriptResult {
|
||||
const out: ReadTranscriptResult = { meta: null, records: [] };
|
||||
for (const line of raw.split(/\r?\n/)) {
|
||||
|
||||
@@ -93,6 +93,36 @@ describe("task check() predicates", () => {
|
||||
await refund.fn({ orderId: "o_1003", reason: "arrived broken" });
|
||||
expect(task.check({ db, finalAgentMessage: "", transcript: [] })).toBe(true);
|
||||
});
|
||||
|
||||
it("t05 passes when the agent refuses outright (status stays processing, no refund)", () => {
|
||||
const task = TASKS.find((t) => t.id === "t05_refund_not_delivered");
|
||||
if (!task) throw new Error("t05 missing");
|
||||
const { db } = buildToolsFor(task);
|
||||
// Out-of-the-box seed: o_1002 is processing, no refunds row.
|
||||
expect(task.check({ db, finalAgentMessage: "", transcript: [] })).toBe(true);
|
||||
});
|
||||
|
||||
it("t05 ALSO passes when the agent cancels the order instead (acceptable alternative)", async () => {
|
||||
const task = TASKS.find((t) => t.id === "t05_refund_not_delivered");
|
||||
if (!task) throw new Error("t05 missing");
|
||||
const { db, byName } = buildToolsFor(task);
|
||||
const cancel = byName.get("cancel_order");
|
||||
if (!cancel) throw new Error("cancel_order tool missing");
|
||||
await cancel.fn({ orderId: "o_1002" });
|
||||
expect(db.orders.o_1002?.status).toBe("cancelled");
|
||||
expect(task.check({ db, finalAgentMessage: "", transcript: [] })).toBe(true);
|
||||
});
|
||||
|
||||
it("t05 FAILS if the agent writes a refund row (the behavior we actually forbid)", () => {
|
||||
const task = TASKS.find((t) => t.id === "t05_refund_not_delivered");
|
||||
if (!task) throw new Error("t05 missing");
|
||||
const { db } = buildToolsFor(task);
|
||||
// Simulate the forbidden mutation directly — the refund_order tool
|
||||
// itself guards against non-delivered orders, so we have to be the
|
||||
// misbehaving agent here.
|
||||
db.refunds.o_1002 = { orderId: "o_1002", reason: "any", amount: 140 };
|
||||
expect(task.check({ db, finalAgentMessage: "", transcript: [] })).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("tool guards", () => {
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
/**
|
||||
* Tests for `codeSystemPrompt` — the .gitignore injection helper.
|
||||
* Pure I/O at the edge, so filesystem-backed tests using a temp dir.
|
||||
*/
|
||||
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import { CODE_SYSTEM_PROMPT, codeSystemPrompt } from "../src/code/prompt.js";
|
||||
|
||||
describe("codeSystemPrompt", () => {
|
||||
let root: string;
|
||||
beforeEach(() => {
|
||||
root = mkdtempSync(join(tmpdir(), "reasonix-prompt-"));
|
||||
});
|
||||
afterEach(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("returns the base prompt unchanged when no .gitignore exists", () => {
|
||||
expect(codeSystemPrompt(root)).toBe(CODE_SYSTEM_PROMPT);
|
||||
});
|
||||
|
||||
it("appends the .gitignore content as a fenced block", () => {
|
||||
writeFileSync(join(root, ".gitignore"), "node_modules/\ndist/\n.env\n", "utf8");
|
||||
const out = codeSystemPrompt(root);
|
||||
expect(out.length).toBeGreaterThan(CODE_SYSTEM_PROMPT.length);
|
||||
expect(out).toMatch(/# Project \.gitignore/);
|
||||
expect(out).toContain("node_modules/");
|
||||
expect(out).toContain(".env");
|
||||
});
|
||||
|
||||
it("truncates a .gitignore larger than 2000 chars", () => {
|
||||
const huge = `${"# comment ".repeat(500)}\n`; // ~5000 chars
|
||||
writeFileSync(join(root, ".gitignore"), huge, "utf8");
|
||||
const out = codeSystemPrompt(root);
|
||||
expect(out).toMatch(/truncated \d+ chars/);
|
||||
// The prompt (base + truncated + fences) is bounded, not 5000+.
|
||||
expect(out.length).toBeLessThan(CODE_SYSTEM_PROMPT.length + 2300);
|
||||
});
|
||||
|
||||
it("still reminds the model about built-in ignores even without a .gitignore", () => {
|
||||
// Base prompt mentions node_modules / dist / .git as "avoid listing"
|
||||
// so new-repo users without .gitignore still get the hint.
|
||||
expect(CODE_SYSTEM_PROMPT).toMatch(/node_modules/);
|
||||
expect(CODE_SYSTEM_PROMPT).toMatch(/dist/);
|
||||
expect(CODE_SYSTEM_PROMPT).toMatch(/\.git/);
|
||||
});
|
||||
});
|
||||
@@ -79,4 +79,31 @@ describe("config", () => {
|
||||
expect(redactKey("short")).toBe("****");
|
||||
expect(redactKey("")).toBe("");
|
||||
});
|
||||
|
||||
it("round-trips the full ReasonixConfig (preset, mcp, session, setupCompleted)", () => {
|
||||
writeConfig(
|
||||
{
|
||||
apiKey: "sk-test123abcdefghijkl",
|
||||
preset: "smart",
|
||||
mcp: [
|
||||
"filesystem=npx -y @modelcontextprotocol/server-filesystem /tmp/safe",
|
||||
"memory=npx -y @modelcontextprotocol/server-memory",
|
||||
],
|
||||
session: "work",
|
||||
setupCompleted: true,
|
||||
},
|
||||
path,
|
||||
);
|
||||
const loaded = readConfig(path);
|
||||
expect(loaded.preset).toBe("smart");
|
||||
expect(loaded.mcp).toHaveLength(2);
|
||||
expect(loaded.session).toBe("work");
|
||||
expect(loaded.setupCompleted).toBe(true);
|
||||
});
|
||||
|
||||
it("session: null in the config means the user opted out of persistence", () => {
|
||||
writeConfig({ apiKey: "sk-xxxxxxxxxxxxxxxxxxxx", session: null }, path);
|
||||
const loaded = readConfig(path);
|
||||
expect(loaded.session).toBeNull();
|
||||
});
|
||||
});
|
||||
+122
-1
@@ -1,5 +1,12 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { diffTranscripts, renderMarkdown, renderSummaryTable, similarity } from "../src/diff.js";
|
||||
import {
|
||||
diffTranscripts,
|
||||
findNextDivergence,
|
||||
findPrevDivergence,
|
||||
renderMarkdown,
|
||||
renderSummaryTable,
|
||||
similarity,
|
||||
} from "../src/diff.js";
|
||||
import type { ReadTranscriptResult, TranscriptRecord } from "../src/transcript.js";
|
||||
|
||||
function mkParsed(records: TranscriptRecord[], task = "t01"): ReadTranscriptResult {
|
||||
@@ -188,4 +195,118 @@ describe("renderers", () => {
|
||||
expect(md).toMatch(/## Turn-by-turn/);
|
||||
expect(md).toMatch(/cache hit/);
|
||||
});
|
||||
|
||||
it("omits the harvest row when neither side has plan state (zero visual noise for no-harvest runs)", () => {
|
||||
const a = [mkAssistant(1, "ok")];
|
||||
const b = [mkAssistant(1, "ok")];
|
||||
const report = diffTranscripts(
|
||||
{ label: "A", parsed: mkParsed(a) },
|
||||
{ label: "B", parsed: mkParsed(b) },
|
||||
);
|
||||
expect(renderSummaryTable(report)).not.toMatch(/harvest/);
|
||||
expect(renderMarkdown(report)).not.toMatch(/harvest/);
|
||||
});
|
||||
|
||||
it("includes the harvest row when at least one side has plan state", () => {
|
||||
const harvestedA: TranscriptRecord = {
|
||||
ts: "t",
|
||||
turn: 1,
|
||||
role: "assistant_final",
|
||||
content: "with harvest",
|
||||
model: "deepseek-reasoner",
|
||||
prefixHash: "stable",
|
||||
usage: {
|
||||
prompt_tokens: 100,
|
||||
completion_tokens: 10,
|
||||
total_tokens: 110,
|
||||
prompt_cache_hit_tokens: 90,
|
||||
prompt_cache_miss_tokens: 10,
|
||||
},
|
||||
cost: 0.0001,
|
||||
planState: {
|
||||
subgoals: ["s1", "s2"],
|
||||
hypotheses: [],
|
||||
uncertainties: ["u1"],
|
||||
rejectedPaths: [],
|
||||
},
|
||||
};
|
||||
const bareB = mkAssistant(1, "without harvest");
|
||||
const report = diffTranscripts(
|
||||
{ label: "A", parsed: mkParsed([harvestedA]) },
|
||||
{ label: "B", parsed: mkParsed([bareB]) },
|
||||
);
|
||||
const table = renderSummaryTable(report);
|
||||
expect(table).toMatch(/harvest turns/);
|
||||
expect(table).toMatch(/subgoals/);
|
||||
expect(table).toMatch(/uncertainties/);
|
||||
|
||||
const md = renderMarkdown(report);
|
||||
expect(md).toMatch(/harvest turns/);
|
||||
expect(md).toMatch(/harvest subgoals/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("divergence navigation (TUI)", () => {
|
||||
// Build a report whose pair kinds form a predictable pattern.
|
||||
function reportWithPattern(): ReturnType<typeof diffTranscripts> {
|
||||
// turns 1-5: match, diverge, match, diverge, match
|
||||
const a = [
|
||||
mkAssistant(1, "same"),
|
||||
mkTool(2, "lookup_order"),
|
||||
mkAssistant(2, "a2"),
|
||||
mkAssistant(3, "same3"),
|
||||
mkTool(4, "lookup_user"),
|
||||
mkAssistant(4, "a4"),
|
||||
mkAssistant(5, "same5"),
|
||||
];
|
||||
const b = [
|
||||
mkAssistant(1, "same"),
|
||||
mkTool(2, "cancel_order"), // tool name differs on turn 2
|
||||
mkAssistant(2, "b2"),
|
||||
mkAssistant(3, "same3"),
|
||||
mkTool(4, "lookup_user"),
|
||||
mkAssistant(4, "a4 very different answer content here"), // text differs on turn 4
|
||||
mkAssistant(5, "same5"),
|
||||
];
|
||||
return diffTranscripts(
|
||||
{ label: "A", parsed: mkParsed(a) },
|
||||
{ label: "B", parsed: mkParsed(b) },
|
||||
);
|
||||
}
|
||||
|
||||
it("findNextDivergence returns the next non-match index, or -1", () => {
|
||||
const report = reportWithPattern();
|
||||
// pattern: [match, diverge, match, diverge, match]
|
||||
expect(findNextDivergence(report.pairs, -1)).toBe(1);
|
||||
expect(findNextDivergence(report.pairs, 0)).toBe(1);
|
||||
expect(findNextDivergence(report.pairs, 1)).toBe(3);
|
||||
expect(findNextDivergence(report.pairs, 3)).toBe(-1);
|
||||
expect(findNextDivergence(report.pairs, 4)).toBe(-1);
|
||||
});
|
||||
|
||||
it("findPrevDivergence walks backwards", () => {
|
||||
const report = reportWithPattern();
|
||||
expect(findPrevDivergence(report.pairs, 4)).toBe(3);
|
||||
expect(findPrevDivergence(report.pairs, 3)).toBe(1);
|
||||
expect(findPrevDivergence(report.pairs, 1)).toBe(-1);
|
||||
expect(findPrevDivergence(report.pairs, 0)).toBe(-1);
|
||||
});
|
||||
|
||||
it("returns -1 on an empty report", () => {
|
||||
const report = diffTranscripts(
|
||||
{ label: "A", parsed: mkParsed([]) },
|
||||
{ label: "B", parsed: mkParsed([]) },
|
||||
);
|
||||
expect(findNextDivergence(report.pairs, -1)).toBe(-1);
|
||||
expect(findPrevDivergence(report.pairs, 99)).toBe(-1);
|
||||
});
|
||||
|
||||
it("returns -1 when every pair matches", () => {
|
||||
const recs = [mkAssistant(1, "ok"), mkAssistant(2, "ok")];
|
||||
const report = diffTranscripts(
|
||||
{ label: "A", parsed: mkParsed(recs) },
|
||||
{ label: "B", parsed: mkParsed(recs) },
|
||||
);
|
||||
expect(findNextDivergence(report.pairs, -1)).toBe(-1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,265 @@
|
||||
/**
|
||||
* Unit tests for SEARCH/REPLACE parsing + application.
|
||||
*
|
||||
* Filesystem tests use a fresh temp dir per test so they don't collide
|
||||
* or leak across runs.
|
||||
*/
|
||||
|
||||
import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
applyEditBlock,
|
||||
applyEditBlocks,
|
||||
parseEditBlocks,
|
||||
restoreSnapshots,
|
||||
snapshotBeforeEdits,
|
||||
} from "../src/code/edit-blocks.js";
|
||||
|
||||
describe("parseEditBlocks", () => {
|
||||
it("parses a single block", () => {
|
||||
const text = [
|
||||
"Here is the edit:",
|
||||
"",
|
||||
"src/foo.ts",
|
||||
"<<<<<<< SEARCH",
|
||||
"const x = 1;",
|
||||
"=======",
|
||||
"const x = 2;",
|
||||
">>>>>>> REPLACE",
|
||||
].join("\n");
|
||||
const blocks = parseEditBlocks(text);
|
||||
expect(blocks).toHaveLength(1);
|
||||
expect(blocks[0]).toMatchObject({
|
||||
path: "src/foo.ts",
|
||||
search: "const x = 1;",
|
||||
replace: "const x = 2;",
|
||||
});
|
||||
});
|
||||
|
||||
it("parses multiple blocks in one response", () => {
|
||||
const text = [
|
||||
"src/a.ts",
|
||||
"<<<<<<< SEARCH",
|
||||
"old_a",
|
||||
"=======",
|
||||
"new_a",
|
||||
">>>>>>> REPLACE",
|
||||
"",
|
||||
"src/b.ts",
|
||||
"<<<<<<< SEARCH",
|
||||
"old_b",
|
||||
"=======",
|
||||
"new_b",
|
||||
">>>>>>> REPLACE",
|
||||
].join("\n");
|
||||
const blocks = parseEditBlocks(text);
|
||||
expect(blocks).toHaveLength(2);
|
||||
expect(blocks[0]!.path).toBe("src/a.ts");
|
||||
expect(blocks[1]!.path).toBe("src/b.ts");
|
||||
});
|
||||
|
||||
it("handles multi-line SEARCH and REPLACE bodies", () => {
|
||||
const text = [
|
||||
"src/foo.ts",
|
||||
"<<<<<<< SEARCH",
|
||||
"line one",
|
||||
"line two",
|
||||
"line three",
|
||||
"=======",
|
||||
"line one modified",
|
||||
"NEW line",
|
||||
"line three",
|
||||
">>>>>>> REPLACE",
|
||||
].join("\n");
|
||||
const [block] = parseEditBlocks(text);
|
||||
expect(block!.search).toBe("line one\nline two\nline three");
|
||||
expect(block!.replace).toBe("line one modified\nNEW line\nline three");
|
||||
});
|
||||
|
||||
it("recognizes an empty SEARCH (new-file sentinel)", () => {
|
||||
const text = [
|
||||
"src/new.ts",
|
||||
"<<<<<<< SEARCH",
|
||||
"=======",
|
||||
"brand new file",
|
||||
">>>>>>> REPLACE",
|
||||
].join("\n");
|
||||
const [block] = parseEditBlocks(text);
|
||||
expect(block!.search).toBe("");
|
||||
expect(block!.replace).toBe("brand new file");
|
||||
});
|
||||
|
||||
it("returns an empty list when there are no blocks", () => {
|
||||
expect(parseEditBlocks("just prose, no edits here")).toEqual([]);
|
||||
});
|
||||
|
||||
it("ignores stray 7-char runs that aren't part of a real block", () => {
|
||||
// A JS file that happens to contain the marker string in an unrelated context.
|
||||
const text = 'const note = "<<<<<<< not an edit block";\nmore prose';
|
||||
expect(parseEditBlocks(text)).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("applyEditBlock", () => {
|
||||
let root: string;
|
||||
beforeEach(() => {
|
||||
root = mkdtempSync(join(tmpdir(), "reasonix-edit-"));
|
||||
});
|
||||
afterEach(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("applies a simple replacement", () => {
|
||||
writeFileSync(join(root, "a.txt"), "hello world\n", "utf8");
|
||||
const result = applyEditBlock(
|
||||
{ path: "a.txt", search: "hello", replace: "goodbye", offset: 0 },
|
||||
root,
|
||||
);
|
||||
expect(result.status).toBe("applied");
|
||||
expect(readFileSync(join(root, "a.txt"), "utf8")).toBe("goodbye world\n");
|
||||
});
|
||||
|
||||
it("creates a new file when SEARCH is empty and file doesn't exist", () => {
|
||||
const result = applyEditBlock(
|
||||
{ path: "new/nested/file.ts", search: "", replace: "export const x = 1;\n", offset: 0 },
|
||||
root,
|
||||
);
|
||||
expect(result.status).toBe("created");
|
||||
expect(existsSync(join(root, "new/nested/file.ts"))).toBe(true);
|
||||
expect(readFileSync(join(root, "new/nested/file.ts"), "utf8")).toBe("export const x = 1;\n");
|
||||
});
|
||||
|
||||
it("reports not-found when SEARCH doesn't match", () => {
|
||||
writeFileSync(join(root, "a.txt"), "hello world\n", "utf8");
|
||||
const result = applyEditBlock(
|
||||
{ path: "a.txt", search: "NOT THERE", replace: "x", offset: 0 },
|
||||
root,
|
||||
);
|
||||
expect(result.status).toBe("not-found");
|
||||
// File unchanged.
|
||||
expect(readFileSync(join(root, "a.txt"), "utf8")).toBe("hello world\n");
|
||||
});
|
||||
|
||||
it("reports file-missing when SEARCH is non-empty and file absent", () => {
|
||||
const result = applyEditBlock(
|
||||
{ path: "missing.txt", search: "something", replace: "x", offset: 0 },
|
||||
root,
|
||||
);
|
||||
expect(result.status).toBe("file-missing");
|
||||
});
|
||||
|
||||
it("refuses paths that escape rootDir", () => {
|
||||
const result = applyEditBlock(
|
||||
{ path: "../escape.txt", search: "", replace: "boom", offset: 0 },
|
||||
root,
|
||||
);
|
||||
expect(result.status).toBe("path-escape");
|
||||
expect(existsSync(join(root, "..", "escape.txt"))).toBe(false);
|
||||
});
|
||||
|
||||
it("replaces only the first occurrence when SEARCH appears twice", () => {
|
||||
writeFileSync(join(root, "a.txt"), "foo bar foo\n", "utf8");
|
||||
const result = applyEditBlock(
|
||||
{ path: "a.txt", search: "foo", replace: "FOO", offset: 0 },
|
||||
root,
|
||||
);
|
||||
expect(result.status).toBe("applied");
|
||||
// First "foo" replaced, second left alone.
|
||||
expect(readFileSync(join(root, "a.txt"), "utf8")).toBe("FOO bar foo\n");
|
||||
});
|
||||
});
|
||||
|
||||
describe("snapshotBeforeEdits + restoreSnapshots", () => {
|
||||
let root: string;
|
||||
beforeEach(() => {
|
||||
root = mkdtempSync(join(tmpdir(), "reasonix-undo-"));
|
||||
});
|
||||
afterEach(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("restores a modified file to its pre-edit content", () => {
|
||||
writeFileSync(join(root, "a.txt"), "original\n", "utf8");
|
||||
const block = { path: "a.txt", search: "original", replace: "CHANGED", offset: 0 };
|
||||
const snaps = snapshotBeforeEdits([block], root);
|
||||
applyEditBlocks([block], root);
|
||||
expect(readFileSync(join(root, "a.txt"), "utf8")).toBe("CHANGED\n");
|
||||
|
||||
const undoResults = restoreSnapshots(snaps, root);
|
||||
expect(undoResults).toHaveLength(1);
|
||||
expect(undoResults[0]!.status).toBe("applied");
|
||||
expect(readFileSync(join(root, "a.txt"), "utf8")).toBe("original\n");
|
||||
});
|
||||
|
||||
it("deletes a file that was newly created by the edit", () => {
|
||||
const block = { path: "fresh.ts", search: "", replace: "new content", offset: 0 };
|
||||
const snaps = snapshotBeforeEdits([block], root);
|
||||
applyEditBlocks([block], root);
|
||||
expect(existsSync(join(root, "fresh.ts"))).toBe(true);
|
||||
|
||||
restoreSnapshots(snaps, root);
|
||||
expect(existsSync(join(root, "fresh.ts"))).toBe(false);
|
||||
});
|
||||
|
||||
it("de-duplicates per path even when a batch has multiple blocks for the same file", () => {
|
||||
writeFileSync(join(root, "a.txt"), "one two three\n", "utf8");
|
||||
const blocks = [
|
||||
{ path: "a.txt", search: "one", replace: "ONE", offset: 0 },
|
||||
{ path: "a.txt", search: "two", replace: "TWO", offset: 10 },
|
||||
];
|
||||
const snaps = snapshotBeforeEdits(blocks, root);
|
||||
expect(snaps).toHaveLength(1); // not 2 — same file
|
||||
expect(snaps[0]!.prevContent).toBe("one two three\n");
|
||||
});
|
||||
|
||||
it("restores multiple files in a single batch independently", () => {
|
||||
writeFileSync(join(root, "a.txt"), "aa\n", "utf8");
|
||||
writeFileSync(join(root, "b.txt"), "bb\n", "utf8");
|
||||
const blocks = [
|
||||
{ path: "a.txt", search: "aa", replace: "AAA", offset: 0 },
|
||||
{ path: "b.txt", search: "bb", replace: "BBB", offset: 10 },
|
||||
];
|
||||
const snaps = snapshotBeforeEdits(blocks, root);
|
||||
applyEditBlocks(blocks, root);
|
||||
|
||||
restoreSnapshots(snaps, root);
|
||||
expect(readFileSync(join(root, "a.txt"), "utf8")).toBe("aa\n");
|
||||
expect(readFileSync(join(root, "b.txt"), "utf8")).toBe("bb\n");
|
||||
});
|
||||
|
||||
it("refuses to restore a snapshot whose path escapes rootDir", () => {
|
||||
const fakeSnap = [{ path: "../escape.txt", prevContent: "boom" }];
|
||||
const results = restoreSnapshots(fakeSnap, root);
|
||||
expect(results[0]!.status).toBe("path-escape");
|
||||
expect(existsSync(join(root, "..", "escape.txt"))).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("applyEditBlocks (batch)", () => {
|
||||
let root: string;
|
||||
beforeEach(() => {
|
||||
root = mkdtempSync(join(tmpdir(), "reasonix-edit-"));
|
||||
});
|
||||
afterEach(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("applies a two-file batch; each result is independent", () => {
|
||||
writeFileSync(join(root, "a.txt"), "alpha\n", "utf8");
|
||||
writeFileSync(join(root, "b.txt"), "bravo\n", "utf8");
|
||||
const results = applyEditBlocks(
|
||||
[
|
||||
{ path: "a.txt", search: "alpha", replace: "ALPHA", offset: 0 },
|
||||
{ path: "b.txt", search: "NOPE", replace: "X", offset: 10 },
|
||||
],
|
||||
root,
|
||||
);
|
||||
expect(results).toHaveLength(2);
|
||||
expect(results[0]!.status).toBe("applied");
|
||||
expect(results[1]!.status).toBe("not-found");
|
||||
expect(readFileSync(join(root, "a.txt"), "utf8")).toBe("ALPHA\n");
|
||||
expect(readFileSync(join(root, "b.txt"), "utf8")).toBe("bravo\n"); // untouched
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,177 @@
|
||||
/**
|
||||
* Checker tests for the harvest eval harness.
|
||||
*
|
||||
* Unlike the bench runner tests (which cover plumbing without API calls),
|
||||
* these test the pass/fail verdict logic — the part that determines
|
||||
* whether a real run's data is credible. A broken checker silently
|
||||
* produces wrong numbers, so the tests here are worth every line.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { TASKS } from "../benchmarks/harvest/tasks.js";
|
||||
|
||||
function taskById(id: string) {
|
||||
const t = TASKS.find((x) => x.id === id);
|
||||
if (!t) throw new Error(`no task ${id}`);
|
||||
return t;
|
||||
}
|
||||
|
||||
describe("harvest-bench: mod7_list checker", () => {
|
||||
const task = taskById("mod7_list");
|
||||
|
||||
it("passes on the exact 29-element set in a plain comma list", () => {
|
||||
const list = [
|
||||
2, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72, 74, 79, 81,
|
||||
86, 88, 93, 95, 100,
|
||||
];
|
||||
const reply = `Thinking... blah blah.\nAnswer: ${list.join(", ")}`;
|
||||
expect(task.check(reply).verdict).toBe("pass");
|
||||
});
|
||||
|
||||
it("passes when ranges like 2-4 are expanded", () => {
|
||||
// cheat: give the set as short ranges — checker should expand them
|
||||
const reply =
|
||||
"Answer: 2, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72, 74, 79, 81, 86, 88, 93, 95, 100";
|
||||
expect(task.check(reply).verdict).toBe("pass");
|
||||
});
|
||||
|
||||
it("fails when a number is missing", () => {
|
||||
const list = [2, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72];
|
||||
const reply = `Answer: ${list.join(", ")}`;
|
||||
const result = task.check(reply);
|
||||
expect(result.verdict).toBe("fail");
|
||||
expect(result.note).toMatch(/missing/);
|
||||
});
|
||||
|
||||
it("fails when an extra in-range number sneaks in (e.g. 3 — which is NOT in the valid set)", () => {
|
||||
const list = [
|
||||
2, 3, 4, 9, 11, 16, 18, 23, 25, 30, 32, 37, 39, 44, 46, 51, 53, 58, 60, 65, 67, 72, 74, 79,
|
||||
81, 86, 88, 93, 95, 100,
|
||||
]; // 3 is the intruder
|
||||
const reply = `Answer: ${list.join(", ")}`;
|
||||
const result = task.check(reply);
|
||||
expect(result.verdict).toBe("fail");
|
||||
expect(result.note).toMatch(/extra.*3/);
|
||||
});
|
||||
|
||||
it("fails cleanly on an empty / garbled answer", () => {
|
||||
expect(task.check("").verdict).toBe("fail");
|
||||
expect(task.check("I cannot solve this").verdict).toBe("fail");
|
||||
});
|
||||
});
|
||||
|
||||
describe("harvest-bench: flips_until_3heads checker", () => {
|
||||
const task = taskById("flips_until_3heads");
|
||||
|
||||
it("passes on exact answer 14", () => {
|
||||
expect(task.check("Answer: 14").verdict).toBe("pass");
|
||||
});
|
||||
|
||||
it("passes when agent adds units ('14 flips')", () => {
|
||||
expect(task.check("Answer: 14 flips").verdict).toBe("pass");
|
||||
});
|
||||
|
||||
it("fails on wrong number", () => {
|
||||
const r = task.check("Answer: 7");
|
||||
expect(r.verdict).toBe("fail");
|
||||
expect(r.note).toMatch(/got 7/);
|
||||
});
|
||||
|
||||
it("fails when no number is present", () => {
|
||||
expect(task.check("Answer: lots").verdict).toBe("fail");
|
||||
});
|
||||
});
|
||||
|
||||
describe("harvest-bench: three_hats checker", () => {
|
||||
const task = taskById("three_hats");
|
||||
|
||||
it("passes on 'red'", () => {
|
||||
expect(task.check("Answer: red").verdict).toBe("pass");
|
||||
});
|
||||
|
||||
it("passes with extra wording around the keyword", () => {
|
||||
expect(task.check("Answer: The third person's hat is red.").verdict).toBe("pass");
|
||||
});
|
||||
|
||||
it("fails on 'blue'", () => {
|
||||
expect(task.check("Answer: blue").verdict).toBe("fail");
|
||||
});
|
||||
|
||||
it("fails when answer mentions both colors (ambiguous)", () => {
|
||||
// We require a clean red-only answer to avoid giving credit for hedging
|
||||
expect(task.check("Answer: red or blue, depends").verdict).toBe("fail");
|
||||
});
|
||||
});
|
||||
|
||||
describe("harvest-bench: pseudoprime_base2 checker", () => {
|
||||
const task = taskById("pseudoprime_base2");
|
||||
|
||||
it("passes on 341", () => {
|
||||
expect(task.check("Reasoning... Answer: 341").verdict).toBe("pass");
|
||||
});
|
||||
|
||||
it("fails on 561 (common V3 wrong answer — Carmichael)", () => {
|
||||
const r = task.check("Answer: 561");
|
||||
expect(r.verdict).toBe("fail");
|
||||
expect(r.note).toMatch(/got 561/);
|
||||
});
|
||||
|
||||
it("passes when answer is wrapped in LaTeX", () => {
|
||||
expect(task.check("Answer: $341$ (= 11 \\cdot 31)").verdict).toBe("pass");
|
||||
});
|
||||
});
|
||||
|
||||
describe("harvest-bench: derangements_d7 checker", () => {
|
||||
const task = taskById("derangements_d7");
|
||||
|
||||
it("passes on 1854", () => {
|
||||
expect(task.check("Answer: 1854").verdict).toBe("pass");
|
||||
});
|
||||
|
||||
it("fails on 265 (D_6) — common off-by-one", () => {
|
||||
expect(task.check("Answer: 265").verdict).toBe("fail");
|
||||
});
|
||||
|
||||
it("fails on 1855 (off by one)", () => {
|
||||
expect(task.check("Answer: 1855").verdict).toBe("fail");
|
||||
});
|
||||
});
|
||||
|
||||
describe("harvest-bench: euler_quadratic_break checker", () => {
|
||||
const task = taskById("euler_quadratic_break");
|
||||
|
||||
it("passes on 40", () => {
|
||||
expect(task.check("Answer: 40").verdict).toBe("pass");
|
||||
});
|
||||
|
||||
it("fails on 41 (the off-by-one trap from n²-n+41)", () => {
|
||||
expect(task.check("Answer: 41").verdict).toBe("fail");
|
||||
});
|
||||
|
||||
it("fails when agent gives f(40) instead of n=40", () => {
|
||||
expect(task.check("Answer: 1681").verdict).toBe("fail");
|
||||
});
|
||||
});
|
||||
|
||||
describe("task set invariants", () => {
|
||||
it("exposes at least 6 tasks (3 easy + 3 hard) with unique ids", () => {
|
||||
expect(TASKS.length).toBeGreaterThanOrEqual(6);
|
||||
const ids = new Set(TASKS.map((t) => t.id));
|
||||
expect(ids.size).toBe(TASKS.length);
|
||||
});
|
||||
|
||||
it("every task checker rejects a deliberately wrong answer (sanity)", () => {
|
||||
const bogus = "Answer: this is not a valid answer to anything";
|
||||
for (const t of TASKS) {
|
||||
const r = t.check(bogus);
|
||||
expect(r.verdict).toBe("fail");
|
||||
}
|
||||
});
|
||||
|
||||
it("every task checker is deterministic on identical input", () => {
|
||||
const sample = "Answer: something";
|
||||
for (const t of TASKS) {
|
||||
expect(t.check(sample).verdict).toBe(t.check(sample).verdict);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,121 @@
|
||||
/**
|
||||
* Tests for the loop's error-message decorator. Scope is narrow:
|
||||
* context-overflow errors get a user-friendly hint, everything else
|
||||
* passes through unchanged.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { formatLoopError, healLoadedMessages, stripHallucinatedToolMarkup } from "../src/loop.js";
|
||||
import type { ChatMessage } from "../src/types.js";
|
||||
|
||||
describe("formatLoopError", () => {
|
||||
it("annotates a DeepSeek 400 'maximum context length' error", () => {
|
||||
const raw = new Error(
|
||||
'DeepSeek 400: {"error":{"message":"This model\'s maximum context length is 131072 tokens. ' +
|
||||
"However, you requested 929452 tokens (929452 in the messages, 0 in the completion). " +
|
||||
'Please reduce the length of the messages or completion."}}',
|
||||
);
|
||||
const out = formatLoopError(raw);
|
||||
expect(out).toMatch(/Context overflow/);
|
||||
expect(out).toMatch(/\/forget/);
|
||||
expect(out).toMatch(/929,452 tokens/); // pretty-printed from the raw JSON
|
||||
});
|
||||
|
||||
it("leaves non-overflow errors unchanged", () => {
|
||||
const raw = new Error("DeepSeek 401: invalid api key");
|
||||
expect(formatLoopError(raw)).toBe("DeepSeek 401: invalid api key");
|
||||
});
|
||||
|
||||
it("tolerates an overflow error without a requested-tokens figure", () => {
|
||||
const raw = new Error("DeepSeek 400: This model's maximum context length is 131072 tokens.");
|
||||
const out = formatLoopError(raw);
|
||||
expect(out).toMatch(/Context overflow/);
|
||||
expect(out).toMatch(/too many tokens/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("healLoadedMessages", () => {
|
||||
it("truncates a giant tool result, leaves user/assistant messages alone", () => {
|
||||
const big = "X".repeat(80_000);
|
||||
const messages: ChatMessage[] = [
|
||||
{ role: "user", content: "read the big file" },
|
||||
{ role: "assistant", content: "", tool_calls: [] },
|
||||
{ role: "tool", tool_call_id: "t1", content: big },
|
||||
{ role: "assistant", content: "here's what I found" },
|
||||
];
|
||||
const { messages: healed, healedCount, healedFrom } = healLoadedMessages(messages, 32_000);
|
||||
expect(healedCount).toBe(1);
|
||||
expect(healedFrom).toBe(80_000);
|
||||
expect(healed[0]).toEqual(messages[0]); // user untouched
|
||||
expect(healed[1]).toEqual(messages[1]); // assistant untouched
|
||||
expect(typeof healed[2]!.content).toBe("string");
|
||||
expect((healed[2]!.content as string).length).toBeLessThan(33_000);
|
||||
expect(healed[2]!.content).toContain("truncated");
|
||||
expect(healed[3]).toEqual(messages[3]); // trailing assistant untouched
|
||||
});
|
||||
|
||||
it("is a no-op when every message fits", () => {
|
||||
const messages: ChatMessage[] = [
|
||||
{ role: "user", content: "hi" },
|
||||
{ role: "tool", tool_call_id: "t1", content: "small result" },
|
||||
];
|
||||
const { messages: healed, healedCount } = healLoadedMessages(messages, 32_000);
|
||||
expect(healedCount).toBe(0);
|
||||
expect(healed).toEqual(messages); // structural equality, same strings
|
||||
});
|
||||
|
||||
it("heals multiple oversized tool messages in one pass", () => {
|
||||
const messages: ChatMessage[] = [
|
||||
{ role: "tool", tool_call_id: "t1", content: "A".repeat(40_000) },
|
||||
{ role: "tool", tool_call_id: "t2", content: "B".repeat(50_000) },
|
||||
{ role: "tool", tool_call_id: "t3", content: "small" },
|
||||
];
|
||||
const { healedCount, healedFrom } = healLoadedMessages(messages, 32_000);
|
||||
expect(healedCount).toBe(2);
|
||||
expect(healedFrom).toBe(90_000);
|
||||
});
|
||||
});
|
||||
|
||||
describe("stripHallucinatedToolMarkup", () => {
|
||||
it("removes a full DSML function_calls block (the R1 hallucination we saw live)", () => {
|
||||
const input = [
|
||||
"Let me look at the file structure.",
|
||||
"",
|
||||
'<|DSML|function_calls> <|DSML|invoke name="filesystem_edit_file">',
|
||||
' <|DSML|parameter name="path" string="true">F:.html</|DSML|parameter>',
|
||||
' <|DSML|parameter name="edits" string="false">[...]</|DSML|parameter>',
|
||||
"</|DSML|invoke> </|DSML|function_calls>",
|
||||
"",
|
||||
"Saved.",
|
||||
].join("\n");
|
||||
const out = stripHallucinatedToolMarkup(input);
|
||||
expect(out).toContain("Let me look at the file structure.");
|
||||
expect(out).toContain("Saved.");
|
||||
expect(out).not.toContain("DSML");
|
||||
expect(out).not.toContain("filesystem_edit_file");
|
||||
});
|
||||
|
||||
it("removes an Anthropic-style <function_calls> block", () => {
|
||||
const input = "Here is the plan.\n<function_calls>\n<tool>...</tool>\n</function_calls>\nDone.";
|
||||
const out = stripHallucinatedToolMarkup(input);
|
||||
expect(out).toContain("Here is the plan.");
|
||||
expect(out).toContain("Done.");
|
||||
expect(out).not.toContain("function_calls");
|
||||
});
|
||||
|
||||
it("strips a truncated DSML opener that never gets closed", () => {
|
||||
const input = 'Before the junk.\n<|DSML|function_calls> <|DSML|invoke name="x"> ...';
|
||||
const out = stripHallucinatedToolMarkup(input);
|
||||
expect(out).toBe("Before the junk.");
|
||||
});
|
||||
|
||||
it("leaves plain prose completely alone", () => {
|
||||
const input = "Just a normal summary with no markup anywhere.";
|
||||
expect(stripHallucinatedToolMarkup(input)).toBe(input);
|
||||
});
|
||||
|
||||
it("returns empty string when ALL content was hallucinated markup", () => {
|
||||
const input = "<|DSML|function_calls>garbage</|DSML|function_calls>";
|
||||
expect(stripHallucinatedToolMarkup(input)).toBe("");
|
||||
});
|
||||
});
|
||||
@@ -156,6 +156,48 @@ describe("CacheFirstLoop (non-streaming)", () => {
|
||||
expect(loop.stats.turns.length).toBe(2); // two model round-trips
|
||||
});
|
||||
|
||||
it("yields tool_start before each tool dispatch so the TUI can show 'running…'", async () => {
|
||||
const client = makeClient([
|
||||
{
|
||||
content: "",
|
||||
tool_calls: [
|
||||
{
|
||||
id: "call_1",
|
||||
type: "function",
|
||||
function: { name: "add", arguments: '{"a":1,"b":2}' },
|
||||
},
|
||||
],
|
||||
},
|
||||
{ content: "done" },
|
||||
]);
|
||||
const tools = new ToolRegistry();
|
||||
tools.register<{ a: number; b: number }, number>({
|
||||
name: "add",
|
||||
parameters: {
|
||||
type: "object",
|
||||
properties: { a: { type: "integer" }, b: { type: "integer" } },
|
||||
required: ["a", "b"],
|
||||
},
|
||||
fn: ({ a, b }) => a + b,
|
||||
});
|
||||
const loop = new CacheFirstLoop({
|
||||
client,
|
||||
prefix: new ImmutablePrefix({ system: "s", toolSpecs: tools.specs() }),
|
||||
tools,
|
||||
stream: false,
|
||||
});
|
||||
|
||||
const roleOrder: { role: string; toolName?: string }[] = [];
|
||||
for await (const ev of loop.step("go")) {
|
||||
if (ev.role === "tool_start" || ev.role === "tool") {
|
||||
roleOrder.push({ role: ev.role, toolName: ev.toolName });
|
||||
}
|
||||
}
|
||||
// tool_start must precede the matching tool result.
|
||||
expect(roleOrder[0]).toEqual({ role: "tool_start", toolName: "add" });
|
||||
expect(roleOrder[1]).toEqual({ role: "tool", toolName: "add" });
|
||||
});
|
||||
|
||||
it("immutable prefix is preserved across turns (cache-stability invariant)", async () => {
|
||||
const sharedFetch = fakeFetch([{ content: "a" }, { content: "b" }]);
|
||||
const client = new DeepSeekClient({ apiKey: "sk-test", fetch: sharedFetch });
|
||||
@@ -186,6 +228,205 @@ describe("CacheFirstLoop (non-streaming)", () => {
|
||||
expect(msgs2.length).toBeGreaterThan(msgs1.length);
|
||||
});
|
||||
|
||||
it("yields a warning event once when tool-call count crosses 70% of budget", async () => {
|
||||
const reg = new ToolRegistry();
|
||||
reg.register({
|
||||
name: "probe",
|
||||
description: "no-op",
|
||||
parameters: { type: "object", properties: {} },
|
||||
fn: async () => "ok",
|
||||
});
|
||||
const callAgain = {
|
||||
content: "",
|
||||
tool_calls: [{ id: "c", type: "function", function: { name: "probe", arguments: "{}" } }],
|
||||
};
|
||||
const summary = { content: "all done" };
|
||||
const responses: FakeResponseShape[] = [callAgain, callAgain, callAgain, callAgain, summary];
|
||||
const client = makeClient(responses);
|
||||
const loop = new CacheFirstLoop({
|
||||
client,
|
||||
prefix: new ImmutablePrefix({ system: "s", toolSpecs: reg.specs() }),
|
||||
tools: reg,
|
||||
stream: false,
|
||||
maxToolIters: 4, // 70% → warn starting at iter >= 2
|
||||
});
|
||||
|
||||
const warnings: string[] = [];
|
||||
for await (const ev of loop.step("go")) {
|
||||
if (ev.role === "warning") warnings.push(ev.content);
|
||||
}
|
||||
// Exactly one warning should fire (guard against the once-per-turn flag
|
||||
// regressing). Content must mention the X/N count and "Esc".
|
||||
expect(warnings).toHaveLength(1);
|
||||
expect(warnings[0]).toMatch(/\d+\/4 tool calls used/);
|
||||
expect(warnings[0]).toMatch(/Esc/);
|
||||
});
|
||||
|
||||
it("abort() mid-step stops immediately without a follow-up API call", async () => {
|
||||
const reg = new ToolRegistry();
|
||||
reg.register({
|
||||
name: "probe",
|
||||
description: "no-op",
|
||||
parameters: { type: "object", properties: {} },
|
||||
fn: async () => "ok",
|
||||
});
|
||||
const chainingToolCall = {
|
||||
content: "",
|
||||
tool_calls: [{ id: "c", type: "function", function: { name: "probe", arguments: "{}" } }],
|
||||
};
|
||||
// Only one chaining response needed — abort should stop the loop
|
||||
// before any follow-up model call. A second response in the array
|
||||
// would indicate the loop made an unwanted extra API call.
|
||||
const fetchSpy = vi.fn() as unknown as typeof fetch;
|
||||
const responses: FakeResponseShape[] = [chainingToolCall];
|
||||
const client = new DeepSeekClient({
|
||||
apiKey: "sk-test",
|
||||
fetch: fakeFetch(responses) as unknown as typeof fetch,
|
||||
});
|
||||
const loop = new CacheFirstLoop({
|
||||
client,
|
||||
prefix: new ImmutablePrefix({ system: "s", toolSpecs: reg.specs() }),
|
||||
tools: reg,
|
||||
stream: false,
|
||||
maxToolIters: 16,
|
||||
});
|
||||
|
||||
// Call abort AFTER the first tool event fires — simulates the user
|
||||
// hitting Esc while the loop is exploring.
|
||||
const events: { role: string; content?: string; forcedSummary?: boolean }[] = [];
|
||||
let aborted = false;
|
||||
for await (const ev of loop.step("go")) {
|
||||
events.push({ role: ev.role, content: ev.content, forcedSummary: ev.forcedSummary });
|
||||
if (!aborted && ev.role === "tool") {
|
||||
aborted = true;
|
||||
loop.abort();
|
||||
}
|
||||
}
|
||||
|
||||
// Warning fires with the abort notice.
|
||||
const warnings = events.filter((e) => e.role === "warning");
|
||||
expect(warnings.some((w) => /aborted at iter/.test(w.content ?? ""))).toBe(true);
|
||||
|
||||
// Synthetic assistant_final is tagged forcedSummary and carries
|
||||
// the stopped-message text. It should NOT contain any model
|
||||
// output because no second API call was made.
|
||||
const finals = events.filter((e) => e.role === "assistant_final");
|
||||
const stopped = finals[finals.length - 1]!;
|
||||
expect(stopped.forcedSummary).toBe(true);
|
||||
expect(stopped.content).toMatch(/aborted by user \(Esc\)/);
|
||||
expect(stopped.content).toMatch(/no summary produced/);
|
||||
|
||||
// Suite ends with `done`.
|
||||
expect(events[events.length - 1]!.role).toBe("done");
|
||||
// Silence unused-var warning.
|
||||
void fetchSpy;
|
||||
});
|
||||
|
||||
it("forces a summary when maxToolIters is exhausted, instead of stopping silently", async () => {
|
||||
// Give a registered tool so the repair layer doesn't strip the fake
|
||||
// tool_calls for referring to an unknown name.
|
||||
const reg = new ToolRegistry();
|
||||
reg.register({
|
||||
name: "probe",
|
||||
description: "no-op",
|
||||
parameters: { type: "object", properties: {} },
|
||||
fn: async () => "ok",
|
||||
});
|
||||
// Every tool-iter response says "call probe again" — infinite loop
|
||||
// absent the iter cap. The (N+1)th response is the forced-summary
|
||||
// call (no tools, returns text).
|
||||
const chainingToolCall = {
|
||||
content: "",
|
||||
tool_calls: [
|
||||
{
|
||||
id: "call_1",
|
||||
type: "function",
|
||||
function: { name: "probe", arguments: "{}" },
|
||||
},
|
||||
],
|
||||
};
|
||||
const responses: FakeResponseShape[] = [
|
||||
chainingToolCall,
|
||||
chainingToolCall,
|
||||
{ content: "done — here's what I found." }, // summary call
|
||||
];
|
||||
const client = makeClient(responses);
|
||||
const loop = new CacheFirstLoop({
|
||||
client,
|
||||
prefix: new ImmutablePrefix({ system: "s", toolSpecs: reg.specs() }),
|
||||
tools: reg,
|
||||
stream: false,
|
||||
maxToolIters: 2, // deliberately tight so we hit the cap fast
|
||||
});
|
||||
|
||||
const events: { role: string; content?: string }[] = [];
|
||||
for await (const ev of loop.step("go")) {
|
||||
events.push({ role: ev.role, content: ev.content });
|
||||
}
|
||||
|
||||
// Multiple assistant_final events are yielded (one per iter) — the
|
||||
// summary is the LAST one, carrying the "tool-call budget" prefix.
|
||||
const finals = events.filter((e) => e.role === "assistant_final");
|
||||
const summary = finals[finals.length - 1];
|
||||
expect(summary).toBeDefined();
|
||||
expect(summary!.content).toMatch(/tool-call budget/);
|
||||
expect(summary!.content).toContain("done — here's what I found.");
|
||||
// Last event is still `done`, preserving the contract used by run().
|
||||
expect(events[events.length - 1]!.role).toBe("done");
|
||||
});
|
||||
|
||||
it("context-guard diverts to summary when promptTokens > 80% of the window, tagging forcedSummary", async () => {
|
||||
const reg = new ToolRegistry();
|
||||
reg.register({
|
||||
name: "probe",
|
||||
description: "no-op",
|
||||
parameters: { type: "object", properties: {} },
|
||||
fn: async () => "ok",
|
||||
});
|
||||
// First response: chaining tool call with a prompt-token count
|
||||
// deliberately over 80% of DeepSeek's 131k window (131k * 0.8 = 104_857).
|
||||
// 120k trips the guard.
|
||||
const responses: FakeResponseShape[] = [
|
||||
{
|
||||
content: "",
|
||||
tool_calls: [{ id: "c", type: "function", function: { name: "probe", arguments: "{}" } }],
|
||||
usage: {
|
||||
prompt_tokens: 120_000,
|
||||
completion_tokens: 50,
|
||||
total_tokens: 120_050,
|
||||
prompt_cache_hit_tokens: 90_000,
|
||||
prompt_cache_miss_tokens: 30_000,
|
||||
},
|
||||
},
|
||||
// Forced-summary response (no tools)
|
||||
{ content: "based on what I saw, X." },
|
||||
];
|
||||
const client = makeClient(responses);
|
||||
const loop = new CacheFirstLoop({
|
||||
client,
|
||||
prefix: new ImmutablePrefix({ system: "s", toolSpecs: reg.specs() }),
|
||||
tools: reg,
|
||||
stream: false,
|
||||
maxToolIters: 64,
|
||||
});
|
||||
|
||||
const events: { role: string; forcedSummary?: boolean; content?: string }[] = [];
|
||||
for await (const ev of loop.step("analyze the repo")) {
|
||||
events.push({ role: ev.role, forcedSummary: ev.forcedSummary, content: ev.content });
|
||||
}
|
||||
|
||||
// A warning must fire about the context guard.
|
||||
const warn = events.find((e) => e.role === "warning");
|
||||
expect(warn).toBeDefined();
|
||||
expect(warn!.content).toMatch(/context \d+\/\d+/);
|
||||
|
||||
// The final assistant_final must be tagged forcedSummary and carry the context-guard prefix.
|
||||
const finals = events.filter((e) => e.role === "assistant_final");
|
||||
const summary = finals[finals.length - 1];
|
||||
expect(summary!.forcedSummary).toBe(true);
|
||||
expect(summary!.content).toMatch(/context budget running low/);
|
||||
});
|
||||
|
||||
it("surfaces an error event when the HTTP call fails with a non-retryable status", async () => {
|
||||
// 401 is non-retryable (bad key). Using this avoids multi-retry waits.
|
||||
const errFetch = vi.fn(async () => new Response("boom", { status: 401 }));
|
||||
|
||||
+93
-1
@@ -1,5 +1,5 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { stripMath } from "../src/cli/ui/markdown.js";
|
||||
import { parseBlocks, stripMath } from "../src/cli/ui/markdown.js";
|
||||
|
||||
describe("stripMath", () => {
|
||||
it("converts \\frac, \\dfrac, \\tfrac uniformly", () => {
|
||||
@@ -67,3 +67,95 @@ describe("stripMath", () => {
|
||||
expect(out).toContain("(d)/(30)");
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseBlocks — SEARCH/REPLACE detection", () => {
|
||||
it("extracts a single SEARCH/REPLACE block into a first-class edit-block", () => {
|
||||
const text = [
|
||||
"Here is the fix:",
|
||||
"",
|
||||
"src/foo.ts",
|
||||
"<<<<<<< SEARCH",
|
||||
"const x = 1;",
|
||||
"=======",
|
||||
"const x = 2;",
|
||||
">>>>>>> REPLACE",
|
||||
].join("\n");
|
||||
const blocks = parseBlocks(text);
|
||||
const edit = blocks.find((b) => b.kind === "edit-block");
|
||||
expect(edit).toBeDefined();
|
||||
if (edit?.kind !== "edit-block") throw new Error("unreachable");
|
||||
expect(edit.filename).toBe("src/foo.ts");
|
||||
expect(edit.search).toBe("const x = 1;");
|
||||
expect(edit.replace).toBe("const x = 2;");
|
||||
});
|
||||
|
||||
it("preserves multi-line SEARCH and REPLACE verbatim (no markdown mangling)", () => {
|
||||
// The original user-reported shape: JSDoc comments inside SEARCH.
|
||||
// Before this fix, `/** ... */` got eaten by bold/italic regex and
|
||||
// `para.join(" ")` collapsed newlines.
|
||||
const text = [
|
||||
"src/code/edit-blocks.ts",
|
||||
"<<<<<<< SEARCH",
|
||||
"/** Edit landed on disk. */",
|
||||
"| 'applied'",
|
||||
"=======",
|
||||
"/** Edit landed on disk. */",
|
||||
"| 'applied-new'",
|
||||
">>>>>>> REPLACE",
|
||||
].join("\n");
|
||||
const [edit] = parseBlocks(text).filter((b) => b.kind === "edit-block");
|
||||
if (edit?.kind !== "edit-block") throw new Error("expected edit-block");
|
||||
// The `/** ... */` and `|` chars survive intact — no `*`-eating,
|
||||
// no newline-flattening.
|
||||
expect(edit.search).toContain("/** Edit landed on disk. */");
|
||||
expect(edit.search).toContain("\n");
|
||||
expect(edit.replace).toContain("'applied-new'");
|
||||
});
|
||||
|
||||
it("recognizes new-file (empty SEARCH) blocks", () => {
|
||||
const text = [
|
||||
"src/new.ts",
|
||||
"<<<<<<< SEARCH",
|
||||
"=======",
|
||||
"export const x = 1;",
|
||||
">>>>>>> REPLACE",
|
||||
].join("\n");
|
||||
const [edit] = parseBlocks(text).filter((b) => b.kind === "edit-block");
|
||||
if (edit?.kind !== "edit-block") throw new Error("expected edit-block");
|
||||
expect(edit.search).toBe("");
|
||||
expect(edit.replace).toBe("export const x = 1;");
|
||||
});
|
||||
|
||||
it("ignores a stray <<<<<<< SEARCH without a filename or close marker", () => {
|
||||
const text = "just some prose with <<<<<<< SEARCH left over in the middle";
|
||||
const blocks = parseBlocks(text);
|
||||
expect(blocks.find((b) => b.kind === "edit-block")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("extracts multiple edit-blocks in one response, keeping the prose between them", () => {
|
||||
const text = [
|
||||
"First change:",
|
||||
"src/a.ts",
|
||||
"<<<<<<< SEARCH",
|
||||
"old_a",
|
||||
"=======",
|
||||
"new_a",
|
||||
">>>>>>> REPLACE",
|
||||
"",
|
||||
"And second:",
|
||||
"src/b.ts",
|
||||
"<<<<<<< SEARCH",
|
||||
"old_b",
|
||||
"=======",
|
||||
"new_b",
|
||||
">>>>>>> REPLACE",
|
||||
].join("\n");
|
||||
const blocks = parseBlocks(text);
|
||||
const edits = blocks.filter((b) => b.kind === "edit-block");
|
||||
expect(edits).toHaveLength(2);
|
||||
const paragraphs = blocks.filter((b) => b.kind === "paragraph");
|
||||
expect(paragraphs.map((p) => (p.kind === "paragraph" ? p.text : ""))).toEqual(
|
||||
expect.arrayContaining(["First change:", "And second:"]),
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,229 @@
|
||||
/**
|
||||
* Tests for inspectMcpServer (pure, runs against the fake transport).
|
||||
* We don't unit-test the CLI wrapper — that's just argv parsing and
|
||||
* stdout printing, covered by the integration story of "types compile".
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { McpClient } from "../src/mcp/client.js";
|
||||
import { inspectMcpServer } from "../src/mcp/inspect.js";
|
||||
import type { McpTransport } from "../src/mcp/stdio.js";
|
||||
import {
|
||||
type JsonRpcMessage,
|
||||
type JsonRpcRequest,
|
||||
MCP_PROTOCOL_VERSION,
|
||||
} from "../src/mcp/types.js";
|
||||
|
||||
// A minimal in-process transport that answers methods from a handler
|
||||
// map. Simpler than the FakeMcpTransport in mcp.test.ts — we only
|
||||
// care about shape-of-response here, not call ordering.
|
||||
class HandlerTransport implements McpTransport {
|
||||
private readonly queue: JsonRpcMessage[] = [];
|
||||
private readonly waiters: Array<(m: JsonRpcMessage | null) => void> = [];
|
||||
private closed = false;
|
||||
constructor(private readonly handlers: Record<string, (req: JsonRpcRequest) => JsonRpcMessage>) {}
|
||||
|
||||
async send(msg: JsonRpcMessage): Promise<void> {
|
||||
if (this.closed) throw new Error("closed");
|
||||
if (!("id" in msg) || !("method" in msg)) return; // notification from client
|
||||
const req = msg as JsonRpcRequest;
|
||||
const handler = this.handlers[req.method];
|
||||
const response = handler
|
||||
? handler(req)
|
||||
: {
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
error: { code: -32601, message: `method not found: ${req.method}` },
|
||||
};
|
||||
const w = this.waiters.shift();
|
||||
if (w) w(response as JsonRpcMessage);
|
||||
else this.queue.push(response as JsonRpcMessage);
|
||||
}
|
||||
|
||||
async *messages(): AsyncIterableIterator<JsonRpcMessage> {
|
||||
while (true) {
|
||||
if (this.queue.length > 0) {
|
||||
yield this.queue.shift()!;
|
||||
continue;
|
||||
}
|
||||
if (this.closed) return;
|
||||
const next = await new Promise<JsonRpcMessage | null>((resolve) => {
|
||||
this.waiters.push(resolve);
|
||||
});
|
||||
if (next === null) return;
|
||||
yield next;
|
||||
}
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
this.closed = true;
|
||||
while (this.waiters.length > 0) this.waiters.shift()!(null);
|
||||
}
|
||||
}
|
||||
|
||||
function initOk(req: JsonRpcRequest): JsonRpcMessage {
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
result: {
|
||||
protocolVersion: MCP_PROTOCOL_VERSION,
|
||||
serverInfo: { name: "test-server", version: "1.2.3" },
|
||||
capabilities: { tools: {}, resources: {}, prompts: {} },
|
||||
instructions: "Use tool X before Y.",
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe("inspectMcpServer", () => {
|
||||
it("reports server info, capabilities, tools, resources, prompts — full-support server", async () => {
|
||||
const transport = new HandlerTransport({
|
||||
initialize: initOk,
|
||||
"tools/list": (req) => ({
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
result: {
|
||||
tools: [
|
||||
{ name: "echo", description: "echoes", inputSchema: { type: "object" } },
|
||||
{ name: "add", description: "a+b", inputSchema: { type: "object" } },
|
||||
],
|
||||
},
|
||||
}),
|
||||
"resources/list": (req) => ({
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
result: {
|
||||
resources: [
|
||||
{ uri: "file:///a.md", name: "a", mimeType: "text/markdown" },
|
||||
{ uri: "custom://b", name: "b" },
|
||||
],
|
||||
},
|
||||
}),
|
||||
"prompts/list": (req) => ({
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
result: {
|
||||
prompts: [
|
||||
{
|
||||
name: "summarize",
|
||||
description: "summarize a doc",
|
||||
arguments: [{ name: "lang", required: true }],
|
||||
},
|
||||
],
|
||||
},
|
||||
}),
|
||||
});
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
const report = await inspectMcpServer(client);
|
||||
|
||||
expect(report.protocolVersion).toBe(MCP_PROTOCOL_VERSION);
|
||||
expect(report.serverInfo).toEqual({ name: "test-server", version: "1.2.3" });
|
||||
expect(report.instructions).toBe("Use tool X before Y.");
|
||||
expect(report.tools).toEqual({
|
||||
supported: true,
|
||||
items: expect.arrayContaining([expect.objectContaining({ name: "echo" })]),
|
||||
});
|
||||
expect(report.resources.supported).toBe(true);
|
||||
if (report.resources.supported) {
|
||||
expect(report.resources.items.map((r) => r.name)).toEqual(["a", "b"]);
|
||||
}
|
||||
expect(report.prompts.supported).toBe(true);
|
||||
if (report.prompts.supported) {
|
||||
expect(report.prompts.items[0]!.arguments?.[0]?.name).toBe("lang");
|
||||
}
|
||||
|
||||
await client.close();
|
||||
});
|
||||
|
||||
it("marks resources+prompts as not supported when server returns -32601", async () => {
|
||||
// Tools-only server: init returns, tools/list works, resources/list
|
||||
// + prompts/list fall through to the default -32601 in HandlerTransport.
|
||||
const transport = new HandlerTransport({
|
||||
initialize: initOk,
|
||||
"tools/list": (req) => ({
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
result: { tools: [] },
|
||||
}),
|
||||
});
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
const report = await inspectMcpServer(client);
|
||||
|
||||
expect(report.tools.supported).toBe(true);
|
||||
expect(report.resources.supported).toBe(false);
|
||||
expect(report.prompts.supported).toBe(false);
|
||||
if (!report.resources.supported) {
|
||||
expect(report.resources.reason).toMatch(/-32601/);
|
||||
}
|
||||
if (!report.prompts.supported) {
|
||||
expect(report.prompts.reason).toMatch(/-32601/);
|
||||
}
|
||||
|
||||
await client.close();
|
||||
});
|
||||
|
||||
it("forwards non-32601 errors as the section reason so user sees real diagnostics", async () => {
|
||||
const transport = new HandlerTransport({
|
||||
initialize: initOk,
|
||||
"tools/list": (req) => ({
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
error: { code: -32000, message: "server is overloaded" },
|
||||
}),
|
||||
"resources/list": (req) => ({
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
result: { resources: [] },
|
||||
}),
|
||||
"prompts/list": (req) => ({
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
result: { prompts: [] },
|
||||
}),
|
||||
});
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
const report = await inspectMcpServer(client);
|
||||
|
||||
expect(report.tools.supported).toBe(false);
|
||||
if (!report.tools.supported) {
|
||||
expect(report.tools.reason).toMatch(/overloaded/);
|
||||
}
|
||||
// Resources and prompts are supported and empty — should not be affected.
|
||||
expect(report.resources.supported).toBe(true);
|
||||
expect(report.prompts.supported).toBe(true);
|
||||
|
||||
await client.close();
|
||||
});
|
||||
});
|
||||
|
||||
describe("McpClient: initialize records serverInfo + protocolVersion + instructions", () => {
|
||||
it("exposes server info, protocol version, instructions from the handshake", async () => {
|
||||
const transport = new HandlerTransport({ initialize: initOk });
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
expect(client.serverInfo).toEqual({ name: "test-server", version: "1.2.3" });
|
||||
expect(client.protocolVersion).toBe(MCP_PROTOCOL_VERSION);
|
||||
expect(client.serverInstructions).toBe("Use tool X before Y.");
|
||||
await client.close();
|
||||
});
|
||||
|
||||
it("leaves serverInstructions undefined when absent from the init result", async () => {
|
||||
const transport = new HandlerTransport({
|
||||
initialize: (req) => ({
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
result: {
|
||||
protocolVersion: MCP_PROTOCOL_VERSION,
|
||||
serverInfo: { name: "minimal", version: "0" },
|
||||
capabilities: { tools: {} },
|
||||
},
|
||||
}),
|
||||
});
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
expect(client.serverInstructions).toBeUndefined();
|
||||
await client.close();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,107 @@
|
||||
/**
|
||||
* Integration test: real subprocess + real transport + bridge + dispatch.
|
||||
*
|
||||
* Distinct from tests/mcp.test.ts (which uses an in-process fake
|
||||
* transport). This one actually spawns the bundled demo MCP server as
|
||||
* a child process, connects via StdioTransport, bridges tools into a
|
||||
* ToolRegistry, and invokes them.
|
||||
*
|
||||
* We don't put this in CI yet — subprocess tests on Windows can be
|
||||
* slow or flaky. Run locally with `npm test`. If it ever becomes
|
||||
* flaky, swap the afterEach cleanup to force-kill and move on.
|
||||
*/
|
||||
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { McpClient } from "../src/mcp/client.js";
|
||||
import { bridgeMcpTools } from "../src/mcp/registry.js";
|
||||
import { StdioTransport } from "../src/mcp/stdio.js";
|
||||
import { ToolRegistry } from "../src/tools.js";
|
||||
|
||||
// Spawning `tsx` directly needs a cross-platform approach. `node --import tsx`
|
||||
// works everywhere Node 20+ is installed (which is our engines target) and
|
||||
// avoids the Windows `.cmd` resolution gotcha in child_process.spawn.
|
||||
const NODE_CMD = process.execPath;
|
||||
const DEMO_SERVER_ARGS = ["--import", "tsx", "examples/mcp-server-demo.ts"];
|
||||
|
||||
describe("MCP integration — real subprocess against bundled demo server", () => {
|
||||
let client: McpClient | null = null;
|
||||
|
||||
afterEach(async () => {
|
||||
if (client) {
|
||||
await client.close();
|
||||
client = null;
|
||||
}
|
||||
});
|
||||
|
||||
it("initializes, lists tools, and calls echo/add end-to-end", async () => {
|
||||
const transport = new StdioTransport({
|
||||
command: NODE_CMD,
|
||||
args: DEMO_SERVER_ARGS,
|
||||
// We're spawning node.exe directly — bypass the shell-true default
|
||||
// that exists for .cmd wrappers (npx etc.). Saves a cmd.exe hop
|
||||
// and the quoting concerns that come with it.
|
||||
shell: false,
|
||||
});
|
||||
client = new McpClient({ transport, requestTimeoutMs: 15_000 });
|
||||
const info = await client.initialize();
|
||||
expect(info.serverInfo.name).toBe("reasonix-demo-mcp");
|
||||
expect(info.capabilities.tools).toBeDefined();
|
||||
|
||||
const { tools } = await client.listTools();
|
||||
const names = tools.map((t) => t.name).sort();
|
||||
expect(names).toEqual(["add", "echo", "get_time"]);
|
||||
|
||||
const echoResult = await client.callTool("echo", { msg: "hello" });
|
||||
const echoText = echoResult.content.map((c) => ("text" in c ? c.text : "")).join("");
|
||||
expect(echoText).toContain("hello");
|
||||
|
||||
const addResult = await client.callTool("add", { a: 17, b: 25 });
|
||||
const addText = addResult.content.map((c) => ("text" in c ? c.text : "")).join("");
|
||||
expect(addText).toContain("42");
|
||||
}, 30_000);
|
||||
|
||||
it("bridges real MCP tools into a Reasonix ToolRegistry", async () => {
|
||||
const transport = new StdioTransport({
|
||||
command: NODE_CMD,
|
||||
args: DEMO_SERVER_ARGS,
|
||||
shell: false,
|
||||
});
|
||||
client = new McpClient({ transport, requestTimeoutMs: 15_000 });
|
||||
await client.initialize();
|
||||
|
||||
const { registry, registeredNames } = await bridgeMcpTools(client, { namePrefix: "demo_" });
|
||||
expect(registeredNames.sort()).toEqual(["demo_add", "demo_echo", "demo_get_time"]);
|
||||
|
||||
// Dispatch through the registry — should round-trip through MCP
|
||||
const out = await registry.dispatch("demo_add", JSON.stringify({ a: 100, b: 1 }));
|
||||
expect(out).toContain("101");
|
||||
}, 30_000);
|
||||
|
||||
it("bridges two MCP servers into a shared registry with different prefixes", async () => {
|
||||
// Two instances of the same demo server, namespaced `a_` and `b_`.
|
||||
// Proves the multi-server CLI wiring: both dispatches go through
|
||||
// their respective subprocesses without cross-talk.
|
||||
const tA = new StdioTransport({ command: NODE_CMD, args: DEMO_SERVER_ARGS, shell: false });
|
||||
const a = new McpClient({ transport: tA, requestTimeoutMs: 15_000 });
|
||||
const tB = new StdioTransport({ command: NODE_CMD, args: DEMO_SERVER_ARGS, shell: false });
|
||||
const b = new McpClient({ transport: tB, requestTimeoutMs: 15_000 });
|
||||
try {
|
||||
await a.initialize();
|
||||
await b.initialize();
|
||||
const shared = new ToolRegistry();
|
||||
const resA = await bridgeMcpTools(a, { registry: shared, namePrefix: "a_" });
|
||||
const resB = await bridgeMcpTools(b, { registry: shared, namePrefix: "b_" });
|
||||
expect(resA.registeredNames).toHaveLength(3);
|
||||
expect(resB.registeredNames).toHaveLength(3);
|
||||
expect(shared.size).toBe(6);
|
||||
|
||||
const outA = await shared.dispatch("a_add", JSON.stringify({ a: 10, b: 20 }));
|
||||
expect(outA).toContain("30");
|
||||
const outB = await shared.dispatch("b_add", JSON.stringify({ a: 1, b: 2 }));
|
||||
expect(outB).toContain("3");
|
||||
} finally {
|
||||
await a.close();
|
||||
await b.close();
|
||||
}
|
||||
}, 30_000);
|
||||
});
|
||||
@@ -0,0 +1,47 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { shellSplit } from "../src/mcp/shell-split.js";
|
||||
|
||||
describe("shellSplit", () => {
|
||||
it("splits a simple space-separated command", () => {
|
||||
expect(shellSplit("npx -y @scope/pkg /tmp")).toEqual(["npx", "-y", "@scope/pkg", "/tmp"]);
|
||||
});
|
||||
|
||||
it("collapses runs of spaces and tabs", () => {
|
||||
expect(shellSplit("a b\tc")).toEqual(["a", "b", "c"]);
|
||||
});
|
||||
|
||||
it("respects double quotes around paths with spaces", () => {
|
||||
expect(shellSplit('npx pkg "my dir/with spaces"')).toEqual([
|
||||
"npx",
|
||||
"pkg",
|
||||
"my dir/with spaces",
|
||||
]);
|
||||
});
|
||||
|
||||
it("respects single quotes and keeps inner backslashes literal", () => {
|
||||
expect(shellSplit("cmd 'a\\b c'")).toEqual(["cmd", "a\\b c"]);
|
||||
});
|
||||
|
||||
it("processes backslash escapes inside double quotes", () => {
|
||||
expect(shellSplit('cmd "a\\"b"')).toEqual(["cmd", 'a"b']);
|
||||
});
|
||||
|
||||
it("passes backslashes through literally OUTSIDE quotes (so Windows paths don't mangle)", () => {
|
||||
// Critical for `reasonix chat --mcp "... C:\\path\\to\\dir"`. Users
|
||||
// who want to escape a space outside quotes can quote the arg.
|
||||
expect(shellSplit("cmd C:\\path\\to\\file.exe")).toEqual(["cmd", "C:\\path\\to\\file.exe"]);
|
||||
});
|
||||
|
||||
it("throws on unterminated double quote", () => {
|
||||
expect(() => shellSplit('cmd "unterminated')).toThrow(/unterminated/);
|
||||
});
|
||||
|
||||
it("throws on unterminated single quote", () => {
|
||||
expect(() => shellSplit("cmd 'still open")).toThrow(/unterminated/);
|
||||
});
|
||||
|
||||
it("returns empty array on whitespace-only input", () => {
|
||||
expect(shellSplit(" ")).toEqual([]);
|
||||
expect(shellSplit("")).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,97 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { parseMcpSpec } from "../src/mcp/spec.js";
|
||||
|
||||
describe("parseMcpSpec: stdio", () => {
|
||||
it("parses a namespaced spec", () => {
|
||||
const spec = parseMcpSpec("fs=npx -y @scope/fs /tmp/dir");
|
||||
expect(spec.transport).toBe("stdio");
|
||||
if (spec.transport !== "stdio") throw new Error("unreachable");
|
||||
expect(spec.name).toBe("fs");
|
||||
expect(spec.command).toBe("npx");
|
||||
expect(spec.args).toEqual(["-y", "@scope/fs", "/tmp/dir"]);
|
||||
});
|
||||
|
||||
it("parses an anonymous spec (no name=)", () => {
|
||||
const spec = parseMcpSpec("npx -y @scope/fs /tmp/dir");
|
||||
expect(spec.transport).toBe("stdio");
|
||||
if (spec.transport !== "stdio") throw new Error("unreachable");
|
||||
expect(spec.name).toBeNull();
|
||||
expect(spec.command).toBe("npx");
|
||||
expect(spec.args).toEqual(["-y", "@scope/fs", "/tmp/dir"]);
|
||||
});
|
||||
|
||||
it("does NOT treat Windows drive letters as a namespace", () => {
|
||||
// `C:\...` matches the colon but NOT the identifier regex [a-zA-Z_]\w* =
|
||||
// So it stays anonymous with the whole path as command.
|
||||
const spec = parseMcpSpec("C:\\path\\to\\server.exe arg1");
|
||||
expect(spec.transport).toBe("stdio");
|
||||
if (spec.transport !== "stdio") throw new Error("unreachable");
|
||||
expect(spec.name).toBeNull();
|
||||
expect(spec.command).toBe("C:\\path\\to\\server.exe");
|
||||
expect(spec.args).toEqual(["arg1"]);
|
||||
});
|
||||
|
||||
it("handles quoted args in the body", () => {
|
||||
const spec = parseMcpSpec('myserver=cmd "path with spaces" --flag');
|
||||
if (spec.transport !== "stdio") throw new Error("unreachable");
|
||||
expect(spec.name).toBe("myserver");
|
||||
expect(spec.command).toBe("cmd");
|
||||
expect(spec.args).toEqual(["path with spaces", "--flag"]);
|
||||
});
|
||||
|
||||
it("trims leading/trailing whitespace", () => {
|
||||
const spec = parseMcpSpec(" fs=npx pkg ");
|
||||
if (spec.transport !== "stdio") throw new Error("unreachable");
|
||||
expect(spec.name).toBe("fs");
|
||||
expect(spec.command).toBe("npx");
|
||||
expect(spec.args).toEqual(["pkg"]);
|
||||
});
|
||||
|
||||
it("throws on empty input", () => {
|
||||
expect(() => parseMcpSpec("")).toThrow(/empty MCP spec/);
|
||||
expect(() => parseMcpSpec(" ")).toThrow(/empty MCP spec/);
|
||||
});
|
||||
|
||||
it("throws when name is given but no command follows", () => {
|
||||
expect(() => parseMcpSpec("fs=")).toThrow(/has name but no command/);
|
||||
expect(() => parseMcpSpec("fs= ")).toThrow(/has name but no command/);
|
||||
});
|
||||
|
||||
it("allows underscores and digits in the name (but not leading digit)", () => {
|
||||
const ok = parseMcpSpec("my_fs2=cmd");
|
||||
expect(ok.name).toBe("my_fs2");
|
||||
// Leading digit → not a valid identifier → whole thing is command
|
||||
// (since `2fs` doesn't match identifier regex).
|
||||
const s = parseMcpSpec("2fs=cmd");
|
||||
if (s.transport !== "stdio") throw new Error("unreachable");
|
||||
expect(s.name).toBeNull();
|
||||
expect(s.command).toBe("2fs=cmd");
|
||||
});
|
||||
});
|
||||
|
||||
describe("parseMcpSpec: sse", () => {
|
||||
it("parses an anonymous https URL", () => {
|
||||
const spec = parseMcpSpec("https://example.com/sse");
|
||||
expect(spec.transport).toBe("sse");
|
||||
if (spec.transport !== "sse") throw new Error("unreachable");
|
||||
expect(spec.name).toBeNull();
|
||||
expect(spec.url).toBe("https://example.com/sse");
|
||||
});
|
||||
|
||||
it("parses a namespaced http URL (localhost dev)", () => {
|
||||
const spec = parseMcpSpec("local=http://127.0.0.1:9000/sse");
|
||||
if (spec.transport !== "sse") throw new Error("unreachable");
|
||||
expect(spec.name).toBe("local");
|
||||
expect(spec.url).toBe("http://127.0.0.1:9000/sse");
|
||||
});
|
||||
|
||||
it("is case-insensitive on the scheme", () => {
|
||||
const spec = parseMcpSpec("HTTPS://example.com/sse");
|
||||
expect(spec.transport).toBe("sse");
|
||||
});
|
||||
|
||||
it("does not match ws:// as SSE (falls through to stdio)", () => {
|
||||
const spec = parseMcpSpec("ws://example.com/mcp");
|
||||
expect(spec.transport).toBe("stdio");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,227 @@
|
||||
/**
|
||||
* SSE transport tests.
|
||||
*
|
||||
* Strategy: spin up a tiny in-process http.Server that implements the
|
||||
* 2024-11-05 MCP HTTP+SSE wire shape — GET emits `event: endpoint` then
|
||||
* stays open; POSTs land a JSON-RPC response on the SSE channel. No real
|
||||
* MCP server needed.
|
||||
*/
|
||||
|
||||
import { type IncomingMessage, type ServerResponse, createServer } from "node:http";
|
||||
import type { AddressInfo } from "node:net";
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import { McpClient } from "../src/mcp/client.js";
|
||||
import { SseTransport } from "../src/mcp/sse.js";
|
||||
import { MCP_PROTOCOL_VERSION } from "../src/mcp/types.js";
|
||||
|
||||
interface FakeSseServer {
|
||||
url: string;
|
||||
requests: Array<{ method: string; url: string; body?: string }>;
|
||||
stop: () => Promise<void>;
|
||||
}
|
||||
|
||||
interface FakeSseOptions {
|
||||
/** Endpoint URL announced in the first SSE event. Relative or absolute. */
|
||||
endpointPath?: string;
|
||||
/** Override the SSE GET path (default `/sse`). */
|
||||
ssePath?: string;
|
||||
/** Override the POST path (default `/messages`). */
|
||||
postPath?: string;
|
||||
/** Auto-answer incoming JSON-RPC requests on the SSE channel. */
|
||||
autoRespond?: (body: unknown) => unknown;
|
||||
/** Return this status for the initial SSE GET instead of 200. */
|
||||
handshakeStatus?: number;
|
||||
}
|
||||
|
||||
function startFakeSseServer(opts: FakeSseOptions = {}): Promise<FakeSseServer> {
|
||||
const ssePath = opts.ssePath ?? "/sse";
|
||||
const postPath = opts.postPath ?? "/messages";
|
||||
const endpointPath = opts.endpointPath ?? postPath;
|
||||
let sseRes: ServerResponse | null = null;
|
||||
const requests: FakeSseServer["requests"] = [];
|
||||
|
||||
const writeFrame = (res: ServerResponse, event: string, data: string) => {
|
||||
res.write(`event: ${event}\ndata: ${data}\n\n`);
|
||||
};
|
||||
|
||||
const server = createServer((req: IncomingMessage, res: ServerResponse) => {
|
||||
if (req.method === "GET" && req.url === ssePath) {
|
||||
requests.push({ method: "GET", url: req.url ?? "" });
|
||||
if (opts.handshakeStatus && opts.handshakeStatus !== 200) {
|
||||
res.statusCode = opts.handshakeStatus;
|
||||
res.end("not ok");
|
||||
return;
|
||||
}
|
||||
res.writeHead(200, {
|
||||
"content-type": "text/event-stream",
|
||||
"cache-control": "no-cache",
|
||||
connection: "keep-alive",
|
||||
});
|
||||
sseRes = res;
|
||||
writeFrame(res, "endpoint", endpointPath);
|
||||
req.on("close", () => {
|
||||
if (sseRes === res) sseRes = null;
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (req.method === "POST" && req.url === postPath) {
|
||||
let body = "";
|
||||
req.setEncoding("utf8");
|
||||
req.on("data", (chunk) => {
|
||||
body += chunk;
|
||||
});
|
||||
req.on("end", () => {
|
||||
requests.push({ method: "POST", url: req.url ?? "", body });
|
||||
res.writeHead(202);
|
||||
res.end();
|
||||
if (opts.autoRespond && sseRes) {
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(body);
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
const reply = opts.autoRespond(parsed);
|
||||
if (reply !== undefined) {
|
||||
writeFrame(sseRes, "message", JSON.stringify(reply));
|
||||
}
|
||||
}
|
||||
});
|
||||
return;
|
||||
}
|
||||
res.writeHead(404);
|
||||
res.end("not found");
|
||||
});
|
||||
|
||||
return new Promise((resolve) => {
|
||||
server.listen(0, "127.0.0.1", () => {
|
||||
const { port } = server.address() as AddressInfo;
|
||||
resolve({
|
||||
url: `http://127.0.0.1:${port}${ssePath}`,
|
||||
requests,
|
||||
stop: () =>
|
||||
new Promise<void>((r) => {
|
||||
sseRes?.end();
|
||||
server.close(() => r());
|
||||
server.closeAllConnections?.();
|
||||
}),
|
||||
});
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
describe("SseTransport: handshake + round-trip", () => {
|
||||
let fake: FakeSseServer | null = null;
|
||||
|
||||
beforeEach(() => {
|
||||
fake = null;
|
||||
});
|
||||
afterEach(async () => {
|
||||
await fake?.stop();
|
||||
});
|
||||
|
||||
it("resolves a relative endpoint URL against the SSE base", async () => {
|
||||
fake = await startFakeSseServer({
|
||||
endpointPath: "/messages",
|
||||
autoRespond: (req) => ({
|
||||
jsonrpc: "2.0",
|
||||
id: (req as { id: number }).id,
|
||||
result: { ok: true },
|
||||
}),
|
||||
});
|
||||
const transport = new SseTransport({ url: fake.url });
|
||||
await transport.send({ jsonrpc: "2.0", id: 1, method: "ping" });
|
||||
|
||||
// Read one incoming message.
|
||||
const iter = transport.messages();
|
||||
const first = await iter.next();
|
||||
expect(first.done).toBe(false);
|
||||
expect(first.value).toMatchObject({ id: 1, result: { ok: true } });
|
||||
await transport.close();
|
||||
});
|
||||
|
||||
it("drives a full McpClient initialize → tools/list round-trip", async () => {
|
||||
fake = await startFakeSseServer({
|
||||
autoRespond: (raw) => {
|
||||
const req = raw as { id?: number; method: string };
|
||||
if (req.id === undefined) return undefined; // notification (initialized)
|
||||
if (req.method === "initialize") {
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
result: {
|
||||
protocolVersion: MCP_PROTOCOL_VERSION,
|
||||
serverInfo: { name: "fake-sse", version: "0.0.0" },
|
||||
capabilities: { tools: { listChanged: false } },
|
||||
},
|
||||
};
|
||||
}
|
||||
if (req.method === "tools/list") {
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
result: {
|
||||
tools: [
|
||||
{
|
||||
name: "ping",
|
||||
description: "responds pong",
|
||||
inputSchema: { type: "object", properties: {} },
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
}
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
error: { code: -32601, message: `no such method ${req.method}` },
|
||||
};
|
||||
},
|
||||
});
|
||||
const client = new McpClient({ transport: new SseTransport({ url: fake.url }) });
|
||||
const info = await client.initialize();
|
||||
expect(info.serverInfo.name).toBe("fake-sse");
|
||||
const { tools } = await client.listTools();
|
||||
expect(tools).toHaveLength(1);
|
||||
expect(tools[0]!.name).toBe("ping");
|
||||
|
||||
await client.close();
|
||||
|
||||
// We should have exactly: GET /sse, POST initialize, POST notifications/initialized, POST tools/list
|
||||
const posts = fake.requests.filter((r) => r.method === "POST");
|
||||
expect(posts).toHaveLength(3);
|
||||
const methods = posts.map((p) => (JSON.parse(p.body!) as { method: string }).method);
|
||||
expect(methods).toEqual(["initialize", "notifications/initialized", "tools/list"]);
|
||||
});
|
||||
|
||||
it("accepts an absolute endpoint URL", async () => {
|
||||
// Spin up a first server just to get a port we can embed in the other.
|
||||
const probe = await startFakeSseServer();
|
||||
const absoluteEndpoint = new URL("/messages", probe.url).toString();
|
||||
await probe.stop();
|
||||
|
||||
fake = await startFakeSseServer({
|
||||
endpointPath: absoluteEndpoint,
|
||||
autoRespond: (req) => ({
|
||||
jsonrpc: "2.0",
|
||||
id: (req as { id: number }).id,
|
||||
result: { pong: true },
|
||||
}),
|
||||
});
|
||||
// Point the SSE transport at THIS server, but have it advertise the
|
||||
// stale probe URL — we care that the client stores it verbatim
|
||||
// rather than resolving it against the base, so the POST will land
|
||||
// on the dead probe port and fail. That's the assertion.
|
||||
const t = new SseTransport({ url: fake.url });
|
||||
await expect(t.send({ jsonrpc: "2.0", id: 1, method: "ping" })).rejects.toThrow();
|
||||
await t.close();
|
||||
});
|
||||
|
||||
it("surfaces a handshake failure as a synthetic error frame", async () => {
|
||||
fake = await startFakeSseServer({ handshakeStatus: 500 });
|
||||
const t = new SseTransport({ url: fake.url });
|
||||
// Any pending send() rejects with the handshake error.
|
||||
await expect(t.send({ jsonrpc: "2.0", id: 1, method: "ping" })).rejects.toThrow(/500/);
|
||||
await t.close();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,541 @@
|
||||
/**
|
||||
* MCP client + bridge tests.
|
||||
*
|
||||
* Strategy: fake the McpTransport with an in-process pair. The fake
|
||||
* server understands initialize / tools/list / tools/call and returns
|
||||
* canned responses, letting us verify the client side without spawning
|
||||
* a real process.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { McpClient } from "../src/mcp/client.js";
|
||||
import { bridgeMcpTools, flattenMcpResult } from "../src/mcp/registry.js";
|
||||
import type { McpTransport } from "../src/mcp/stdio.js";
|
||||
import {
|
||||
type CallToolResult,
|
||||
type GetPromptResult,
|
||||
type JsonRpcMessage,
|
||||
type JsonRpcRequest,
|
||||
type ListPromptsResult,
|
||||
type ListResourcesResult,
|
||||
MCP_PROTOCOL_VERSION,
|
||||
type McpTool,
|
||||
type ReadResourceResult,
|
||||
} from "../src/mcp/types.js";
|
||||
|
||||
// ---------- fake transport ----------
|
||||
|
||||
interface FakeServerOptions {
|
||||
tools: McpTool[];
|
||||
/** Server's response per (name, args). Called for tools/call. */
|
||||
callHandler?: (name: string, args: Record<string, unknown>) => CallToolResult;
|
||||
/** Return an error from tools/call instead of a result. */
|
||||
errorFor?: Set<string>;
|
||||
/** Track every call the server received. */
|
||||
received?: JsonRpcRequest[];
|
||||
/** resources/list response. Optional — omit to return empty. */
|
||||
listResources?: () => ListResourcesResult;
|
||||
/** resources/read response keyed by URI. Throw-returns method-not-found for unknowns. */
|
||||
readResource?: (uri: string) => ReadResourceResult;
|
||||
/** prompts/list response. */
|
||||
listPrompts?: () => ListPromptsResult;
|
||||
/** prompts/get response keyed by name. */
|
||||
getPrompt?: (name: string, args?: Record<string, string>) => GetPromptResult;
|
||||
/** Initialize capabilities override — defaults advertise tools only. */
|
||||
capabilities?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
/**
|
||||
* In-process MCP server. Responds directly in `send()` by pushing a
|
||||
* response onto the messages queue. Synchronous-enough to make tests
|
||||
* deterministic.
|
||||
*/
|
||||
class FakeMcpTransport implements McpTransport {
|
||||
private readonly queue: JsonRpcMessage[] = [];
|
||||
private readonly waiters: Array<(m: JsonRpcMessage | null) => void> = [];
|
||||
private closed = false;
|
||||
constructor(private readonly opts: FakeServerOptions) {
|
||||
opts.received = opts.received ?? [];
|
||||
}
|
||||
|
||||
async send(msg: JsonRpcMessage): Promise<void> {
|
||||
if (this.closed) throw new Error("fake transport closed");
|
||||
if (!("method" in msg)) return; // response frames from client? never happens
|
||||
if (!("id" in msg)) {
|
||||
// notification — e.g. notifications/initialized — acknowledge silently
|
||||
this.opts.received!.push(msg as JsonRpcRequest);
|
||||
return;
|
||||
}
|
||||
const req = msg as JsonRpcRequest;
|
||||
this.opts.received!.push(req);
|
||||
const response = this.handle(req);
|
||||
this.push(response);
|
||||
}
|
||||
|
||||
async *messages(): AsyncIterableIterator<JsonRpcMessage> {
|
||||
while (true) {
|
||||
if (this.queue.length > 0) {
|
||||
yield this.queue.shift()!;
|
||||
continue;
|
||||
}
|
||||
if (this.closed) return;
|
||||
const next = await new Promise<JsonRpcMessage | null>((resolve) => {
|
||||
this.waiters.push(resolve);
|
||||
});
|
||||
if (next === null) return;
|
||||
yield next;
|
||||
}
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
this.closed = true;
|
||||
while (this.waiters.length > 0) this.waiters.shift()!(null);
|
||||
}
|
||||
|
||||
private handle(req: JsonRpcRequest): JsonRpcMessage {
|
||||
switch (req.method) {
|
||||
case "initialize":
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
result: {
|
||||
protocolVersion: MCP_PROTOCOL_VERSION,
|
||||
serverInfo: { name: "fake-mcp", version: "0.0.0" },
|
||||
capabilities: this.opts.capabilities ?? { tools: { listChanged: false } },
|
||||
},
|
||||
};
|
||||
case "tools/list":
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
result: { tools: this.opts.tools },
|
||||
};
|
||||
case "tools/call": {
|
||||
const params = req.params as { name: string; arguments?: Record<string, unknown> };
|
||||
if (this.opts.errorFor?.has(params.name)) {
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
error: { code: -32001, message: `server-side error for ${params.name}` },
|
||||
};
|
||||
}
|
||||
const result = this.opts.callHandler
|
||||
? this.opts.callHandler(params.name, params.arguments ?? {})
|
||||
: { content: [{ type: "text" as const, text: `called ${params.name}` }] };
|
||||
return { jsonrpc: "2.0", id: req.id, result };
|
||||
}
|
||||
case "resources/list":
|
||||
if (!this.opts.listResources) {
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
error: { code: -32601, message: "method not found: resources/list" },
|
||||
};
|
||||
}
|
||||
return { jsonrpc: "2.0", id: req.id, result: this.opts.listResources() };
|
||||
case "resources/read": {
|
||||
if (!this.opts.readResource) {
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
error: { code: -32601, message: "method not found: resources/read" },
|
||||
};
|
||||
}
|
||||
const { uri } = req.params as { uri: string };
|
||||
return { jsonrpc: "2.0", id: req.id, result: this.opts.readResource(uri) };
|
||||
}
|
||||
case "prompts/list":
|
||||
if (!this.opts.listPrompts) {
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
error: { code: -32601, message: "method not found: prompts/list" },
|
||||
};
|
||||
}
|
||||
return { jsonrpc: "2.0", id: req.id, result: this.opts.listPrompts() };
|
||||
case "prompts/get": {
|
||||
if (!this.opts.getPrompt) {
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
error: { code: -32601, message: "method not found: prompts/get" },
|
||||
};
|
||||
}
|
||||
const { name, arguments: a } = req.params as {
|
||||
name: string;
|
||||
arguments?: Record<string, string>;
|
||||
};
|
||||
return { jsonrpc: "2.0", id: req.id, result: this.opts.getPrompt(name, a) };
|
||||
}
|
||||
default:
|
||||
return {
|
||||
jsonrpc: "2.0",
|
||||
id: req.id,
|
||||
error: { code: -32601, message: `method not found: ${req.method}` },
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
private push(msg: JsonRpcMessage): void {
|
||||
const w = this.waiters.shift();
|
||||
if (w) w(msg);
|
||||
else this.queue.push(msg);
|
||||
}
|
||||
}
|
||||
|
||||
// ---------- tests ----------
|
||||
|
||||
describe("McpClient: initialize handshake", () => {
|
||||
it("completes initialize and sends notifications/initialized", async () => {
|
||||
const received: JsonRpcRequest[] = [];
|
||||
const transport = new FakeMcpTransport({ tools: [], received });
|
||||
const client = new McpClient({ transport });
|
||||
const info = await client.initialize();
|
||||
expect(info.protocolVersion).toBe(MCP_PROTOCOL_VERSION);
|
||||
expect(info.serverInfo.name).toBe("fake-mcp");
|
||||
|
||||
// Client should have sent two messages: initialize + notifications/initialized
|
||||
expect(received).toHaveLength(2);
|
||||
expect(received[0]!.method).toBe("initialize");
|
||||
expect(received[1]!.method).toBe("notifications/initialized");
|
||||
|
||||
await client.close();
|
||||
});
|
||||
|
||||
it("refuses listTools before initialize", async () => {
|
||||
const client = new McpClient({ transport: new FakeMcpTransport({ tools: [] }) });
|
||||
await expect(client.listTools()).rejects.toThrow(/not initialized/);
|
||||
await client.close();
|
||||
});
|
||||
|
||||
it("refuses a second initialize call", async () => {
|
||||
const client = new McpClient({ transport: new FakeMcpTransport({ tools: [] }) });
|
||||
await client.initialize();
|
||||
await expect(client.initialize()).rejects.toThrow(/already initialized/);
|
||||
await client.close();
|
||||
});
|
||||
});
|
||||
|
||||
describe("McpClient: tools/list + tools/call", () => {
|
||||
const SAMPLE_TOOLS: McpTool[] = [
|
||||
{
|
||||
name: "echo",
|
||||
description: "echoes its input",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: { msg: { type: "string" } },
|
||||
required: ["msg"],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "add",
|
||||
description: "a+b",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: { a: { type: "integer" }, b: { type: "integer" } },
|
||||
required: ["a", "b"],
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
it("lists tools and invokes them", async () => {
|
||||
const transport = new FakeMcpTransport({
|
||||
tools: SAMPLE_TOOLS,
|
||||
callHandler: (name, args) => {
|
||||
if (name === "echo") {
|
||||
return { content: [{ type: "text", text: String(args.msg ?? "") }] };
|
||||
}
|
||||
if (name === "add") {
|
||||
const sum = Number(args.a) + Number(args.b);
|
||||
return { content: [{ type: "text", text: String(sum) }] };
|
||||
}
|
||||
return { content: [{ type: "text", text: "?" }] };
|
||||
},
|
||||
});
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
|
||||
const { tools } = await client.listTools();
|
||||
expect(tools).toHaveLength(2);
|
||||
expect(tools.map((t) => t.name)).toEqual(["echo", "add"]);
|
||||
|
||||
const echoResult = await client.callTool("echo", { msg: "hi" });
|
||||
expect(echoResult.content[0]).toEqual({ type: "text", text: "hi" });
|
||||
|
||||
const addResult = await client.callTool("add", { a: 17, b: 25 });
|
||||
expect(addResult.content[0]).toEqual({ type: "text", text: "42" });
|
||||
|
||||
await client.close();
|
||||
});
|
||||
|
||||
it("surfaces server errors as rejected promises", async () => {
|
||||
const transport = new FakeMcpTransport({
|
||||
tools: SAMPLE_TOOLS,
|
||||
errorFor: new Set(["echo"]),
|
||||
});
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
await expect(client.callTool("echo", { msg: "x" })).rejects.toThrow(/MCP -32001/);
|
||||
await client.close();
|
||||
});
|
||||
});
|
||||
|
||||
describe("bridgeMcpTools (MCP → ToolRegistry)", () => {
|
||||
it("registers every MCP tool into a ToolRegistry and dispatch calls through the client", async () => {
|
||||
const transport = new FakeMcpTransport({
|
||||
tools: [
|
||||
{
|
||||
name: "echo",
|
||||
description: "echoes",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: { msg: { type: "string" } },
|
||||
required: ["msg"],
|
||||
},
|
||||
},
|
||||
],
|
||||
callHandler: (_name, args) => ({
|
||||
content: [{ type: "text", text: `you said: ${args.msg}` }],
|
||||
}),
|
||||
});
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
|
||||
const { registry, registeredNames } = await bridgeMcpTools(client);
|
||||
expect(registeredNames).toEqual(["echo"]);
|
||||
expect(registry.has("echo")).toBe(true);
|
||||
|
||||
// Dispatching through the registry should go through the MCP transport
|
||||
const result = await registry.dispatch("echo", JSON.stringify({ msg: "hello" }));
|
||||
expect(result).toContain("you said: hello");
|
||||
|
||||
await client.close();
|
||||
});
|
||||
|
||||
it("applies a name prefix when collisions could happen", async () => {
|
||||
const transport = new FakeMcpTransport({
|
||||
tools: [
|
||||
{
|
||||
name: "search",
|
||||
description: "fs search",
|
||||
inputSchema: { type: "object", properties: {} },
|
||||
},
|
||||
],
|
||||
});
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
|
||||
const { registeredNames, registry } = await bridgeMcpTools(client, { namePrefix: "fs_" });
|
||||
expect(registeredNames).toEqual(["fs_search"]);
|
||||
expect(registry.has("fs_search")).toBe(true);
|
||||
expect(registry.has("search")).toBe(false);
|
||||
|
||||
await client.close();
|
||||
});
|
||||
});
|
||||
|
||||
describe("flattenMcpResult", () => {
|
||||
it("joins text blocks with newlines", () => {
|
||||
const out = flattenMcpResult({
|
||||
content: [
|
||||
{ type: "text", text: "line one" },
|
||||
{ type: "text", text: "line two" },
|
||||
],
|
||||
});
|
||||
expect(out).toBe("line one\nline two");
|
||||
});
|
||||
|
||||
it("prefixes error results", () => {
|
||||
const out = flattenMcpResult({
|
||||
content: [{ type: "text", text: "bad input" }],
|
||||
isError: true,
|
||||
});
|
||||
expect(out).toMatch(/^ERROR: /);
|
||||
expect(out).toContain("bad input");
|
||||
});
|
||||
|
||||
it("renders image blocks as placeholders", () => {
|
||||
const out = flattenMcpResult({
|
||||
content: [{ type: "image", data: "abc123", mimeType: "image/png" }],
|
||||
});
|
||||
expect(out).toContain("[image image/png");
|
||||
});
|
||||
|
||||
it("truncates oversized results when maxChars is given, keeping head + tail", () => {
|
||||
const big = `${"X".repeat(50_000)}END_MARKER`;
|
||||
const out = flattenMcpResult({ content: [{ type: "text", text: big }] }, { maxChars: 10_000 });
|
||||
expect(out.length).toBeLessThan(11_000); // cap + a small envelope
|
||||
expect(out).toContain("truncated");
|
||||
// tail preservation: the distinctive END_MARKER at the original's end must survive
|
||||
expect(out).toContain("END_MARKER");
|
||||
// head preservation: first chars must survive
|
||||
expect(out.startsWith("X")).toBe(true);
|
||||
});
|
||||
|
||||
it("does NOT truncate when below the cap", () => {
|
||||
const small = "hello world";
|
||||
const out = flattenMcpResult({ content: [{ type: "text", text: small }] }, { maxChars: 1000 });
|
||||
expect(out).toBe("hello world");
|
||||
});
|
||||
});
|
||||
|
||||
describe("bridgeMcpTools: result-size cap", () => {
|
||||
it("caps a giant MCP tool result before handing it to the registry", async () => {
|
||||
// Minimal local fake — just enough to exercise the dispatch path.
|
||||
const transport: McpTransport = {
|
||||
async send() {},
|
||||
async *messages() {},
|
||||
async close() {},
|
||||
};
|
||||
const big = "A".repeat(200_000);
|
||||
const client = {
|
||||
listTools: async () => ({
|
||||
tools: [
|
||||
{
|
||||
name: "dump",
|
||||
description: "returns a massive string",
|
||||
inputSchema: { type: "object", properties: {} },
|
||||
},
|
||||
],
|
||||
}),
|
||||
callTool: async () => ({ content: [{ type: "text" as const, text: big }] }),
|
||||
} as unknown as McpClient;
|
||||
// Default cap (32k): enough to confirm the feature bites.
|
||||
const { registry } = await bridgeMcpTools(client, { namePrefix: "x_" });
|
||||
const out = await registry.dispatch("x_dump", "{}");
|
||||
expect(out.length).toBeLessThan(33_000);
|
||||
expect(out).toContain("truncated");
|
||||
// Sanity-silence TS about the unused transport binding.
|
||||
void transport;
|
||||
});
|
||||
});
|
||||
|
||||
describe("McpClient: resources", () => {
|
||||
it("lists resources and reads one by URI", async () => {
|
||||
const transport = new FakeMcpTransport({
|
||||
tools: [],
|
||||
capabilities: { tools: {}, resources: { listChanged: false } },
|
||||
listResources: () => ({
|
||||
resources: [
|
||||
{ uri: "file:///a.md", name: "a", mimeType: "text/markdown" },
|
||||
{ uri: "custom://b", name: "b" },
|
||||
],
|
||||
}),
|
||||
readResource: (uri) => ({
|
||||
contents: [{ uri, mimeType: "text/plain", text: `body of ${uri}` }],
|
||||
}),
|
||||
});
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
|
||||
const list = await client.listResources();
|
||||
expect(list.resources).toHaveLength(2);
|
||||
expect(list.resources[0]!.uri).toBe("file:///a.md");
|
||||
|
||||
const read = await client.readResource("file:///a.md");
|
||||
expect(read.contents).toHaveLength(1);
|
||||
const block = read.contents[0]!;
|
||||
expect("text" in block ? block.text : "").toBe("body of file:///a.md");
|
||||
expect(block.mimeType).toBe("text/plain");
|
||||
|
||||
await client.close();
|
||||
});
|
||||
|
||||
it("surfaces method-not-found as an error when the server lacks resources support", async () => {
|
||||
// Default FakeMcpTransport (no listResources handler) → −32601.
|
||||
const transport = new FakeMcpTransport({ tools: [] });
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
await expect(client.listResources()).rejects.toThrow(/-32601/);
|
||||
await client.close();
|
||||
});
|
||||
|
||||
it("advertises resources + prompts in the initialize capabilities payload", async () => {
|
||||
const received: JsonRpcRequest[] = [];
|
||||
const transport = new FakeMcpTransport({ tools: [], received });
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
const init = received.find((r) => r.method === "initialize")!;
|
||||
const params = init.params as { capabilities: Record<string, unknown> };
|
||||
// The client now claims to support all three method families.
|
||||
expect(params.capabilities).toHaveProperty("tools");
|
||||
expect(params.capabilities).toHaveProperty("resources");
|
||||
expect(params.capabilities).toHaveProperty("prompts");
|
||||
await client.close();
|
||||
});
|
||||
|
||||
it("propagates the pagination cursor to resources/list", async () => {
|
||||
const received: JsonRpcRequest[] = [];
|
||||
const transport = new FakeMcpTransport({
|
||||
tools: [],
|
||||
received,
|
||||
listResources: () => ({ resources: [], nextCursor: undefined }),
|
||||
});
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
await client.listResources("page2");
|
||||
const listReq = received.find((r) => r.method === "resources/list")!;
|
||||
expect((listReq.params as { cursor?: string }).cursor).toBe("page2");
|
||||
await client.close();
|
||||
});
|
||||
});
|
||||
|
||||
describe("McpClient: prompts", () => {
|
||||
it("lists prompts and fetches a rendered one with arguments", async () => {
|
||||
const transport = new FakeMcpTransport({
|
||||
tools: [],
|
||||
capabilities: { tools: {}, prompts: { listChanged: false } },
|
||||
listPrompts: () => ({
|
||||
prompts: [
|
||||
{
|
||||
name: "summarize",
|
||||
description: "summarize a document",
|
||||
arguments: [{ name: "lang", required: true }],
|
||||
},
|
||||
],
|
||||
}),
|
||||
getPrompt: (name, args) => ({
|
||||
description: `rendered ${name}`,
|
||||
messages: [
|
||||
{
|
||||
role: "user",
|
||||
content: { type: "text", text: `please summarize in ${args?.lang ?? "?"}` },
|
||||
},
|
||||
],
|
||||
}),
|
||||
});
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
|
||||
const list = await client.listPrompts();
|
||||
expect(list.prompts).toHaveLength(1);
|
||||
expect(list.prompts[0]!.name).toBe("summarize");
|
||||
expect(list.prompts[0]!.arguments?.[0]?.required).toBe(true);
|
||||
|
||||
const got = await client.getPrompt("summarize", { lang: "zh" });
|
||||
expect(got.messages).toHaveLength(1);
|
||||
const msg = got.messages[0]!;
|
||||
expect(msg.role).toBe("user");
|
||||
expect(msg.content.type).toBe("text");
|
||||
if (msg.content.type === "text") {
|
||||
expect(msg.content.text).toContain("zh");
|
||||
}
|
||||
|
||||
await client.close();
|
||||
});
|
||||
|
||||
it("omits the arguments field when caller passes no args", async () => {
|
||||
const received: JsonRpcRequest[] = [];
|
||||
const transport = new FakeMcpTransport({
|
||||
tools: [],
|
||||
received,
|
||||
getPrompt: () => ({ messages: [] }),
|
||||
});
|
||||
const client = new McpClient({ transport });
|
||||
await client.initialize();
|
||||
await client.getPrompt("hello");
|
||||
const getReq = received.find((r) => r.method === "prompts/get")!;
|
||||
expect(getReq.params).toEqual({ name: "hello" });
|
||||
await client.close();
|
||||
});
|
||||
});
|
||||
@@ -47,4 +47,33 @@ describe("ToolCallRepair pipeline", () => {
|
||||
expect(calls.length).toBe(1);
|
||||
expect(report.scavenged).toBe(0);
|
||||
});
|
||||
|
||||
it("scavenges DSML tool calls from the content channel (regular turn, not just reasoning)", () => {
|
||||
// R1 sometimes emits the DSML envelope in the content stream
|
||||
// instead of the proper tool_calls field. Before this wire-up,
|
||||
// the model's intent was silently dropped.
|
||||
const repair = new ToolCallRepair({
|
||||
allowedToolNames: new Set(["filesystem_read_file"]),
|
||||
});
|
||||
const content = [
|
||||
"I'll read the file next.",
|
||||
'<|DSML|invoke name="filesystem_read_file">',
|
||||
' <|DSML|parameter name="path" string="true">README.md</|DSML|parameter>',
|
||||
"</|DSML|invoke>",
|
||||
].join("\n");
|
||||
const { calls, report } = repair.process([], null, content);
|
||||
expect(calls.length).toBe(1);
|
||||
expect(calls[0]!.function.name).toBe("filesystem_read_file");
|
||||
expect(JSON.parse(calls[0]!.function.arguments)).toEqual({ path: "README.md" });
|
||||
expect(report.scavenged).toBe(1);
|
||||
});
|
||||
|
||||
it("does not double-count when DSML appears in both reasoning and content", () => {
|
||||
const repair = new ToolCallRepair({ allowedToolNames: new Set(["search"]) });
|
||||
const dsml =
|
||||
'<|DSML|invoke name="search"><|DSML|parameter name="q" string="true">ts</|DSML|parameter></|DSML|invoke>';
|
||||
const { calls, report } = repair.process([], dsml, dsml);
|
||||
expect(calls.length).toBe(1);
|
||||
expect(report.scavenged).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -44,4 +44,60 @@ describe("scavengeToolCalls", () => {
|
||||
const r = scavengeToolCalls(reasoning, { allowedNames: allowed, maxCalls: 2 });
|
||||
expect(r.calls.length).toBe(2);
|
||||
});
|
||||
|
||||
it("extracts a DSML invoke block with string + JSON parameters", () => {
|
||||
const dsmlAllowed = new Set(["filesystem_edit_file"]);
|
||||
const input = [
|
||||
"Let me make the edit.",
|
||||
"",
|
||||
'<|DSML|function_calls> <|DSML|invoke name="filesystem_edit_file">',
|
||||
' <|DSML|parameter name="path" string="true">F:/x.html</|DSML|parameter>',
|
||||
' <|DSML|parameter name="edits" string="false">[{"oldText":"a","newText":"b"}]</|DSML|parameter>',
|
||||
"</|DSML|invoke> </|DSML|function_calls>",
|
||||
].join("\n");
|
||||
const r = scavengeToolCalls(input, { allowedNames: dsmlAllowed });
|
||||
expect(r.calls.length).toBe(1);
|
||||
const call = r.calls[0]!;
|
||||
expect(call.function.name).toBe("filesystem_edit_file");
|
||||
const args = JSON.parse(call.function.arguments);
|
||||
expect(args.path).toBe("F:/x.html");
|
||||
expect(args.edits).toEqual([{ oldText: "a", newText: "b" }]);
|
||||
expect(r.notes[0]).toMatch(/DSML/);
|
||||
});
|
||||
|
||||
it("accepts ASCII pipe DSML variant too", () => {
|
||||
const dsmlAllowed = new Set(["search"]);
|
||||
const input =
|
||||
'<|DSML|invoke name="search"><|DSML|parameter name="q" string="true">ts</|DSML|parameter></|DSML|invoke>';
|
||||
const r = scavengeToolCalls(input, { allowedNames: dsmlAllowed });
|
||||
expect(r.calls.length).toBe(1);
|
||||
expect(JSON.parse(r.calls[0]!.function.arguments)).toEqual({ q: "ts" });
|
||||
});
|
||||
|
||||
it("DSML call with unknown tool name is skipped (allow-list guards us)", () => {
|
||||
const input =
|
||||
'<|DSML|invoke name="rm_rf_slash"><|DSML|parameter name="x" string="true">y</|DSML|parameter></|DSML|invoke>';
|
||||
const r = scavengeToolCalls(input, { allowedNames: allowed });
|
||||
expect(r.calls).toEqual([]);
|
||||
});
|
||||
|
||||
it("DSML 'string=false' with malformed JSON falls back to literal text (loses no data)", () => {
|
||||
const dsmlAllowed = new Set(["search"]);
|
||||
const input =
|
||||
'<|DSML|invoke name="search"><|DSML|parameter name="q" string="false">not valid json</|DSML|parameter></|DSML|invoke>';
|
||||
const r = scavengeToolCalls(input, { allowedNames: dsmlAllowed });
|
||||
expect(r.calls.length).toBe(1);
|
||||
expect(JSON.parse(r.calls[0]!.function.arguments)).toEqual({ q: "not valid json" });
|
||||
});
|
||||
|
||||
it("does not double-count: JSON args inside a DSML block don't become separate calls", () => {
|
||||
const dsmlAllowed = new Set(["filesystem_edit_file"]);
|
||||
// The inner JSON is a param value, not a standalone scavenge target.
|
||||
const input =
|
||||
'<|DSML|invoke name="filesystem_edit_file"><|DSML|parameter name="edits" string="false">{"name": "filesystem_edit_file", "arguments": {}}</|DSML|parameter></|DSML|invoke>';
|
||||
const r = scavengeToolCalls(input, { allowedNames: dsmlAllowed });
|
||||
// Expect exactly one call — the DSML wrapper. If Pattern B also
|
||||
// fired on the inner JSON we'd see two.
|
||||
expect(r.calls.length).toBe(1);
|
||||
});
|
||||
});
|
||||
+118
-1
@@ -1,5 +1,5 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { computeReplayStats } from "../src/replay.js";
|
||||
import { computeCumulativeStats, computeReplayStats, groupRecordsByTurn } from "../src/replay.js";
|
||||
import type { TranscriptRecord } from "../src/transcript.js";
|
||||
|
||||
const mkAssistant = (
|
||||
@@ -68,4 +68,121 @@ describe("computeReplayStats", () => {
|
||||
expect(stats.totalCostUsd).toBe(0);
|
||||
expect(stats.cacheHitRatio).toBe(0);
|
||||
});
|
||||
|
||||
it("counts harvest signal across turns with planState", () => {
|
||||
const recs: TranscriptRecord[] = [
|
||||
{
|
||||
ts: "t",
|
||||
turn: 1,
|
||||
role: "assistant_final",
|
||||
content: "a",
|
||||
planState: {
|
||||
subgoals: ["s1", "s2"],
|
||||
hypotheses: ["h1"],
|
||||
uncertainties: ["u1"],
|
||||
rejectedPaths: [],
|
||||
},
|
||||
},
|
||||
{
|
||||
ts: "t",
|
||||
turn: 2,
|
||||
role: "assistant_final",
|
||||
content: "b",
|
||||
planState: {
|
||||
subgoals: ["s3"],
|
||||
hypotheses: [],
|
||||
uncertainties: ["u2", "u3"],
|
||||
rejectedPaths: ["r1"],
|
||||
},
|
||||
},
|
||||
{
|
||||
ts: "t",
|
||||
turn: 3,
|
||||
role: "assistant_final",
|
||||
content: "c",
|
||||
// no planState — harvest was off or produced empty
|
||||
},
|
||||
];
|
||||
const stats = computeReplayStats(recs);
|
||||
expect(stats.harvestedTurns).toBe(2);
|
||||
expect(stats.totalSubgoals).toBe(3);
|
||||
expect(stats.totalUncertainties).toBe(3);
|
||||
});
|
||||
|
||||
it("reports zero harvest signal for transcripts without planState", () => {
|
||||
const recs: TranscriptRecord[] = [
|
||||
mkAssistant(1, 100, 900, 50, 0.0003),
|
||||
mkAssistant(2, 100, 900, 50, 0.0003),
|
||||
];
|
||||
const stats = computeReplayStats(recs);
|
||||
expect(stats.harvestedTurns).toBe(0);
|
||||
expect(stats.totalSubgoals).toBe(0);
|
||||
expect(stats.totalUncertainties).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("groupRecordsByTurn (replay nav)", () => {
|
||||
it("groups records by turn and preserves in-turn order", () => {
|
||||
const recs: TranscriptRecord[] = [
|
||||
{ ts: "t", turn: 1, role: "user", content: "q1" },
|
||||
mkAssistant(1, 100, 0, 10, 0),
|
||||
{ ts: "t", turn: 1, role: "tool", content: "r", tool: "foo", args: "{}" },
|
||||
{ ts: "t", turn: 2, role: "user", content: "q2" },
|
||||
mkAssistant(2, 100, 0, 10, 0),
|
||||
];
|
||||
const pages = groupRecordsByTurn(recs);
|
||||
expect(pages).toHaveLength(2);
|
||||
expect(pages[0]!.turn).toBe(1);
|
||||
expect(pages[0]!.records).toHaveLength(3);
|
||||
expect(pages[0]!.records[0]!.role).toBe("user");
|
||||
expect(pages[0]!.records[2]!.role).toBe("tool");
|
||||
expect(pages[1]!.turn).toBe(2);
|
||||
expect(pages[1]!.records).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("returns pages sorted by turn even if records appear out of order", () => {
|
||||
const recs: TranscriptRecord[] = [
|
||||
{ ts: "t", turn: 3, role: "user", content: "c" },
|
||||
{ ts: "t", turn: 1, role: "user", content: "a" },
|
||||
{ ts: "t", turn: 2, role: "user", content: "b" },
|
||||
];
|
||||
const pages = groupRecordsByTurn(recs);
|
||||
expect(pages.map((p) => p.turn)).toEqual([1, 2, 3]);
|
||||
});
|
||||
|
||||
it("handles an empty record list", () => {
|
||||
expect(groupRecordsByTurn([])).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("computeCumulativeStats (replay nav)", () => {
|
||||
it("grows monotonically as the cursor advances", () => {
|
||||
const pages = groupRecordsByTurn([
|
||||
mkAssistant(1, 100, 900, 10, 0.001),
|
||||
mkAssistant(2, 200, 800, 10, 0.002),
|
||||
mkAssistant(3, 300, 700, 10, 0.003),
|
||||
]);
|
||||
const s0 = computeCumulativeStats(pages, 0);
|
||||
const s1 = computeCumulativeStats(pages, 1);
|
||||
const s2 = computeCumulativeStats(pages, 2);
|
||||
expect(s0.turns).toBe(1);
|
||||
expect(s1.turns).toBe(2);
|
||||
expect(s2.turns).toBe(3);
|
||||
expect(s0.totalCostUsd).toBeLessThan(s1.totalCostUsd);
|
||||
expect(s1.totalCostUsd).toBeLessThan(s2.totalCostUsd);
|
||||
});
|
||||
|
||||
it("returns empty stats for upToIdx < 0", () => {
|
||||
const pages = groupRecordsByTurn([mkAssistant(1, 100, 900, 10, 0.001)]);
|
||||
const s = computeCumulativeStats(pages, -1);
|
||||
expect(s.turns).toBe(0);
|
||||
expect(s.totalCostUsd).toBe(0);
|
||||
expect(s.prefixHashes).toEqual([]);
|
||||
});
|
||||
|
||||
it("clamps gracefully when upToIdx exceeds page count", () => {
|
||||
const pages = groupRecordsByTurn([mkAssistant(1, 100, 900, 10, 0.001)]);
|
||||
const s = computeCumulativeStats(pages, 99);
|
||||
expect(s.turns).toBe(1); // only the one real page contributes
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,123 @@
|
||||
/**
|
||||
* Precedence test for `resolveDefaults` — the glue between CLI flags
|
||||
* and `~/.reasonix/config.json`. Bugs here would look like "my config
|
||||
* doesn't do anything" or "--model was ignored"; both would be
|
||||
* invisible to the rest of the test suite, which is why this one exists.
|
||||
*/
|
||||
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import { resolveDefaults } from "../src/cli/resolve.js";
|
||||
import { writeConfig } from "../src/config.js";
|
||||
|
||||
// resolve.ts reads the real ~/.reasonix/config.json via readConfig().
|
||||
// Redirect HOME to a temp dir for each test so we never touch the
|
||||
// user's real config and we start each case with a clean slate.
|
||||
describe("resolveDefaults", () => {
|
||||
let home: string;
|
||||
const origHome = process.env.HOME;
|
||||
const origUserProfile = process.env.USERPROFILE;
|
||||
|
||||
beforeEach(() => {
|
||||
home = mkdtempSync(join(tmpdir(), "reasonix-resolve-"));
|
||||
process.env.HOME = home;
|
||||
process.env.USERPROFILE = home; // node:os homedir() uses this on Windows
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
if (origHome === undefined) {
|
||||
// biome-ignore lint/performance/noDelete: process.env must lose the key, not hold "undefined"
|
||||
delete process.env.HOME;
|
||||
} else {
|
||||
process.env.HOME = origHome;
|
||||
}
|
||||
if (origUserProfile === undefined) {
|
||||
// biome-ignore lint/performance/noDelete: same reason as HOME
|
||||
delete process.env.USERPROFILE;
|
||||
} else {
|
||||
process.env.USERPROFILE = origUserProfile;
|
||||
}
|
||||
});
|
||||
|
||||
it("empty flags + empty config → hard-coded fast preset", () => {
|
||||
const r = resolveDefaults({});
|
||||
expect(r.model).toBe("deepseek-chat");
|
||||
expect(r.harvest).toBe(false);
|
||||
expect(r.branch).toBeUndefined();
|
||||
expect(r.mcp).toEqual([]);
|
||||
expect(r.session).toBe("default");
|
||||
});
|
||||
|
||||
it("config.preset 'smart' flips harvest on and switches model", () => {
|
||||
writeConfig({ preset: "smart" }, join(home, ".reasonix", "config.json"));
|
||||
const r = resolveDefaults({});
|
||||
expect(r.model).toBe("deepseek-reasoner");
|
||||
expect(r.harvest).toBe(true);
|
||||
expect(r.branch).toBeUndefined();
|
||||
});
|
||||
|
||||
it("--preset max overrides config.preset=fast", () => {
|
||||
writeConfig({ preset: "fast" }, join(home, ".reasonix", "config.json"));
|
||||
const r = resolveDefaults({ preset: "max" });
|
||||
expect(r.model).toBe("deepseek-reasoner");
|
||||
expect(r.branch).toBe(3);
|
||||
});
|
||||
|
||||
it("--model wins even when --preset is set", () => {
|
||||
const r = resolveDefaults({ preset: "max", model: "deepseek-chat" });
|
||||
expect(r.model).toBe("deepseek-chat");
|
||||
// preset still controls harvest/branch
|
||||
expect(r.harvest).toBe(true);
|
||||
expect(r.branch).toBe(3);
|
||||
});
|
||||
|
||||
it("--mcp overrides config.mcp wholesale (no merging)", () => {
|
||||
writeConfig(
|
||||
{ mcp: ["fs=npx -y @modelcontextprotocol/server-filesystem /tmp/old"] },
|
||||
join(home, ".reasonix", "config.json"),
|
||||
);
|
||||
const r = resolveDefaults({ mcp: ["new=cmd arg"] });
|
||||
expect(r.mcp).toEqual(["new=cmd arg"]);
|
||||
});
|
||||
|
||||
it("empty --mcp array falls through to config.mcp", () => {
|
||||
writeConfig(
|
||||
{ mcp: ["fs=npx -y @modelcontextprotocol/server-filesystem /tmp/safe"] },
|
||||
join(home, ".reasonix", "config.json"),
|
||||
);
|
||||
const r = resolveDefaults({ mcp: [] });
|
||||
expect(r.mcp).toHaveLength(1);
|
||||
expect(r.mcp[0]).toContain("filesystem");
|
||||
});
|
||||
|
||||
it("--no-config ignores the config entirely", () => {
|
||||
writeConfig({ preset: "max", mcp: ["x=cmd"] }, join(home, ".reasonix", "config.json"));
|
||||
const r = resolveDefaults({ noConfig: true });
|
||||
expect(r.model).toBe("deepseek-chat"); // fast defaults
|
||||
expect(r.mcp).toEqual([]);
|
||||
});
|
||||
|
||||
it("--no-session beats config.session", () => {
|
||||
writeConfig({ session: "work" }, join(home, ".reasonix", "config.json"));
|
||||
const r = resolveDefaults({ session: false });
|
||||
expect(r.session).toBeUndefined();
|
||||
});
|
||||
|
||||
it("config.session=null means ephemeral by default", () => {
|
||||
writeConfig({ session: null }, join(home, ".reasonix", "config.json"));
|
||||
const r = resolveDefaults({});
|
||||
expect(r.session).toBeUndefined();
|
||||
});
|
||||
|
||||
it("--branch 5 wins; --branch 1 means 'off'", () => {
|
||||
expect(resolveDefaults({ branch: 5 }).branch).toBe(5);
|
||||
expect(resolveDefaults({ branch: 1 }).branch).toBeUndefined();
|
||||
});
|
||||
|
||||
it("--branch 99 caps at 8", () => {
|
||||
expect(resolveDefaults({ branch: 99 }).branch).toBe(8);
|
||||
});
|
||||
});
|
||||
+376
-3
@@ -1,6 +1,11 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { handleSlash, parseSlash } from "../src/cli/ui/slash.js";
|
||||
import { DeepSeekClient } from "../src/client.js";
|
||||
import {
|
||||
SLASH_COMMANDS,
|
||||
handleSlash,
|
||||
parseSlash,
|
||||
suggestSlashCommands,
|
||||
} from "../src/cli/ui/slash.js";
|
||||
import { DeepSeekClient, Usage } from "../src/client.js";
|
||||
import { CacheFirstLoop } from "../src/loop.js";
|
||||
import { ImmutablePrefix } from "../src/memory.js";
|
||||
|
||||
@@ -49,7 +54,9 @@ describe("handleSlash", () => {
|
||||
it("/status reflects current loop config", () => {
|
||||
const loop = makeLoop();
|
||||
const r = handleSlash("status", [], loop);
|
||||
expect(r.info).toMatch(/model=/);
|
||||
// New format: indented table with labeled rows ("model", "flags",
|
||||
// "ctx", "mcp", "session"). Harvest/branch live on the flags row.
|
||||
expect(r.info).toMatch(/model\s+deepseek-/);
|
||||
expect(r.info).toMatch(/harvest=off/);
|
||||
expect(r.info).toMatch(/branch=off/);
|
||||
});
|
||||
@@ -116,6 +123,66 @@ describe("handleSlash", () => {
|
||||
expect(r.info).toMatch(/unknown command/);
|
||||
});
|
||||
|
||||
it("/mcp with no servers attached points at reasonix setup", () => {
|
||||
const r = handleSlash("mcp", [], makeLoop());
|
||||
expect(r.info).toMatch(/no MCP servers/);
|
||||
expect(r.info).toMatch(/reasonix setup/);
|
||||
});
|
||||
|
||||
it("/mcp shows the spec strings from SlashContext", () => {
|
||||
const r = handleSlash("mcp", [], makeLoop(), {
|
||||
mcpSpecs: [
|
||||
"filesystem=npx -y @modelcontextprotocol/server-filesystem /tmp",
|
||||
"kb=https://kb.example.com/sse",
|
||||
],
|
||||
});
|
||||
expect(r.info).toMatch(/MCP servers \(2\)/);
|
||||
expect(r.info).toMatch(/server-filesystem/);
|
||||
expect(r.info).toMatch(/kb.example.com/);
|
||||
});
|
||||
|
||||
it("/setup prints instructions to exit and run reasonix setup", () => {
|
||||
const r = handleSlash("setup", [], makeLoop());
|
||||
expect(r.info).toMatch(/reasonix setup/);
|
||||
expect(r.exit).toBeUndefined(); // /setup doesn't auto-exit — user presses /exit
|
||||
});
|
||||
|
||||
it("/compact says 'nothing to compact' when no tool messages exceed the cap", () => {
|
||||
const loop = makeLoop();
|
||||
loop.log.append({ role: "user", content: "hi" });
|
||||
loop.log.append({ role: "tool", tool_call_id: "t1", content: "short result" });
|
||||
const r = handleSlash("compact", [], loop);
|
||||
expect(r.info).toMatch(/nothing to compact/);
|
||||
});
|
||||
|
||||
it("/compact shrinks oversized tool results and reports chars saved", () => {
|
||||
const loop = makeLoop();
|
||||
loop.log.append({ role: "user", content: "read a big file" });
|
||||
loop.log.append({ role: "tool", tool_call_id: "t1", content: "Z".repeat(20_000) });
|
||||
const r = handleSlash("compact", [], loop);
|
||||
expect(r.info).toMatch(/compacted 1 tool result/);
|
||||
expect(r.info).toMatch(/saved/);
|
||||
// After compaction the tool message length should be below the default 4k cap + envelope.
|
||||
const toolEntry = loop.log.entries.find((m) => m.role === "tool");
|
||||
expect(typeof toolEntry?.content).toBe("string");
|
||||
expect((toolEntry?.content as string).length).toBeLessThan(5_000);
|
||||
});
|
||||
|
||||
it("/compact honors a custom cap argument", () => {
|
||||
const loop = makeLoop();
|
||||
loop.log.append({ role: "tool", tool_call_id: "t1", content: "A".repeat(10_000) });
|
||||
// 2000-char cap should shrink the 10k message
|
||||
const r = handleSlash("compact", ["2000"], loop);
|
||||
expect(r.info).toMatch(/compacted 1/);
|
||||
const toolEntry = loop.log.entries.find((m) => m.role === "tool");
|
||||
expect((toolEntry?.content as string).length).toBeLessThan(2_500);
|
||||
});
|
||||
|
||||
it("/help mentions /compact", () => {
|
||||
const r = handleSlash("help", [], makeLoop());
|
||||
expect(r.info).toMatch(/\/compact/);
|
||||
});
|
||||
|
||||
it("/preset fast = deepseek-chat, no harvest, no branch", () => {
|
||||
const loop = makeLoop();
|
||||
handleSlash("model", ["deepseek-reasoner"], loop);
|
||||
@@ -162,6 +229,312 @@ describe("handleSlash", () => {
|
||||
expect(r.info).toMatch(/\/forget/);
|
||||
});
|
||||
|
||||
it("/help mentions /mcp and /setup", () => {
|
||||
const r = handleSlash("help", [], makeLoop());
|
||||
expect(r.info).toMatch(/\/mcp/);
|
||||
expect(r.info).toMatch(/\/setup/);
|
||||
});
|
||||
|
||||
it("/undo outside code mode says it's not available", () => {
|
||||
const r = handleSlash("undo", [], makeLoop());
|
||||
expect(r.info).toMatch(/only available inside .reasonix code/);
|
||||
});
|
||||
|
||||
it("/undo in code mode invokes the callback", () => {
|
||||
const r = handleSlash("undo", [], makeLoop(), {
|
||||
codeUndo: () => "▸ restored 2 file(s)",
|
||||
});
|
||||
expect(r.info).toMatch(/restored 2 file/);
|
||||
});
|
||||
|
||||
it("/help mentions /undo and /commit", () => {
|
||||
const r = handleSlash("help", [], makeLoop());
|
||||
expect(r.info).toMatch(/\/undo/);
|
||||
expect(r.info).toMatch(/\/commit/);
|
||||
});
|
||||
|
||||
it("/commit outside code mode says it's not available", () => {
|
||||
const r = handleSlash("commit", ["foo"], makeLoop());
|
||||
expect(r.info).toMatch(/only available inside .reasonix code/);
|
||||
});
|
||||
|
||||
it("/commit with no message prints usage", () => {
|
||||
const r = handleSlash("commit", [], makeLoop(), { codeRoot: "/tmp" });
|
||||
expect(r.info).toMatch(/usage: \/commit/);
|
||||
});
|
||||
|
||||
it("/apply outside code mode says it's not available", () => {
|
||||
const r = handleSlash("apply", [], makeLoop());
|
||||
expect(r.info).toMatch(/only available inside .reasonix code/);
|
||||
});
|
||||
|
||||
it("/apply in code mode invokes the callback", () => {
|
||||
const r = handleSlash("apply", [], makeLoop(), {
|
||||
codeApply: () => "▸ 2/2 edits applied",
|
||||
});
|
||||
expect(r.info).toMatch(/2\/2 edits applied/);
|
||||
});
|
||||
|
||||
it("/discard outside code mode says it's not available", () => {
|
||||
const r = handleSlash("discard", [], makeLoop());
|
||||
expect(r.info).toMatch(/only available inside .reasonix code/);
|
||||
});
|
||||
|
||||
it("/discard in code mode invokes the callback", () => {
|
||||
const r = handleSlash("discard", [], makeLoop(), {
|
||||
codeDiscard: () => "▸ discarded 3 pending",
|
||||
});
|
||||
expect(r.info).toMatch(/discarded 3 pending/);
|
||||
});
|
||||
|
||||
it("/help mentions /apply and /discard", () => {
|
||||
const r = handleSlash("help", [], makeLoop());
|
||||
expect(r.info).toMatch(/\/apply/);
|
||||
expect(r.info).toMatch(/\/discard/);
|
||||
});
|
||||
|
||||
it("/think says no reasoning cached when scratch is empty", () => {
|
||||
const r = handleSlash("think", [], makeLoop());
|
||||
expect(r.info).toMatch(/no reasoning cached/);
|
||||
});
|
||||
|
||||
it("/think dumps the full reasoning when scratch has content", () => {
|
||||
const loop = makeLoop();
|
||||
loop.scratch.reasoning = "lots of R1 deliberation here over many sentences";
|
||||
const r = handleSlash("think", [], loop);
|
||||
expect(r.info).toMatch(/full thinking/);
|
||||
expect(r.info).toContain("lots of R1 deliberation");
|
||||
});
|
||||
|
||||
it("/help mentions /think", () => {
|
||||
const r = handleSlash("help", [], makeLoop());
|
||||
expect(r.info).toMatch(/\/think/);
|
||||
});
|
||||
|
||||
it("/retry returns info + resubmit when there's a prior user message", () => {
|
||||
const loop = makeLoop();
|
||||
loop.log.append({ role: "user", content: "hello" });
|
||||
loop.log.append({ role: "assistant", content: "hi back" });
|
||||
const r = handleSlash("retry", [], loop);
|
||||
expect(r.resubmit).toBe("hello");
|
||||
expect(r.info).toMatch(/retrying/);
|
||||
// After retry, the log should be empty (last user message and
|
||||
// everything after were dropped; user will be re-pushed on next
|
||||
// successful turn).
|
||||
expect(loop.log.length).toBe(0);
|
||||
});
|
||||
|
||||
it("/retry says nothing to retry when log has no user messages", () => {
|
||||
const r = handleSlash("retry", [], makeLoop());
|
||||
expect(r.info).toMatch(/nothing to retry/);
|
||||
expect(r.resubmit).toBeUndefined();
|
||||
});
|
||||
|
||||
it("/help mentions /retry", () => {
|
||||
const r = handleSlash("help", [], makeLoop());
|
||||
expect(r.info).toMatch(/\/retry/);
|
||||
});
|
||||
|
||||
it("/tool with no history says none yet", () => {
|
||||
const r = handleSlash("tool", [], makeLoop());
|
||||
expect(r.info).toMatch(/no tool calls yet/);
|
||||
});
|
||||
|
||||
it("/tool with history lists entries, most recent first", () => {
|
||||
const r = handleSlash("tool", [], makeLoop(), {
|
||||
toolHistory: () => [
|
||||
{ toolName: "fs_read_file", text: "old content" },
|
||||
{ toolName: "fs_list_directory", text: "newer content" },
|
||||
],
|
||||
});
|
||||
expect(r.info).toMatch(/Tool calls in this session \(2/);
|
||||
// Most recent is #1 — the list must order so fs_list_directory
|
||||
// (newer) appears before fs_read_file (older).
|
||||
const idxNewer = r.info!.indexOf("fs_list_directory");
|
||||
const idxOlder = r.info!.indexOf("fs_read_file");
|
||||
expect(idxNewer).toBeGreaterThan(-1);
|
||||
expect(idxOlder).toBeGreaterThan(idxNewer);
|
||||
});
|
||||
|
||||
it("/tool N dumps the Nth-most-recent tool output in full", () => {
|
||||
const big = "X".repeat(2000);
|
||||
const r = handleSlash("tool", ["1"], makeLoop(), {
|
||||
toolHistory: () => [
|
||||
{ toolName: "fs_read_file", text: "older" },
|
||||
{ toolName: "fs_list_directory", text: big },
|
||||
],
|
||||
});
|
||||
expect(r.info).toMatch(/tool<fs_list_directory>/);
|
||||
expect(r.info).toMatch(/2000 chars/);
|
||||
// Full 2000 X's included — not truncated.
|
||||
expect(r.info).toContain(big);
|
||||
});
|
||||
|
||||
it("/tool 2 reaches one call back from the most recent", () => {
|
||||
const r = handleSlash("tool", ["2"], makeLoop(), {
|
||||
toolHistory: () => [
|
||||
{ toolName: "fs_read_file", text: "target" },
|
||||
{ toolName: "fs_list_directory", text: "most recent" },
|
||||
],
|
||||
});
|
||||
expect(r.info).toMatch(/tool<fs_read_file>/);
|
||||
expect(r.info).toContain("target");
|
||||
});
|
||||
|
||||
it("/tool N past history length reports bounds", () => {
|
||||
const r = handleSlash("tool", ["5"], makeLoop(), {
|
||||
toolHistory: () => [{ toolName: "fs_read_file", text: "one" }],
|
||||
});
|
||||
expect(r.info).toMatch(/only 1 tool call/);
|
||||
});
|
||||
|
||||
it("/tool with non-numeric arg returns usage", () => {
|
||||
const r = handleSlash("tool", ["huh"], makeLoop(), {
|
||||
toolHistory: () => [{ toolName: "fs_read_file", text: "one" }],
|
||||
});
|
||||
expect(r.info).toMatch(/usage: \/tool/);
|
||||
});
|
||||
|
||||
it("/tool list trims the display to 10 most recent but hints at older", () => {
|
||||
const many = Array.from({ length: 15 }, (_, i) => ({
|
||||
toolName: `t${i}`,
|
||||
text: `call-${i}`,
|
||||
}));
|
||||
const r = handleSlash("tool", [], makeLoop(), { toolHistory: () => many });
|
||||
// Most recent (#1 = t14) and 10th back (#10 = t5) should be shown.
|
||||
expect(r.info).toContain("t14");
|
||||
expect(r.info).toContain("t5");
|
||||
// t4 would be #11 — beyond the first-page cutoff.
|
||||
expect(r.info).toMatch(/5 earlier.*reach with \/tool N/);
|
||||
});
|
||||
|
||||
it("/help mentions /tool", () => {
|
||||
const r = handleSlash("help", [], makeLoop());
|
||||
expect(r.info).toMatch(/\/tool/);
|
||||
});
|
||||
|
||||
it("SLASH_COMMANDS registry contains every handler switch case", () => {
|
||||
// Spot-check a handful so the registry doesn't silently drift
|
||||
// from `handleSlash`. If a new case lands in handleSlash, it
|
||||
// should also show up in suggestions — bump this list when
|
||||
// adding.
|
||||
const names = SLASH_COMMANDS.map((s) => s.cmd);
|
||||
for (const required of [
|
||||
"help",
|
||||
"status",
|
||||
"preset",
|
||||
"model",
|
||||
"branch",
|
||||
"mcp",
|
||||
"tool",
|
||||
"think",
|
||||
"retry",
|
||||
"compact",
|
||||
"sessions",
|
||||
"clear",
|
||||
"exit",
|
||||
"apply",
|
||||
"discard",
|
||||
"undo",
|
||||
"commit",
|
||||
]) {
|
||||
expect(names, `registry missing /${required}`).toContain(required);
|
||||
}
|
||||
});
|
||||
|
||||
it("suggestSlashCommands filters by prefix", () => {
|
||||
expect(suggestSlashCommands("h").map((s) => s.cmd)).toEqual(["help", "harvest"]);
|
||||
// Case-insensitive.
|
||||
expect(suggestSlashCommands("HE").map((s) => s.cmd)).toEqual(["help"]);
|
||||
// Empty prefix returns everything (non-contextual).
|
||||
expect(suggestSlashCommands("").length).toBeGreaterThan(5);
|
||||
});
|
||||
|
||||
it("suggestSlashCommands hides code-mode-only entries when codeMode=false", () => {
|
||||
const names = suggestSlashCommands("", false).map((s) => s.cmd);
|
||||
expect(names).not.toContain("apply");
|
||||
expect(names).not.toContain("undo");
|
||||
});
|
||||
|
||||
it("suggestSlashCommands shows code-mode-only entries when codeMode=true", () => {
|
||||
const names = suggestSlashCommands("", true).map((s) => s.cmd);
|
||||
expect(names).toContain("apply");
|
||||
expect(names).toContain("undo");
|
||||
});
|
||||
|
||||
it("/mcp with mcpServers renders per-server tools+resources+prompts", () => {
|
||||
const r = handleSlash("mcp", [], makeLoop(), {
|
||||
mcpServers: [
|
||||
{
|
||||
label: "fs",
|
||||
spec: "fs=npx -y @scope/fs /tmp",
|
||||
toolCount: 4,
|
||||
report: {
|
||||
protocolVersion: "2024-11-05",
|
||||
serverInfo: { name: "fs-server", version: "1.0.0" },
|
||||
capabilities: { tools: {}, resources: {} },
|
||||
tools: { supported: true, items: [] },
|
||||
resources: {
|
||||
supported: true,
|
||||
items: [
|
||||
{ uri: "file:///a", name: "docs" },
|
||||
{ uri: "file:///b", name: "readme" },
|
||||
],
|
||||
},
|
||||
prompts: { supported: false, reason: "method not found (-32601)" },
|
||||
},
|
||||
},
|
||||
],
|
||||
});
|
||||
expect(r.info).toMatch(/\[fs\].*fs-server v1\.0\.0/);
|
||||
expect(r.info).toMatch(/tools\s+4/);
|
||||
expect(r.info).toMatch(/resources\s+2\s+\[docs, readme\]/);
|
||||
expect(r.info).toMatch(/prompts\s+\(not supported\)/);
|
||||
});
|
||||
|
||||
it("/mcp falls back to the spec-only list when mcpServers is absent", () => {
|
||||
const r = handleSlash("mcp", [], makeLoop(), {
|
||||
mcpSpecs: ["filesystem=npx -y @scope/fs /tmp"],
|
||||
});
|
||||
expect(r.info).toMatch(/MCP servers \(1\)/);
|
||||
expect(r.info).toMatch(/server-filesystem|fs/);
|
||||
});
|
||||
|
||||
it("/status shows ctx / session / mcp / pending lines with rich detail", () => {
|
||||
const loop = makeLoop();
|
||||
// Make it look like one turn ran so lastPromptTokens > 0.
|
||||
loop.stats.record(1, loop.model, new Usage(42_000, 50, 42_050, 40_000, 2_000));
|
||||
loop.log.append({ role: "user", content: "hi" });
|
||||
loop.log.append({ role: "assistant", content: "there" });
|
||||
const r = handleSlash("status", [], loop, {
|
||||
mcpSpecs: ["filesystem=npx -y @scope/fs /tmp", "mem=npx -y @scope/mem"],
|
||||
pendingEditCount: 3,
|
||||
});
|
||||
expect(r.info).toMatch(/model\s+deepseek-/);
|
||||
expect(r.info).toMatch(/ctx\s+\d+\.?\d*k?\/\d+k/);
|
||||
expect(r.info).toMatch(/mcp\s+2 server\(s\)/);
|
||||
expect(r.info).toMatch(/session.*\(ephemeral|session\s+"/);
|
||||
expect(r.info).toMatch(/edits\s+3 pending/);
|
||||
});
|
||||
|
||||
it("/status with pendingEditCount=0 hides the edits line", () => {
|
||||
const r = handleSlash("status", [], makeLoop(), { pendingEditCount: 0 });
|
||||
expect(r.info).not.toMatch(/pending/);
|
||||
});
|
||||
|
||||
it("/commit strips surrounding double quotes from the message", () => {
|
||||
// We can't exercise git without a real repo; instead, rely on the
|
||||
// fact that /commit fails (no git repo at /nonexistent) but the
|
||||
// failure output should reveal the stripped message in the
|
||||
// arguments we passed. We mirror this by just confirming usage
|
||||
// ISN'T printed — meaning the parser accepted a non-empty message.
|
||||
const r = handleSlash("commit", ['"fix: tests"'], makeLoop(), { codeRoot: "/nonexistent" });
|
||||
expect(r.info).not.toMatch(/usage: \/commit/);
|
||||
// It WILL say git failed since /nonexistent isn't a git repo, but
|
||||
// we don't assert the exact message — it varies by platform.
|
||||
expect(r.info).toMatch(/git (add|commit) failed/);
|
||||
});
|
||||
|
||||
it("/sessions returns a hint when none exist", () => {
|
||||
const r = handleSlash("sessions", [], makeLoop());
|
||||
expect(r.info).toMatch(/no saved sessions yet|Saved sessions/);
|
||||
|
||||
@@ -42,4 +42,13 @@ describe("SessionStats", () => {
|
||||
expect(stats.turns.length).toBe(2);
|
||||
expect(stats.aggregateCacheHitRatio).toBeCloseTo(240 / 300);
|
||||
});
|
||||
|
||||
it("summary.lastPromptTokens tracks the most recent turn only", () => {
|
||||
const stats = new SessionStats();
|
||||
expect(stats.summary().lastPromptTokens).toBe(0);
|
||||
stats.record(1, "deepseek-chat", new Usage(5_000, 100, 5_100, 4_000, 1_000));
|
||||
expect(stats.summary().lastPromptTokens).toBe(5_000);
|
||||
stats.record(2, "deepseek-chat", new Usage(42_000, 200, 42_200, 40_000, 2_000));
|
||||
expect(stats.summary().lastPromptTokens).toBe(42_000);
|
||||
});
|
||||
});
|
||||
@@ -111,4 +111,37 @@ describe("transcript writer / reader round-trip", () => {
|
||||
const { records } = parseTranscript(raw);
|
||||
expect(records).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("round-trips planState from LoopEvent to transcript", () => {
|
||||
const assistantEv: LoopEvent = {
|
||||
turn: 1,
|
||||
role: "assistant_final",
|
||||
content: "The answer is 42.",
|
||||
planState: {
|
||||
subgoals: ["compute x", "verify via alt path"],
|
||||
hypotheses: ["x could be between 40 and 50"],
|
||||
uncertainties: ["precision of intermediate rounding"],
|
||||
rejectedPaths: ["brute-force enumeration"],
|
||||
},
|
||||
};
|
||||
const rec = recordFromLoopEvent(assistantEv, {
|
||||
model: "deepseek-reasoner",
|
||||
prefixHash: "stable",
|
||||
});
|
||||
expect(rec.planState).toBeDefined();
|
||||
expect(rec.planState?.subgoals).toEqual(["compute x", "verify via alt path"]);
|
||||
expect(rec.planState?.uncertainties).toHaveLength(1);
|
||||
expect(rec.planState?.rejectedPaths[0]).toBe("brute-force enumeration");
|
||||
});
|
||||
|
||||
it("omits empty planState from the record (no bytes wasted when harvest was off)", () => {
|
||||
const assistantEv: LoopEvent = {
|
||||
turn: 1,
|
||||
role: "assistant_final",
|
||||
content: "short reply",
|
||||
planState: { subgoals: [], hypotheses: [], uncertainties: [], rejectedPaths: [] },
|
||||
};
|
||||
const rec = recordFromLoopEvent(assistantEv, { model: "deepseek-chat", prefixHash: "x" });
|
||||
expect(rec.planState).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,47 @@
|
||||
/**
|
||||
* Pure-function tests for the wizard. The Ink rendering is tested
|
||||
* manually — vitest + ink-testing-library would be another dev dep for
|
||||
* negligible coverage (arrow keys → setIndex, already obvious). What
|
||||
* we DO test: the data-transform surface, because the wizard's output
|
||||
* goes straight into config.json and then straight into the `--mcp`
|
||||
* parser — any bug shows up as "my saved config silently doesn't
|
||||
* bridge the server I picked".
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { buildSpec } from "../src/cli/ui/Wizard.js";
|
||||
import { parseMcpSpec } from "../src/mcp/spec.js";
|
||||
|
||||
describe("Wizard.buildSpec → parseMcpSpec round-trip", () => {
|
||||
it("builds a filesystem spec the parser accepts", () => {
|
||||
const spec = buildSpec("filesystem", { filesystem: "/tmp/safe" });
|
||||
expect(spec).toBe("filesystem=npx -y @modelcontextprotocol/server-filesystem /tmp/safe");
|
||||
const parsed = parseMcpSpec(spec);
|
||||
if (parsed.transport !== "stdio") throw new Error("expected stdio");
|
||||
expect(parsed.name).toBe("filesystem");
|
||||
expect(parsed.command).toBe("npx");
|
||||
expect(parsed.args).toEqual(["-y", "@modelcontextprotocol/server-filesystem", "/tmp/safe"]);
|
||||
});
|
||||
|
||||
it("omits the trailing userArg when the catalog entry needs none", () => {
|
||||
const spec = buildSpec("memory", {});
|
||||
expect(spec).toBe("memory=npx -y @modelcontextprotocol/server-memory");
|
||||
const parsed = parseMcpSpec(spec);
|
||||
expect(parsed.name).toBe("memory");
|
||||
});
|
||||
|
||||
it("quotes directory paths that contain spaces", () => {
|
||||
const spec = buildSpec("filesystem", { filesystem: "/Users/me/My Documents" });
|
||||
// Inside quotes, the parser should re-join the path as a single arg.
|
||||
const parsed = parseMcpSpec(spec);
|
||||
if (parsed.transport !== "stdio") throw new Error("expected stdio");
|
||||
expect(parsed.args.at(-1)).toBe("/Users/me/My Documents");
|
||||
});
|
||||
|
||||
it("returns the name bare when the catalog entry is unknown", () => {
|
||||
// Defensive: if someone manually edits config.json and the wizard
|
||||
// sees an unfamiliar name on re-run, we degrade gracefully rather
|
||||
// than throwing.
|
||||
expect(buildSpec("not-in-catalog", {})).toBe("not-in-catalog");
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user