mirror of
https://github.com/ouroboric-dev/ouroboric.git
synced 2026-11-07 06:15:57 -10:00
The smoke says which ygg it runs (B-111); the reviewer bench (B-108); repair-path and implementer options (B-109, B-110) (#68)
* The real-agent smoke says which ygg it runs (B-111) Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * The reviewer bench is a script with a test and planted-fault leaves (B-108) scripts/ygg-reviewer-bench.sh runs N OpenCode reviewer sessions of one model on a fresh copy of a leaf, with the smoke reviewer's prompt as the engine builds it (odin testdata/smoke-grade/prompts/review.md, the smoke-notes brief, and jormungandr's ReviewRequest), and the OpenCode adapter's settings (strict OpenRouter provider, effort high, read-only, 10 turns). It prints the journal's table: valid reports, findings, fault found, tokens and cost. A cost of 0 with tokens prints as unknown. --dry-run prints the commands and spends nothing. The leaves P0 to P3 are the product of the failed smoke run and its three planted faults. P4 is new: DELETE removes another person's note, and it shows only when server.js and notes.js are read together. Co-Authored-By: Claude Opus 5.5 <[email protected]> * A bench session gets only the variables the script names, and no global OpenCode config (B-108) The engine's OpenCode session gets only the adapter's variables. The bench passed the caller's whole environment, so an OPENCODE_* variable or ~/.config/opencode could change a session. Now each session runs under env -i with PATH, HOME, TMPDIR, XDG_CACHE_HOME, the key and the OpenCode settings, and has its own empty XDG_CONFIG_HOME. The header says the bench has no token limit, only the timeout, and lists session-N.exit. Co-Authored-By: Claude Opus 5.5 <[email protected]> * The reviewer bench's scorer is its own file, ygg-reviewer-bench.py (B-108) The Python scorer moves out of the shell script, as qwen-measure.py is, and opens its files in with blocks. Its output is unchanged: the tests pass as they were, and a replayed real session gives the same summary byte for byte. Co-Authored-By: Claude Opus 5.5 <[email protected]> * The real-agent smoke can start from a planted-fault leaf, say whether a finding took the repair path, and change the implementer (B-109, B-110) --from-leaf LEAF copies odin's smoke pack into the run's folder with LEAF as the template of repos/notes and every leaf file in the first commit. ygg-repair-path.sh reads the run's ledger and prints "repair path: finding raised, fixed, closed" or the missing step; the smoke adds that line to its report. --implementer AGENT:MODEL replaces the implementer line of the run's factory.yaml, such as opencode:openrouter/qwen/qwen3.8-27b. Co-Authored-By: Claude Opus 5.5 <[email protected]> * The real-agent smoke checks the repair path only from a leaf, copies the whole report with --clip, and keeps a leaf's start.sh shebang (B-109, B-111) The repair-path line runs only with --from-leaf, so a plain run does not end with a missing step. --clip no longer goes to ygg-evidence.sh: the whole report.txt, with the ygg line and the repair path, goes to clip.exe at the end. A run from a leaf starts an executable start.sh as a program and falls back to sh. A dry run prints what happens after the run. Co-Authored-By: Claude Opus 5.5 <[email protected]> * AGENTS.md lists the reviewer bench and the smoke's new options (B-108, B-109, B-110) Co-Authored-By: Claude Opus 5.5 <[email protected]> * The real-agent smoke's test runs the tail after the run against a fake ygg, behind a guard (B-109, B-111) run_real runs the script without --dry-run only when YGG_CMD is an executable in the test's temp folder and the engine check is off; the folder and the credentials file are the test's. The fake ygg's run writes a made-up ledger. The cases: a run from a leaf adds the repair-path line to report.txt, a plain run adds none, and --clip gives clip.exe the whole report.txt with the ygg line. Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Bruno Guilhermo de Barros Valério <[email protected]> Co-authored-by: Claude Sonnet 5.5 <[email protected]>
This commit is contained in:
30 files changed
+2096
-12
No files matched your search
@@ -39,7 +39,8 @@ folder.
|
||||
| Any real-engine run (Podman, devcontainer, the sandbox): the engine, registry config and checks | `scripts/ygg-engine.sh check\|env\|run --folder DIR`. Never set `CONTAINER_HOST` by hand: the root socket is the wrong one |
|
||||
| The realms smoke runs (v0-H13, v0-H15) and their evidence | `scripts/ygg-smoke-realms.sh`, `scripts/ygg-smoke-loop.sh`, `scripts/ygg-evidence.sh` |
|
||||
| The grader smoke on a real engine (v0-H16), with the public sample suite | `scripts/ygg-smoke-grade.sh [--build]`. Never a hidden suite |
|
||||
| The real-agent smoke (v0-H17): costs money; `--dry-run` first; the cost from the ledger | `scripts/ygg-smoke-real.sh --dry-run`, then without it; `scripts/ygg-cost.sh LEDGER`. It never prints the credentials file |
|
||||
| The real-agent smoke (v0-H17): costs money; `--dry-run` first; the cost from the ledger; a start from a planted-fault leaf; another implementer; the repair path from the ledger | `scripts/ygg-smoke-real.sh --dry-run`, then without it; `--from-leaf LEAF`, `--implementer AGENT:MODEL`; `scripts/ygg-cost.sh LEDGER`, `scripts/ygg-repair-path.sh LEDGER`. It never prints the credentials file |
|
||||
| Compare reviewer models: costs money unless `--dry-run`; on the planted-fault leaves in `scripts/testdata/reviewer-bench` | `scripts/ygg-reviewer-bench.sh --model MODEL --leaf DIR --sessions N --dry-run`, then without it |
|
||||
| Build a plan from reference commits, or execute one task by task | `scripts/plan-*` (`plan-generate.py`, `plan-replay.py`, `plan-states.sh`, `plan-red.sh`, `plan-defences.py`; `plan-blocks.py`, `plan-task-start.sh`, `plan-task-done.sh`, `plan-task-land.sh`) |
|
||||
| Check that the repos have the required setup, and that the realms' shared files match | `scripts/as-bot.sh --read all scripts/check-repos.sh`; `--check shared` runs the realms check alone |
|
||||
| Scan git history for secrets | `scripts/scan-history.sh` |
|
||||
|
||||
@@ -0,0 +1,3 @@
|
||||
# The leaves are copies of products an agent wrote; the reviewers read them as they are.
|
||||
extends: ../../../.markdownlint.yaml
|
||||
MD034: false
|
||||
+1
@@ -0,0 +1 @@
|
||||
The toy app is a test fixture of Odin, AGPL-3.0-only.
|
||||
@@ -0,0 +1,3 @@
|
||||
Notes is a tiny web product where people keep private notes, with sign-up, log-in and per-user notes.
|
||||
It keeps its data in memory and uses only Node's standard library.
|
||||
Start it with `./start.sh`, then open http://localhost:3000/.
|
||||
+192
@@ -0,0 +1,192 @@
|
||||
const http = require('http');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const users = new Map(); // email -> {id, salt, hash}
|
||||
const sessions = new Map(); // token -> email
|
||||
const notes = new Map(); // email -> [{id, text}]
|
||||
let nextId = 1;
|
||||
|
||||
const PAGE = `<!doctype html>
|
||||
<html><head><meta charset="utf-8"><title>Notes</title></head>
|
||||
<body>
|
||||
<h1>Notes</h1>
|
||||
<div id="out">
|
||||
<label>Email <input id="email" type="email" autocomplete="username"></label>
|
||||
<label>Password <input id="password" type="password" autocomplete="current-password"></label>
|
||||
<button id="signup">Sign up</button>
|
||||
<button id="login">Log in</button>
|
||||
<p id="msg"></p>
|
||||
</div>
|
||||
<div id="in" hidden>
|
||||
<button id="logout">Log out</button>
|
||||
<div>
|
||||
<label>Note <input id="note" maxlength="500"></label>
|
||||
<button id="add">Add note</button>
|
||||
</div>
|
||||
<ul id="list"></ul>
|
||||
</div>
|
||||
<script>
|
||||
const $ = id => document.getElementById(id);
|
||||
async function api(method, url, body) {
|
||||
const r = await fetch(url, {method, headers: {'Content-Type': 'application/json'},
|
||||
body: body === undefined ? undefined : JSON.stringify(body)});
|
||||
let data = null;
|
||||
try { data = await r.json(); } catch (e) {}
|
||||
return {status: r.status, data};
|
||||
}
|
||||
function show(signedIn) {
|
||||
$('out').hidden = signedIn;
|
||||
$('in').hidden = !signedIn;
|
||||
}
|
||||
async function load() {
|
||||
const r = await api('GET', '/api/notes');
|
||||
if (r.status !== 200) { show(false); return; }
|
||||
show(true);
|
||||
const ul = $('list');
|
||||
ul.textContent = '';
|
||||
for (const n of r.data) {
|
||||
const li = document.createElement('li');
|
||||
const s = document.createElement('span');
|
||||
s.textContent = n.text;
|
||||
const b = document.createElement('button');
|
||||
b.textContent = 'Delete';
|
||||
b.onclick = async () => { await api('DELETE', '/api/notes/' + n.id); load(); };
|
||||
li.append(s, ' ', b);
|
||||
ul.append(li);
|
||||
}
|
||||
}
|
||||
async function auth(path) {
|
||||
const r = await api('POST', path, {email: $('email').value, password: $('password').value});
|
||||
if (r.status === 200 || r.status === 201) { $('msg').textContent = ''; $('password').value = ''; load(); }
|
||||
else $('msg').textContent = 'refused';
|
||||
}
|
||||
$('signup').onclick = () => auth('/api/signup');
|
||||
$('login').onclick = () => auth('/api/login');
|
||||
$('logout').onclick = async () => { await api('POST', '/api/logout'); $('msg').textContent = ''; show(false); };
|
||||
$('add').onclick = async () => {
|
||||
const r = await api('POST', '/api/notes', {text: $('note').value});
|
||||
if (r.status === 201) $('note').value = '';
|
||||
load();
|
||||
};
|
||||
load();
|
||||
</script>
|
||||
</body></html>`;
|
||||
|
||||
function send(res, status, obj, headers = {}) {
|
||||
if (obj === undefined) { res.writeHead(status, headers); return res.end(); }
|
||||
res.writeHead(status, {'Content-Type': 'application/json', ...headers});
|
||||
res.end(JSON.stringify(obj));
|
||||
}
|
||||
|
||||
function readBody(req) {
|
||||
return new Promise(resolve => {
|
||||
let size = 0; const chunks = [];
|
||||
req.on('data', c => {
|
||||
size += c.length;
|
||||
if (size > 65536) { req.destroy(); return; }
|
||||
chunks.push(c);
|
||||
});
|
||||
req.on('end', () => {
|
||||
try {
|
||||
const v = JSON.parse(Buffer.concat(chunks).toString('utf8'));
|
||||
resolve(v && typeof v === 'object' && !Array.isArray(v) ? v : null);
|
||||
} catch (e) { resolve(null); }
|
||||
});
|
||||
req.on('error', () => resolve(null));
|
||||
});
|
||||
}
|
||||
|
||||
function cookieToken(req) {
|
||||
for (const part of (req.headers.cookie || '').split(';')) {
|
||||
const i = part.indexOf('=');
|
||||
if (i > 0 && part.slice(0, i).trim() === 'sid') return part.slice(i + 1).trim();
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function currentUser(req) {
|
||||
const t = cookieToken(req);
|
||||
return t && sessions.has(t) ? sessions.get(t) : null;
|
||||
}
|
||||
|
||||
function hashPw(password, salt) {
|
||||
return crypto.scryptSync(password, salt, 64);
|
||||
}
|
||||
|
||||
function startSession(res, status, email) {
|
||||
const token = crypto.randomBytes(32).toString('hex');
|
||||
sessions.set(token, email);
|
||||
send(res, status, {email}, {'Set-Cookie': `sid=${token}; Path=/; HttpOnly; SameSite=Strict`});
|
||||
}
|
||||
|
||||
const EMAIL_RE = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
|
||||
|
||||
const server = http.createServer(async (req, res) => {
|
||||
try {
|
||||
const url = new URL(req.url, 'http://x');
|
||||
const path = url.pathname;
|
||||
const m = req.method;
|
||||
|
||||
if (path === '/' && (m === 'GET' || m === 'HEAD')) {
|
||||
res.writeHead(200, {'Content-Type': 'text/html; charset=utf-8', 'Cache-Control': 'no-store'});
|
||||
return res.end(m === 'HEAD' ? undefined : PAGE);
|
||||
}
|
||||
|
||||
if (path === '/api/signup' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.email !== 'string' || typeof b.password !== 'string') return send(res, 400, {error: 'bad request'});
|
||||
const email = b.email.trim().toLowerCase();
|
||||
if (!EMAIL_RE.test(email) || email.length > 254 || b.password.length < 8 || b.password.length > 1024)
|
||||
return send(res, 400, {error: 'bad email or password'});
|
||||
if (users.has(email)) return send(res, 409, {error: 'email used'});
|
||||
const salt = crypto.randomBytes(16);
|
||||
users.set(email, {salt, hash: hashPw(b.password, salt)});
|
||||
notes.set(email, []);
|
||||
return startSession(res, 201, email);
|
||||
}
|
||||
|
||||
if (path === '/api/login' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.email !== 'string' || typeof b.password !== 'string' || b.password.length > 1024)
|
||||
return send(res, 401, {error: 'refused'});
|
||||
const u = users.get(b.email.trim().toLowerCase());
|
||||
if (!u) { hashPw(b.password, Buffer.alloc(16)); return send(res, 401, {error: 'refused'}); }
|
||||
if (!crypto.timingSafeEqual(hashPw(b.password, u.salt), u.hash)) return send(res, 401, {error: 'refused'});
|
||||
return startSession(res, 200, b.email.trim().toLowerCase());
|
||||
}
|
||||
|
||||
if (path === '/api/logout' && m === 'POST') {
|
||||
const t = cookieToken(req);
|
||||
if (t) sessions.delete(t);
|
||||
return send(res, 204, undefined, {'Set-Cookie': 'sid=; Path=/; HttpOnly; SameSite=Strict; Max-Age=0'});
|
||||
}
|
||||
|
||||
if (path === '/api/notes' || path.startsWith('/api/notes/')) {
|
||||
const email = currentUser(req);
|
||||
if (!email) return send(res, 401, {error: 'unauthorized'});
|
||||
const list = notes.get(email);
|
||||
if (path === '/api/notes' && m === 'GET') return send(res, 200, list.slice().reverse().map(n => ({id: n.id, text: n.text})));
|
||||
if (path === '/api/notes' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.text !== 'string' || b.text.length < 1 || b.text.length > 500) return send(res, 400, {error: 'bad text'});
|
||||
const n = {id: nextId++, text: b.text};
|
||||
list.push(n);
|
||||
return send(res, 201, {id: n.id, text: n.text});
|
||||
}
|
||||
const dm = path.match(/^\/api\/notes\/(\d+)$/);
|
||||
if (dm && m === 'DELETE') {
|
||||
const id = Number(dm[1]);
|
||||
const i = list.findIndex(n => n.id === id);
|
||||
if (i < 0) return send(res, 404, {error: 'not found'});
|
||||
list.splice(i, 1);
|
||||
return send(res, 204);
|
||||
}
|
||||
if (dm) return send(res, 404, {error: 'not found'});
|
||||
}
|
||||
send(res, 404, {error: 'not found'});
|
||||
} catch (e) {
|
||||
send(res, 500, {error: 'server error'});
|
||||
}
|
||||
});
|
||||
|
||||
server.listen(3000, '0.0.0.0');
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
#!/bin/sh
|
||||
cd "$(dirname "$0")" && exec node server.js
|
||||
+1
@@ -0,0 +1 @@
|
||||
The toy app is a test fixture of Odin, AGPL-3.0-only.
|
||||
@@ -0,0 +1,3 @@
|
||||
Notes is a tiny web product where people keep private notes, with sign-up, log-in and per-user notes.
|
||||
It keeps its data in memory and uses only Node's standard library.
|
||||
Start it with `./start.sh`, then open http://localhost:3000/.
|
||||
+192
@@ -0,0 +1,192 @@
|
||||
const http = require('http');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const users = new Map(); // email -> {id, salt, hash}
|
||||
const sessions = new Map(); // token -> email
|
||||
const notes = new Map(); // email -> [{id, text}]
|
||||
let nextId = 1;
|
||||
|
||||
const PAGE = `<!doctype html>
|
||||
<html><head><meta charset="utf-8"><title>Notes</title></head>
|
||||
<body>
|
||||
<h1>Notes</h1>
|
||||
<div id="out">
|
||||
<label>Email <input id="email" type="email" autocomplete="username"></label>
|
||||
<label>Password <input id="password" type="password" autocomplete="current-password"></label>
|
||||
<button id="signup">Sign up</button>
|
||||
<button id="login">Log in</button>
|
||||
<p id="msg"></p>
|
||||
</div>
|
||||
<div id="in" hidden>
|
||||
<button id="logout">Log out</button>
|
||||
<div>
|
||||
<label>Note <input id="note" maxlength="500"></label>
|
||||
<button id="add">Add note</button>
|
||||
</div>
|
||||
<ul id="list"></ul>
|
||||
</div>
|
||||
<script>
|
||||
const $ = id => document.getElementById(id);
|
||||
async function api(method, url, body) {
|
||||
const r = await fetch(url, {method, headers: {'Content-Type': 'application/json'},
|
||||
body: body === undefined ? undefined : JSON.stringify(body)});
|
||||
let data = null;
|
||||
try { data = await r.json(); } catch (e) {}
|
||||
return {status: r.status, data};
|
||||
}
|
||||
function show(signedIn) {
|
||||
$('out').hidden = signedIn;
|
||||
$('in').hidden = !signedIn;
|
||||
}
|
||||
async function load() {
|
||||
const r = await api('GET', '/api/notes');
|
||||
if (r.status !== 200) { show(false); return; }
|
||||
show(true);
|
||||
const ul = $('list');
|
||||
ul.textContent = '';
|
||||
for (const n of r.data) {
|
||||
const li = document.createElement('li');
|
||||
const s = document.createElement('span');
|
||||
s.textContent = n.text;
|
||||
const b = document.createElement('button');
|
||||
b.textContent = 'Delete';
|
||||
b.onclick = async () => { await api('DELETE', '/api/notes/' + n.id); load(); };
|
||||
li.append(s, ' ', b);
|
||||
ul.append(li);
|
||||
}
|
||||
}
|
||||
async function auth(path) {
|
||||
const r = await api('POST', path, {email: $('email').value, password: $('password').value});
|
||||
if (r.status === 200 || r.status === 201) { $('msg').textContent = ''; $('password').value = ''; load(); }
|
||||
else $('msg').textContent = 'refused';
|
||||
}
|
||||
$('signup').onclick = () => auth('/api/signup');
|
||||
$('login').onclick = () => auth('/api/login');
|
||||
$('logout').onclick = async () => { await api('POST', '/api/logout'); $('msg').textContent = ''; show(false); };
|
||||
$('add').onclick = async () => {
|
||||
const r = await api('POST', '/api/notes', {text: $('note').value});
|
||||
if (r.status === 201) $('note').value = '';
|
||||
load();
|
||||
};
|
||||
load();
|
||||
</script>
|
||||
</body></html>`;
|
||||
|
||||
function send(res, status, obj, headers = {}) {
|
||||
if (obj === undefined) { res.writeHead(status, headers); return res.end(); }
|
||||
res.writeHead(status, {'Content-Type': 'application/json', ...headers});
|
||||
res.end(JSON.stringify(obj));
|
||||
}
|
||||
|
||||
function readBody(req) {
|
||||
return new Promise(resolve => {
|
||||
let size = 0; const chunks = [];
|
||||
req.on('data', c => {
|
||||
size += c.length;
|
||||
if (size > 65536) { req.destroy(); return; }
|
||||
chunks.push(c);
|
||||
});
|
||||
req.on('end', () => {
|
||||
try {
|
||||
const v = JSON.parse(Buffer.concat(chunks).toString('utf8'));
|
||||
resolve(v && typeof v === 'object' && !Array.isArray(v) ? v : null);
|
||||
} catch (e) { resolve(null); }
|
||||
});
|
||||
req.on('error', () => resolve(null));
|
||||
});
|
||||
}
|
||||
|
||||
function cookieToken(req) {
|
||||
for (const part of (req.headers.cookie || '').split(';')) {
|
||||
const i = part.indexOf('=');
|
||||
if (i > 0 && part.slice(0, i).trim() === 'sid') return part.slice(i + 1).trim();
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function currentUser(req) {
|
||||
const t = cookieToken(req);
|
||||
return t && sessions.has(t) ? sessions.get(t) : null;
|
||||
}
|
||||
|
||||
function hashPw(password, salt) {
|
||||
return crypto.scryptSync(password, salt, 64);
|
||||
}
|
||||
|
||||
function startSession(res, status, email) {
|
||||
const token = crypto.randomBytes(32).toString('hex');
|
||||
sessions.set(token, email);
|
||||
send(res, status, {email}, {'Set-Cookie': `sid=${token}; Path=/; HttpOnly; SameSite=Strict`});
|
||||
}
|
||||
|
||||
const EMAIL_RE = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
|
||||
|
||||
const server = http.createServer(async (req, res) => {
|
||||
try {
|
||||
const url = new URL(req.url, 'http://x');
|
||||
const path = url.pathname;
|
||||
const m = req.method;
|
||||
|
||||
if (path === '/' && (m === 'GET' || m === 'HEAD')) {
|
||||
res.writeHead(200, {'Content-Type': 'text/html; charset=utf-8', 'Cache-Control': 'no-store'});
|
||||
return res.end(m === 'HEAD' ? undefined : PAGE);
|
||||
}
|
||||
|
||||
if (path === '/api/signup' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.email !== 'string' || typeof b.password !== 'string') return send(res, 400, {error: 'bad request'});
|
||||
const email = b.email.trim().toLowerCase();
|
||||
if (!EMAIL_RE.test(email) || email.length > 254 || b.password.length < 8 || b.password.length > 1024)
|
||||
return send(res, 400, {error: 'bad email or password'});
|
||||
if (users.has(email)) return send(res, 409, {error: 'email used'});
|
||||
const salt = crypto.randomBytes(16);
|
||||
users.set(email, {salt, hash: hashPw(b.password, salt)});
|
||||
notes.set(email, []);
|
||||
return startSession(res, 201, email);
|
||||
}
|
||||
|
||||
if (path === '/api/login' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.email !== 'string' || typeof b.password !== 'string' || b.password.length > 1024)
|
||||
return send(res, 401, {error: 'refused'});
|
||||
const u = users.get(b.email.trim().toLowerCase());
|
||||
if (!u) { hashPw(b.password, Buffer.alloc(16)); return send(res, 401, {error: 'refused'}); }
|
||||
if (!crypto.timingSafeEqual(hashPw(b.password, u.salt), u.hash)) return send(res, 401, {error: 'refused'});
|
||||
return startSession(res, 200, b.email.trim().toLowerCase());
|
||||
}
|
||||
|
||||
if (path === '/api/logout' && m === 'POST') {
|
||||
const t = cookieToken(req);
|
||||
if (t) sessions.delete(t);
|
||||
return send(res, 204, undefined, {'Set-Cookie': 'sid=; Path=/; HttpOnly; SameSite=Strict; Max-Age=0'});
|
||||
}
|
||||
|
||||
if (path === '/api/notes' || path.startsWith('/api/notes/')) {
|
||||
const email = currentUser(req);
|
||||
if (!email) return send(res, 401, {error: 'unauthorized'});
|
||||
const list = notes.get(email);
|
||||
if (path === '/api/notes' && m === 'GET') return send(res, 200, list.slice().reverse().map(n => ({id: n.id, text: n.text})));
|
||||
if (path === '/api/notes' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.text !== 'string' || b.text.length < 1 || b.text.length > 500) return send(res, 400, {error: 'bad text'});
|
||||
const n = {id: nextId++, text: b.text};
|
||||
list.push(n);
|
||||
return send(res, 201, {id: n.id, text: n.text});
|
||||
}
|
||||
const dm = path.match(/^\/api\/notes\/(\d+)$/);
|
||||
if (dm && m === 'DELETE') {
|
||||
const id = Number(dm[1]);
|
||||
const i = list.findIndex(n => n.id === id);
|
||||
if (i < 0) return send(res, 404, {error: 'not found'});
|
||||
list.splice(i, 1);
|
||||
return send(res, 204);
|
||||
}
|
||||
if (dm) return send(res, 404, {error: 'not found'});
|
||||
}
|
||||
send(res, 404, {error: 'not found'});
|
||||
} catch (e) {
|
||||
send(res, 500, {error: 'server error'});
|
||||
}
|
||||
});
|
||||
|
||||
server.listen(3000, '0.0.0.0');
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
#!/bin/sh
|
||||
cd "$(dirname "$0")" && exec node servr.js
|
||||
+1
@@ -0,0 +1 @@
|
||||
The toy app is a test fixture of Odin, AGPL-3.0-only.
|
||||
@@ -0,0 +1,3 @@
|
||||
Notes is a tiny web product where people keep private notes, with sign-up, log-in and per-user notes.
|
||||
It keeps its data in memory and uses only Node's standard library.
|
||||
Start it with `./start.sh`, then open http://localhost:3000/.
|
||||
+192
@@ -0,0 +1,192 @@
|
||||
const http = require('http');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const users = new Map(); // email -> {id, salt, hash}
|
||||
const sessions = new Map(); // token -> email
|
||||
const notes = new Map(); // email -> [{id, text}]
|
||||
let nextId = 1;
|
||||
|
||||
const PAGE = `<!doctype html>
|
||||
<html><head><meta charset="utf-8"><title>Notes</title></head>
|
||||
<body>
|
||||
<h1>Notes</h1>
|
||||
<div id="out">
|
||||
<label>Email <input id="email" type="email" autocomplete="username"></label>
|
||||
<label>Password <input id="password" type="password" autocomplete="current-password"></label>
|
||||
<button id="signup">Sign up</button>
|
||||
<button id="login">Log in</button>
|
||||
<p id="msg"></p>
|
||||
</div>
|
||||
<div id="in" hidden>
|
||||
<button id="logout">Log out</button>
|
||||
<div>
|
||||
<label>Note <input id="note" maxlength="500"></label>
|
||||
<button id="add">Add note</button>
|
||||
</div>
|
||||
<ul id="list"></ul>
|
||||
</div>
|
||||
<script>
|
||||
const $ = id => document.getElementById(id);
|
||||
async function api(method, url, body) {
|
||||
const r = await fetch(url, {method, headers: {'Content-Type': 'application/json'},
|
||||
body: body === undefined ? undefined : JSON.stringify(body)});
|
||||
let data = null;
|
||||
try { data = await r.json(); } catch (e) {}
|
||||
return {status: r.status, data};
|
||||
}
|
||||
function show(signedIn) {
|
||||
$('out').hidden = signedIn;
|
||||
$('in').hidden = !signedIn;
|
||||
}
|
||||
async function load() {
|
||||
const r = await api('GET', '/api/notes');
|
||||
if (r.status !== 200) { show(false); return; }
|
||||
show(true);
|
||||
const ul = $('list');
|
||||
ul.textContent = '';
|
||||
for (const n of r.data) {
|
||||
const li = document.createElement('li');
|
||||
const s = document.createElement('span');
|
||||
s.textContent = n.text;
|
||||
const b = document.createElement('button');
|
||||
b.textContent = 'Delete';
|
||||
b.onclick = async () => { await api('DELETE', '/api/notes/' + n.id); load(); };
|
||||
li.append(s, ' ', b);
|
||||
ul.append(li);
|
||||
}
|
||||
}
|
||||
async function auth(path) {
|
||||
const r = await api('POST', path, {email: $('email').value, password: $('password').value});
|
||||
if (r.status === 200 || r.status === 201) { $('msg').textContent = ''; $('password').value = ''; load(); }
|
||||
else $('msg').textContent = 'refused';
|
||||
}
|
||||
$('signup').onclick = () => auth('/api/signup');
|
||||
$('login').onclick = () => auth('/api/login');
|
||||
$('logout').onclick = async () => { await api('POST', '/api/logout'); $('msg').textContent = ''; show(false); };
|
||||
$('add').onclick = async () => {
|
||||
const r = await api('POST', '/api/notes', {text: $('note').value});
|
||||
if (r.status === 201) $('note').value = '';
|
||||
load();
|
||||
};
|
||||
load();
|
||||
</script>
|
||||
</body></html>`;
|
||||
|
||||
function send(res, status, obj, headers = {}) {
|
||||
if (obj === undefined) { res.writeHead(status, headers); return res.end(); }
|
||||
res.writeHead(status, {'Content-Type': 'application/json', ...headers});
|
||||
res.end(JSON.stringify(obj));
|
||||
}
|
||||
|
||||
function readBody(req) {
|
||||
return new Promise(resolve => {
|
||||
let size = 0; const chunks = [];
|
||||
req.on('data', c => {
|
||||
size += c.length;
|
||||
if (size > 65536) { req.destroy(); return; }
|
||||
chunks.push(c);
|
||||
});
|
||||
req.on('end', () => {
|
||||
try {
|
||||
const v = JSON.parse(Buffer.concat(chunks).toString('utf8'));
|
||||
resolve(v && typeof v === 'object' && !Array.isArray(v) ? v : null);
|
||||
} catch (e) { resolve(null); }
|
||||
});
|
||||
req.on('error', () => resolve(null));
|
||||
});
|
||||
}
|
||||
|
||||
function cookieToken(req) {
|
||||
for (const part of (req.headers.cookie || '').split(';')) {
|
||||
const i = part.indexOf('=');
|
||||
if (i > 0 && part.slice(0, i).trim() === 'sid') return part.slice(i + 1).trim();
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function currentUser(req) {
|
||||
const t = cookieToken(req);
|
||||
return t && sessions.has(t) ? sessions.get(t) : null;
|
||||
}
|
||||
|
||||
function hashPw(password, salt) {
|
||||
return crypto.scryptSync(password, salt, 64);
|
||||
}
|
||||
|
||||
function startSession(res, status, email) {
|
||||
const token = crypto.randomBytes(32).toString('hex');
|
||||
sessions.set(token, email);
|
||||
send(res, status, {email}, {'Set-Cookie': `sid=${token}; Path=/; HttpOnly; SameSite=Strict`});
|
||||
}
|
||||
|
||||
const EMAIL_RE = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
|
||||
|
||||
const server = http.createServer(async (req, res) => {
|
||||
try {
|
||||
const url = new URL(req.url, 'http://x');
|
||||
const path = url.pathname;
|
||||
const m = req.method;
|
||||
|
||||
if (path === '/' && (m === 'GET' || m === 'HEAD')) {
|
||||
res.writeHead(200, {'Content-Type': 'text/html; charset=utf-8', 'Cache-Control': 'no-store'});
|
||||
return res.end(m === 'HEAD' ? undefined : PAGE);
|
||||
}
|
||||
|
||||
if (path === '/api/signup' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.email !== 'string' || typeof b.password !== 'string') return send(res, 400, {error: 'bad request'});
|
||||
const email = b.email.trim().toLowerCase();
|
||||
if (!EMAIL_RE.test(email) || email.length > 254 || b.password.length < 8 || b.password.length > 1024)
|
||||
return send(res, 400, {error: 'bad email or password'});
|
||||
if (users.has(email)) return send(res, 409, {error: 'email used'});
|
||||
const salt = crypto.randomBytes(16);
|
||||
users.set(email, {salt, hash: hashPw(b.password, salt)});
|
||||
notes.set(email, []);
|
||||
return startSession(res, 201, email);
|
||||
}
|
||||
|
||||
if (path === '/api/login' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.email !== 'string' || typeof b.password !== 'string' || b.password.length > 1024)
|
||||
return send(res, 401, {error: 'refused'});
|
||||
const u = users.get(b.email.trim().toLowerCase());
|
||||
if (!u) { hashPw(b.password, Buffer.alloc(16)); return send(res, 401, {error: 'refused'}); }
|
||||
if (!crypto.timingSafeEqual(hashPw(b.password, u.salt), u.hash)) return send(res, 401, {error: 'refused'});
|
||||
return startSession(res, 200, b.email.trim().toLowerCase());
|
||||
}
|
||||
|
||||
if (path === '/api/logout' && m === 'POST') {
|
||||
const t = cookieToken(req);
|
||||
if (t) sessions.delete(t);
|
||||
return send(res, 204, undefined, {'Set-Cookie': 'sid=; Path=/; HttpOnly; SameSite=Strict; Max-Age=0'});
|
||||
}
|
||||
|
||||
if (path === '/api/notes' || path.startsWith('/api/notes/')) {
|
||||
const email = currentUser(req);
|
||||
if (!email) return send(res, 401, {error: 'unauthorized'});
|
||||
const list = notes.get(email);
|
||||
if (path === '/api/notes' && m === 'GET') return send(res, 200, list.slice().reverse().map(n => ({id: n.id, text: n.text})));
|
||||
if (path === '/api/notes' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.text !== 'string' || b.text.length < 1 || b.text.length > 500) return send(res, 400, {error: 'bad text'});
|
||||
const n = {id: nextId++, text: b.text};
|
||||
list.push(n);
|
||||
return send(res, 201, {id: n.id, text: n.text});
|
||||
}
|
||||
const dm = path.match(/^\/api\/notes\/(\d+)$/);
|
||||
if (dm && m === 'DELETE') {
|
||||
const id = Number(dm[1]);
|
||||
const i = list.findIndex(n => n.id === id);
|
||||
if (i < 0) return send(res, 404, {error: 'not found'});
|
||||
list.splice(i, 1);
|
||||
return send(res, 204);
|
||||
}
|
||||
if (dm) return send(res, 404, {error: 'not found'});
|
||||
}
|
||||
send(res, 404, {error: 'not found'});
|
||||
} catch (e) {
|
||||
send(res, 500, {error: 'server error'});
|
||||
}
|
||||
});
|
||||
|
||||
server.listen(8080, '0.0.0.0');
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
#!/bin/sh
|
||||
cd "$(dirname "$0")" && exec node server.js
|
||||
+1
@@ -0,0 +1 @@
|
||||
The toy app is a test fixture of Odin, AGPL-3.0-only.
|
||||
@@ -0,0 +1,3 @@
|
||||
Notes is a tiny web product where people keep private notes, with sign-up, log-in and per-user notes.
|
||||
It keeps its data in memory and uses only Node's standard library.
|
||||
Start it with `./start.sh`, then open http://localhost:3000/.
|
||||
+192
@@ -0,0 +1,192 @@
|
||||
const http = require('http');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const users = new Map(); // email -> {id, salt, hash}
|
||||
const sessions = new Map(); // token -> email
|
||||
const notes = new Map(); // email -> [{id, text}]
|
||||
let nextId = 1;
|
||||
|
||||
const PAGE = `<!doctype html>
|
||||
<html><head><meta charset="utf-8"><title>Notes</title></head>
|
||||
<body>
|
||||
<h1>Notes</h1>
|
||||
<div id="out">
|
||||
<label>Email <input id="email" type="email" autocomplete="username"></label>
|
||||
<label>Password <input id="password" type="password" autocomplete="current-password"></label>
|
||||
<button id="signup">Sign up</button>
|
||||
<button id="login">Log in</button>
|
||||
<p id="msg"></p>
|
||||
</div>
|
||||
<div id="in" hidden>
|
||||
<button id="logout">Log out</button>
|
||||
<div>
|
||||
<label>Note <input id="note" maxlength="500"></label>
|
||||
<button id="add">Add note</button>
|
||||
</div>
|
||||
<ul id="list"></ul>
|
||||
</div>
|
||||
<script>
|
||||
const $ = id => document.getElementById(id);
|
||||
async function api(method, url, body) {
|
||||
const r = await fetch(url, {method, headers: {'Content-Type': 'application/json'},
|
||||
body: body === undefined ? undefined : JSON.stringify(body)});
|
||||
let data = null;
|
||||
try { data = await r.json(); } catch (e) {}
|
||||
return {status: r.status, data};
|
||||
}
|
||||
function show(signedIn) {
|
||||
$('out').hidden = signedIn;
|
||||
$('in').hidden = !signedIn;
|
||||
}
|
||||
async function load() {
|
||||
const r = await api('GET', '/api/notes');
|
||||
if (r.status !== 200) { show(false); return; }
|
||||
show(true);
|
||||
const ul = $('list');
|
||||
ul.textContent = '';
|
||||
for (const n of r.data) {
|
||||
const li = document.createElement('li');
|
||||
const s = document.createElement('span');
|
||||
s.textContent = n.text;
|
||||
const b = document.createElement('button');
|
||||
b.textContent = 'Delete';
|
||||
b.onclick = async () => { await api('DELETE', '/api/notes/' + n.id); load(); };
|
||||
li.append(s, ' ', b);
|
||||
ul.append(li);
|
||||
}
|
||||
}
|
||||
async function auth(path) {
|
||||
const r = await api('POST', path, {email: $('email').value, password: $('password').value});
|
||||
if (r.status === 200 || r.status === 201) { $('msg').textContent = ''; $('password').value = ''; load(); }
|
||||
else $('msg').textContent = 'refused';
|
||||
}
|
||||
$('signup').onclick = () => auth('/api/signup');
|
||||
$('login').onclick = () => auth('/api/login');
|
||||
$('logout').onclick = async () => { await api('POST', '/api/logout'); $('msg').textContent = ''; show(false); };
|
||||
$('add').onclick = async () => {
|
||||
const r = await api('POST', '/api/notes', {text: $('note').value});
|
||||
if (r.status === 201) $('note').value = '';
|
||||
load();
|
||||
};
|
||||
load();
|
||||
</script>
|
||||
</body></html>`;
|
||||
|
||||
function send(res, status, obj, headers = {}) {
|
||||
if (obj === undefined) { res.writeHead(status, headers); return res.end(); }
|
||||
res.writeHead(status, {'Content-Type': 'application/json', ...headers});
|
||||
res.end(JSON.stringify(obj));
|
||||
}
|
||||
|
||||
function readBody(req) {
|
||||
return new Promise(resolve => {
|
||||
let size = 0; const chunks = [];
|
||||
req.on('data', c => {
|
||||
size += c.length;
|
||||
if (size > 65536) { req.destroy(); return; }
|
||||
chunks.push(c);
|
||||
});
|
||||
req.on('end', () => {
|
||||
try {
|
||||
const v = JSON.parse(Buffer.concat(chunks).toString('utf8'));
|
||||
resolve(v && typeof v === 'object' && !Array.isArray(v) ? v : null);
|
||||
} catch (e) { resolve(null); }
|
||||
});
|
||||
req.on('error', () => resolve(null));
|
||||
});
|
||||
}
|
||||
|
||||
function cookieToken(req) {
|
||||
for (const part of (req.headers.cookie || '').split(';')) {
|
||||
const i = part.indexOf('=');
|
||||
if (i > 0 && part.slice(0, i).trim() === 'sid') return part.slice(i + 1).trim();
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function currentUser(req) {
|
||||
const t = cookieToken(req);
|
||||
return t && sessions.has(t) ? sessions.get(t) : null;
|
||||
}
|
||||
|
||||
function hashPw(password, salt) {
|
||||
return crypto.scryptSync(password, salt, 64);
|
||||
}
|
||||
|
||||
function startSession(res, status, email) {
|
||||
const token = crypto.randomBytes(32).toString('hex');
|
||||
sessions.set(token, email);
|
||||
send(res, status, {email}, {'Set-Cookie': `sid=${token}; Path=/; HttpOnly; SameSite=Strict`});
|
||||
}
|
||||
|
||||
const EMAIL_RE = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
|
||||
|
||||
const server = http.createServer(async (req, res) => {
|
||||
try {
|
||||
const url = new URL(req.url, 'http://x');
|
||||
const path = url.pathname;
|
||||
const m = req.method;
|
||||
|
||||
if (path === '/' && (m === 'GET' || m === 'HEAD')) {
|
||||
res.writeHead(200, {'Content-Type': 'text/html; charset=utf-8', 'Cache-Control': 'no-store'});
|
||||
return res.end(m === 'HEAD' ? undefined : PAGE);
|
||||
}
|
||||
|
||||
if (path === '/api/signup' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.email !== 'string' || typeof b.password !== 'string') return send(res, 400, {error: 'bad request'});
|
||||
const email = b.email.trim().toLowerCase();
|
||||
if (!EMAIL_RE.test(email) || email.length > 254 || b.password.length < 8 || b.password.length > 1024)
|
||||
return send(res, 400, {error: 'bad email or password'});
|
||||
if (users.has(email)) return send(res, 409, {error: 'email used'});
|
||||
const salt = crypto.randomBytes(16);
|
||||
users.set(email, {salt, hash: hashPw(b.password, salt)});
|
||||
notes.set(email, []);
|
||||
return startSession(res, 201, email);
|
||||
}
|
||||
|
||||
if (path === '/api/login' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.email !== 'string' || typeof b.password !== 'string' || b.password.length > 1024)
|
||||
return send(res, 401, {error: 'refused'});
|
||||
const u = users.get(b.email.trim().toLowerCase());
|
||||
if (!u) { hashPw(b.password, Buffer.alloc(16)); return send(res, 401, {error: 'refused'}); }
|
||||
if (!crypto.timingSafeEqual(hashPw(b.password, u.salt), u.hash)) return send(res, 401, {error: 'refused'});
|
||||
return startSession(res, 200, b.email.trim().toLowerCase());
|
||||
}
|
||||
|
||||
if (path === '/api/logout' && m === 'POST') {
|
||||
const t = cookieToken(req);
|
||||
if (t) sessions.delete(t);
|
||||
return send(res, 204, undefined, {'Set-Cookie': 'sid=; Path=/; HttpOnly; SameSite=Strict; Max-Age=0'});
|
||||
}
|
||||
|
||||
if (path === '/api/notes' || path.startsWith('/api/notes/')) {
|
||||
const email = currentUser(req);
|
||||
if (!email) return send(res, 401, {error: 'unauthorized'});
|
||||
const list = notes.get(email);
|
||||
if (path === '/api/notes' && m === 'GET') return send(res, 200, list.slice().map(n => ({id: n.id, text: n.text})));
|
||||
if (path === '/api/notes' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.text !== 'string' || b.text.length < 1 || b.text.length > 500) return send(res, 400, {error: 'bad text'});
|
||||
const n = {id: nextId++, text: b.text};
|
||||
list.push(n);
|
||||
return send(res, 201, {id: n.id, text: n.text});
|
||||
}
|
||||
const dm = path.match(/^\/api\/notes\/(\d+)$/);
|
||||
if (dm && m === 'DELETE') {
|
||||
const id = Number(dm[1]);
|
||||
const i = list.findIndex(n => n.id === id);
|
||||
if (i < 0) return send(res, 404, {error: 'not found'});
|
||||
list.splice(i, 1);
|
||||
return send(res, 204);
|
||||
}
|
||||
if (dm) return send(res, 404, {error: 'not found'});
|
||||
}
|
||||
send(res, 404, {error: 'not found'});
|
||||
} catch (e) {
|
||||
send(res, 500, {error: 'server error'});
|
||||
}
|
||||
});
|
||||
|
||||
server.listen(3000, '0.0.0.0');
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
#!/bin/sh
|
||||
cd "$(dirname "$0")" && exec node server.js
|
||||
+1
@@ -0,0 +1 @@
|
||||
The toy app is a test fixture of Odin, AGPL-3.0-only.
|
||||
@@ -0,0 +1,3 @@
|
||||
Notes is a tiny web product where people keep private notes, with sign-up, log-in and per-user notes.
|
||||
It keeps its data in memory and uses only Node's standard library.
|
||||
Start it with `./start.sh`, then open http://localhost:3000/.
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
// The notes of every person, kept in memory. A note id is unique across all people.
|
||||
const all = new Map(); // id -> {id, owner, text}, oldest first
|
||||
let nextId = 1;
|
||||
|
||||
// of returns the notes of one person: the server reaches notes only through it.
|
||||
function of(owner) {
|
||||
return {
|
||||
// list returns the person's notes, newest first.
|
||||
list() {
|
||||
return [...all.values()].filter(n => n.owner === owner).reverse().map(n => ({id: n.id, text: n.text}));
|
||||
},
|
||||
add(text) {
|
||||
const n = {id: nextId++, owner, text};
|
||||
all.set(n.id, n);
|
||||
return {id: n.id, text: n.text};
|
||||
},
|
||||
// remove deletes a note and says whether it was there.
|
||||
remove(id) {
|
||||
return all.delete(id);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
module.exports = {of};
|
||||
+186
@@ -0,0 +1,186 @@
|
||||
const http = require('http');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const users = new Map(); // email -> {id, salt, hash}
|
||||
const sessions = new Map(); // token -> email
|
||||
const notes = require('./notes');
|
||||
|
||||
const PAGE = `<!doctype html>
|
||||
<html><head><meta charset="utf-8"><title>Notes</title></head>
|
||||
<body>
|
||||
<h1>Notes</h1>
|
||||
<div id="out">
|
||||
<label>Email <input id="email" type="email" autocomplete="username"></label>
|
||||
<label>Password <input id="password" type="password" autocomplete="current-password"></label>
|
||||
<button id="signup">Sign up</button>
|
||||
<button id="login">Log in</button>
|
||||
<p id="msg"></p>
|
||||
</div>
|
||||
<div id="in" hidden>
|
||||
<button id="logout">Log out</button>
|
||||
<div>
|
||||
<label>Note <input id="note" maxlength="500"></label>
|
||||
<button id="add">Add note</button>
|
||||
</div>
|
||||
<ul id="list"></ul>
|
||||
</div>
|
||||
<script>
|
||||
const $ = id => document.getElementById(id);
|
||||
async function api(method, url, body) {
|
||||
const r = await fetch(url, {method, headers: {'Content-Type': 'application/json'},
|
||||
body: body === undefined ? undefined : JSON.stringify(body)});
|
||||
let data = null;
|
||||
try { data = await r.json(); } catch (e) {}
|
||||
return {status: r.status, data};
|
||||
}
|
||||
function show(signedIn) {
|
||||
$('out').hidden = signedIn;
|
||||
$('in').hidden = !signedIn;
|
||||
}
|
||||
async function load() {
|
||||
const r = await api('GET', '/api/notes');
|
||||
if (r.status !== 200) { show(false); return; }
|
||||
show(true);
|
||||
const ul = $('list');
|
||||
ul.textContent = '';
|
||||
for (const n of r.data) {
|
||||
const li = document.createElement('li');
|
||||
const s = document.createElement('span');
|
||||
s.textContent = n.text;
|
||||
const b = document.createElement('button');
|
||||
b.textContent = 'Delete';
|
||||
b.onclick = async () => { await api('DELETE', '/api/notes/' + n.id); load(); };
|
||||
li.append(s, ' ', b);
|
||||
ul.append(li);
|
||||
}
|
||||
}
|
||||
async function auth(path) {
|
||||
const r = await api('POST', path, {email: $('email').value, password: $('password').value});
|
||||
if (r.status === 200 || r.status === 201) { $('msg').textContent = ''; $('password').value = ''; load(); }
|
||||
else $('msg').textContent = 'refused';
|
||||
}
|
||||
$('signup').onclick = () => auth('/api/signup');
|
||||
$('login').onclick = () => auth('/api/login');
|
||||
$('logout').onclick = async () => { await api('POST', '/api/logout'); $('msg').textContent = ''; show(false); };
|
||||
$('add').onclick = async () => {
|
||||
const r = await api('POST', '/api/notes', {text: $('note').value});
|
||||
if (r.status === 201) $('note').value = '';
|
||||
load();
|
||||
};
|
||||
load();
|
||||
</script>
|
||||
</body></html>`;
|
||||
|
||||
function send(res, status, obj, headers = {}) {
|
||||
if (obj === undefined) { res.writeHead(status, headers); return res.end(); }
|
||||
res.writeHead(status, {'Content-Type': 'application/json', ...headers});
|
||||
res.end(JSON.stringify(obj));
|
||||
}
|
||||
|
||||
function readBody(req) {
|
||||
return new Promise(resolve => {
|
||||
let size = 0; const chunks = [];
|
||||
req.on('data', c => {
|
||||
size += c.length;
|
||||
if (size > 65536) { req.destroy(); return; }
|
||||
chunks.push(c);
|
||||
});
|
||||
req.on('end', () => {
|
||||
try {
|
||||
const v = JSON.parse(Buffer.concat(chunks).toString('utf8'));
|
||||
resolve(v && typeof v === 'object' && !Array.isArray(v) ? v : null);
|
||||
} catch (e) { resolve(null); }
|
||||
});
|
||||
req.on('error', () => resolve(null));
|
||||
});
|
||||
}
|
||||
|
||||
function cookieToken(req) {
|
||||
for (const part of (req.headers.cookie || '').split(';')) {
|
||||
const i = part.indexOf('=');
|
||||
if (i > 0 && part.slice(0, i).trim() === 'sid') return part.slice(i + 1).trim();
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function currentUser(req) {
|
||||
const t = cookieToken(req);
|
||||
return t && sessions.has(t) ? sessions.get(t) : null;
|
||||
}
|
||||
|
||||
function hashPw(password, salt) {
|
||||
return crypto.scryptSync(password, salt, 64);
|
||||
}
|
||||
|
||||
function startSession(res, status, email) {
|
||||
const token = crypto.randomBytes(32).toString('hex');
|
||||
sessions.set(token, email);
|
||||
send(res, status, {email}, {'Set-Cookie': `sid=${token}; Path=/; HttpOnly; SameSite=Strict`});
|
||||
}
|
||||
|
||||
const EMAIL_RE = /^[^\s@]+@[^\s@]+\.[^\s@]+$/;
|
||||
|
||||
const server = http.createServer(async (req, res) => {
|
||||
try {
|
||||
const url = new URL(req.url, 'http://x');
|
||||
const path = url.pathname;
|
||||
const m = req.method;
|
||||
|
||||
if (path === '/' && (m === 'GET' || m === 'HEAD')) {
|
||||
res.writeHead(200, {'Content-Type': 'text/html; charset=utf-8', 'Cache-Control': 'no-store'});
|
||||
return res.end(m === 'HEAD' ? undefined : PAGE);
|
||||
}
|
||||
|
||||
if (path === '/api/signup' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.email !== 'string' || typeof b.password !== 'string') return send(res, 400, {error: 'bad request'});
|
||||
const email = b.email.trim().toLowerCase();
|
||||
if (!EMAIL_RE.test(email) || email.length > 254 || b.password.length < 8 || b.password.length > 1024)
|
||||
return send(res, 400, {error: 'bad email or password'});
|
||||
if (users.has(email)) return send(res, 409, {error: 'email used'});
|
||||
const salt = crypto.randomBytes(16);
|
||||
users.set(email, {salt, hash: hashPw(b.password, salt)});
|
||||
return startSession(res, 201, email);
|
||||
}
|
||||
|
||||
if (path === '/api/login' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.email !== 'string' || typeof b.password !== 'string' || b.password.length > 1024)
|
||||
return send(res, 401, {error: 'refused'});
|
||||
const u = users.get(b.email.trim().toLowerCase());
|
||||
if (!u) { hashPw(b.password, Buffer.alloc(16)); return send(res, 401, {error: 'refused'}); }
|
||||
if (!crypto.timingSafeEqual(hashPw(b.password, u.salt), u.hash)) return send(res, 401, {error: 'refused'});
|
||||
return startSession(res, 200, b.email.trim().toLowerCase());
|
||||
}
|
||||
|
||||
if (path === '/api/logout' && m === 'POST') {
|
||||
const t = cookieToken(req);
|
||||
if (t) sessions.delete(t);
|
||||
return send(res, 204, undefined, {'Set-Cookie': 'sid=; Path=/; HttpOnly; SameSite=Strict; Max-Age=0'});
|
||||
}
|
||||
|
||||
if (path === '/api/notes' || path.startsWith('/api/notes/')) {
|
||||
const email = currentUser(req);
|
||||
if (!email) return send(res, 401, {error: 'unauthorized'});
|
||||
const mine = notes.of(email);
|
||||
if (path === '/api/notes' && m === 'GET') return send(res, 200, mine.list());
|
||||
if (path === '/api/notes' && m === 'POST') {
|
||||
const b = await readBody(req);
|
||||
if (!b || typeof b.text !== 'string' || b.text.length < 1 || b.text.length > 500) return send(res, 400, {error: 'bad text'});
|
||||
return send(res, 201, mine.add(b.text));
|
||||
}
|
||||
const dm = path.match(/^\/api\/notes\/(\d+)$/);
|
||||
if (dm && m === 'DELETE') {
|
||||
const id = Number(dm[1]);
|
||||
if (!mine.remove(id)) return send(res, 404, {error: 'not found'});
|
||||
return send(res, 204);
|
||||
}
|
||||
if (dm) return send(res, 404, {error: 'not found'});
|
||||
}
|
||||
send(res, 404, {error: 'not found'});
|
||||
} catch (e) {
|
||||
send(res, 500, {error: 'server error'});
|
||||
}
|
||||
});
|
||||
|
||||
server.listen(3000, '0.0.0.0');
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
#!/bin/sh
|
||||
cd "$(dirname "$0")" && exec node server.js
|
||||
Executable
+101
@@ -0,0 +1,101 @@
|
||||
#!/usr/bin/env bash
|
||||
# Tests for ygg-repair-path.sh: does a run's ledger show one finding raised, fixed and closed (B-109).
|
||||
# scripts/tests/ygg-repair-path.test.sh
|
||||
# The ledgers are made up here, with the field names of gungnir event.go and the order of odin's loop
|
||||
# (internal/engine/loop.go): the implementer's node_finished on the loop node, the reviewer's on node-reviewer.
|
||||
# shellcheck disable=SC2329 # check calls has and hasnt by name
|
||||
set -uo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
S="$HERE/.."
|
||||
T="$(mktemp -d)"
|
||||
trap 'rm -rf -- "$T"' EXIT
|
||||
fail=0
|
||||
check() {
|
||||
local d="$1"; shift
|
||||
if "$@"; then echo "ok $d"; else echo "FAIL $d"; fail=1; [ -z "${VERBOSE:-}" ] || sed 's/^/ | /' "$T/out"; fi
|
||||
}
|
||||
run() { "$S/ygg-repair-path.sh" "$@" >"$T/out" 2>&1; rc=$?; }
|
||||
has() { grep -q -F -- "$1" "$T/out"; }
|
||||
hasnt() { ! grep -q -F -- "$1" "$T/out"; }
|
||||
|
||||
# One ledger line: event, node, attempt, and optionally a finding id and its source.
|
||||
ev() {
|
||||
local name="$1" node="$2" attempt="$3" id="${4:-}" src="${5:-review}"
|
||||
if [ -n "$id" ]; then
|
||||
printf '{"schema":"1","seq":0,"ygg.run.id":"r","event":"%s","ygg.node.id":"%s","ygg.attempt":%s,"ygg.finding.id":"%s","ygg.finding.source":"%s","ygg.note":"server.js:3: the port is 8080","prev":""}\n' \
|
||||
"$name" "$node" "$attempt" "$id" "$src"
|
||||
else
|
||||
printf '{"schema":"1","seq":0,"ygg.run.id":"r","event":"%s","ygg.node.id":"%s","ygg.attempt":%s,"gen_ai.agent.name":"implementer","prev":""}\n' \
|
||||
"$name" "$node" "$attempt"
|
||||
fi
|
||||
}
|
||||
|
||||
# --- the whole repair path: raised in attempt 1, the implementer runs in attempt 2, closed in attempt 2
|
||||
{
|
||||
ev run_started "" 0
|
||||
ev node_finished build 1
|
||||
ev node_finished build-review 1
|
||||
ev finding_raised build 1 F1
|
||||
ev loop_attempt build 2
|
||||
ev node_finished build 2
|
||||
ev node_finished build-review 2
|
||||
ev finding_closed build 2 F1
|
||||
echo 'not json'
|
||||
} >"$T/full.jsonl"
|
||||
run "$T/full.jsonl"
|
||||
check "the whole path exits 0" test "$rc" -eq 0
|
||||
check "it says raised, fixed, closed" has "repair path: finding raised, fixed, closed"
|
||||
check "it names the finding and its source" has "finding F1 from review"
|
||||
check "it names the attempts" has "raised in attempt 1, closed in attempt 2"
|
||||
check "it quotes the finding" has "server.js:3: the port is 8080"
|
||||
|
||||
# --- no finding at all
|
||||
{ ev node_finished build 1; ev node_finished build-review 1; } >"$T/none.jsonl"
|
||||
run "$T/none.jsonl"
|
||||
check "no finding exits 1" test "$rc" -eq 1
|
||||
check "no finding names the missing step raised" has "repair path: missing step raised: no finding was raised"
|
||||
|
||||
# --- raised, but no implementer session after it (only the reviewer's line, and one before the raise)
|
||||
{ ev node_finished build 1; ev finding_raised build 1 F1; ev node_finished build-review 1; } >"$T/unfixed.jsonl"
|
||||
run "$T/unfixed.jsonl"
|
||||
check "raised but not fixed exits 1" test "$rc" -eq 1
|
||||
check "it names the missing step fixed" has "repair path: missing step fixed: finding F1 was raised in attempt 1, but no implementer session of node build finished after it"
|
||||
|
||||
# --- raised and an implementer session after it, but never closed
|
||||
{ ev finding_raised build 1 F1; ev node_finished build 2; ev node_finished build-review 2; } >"$T/open.jsonl"
|
||||
run "$T/open.jsonl"
|
||||
check "raised and fixed but not closed exits 1" test "$rc" -eq 1
|
||||
check "it names the missing step closed" has "repair path: missing step closed: finding F1 was raised and an implementer session finished after it, but it was not closed"
|
||||
check "it does not claim an overrule" hasnt "overruled"
|
||||
|
||||
# --- overruled by a check: not closed by its source
|
||||
{ ev finding_raised build 1 F1; ev node_finished build 2; ev finding_overruled build 2 F1; } >"$T/overruled.jsonl"
|
||||
run "$T/overruled.jsonl"
|
||||
check "an overruled finding exits 1" test "$rc" -eq 1
|
||||
check "it says the finding was overruled, not closed" has "it was overruled by a check, not closed"
|
||||
|
||||
# --- two findings: one stays open, the other completes the path; the counts are printed
|
||||
{
|
||||
ev finding_raised build 1 F1; ev finding_raised build 1 F2 check:unit
|
||||
ev node_finished build 2; ev finding_closed build 2 F2
|
||||
} >"$T/two.jsonl"
|
||||
run "$T/two.jsonl"
|
||||
check "one complete finding of two exits 0" test "$rc" -eq 0
|
||||
check "it names the finding that completed the path" has "finding F2 from check:unit"
|
||||
check "it counts the findings" has "2 finding(s) raised, 1 closed"
|
||||
|
||||
# --- a closed line before the fix does not count
|
||||
{ ev finding_raised build 1 F1; ev finding_closed build 1 F1; ev node_finished build 2; } >"$T/early.jsonl"
|
||||
run "$T/early.jsonl"
|
||||
check "a close before the fix is not the repair path" test "$rc" -eq 1
|
||||
check "and the missing step is closed" has "missing step closed"
|
||||
|
||||
# --- usage
|
||||
run
|
||||
check "no argument exits 2" test "$rc" -eq 2
|
||||
run "$T/no-such.jsonl"
|
||||
check "a missing ledger exits 2" test "$rc" -eq 2
|
||||
check "and says so" has "no ledger at"
|
||||
|
||||
[ "$fail" -eq 0 ] && echo "all ygg-repair-path tests passed"
|
||||
exit "$fail"
|
||||
@@ -0,0 +1,210 @@
|
||||
#!/usr/bin/env bash
|
||||
# Tests for ygg-reviewer-bench.sh (B-108). scripts/tests/ygg-reviewer-bench.test.sh
|
||||
# A fake opencode (YGG_REVIEWER_BENCH_OPENCODE) prints canned sessions: no model, no network, no money.
|
||||
# Each check is a condition string run by eval: $rc and the helpers are read there, so the linter cannot
|
||||
# see them used.
|
||||
# shellcheck disable=SC2016,SC2034,SC2329
|
||||
set -uo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
S="$HERE/.."
|
||||
B="$S/ygg-reviewer-bench.sh"
|
||||
F="$S/testdata/reviewer-bench"
|
||||
T="$(mktemp -d)"
|
||||
trap 'rm -rf -- "$T"' EXIT
|
||||
fail=0
|
||||
check() {
|
||||
local d="$1"; shift
|
||||
if "$@"; then echo "ok $d"; else echo "FAIL $d"; fail=1; [ -z "${VERBOSE:-}" ] || sed 's/^/ | /' "$T/out"; fi
|
||||
}
|
||||
run() { "$@" >"$T/out" 2>&1; rc=$?; }
|
||||
has() { grep -q -F -- "$1" "$T/out"; }
|
||||
hasnt() { ! grep -q -F -- "$1" "$T/out"; }
|
||||
calls() { cat "$T/calls/count" 2>/dev/null || echo 0; }
|
||||
envhas() { grep -qx -F -- "$1" "$T/calls/1.env"; }
|
||||
export HOME="$T/home" TMPDIR="$T/tmp"
|
||||
mkdir -p "$HOME" "$TMPDIR"
|
||||
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1
|
||||
unset OPENROUTER_API_KEY ANTHROPIC_API_KEY CLAUDE_CODE_OAUTH_TOKEN
|
||||
|
||||
# A made-up odin with the reviewer prompt, and a made-up mimir with the brief.
|
||||
O="$T/odin"; mkdir -p "$O/testdata/smoke-grade/prompts"
|
||||
printf 'Review the work in this repository. REVIEW-PROMPT-MARK\n' >"$O/testdata/smoke-grade/prompts/review.md"
|
||||
M="$T/mimir"; mkdir -p "$M/smoke-notes"
|
||||
printf '# Notes: product brief\n\nBRIEF-MARK\n' >"$M/smoke-notes/instruction.md"
|
||||
common=(--odin "$O" --tasks "$M")
|
||||
|
||||
# The fake opencode: it keeps what it was given, writes a file into its folder, prints the canned session
|
||||
# of its number and exits with the canned code (0 by default).
|
||||
mkdir -p "$T/bin"
|
||||
cat >"$T/bin/opencode" <<EOF
|
||||
#!/usr/bin/env bash
|
||||
n=\$((\$(cat "$T/calls/count" 2>/dev/null || echo 0) + 1))
|
||||
echo "\$n" >"$T/calls/count"
|
||||
printf '%s\n' "\$@" >"$T/calls/\$n.args"
|
||||
env >"$T/calls/\$n.env"
|
||||
pwd >"$T/calls/\$n.pwd"
|
||||
ls -A >"$T/calls/\$n.ls"
|
||||
cat >"$T/calls/\$n.stdin"
|
||||
echo planted >"written-by-session-\$n"
|
||||
cat "$T/answers/\$n.jsonl" 2>/dev/null
|
||||
exit "\$(cat "$T/answers/\$n.exit" 2>/dev/null || echo 0)"
|
||||
EOF
|
||||
chmod +x "$T/bin/opencode"
|
||||
export YGG_REVIEWER_BENCH_OPENCODE="$T/bin/opencode"
|
||||
fresh() { rm -rf -- "$T/calls" "$T/answers"; mkdir -p "$T/calls" "$T/answers"; }
|
||||
|
||||
# canned N TEXT [COST]: session N reads in a first step (with a sentence that is not the answer) and answers
|
||||
# TEXT in a second. In all: 3000 new tokens in, 7000 from the cache, 100 out and 400 of reasoning, COST dollars.
|
||||
canned() {
|
||||
python3 -I - "$T/answers/$1.jsonl" "$2" "${3:-0.004}" <<'PY'
|
||||
import json, sys
|
||||
path, text, cost = sys.argv[1], sys.argv[2], float(sys.argv[3])
|
||||
def fin(reason, i, o, r, cr, c):
|
||||
return {"type": "step_finish", "part": {"type": "step-finish", "reason": reason, "cost": c,
|
||||
"tokens": {"total": i + o + r + cr, "input": i, "output": o, "reasoning": r, "cache": {"write": 0, "read": cr}}}}
|
||||
ev = [{"type": "step_start", "part": {"type": "step-start"}},
|
||||
{"type": "text", "part": {"type": "text", "text": "I will read start.sh first."}},
|
||||
{"type": "tool_use", "part": {"type": "tool", "tool": "read", "state": {"status": "completed", "input": {"filePath": "start.sh"}}}},
|
||||
fin("tool-calls", 1000, 50, 0, 0, cost / 2),
|
||||
{"type": "step_start", "part": {"type": "step-start"}},
|
||||
{"type": "text", "part": {"type": "text", "text": text}},
|
||||
fin("stop", 2000, 50, 400, 7000, cost / 2)]
|
||||
with open(path, "w") as f:
|
||||
for e in ev:
|
||||
f.write(json.dumps(e) + "\n")
|
||||
PY
|
||||
}
|
||||
finding() { # finding ID LOCATION CLAIM [SOURCE] [STATUS]: one finding as JSON
|
||||
printf '{"id": "%s", "source": "%s", "severity": "major", "location": "%s", "claim": "%s", "status": "%s"}' \
|
||||
"$1" "${4:-review}" "$2" "$3" "${5:-open}"
|
||||
}
|
||||
fence() { printf '```json\n%s\n```' "$1"; }
|
||||
P4RE='owner|another (user|person|account)|other (users?|person|people|accounts?)|any (user|person)|not theirs|not the caller|idor'
|
||||
|
||||
# --- the arguments ---
|
||||
run bash "$B"
|
||||
check "no arguments: usage, exit 2" eval '[ $rc = 2 ] && has "usage:"'
|
||||
run bash "$B" --model qwen/qwen3.8-27b --leaf "$F/P4" --sessions 0 "${common[@]}"
|
||||
check "zero sessions is refused" eval '[ $rc = 2 ] && has "--sessions"'
|
||||
run bash "$B" --model qwen/qwen3.8-27b --leaf "$T/no-such-leaf" --sessions 1 "${common[@]}"
|
||||
check "a missing leaf is refused" eval '[ $rc = 2 ] && has "no-such-leaf"'
|
||||
run bash "$B" --model qwen/qwen3.8-27b --leaf "$F/P4" --sessions 1 --fault '(' "${common[@]}"
|
||||
check "a fault pattern that is not a regular expression is refused" eval '[ $rc = 2 ] && has "--fault"'
|
||||
run bash "$B" --model qwen/qwen3.8-27b --leaf "$F/P4" --sessions 1 --effort max "${common[@]}"
|
||||
check "an unknown effort is refused" eval '[ $rc = 2 ] && has "--effort"'
|
||||
run bash "$B" --model 'qwen/x"y' --leaf "$F/P4" --sessions 1 "${common[@]}"
|
||||
check "a model name with a quote is refused" eval '[ $rc = 2 ] && has "--model"'
|
||||
run bash "$B" --model qwen/qwen3.8-27b --leaf "$F/P4" --sessions 1 --odin "$T/nowhere" --tasks "$M"
|
||||
check "an odin without the smoke reviewer prompt is refused" eval '[ $rc = 2 ] && has "review.md"'
|
||||
|
||||
# --- the dry run: prints the commands, runs nothing, needs no key ---
|
||||
fresh
|
||||
run bash "$B" --dry-run --model qwen/qwen3.8-27b --leaf "$F/P4" --sessions 8 --fault "$P4RE" --out "$T/dry" "${common[@]}"
|
||||
check "dry run: exit 0, and opencode never runs" eval '[ $rc = 0 ] && [ "$(calls)" = 0 ] && has "spends nothing"'
|
||||
check "dry run: the command is opencode run, standalone, JSON events, the OpenRouter model" \
|
||||
eval 'has "run --standalone --format json -m openrouter/qwen/qwen3.8-27b"'
|
||||
check "dry run: strict provider settings (no data kept, zero retention, no fallback), effort high" \
|
||||
eval 'has "\"data_collection\":\"deny\"" && has "\"zdr\":true" && has "\"allow_fallbacks\":false" && has "\"effort\":\"high\""'
|
||||
check "dry run: read-only permissions, and the graph's 10 turns" \
|
||||
eval 'has "\"edit\":\"deny\"" && has "\"bash\":\"deny\"" && has "\"external_directory\":\"deny\"" && has "\"steps\":10"'
|
||||
check "dry run: the key is a reference to the environment, never a value" eval 'has "{env:OPENROUTER_API_KEY}"'
|
||||
check "dry run: XDG_DATA_HOME is set, outside HOME" \
|
||||
eval 'has "XDG_DATA_HOME=" && ! grep -q "XDG_DATA_HOME=[^ ]*$HOME" "$T/out"'
|
||||
check "dry run: the prompt is written, from odin's review.md, the brief and the report format" \
|
||||
eval 'grep -q REVIEW-PROMPT-MARK "$T/dry/prompt.md" && grep -q "^## Brief$" "$T/dry/prompt.md" && grep -q BRIEF-MARK "$T/dry/prompt.md" && grep -q "Your id is review" "$T/dry/prompt.md"'
|
||||
run bash "$B" --dry-run --model openrouter/qwen/qwen3.8-27b --leaf "$F/P4" --sessions 1 --effort off --out "$T/dry2" "${common[@]}"
|
||||
check "dry run: an openrouter/ prefix is accepted, and effort off names no effort" \
|
||||
eval '[ $rc = 0 ] && has "-m openrouter/qwen/qwen3.8-27b" && hasnt "openrouter/openrouter" && hasnt "\"effort\""'
|
||||
|
||||
# --- a real run needs the key ---
|
||||
fresh
|
||||
run bash "$B" --model qwen/qwen3.8-27b --leaf "$F/P4" --sessions 1 --out "$T/nokey" "${common[@]}"
|
||||
check "no OPENROUTER_API_KEY: refused before any session" eval '[ $rc = 2 ] && has "OPENROUTER_API_KEY" && [ "$(calls)" = 0 ]'
|
||||
|
||||
# --- five sessions with the fake: the table ---
|
||||
fresh
|
||||
canned 1 "The work breaks the brief.
|
||||
|
||||
$(fence "{\"findings\": [$(finding cross-user-delete notes.js:18 "remove deletes any note by id: one person's DELETE reaches another user's note")]}")"
|
||||
canned 2 $'\n\n{"findings": []}'
|
||||
canned 3 "What would you like help with?"
|
||||
canned 4 "$(fence "{\"findings\": [$(finding find-index server.js:174 "findIndex is not a JavaScript method")]}")"
|
||||
canned 5 "$(fence '{"findings": []}')
|
||||
That is all."
|
||||
export OPENROUTER_API_KEY=made-up-key ANTHROPIC_API_KEY=made-up-a CLAUDE_CODE_OAUTH_TOKEN=made-up-b
|
||||
# Host settings that must not reach a session: OpenCode variables and anything else of the caller's shell.
|
||||
export OPENCODE_PERMISSION=x OPENCODE_CONFIG=/x HOST_ONLY_VAR=y
|
||||
run bash "$B" --model qwen/qwen3.8-27b --leaf "$F/P4" --sessions 5 --fault "$P4RE" --out "$T/five" "${common[@]}"
|
||||
check "five sessions: exit 0 and five calls" eval '[ $rc = 0 ] && [ "$(calls)" = 5 ]'
|
||||
check "the table: 3 valid reports, findings per session, 1 fault found, average tokens and cost" \
|
||||
eval 'has "| P4 | qwen/qwen3.8-27b | 5 | 3 | 1, 0, -, 1, - | 1 | 10.0k (3.0k) | 0.5k | \$0.004 |"'
|
||||
check "the table has the journal's columns" \
|
||||
eval 'has "| Leaf | Model | Sessions | Valid reports | Findings per session | Fault found | Tokens in (new) | Tokens out | Cost per session |"'
|
||||
check "each session says why its report is not valid" \
|
||||
eval 'has "session 3: not a valid report: the answer does not end with a JSON document" && has "session 5: not a valid report: text after the report"'
|
||||
check "the claims are printed, and the one that matches the fault is marked" \
|
||||
eval 'has "fault found" && has "reaches another user" && has "findIndex is not a JavaScript method"'
|
||||
check "the summary and each session's events are kept in the out folder" \
|
||||
eval 'grep -q "| P4 |" "$T/five/summary.md" && [ -s "$T/five/session-1.jsonl" ] && [ -s "$T/five/session-5.jsonl" ]'
|
||||
check "the runner gets the command line: run, standalone, JSON events, the model" \
|
||||
eval '[ "$(cat "$T/calls/1.args")" = "$(printf "run\n--standalone\n--format\njson\n-m\nopenrouter/qwen/qwen3.8-27b")" ]'
|
||||
check "the prompt goes on stdin: odin's review.md, the brief, the report format, the reviewer id and the passed check" \
|
||||
eval 'grep -q REVIEW-PROMPT-MARK "$T/calls/1.stdin" && grep -q BRIEF-MARK "$T/calls/1.stdin" && grep -q "end your answer with the document" "$T/calls/1.stdin" && grep -q "Your id is review" "$T/calls/1.stdin" && grep -q "The checks that passed on this work: unit." "$T/calls/1.stdin"'
|
||||
check "the session runs in a copy of the leaf, never in the leaf itself" \
|
||||
eval '[ "$(cat "$T/calls/1.pwd")" != "$(cd "$F/P4" && pwd)" ] && grep -qx notes.js "$T/calls/1.ls" && grep -qx server.js "$T/calls/1.ls" && ! ls "$F/P4" | grep -q written-by'
|
||||
check "each session gets a fresh copy" eval '[ -s "$T/calls/2.ls" ] && ! grep -q written-by "$T/calls/2.ls"'
|
||||
check "the copy is a git repository with one commit, so OpenCode's project is the copy" \
|
||||
eval 'grep -qx .git "$T/calls/1.ls"'
|
||||
check "the OpenRouter key reaches the runner; the Anthropic credentials do not" \
|
||||
eval 'grep -qx OPENROUTER_API_KEY=made-up-key "$T/calls/1.env" && ! grep -Eq "^(ANTHROPIC_API_KEY|CLAUDE_CODE_OAUTH_TOKEN)=" "$T/calls/1.env"'
|
||||
check "OpenCode reads no project or Claude Code config and does not update, share or fetch models" \
|
||||
eval 'envhas OPENCODE_DISABLE_PROJECT_CONFIG=1 && envhas OPENCODE_DISABLE_CLAUDE_CODE=1 && envhas OPENCODE_DISABLE_EXTERNAL_SKILLS=1 && envhas OPENCODE_DISABLE_AUTOUPDATE=true && envhas OPENCODE_DISABLE_SHARE=true && envhas OPENCODE_DISABLE_MODELS_FETCH=true'
|
||||
d1="$(sed -n 's/^XDG_DATA_HOME=//p' "$T/calls/1.env")"
|
||||
d2="$(sed -n 's/^XDG_DATA_HOME=//p' "$T/calls/2.env")"
|
||||
c1="$(sed -n 's/^XDG_CONFIG_HOME=//p' "$T/calls/1.env")"
|
||||
c2="$(sed -n 's/^XDG_CONFIG_HOME=//p' "$T/calls/2.env")"
|
||||
check "no OpenCode variable and no other variable of the caller's shell reaches the session" \
|
||||
eval '[ -s "$T/calls/1.env" ] && ! grep -Eq "^(OPENCODE_PERMISSION|OPENCODE_CONFIG|HOST_ONLY_VAR)=" "$T/calls/1.env"'
|
||||
check "each session has its own XDG_CONFIG_HOME, in the work folder, so no global OpenCode config applies" \
|
||||
eval '[ -n "$c1" ] && [ "$c1" != "$c2" ] && case "$c1" in "$TMPDIR"/ygg-reviewer-bench-work.*/config-1) true ;; *) false ;; esac'
|
||||
check "each session has its own XDG_DATA_HOME, outside HOME" \
|
||||
eval '[ -n "$d1" ] && [ "$d1" != "$d2" ] && case "$d1" in "$HOME"/*) false ;; *) true ;; esac'
|
||||
check "the config reaches the runner with the strict provider settings and the read-only permissions" \
|
||||
eval 'grep -q "^OPENCODE_CONFIG_CONTENT=.*\"zdr\":true" "$T/calls/1.env" && grep -q "^OPENCODE_CONFIG_CONTENT=.*\"question\":\"deny\"" "$T/calls/1.env"'
|
||||
|
||||
# --- a model that OpenRouter reports no cost for ---
|
||||
fresh
|
||||
canned 1 '{"findings": []}' 0
|
||||
canned 2 '{"findings": []}' 0
|
||||
run bash "$B" --model anthropic/claude-haiku-5.5 --leaf "$F/P0" --sessions 2 --out "$T/zero" "${common[@]}"
|
||||
check "cost 0 with tokens prints as unknown, never \$0.00" \
|
||||
eval '[ $rc = 0 ] && has "| P0 | anthropic/claude-haiku-5.5 | 2 | 2 | 0, 0 | - | 10.0k (3.0k) | 0.5k | unknown |" && hasnt "\$0.00"'
|
||||
|
||||
# --- what counts as a valid report: the engine's rules ---
|
||||
fresh
|
||||
canned 1 "$(fence '{"findings": []}') and $(fence '{"findings": []}')"
|
||||
canned 2 "{\"findings\": [$(finding a server.js:1 "x" reviewer)]}"
|
||||
canned 3 "{\"findings\": [$(finding a server.js "x")]}"
|
||||
canned 4 "{\"findings\": [$(finding a server.js:1 "x" review fixed)]}"
|
||||
canned 5 "{\"findings\": [$(finding a server.js:1 "x"), $(finding a server.js:2 "y")]}"
|
||||
canned 6 'An example: {"findings": []} and my answer: {"findings": []}'
|
||||
canned 7 "Here is my report: {\"findings\": [$(finding port start.sh:2 "the server listens on 8080")]}"
|
||||
canned 8 '{"findings": []}'
|
||||
echo 1 >"$T/answers/8.exit"
|
||||
canned 9 "{\"findings\": [$(finding check-unit server.js:1 "x")]}"
|
||||
canned 10 '{"findings": [], "summary": "fine"}'
|
||||
run bash "$B" --model qwen/qwen3.8-27b --leaf "$F/P2" --sessions 10 --out "$T/rules" "${common[@]}"
|
||||
check "only the bare report after prose is valid; with no --fault the fault column is -" \
|
||||
eval '[ $rc = 0 ] && has "| P2 | qwen/qwen3.8-27b | 10 | 1 | -, -, -, -, -, -, 1, -, -, - | - |"'
|
||||
check "the reasons: two fences, a source, a location, a closed finding, an id twice, two documents, the exit code, a check id, an unknown field" \
|
||||
eval 'has "session 1: not a valid report: more than one code fence" && has "session 2: not a valid report" && has "session 3: not a valid report" && has "session 4: not a valid report" && has "session 5: not a valid report" && has "session 6: not a valid report: more than one document" && has "session 8: not a valid report: opencode exited 1" && has "session 9: not a valid report" && has "session 10: not a valid report"'
|
||||
|
||||
# --- the fixtures ---
|
||||
lines() { diff "$1" "$2" | grep -c '^[<>]'; }
|
||||
check "P1 to P3 are P0 with one line changed: the start.sh typo, the port, the order" \
|
||||
eval '[ "$(lines "$F/P0/start.sh" "$F/P1/start.sh")" = 2 ] && grep -q "node servr.js" "$F/P1/start.sh" && [ "$(lines "$F/P0/server.js" "$F/P2/server.js")" = 2 ] && grep -q "listen(8080" "$F/P2/server.js" && [ "$(lines "$F/P0/server.js" "$F/P3/server.js")" = 2 ]'
|
||||
check "P4 reaches notes only through notes.of(email), and its remove ignores the owner" \
|
||||
eval 'grep -q "notes.of(email)" "$F/P4/server.js" && grep -q "return all.delete(id);" "$F/P4/notes.js" && ! grep -q "^const notes = new Map" "$F/P4/server.js"'
|
||||
check "no fixture holds a .git" eval '[ -z "$(find "$F" -name .git)" ]'
|
||||
check "the P4 fault pattern of the script's header is the one tested here" eval 'grep -qF -- "$P4RE" "$B"'
|
||||
exit $fail
|
||||
@@ -0,0 +1,250 @@
|
||||
#!/usr/bin/env bash
|
||||
# Tests for ygg-smoke-real.sh: which `ygg` it says it runs (B-111), the run from a planted-fault leaf (B-109)
|
||||
# and another implementer (B-110). scripts/tests/ygg-smoke-real.test.sh
|
||||
# Each case uses a fake `ygg`, a fake `go` and made-up credentials: no model, no engine, no money. Most cases are
|
||||
# --dry-run. The cases of the run's tail run without it, only through run_real, which refuses unless YGG_CMD is a
|
||||
# fake in this test's temp folder.
|
||||
# shellcheck disable=SC2329,SC2016 # check calls has and hasnt by name; the $ in the patterns is literal
|
||||
set -uo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
S="$HERE/.."
|
||||
T="$(mktemp -d)"
|
||||
trap 'rm -rf -- "$T"' EXIT
|
||||
fail=0
|
||||
check() {
|
||||
local d="$1"; shift
|
||||
if "$@"; then echo "ok $d"; else echo "FAIL $d"; fail=1; [ -z "${VERBOSE:-}" ] || sed 's/^/ | /' "$T/out"; fi
|
||||
}
|
||||
# The script names its folder by the second, so each run waits for the next second.
|
||||
run() { sleep 1; "$@" >"$T/out" 2>&1; rc=$?; }
|
||||
has() { grep -q -F -- "$1" "$T/out"; }
|
||||
hasnt() { ! grep -q -F -- "$1" "$T/out"; }
|
||||
export HOME="$T/home"; mkdir -p "$HOME"
|
||||
export YGG_ENGINE_CHECK=0 # the engine environment has its own test (ygg-engine.test.sh)
|
||||
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1
|
||||
export YGG_BIND="$T/bind"; mkdir -p "$YGG_BIND"
|
||||
|
||||
# A made-up mimir and a made-up odin (a git repo, so the build case can name its commit).
|
||||
M="$T/mimir"; mkdir -p "$M/smoke-notes/sample-suite/tests"
|
||||
printf '[metadata]\nsuite = "v1"\n' >"$M/smoke-notes/task.toml"
|
||||
printf '# brief\n' >"$M/smoke-notes/instruction.md"
|
||||
printf '{}\n' >"$M/smoke-notes/sample-suite/tests/manifest.json"
|
||||
printf '#!/bin/bash\n' >"$M/smoke-notes/sample-suite/tests/test.sh"
|
||||
O="$T/odin"; mkdir -p "$O/testdata/smoke-grade/templates/notes"
|
||||
printf 'name: smoke-grade\nleaves:\n repos/notes: { description: the notes product, template: templates/notes, first_commit: [LICENSE] }\n' \
|
||||
>"$O/testdata/smoke-grade/pack.yaml"
|
||||
printf 'licence\n' >"$O/testdata/smoke-grade/templates/notes/LICENSE"
|
||||
printf 'old\n' >"$O/testdata/smoke-grade/templates/notes/OLD.md"
|
||||
printf 'roles:\n implementer: { agent: claude-code, model: sonnet }\n reviewer: { agent: opencode, model: openrouter/qwen/qwen3.8-27b }\n' \
|
||||
>"$O/testdata/smoke-grade/factory.real.example.yaml"
|
||||
printf 'sandbox:\n engine: @ENGINE@\n mounts:\n - { host: @FOLDER@, engine: @BIND@ }\ngrader: { tasks_dir: @TASKS@, suite_dir: @SUITES@ }\n' \
|
||||
>>"$O/testdata/smoke-grade/factory.real.example.yaml"
|
||||
git -C "$O" init -q -b main
|
||||
git -C "$O" add testdata
|
||||
git -C "$O" -c user.name=t -c user.email=[email protected] commit -q -m odin
|
||||
odin_sha="$(git -C "$O" rev-parse --short HEAD)"
|
||||
D="$T/it"; mkdir -p "$D"
|
||||
printf 'engine: unix://%s/engine.sock\n' "$T" >"$D/factory.yaml"
|
||||
printf 'CLAUDE_CODE_OAUTH_TOKEN=made-up-a\nOPENROUTER_API_KEY=made-up-b\n' >"$T/factory.env"
|
||||
rargs=(--folder "$D" --tasks "$M" --odin "$O" --env "$T/factory.env" --dry-run)
|
||||
|
||||
# A fake ygg that accepts every verb, and a fake go whose build writes that same fake to the -o path.
|
||||
mkdir -p "$T/pathbin" "$T/other" "$T/gobin"
|
||||
printf '#!/usr/bin/env bash\nexit 0\n' >"$T/pathbin/ygg"
|
||||
cp "$T/pathbin/ygg" "$T/other/ygg-custom"
|
||||
cat >"$T/gobin/go" <<'EOF2'
|
||||
#!/usr/bin/env bash
|
||||
[ "$1" = build ] && [ "$2" = -o ] && { printf '#!/usr/bin/env bash\nexit 0\n' >"$3"; chmod +x "$3"; exit 0; }
|
||||
exit 1
|
||||
EOF2
|
||||
chmod +x "$T/pathbin/ygg" "$T/other/ygg-custom" "$T/gobin/go"
|
||||
touch -d '2026-01-02 03:04:05' "$T/pathbin/ygg" "$T/other/ygg-custom"
|
||||
BASEPATH="$PATH"
|
||||
|
||||
# --- a ygg from PATH
|
||||
PATH="$T/pathbin:$BASEPATH" run "$S/ygg-smoke-real.sh" "${rargs[@]}"
|
||||
check "a dry run with a ygg on PATH exits 0" test "$rc" -eq 0
|
||||
check "it says the path of the ygg that runs" has "smoke-real: ygg: $T/pathbin/ygg"
|
||||
check "it says the mtime of that binary" has "modified 2026-01-02"
|
||||
check "it says no commit is known for a binary it did not build" has "odin commit unknown"
|
||||
check "the line comes before the doctor" bash -c "grep -n 'smoke-real: ygg:' '$T/out' | head -n 1 | cut -d: -f1 | { read -r a; b=\$(grep -n 'ygg doctor on that config' '$T/out' | cut -d: -f1); [ \"\$a\" -lt \"\$b\" ]; }"
|
||||
|
||||
# --- YGG_CMD
|
||||
PATH="$T/pathbin:$BASEPATH" YGG_CMD="$T/other/ygg-custom" run "$S/ygg-smoke-real.sh" "${rargs[@]}"
|
||||
check "YGG_CMD wins over PATH and the line names it" has "smoke-real: ygg: $T/other/ygg-custom"
|
||||
check "and does not name the PATH one" hasnt "$T/pathbin/ygg"
|
||||
|
||||
# --- no ygg: built from --odin, clean tree
|
||||
PATH="$T/gobin:$BASEPATH" run "$S/ygg-smoke-real.sh" "${rargs[@]}"
|
||||
check "a build from --odin exits 0" test "$rc" -eq 0
|
||||
check "the line names the built binary" has "/bin/ygg"
|
||||
check "the line names the odin commit" has "built from $O at commit $odin_sha"
|
||||
check "a clean tree is said to be clean" has "tree clean"
|
||||
|
||||
# --- no ygg: built from --odin, dirty tree
|
||||
printf 'x\n' >"$O/untracked.txt"
|
||||
PATH="$T/gobin:$BASEPATH" run "$S/ygg-smoke-real.sh" "${rargs[@]}"
|
||||
check "a dirty odin tree is said to be dirty" has "tree DIRTY"
|
||||
|
||||
# The run's folder, from the line "smoke-real: made DIR".
|
||||
work() { sed -n 's/^smoke-real: made //p' "$T/out" | head -n 1; }
|
||||
export PATH="$T/pathbin:$BASEPATH"
|
||||
|
||||
# --- the default run: odin's pack, the empty template, the example's implementer (B-109, B-110)
|
||||
run "$S/ygg-smoke-real.sh" "${rargs[@]}"
|
||||
W="$(work)"
|
||||
check "a default run uses odin's pack as the graph" has "--graph '$O/testdata/smoke-grade'"
|
||||
check "a default run copies no pack" test ! -e "$W/pack"
|
||||
check "a default run's start.sh runs the leaf's start.sh as a program" grep -q -F 'exec ./start.sh' "$W/project/start.sh"
|
||||
check "a default run keeps the example's implementer" grep -q -F 'implementer: { agent: claude-code, model: sonnet }' "$W/factory.yaml"
|
||||
check "a default run prints the implementer" has "smoke-real: the implementer: { agent: claude-code, model: sonnet }"
|
||||
check "a default run says nothing about the repair path" hasnt "repair"
|
||||
|
||||
# --- --clip: the whole report.txt goes to the clipboard after the run, with clip.exe (B-111, B-109)
|
||||
mkdir -p "$T/clipbin"; printf '#!/bin/sh\ncat >/dev/null\n' >"$T/clipbin/clip.exe"; chmod +x "$T/clipbin/clip.exe"
|
||||
PATH="$T/clipbin:$PATH" run "$S/ygg-smoke-real.sh" "${rargs[@]}" --clip
|
||||
check "a dry run with --clip and clip.exe exits 0" test "$rc" -eq 0
|
||||
check "it says report.txt goes to the clipboard after the run" has "smoke-real: after the run: report.txt is copied to the clipboard with clip.exe"
|
||||
NOCLIP="$(tr ':' '\n' <<<"$PATH" | while IFS= read -r d; do [ -e "$d/clip.exe" ] || printf '%s:' "$d"; done)"
|
||||
PATH="${NOCLIP%:}" run "$S/ygg-smoke-real.sh" "${rargs[@]}" --clip
|
||||
check "a dry run with --clip and no clip.exe still exits 0" test "$rc" -eq 0
|
||||
check "it says --clip needs clip.exe" has "smoke-real: --clip needs clip.exe (WSL); the report will be in report.txt only"
|
||||
check "the evidence script never gets --clip" bash -c "! grep -q -E 'ygg-evidence\.sh.*(--clip|clip\[)' '$S/ygg-smoke-real.sh'"
|
||||
|
||||
# --- --from-leaf: the run starts from a planted-fault leaf (B-109)
|
||||
L="$T/leaf"; mkdir -p "$L/lib" "$L/.git"
|
||||
printf 'licence\n' >"$L/LICENSE"; printf '# notes\n' >"$L/README.md"; printf 'listen(8080)\n' >"$L/server.js"
|
||||
printf 'x\n' >"$L/lib/notes.js"; printf 'git\n' >"$L/.git/HEAD"
|
||||
printf '#!/bin/sh\nexec node server.js\n' >"$L/start.sh"; chmod +x "$L/start.sh"
|
||||
cp "$O/testdata/smoke-grade/pack.yaml" "$T/pack.before"
|
||||
run "$S/ygg-smoke-real.sh" "${rargs[@]}" --from-leaf "$L"
|
||||
W="$(work)"
|
||||
check "a dry run from a leaf exits 0" test "$rc" -eq 0
|
||||
check "it says which leaf the run starts from" has "smoke-real: the leaf repos/notes starts from $L"
|
||||
check "the graph is the run's own copy of the pack" has "--graph '$W/pack'"
|
||||
check "the copy's template holds the leaf's files" cmp -s "$L/server.js" "$W/pack/templates/notes/server.js"
|
||||
check "and its nested files" cmp -s "$L/lib/notes.js" "$W/pack/templates/notes/lib/notes.js"
|
||||
check "but not its .git" test ! -e "$W/pack/templates/notes/.git"
|
||||
check "the old template's files are gone" bash -c "[ \"\$(ls '$W/pack/templates/notes' | tr '\n' ' ')\" = 'LICENSE README.md lib server.js start.sh ' ]"
|
||||
check "every leaf file is in the first commit" grep -q -F 'first_commit: [LICENSE, README.md, lib/notes.js, server.js, start.sh]' "$W/pack/pack.yaml"
|
||||
check "the rest of the pack line stays" grep -q -F 'repos/notes: { description: the notes product, template: templates/notes, first_commit:' "$W/pack/pack.yaml"
|
||||
check "odin's own pack is not changed" cmp -s "$T/pack.before" "$O/testdata/smoke-grade/pack.yaml"
|
||||
check "start.sh runs the leaf's start.sh as a program when it is executable" grep -q -F 'if [ -x ./start.sh ]; then exec ./start.sh; else exec sh ./start.sh; fi' "$W/project/start.sh"
|
||||
# The wrapper itself, run against a leaf start.sh with and without the x bit (odin writes first-commit files 0644).
|
||||
mkdir -p "$T/app/repos/notes"; cp "$W/project/start.sh" "$T/app/start.sh"
|
||||
printf '#!/bin/sh\necho started\n' >"$T/app/repos/notes/start.sh"; chmod 644 "$T/app/repos/notes/start.sh"
|
||||
check "the wrapper runs a 0644 start.sh with sh" bash -c "[ \"\$('$T/app/start.sh')\" = started ]"
|
||||
printf '#!/usr/bin/env bash\n[[ -n x ]] && echo bash-started\n' >"$T/app/repos/notes/start.sh"; chmod 755 "$T/app/repos/notes/start.sh"
|
||||
check "the wrapper runs an executable start.sh by its own shebang" bash -c "[ \"\$('$T/app/start.sh')\" = bash-started ]"
|
||||
check "a run from a leaf says it checks the repair path after the run" has "smoke-real: after the run: ygg-repair-path.sh checks the repair path in the run's ledger (--from-leaf)"
|
||||
run "$S/ygg-smoke-real.sh" "${rargs[@]}" --from-leaf "$L" --build
|
||||
check "the dry run names the sandbox build on the copy" has "--realm '$(work)/pack'"
|
||||
|
||||
run "$S/ygg-smoke-real.sh" "${rargs[@]}" --from-leaf "$T/no-such-leaf"
|
||||
check "a missing leaf folder is refused" test "$rc" -eq 2
|
||||
check "and named" has "no leaf folder at $T/no-such-leaf"
|
||||
L2="$T/leaf-odd"; mkdir -p "$L2"; printf 'x\n' >"$L2/a b.js"
|
||||
run "$S/ygg-smoke-real.sh" "${rargs[@]}" --from-leaf "$L2"
|
||||
check "a leaf file name that needs quoting in YAML is refused" test "$rc" -eq 2
|
||||
check "and named" has "a b.js"
|
||||
L3="$T/leaf-empty"; mkdir -p "$L3/.git"
|
||||
run "$S/ygg-smoke-real.sh" "${rargs[@]}" --from-leaf "$L3"
|
||||
check "a leaf with no files is refused" test "$rc" -eq 2
|
||||
|
||||
# --- --implementer: the implementer role of the run's factory.yaml (B-110)
|
||||
run "$S/ygg-smoke-real.sh" "${rargs[@]}" --implementer opencode:openrouter/qwen/qwen3.8-27b
|
||||
W="$(work)"
|
||||
check "a dry run with another implementer exits 0" test "$rc" -eq 0
|
||||
check "the run's factory.yaml has that implementer" grep -q -E '^ implementer: \{ agent: opencode, model: openrouter/qwen/qwen3\.8-27b \}' "$W/factory.yaml"
|
||||
check "it has one implementer line" test "$(grep -c 'implementer:' "$W/factory.yaml")" -eq 1
|
||||
check "the reviewer is not changed" grep -q -F 'reviewer: { agent: opencode, model: openrouter/qwen/qwen3.8-27b }' "$W/factory.yaml"
|
||||
check "it prints the implementer" has "smoke-real: the implementer: { agent: opencode, model: openrouter/qwen/qwen3.8-27b }"
|
||||
run "$S/ygg-smoke-real.sh" "${rargs[@]}" --implementer claude-code:opus
|
||||
check "claude-code with another model is accepted" grep -q -F 'implementer: { agent: claude-code, model: opus }' "$(work)/factory.yaml"
|
||||
for bad in opencode qwen opencode: :sonnet scripted:x gemini:x opencode:qwen/qwen3.8-27b 'opencode:openrouter/a b' 'claude-code:a}'; do
|
||||
run "$S/ygg-smoke-real.sh" "${rargs[@]}" --implementer "$bad"
|
||||
check "--implementer '$bad' is refused" test "$rc" -eq 2
|
||||
done
|
||||
check "a refusal says what is allowed" has "--implementer is AGENT:MODEL"
|
||||
|
||||
# --- the run's tail, without --dry-run: the repair path and --clip (B-109, B-111)
|
||||
# A non-dry run reaches the code after `ygg run`. run_real refuses unless every guard holds: YGG_CMD is an
|
||||
# executable under this test's temp folder, the engine check is off, and the folder and the credentials file are the
|
||||
# test's own (dummy values). No later edit can make it reach a real or built ygg, or the founder's credentials.
|
||||
run_real() {
|
||||
case "${YGG_CMD:-}" in "$T"/*) ;; *) echo "FAIL run_real refused: YGG_CMD '${YGG_CMD:-}' is not under $T"; fail=1; return 1 ;; esac
|
||||
[ -x "$YGG_CMD" ] || { echo "FAIL run_real refused: $YGG_CMD is not executable"; fail=1; return 1; }
|
||||
[ "${YGG_ENGINE_CHECK:-}" = 0 ] || { echo "FAIL run_real refused: YGG_ENGINE_CHECK is not 0"; fail=1; return 1; }
|
||||
run "$S/ygg-smoke-real.sh" --folder "$D" --tasks "$M" --odin "$O" --env "$T/factory.env" "$@"
|
||||
}
|
||||
# The fake ygg: doctor and status exit 0; run writes a made-up ledger with one finding raised, fixed and closed.
|
||||
FY="$T/fakeygg/ygg"; mkdir -p "$T/fakeygg"
|
||||
cat >"$FY" <<'EOF2'
|
||||
#!/usr/bin/env bash
|
||||
[ "$1" = run ] || exit 0
|
||||
while [ $# -gt 0 ]; do [ "$1" = --project ] && project="$2"; shift; done
|
||||
d="$project/.factory/runs/20260101-000000-abcdef"; mkdir -p "$d"
|
||||
{
|
||||
echo '{"seq":1,"event":"run_started","ygg.run.id":"20260101-000000-abcdef"}'
|
||||
echo '{"seq":2,"event":"node_finished","ygg.node.id":"build","ygg.attempt":1}'
|
||||
echo '{"seq":3,"event":"finding_raised","ygg.node.id":"build","ygg.attempt":1,"ygg.finding.id":"F1","ygg.finding.source":"review","ygg.note":"server.js:3: the port is 8080"}'
|
||||
echo '{"seq":4,"event":"node_finished","ygg.node.id":"build","ygg.attempt":2}'
|
||||
echo '{"seq":5,"event":"finding_closed","ygg.node.id":"build","ygg.attempt":2,"ygg.finding.id":"F1","ygg.finding.source":"review"}'
|
||||
echo '{"seq":6,"event":"run_finished","ygg.run.id":"20260101-000000-abcdef"}'
|
||||
} >"$d/ledger.jsonl"
|
||||
echo "fake ygg run"
|
||||
EOF2
|
||||
chmod +x "$FY"
|
||||
# The fake clip.exe keeps what it is given.
|
||||
mkdir -p "$T/fakeclip"; printf '#!/bin/sh\ncat >"%s"\n' "$T/clipped.txt" >"$T/fakeclip/clip.exe"; chmod +x "$T/fakeclip/clip.exe"
|
||||
CLIPPATH="$T/fakeclip:$PATH"
|
||||
|
||||
PATH="$CLIPPATH" YGG_CMD="$FY" run_real --from-leaf "$L"
|
||||
W="$(work)"
|
||||
check "a run from a leaf with the fake ygg exits 0" test "$rc" -eq 0
|
||||
check "the ygg that ran is the fake" has "smoke-real: ygg: $FY, modified"
|
||||
check "report.txt has the ygg line" grep -q -F "smoke-real: ygg: $FY" "$W/report.txt"
|
||||
check "report.txt has the repair-path line" grep -q -F "repair path: finding raised, fixed, closed (finding F1 from review" "$W/report.txt"
|
||||
check "the repair-path line is printed" has "repair path: finding raised, fixed, closed"
|
||||
check "without --clip nothing goes to clip.exe" test ! -e "$T/clipped.txt"
|
||||
check "no credential value reaches the output" hasnt "made-up-"
|
||||
check "no credential value reaches report.txt" bash -c "! grep -q made-up- '$W/report.txt'"
|
||||
|
||||
PATH="$CLIPPATH" YGG_CMD="$FY" run_real
|
||||
W="$(work)"
|
||||
check "a run without --from-leaf exits 0" test "$rc" -eq 0
|
||||
check "the ygg that ran is the fake (no leaf)" has "smoke-real: ygg: $FY, modified"
|
||||
check "a run without --from-leaf prints no repair-path line" hasnt "repair path"
|
||||
check "and report.txt has none" bash -c "! grep -q 'repair path' '$W/report.txt'"
|
||||
|
||||
PATH="$CLIPPATH" YGG_CMD="$FY" run_real --from-leaf "$L" --clip
|
||||
W="$(work)"
|
||||
check "a run with --clip exits 0" test "$rc" -eq 0
|
||||
check "the ygg that ran is the fake (clip)" has "smoke-real: ygg: $FY, modified"
|
||||
check "clip.exe gets the whole report.txt" cmp -s "$T/clipped.txt" "$W/report.txt"
|
||||
check "the clipboard has the ygg line" grep -q -F "smoke-real: ygg: $FY" "$T/clipped.txt"
|
||||
check "the clipboard has the repair-path line" grep -q -F "repair path: finding raised, fixed, closed" "$T/clipped.txt"
|
||||
check "the clipboard has the evidence" grep -q -F "# ygg evidence: run 20260101-000000-abcdef" "$T/clipped.txt"
|
||||
check "it says report.txt was copied" has "(report.txt copied to the clipboard)"
|
||||
|
||||
# The guard itself, in a subshell so its expected refusal does not count as a failure here.
|
||||
(YGG_CMD=ygg run_real) >"$T/refusal"
|
||||
check "run_real refuses a YGG_CMD outside the temp folder" grep -q -F "run_real refused: YGG_CMD 'ygg' is not under" "$T/refusal"
|
||||
(YGG_CMD="$FY" YGG_ENGINE_CHECK=1 run_real) >"$T/refusal"
|
||||
check "run_real refuses when the engine check is on" grep -q -F "run_real refused: YGG_ENGINE_CHECK is not 0" "$T/refusal"
|
||||
|
||||
# --- the real odin files, when this checkout has them: the edits must match their lines
|
||||
R="${YGG_REAL_ODIN:-$S/../../yggdrasil/odin}"
|
||||
if [ -f "$R/testdata/smoke-grade/pack.yaml" ] && [ -f "$R/testdata/smoke-grade/factory.real.example.yaml" ]; then
|
||||
run "$S/ygg-smoke-real.sh" --folder "$D" --tasks "$M" --odin "$R" --env "$T/factory.env" --dry-run \
|
||||
--from-leaf "$S/testdata/reviewer-bench/P4" --implementer opencode:openrouter/qwen/qwen3.8-27b
|
||||
W="$(work)"
|
||||
check "real odin: the dry run exits 0" test "$rc" -eq 0
|
||||
check "real odin: the pack lists the P4 files" grep -q -F 'first_commit: [LICENSE, README.md, notes.js, server.js, start.sh]' "$W/pack/pack.yaml"
|
||||
check "real odin: the implementer is set" grep -q -F 'implementer: { agent: opencode, model: openrouter/qwen/qwen3.8-27b }' "$W/factory.yaml"
|
||||
else
|
||||
echo "skip the real odin files: none at $R (set YGG_REAL_ODIN)"
|
||||
fi
|
||||
|
||||
[ "$fail" -eq 0 ] && echo "all ygg-smoke-real tests passed"
|
||||
exit "$fail"
|
||||
Executable
+73
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env bash
|
||||
# Say whether one ygg run used the repair path of a loop, read from its ledger (B-109).
|
||||
# ygg-repair-path.sh LEDGER
|
||||
# The repair path is three steps, in this order in the ledger:
|
||||
# 1. raised: a finding_raised line, on the loop's node N, for finding F;
|
||||
# 2. fixed: an implementer session of the loop finished after it: a node_finished line on node N (odin logs the
|
||||
# reviewers' sessions on N-<reviewer id>, so they do not count);
|
||||
# 3. closed: a finding_closed line for F after that, which its source logs when it confirms the fix.
|
||||
# The event names are gungnir's (event.go); the order is odin's loop (internal/engine/loop.go). A finding that a
|
||||
# check overruled is not closed by its source, so it does not complete the path.
|
||||
# It prints one line, "repair path: finding raised, fixed, closed (...)" or "repair path: missing step STEP: ..."
|
||||
# for the finding that came furthest, and the counts. Exit 0 when one finding took the whole path, 1 when none
|
||||
# did, 2 for a refusal. The ledger holds no credential; this script reads nothing else.
|
||||
set -euo pipefail
|
||||
[ $# -eq 1 ] || { echo "usage: ygg-repair-path.sh LEDGER" >&2; exit 2; }
|
||||
[ -f "$1" ] || { echo "ygg-repair-path: no ledger at $1" >&2; exit 2; }
|
||||
python3 -I - "$1" <<'PY'
|
||||
import json, sys
|
||||
|
||||
events = []
|
||||
with open(sys.argv[1], encoding="utf-8") as f:
|
||||
for line in f:
|
||||
try:
|
||||
e = json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
if isinstance(e, dict):
|
||||
events.append(e)
|
||||
|
||||
# For each finding, in the order it was first raised: the step it reached.
|
||||
findings = {}
|
||||
for e in events:
|
||||
name, node, fid = e.get("event"), e.get("ygg.node.id", ""), e.get("ygg.finding.id", "")
|
||||
if name == "finding_raised":
|
||||
if fid and fid not in findings:
|
||||
findings[fid] = {"node": node, "source": e.get("ygg.finding.source", "?"), "note": e.get("ygg.note", ""),
|
||||
"raised": e.get("ygg.attempt", "?"), "fixed": False, "closed": None, "overruled": False}
|
||||
elif name == "node_finished":
|
||||
for r in findings.values():
|
||||
if r["node"] == node:
|
||||
r["fixed"] = True
|
||||
elif name in ("finding_closed", "finding_overruled"):
|
||||
r = findings.get(fid)
|
||||
if r is None or not r["fixed"]:
|
||||
continue # a close before any fix is not the repair path
|
||||
if name == "finding_closed" and r["closed"] is None:
|
||||
r["closed"] = e.get("ygg.attempt", "?")
|
||||
elif name == "finding_overruled":
|
||||
r["overruled"] = True
|
||||
|
||||
closed = sum(1 for r in findings.values() if r["closed"] is not None)
|
||||
counts = "%d finding(s) raised, %d closed" % (len(findings), closed)
|
||||
if not findings:
|
||||
print("repair path: missing step raised: no finding was raised (%s)" % counts)
|
||||
sys.exit(1)
|
||||
done = [(fid, r) for fid, r in findings.items() if r["closed"] is not None]
|
||||
if done:
|
||||
fid, r = done[0]
|
||||
print("repair path: finding raised, fixed, closed (finding %s from %s, raised in attempt %s, closed in attempt %s: %s; %s)"
|
||||
% (fid, r["source"], r["raised"], r["closed"], r["note"], counts))
|
||||
sys.exit(0)
|
||||
fixed = [(fid, r) for fid, r in findings.items() if r["fixed"]]
|
||||
if fixed:
|
||||
fid, r = fixed[0]
|
||||
why = "; it was overruled by a check, not closed by its source" if r["overruled"] else ""
|
||||
print("repair path: missing step closed: finding %s was raised and an implementer session finished after it, but it was not closed%s (%s)"
|
||||
% (fid, why, counts))
|
||||
sys.exit(1)
|
||||
fid, r = next(iter(findings.items()))
|
||||
print("repair path: missing step fixed: finding %s was raised in attempt %s, but no implementer session of node %s finished after it (%s)"
|
||||
% (fid, r["raised"], r["node"], counts))
|
||||
sys.exit(1)
|
||||
PY
|
||||
Executable
+165
@@ -0,0 +1,165 @@
|
||||
#!/usr/bin/env python3
|
||||
# The scorer of ygg-reviewer-bench.sh (B-108). Run it as `python3 -I`:
|
||||
# ygg-reviewer-bench.py OUT N FAULT LEAF MODEL
|
||||
# It reads OUT/session-1.jsonl to OUT/session-N.jsonl (the OpenCode events) and OUT/session-N.exit, and prints for
|
||||
# each session whether the answer is a valid report, then the table. FAULT is a Python regex, or empty for none.
|
||||
# It reads the events as gleipnir's opencode adapter does (the answer is the text of the last step that said
|
||||
# anything; tokens and cost are the sums of the step_finish events) and the answer as the engine does
|
||||
# (gungnir/schema/event.go ParseReport and lastDocument, the report and finding schemas, and the rules of
|
||||
# jormungandr Review for a first review).
|
||||
import json, re, sys
|
||||
out, n, fault, leaf, model = sys.argv[1], int(sys.argv[2]), sys.argv[3], sys.argv[4], sys.argv[5]
|
||||
pat = re.compile(fault, re.I) if fault else None
|
||||
FENCE, LOC = "`" * 3, re.compile(r"[^\s:]+:[0-9]+(-[0-9]+)?")
|
||||
LIMITS = {"id": 100, "source": 100, "location": 300, "claim": 4000, "proof": 2000}
|
||||
class Bad(Exception):
|
||||
pass
|
||||
def has_document(text):
|
||||
dec = json.JSONDecoder()
|
||||
for i, c in enumerate(text):
|
||||
if c == "{":
|
||||
try:
|
||||
dec.raw_decode(text, i)
|
||||
return True
|
||||
except ValueError:
|
||||
pass
|
||||
return False
|
||||
def last_document(s):
|
||||
s, k = s.strip(), s.count(FENCE)
|
||||
if k == 0:
|
||||
starts = [i for i, c in enumerate(s) if c == "{"]
|
||||
for i in starts:
|
||||
try:
|
||||
json.loads(s[i:])
|
||||
break
|
||||
except ValueError:
|
||||
pass
|
||||
else:
|
||||
raise Bad("the answer does not end with a JSON document")
|
||||
prose, data = s[:i], s[i:]
|
||||
elif k == 1:
|
||||
raise Bad("a code fence is not closed")
|
||||
elif k > 2:
|
||||
raise Bad("more than one code fence")
|
||||
elif not s.endswith(FENCE):
|
||||
raise Bad("text after the report")
|
||||
else:
|
||||
o = s.index(FENCE)
|
||||
prose, nl, body = s[:o], *s[o:].partition("\n")[1:]
|
||||
if not nl:
|
||||
raise Bad("a code fence is not closed")
|
||||
body = body.strip()
|
||||
data = (body[:-3] if body.endswith(FENCE) else body).strip()
|
||||
if has_document(prose):
|
||||
raise Bad("more than one document")
|
||||
return data
|
||||
def findings(answer):
|
||||
try:
|
||||
doc = json.loads(last_document(answer))
|
||||
except ValueError as e:
|
||||
raise Bad(f"not JSON: {e}")
|
||||
if not isinstance(doc, dict) or set(doc) != {"findings"} or not isinstance(doc["findings"], list):
|
||||
raise Bad('the report is not {"findings": [...]} alone')
|
||||
fs, seen = doc["findings"], set()
|
||||
if len(fs) > 200:
|
||||
raise Bad("more than 200 findings")
|
||||
for f in fs:
|
||||
if not isinstance(f, dict):
|
||||
raise Bad("a finding is not an object")
|
||||
if set(f) - set(LIMITS) - {"severity", "status"}:
|
||||
raise Bad(f"a finding has an unknown field: {sorted(set(f) - set(LIMITS) - {'severity', 'status'})}")
|
||||
for k in ("id", "source", "severity", "location", "claim", "status") + (("proof",) if "proof" in f else ()):
|
||||
if not isinstance(f.get(k), str) or (k != "location" and not f[k]):
|
||||
raise Bad(f'a finding has no text "{k}"')
|
||||
if len(f[k]) > LIMITS.get(k, 100):
|
||||
raise Bad(f'a finding\'s "{k}" is too long')
|
||||
fid = f["id"]
|
||||
if f["severity"] not in ("blocker", "major", "minor"):
|
||||
raise Bad(f"finding {fid}: severity {f['severity']!r} is not blocker, major or minor")
|
||||
if not LOC.fullmatch(f["location"]):
|
||||
raise Bad(f"finding {fid}: location {f['location']!r} is not file:line")
|
||||
if f["status"] not in ("open", "disputed"):
|
||||
raise Bad(f"finding {fid}: status {f['status']!r} in a first review")
|
||||
if f["source"] != "review":
|
||||
raise Bad(f"finding {fid} says its source is {f['source']!r}, not review")
|
||||
if fid.startswith("check-"):
|
||||
raise Bad(f"finding {fid} uses the id space of the checks")
|
||||
if fid in seen:
|
||||
raise Bad(f"finding id {fid} is used twice")
|
||||
seen.add(fid)
|
||||
return fs
|
||||
def session(i):
|
||||
text = answer = failed = ""
|
||||
steps, broken, cost, use = 0, False, 0.0, {"in": 0, "cw": 0, "read": 0, "out": 0}
|
||||
with open(f"{out}/session-{i}.jsonl", encoding="utf-8", errors="replace") as fh:
|
||||
lines = fh.readlines()
|
||||
for line in lines:
|
||||
line = line.strip()
|
||||
try:
|
||||
e = json.loads(line) if line.startswith("{") else None
|
||||
except ValueError:
|
||||
e = None
|
||||
if not isinstance(e, dict):
|
||||
continue
|
||||
t, p = e.get("type"), e.get("part") or {}
|
||||
if t == "step_start":
|
||||
text = ""
|
||||
elif t == "text":
|
||||
text += p.get("text") or ""
|
||||
elif t == "error":
|
||||
err = e.get("error") or {}
|
||||
failed = f"{err.get('name', 'error')}: {(err.get('data') or {}).get('message', '')}"
|
||||
elif t == "step_finish":
|
||||
tk, c = p.get("tokens") or {}, p.get("cost")
|
||||
nums = [tk.get("input"), tk.get("output"), (tk.get("cache") or {}).get("read"), (tk.get("cache") or {}).get("write"), c]
|
||||
if not all(isinstance(x, (int, float)) for x in nums):
|
||||
broken = True
|
||||
continue
|
||||
steps += 1
|
||||
use["in"] += tk["input"]
|
||||
use["cw"] += tk["cache"]["write"]
|
||||
use["read"] += tk["cache"]["read"]
|
||||
use["out"] += tk["output"] + (tk.get("reasoning") or 0)
|
||||
cost += c
|
||||
if text.strip():
|
||||
answer = text
|
||||
with open(f"{out}/session-{i}.exit") as fh:
|
||||
rc = int(fh.read().strip() or 0)
|
||||
known = not broken and not (cost == 0 and sum(use.values()) > 0)
|
||||
try:
|
||||
if rc == 124:
|
||||
raise Bad("the session timed out after 5 minutes")
|
||||
if failed:
|
||||
raise Bad(failed)
|
||||
if rc != 0:
|
||||
raise Bad(f"opencode exited {rc}")
|
||||
if broken:
|
||||
raise Bad("a step_finish event lacks tokens or cost")
|
||||
fs, why = findings(answer), ""
|
||||
except Bad as e:
|
||||
fs, why = None, str(e)
|
||||
return fs, why, use, cost if known else None
|
||||
k = lambda v: f"{v / 1000:.1f}k"
|
||||
usd = lambda c: "unknown" if c is None else f"${c:.3f}"
|
||||
rows = [session(i) for i in range(1, n + 1)]
|
||||
found = 0
|
||||
for i, (fs, why, use, cost) in enumerate(rows, 1):
|
||||
usage = f"{k(use['in'] + use['cw'] + use['read'])} in ({k(use['in'] + use['cw'])} new), {k(use['out'])} out, {usd(cost)}"
|
||||
if fs is None:
|
||||
print(f"session {i}: not a valid report: {why}; {usage}")
|
||||
continue
|
||||
hits = [f for f in fs if pat and pat.search(f["claim"] + " " + f["location"])]
|
||||
found += bool(hits)
|
||||
print(f"session {i}: valid, {len(fs)} finding(s){', fault found' if hits else ''}; {usage}")
|
||||
for f in fs:
|
||||
claim = " ".join(f["claim"].split())
|
||||
print(f" - {f['severity']} {f['location']}{' [fault]' if f in hits else ''}: {claim[:300]}")
|
||||
avg = lambda key: sum(r[2][key] for r in rows) / n
|
||||
costs = [r[3] for r in rows]
|
||||
print()
|
||||
print("| Leaf | Model | Sessions | Valid reports | Findings per session | Fault found | Tokens in (new) | Tokens out | Cost per session |")
|
||||
print("| --- | --- | --- | --- | --- | --- | --- | --- | --- |")
|
||||
print(f"| {leaf} | {model} | {n} | {sum(r[0] is not None for r in rows)} | "
|
||||
f"{', '.join('-' if r[0] is None else str(len(r[0])) for r in rows)} | {found if pat else '-'} | "
|
||||
f"{k(avg('in') + avg('cw') + avg('read'))} ({k(avg('in') + avg('cw'))}) | {k(avg('out'))} | "
|
||||
f"{usd(None if None in costs else sum(costs) / n)} |")
|
||||
Executable
+168
@@ -0,0 +1,168 @@
|
||||
#!/usr/bin/env bash
|
||||
# Compare reviewer models on a leaf with a known planted fault (B-108; docs/6-journal/2026-10-20-choosing-the-
|
||||
# reviewer-model.md). IT COSTS MONEY unless --dry-run: each session is one OpenCode session on OpenRouter (a paid
|
||||
# API: OPENROUTER_API_KEY), about 1 cent for qwen/qwen3.8-27b.
|
||||
# ygg-reviewer-bench.sh --model MODEL --leaf DIR --sessions N [--fault REGEX] [--effort low|medium|high|off]
|
||||
# [--out OUT] [--odin DIR] [--tasks DIR] [--dry-run]
|
||||
# Each session reviews a fresh copy of the leaf (one git commit, no .git of the original) with the prompt the
|
||||
# engine gives the smoke's reviewer:
|
||||
# 1. odin's smoke pack reviewer prompt, testdata/smoke-grade/prompts/review.md (odin PRs #21 and #22);
|
||||
# 2. "## Brief" and the brief of the task, mimir smoke-notes/instruction.md;
|
||||
# 3. the engine's answer request: jormungandr prompt.go, ReviewRequest and ReportFormat (as of jormungandr
|
||||
# ec8f18c), for the reviewer id "review" and the passed check "unit".
|
||||
# The settings are those of the OpenCode adapter (gleipnir opencode/opencode.go): strict privacy on OpenRouter (no
|
||||
# data kept, zero retention, no fallback), the effort named on every request (high), read-only permissions, 10
|
||||
# turns (the graph's max_turns) and 5 minutes. There is no token limit: gleipnir ends a session at the node's
|
||||
# max_tokens, and the bench has only the 300-second timeout. As in qwen-opencode.sh, opencode runs --standalone,
|
||||
# with no project or Claude Code config, and no update, share or model fetch. As in the engine, a session gets
|
||||
# only the variables the script names (env -i): PATH, HOME, TMPDIR, XDG_CACHE_HOME (if set), OPENROUTER_API_KEY
|
||||
# and the OpenCode settings. No OPENCODE_* variable and no Anthropic credential of the caller reaches it. Each
|
||||
# session has its own empty XDG_CONFIG_HOME and XDG_DATA_HOME, in a folder under TMPDIR, outside HOME, so no
|
||||
# global OpenCode config applies. The cache (models catalogue, provider package) is the host's.
|
||||
# It prints for each session whether the answer is a valid report, by the engine's rules (gungnir ParseReport:
|
||||
# prose before the report is ignored, nothing may follow it; and jormungandr Review on a first review), and the
|
||||
# claims of its findings; then the journal's table. A finding is the fault when REGEX (Python, case ignored)
|
||||
# matches its claim or location: read the claims as well. "Tokens in" counts cache reads, "(new)" does not;
|
||||
# "Tokens out" includes reasoning. The cost is OpenCode's own figure. A session with tokens and cost 0 (OpenRouter
|
||||
# reports no cost for the Claude models) makes the cost "unknown".
|
||||
# OUT (new or empty; default a new folder under TMPDIR) keeps prompt.md, session-N.jsonl (the events),
|
||||
# session-N.err, session-N.exit (the exit code) and summary.md. The scorer is ygg-reviewer-bench.py.
|
||||
# The leaves in scripts/testdata/reviewer-bench, and their --fault:
|
||||
# P0 no known fault: the product the failed smoke run built (no --fault)
|
||||
# P1 start.sh runs servr.js 'servr'
|
||||
# P2 the server listens on 8080, not 3000 '8080'
|
||||
# P3 GET /api/notes returns the oldest first 'oldest|newest|revers|insertion order'
|
||||
# P4 DELETE /api/notes/ID deletes another person's note. server.js reaches notes through notes.of(email);
|
||||
# remove in notes.js ignores the owner. Neither file alone shows it.
|
||||
# 'owner|another (user|person|account)|other (users?|person|people|accounts?)|any (user|person)|not theirs|not the caller|idor'
|
||||
# YGG_REVIEWER_BENCH_OPENCODE overrides the opencode command (the test's fake). Exit 0, or 2 for a refusal.
|
||||
set -euo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
usage() {
|
||||
echo "usage: ygg-reviewer-bench.sh --model MODEL --leaf DIR --sessions N [--fault REGEX] [--effort low|medium|high|off] [--out OUT] [--odin DIR] [--tasks DIR] [--dry-run]" >&2
|
||||
exit 2
|
||||
}
|
||||
fail() { echo "reviewer-bench: $*" >&2; exit 2; }
|
||||
model="" leaf="" sessions="" fault="" effort=high out="" dry=0
|
||||
odin="$HERE/../../yggdrasil/odin" tasks="$HERE/../../mimir"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--dry-run) dry=1; shift ;;
|
||||
--model) [ $# -ge 2 ] || usage; model="$2"; shift 2 ;;
|
||||
--leaf) [ $# -ge 2 ] || usage; leaf="$2"; shift 2 ;;
|
||||
--sessions) [ $# -ge 2 ] || usage; sessions="$2"; shift 2 ;;
|
||||
--fault) [ $# -ge 2 ] || usage; fault="$2"; shift 2 ;;
|
||||
--effort) [ $# -ge 2 ] || usage; effort="$2"; shift 2 ;;
|
||||
--out) [ $# -ge 2 ] || usage; out="$2"; shift 2 ;;
|
||||
--odin) [ $# -ge 2 ] || usage; odin="$2"; shift 2 ;;
|
||||
--tasks) [ $# -ge 2 ] || usage; tasks="$2"; shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$model" ] && [ -n "$leaf" ] && [ -n "$sessions" ] || usage
|
||||
[[ $sessions =~ ^[1-9][0-9]*$ ]] || fail "--sessions is a whole number above 0, not '$sessions'"
|
||||
id="${model#openrouter/}"
|
||||
[[ $id =~ ^[A-Za-z0-9][A-Za-z0-9._:/-]*$ ]] || fail "--model '$model' is not an OpenRouter model id, such as qwen/qwen3.8-27b"
|
||||
case "$effort" in low | medium | high | off) ;; *) fail "--effort is low, medium, high or off, not '$effort'" ;; esac
|
||||
for tool in python3 git tar timeout; do command -v "$tool" >/dev/null 2>&1 || fail "$tool is not installed"; done
|
||||
[ -d "$leaf" ] || fail "no leaf folder at $leaf (--leaf)"
|
||||
leaf="$(cd "$leaf" && pwd)"
|
||||
if [ -n "$fault" ] && ! python3 -I -c 'import re, sys; re.compile(sys.argv[1])' "$fault" 2>/dev/null; then
|
||||
fail "--fault '$fault' is not a Python regular expression"
|
||||
fi
|
||||
review="$odin/testdata/smoke-grade/prompts/review.md"
|
||||
brief="$tasks/smoke-notes/instruction.md"
|
||||
[ -f "$review" ] || fail "$review is missing (--odin): it is the smoke reviewer's prompt"
|
||||
[ -f "$brief" ] || fail "$brief is missing (--tasks): it is the brief of the task"
|
||||
[ "$dry" = 1 ] || [ -n "${OPENROUTER_API_KEY:-}" ] || fail "OPENROUTER_API_KEY is not set; use a dedicated OpenRouter key with a credit limit"
|
||||
if [ -z "$out" ]; then
|
||||
out="$(mktemp -d "${TMPDIR:-/tmp}/ygg-reviewer-bench.XXXXXX")"
|
||||
elif [ -d "$out" ] && [ -n "$(ls -A "$out")" ]; then
|
||||
fail "--out $out is not empty; give a new folder"
|
||||
fi
|
||||
mkdir -p "$out"
|
||||
out="$(cd "$out" && pwd)"
|
||||
work="$(mktemp -d "${TMPDIR:-/tmp}/ygg-reviewer-bench-work.XXXXXX")"
|
||||
trap 'rm -rf -- "$work"' EXIT
|
||||
|
||||
# The prompt, as the engine builds it (odin internal/engine/loop.go, Review): the reviewer's prompt file, the
|
||||
# brief, then jormungandr.ReviewRequest("review", no open findings, passed ["unit"]).
|
||||
{
|
||||
cat "$review"
|
||||
printf '\n\n## Brief\n\n'
|
||||
cat "$brief"
|
||||
printf '\n\n'
|
||||
cat <<'EOF'
|
||||
## Your answer
|
||||
|
||||
Your id is review. Use it as the "source" of every finding.
|
||||
|
||||
Answer with one JSON document: end your answer with the document; nothing after it. Any text before it is ignored. The document, bare or in one code fence:
|
||||
|
||||
{"findings": [ {"id": "...", "source": "<your id>", "severity": "blocker|major|minor", "location": "path/to/file:LINE", "claim": "what is wrong", "status": "open"} ]}
|
||||
|
||||
- "id" names the problem and stays the same every time you report it.
|
||||
- "location" is a file and a line, or a range such as path/to/file:10-20.
|
||||
- "status" is "open" for a problem you still see. Say "fixed" only for a problem listed below that you checked
|
||||
is gone, and then add "proof": the file:line you read, and the name of one of the checks listed below
|
||||
that passed. A finding without both is refused.
|
||||
- A blocker or major finding of yours that the implementer disputed and a check settled is closed ("overruled"):
|
||||
it is not in the list below. Report it again only with proof: a failing check, or a different location than
|
||||
before. A repeat with neither is ignored.
|
||||
- If you find nothing, answer {"findings": []}.
|
||||
|
||||
The checks that passed on this work: unit.
|
||||
|
||||
The implementer may dispute one of your findings. A dispute is settled by a check: if it holds, the finding is overruled and closed. If you see the same problem again, report it with proof: a failing check, or a different location than the overruled finding. A repeat with neither is ignored.
|
||||
|
||||
EOF
|
||||
} >"$out/prompt.md"
|
||||
|
||||
# The session's config, as gleipnir's ConfigJSON makes it for a read-only reviewer under privacy: strict.
|
||||
reasoning=""
|
||||
[ "$effort" = off ] || reasoning="\"reasoning\":{\"effort\":\"$effort\"},"
|
||||
# shellcheck disable=SC2016 # "$schema" is a JSON key, not a variable
|
||||
conf="$(printf '{"$schema":"https://opencode.ai/config.json","autoupdate":false,"share":"disabled","provider":{"openrouter":{"options":{"apiKey":"{env:OPENROUTER_API_KEY}"},"models":{"%s":{"options":{%s"provider":{"data_collection":"deny","zdr":true,"allow_fallbacks":false}}}}}},"small_model":"openrouter/%s","agent":{"build":{"steps":10}},"permission":{"edit":"deny","bash":"deny","webfetch":"deny","external_directory":"deny","task":"deny","question":"deny"}}' \
|
||||
"$id" "$reasoning" "$id")"
|
||||
# As in the engine, a session gets only the variables named here (env -i): no OPENCODE_* and nothing else of the
|
||||
# caller's shell. The key is added in the real run only, so the dry run never prints it.
|
||||
env_args=("PATH=$PATH" "HOME=$HOME")
|
||||
[ -z "${TMPDIR:-}" ] || env_args+=("TMPDIR=$TMPDIR")
|
||||
[ -z "${XDG_CACHE_HOME:-}" ] || env_args+=("XDG_CACHE_HOME=$XDG_CACHE_HOME")
|
||||
env_args+=("OPENCODE_CONFIG_CONTENT=$conf" OPENCODE_DISABLE_PROJECT_CONFIG=1 OPENCODE_DISABLE_CLAUDE_CODE=1 OPENCODE_DISABLE_EXTERNAL_SKILLS=1
|
||||
OPENCODE_DISABLE_AUTOUPDATE=true OPENCODE_DISABLE_SHARE=true OPENCODE_DISABLE_MODELS_FETCH=true)
|
||||
bin="${YGG_REVIEWER_BENCH_OPENCODE:-opencode}"
|
||||
cmd=(run --standalone --format json -m "openrouter/$id")
|
||||
|
||||
if [ "$dry" = 1 ]; then
|
||||
q() { printf "'%s'" "${1//\'/\'\\\'\'}"; }
|
||||
echo "Dry run: it spends nothing. Each of the $sessions sessions would copy $leaf (without .git) to"
|
||||
echo "$work/leaf-N, commit it there, and run in that folder:"
|
||||
printf ' env -i'
|
||||
for a in "${env_args[@]}"; do printf ' %s=%s' "${a%%=*}" "$(q "${a#*=}")"; done
|
||||
# shellcheck disable=SC2016 # the key is shown as a reference, never as its value
|
||||
printf ' OPENROUTER_API_KEY="$OPENROUTER_API_KEY"'
|
||||
printf ' XDG_DATA_HOME=%s XDG_CONFIG_HOME=%s timeout 300 %s %s < %s\n' "$(q "$work/data-N")" "$(q "$work/config-N")" \
|
||||
"$(q "$bin")" "${cmd[*]}" "$(q "$out/prompt.md")"
|
||||
echo "The prompt is in $out/prompt.md."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
for i in $(seq 1 "$sessions"); do
|
||||
copy="$work/leaf-$i"
|
||||
mkdir -p "$copy" "$work/data-$i" "$work/config-$i"
|
||||
tar -C "$leaf" --exclude=.git -cf - . | tar -C "$copy" -xf -
|
||||
git -C "$copy" init -q
|
||||
git -C "$copy" add -A
|
||||
git -C "$copy" -c user.name=reviewer-bench -c user.email=reviewer-bench@localhost -c commit.gpgsign=false \
|
||||
commit -q --no-verify -m "The leaf under review"
|
||||
echo "session $i of $sessions" >&2
|
||||
rc=0
|
||||
(cd "$copy" && env -i "${env_args[@]}" OPENROUTER_API_KEY="$OPENROUTER_API_KEY" XDG_DATA_HOME="$work/data-$i" \
|
||||
XDG_CONFIG_HOME="$work/config-$i" timeout 300 "$bin" "${cmd[@]}") \
|
||||
<"$out/prompt.md" >"$out/session-$i.jsonl" 2>"$out/session-$i.err" || rc=$?
|
||||
echo "$rc" >"$out/session-$i.exit"
|
||||
done
|
||||
|
||||
python3 -I "$HERE/ygg-reviewer-bench.py" "$out" "$sessions" "$fault" "$(basename "$leaf")" "$id" | tee "$out/summary.md"
|
||||
echo "The prompt, the events of each session and this summary are in $out."
|
||||
+116
-11
@@ -2,6 +2,7 @@
|
||||
# Run the real-agent smoke (v0-H17): the task smoke-notes built by the real Claude Code and the real OpenCode on
|
||||
# OpenRouter, then graded. IT COSTS MONEY: the caps are in odin/testdata/smoke-grade/factory.real.example.yaml.
|
||||
# ygg-smoke-real.sh [--dry-run] [--build] [--folder DIR] [--env FILE] [--tasks DIR] [--odin DIR] [--wall TIME] [--clip]
|
||||
# [--from-leaf LEAF] [--implementer AGENT:MODEL]
|
||||
# The suite is the PUBLIC sample suite of the task (no hidden suite of smoke-notes exists); the agents never mount
|
||||
# it. Needs DIR/factory.yaml (it names the engine; this script never writes it), the credentials file, the
|
||||
# devcontainer CLI 0.89.0, jq and python3, the engine up, the bind mount present, and a checkout of `mimir`. It
|
||||
@@ -10,12 +11,23 @@
|
||||
# CLAUDE_CODE_OAUTH_TOKEN and OPENROUTER_API_KEY is on one line with a value after it (grep -c counts the
|
||||
# lines; no value is printed or kept);
|
||||
# 3. makes DIR/smoke-real-<time> with the project, the public suite and the factory.yaml filled from the
|
||||
# example (its caps are the real ones);
|
||||
# 4. runs `ygg doctor` on that config. A config that doctor refuses stops the script before any money is spent;
|
||||
# 5. with --dry-run, stops here and prints what it would run;
|
||||
# example (its caps are the real ones). With --implementer AGENT:MODEL, its implementer line is replaced
|
||||
# (AGENT is claude-code or opencode; an opencode MODEL is openrouter/<model>), so one role changes (B-110).
|
||||
# With --from-leaf LEAF, the leaf repos/notes starts from the files of LEAF (without .git) instead of the
|
||||
# pack's template (B-109): the pack is copied to DIR/smoke-real-<time>/pack, LEAF becomes the copy's
|
||||
# template, and every file of LEAF goes in the leaf's first commit. odin writes those files without the
|
||||
# execute bit, so the project's start.sh then runs the leaf's start.sh as a program when it is executable,
|
||||
# and with sh when it is not. The planted-fault leaves are in scripts/testdata/reviewer-bench (see
|
||||
# ygg-reviewer-bench.sh);
|
||||
# 4. prints which ygg runs (path, modified time; for a build from --odin, the odin commit and tree state), then
|
||||
# runs `ygg doctor` on that config. A config that doctor refuses stops the script before any money is spent;
|
||||
# 5. with --dry-run, stops here and prints what it would run, and what it would do after the run;
|
||||
# 6. otherwise runs `ygg run` under `timeout` (--wall, default 90m) with the credentials loaded in a subshell
|
||||
# only, and keeps the raw output in DIR/smoke-real-<time>/run.txt (shown live: do not paste that file);
|
||||
# 7. prints the evidence report (credentials removed), the score, and the COST read from the ledger.
|
||||
# 7. prints the evidence report (credentials removed) and keeps it in DIR/smoke-real-<time>/report.txt, with
|
||||
# the ygg line; then the score; with --from-leaf, whether a finding took the repair path (ygg-repair-path.sh:
|
||||
# raised, fixed, closed, or the missing step; also added to report.txt); and the COST read from the ledger.
|
||||
# --clip copies the whole report.txt to the Windows clipboard (clip.exe, in WSL).
|
||||
# The credentials are loaded by `. FILE` in the subshell that runs ygg, as the other smoke scripts do, and nothing
|
||||
# in this script echoes, greps for a value, or copies them. The exit code is the run's (0 done, 1 a node failed,
|
||||
# 3 stopped, 4 paused, 5 done with tasks given up, 124 the wall clock ended it) or 2 for a refusal before it.
|
||||
@@ -24,9 +36,10 @@
|
||||
# (default 0.89.0). The engine environment comes from ygg-engine.sh (YGG_ENGINE_CHECK=0 skips it).
|
||||
set -euo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
dir="$HOME/ygg-it" envfile="$HOME/.config/ouroboric/factory.env" tasks="$HERE/../../mimir" wall=90m dry=0 build=0 clip=()
|
||||
dir="$HOME/ygg-it" envfile="$HOME/.config/ouroboric/factory.env" tasks="$HERE/../../mimir" wall=90m dry=0 build=0 clip=0
|
||||
leaf="" implementer=""
|
||||
odin="${YGG_ODIN:-$HERE/../../yggdrasil/odin}"
|
||||
usage() { echo "usage: ygg-smoke-real.sh [--dry-run] [--build] [--folder DIR] [--env FILE] [--tasks DIR] [--odin DIR] [--wall TIME] [--clip]" >&2; exit 2; }
|
||||
usage() { echo "usage: ygg-smoke-real.sh [--dry-run] [--build] [--folder DIR] [--env FILE] [--tasks DIR] [--odin DIR] [--wall TIME] [--clip] [--from-leaf LEAF] [--implementer AGENT:MODEL]" >&2; exit 2; }
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--dry-run) dry=1; shift ;;
|
||||
@@ -36,7 +49,9 @@ while [ $# -gt 0 ]; do
|
||||
--tasks) [ $# -ge 2 ] || usage; tasks="$2"; shift 2 ;;
|
||||
--odin) [ $# -ge 2 ] || usage; odin="$2"; shift 2 ;;
|
||||
--wall) [ $# -ge 2 ] || usage; wall="$2"; shift 2 ;;
|
||||
--clip) clip=(--clip); shift ;;
|
||||
--clip) clip=1; shift ;;
|
||||
--from-leaf) [ $# -ge 2 ] || usage; leaf="$2"; shift 2 ;;
|
||||
--implementer) [ $# -ge 2 ] || usage; implementer="$2"; shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
@@ -44,6 +59,18 @@ fail() { echo "smoke-real: $*" >&2; exit 2; }
|
||||
task=smoke-notes
|
||||
bind="${YGG_BIND:-/mnt/wsl/ygg-bind}"
|
||||
[[ $wall =~ ^[1-9][0-9]*[smh]$ ]] || fail "--wall is a number and s, m or h, such as 90m"
|
||||
# The implementer goes into a YAML line as it is, so its model has only the characters a model id needs.
|
||||
if [ -n "$implementer" ]; then
|
||||
impl_agent="${implementer%%:*}" impl_model="${implementer#*:}"
|
||||
ok=0
|
||||
if [[ $implementer == *:* ]] && [[ $impl_model =~ ^[A-Za-z0-9._/:-]+$ ]]; then
|
||||
case "$impl_agent" in
|
||||
claude-code) ok=1 ;;
|
||||
opencode) [[ $impl_model == openrouter/?* ]] && ok=1 ;;
|
||||
esac
|
||||
fi
|
||||
[ "$ok" = 1 ] || fail "--implementer is AGENT:MODEL, with AGENT claude-code or opencode; an opencode MODEL is openrouter/<model>, such as opencode:openrouter/qwen/qwen3.8-27b"
|
||||
fi
|
||||
|
||||
export PATH="$dir/dc/node_modules/.bin:$PATH"
|
||||
if [ "${YGG_ENGINE_CHECK:-1}" = 1 ]; then
|
||||
@@ -65,6 +92,22 @@ done
|
||||
suite="$(sed -n 's/^suite[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' "$tasks/$task/task.toml" | head -n 1)"
|
||||
[ -n "$suite" ] || fail "$tasks/$task/task.toml has no metadata.suite"
|
||||
|
||||
# The leaf to start from: its files, without .git, each a plain name pack.yaml can list without quotes.
|
||||
if [ -n "$leaf" ]; then
|
||||
[ -d "$leaf" ] || fail "no leaf folder at $leaf (--from-leaf)"
|
||||
leaf="$(cd "$leaf" && pwd)"
|
||||
odd="$(cd "$leaf" && find . -name .git -prune -o ! -type f ! -type d -print | head -n 1)"
|
||||
[ -z "$odd" ] || fail "$leaf has $odd, which is neither a file nor a folder; odin's first commit takes plain files only"
|
||||
leaf_files="$(cd "$leaf" && find . -name .git -prune -o -type f -print | sed 's|^\./||' | LC_ALL=C sort)"
|
||||
[ -n "$leaf_files" ] || fail "$leaf has no files"
|
||||
while IFS= read -r f; do
|
||||
[[ $f =~ ^[A-Za-z0-9._/-]+$ ]] || fail "the leaf file '$f' has a name pack.yaml cannot list without quotes"
|
||||
done <<<"$leaf_files"
|
||||
leaf_files="$(printf '%s\n' "$leaf_files" | paste -sd, - | sed 's/,/, /g')"
|
||||
template="$(sed -n 's|^ repos/notes: {.*template: \([^,} ]*\).*|\1|p' "$realm/pack.yaml")"
|
||||
[ -n "$template" ] || fail "$realm/pack.yaml has no line for the leaf repos/notes with a template"
|
||||
fi
|
||||
|
||||
# The credentials file: its existence, and the two names. grep -c prints a count and nothing else; the file's
|
||||
# lines are never shown, stored or compared. A name needs one line, with something after the equals sign.
|
||||
[ -f "$envfile" ] || fail "the credentials file $envfile is missing"
|
||||
@@ -80,13 +123,31 @@ project="$work/project"
|
||||
git -C "$project" init -q -b main
|
||||
printf 'root: true\nowner: ouroboric-dev\nvisibility: private\n' >"$project/umbrella.yaml"
|
||||
# The grader's /app is the project. The product is built in the leaf; this file starts it.
|
||||
# odin writes a leaf's first-commit files without the execute bit, so a run from a leaf runs its start.sh with sh
|
||||
# unless the implementer made it executable. sh is dash in the grader's image, so the shebang wins when it can.
|
||||
starter=./start.sh
|
||||
[ -z "$leaf" ] || starter="if [ -x ./start.sh ]; then exec ./start.sh; else exec sh ./start.sh; fi"
|
||||
# shellcheck disable=SC2016
|
||||
printf '#!/bin/sh\ncd "$(dirname "$0")/repos/notes" && exec ./start.sh\n' >"$project/start.sh"
|
||||
if [ -z "$leaf" ]; then
|
||||
printf '#!/bin/sh\ncd "$(dirname "$0")/repos/notes" && exec %s\n' "$starter" >"$project/start.sh"
|
||||
else
|
||||
printf '#!/bin/sh\ncd "$(dirname "$0")/repos/notes" || exit 1\n%s\n' "$starter" >"$project/start.sh"
|
||||
fi
|
||||
chmod +x "$project/start.sh"
|
||||
git -C "$project" add umbrella.yaml start.sh
|
||||
git -C "$project" -c user.name=ygg -c user.email=[email protected] commit -q -m "the real-agent smoke project"
|
||||
echo "smoke-real: made $work"
|
||||
echo "smoke-real: the suite is the PUBLIC sample suite $public/tests (suite $suite)"
|
||||
if [ -n "$leaf" ]; then
|
||||
# The run's own copy of the pack, with LEAF as the template of repos/notes. odin's pack is not changed.
|
||||
mkdir -p "$work/pack/$template"
|
||||
tar -C "$realm" --exclude="./$template" -cf - . | tar -C "$work/pack" -xf -
|
||||
tar -C "$leaf" --exclude=.git -cf - . | tar -C "$work/pack/$template" -xf -
|
||||
sed -i "/^ repos\/notes: {/ s|first_commit: \[[^]]*\]|first_commit: [$leaf_files]|" "$work/pack/pack.yaml"
|
||||
grep -q -F "first_commit: [$leaf_files]" "$work/pack/pack.yaml" || fail "the first commit of repos/notes could not be set in $work/pack/pack.yaml"
|
||||
realm="$work/pack"
|
||||
echo "smoke-real: the leaf repos/notes starts from $leaf: $realm is odin's pack with that leaf as its template, and every file in the first commit"
|
||||
fi
|
||||
|
||||
# The config: the example with its places filled. Its caps are the example's.
|
||||
text="$(<"$example")"
|
||||
@@ -97,26 +158,62 @@ text="${text//@TASKS@/$tasks}"
|
||||
text="${text//@SUITES@/$work/suites}"
|
||||
printf '%s\n' "$text" >"$work/factory.yaml"
|
||||
if grep -q '@[A-Z]*@' "$work/factory.yaml"; then fail "$work/factory.yaml still has a place to fill: $(grep -o '@[A-Z]*@' "$work/factory.yaml" | sort -u | tr '\n' ' ')"; fi
|
||||
if [ -n "$implementer" ]; then
|
||||
n="$(grep -c '^ implementer:' "$work/factory.yaml" || true)"
|
||||
[ "$n" = 1 ] || fail "$example has $n implementer line(s) under roles; --implementer needs exactly 1"
|
||||
sed -i "s|^ implementer:.*| implementer: { agent: $impl_agent, model: $impl_model } # set by ygg-smoke-real.sh --implementer|" "$work/factory.yaml"
|
||||
fi
|
||||
echo "smoke-real: the implementer: $(sed -n 's/^ implementer:[[:space:]]*\({[^}]*}\).*/\1/p' "$work/factory.yaml")"
|
||||
|
||||
ygg="${YGG_CMD:-}"
|
||||
# Which ygg runs. `ygg` has no version command, so the line gives the path and the modified time of the binary;
|
||||
# for a build from --odin it also gives the odin commit and whether that tree is dirty. YGG_CMD and a ygg on PATH
|
||||
# are run as they are: the line is what shows a stale one.
|
||||
ygg="${YGG_CMD:-}" built=0
|
||||
if [ -z "$ygg" ]; then
|
||||
if command -v ygg >/dev/null 2>&1; then ygg=ygg
|
||||
else
|
||||
mkdir -p "$work/bin"
|
||||
(cd "$odin" && go build -o "$work/bin/ygg" ./cmd/ygg) || fail "go build of ygg failed in $odin"
|
||||
ygg="'$work/bin/ygg'"
|
||||
ygg="'$work/bin/ygg'" built=1
|
||||
fi
|
||||
fi
|
||||
first="${ygg%% *}"
|
||||
if [ "$built" -eq 1 ]; then first="$work/bin/ygg"; fi
|
||||
bin="$(command -v -- "$first" 2>/dev/null || true)"
|
||||
if [ -n "$bin" ]; then
|
||||
stamp="$(date -r "$bin" '+%Y-%m-%d %H:%M:%S' 2>/dev/null || echo unknown)"
|
||||
ygg_line="smoke-real: ygg: $bin, modified $stamp"
|
||||
else
|
||||
ygg_line="smoke-real: ygg: $first (not found on PATH)"
|
||||
fi
|
||||
if [ "$built" -eq 1 ]; then
|
||||
sha="$(git -C "$odin" rev-parse --short HEAD 2>/dev/null || echo unknown)"
|
||||
if [ -n "$(git -C "$odin" status --porcelain 2>/dev/null)" ]; then tree="tree DIRTY"; else tree="tree clean"; fi
|
||||
ygg_line="$ygg_line, built from $odin at commit $sha, $tree"
|
||||
else
|
||||
ygg_line="$ygg_line, odin commit unknown (not built by this script)"
|
||||
fi
|
||||
echo "$ygg_line"
|
||||
|
||||
echo "smoke-real: the caps in $work/factory.yaml:"
|
||||
{ grep -E '^ per_(run|day):' "$work/factory.yaml" || true; } | sed 's/^/ /'
|
||||
echo "smoke-real: ygg doctor on that config"
|
||||
if ! eval "$ygg doctor --project '$project' --config '$work/factory.yaml'"; then fail "ygg doctor refused the config; nothing was spent"; fi
|
||||
|
||||
# What happens after the run. The dry run prints it from the same settings, since only a real run reaches it.
|
||||
after_run_notes() {
|
||||
if [ -n "$leaf" ]; then echo "smoke-real: after the run: ygg-repair-path.sh checks the repair path in the run's ledger (--from-leaf)"; fi
|
||||
if [ "$clip" -eq 1 ]; then
|
||||
if command -v clip.exe >/dev/null 2>&1; then echo "smoke-real: after the run: report.txt is copied to the clipboard with clip.exe"
|
||||
else echo "smoke-real: --clip needs clip.exe (WSL); the report will be in report.txt only"; fi
|
||||
fi
|
||||
}
|
||||
|
||||
run="$ygg run --project '$project' --config '$work/factory.yaml' --graph '$realm' --brief '$tasks/$task/instruction.md'"
|
||||
if [ "$dry" -eq 1 ]; then
|
||||
[ "$build" -eq 0 ] || echo "smoke-real: would run: $ygg sandbox build --config '$work/factory.yaml' --realm '$realm'"
|
||||
echo "smoke-real: would run: timeout --signal=INT --kill-after=2m $wall $run"
|
||||
after_run_notes
|
||||
echo "smoke-real: dry run: nothing was started and the credentials were not loaded"
|
||||
exit 0
|
||||
fi
|
||||
@@ -139,11 +236,12 @@ echo "smoke-real: the run ended with exit code $rc (0 done, 1 a node failed, 3 s
|
||||
|
||||
report="$work/report.txt"
|
||||
( { set +x; } 2>/dev/null # as above
|
||||
echo "$ygg_line"
|
||||
set -a
|
||||
# shellcheck source=/dev/null
|
||||
. "$envfile"
|
||||
set +a
|
||||
YGG_CMD="$ygg" "$HERE/ygg-evidence.sh" --config "$work/factory.yaml" --output "$out" ${clip[@]+"${clip[@]}"} "$project"
|
||||
YGG_CMD="$ygg" "$HERE/ygg-evidence.sh" --config "$work/factory.yaml" --output "$out" "$project"
|
||||
) | tee "$report" || echo "smoke-real: no evidence report: the run left no ledger" >&2
|
||||
|
||||
# The score and the cost come from the run folder and its ledger; neither holds a credential.
|
||||
@@ -154,7 +252,14 @@ else
|
||||
echo "smoke-real: no score.json: the run did not reach the grade node"
|
||||
fi
|
||||
if [ -n "$id" ] && [ -f "$project/.factory/runs/$id/ledger.jsonl" ]; then
|
||||
# Only a run from a planted-fault leaf is meant to take the repair path; on any other run the line would read as a failure.
|
||||
if [ -n "$leaf" ]; then "$HERE/ygg-repair-path.sh" "$project/.factory/runs/$id/ledger.jsonl" | tee -a "$report" || true; fi
|
||||
"$HERE/ygg-cost.sh" "$project/.factory/runs/$id/ledger.jsonl" "$work/factory.yaml" || echo "smoke-real: the cost could not be read from the ledger" >&2
|
||||
fi
|
||||
# The whole report.txt (the ygg line, the redacted evidence, the repair path) is what the founder pastes.
|
||||
if [ "$clip" -eq 1 ]; then
|
||||
if command -v clip.exe >/dev/null 2>&1 && [ -f "$report" ]; then clip.exe <"$report"; echo "(report.txt copied to the clipboard)" >&2
|
||||
else echo "smoke-real: --clip needs clip.exe (WSL); the report is in $report" >&2; fi
|
||||
fi
|
||||
echo "smoke-real: the folder is $work" >&2
|
||||
exit "$rc"
|
||||
Reference in new issue
Block a user