mirror of
https://github.com/ouroboric-dev/ouroboric.git
synced 2026-11-07 06:15:57 -10:00
Phase 6: journal-check.sh and the proof-run script (v0-T285, v0-T290, v0-T291) (#77)
* Check the journal: journal-check.sh and the factory-change tag (v0-T285) Co-Authored-By: Claude Haiku 5.5 <[email protected]> * journal-check compares the whole index row, refuses a duplicate, and reads quoted and block-style lists (v0-T285 review) Co-Authored-By: Claude Haiku 5.5 <[email protected]> * journal-check reads bare scalar lists and tags in any case (v0-T285 review) Co-Authored-By: Claude Haiku 5.5 <[email protected]> * The proof-run script: setup and the dry run (v0-T290) Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * The proof-run script resolves folders physically, never enters the suites folder, and refuses unsafe paths (v0-T290 review) Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * The proof-run script: the run, the grade and the evidence (v0-T291) Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * The proof-run script gives the credentials to the run only, survives any byte, keeps the run's exit code and passes signals on (v0-T291 review) Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * The proof-run script kills the run's whole process group at the wall clock, starts no run after a stop, and redacts or withholds (v0-T291 review) Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * The proof-run script never echoes a bad credentials line and stops the build on any signal (v0-T291 review) Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * The proof-run test fails if a Ctrl-C does not stop the build within five seconds (v0-T291 review) Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * The proof-run test never waits without a bound (v0-T291 review) Co-Authored-By: Claude Sonnet 5.5 <[email protected]> * The proof-run script survives a terminal's Ctrl-C and hang-up, compares across proof folders, continues a stopped run itself, and cleans up after an abnormal end (ouroboric review) Co-Authored-By: Claude Opus 5.5 <[email protected]> * The proof-run script keeps its report after a Ctrl-C and a closed terminal, and writes proof.meta private (ouroboric review) Co-Authored-By: Claude Opus 5.5 <[email protected]> * The proof-run script keeps only its own files private and runs ygg with the caller's umask (ouroboric review) Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Bruno Guilhermo de Barros Valério <[email protected]> Co-authored-by: Claude Haiku 5.5 <[email protected]>
This commit is contained in:
8 files changed
+1490
-1
No files matched your search
@@ -1,4 +1,7 @@
|
||||
extends: .yamllint.yaml
|
||||
|
||||
# The pieces are fragments indented to sit under repos: (the piece contract), not YAML documents.
|
||||
ignore: scripts/templates/ci/pieces/
|
||||
# The proof-run example is a template: its places between two at signs are filled by ygg-proof-run.sh.
|
||||
ignore: |
|
||||
scripts/templates/ci/pieces/
|
||||
scripts/testdata/proof/
|
||||
@@ -41,6 +41,7 @@ folder.
|
||||
| The grader smoke on a real engine (v0-H16), with the public sample suite | `scripts/ygg-smoke-grade.sh [--build]`. Never a hidden suite by default; `--task ID --suite-dir DIR --reference DIR [--suite-engine DIR]` points the grader at a suite store as it is (never copied or read) and grades the task's reference solution; its output holds numbers only |
|
||||
| The real-agent smoke (v0-H17): costs money; `--dry-run` first; the cost from the ledger; a start from a planted-fault leaf; another implementer; a note before the brief; the repair path from the ledger | `scripts/ygg-smoke-real.sh --dry-run`, then without it; `--from-leaf LEAF`, `--implementer AGENT:MODEL`, `--brief-note FILE` (narrows the implementer's task; the run is graded against a derived task version in its folder; the ygg line says if the build used go.work); `scripts/ygg-cost.sh LEDGER`, `scripts/ygg-repair-path.sh LEDGER`. It never prints the credentials file |
|
||||
| Compare reviewer models: costs money unless `--dry-run`; on the planted-fault leaves in `scripts/testdata/reviewer-bench`; runs the OpenCode version gleipnir pins (installed once with npm; `--opencode BIN`, `--any-opencode`) | `scripts/ygg-reviewer-bench.sh --model MODEL --leaf DIR --sessions N --dry-run`, then without it |
|
||||
| A Phase 6 proof run (v0-T290, v0-T291): costs money without `--dry-run`; `--dry-run` first; the `software` graph on the real task with three roles and caps you set, then the grade and the evidence; never reads the suites folder or prints the credentials | `scripts/ygg-proof-run.sh --name NAME --dry-run [--planner AGENT:MODEL --implementer AGENT:MODEL --reviewer AGENT:MODEL --run-usd N ...]`, then without `--dry-run`. `--build` builds the sandbox images first; `--baseline RUN_ID` compares with a run of an earlier proof run; `--continue RUN_ID --approve GATE` or `--resume` continues a stopped or paused run; `--clip` copies the report. The run's output is in `run.txt` and the evidence in `report.txt`, in `DIR/proof-NAME-TIME`. The example config is `scripts/testdata/proof/factory.proof.example.yaml` |
|
||||
| Build a plan from reference commits, or execute one task by task | `scripts/plan-*` (`plan-generate.py`, `plan-replay.py`, `plan-states.sh`, `plan-red.sh`, `plan-defences.py`; `plan-blocks.py`, `plan-task-start.sh`, `plan-task-done.sh`, `plan-task-land.sh`) |
|
||||
| Check that the repos have the required setup, and that the realms' shared files match | `scripts/as-bot.sh --read all scripts/check-repos.sh`; `--check shared` runs the realms check alone |
|
||||
| Scan git history for secrets | `scripts/scan-history.sh` |
|
||||
@@ -48,6 +49,7 @@ folder.
|
||||
| Write the licence files or the base CI into a folder | `scripts/add-licences.sh`, `scripts/add-ci.sh`. Founder only |
|
||||
| Push the prepared `base-ci` branches of existing repos and open their PRs, then require `lint` after the merge | `scripts/rollout-ci.sh --org <org> push\|protect <repo>...`. Founder only |
|
||||
| Prove that a Markdown change is formatting only, such as `markdownlint-cli2 --fix` output | `scripts/md-same-text.sh <base-ref> <head-ref>` |
|
||||
| Check the journal against spec §10.5 and design criterion 6: front matter, sections, the index, evidence once resolved, a `factory-change` entry citing `ygg metrics --compare` | `scripts/journal-check.sh [DIR]` |
|
||||
| Scaffold a new Go component | `scripts/scaffold-component.sh` |
|
||||
| Render and check diagrams | `scripts/diagrams.sh` |
|
||||
|
||||
|
||||
@@ -5,6 +5,14 @@ considered, what we chose, and the measured result. In the naming story these en
|
||||
**sagas**; a future content component (**Bragi**) will turn them into blog posts. Spec:
|
||||
[§13.5](https://github.com/ouroboric-dev/yggdrasil/blob/main/docs/2-engineering/specs/2026-10-11-yggdrasil-v0-spec.md#105-journey-journal).
|
||||
|
||||
## Checking
|
||||
|
||||
`scripts/journal-check.sh` reads this folder and fails when an entry breaks the rules above: a missing key, a status that
|
||||
is not `open`, `resolved` or `abandoned`, sections out of order, a resolved entry with no evidence or a Result that is still
|
||||
"pending", an entry that the index below does not list or lists differently, or a raw ledger line. An entry that records a
|
||||
change to the factory carries the tag `factory-change`; once it is resolved, its Result cites the before and after with
|
||||
`ygg metrics --compare <runA> <runB>` and names both run ids (design doc §2.3, criterion 6).
|
||||
|
||||
## Entries
|
||||
|
||||
| Date | Entry | Tags | Status | Post candidate |
|
||||
|
||||
Executable
+162
@@ -0,0 +1,162 @@
|
||||
#!/usr/bin/env bash
|
||||
# Check the journey journal against spec §10.5 and design success criterion 6.
|
||||
# journal-check.sh [JOURNAL_DIR] default: docs/6-journal beside this script's repo
|
||||
# Every entry (a file named YYYY-MM-DD-*.md) must have:
|
||||
# - front matter with date (the file name's), tags, status (open, resolved or abandoned), post-candidate (yes or
|
||||
# no) and evidence (a list, flow or block style; not empty once the entry is resolved);
|
||||
# - the sections Challenge, Context, Options, Decision, Result and Lessons, in that order (headings inside fenced
|
||||
# code blocks do not count);
|
||||
# - a Result that is not "pending" once the entry is resolved;
|
||||
# - one row in README.md with the same date, status and post candidate, and no row for a file that is not there;
|
||||
# - no raw ledger line (a private run's ledger is quoted by aggregate numbers only).
|
||||
# An entry tagged factory-change that is resolved or abandoned must cite its before and after in its Result: the
|
||||
# words `ygg metrics --compare` and two different run ids (criterion 6: a factory change cites `--compare`).
|
||||
# It reads and prints; it changes nothing. Exit 0 when the journal passes, 1 when a rule fails, 2 on bad usage.
|
||||
set -euo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
dir="${1:-$HERE/../docs/6-journal}"
|
||||
[ $# -le 1 ] || { echo "usage: journal-check.sh [JOURNAL_DIR]" >&2; exit 2; }
|
||||
[ -d "$dir" ] || { echo "journal-check: no journal at $dir" >&2; exit 2; }
|
||||
exec python3 -I - "$dir" <<'PY'
|
||||
import os, re, sys
|
||||
|
||||
jdir = sys.argv[1]
|
||||
SECTIONS = ["Challenge", "Context", "Options", "Decision", "Result", "Lessons"]
|
||||
STATUS = {"open", "resolved", "abandoned"}
|
||||
RUN = re.compile(r"\b\d{8}-\d{6}-[0-9a-f]{6}\b")
|
||||
FENCE = re.compile(r"^\s*(`{3,}|~{3,})")
|
||||
LEDGER = re.compile(r'^\s*(?:[-*]\s+)?\{\s*"schema"\s*:\s*"ygg\.ledger/', re.M)
|
||||
ROW = re.compile(r"^\| (\d{4}-\d\d-\d\d) \| \[[^\]]*\]\(([^)]+)\) \| [^|]*\| (\w+) \| (\w+) \|", re.M)
|
||||
problems = []
|
||||
|
||||
|
||||
def bad(file, msg):
|
||||
problems.append(f"{file}: {msg}")
|
||||
|
||||
|
||||
def clean(s):
|
||||
return s.strip().strip("\"'").strip()
|
||||
|
||||
|
||||
def parse_front(text):
|
||||
# Scalars go in fm; list items (block style: " - item" under a key) go in lists.
|
||||
fm, lists, cur = {}, {}, None
|
||||
for line in text.split("\n"):
|
||||
m = re.match(r"^([\w-]+):(.*)$", line)
|
||||
if m:
|
||||
cur = m.group(1)
|
||||
fm[cur] = m.group(2).strip()
|
||||
lists.setdefault(cur, [])
|
||||
continue
|
||||
m = re.match(r"^\s*-\s+(.*)$", line)
|
||||
if m and cur is not None:
|
||||
lists[cur].append(m.group(1))
|
||||
return fm, lists
|
||||
|
||||
|
||||
def items(fm, lists, key):
|
||||
# A list in flow style ([a, "b"]), a bare comma-separated scalar (a, b) or block style; quotes and spaces are
|
||||
# stripped from each item.
|
||||
v = fm.get(key, "")
|
||||
if v.startswith("["):
|
||||
raw = v[1:].rsplit("]", 1)[0].split(",")
|
||||
elif v:
|
||||
raw = v.split(",")
|
||||
else:
|
||||
raw = lists.get(key, [])
|
||||
return [c for c in (clean(x) for x in raw) if c]
|
||||
|
||||
|
||||
def visible(body):
|
||||
# The body with the headings inside fenced code blocks blanked, so they neither count as sections nor end Result.
|
||||
out, fence = [], None
|
||||
for line in body.split("\n"):
|
||||
m = FENCE.match(line)
|
||||
if fence is None:
|
||||
if m:
|
||||
fence = m.group(1)
|
||||
out.append(line)
|
||||
elif m and m.group(1)[0] == fence[0] and len(m.group(1)) >= len(fence) and line.strip() == m.group(1):
|
||||
fence = None
|
||||
out.append(line)
|
||||
else:
|
||||
out.append("" if re.match(r"^\s*#{1,6}\s", line) else line)
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
entries = sorted(f for f in os.listdir(jdir) if re.match(r"\d{4}-\d\d-\d\d-.*\.md$", f))
|
||||
facts = {}
|
||||
for f in entries:
|
||||
text = open(os.path.join(jdir, f), encoding="utf-8").read()
|
||||
m = re.match(r"---\n(.*?)\n---\n", text, re.S)
|
||||
if not m:
|
||||
bad(f, "front matter is missing")
|
||||
continue
|
||||
fm, lists = parse_front(m.group(1))
|
||||
for key in ("date", "tags", "status", "post-candidate", "evidence"):
|
||||
if key not in fm:
|
||||
bad(f, f"front matter lacks {key}")
|
||||
date, status, post = clean(fm.get("date", "")), clean(fm.get("status", "")), clean(fm.get("post-candidate", ""))
|
||||
if "status" in fm and status not in STATUS:
|
||||
bad(f, f"status '{status}' is not open, resolved or abandoned")
|
||||
if "post-candidate" in fm and post not in ("yes", "no"):
|
||||
bad(f, f"post-candidate '{post}' is not yes or no")
|
||||
if "date" in fm and date != f[:10]:
|
||||
bad(f, f"date {date} is not the file name's")
|
||||
body = text[m.end():]
|
||||
seen = visible(body)
|
||||
heads = re.findall(r"^## (\w+)\s*$", seen, re.M)
|
||||
for s in SECTIONS:
|
||||
if s not in heads:
|
||||
bad(f, f"section {s} is missing")
|
||||
present = [h for h in heads if h in SECTIONS]
|
||||
if all(s in heads for s in SECTIONS) and present != SECTIONS:
|
||||
bad(f, "the sections are not in the order " + ", ".join(SECTIONS))
|
||||
result = ""
|
||||
rm = re.search(r"^## Result\s*$(.*?)(?=^## |\Z)", seen, re.S | re.M)
|
||||
if rm:
|
||||
result = rm.group(1).strip()
|
||||
evidence = items(fm, lists, "evidence")
|
||||
if status == "resolved" and not evidence:
|
||||
bad(f, "resolved but evidence is empty")
|
||||
if status == "resolved" and result.lower().rstrip(".") == "pending":
|
||||
bad(f, "resolved but its Result is pending")
|
||||
tags = items(fm, lists, "tags")
|
||||
if "factory-change" in [t.lower() for t in tags] and status in ("resolved", "abandoned"):
|
||||
if "ygg metrics --compare" not in result:
|
||||
bad(f, "tagged factory-change but its Result does not cite ygg metrics --compare")
|
||||
elif len(set(RUN.findall(result))) < 2:
|
||||
bad(f, "its ygg metrics --compare citation does not name two different run ids")
|
||||
if LEDGER.search(body):
|
||||
bad(f, "holds a raw ledger line; quote aggregate numbers and link the run by id")
|
||||
facts[f] = (date, status, post)
|
||||
|
||||
index = os.path.join(jdir, "README.md")
|
||||
if os.path.isfile(index):
|
||||
rows = {}
|
||||
for d, link, status, post in ROW.findall(open(index, encoding="utf-8").read()):
|
||||
rows.setdefault(link, []).append((d, status, post))
|
||||
for f in entries:
|
||||
if f not in rows:
|
||||
bad(f, "not in README.md")
|
||||
continue
|
||||
if len(rows[f]) > 1:
|
||||
bad(f, f"has {len(rows[f])} rows in README.md")
|
||||
if f in facts:
|
||||
d, status, post = rows[f][0]
|
||||
fd, fs, fp = facts[f]
|
||||
if d != fd:
|
||||
bad(f, f"README.md row says date {d}, the entry says {fd}")
|
||||
if (status, post) != (fs, fp):
|
||||
bad(f, f"README.md says {status}/{post}, the entry says {fs}/{fp}")
|
||||
for link in rows:
|
||||
if link not in entries:
|
||||
bad("README.md", f"lists {link}, which is not there")
|
||||
else:
|
||||
bad("README.md", "is missing")
|
||||
|
||||
for p in problems:
|
||||
print(p)
|
||||
print(f"journal-check: {len(entries)} entries, {len(problems)} problem(s)")
|
||||
sys.exit(1 if problems else 0)
|
||||
PY
|
||||
@@ -0,0 +1,27 @@
|
||||
# The factory.yaml of a proof run (v0-H19 to v0-H23), with the places between two at signs that
|
||||
# scripts/ygg-proof-run.sh fills from its flags. It never writes your factory.yaml. The credentials come from the
|
||||
# environment, never from this file. A "usd" line is written only when a role pays per token (an opencode role):
|
||||
# the caps then count what the API billed, not the cost figure of a subscription's sessions.
|
||||
mode: @MODE@
|
||||
privacy: strict
|
||||
roles:
|
||||
planner: { agent: @PLANNER_AGENT@, model: @PLANNER_MODEL@ }
|
||||
implementer: { agent: @IMPLEMENTER_AGENT@, model: @IMPLEMENTER_MODEL@ }
|
||||
reviewer: { agent: @REVIEWER_AGENT@, model: @REVIEWER_MODEL@ }
|
||||
budget:
|
||||
per_run: { tokens: @RUN_TOKENS@, turns: @RUN_TURNS@@RUN_USD@ }
|
||||
per_day: { tokens: @DAY_TOKENS@, turns: @DAY_TURNS@@DAY_USD@ }
|
||||
sandbox:
|
||||
engine: @ENGINE@
|
||||
mounts:
|
||||
- { host: @FOLDER@, engine: @BIND@ }
|
||||
realms:
|
||||
dir: @REALMS@
|
||||
graphs:
|
||||
dir: @GRAPHS@
|
||||
grader:
|
||||
tasks_dir: @TASKS@
|
||||
suite_dir: @SUITES@
|
||||
timeout: 30m
|
||||
memory: none
|
||||
cpus: 0
|
||||
Executable
+176
@@ -0,0 +1,176 @@
|
||||
#!/usr/bin/env bash
|
||||
# Tests for journal-check.sh: a good journal passes, and each rule of the checker fails on a journal that breaks it.
|
||||
# scripts/tests/journal-check.test.sh Exit 0 when every case behaves.
|
||||
set -uo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
S="$HERE/.."
|
||||
T="$(mktemp -d)"
|
||||
trap 'rm -rf -- "$T"' EXIT
|
||||
fail=0
|
||||
check() {
|
||||
local d="$1"; shift
|
||||
if "$@"; then echo "ok $d"; else echo "FAIL $d"; fail=1; [ -z "${VERBOSE:-}" ] || sed 's/^/ | /' "$T/out"; fi
|
||||
}
|
||||
run() { "$@" >"$T/out" 2>&1; rc=$?; }
|
||||
# shellcheck disable=SC2329 # has runs through check
|
||||
has() { grep -q -F -- "$1" "$T/out"; }
|
||||
|
||||
RUN_A=20261023-090000-a1b2c3
|
||||
RUN_B=20261024-090000-d4e5f6
|
||||
|
||||
# entry FILE STATUS POST TAGS EVIDENCE RESULT: writes one entry with every section.
|
||||
entry() {
|
||||
local file="$1" status="$2" post="$3" tags="$4" evidence="$5" result="$6"
|
||||
local date="${file:0:10}"
|
||||
cat > "$J/$file" <<EOT
|
||||
---
|
||||
date: $date
|
||||
tags: [$tags]
|
||||
status: $status
|
||||
post-candidate: $post
|
||||
evidence: [$evidence]
|
||||
---
|
||||
## Challenge
|
||||
What went wrong.
|
||||
## Context
|
||||
What we knew.
|
||||
## Options
|
||||
What we considered.
|
||||
## Decision
|
||||
What we chose.
|
||||
## Result
|
||||
$result
|
||||
## Lessons
|
||||
What we would tell someone.
|
||||
EOT
|
||||
}
|
||||
index() { # index LINE...: writes README.md with the table header and the given rows
|
||||
{ echo "# Journal"; echo; echo "| Date | Entry | Tags | Status | Post candidate |"; echo "| --- | --- | --- | --- | --- |"; printf '%s\n' "$@"; } > "$J/README.md"
|
||||
}
|
||||
fresh() { J="$T/journal-$1"; mkdir -p "$J"; }
|
||||
|
||||
fresh good
|
||||
entry 2026-10-23-first.md resolved yes "harness, cost" '"run 20261023-090000-a1b2c3"' "Measured: it took 3 attempts, then 2."
|
||||
entry 2026-10-24-open.md open no "loop" "" "pending"
|
||||
entry 2026-10-25-change.md resolved yes "harness, factory-change" '"run 20261023-090000-a1b2c3", "run 20261024-090000-d4e5f6"' "ygg metrics --compare $RUN_A $RUN_B: first pass 50% to 75%, reward held."
|
||||
index "| 2026-10-23 | [first](2026-10-23-first.md) | harness, cost | resolved | yes |" \
|
||||
"| 2026-10-24 | [open](2026-10-24-open.md) | loop | open | no |" \
|
||||
"| 2026-10-25 | [change](2026-10-25-change.md) | harness, factory-change | resolved | yes |"
|
||||
run "$S/journal-check.sh" "$J"
|
||||
check "a good journal passes" test "$rc" -eq 0
|
||||
check "and says how many entries it read" has "3 entries"
|
||||
|
||||
# shellcheck disable=SC2329 # runs through check
|
||||
failed_with() { [ "$rc" = 1 ] && has "$1"; }
|
||||
bad() { # bad NAME EXPECT: run the checker on the current $J and expect exit 1 and a message with EXPECT
|
||||
run "$S/journal-check.sh" "$J"
|
||||
check "$1" failed_with "$2"
|
||||
}
|
||||
|
||||
fresh nofm; entry 2026-10-23-a.md open no "x" "" "pending"; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | open | no |"; sed -i '1d' "$J/2026-10-23-a.md"
|
||||
bad "an entry with no front matter fails" "2026-10-23-a.md: front matter"
|
||||
fresh nokey; entry 2026-10-23-a.md open no "x" "" "pending"; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | open | no |"; sed -i '/^post-candidate:/d' "$J/2026-10-23-a.md"
|
||||
bad "a missing key fails" "2026-10-23-a.md: front matter lacks post-candidate"
|
||||
fresh badstat; entry 2026-10-23-a.md "done" no "x" "" "pending"; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | done | no |"
|
||||
bad "a status that is not open, resolved or abandoned fails" "status 'done'"
|
||||
fresh baddate; entry 2026-10-23-a.md open no "x" "" "pending"; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | open | no |"; sed -i 's/^date: .*/date: 2026-10-22/' "$J/2026-10-23-a.md"
|
||||
bad "a date that is not the file name's fails" "date 2026-10-22 is not the file name's"
|
||||
fresh nosec; entry 2026-10-23-a.md open no "x" "" "pending"; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | open | no |"; sed -i '/^## Options$/,+1d' "$J/2026-10-23-a.md"
|
||||
bad "a missing section fails" "section Options"
|
||||
fresh order; entry 2026-10-23-a.md open no "x" "" "pending"; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | open | no |"; sed -i 's/^## Context$/## ContextX/;s/^## Options$/## Context/;s/^## ContextX$/## Options/' "$J/2026-10-23-a.md"
|
||||
bad "sections out of order fail" "sections are not in the order"
|
||||
fresh noevid; entry 2026-10-23-a.md resolved no "x" "" "Measured: fine."; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | resolved | no |"
|
||||
bad "a resolved entry with no evidence fails" "resolved but evidence is empty"
|
||||
fresh pend; entry 2026-10-23-a.md resolved no "x" '"run 20261023-090000-a1b2c3"' "pending"; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | resolved | no |"
|
||||
bad "a resolved entry whose result is pending fails" "resolved but its Result is pending"
|
||||
fresh noidx; entry 2026-10-23-a.md open no "x" "" "pending"; index
|
||||
bad "an entry missing from the index fails" "2026-10-23-a.md: not in README.md"
|
||||
fresh difidx; entry 2026-10-23-a.md open no "x" "" "pending"; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | resolved | yes |"
|
||||
bad "an index row that disagrees with the entry fails" "README.md says resolved/yes"
|
||||
fresh ghost; entry 2026-10-23-a.md open no "x" "" "pending"; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | open | no |" "| 2026-10-24 | [gone](2026-10-24-gone.md) | x | open | no |"
|
||||
bad "an index row for a file that is not there fails" "README.md: lists 2026-10-24-gone.md"
|
||||
fresh nocomp; entry 2026-10-23-a.md resolved yes "harness, factory-change" '"run 20261023-090000-a1b2c3"' "It got better."; index "| 2026-10-23 | [a](2026-10-23-a.md) | harness, factory-change | resolved | yes |"
|
||||
bad "a factory change with no ygg metrics --compare fails" "factory-change but its Result does not cite ygg metrics --compare"
|
||||
fresh onerun; entry 2026-10-23-a.md resolved yes "factory-change" '"run 20261023-090000-a1b2c3"' "ygg metrics --compare $RUN_A: better."; index "| 2026-10-23 | [a](2026-10-23-a.md) | factory-change | resolved | yes |"
|
||||
bad "a comparison that names one run fails" "two different run ids"
|
||||
fresh ledger; entry 2026-10-23-a.md open no "x" "" '{"schema":"ygg.ledger/v1","seq":1,"event":"run_started"}'; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | open | no |"
|
||||
bad "a raw ledger line in an entry fails" "a raw ledger line"
|
||||
|
||||
# write_entry FILE FRONT RESULT: an entry whose front matter is the given text, for the quoted and block-style cases.
|
||||
write_entry() {
|
||||
local file="$1" front="$2" result="$3"
|
||||
{ echo "---"; printf '%s\n' "$front"; echo "---"; echo "## Challenge"; echo "What went wrong."; echo "## Context"; echo "What we knew."; echo "## Options"; echo "What we considered."; echo "## Decision"; echo "What we chose."; echo "## Result"; echo "$result"; echo "## Lessons"; echo "What we would tell someone."; } > "$J/$file"
|
||||
}
|
||||
FRONT_BLOCK_TAG=$'date: 2026-10-23\ntags:\n - factory-change\nstatus: resolved\npost-candidate: yes\nevidence:\n - "run 20261023-090000-a1b2c3"'
|
||||
FRONT_BLOCK_EVID=$'date: 2026-10-23\ntags: [x]\nstatus: resolved\npost-candidate: no\nevidence:\n - "run 20261023-090000-a1b2c3"'
|
||||
|
||||
fresh idxdate; entry 2026-10-23-a.md open no "x" "" "pending"; index "| 2026-10-22 | [a](2026-10-23-a.md) | x | open | no |"
|
||||
bad "an index row with a different date fails" "README.md row says date 2026-10-22"
|
||||
fresh dup; entry 2026-10-23-a.md open no "x" "" "pending"; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | open | no |" "| 2026-10-23 | [a](2026-10-23-a.md) | x | open | no |"
|
||||
bad "a duplicate index row fails" "2026-10-23-a.md: has 2 rows in README.md"
|
||||
fresh qtag; entry 2026-10-23-a.md resolved yes '"factory-change"' '"run 20261023-090000-a1b2c3"' "It got better."; index "| 2026-10-23 | [a](2026-10-23-a.md) | factory-change | resolved | yes |"
|
||||
bad "a double-quoted factory-change tag still needs the comparison" "factory-change but its Result does not cite ygg metrics --compare"
|
||||
fresh qtag1; entry 2026-10-23-a.md resolved yes "'factory-change'" '"run 20261023-090000-a1b2c3"' "It got better."; index "| 2026-10-23 | [a](2026-10-23-a.md) | factory-change | resolved | yes |"
|
||||
bad "a single-quoted factory-change tag still needs the comparison" "factory-change but its Result does not cite ygg metrics --compare"
|
||||
fresh btag; write_entry 2026-10-23-a.md "$FRONT_BLOCK_TAG" "It got better."; index "| 2026-10-23 | [a](2026-10-23-a.md) | factory-change | resolved | yes |"
|
||||
bad "a block-style factory-change tag still needs the comparison" "factory-change but its Result does not cite ygg metrics --compare"
|
||||
fresh bev; write_entry 2026-10-23-a.md "$FRONT_BLOCK_EVID" "Measured: fine."; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | resolved | no |"
|
||||
run "$S/journal-check.sh" "$J"
|
||||
check "a block-style evidence list counts as evidence" test "$rc" -eq 0
|
||||
fresh fence; cat > "$J/2026-10-23-a.md" <<'EOT'
|
||||
---
|
||||
date: 2026-10-23
|
||||
tags: [x]
|
||||
status: open
|
||||
post-candidate: no
|
||||
evidence: []
|
||||
---
|
||||
## Challenge
|
||||
What went wrong.
|
||||
## Context
|
||||
An example from a note:
|
||||
```markdown
|
||||
## Result
|
||||
pending
|
||||
```
|
||||
What we knew.
|
||||
## Options
|
||||
What we considered.
|
||||
## Decision
|
||||
What we chose.
|
||||
## Result
|
||||
pending
|
||||
## Lessons
|
||||
What we would tell someone.
|
||||
EOT
|
||||
index "| 2026-10-23 | [a](2026-10-23-a.md) | x | open | no |"
|
||||
run "$S/journal-check.sh" "$J"
|
||||
check "a fenced heading inside Context does not break the order rule" test "$rc" -eq 0
|
||||
fresh ledgerind; entry 2026-10-23-a.md open no "x" "" ' {"schema":"ygg.ledger/v1","seq":1}'; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | open | no |"
|
||||
bad "an indented raw ledger line fails" "a raw ledger line"
|
||||
fresh ledgersp; entry 2026-10-23-a.md open no "x" "" '{"schema": "ygg.ledger/v1","seq":1}'; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | open | no |"
|
||||
bad "a raw ledger line with a space after the colon fails" "a raw ledger line"
|
||||
FRONT_BARE_TAG=$'date: 2026-10-23\ntags: factory-change\nstatus: resolved\npost-candidate: yes\nevidence: ["run 20261023-090000-a1b2c3"]'
|
||||
FRONT_BARE_TAGS=$'date: 2026-10-23\ntags: a, factory-change\nstatus: resolved\npost-candidate: yes\nevidence: ["run 20261023-090000-a1b2c3"]'
|
||||
FRONT_BARE_EVID=$'date: 2026-10-23\ntags: [x]\nstatus: resolved\npost-candidate: no\nevidence: run-x'
|
||||
|
||||
fresh btagbare; write_entry 2026-10-23-a.md "$FRONT_BARE_TAG" "It got better."; index "| 2026-10-23 | [a](2026-10-23-a.md) | factory-change | resolved | yes |"
|
||||
bad "a bare-scalar factory-change tag still needs the comparison" "factory-change but its Result does not cite ygg metrics --compare"
|
||||
fresh btagtwo; write_entry 2026-10-23-a.md "$FRONT_BARE_TAGS" "It got better."; index "| 2026-10-23 | [a](2026-10-23-a.md) | a, factory-change | resolved | yes |"
|
||||
bad "a comma-separated scalar tag list with factory-change still needs the comparison" "factory-change but its Result does not cite ygg metrics --compare"
|
||||
fresh tagcase; entry 2026-10-23-a.md resolved yes "Factory-Change" '"run 20261023-090000-a1b2c3"' "It got better."; index "| 2026-10-23 | [a](2026-10-23-a.md) | factory-change | resolved | yes |"
|
||||
bad "a Factory-Change tag in another case still needs the comparison" "factory-change but its Result does not cite ygg metrics --compare"
|
||||
fresh evscalar; write_entry 2026-10-23-a.md "$FRONT_BARE_EVID" "Measured: fine."; index "| 2026-10-23 | [a](2026-10-23-a.md) | x | resolved | no |"
|
||||
run "$S/journal-check.sh" "$J"
|
||||
check "a scalar evidence value counts as evidence" test "$rc" -eq 0
|
||||
|
||||
fresh ok2; entry 2026-10-23-a.md open no "factory-change" "" "pending"; index "| 2026-10-23 | [a](2026-10-23-a.md) | factory-change | open | no |"
|
||||
run "$S/journal-check.sh" "$J"
|
||||
check "an open factory-change entry may still be waiting for its comparison" test "$rc" -eq 0
|
||||
run "$S/journal-check.sh" "$T/no-such-folder"
|
||||
check "a folder that is not there is a usage error" test "$rc" -eq 2
|
||||
check "and the message says there is no journal" has "no journal"
|
||||
run "$S/journal-check.sh" "$S/../docs/6-journal"
|
||||
check "the real journal passes" test "$rc" -eq 0
|
||||
|
||||
exit "$fail"
|
||||
Executable
+597
@@ -0,0 +1,597 @@
|
||||
#!/usr/bin/env bash
|
||||
# Tests for ygg-proof-run.sh (v0-T290): the refusals, the filled factory.yaml, the dry run, the credentials by name
|
||||
# and the suites folder that is never read. scripts/tests/ygg-proof-run.test.sh
|
||||
# Each case uses a fake `ygg` (it records its arguments and answers doctor), a fake `devcontainer`, a tiny task, a
|
||||
# tiny suites folder and a fake credentials file: no engine, no model, no network. v0-T291: the fake ygg also
|
||||
# answers run, grade, metrics, report and status, and logs the credential names of its environment. The ouroboric
|
||||
# review: it also answers approve, resume and metrics --compare (which reads both runs from the one project it gets).
|
||||
# shellcheck disable=SC2329,SC2016 # check calls has and hasnt by name; the $ in the patterns is literal
|
||||
set -uo pipefail
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
S="$HERE/.."
|
||||
SCRIPT="$S/ygg-proof-run.sh"
|
||||
T="$(mktemp -d)"
|
||||
trap 'rm -rf -- "$T"' EXIT
|
||||
fail=0
|
||||
check() {
|
||||
local d="$1"; shift
|
||||
if "$@"; then echo "ok $d"; else echo "FAIL $d"; fail=1; [ -z "${VERBOSE:-}" ] || sed 's/^/ | /' "$T/out"; fi
|
||||
}
|
||||
# The script names its folder by the second, so each run that makes a project waits for the next second.
|
||||
run() { sleep 1; "$@" >"$T/out" 2>&1; rc=$?; }
|
||||
runq() { rm -rf "$D"/proof-*; "$@" >"$T/out" 2>&1; rc=$?; } # a refusal makes no project: no wait
|
||||
has() { grep -q -F -- "$1" "$T/out"; }
|
||||
hasnt() { ! grep -q -F -- "$1" "$T/out"; }
|
||||
export HOME="$T/home"; mkdir -p "$HOME"
|
||||
umask 022 # the caller's umask: the project and what ygg makes follow it; only the script's own files are private
|
||||
mode_is() { [ "$(stat -c %a "$2")" = "$1" ]; } # MODE PATH
|
||||
export YGG_ENGINE_CHECK=0
|
||||
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1
|
||||
export YGG_BIND="$T/bind"; mkdir -p "$YGG_BIND"
|
||||
export YGG_PROOF_TASK=tiny-task
|
||||
|
||||
D="$T/it"; mkdir -p "$D"
|
||||
printf 'sandbox:\n engine: unix:///run/user/1000/podman/podman.sock\n' >"$D/factory.yaml"
|
||||
# A tiny task (key names and format as mimir's task.toml), a tiny suites folder, a fake umbrella.
|
||||
TK="$T/tasks"; mkdir -p "$TK/tiny-task"
|
||||
printf '# brief\n' >"$TK/tiny-task/instruction.md"
|
||||
sha="$(sha256sum "$TK/tiny-task/instruction.md" | cut -d' ' -f1)"
|
||||
cat >"$TK/tiny-task/task.toml" <<EOF2
|
||||
schema_version = "1.4"
|
||||
[task]
|
||||
name = "ouroboric-dev/tiny-task"
|
||||
version = "1.0.0"
|
||||
[metadata]
|
||||
brief_sha256 = "$sha"
|
||||
suite = "v7"
|
||||
EOF2
|
||||
SU="$T/suites"; mkdir -p "$SU/tiny-task/v7"
|
||||
YG="$T/ygg-umbrella"; mkdir -p "$YG/voluspa" "$YG/midgard-react"
|
||||
# The fake credentials: made-up values that must never show up anywhere.
|
||||
ENVF="$T/factory.env"
|
||||
printf 'CLAUDE_CODE_OAUTH_TOKEN=fake-claude-VALUE-1\nOPENROUTER_API_KEY=fake-router-VALUE-2\n' >"$ENVF"
|
||||
# Fakes: ygg records its arguments, doctor exits 0; devcontainer answers a version.
|
||||
FB="$T/fakebin"; mkdir -p "$FB"
|
||||
export FAKE_DIR="$T"
|
||||
cat >"$FB/ygg" <<'EOF3'
|
||||
#!/bin/sh
|
||||
# Fake ygg (v0-T291): records its arguments and the credential NAMES in its environment, answers each subcommand.
|
||||
# FAKE_RUN_RC is the exit code of `run`; FAKE_PLANT is a value it prints from run, metrics, report, grade and status.
|
||||
# FAKE_CONT_RC is the exit code of `approve` and `resume`.
|
||||
echo "$*" >>"$FAKE_DIR/ygg.calls"
|
||||
printf '%s: %s\n' "$1" "$(env | grep -E '^(CLAUDE_CODE_OAUTH_TOKEN|OPENROUTER_API_KEY)=' | cut -d= -f1 | sort | tr '\n' ' ')" >>"$FAKE_DIR/ygg.env"
|
||||
proj="" prev=""
|
||||
for a in "$@"; do [ "$prev" = --project ] && proj="$a"; prev="$a"; done
|
||||
case "$1" in
|
||||
run)
|
||||
d="$proj/.factory/runs/RUN-1"; mkdir -p "$d"
|
||||
echo '{"event": "run_started"}' >"$d/ledger.jsonl"
|
||||
echo '{"pending":"ship-gate","pending_gate":"soft"}' >"$d/run.json"
|
||||
echo "the run runs"
|
||||
[ -z "${FAKE_PLANT:-}" ] || echo "run saw $FAKE_PLANT"
|
||||
[ -z "${FAKE_BAD:-}" ] || printf 'bad \377 byte %s\n' "${FAKE_PLANT:-}"
|
||||
# FAKE_CHILD: the run starts a child (SIGINT at its default, as a real tool's) that must not outlive it.
|
||||
if [ -n "${FAKE_CHILD:-}" ]; then
|
||||
python3 -I -c 'import os, signal, sys, time; signal.signal(signal.SIGINT, signal.SIG_DFL); open(sys.argv[1], "w").write(str(os.getpid())); time.sleep(20)' "$FAKE_DIR/child.pid" &
|
||||
fi
|
||||
if [ -n "${FAKE_SLEEP:-}" ]; then echo $$ >"$FAKE_DIR/run.pid"; exec sleep "$FAKE_SLEEP"; fi
|
||||
# FAKE_TRAP: the run shuts down on INT, HUP or TERM as ygg does: it prints for a second, then ends stopped (3).
|
||||
if [ -n "${FAKE_TRAP:-}" ]; then
|
||||
for s in INT HUP TERM; do
|
||||
trap "echo 'the run got $s; shutting down'; i=0; while [ \$i -lt 5 ]; do echo \"shutdown line \$i\"; sleep 0.2; i=\$((i+1)); done; echo 'the run shut down'; exit 3" "$s"
|
||||
done
|
||||
echo $$ >"$FAKE_DIR/run.pid"
|
||||
while :; do sleep 0.2; done
|
||||
fi
|
||||
exit "${FAKE_RUN_RC:-0}" ;;
|
||||
approve | resume)
|
||||
echo "$1 runs"
|
||||
[ -z "${FAKE_PLANT:-}" ] || echo "$1 saw $FAKE_PLANT"
|
||||
exit "${FAKE_CONT_RC:-0}" ;;
|
||||
metrics)
|
||||
# The comparison reads both runs from the one project it is given, as ygg does.
|
||||
if [ "$2" = --compare ]; then
|
||||
for r in "$3" "$4"; do
|
||||
[ -f "$proj/.factory/runs/$r/run.json" ] && [ -f "$proj/.factory/runs/$r/ledger.jsonl" ] || { echo "ygg: project $proj has no run $r" >&2; exit 2; }
|
||||
done
|
||||
echo "compare $3 to $4: $(cat "$proj/.factory/runs/$3/run.json")"
|
||||
exit 0
|
||||
fi
|
||||
echo "$1 output"
|
||||
[ -z "${FAKE_PLANT:-}" ] || echo "$1 leaked $FAKE_PLANT"
|
||||
exit 0 ;;
|
||||
grade | report | status)
|
||||
echo "$1 output"
|
||||
[ -z "${FAKE_PLANT:-}" ] || echo "$1 leaked $FAKE_PLANT"
|
||||
exit 0 ;;
|
||||
sandbox)
|
||||
if [ -n "${FAKE_BUILD_SLEEP:-}" ]; then echo $$ >"$FAKE_DIR/build.pid"; exec sleep "$FAKE_BUILD_SLEEP"; fi
|
||||
exit 0 ;;
|
||||
*) exit 0 ;;
|
||||
esac
|
||||
EOF3
|
||||
REALTIMEOUT="$(command -v timeout)"
|
||||
printf '#!/bin/sh\necho "$*" >>"%s/timeout.args"\nexec "%s" "$@"\n' "$T" "$REALTIMEOUT" >"$FB/timeout"
|
||||
chmod +x "$FB/timeout"
|
||||
printf '#!/bin/sh\necho 0.89.0\n' >"$FB/devcontainer"
|
||||
chmod +x "$FB/ygg" "$FB/devcontainer"
|
||||
export YGG_CMD="$FB/ygg" PATH="$FB:$PATH"
|
||||
|
||||
BASE=(--folder "$D" --env "$ENVF" --tasks "$TK" --suites "$SU" --yggdrasil "$YG")
|
||||
proof() { "$SCRIPT" "${BASE[@]}" "$@"; }
|
||||
made() { find "$D" -maxdepth 1 -name 'proof-*' | head -n 1; }
|
||||
refuses() { [ "$rc" -eq 2 ] && has "$1" && [ -z "$(made)" ]; }
|
||||
|
||||
check "the script is executable" test -x "$SCRIPT"
|
||||
check "the example is there" test -f "$S/testdata/proof/factory.proof.example.yaml"
|
||||
|
||||
# --- refusals: exit 2, a reason, no project made
|
||||
runq "$SCRIPT" "${BASE[@]}"; check "no --name is refused" refuses "--name"
|
||||
runq proof --name Bad_Name; check "a name outside a-z0-9- is refused" refuses "a-z0-9-"
|
||||
runq proof --name ok --mode wild; check "a bad mode is refused" refuses "--mode"
|
||||
runq proof --name ok --planner nonsense; check "a role without a colon is refused" refuses "--planner"
|
||||
runq proof --name ok --implementer gemini:flash; check "a role with another agent is refused" refuses "--implementer"
|
||||
runq proof --name ok --reviewer opencode:qwen/x; check "an opencode model without openrouter/ is refused" refuses "--reviewer"
|
||||
runq proof --name ok --reviewer 'opencode:openrouter/a b'; check "a model with a space is refused" refuses "--reviewer"
|
||||
runq proof --name ok --planner 'claude-code:x
|
||||
y: z'; check "a model with a newline is refused" refuses "--planner"
|
||||
runq proof --name ok --run-turns many; check "a cap that is not a number is refused" refuses "--run-turns"
|
||||
runq proof --name ok --day-usd 1x; check "a usd cap that is not a number is refused" refuses "--day-usd"
|
||||
runq "$SCRIPT" --folder "$D" --env "$ENVF" --tasks "$T/nope" --suites "$SU" --yggdrasil "$YG" --name ok
|
||||
check "a missing task folder is refused" refuses "no task"
|
||||
cp -R "$TK" "$T/tasks2"; echo changed >>"$T/tasks2/tiny-task/instruction.md"
|
||||
runq "$SCRIPT" --folder "$D" --env "$ENVF" --tasks "$T/tasks2" --suites "$SU" --yggdrasil "$YG" --name ok
|
||||
check "a changed brief is refused as not the frozen brief" refuses "not the frozen brief"
|
||||
mkdir -p "$T/emptysu"
|
||||
runq "$SCRIPT" --folder "$D" --env "$ENVF" --tasks "$TK" --suites "$T/emptysu" --yggdrasil "$YG" --name ok
|
||||
check "a suites folder without the task's suite is refused" refuses "tiny-task/v7"
|
||||
mkdir -p "$TK/inside/tiny-task/v7"
|
||||
runq "$SCRIPT" --folder "$D" --env "$ENVF" --tasks "$TK" --suites "$TK/inside" --yggdrasil "$YG" --name ok
|
||||
check "a suites folder inside the tasks folder is refused" refuses "inside"
|
||||
mkdir -p "$SU/tasksin"; cp -R "$TK/tiny-task" "$SU/tasksin/"
|
||||
runq "$SCRIPT" --folder "$D" --env "$ENVF" --tasks "$SU/tasksin" --suites "$SU" --yggdrasil "$YG" --name ok
|
||||
check "a tasks folder inside the suites folder is refused" refuses "inside"
|
||||
runq "$SCRIPT" --folder "$D" --env "$ENVF" --tasks "$TK" --suites "$D/hidden" --yggdrasil "$YG" --name ok
|
||||
check "a missing suites folder (and so not a hidden one in the project area) is refused" refuses "tiny-task/v7"
|
||||
mkdir -p "$T/p-su/tiny-task/v7"
|
||||
printf 'CLAUDE_CODE_OAUTH_TOKEN=\nOPENROUTER_API_KEY=fake-router-VALUE-2\n' >"$T/env-empty"
|
||||
runq proof --name ok --env "$T/env-empty"; check "a credentials file lacking a needed name is refused" refuses "CLAUDE_CODE_OAUTH_TOKEN"
|
||||
check "the refusal does not print the other value" hasnt "fake-router-VALUE-2"
|
||||
runq proof --name ok --env "$T/missing.env"; check "a missing credentials file is refused" refuses "credentials file"
|
||||
|
||||
# --- review round 1: physical paths, unsafe paths, a model ending in a colon
|
||||
ln -s "$TK/inside" "$T/sl-suites"
|
||||
runq "$SCRIPT" --folder "$D" --env "$ENVF" --tasks "$TK" --suites "$T/sl-suites" --yggdrasil "$YG" --name ok
|
||||
check "a suites folder that is a symlink into the tasks folder is refused" refuses "inside"
|
||||
ln -s "$SU/tasksin" "$T/sl-tasks"
|
||||
runq "$SCRIPT" --folder "$D" --env "$ENVF" --tasks "$T/sl-tasks" --suites "$SU" --yggdrasil "$YG" --name ok
|
||||
check "a tasks folder that is a symlink into the suites folder is refused" refuses "inside"
|
||||
for odd in "a #b" "a, b" "a: b" "a}b" "a'b"; do
|
||||
mkdir -p "$T/$odd"; cp "$D/factory.yaml" "$T/$odd/"
|
||||
runq "$SCRIPT" --folder "$T/$odd" --env "$ENVF" --tasks "$TK" --suites "$SU" --yggdrasil "$YG" --name ok
|
||||
check "a --folder with '$odd' is refused, naming the flag" refuses "--folder"
|
||||
done
|
||||
runq "$SCRIPT" --folder "$D" --env "$ENVF" --tasks "$TK" --suites "$SU" --yggdrasil "$T/a, b" --name ok
|
||||
check "a --yggdrasil with a comma and space is refused, naming the flag" refuses "--yggdrasil"
|
||||
runq "$SCRIPT" --folder "$D" --env "$ENVF" --tasks "$T/a #b" --suites "$SU" --yggdrasil "$YG" --name ok
|
||||
check "a --tasks with ' #' is refused, naming the flag" refuses "--tasks"
|
||||
runq "$SCRIPT" --folder "$D" --env "$ENVF" --tasks "$TK" --suites "$T/a: b" --yggdrasil "$YG" --name ok
|
||||
check "a --suites with ': ' is refused, naming the flag" refuses "--suites"
|
||||
runq proof --name ok --planner claude-code:sonnet:
|
||||
check "a model that ends in a colon is refused" refuses "--planner"
|
||||
check "the script never enters the suites folder with cd" bash -c '! grep -E "(^|[^a-z_])cd[^#]*suites" "$1"' _ "$SCRIPT"
|
||||
check "the script never evals" bash -c '! grep -E "^[^#]*\beval \"\$ygg" "$1"' _ "$SCRIPT"
|
||||
|
||||
# --- B-130: only the names the roles need
|
||||
printf 'CLAUDE_CODE_OAUTH_TOKEN=fake-claude-VALUE-1\n' >"$T/env-claude"
|
||||
printf 'OPENROUTER_API_KEY=fake-router-VALUE-2\n' >"$T/env-router"
|
||||
AC=(--planner claude-code:sonnet --implementer claude-code:sonnet --reviewer claude-code:opus)
|
||||
AO=(--planner opencode:openrouter/a/b --implementer opencode:openrouter/c/d --reviewer opencode:openrouter/e/f)
|
||||
run proof --name allclaude --dry-run --env "$T/env-claude" "${AC[@]}"
|
||||
check "all-Claude without the OpenRouter key goes through" test "$rc" -eq 0
|
||||
check "all-Claude does not ask for OPENROUTER_API_KEY" hasnt OPENROUTER_API_KEY
|
||||
run proof --name allopen --dry-run --env "$T/env-router" "${AO[@]}"
|
||||
check "all-OpenCode without the Claude token goes through" test "$rc" -eq 0
|
||||
check "all-OpenCode does not ask for CLAUDE_CODE_OAUTH_TOKEN" hasnt CLAUDE_CODE_OAUTH_TOKEN
|
||||
runq proof --name mixed --env "$T/env-claude"
|
||||
check "the default mix without the OpenRouter key is refused" refuses "OPENROUTER_API_KEY"
|
||||
|
||||
# --- the filled factory.yaml: defaults (mixed roles, so a usd line), then all-Claude (no usd line)
|
||||
rm -rf "$D"/proof-*
|
||||
run proof --name first --dry-run
|
||||
check "a dry run with the defaults exits 0" test "$rc" -eq 0
|
||||
P="$(made)"; FY="$P/factory.yaml"
|
||||
check "the project is a git repository" test -d "$P/project/.git"
|
||||
check "the umbrella.yaml sets root: true" grep -q '^root: true$' "$P/project/umbrella.yaml"
|
||||
check "the project folder is DIR/proof-<name>-<time>/project" bash -c '[[ $1 =~ /proof-first-[0-9]{8}-[0-9]{6}$ ]]' _ "$P"
|
||||
check "the config has the default mode" grep -q '^mode: key-points$' "$FY"
|
||||
check "the config has the planner" grep -q 'planner: { agent: claude-code, model: sonnet }' "$FY"
|
||||
check "the config has the reviewer" grep -q 'reviewer: { agent: opencode, model: openrouter/qwen/qwen3.8-27b }' "$FY"
|
||||
check "the config has the run caps with usd" grep -q 'per_run: { tokens: 12000000, turns: 1500, usd: 5 }' "$FY"
|
||||
check "the config has the day caps with usd" grep -q 'per_day: { tokens: 24000000, turns: 2000, usd: 10 }' "$FY"
|
||||
check "the config has the tasks folder" grep -q -x " tasks_dir: $TK" "$FY"
|
||||
check "the config has the suites folder" grep -q -x " suite_dir: $SU" "$FY"
|
||||
check "the config has the realms and graphs" grep -q "dir: $YG/voluspa" "$FY"
|
||||
check "the config has the engine" grep -q 'engine: unix:///run/user/1000/podman/podman.sock' "$FY"
|
||||
check "no place is left to fill" bash -c '! grep -q "@[A-Z_]*@" "$1"' _ "$FY"
|
||||
check "proof.meta is private (600)" bash -c '[ "$(stat -c %a "$1")" = 600 ]' _ "$P/proof.meta"
|
||||
check "no credential value is in the config" bash -c '! grep -r -q "VALUE" "$1"' _ "$P"
|
||||
check "no credential value is in the output" bash -c '! grep -q "VALUE" "$1"' _ "$T/out"
|
||||
check "the plan names the graph" has "--graph"
|
||||
rm -rf "$D"/proof-*
|
||||
run proof --name second --dry-run --mode unattended --run-turns 10 --run-tokens 20 --day-turns 30 --day-tokens 40 "${AC[@]}"
|
||||
P="$(made)"; FY="$P/factory.yaml"
|
||||
check "the mode asked for is in the config" grep -q '^mode: unattended$' "$FY"
|
||||
check "the caps asked for are in the config" grep -q 'per_run: { tokens: 20, turns: 10 }' "$FY"
|
||||
check "the day caps asked for are in the config" grep -q 'per_day: { tokens: 40, turns: 30 }' "$FY"
|
||||
check "no usd line when no role is paid" bash -c '! grep -q "usd:" "$1"' _ "$FY"
|
||||
check "the reviewer asked for is in the config" grep -q 'reviewer: { agent: claude-code, model: opus }' "$FY"
|
||||
|
||||
# --- the dry run
|
||||
rm -rf "$D"/proof-* "$T/ygg.calls"
|
||||
run proof --name dry --dry-run
|
||||
check "doctor ran" grep -q '^doctor ' "$T/ygg.calls"
|
||||
check "ygg run never ran" bash -c '! grep -q "^run" "$1"' _ "$T/ygg.calls"
|
||||
check "the dry run says nothing was started and the credentials were not loaded" has "nothing was started and the credentials were not loaded"
|
||||
check "doctor was given the project and the config" grep -q -- "--project .*/project --config .*/factory.yaml" "$T/ygg.calls"
|
||||
|
||||
# --- the suites folder is never listed or read
|
||||
SH="$T/shims"; mkdir -p "$SH"; : >"$T/shim.log"
|
||||
for tool in ls find cat head tail grep sed awk du stat; do
|
||||
real="$(command -v "$tool")"
|
||||
printf '#!/bin/sh\nprintf "%%s\\n" "$*" >>"%s"\nexec "%s" "$@"\n' "$T/shim.log" "$real" >"$SH/$tool"
|
||||
chmod +x "$SH/$tool"
|
||||
done
|
||||
rm -rf "$D"/proof-*
|
||||
PATH="$SH:$PATH" run proof --name shims --dry-run
|
||||
check "the shimmed dry run exits 0" test "$rc" -eq 0
|
||||
check "the shims were called" test -s "$T/shim.log"
|
||||
check "no reader was called with the suites folder" bash -c '! grep -q -F -- "$1" "$2"' _ "$SU" "$T/shim.log"
|
||||
check "no reader was called with a folder inside the suites folder" bash -c '! grep -q -F -- "tiny-task/v7" "$1"' _ "$T/shim.log"
|
||||
|
||||
# --- v0-T291: the run, the grade and the evidence
|
||||
calls() { grep -v '^status ' "$T/ygg.calls" | cut -d' ' -f1 | paste -sd, -; }
|
||||
fresh() { rm -rf "$D"/proof-* "$T/ygg.calls" "$T/ygg.env"; }
|
||||
runit() { fresh; FAKE_RUN_RC="$1" run proof --name go "${@:2}"; }
|
||||
RUND() { find "$D" -maxdepth 1 -name 'proof-go-*' | head -n 1; }
|
||||
runit 0
|
||||
check "exit 0: the script exits 0" test "$rc" -eq 0
|
||||
check "exit 0: doctor, run, grade, metrics, report in that order" test "$(calls)" = "doctor,run,grade,metrics,report"
|
||||
check "run.txt holds the run's output" grep -q 'the run runs' "$(RUND)/run.txt"
|
||||
check "report.txt holds the metrics and the report" bash -c 'grep -q "metrics output" "$1" && grep -q "report output" "$1"' _ "$(RUND)/report.txt"
|
||||
check "the run's output was shown" has "the run runs"
|
||||
check "the exit code is printed" has "exit code 0"
|
||||
runit 5
|
||||
check "exit 5: the script exits 5" test "$rc" -eq 5
|
||||
check "exit 5: doctor, run, grade, metrics, report" test "$(calls)" = "doctor,run,grade,metrics,report"
|
||||
for c in 1 3 4 124; do
|
||||
runit "$c"
|
||||
check "exit $c: the script exits $c" test "$rc" -eq "$c"
|
||||
case "$c" in
|
||||
3 | 4) check "exit $c: no grade, but metrics and report" test "$(calls)" = "doctor,run,metrics,report" ;;
|
||||
*) check "exit $c: no grade; doctor after the abnormal end, then metrics and report" test "$(calls)" = "doctor,run,doctor,metrics,report" ;;
|
||||
esac
|
||||
check "exit $c: the exit code is printed" has "exit code $c"
|
||||
done
|
||||
runit 3
|
||||
check "exit 3: the output ends with the script's own --continue --approve command" bash -c 'tail -n 1 "$1" | grep -q -F "ygg-proof-run.sh --continue RUN-1 --approve ship-gate"' _ "$T/out"
|
||||
check "exit 3: the --continue command names the folder and the credentials file" bash -c 'tail -n 1 "$1" | grep -q -F -- "--folder $2" && tail -n 1 "$1" | grep -q -F -- "--env $3"' _ "$T/out" "$D" "$ENVF"
|
||||
check "exit 3: the raw approve command is still there, as a fallback" bash -c 'grep -F "ygg-engine.sh run" "$1" | grep -q -F "approve RUN-1 ship-gate --project"' _ "$T/out"
|
||||
runit 4
|
||||
check "exit 4: the output ends with the script's own --continue --resume command" bash -c 'tail -n 1 "$1" | grep -q -F "ygg-proof-run.sh --continue RUN-1 --resume"' _ "$T/out"
|
||||
check "exit 4: the raw resume command is still there, as a fallback" bash -c 'grep -F "ygg-engine.sh run" "$1" | grep -q -F "resume RUN-1 --project"' _ "$T/out"
|
||||
runit 0 --mode unattended
|
||||
RL="$(grep '^run ' "$T/ygg.calls")"
|
||||
check "run gets the task's instruction.md as the brief" bash -c 'grep -q -F -- "--brief $1/tiny-task/instruction.md" <<<"$2"' _ "$TK" "$RL"
|
||||
check "run gets --graph software" bash -c 'grep -q -F -- "--graph software" <<<"$1"' _ "$RL"
|
||||
check "run gets the mode asked for" bash -c 'grep -q -F -- "--mode unattended" <<<"$1"' _ "$RL"
|
||||
check "run gets the filled config and the project" bash -c 'grep -q -E -- "--config [^ ]*/proof-go-[0-9-]+/factory.yaml" <<<"$1" && grep -q -E -- "--project [^ ]*/project" <<<"$1"' _ "$RL"
|
||||
check "no credential value is in any argument" bash -c '! grep -q VALUE "$1"' _ "$T/ygg.calls"
|
||||
check "run has both credentials in its environment" grep -q -x 'run: CLAUDE_CODE_OAUTH_TOKEN OPENROUTER_API_KEY ' "$T/ygg.env"
|
||||
check "doctor (the script's own shell) has no credential" grep -q -x 'doctor: ' "$T/ygg.env"
|
||||
# A baseline is a run of an earlier proof run, in its own proof folder and so its own project.
|
||||
OLD=20200101-000000-abcdef
|
||||
plant_base() { # [ID [FOLDER]]: a finished run in another proof folder
|
||||
local b="$D/${2:-proof-old-20200101-000000}/project/.factory/runs/${1:-$OLD}"
|
||||
mkdir -p "$b"
|
||||
echo '{"pending":"","marker":"BASELINE-MARK"}' >"$b/run.json"
|
||||
echo '{"event": "run_started"}' >"$b/ledger.jsonl"
|
||||
}
|
||||
fresh; plant_base; FAKE_RUN_RC=0 run proof --name go --baseline "$OLD"
|
||||
check "--baseline: metrics --compare follows metrics, on a project that holds both runs" test "$(grep -E '^metrics' "$T/ygg.calls" | paste -sd'|' -)" = "metrics RUN-1 --project $(RUND)/project|metrics --compare $OLD RUN-1 --project $(RUND)/compare"
|
||||
check "--baseline: the baseline in another proof folder is found and compared" grep -q -F "compare $OLD to RUN-1: {\"pending\":\"\",\"marker\":\"BASELINE-MARK\"}" "$(RUND)/report.txt"
|
||||
check "--baseline: the comparison is printed as given, with no verdict of the script" bash -c '! grep -q -i -E "better|worse|improv|verdict" "$1"' _ "$T/out"
|
||||
fresh; plant_base; FAKE_RUN_RC=3 run proof --name go --baseline "$OLD"
|
||||
check "--baseline with exit 3: the output still ends with the --continue command" bash -c 'tail -n 1 "$1" | grep -q -F -- "--continue RUN-1 --approve ship-gate"' _ "$T/out"
|
||||
fresh
|
||||
FAKE_PLANT=fake-claude-VALUE-1 FAKE_RUN_RC=0 run proof --name go
|
||||
check "a planted credential value is not in the printed output" hasnt "fake-claude-VALUE-1"
|
||||
check "a planted credential value is not in report.txt" bash -c '! grep -q -F fake-claude-VALUE-1 "$1"' _ "$(RUND)/report.txt"
|
||||
check "a planted credential value is not in run.txt" bash -c '! grep -q -F fake-claude-VALUE-1 "$1"' _ "$(RUND)/run.txt"
|
||||
check "the planted value was replaced, not dropped" grep -q 'REDACTED' "$(RUND)/report.txt"
|
||||
fresh
|
||||
FAKE_RUN_RC=0 run proof --name go --build
|
||||
check "--build: sandbox build --all --config runs after doctor and before the run" test "$(calls)" = "doctor,sandbox,run,grade,metrics,report"
|
||||
check "--build: the build call has --all and --config" grep -q -E '^sandbox build --all --config [^ ]*/factory.yaml$' "$T/ygg.calls"
|
||||
fresh
|
||||
FAKE_RUN_RC=0 run proof --name go --dry-run
|
||||
check "dry run: run and the build still never run" bash -c '! grep -q "^run\|^grade" "$1"' _ "$T/ygg.calls"
|
||||
fresh
|
||||
PATH="$SH:$PATH" FAKE_RUN_RC=0 run proof --name go --build
|
||||
check "the shimmed full run exits 0" test "$rc" -eq 0
|
||||
check "the shimmed full run called the readers" test -s "$T/shim.log"
|
||||
check "the full run called no reader with the suites folder" bash -c '! grep -q -F -- "$1" "$2"' _ "$SU" "$T/shim.log"
|
||||
check "the full run called no reader with a folder inside the suites folder" bash -c '! grep -q -F -- "tiny-task/v7" "$1"' _ "$T/shim.log"
|
||||
|
||||
# --- v0-T291 review: least privilege, any byte, the run's exit code, signals, umask
|
||||
runit 0
|
||||
check "least privilege: only run has the credentials" bash -c 'grep -q -x "run: CLAUDE_CODE_OAUTH_TOKEN OPENROUTER_API_KEY " "$1" && ! grep -E "^(grade|metrics|report|status|doctor): ." "$1"' _ "$T/ygg.env"
|
||||
check "least privilege: grade, metrics, report and status ran without them" bash -c 'for s in grade metrics report status; do grep -q -x "$s: " "$1" || exit 1; done' _ "$T/ygg.env"
|
||||
fresh; plant_base; FAKE_RUN_RC=0 run proof --name go --build --baseline "$OLD"
|
||||
check "least privilege: sandbox build and metrics --compare ran without them" bash -c 'grep -q -x "sandbox: " "$1" && ! grep -E "^(sandbox|metrics): ." "$1"' _ "$T/ygg.env"
|
||||
fresh
|
||||
LC_ALL=C.UTF-8 FAKE_BAD=1 FAKE_PLANT=fake-claude-VALUE-1 FAKE_RUN_RC=3 run proof --name go
|
||||
check "an invalid UTF-8 byte does not stop the run: its exit code 3 is kept" test "$rc" -eq 3
|
||||
check "an invalid byte: the planted value is gone from run.txt" bash -c '! grep -a -q -F fake-claude-VALUE-1 "$1"' _ "$(RUND)/run.txt"
|
||||
check "an invalid byte: the byte survives in run.txt" python3 -I -c 'import sys; sys.exit(0 if b"bad \xff byte [REDACTED]" in open(sys.argv[1],"rb").read() else 1)' "$(RUND)/run.txt"
|
||||
check "an invalid byte: the report is still made" grep -q 'metrics output' "$(RUND)/report.txt"
|
||||
# A filter that fails: the run's exit code and the hint stay.
|
||||
FP="$T/failpy"; mkdir -p "$FP"; REALPY="$(command -v python3)"
|
||||
printf '#!/bin/sh\ncase "$*" in *"-u -c"*) exit 1 ;; esac\nexec "%s" "$@"\n' "$REALPY" >"$FP/python3"; chmod +x "$FP/python3"
|
||||
fresh
|
||||
PATH="$FP:$PATH" FAKE_PLANT=fake-claude-VALUE-1 FAKE_RUN_RC=3 run proof --name go
|
||||
check "a failing output filter: the script still exits with the run's code 3" test "$rc" -eq 3
|
||||
check "a failing output filter: the output ends with the --continue command" bash -c 'tail -n 1 "$1" | grep -q -F -- "--continue RUN-1 --approve ship-gate"' _ "$T/out"
|
||||
check "a failing output filter: no credential value is printed" hasnt "fake-claude-VALUE-1"
|
||||
check "a failing output filter: the metrics call still happened" grep -q '^metrics ' "$T/ygg.calls"
|
||||
# Signals: TERM to the script reaches the run.
|
||||
fresh; rm -f "$T/run.pid"
|
||||
sleep 1
|
||||
FAKE_SLEEP=20 FAKE_RUN_RC=0 "$SCRIPT" "${BASE[@]}" --name go >"$T/out" 2>&1 &
|
||||
spid=$!
|
||||
for _ in $(seq 1 50); do [ -s "$T/run.pid" ] && break; sleep 0.1; done
|
||||
rpid="$(cat "$T/run.pid" 2>/dev/null || echo 0)"
|
||||
check "the fake run is running before the signal" kill -0 "$rpid"
|
||||
kill -TERM "$spid" 2>/dev/null
|
||||
for _ in $(seq 1 50); do kill -0 "$spid" 2>/dev/null || break; sleep 0.1; done
|
||||
check "TERM to the script ends the script" bash -c '! kill -0 "$1" 2>/dev/null' _ "$spid"
|
||||
check "TERM to the script ends the run too" bash -c '! kill -0 "$1" 2>/dev/null' _ "$rpid"
|
||||
kill -KILL "$spid" "$rpid" 2>/dev/null; wait "$spid" 2>/dev/null
|
||||
check "the run's files are private (600)" mode_is 600 "$(RUND)/run.txt"
|
||||
# Only the script's own files are private: the project and what ygg makes in it keep the caller's umask (022 here).
|
||||
runit 0
|
||||
check "umask: the project folder follows the caller's umask (755)" mode_is 755 "$(RUND)/project"
|
||||
check "umask: the run folder ygg made follows it (755)" mode_is 755 "$(RUND)/project/.factory/runs/RUN-1"
|
||||
check "umask: a file ygg made follows it (644)" mode_is 644 "$(RUND)/project/.factory/runs/RUN-1/run.json"
|
||||
check "umask: the project's git files follow it (644)" mode_is 644 "$(RUND)/project/umbrella.yaml"
|
||||
for f in proof.meta run.txt report.txt run.rc; do check "umask: the script's own $f is private (600)" mode_is 600 "$(RUND)/$f"; done
|
||||
check "the exit line labels 137 and 141" bash -c 'grep -q "137" "$1" && grep -q "141" "$1"' _ "$SCRIPT"
|
||||
|
||||
# --- v0-T291 review round 2: the process group, a stop before the run, a filter that fails closed
|
||||
rm -f "$T/timeout.args"; fresh
|
||||
FAKE_RUN_RC=0 run proof --name go --dry-run
|
||||
DRYL="$(grep 'would run: timeout' "$T/out" | head -n 1)"
|
||||
check "the dry run shows the timeout command without --foreground" bash -c '[[ $1 == *"timeout --signal=INT --kill-after=2m 8h "* && $1 != *foreground* ]]' _ "$DRYL"
|
||||
fresh
|
||||
FAKE_RUN_RC=0 run proof --name go
|
||||
check "the timeout that ran is the one the dry run showed" test "$(head -n 1 "$T/timeout.args" | cut -d' ' -f1-3)" = "--signal=INT --kill-after=2m 8h"
|
||||
check "timeout runs without --foreground (it gets its own process group)" bash -c '! grep -q foreground "$1"' _ "$T/timeout.args"
|
||||
check "the script header shows the same timeout command" bash -c 'grep -q -F "timeout --signal=INT --kill-after=2m WALL" "$1" && ! grep -q "timeout --foreground" "$1"' _ "$SCRIPT"
|
||||
# TERM while the sandbox build sleeps: the run is never started.
|
||||
fresh; rm -f "$T/build.pid" "$T/run.pid"; sleep 1
|
||||
FAKE_BUILD_SLEEP=20 FAKE_RUN_RC=0 "$SCRIPT" "${BASE[@]}" --name go --build >"$T/out" 2>&1 &
|
||||
spid=$!
|
||||
for _ in $(seq 1 50); do [ -s "$T/build.pid" ] && break; sleep 0.1; done
|
||||
bpid="$(cat "$T/build.pid" 2>/dev/null || echo 0)"
|
||||
check "the fake build is running before the signal" kill -0 "$bpid"
|
||||
kill -TERM "$spid" 2>/dev/null
|
||||
for _ in $(seq 1 50); do kill -0 "$spid" 2>/dev/null || break; sleep 0.1; done
|
||||
kill -KILL "$spid" 2>/dev/null || true # never wait without a bound
|
||||
wait "$spid" 2>/dev/null; rc=$?
|
||||
check "TERM during the build: the script exits 143" test "$rc" -eq 143
|
||||
check "TERM during the build: ygg run is never called" bash -c '! grep -q "^run " "$1"' _ "$T/ygg.calls"
|
||||
check "TERM during the build: the output says the run was not started" has "not started"
|
||||
check "TERM during the build: the build is gone" bash -c '! kill -0 "$1" 2>/dev/null' _ "$bpid"
|
||||
kill -KILL "$spid" "$bpid" 2>/dev/null
|
||||
# An env file that cannot be loaded: the filter withholds, the run is not started, no value is shown.
|
||||
printf 'CLAUDE_CODE_OAUTH_TOKEN=fake-claude-VALUE-1\nOPENROUTER_API_KEY=fake-router-VALUE-2\nif then (\n' >"$T/env-broken"
|
||||
fresh
|
||||
FAKE_PLANT=fake-claude-VALUE-1 FAKE_RUN_RC=0 run proof --name go --env "$T/env-broken"
|
||||
check "an env file that cannot be sourced: no credential value is printed" bash -c '! grep -q VALUE "$1"' _ "$T/out"
|
||||
check "an env file that cannot be sourced: the output is withheld, and it says so" has "withheld"
|
||||
check "an env file that cannot be sourced: ygg run is not called" bash -c '! grep -q "^run " "$1"' _ "$T/ygg.calls"
|
||||
check "an env file that cannot be sourced: the exit code is not 0" test "$rc" -ne 0
|
||||
check "an env file that cannot be sourced: no credential value in the kept files" bash -c '! grep -r -a -q VALUE "$1"' _ "$(RUND)"
|
||||
|
||||
# --- v0-T291 review round 3: a bad credentials line is never echoed; INT stops the build too
|
||||
printf 'CLAUDE_CODE_OAUTH_TOKEN=fake-claude-VALUE-1 (\nOPENROUTER_API_KEY=fake-router-VALUE-2\n' >"$T/env-badline"
|
||||
fresh
|
||||
FAKE_PLANT=fake-claude-VALUE-1 FAKE_RUN_RC=0 run proof --name go --env "$T/env-badline"
|
||||
check "a bad credentials line: the value is nowhere in stdout or stderr" bash -c '! grep -q VALUE "$1"' _ "$T/out"
|
||||
check "a bad credentials line: the value is in no kept file" bash -c '! grep -r -a -q VALUE "$1"' _ "$(RUND)"
|
||||
check "a bad credentials line: it says the file could not be read" has "credentials file could not be read"
|
||||
check "a bad credentials line: ygg run is not called" bash -c '! grep -q "^run " "$1"' _ "$T/ygg.calls"
|
||||
# INT during the build (the script is started with SIGINT at its default, as a terminal would leave it).
|
||||
fresh; rm -f "$T/build.pid" "$T/run.pid"; sleep 1
|
||||
FAKE_BUILD_SLEEP=20 FAKE_RUN_RC=0 python3 -I -c 'import os, signal, sys; signal.signal(signal.SIGINT, signal.SIG_DFL); os.execv(sys.argv[1], sys.argv[1:])' "$SCRIPT" "${BASE[@]}" --name go --build >"$T/out" 2>&1 &
|
||||
spid=$!
|
||||
for _ in $(seq 1 50); do [ -s "$T/build.pid" ] && break; sleep 0.1; done
|
||||
bpid="$(cat "$T/build.pid" 2>/dev/null || echo 0)"
|
||||
check "the fake build is running before INT" kill -0 "$bpid"
|
||||
kill -INT "$spid" 2>/dev/null
|
||||
# The build must stop because of the signal, within 5 s (the fake build sleeps 20 s and ignores INT).
|
||||
for _ in $(seq 1 50); do kill -0 "$bpid" 2>/dev/null || break; sleep 0.1; done
|
||||
check "INT during the build: the build is gone within 5 s" bash -c '! kill -0 "$1" 2>/dev/null' _ "$bpid"
|
||||
for _ in $(seq 1 50); do kill -0 "$spid" 2>/dev/null || break; sleep 0.1; done
|
||||
check "INT during the build: the script is gone within 10 s" bash -c '! kill -0 "$1" 2>/dev/null' _ "$spid"
|
||||
kill -KILL "$bpid" 2>/dev/null
|
||||
kill -KILL "$spid" 2>/dev/null || true # never wait without a bound
|
||||
wait "$spid" 2>/dev/null; rc=$?
|
||||
check "INT during the build: the script exits 130" test "$rc" -eq 130
|
||||
check "INT during the build: ygg run is never called" bash -c '! grep -q "^run " "$1"' _ "$T/ygg.calls"
|
||||
check "INT during the build: the output says the run was not started" has "not started"
|
||||
check "INT during the build: the build is gone" bash -c '! kill -0 "$1" 2>/dev/null' _ "$bpid"
|
||||
kill -KILL "$spid" "$bpid" 2>/dev/null
|
||||
|
||||
# --- the ouroboric review (C1): a terminal's Ctrl-C and hang-up reach the script's whole process group
|
||||
# The script is started as a terminal starts it: in its own process group, with SIGINT and SIGHUP at their defaults.
|
||||
startg() { # ARGS...: the script in the background; spid is its pid and its process group
|
||||
python3 -I -c 'import os, signal, sys; os.setpgid(0, 0); signal.signal(signal.SIGINT, signal.SIG_DFL); signal.signal(signal.SIGHUP, signal.SIG_DFL); os.execv(sys.argv[1], sys.argv[1:])' "$SCRIPT" "${BASE[@]}" "$@" >"$T/out" 2>&1 &
|
||||
spid=$!
|
||||
}
|
||||
waitfor() { for _ in $(seq 1 50); do [ -s "$1" ] && break; sleep 0.1; done; } # FILE: at most 5 s
|
||||
gone() { for _ in $(seq 1 150); do kill -0 "$1" 2>/dev/null || return 0; sleep 0.1; done; return 1; } # PID: at most 15 s
|
||||
pgid_of() { ps -o pgid= -p "$1" 2>/dev/null | tr -d ' '; }
|
||||
reap() { # what a failing case leaves: the script's group and the fake run's group (timeout's)
|
||||
kill -KILL -- "-$spid" 2>/dev/null; [ -z "${rpg:-}" ] || kill -KILL -- "-$rpg" 2>/dev/null
|
||||
kill -KILL "$spid" 2>/dev/null || true # never wait without a bound
|
||||
wait "$spid" 2>/dev/null
|
||||
}
|
||||
for sig in INT HUP; do
|
||||
fresh; rm -f "$T/run.pid"; sleep 1
|
||||
FAKE_TRAP=1 startg --name go
|
||||
waitfor "$T/run.pid"; rpid="$(cat "$T/run.pid" 2>/dev/null || echo 0)"; rpg="$(pgid_of "$rpid")"
|
||||
check "$sig to the group: the fake run is running before the signal" kill -0 "$rpid"
|
||||
sleep 0.5
|
||||
kill -s "$sig" -- "-$spid" 2>/dev/null
|
||||
run_alive=1 rc=999
|
||||
if gone "$spid"; then
|
||||
kill -0 "$rpid" 2>/dev/null || run_alive=0 # the moment the script has ended, the run must be over
|
||||
wait "$spid" 2>/dev/null; rc=$?
|
||||
fi
|
||||
check "$sig to the group: the script ends (within 15 s)" test "$rc" -ne 999
|
||||
check "$sig to the group: the script does not end before the run" test "$run_alive" -eq 0
|
||||
check "$sig to the group: the script's exit code is the run's (3)" test "$rc" -eq 3
|
||||
check "$sig to the group: the run's shutdown lines are in run.txt" bash -c 'grep -q "shutdown line 4" "$1" && grep -q "the run shut down" "$1"' _ "$(RUND)/run.txt"
|
||||
check "$sig to the group: the run got the signal once" test "$(grep -c "the run got" "$(RUND)/run.txt")" = 1
|
||||
check "$sig to the group: report.txt is made after the run" grep -q 'metrics output' "$(RUND)/report.txt"
|
||||
reap
|
||||
done
|
||||
# A terminal that closes: the kernel hangs up the session (the script leads it here), and every later write to the
|
||||
# terminal fails. The script must still stop the run, keep its exit code and make the report. FIRST is a signal that
|
||||
# comes just before, while the run is still shutting down (a Ctrl-C to the script's group, a TERM to the script), or
|
||||
# "none".
|
||||
cat >"$T/pty.py" <<'PY2'
|
||||
import os, pty, select, signal, sys, time
|
||||
first, ready, argv = sys.argv[1], sys.argv[2], sys.argv[3:]
|
||||
pid, fd = pty.fork()
|
||||
if pid == 0:
|
||||
signal.signal(signal.SIGINT, signal.SIG_DFL)
|
||||
signal.signal(signal.SIGHUP, signal.SIG_DFL)
|
||||
os.execv(argv[0], argv)
|
||||
def drain(secs):
|
||||
end = time.time() + secs
|
||||
while time.time() < end:
|
||||
r, _, _ = select.select([fd], [], [], 0.1)
|
||||
if r:
|
||||
try:
|
||||
os.read(fd, 4096)
|
||||
except OSError:
|
||||
return
|
||||
end = time.time() + 5
|
||||
while time.time() < end and not os.path.exists(ready):
|
||||
drain(0.1)
|
||||
drain(0.5)
|
||||
if first != "none":
|
||||
if first == "INT":
|
||||
os.killpg(pid, signal.SIGINT) # a Ctrl-C reaches the script's whole process group (it leads its own)
|
||||
else:
|
||||
os.kill(pid, getattr(signal, "SIG" + first)) # a TERM is sent to the script
|
||||
drain(0.3) # the run is shutting down (it takes a second) when the terminal closes
|
||||
os.close(fd) # the terminal closes
|
||||
end = time.time() + 15
|
||||
while time.time() < end:
|
||||
p, status = os.waitpid(pid, os.WNOHANG)
|
||||
if p:
|
||||
print("rc=%d" % os.waitstatus_to_exitcode(status))
|
||||
sys.exit(0)
|
||||
time.sleep(0.1)
|
||||
os.kill(pid, signal.SIGKILL)
|
||||
os.waitpid(pid, 0)
|
||||
print("rc=still-running")
|
||||
PY2
|
||||
for first in none INT TERM; do
|
||||
case "$first" in none) what="a closed terminal" ;; *) what="$first, then a closed terminal" ;; esac
|
||||
fresh; rm -f "$T/run.pid"; sleep 1
|
||||
FAKE_TRAP=1 timeout 60 python3 -I "$T/pty.py" "$first" "$T/run.pid" "$SCRIPT" "${BASE[@]}" --name go >"$T/pty.out" 2>&1
|
||||
rpid="$(cat "$T/run.pid" 2>/dev/null || echo 0)"; rpg="$(pgid_of "$rpid")"
|
||||
check "$what: the script ends with the run's exit code (3)" grep -q -x 'rc=3' "$T/pty.out"
|
||||
check "$what: the run is stopped, no process is left" bash -c '[ "$1" != 0 ] && ! kill -0 "$1" 2>/dev/null && { [ -z "$2" ] || ! pgrep -g "$2" >/dev/null; }' _ "$rpid" "$rpg"
|
||||
check "$what: the run's shutdown lines are in run.txt" grep -q "the run shut down" "$(RUND)/run.txt"
|
||||
check "$what: report.txt is made" grep -q 'metrics output' "$(RUND)/report.txt"
|
||||
check "$what: what the script prints after it is kept in hangup.txt" bash -c 'grep -q "exit code 3" "$1" && grep -q -F -- "--continue RUN-1" "$1"' _ "$(RUND)/hangup.txt"
|
||||
check "$what: hangup.txt, run.stop and run.hup are private (600)" bash -c 'for f in hangup.txt run.hup; do [ "$(stat -c %a "$1/$f")" = 600 ] || exit 1; done; [ ! -e "$1/run.stop" ] || [ "$(stat -c %a "$1/run.stop")" = 600 ]' _ "$(RUND)"
|
||||
[ -z "$rpg" ] || kill -KILL -- "-$rpg" 2>/dev/null
|
||||
done
|
||||
|
||||
# --- the ouroboric review (T2): the real wall clock ends the run and what it started
|
||||
fresh; rm -f "$T/run.pid" "$T/child.pid"; sleep 1
|
||||
FAKE_SLEEP=20 FAKE_CHILD=1 timeout 60 "$SCRIPT" "${BASE[@]}" --name go --wall 2s >"$T/out" 2>&1; rc=$?
|
||||
rpid="$(cat "$T/run.pid" 2>/dev/null || echo 0)"; cpid="$(cat "$T/child.pid" 2>/dev/null || echo 0)"
|
||||
check "--wall 2s: the script exits 124" test "$rc" -eq 124
|
||||
check "--wall 2s: the fake run is gone" bash -c '[ "$1" != 0 ] && ! kill -0 "$1" 2>/dev/null' _ "$rpid"
|
||||
check "--wall 2s: the child the run started is gone" bash -c '[ "$1" != 0 ] && ! kill -0 "$1" 2>/dev/null' _ "$cpid"
|
||||
kill -KILL "$rpid" "$cpid" 2>/dev/null
|
||||
|
||||
# --- the ouroboric review (I2): after an abnormal end, ygg doctor stops what the run left
|
||||
runit 137
|
||||
check "exit 137: the script exits 137" test "$rc" -eq 137
|
||||
check "exit 137: doctor runs after the run, before the evidence" test "$(calls)" = "doctor,run,doctor,metrics,report"
|
||||
check "exit 137: that doctor gets the project and the config" bash -c 'sed -n 3p "$1" | grep -q -E -- "^doctor --project [^ ]*/proof-go-[0-9-]+/project --config [^ ]*/proof-go-[0-9-]+/factory.yaml$"' _ "$T/ygg.calls"
|
||||
check "exit 137: that doctor has no credential" test "$(grep -c -x 'doctor: ' "$T/ygg.env")" = 2
|
||||
check "exit 137: the doctor's output is in report.txt" grep -q -F '## ygg doctor' "$(RUND)/report.txt"
|
||||
runit 0
|
||||
check "exit 0: no doctor after the run" test "$(calls)" = "doctor,run,grade,metrics,report"
|
||||
|
||||
# --- the ouroboric review (C2): a baseline is found in its own proof folder, or refused before anything starts
|
||||
nothing_started() { [ "$rc" -eq 2 ] && [ ! -s "$T/ygg.calls" ] && [ -z "$(find "$D" -maxdepth 1 -name 'proof-go-*')" ]; }
|
||||
fresh; plant_base; run proof --name go --baseline 20200101-000000-ffffff
|
||||
check "a baseline no proof folder holds: exit 2, nothing started, no folder made" nothing_started
|
||||
check "a baseline no proof folder holds: it says so" has "no proof folder under"
|
||||
fresh; plant_base; plant_base "$OLD" proof-old2-20200102-000000; run proof --name go --baseline "$OLD"
|
||||
check "a baseline two proof folders hold: exit 2, nothing started, no folder made" nothing_started
|
||||
check "a baseline two proof folders hold: it says so" has "2 proof folders under"
|
||||
fresh; plant_base; run proof --name go --dry-run --baseline "$OLD"
|
||||
check "the dry run names the baseline it found and its folder" bash -c 'grep -F "baseline run $2" "$1" | grep -q -F "proof-old-20200101-000000"' _ "$T/out" "$OLD"
|
||||
|
||||
# --- the ouroboric review (I1): --continue continues a stopped or paused run inside the script
|
||||
CBASE=(--folder "$D" --env "$ENVF")
|
||||
cont() { "$SCRIPT" "${CBASE[@]}" "$@"; }
|
||||
runc() { "$@" >"$T/out" 2>&1; rc=$?; } # a continuation makes no folder: no wait
|
||||
clearlogs() { rm -f "$T/ygg.calls" "$T/ygg.env" "$T/timeout.args"; }
|
||||
fresh; plant_base; FAKE_RUN_RC=3 run proof --name go --baseline "$OLD"; W="$(RUND)"; clearlogs
|
||||
FAKE_PLANT=fake-claude-VALUE-1 runc cont --continue RUN-1 --approve ship-gate --note 'looks right'
|
||||
check "--continue --approve after 3: exit 0, the continuation's code" test "$rc" -eq 0
|
||||
check "--continue --approve: doctor, approve, grade, metrics, the compare, report" test "$(calls)" = "doctor,approve,grade,metrics,metrics,report"
|
||||
check "--continue --approve: approve gets the run, the gate, the project and the note" grep -q -x -F "approve RUN-1 ship-gate --project $W/project --note looks right" "$T/ygg.calls"
|
||||
check "--continue --approve: it runs under the same timeout" bash -c 'head -n 1 "$1" | grep -q -F -- "--signal=INT --kill-after=2m 8h $2 approve RUN-1 ship-gate"' _ "$T/timeout.args" "$FB/ygg"
|
||||
check "--continue: only approve has the credentials" bash -c 'grep -q -x "approve: CLAUDE_CODE_OAUTH_TOKEN OPENROUTER_API_KEY " "$1" && [ "$(grep -c -v -E "^approve: " "$1")" = "$(grep -c -E "^[a-z]+: $" "$1")" ]' _ "$T/ygg.env"
|
||||
check "--continue: the output is added to the folder's run.txt" bash -c 'grep -q "the run runs" "$1" && grep -q "approve runs" "$1"' _ "$W/run.txt"
|
||||
check "--continue: a planted value is filtered from run.txt and the output" bash -c '! grep -q -F fake-claude-VALUE-1 "$1" && ! grep -q -F fake-claude-VALUE-1 "$2" && grep -q REDACTED "$1"' _ "$W/run.txt" "$T/out"
|
||||
check "--continue: the recorded baseline is compared" grep -q -x -F "metrics --compare $OLD RUN-1 --project $W/compare" "$T/ygg.calls"
|
||||
check "--continue: the report is added to report.txt" bash -c '[ "$(grep -c "^# proof run go: run RUN-1" "$1")" = 2 ] && grep -q "^## ygg grade" "$1"' _ "$W/report.txt"
|
||||
check "--continue: run.txt and report.txt stay private, the compare folder is private" bash -c '[ "$(stat -c %a "$1/run.txt")" = 600 ] && [ "$(stat -c %a "$1/report.txt")" = 600 ] && [ "$(stat -c %a "$1/compare")" = 700 ]' _ "$W"
|
||||
check "--continue: no new proof folder" test "$(find "$D" -maxdepth 1 -name 'proof-go-*' | wc -l)" = 1
|
||||
fresh; FAKE_RUN_RC=4 run proof --name go; W="$(RUND)"; clearlogs
|
||||
FAKE_CONT_RC=3 runc cont --continue RUN-1 --resume
|
||||
check "--continue --resume after 4: resume gets the run and the project" grep -q -x -F "resume RUN-1 --project $W/project" "$T/ygg.calls"
|
||||
check "--continue --resume ending 3: no grade, metrics and report" test "$(calls)" = "doctor,resume,metrics,report"
|
||||
check "--continue --resume ending 3: the exit code is 3" test "$rc" -eq 3
|
||||
check "--continue --resume ending 3: the output ends with the next --continue command" bash -c 'tail -n 1 "$1" | grep -q -F "ygg-proof-run.sh --continue RUN-1 --approve ship-gate"' _ "$T/out"
|
||||
clearlogs; FAKE_CONT_RC=0 runc cont --continue RUN-1 --resume
|
||||
check "--continue --resume ending 0: doctor, resume, grade, metrics, report" test "$(calls)" = "doctor,resume,grade,metrics,report"
|
||||
clearlogs; runc cont --continue RUN-1 --approve ship-gate --dry-run
|
||||
check "--continue --dry-run: exit 0, approve never called" bash -c '[ "$1" -eq 0 ] && ! grep -q "^approve" "$2"' _ "$rc" "$T/ygg.calls"
|
||||
check "--continue --dry-run: it shows the command under the timeout" bash -c 'grep "would run: timeout --signal=INT --kill-after=2m 8h" "$1" | grep -q -F "approve RUN-1 ship-gate --project"' _ "$T/out"
|
||||
refusesc() { [ "$rc" -eq 2 ] && has "$1" && [ ! -s "$T/ygg.calls" ]; } # exit 2, the reason, nothing started
|
||||
clearlogs; runc cont --continue NOPE-RUN --resume; check "--continue with no such run: refused before anything starts" refusesc "no proof folder under"
|
||||
clearlogs; runc cont --continue RUN-1 --approve ship-gate --resume; check "--continue with --approve and --resume: refused" refusesc "not both"
|
||||
clearlogs; runc cont --continue RUN-1; check "--continue with neither --approve nor --resume: refused" refusesc "--approve GATE or --resume"
|
||||
clearlogs; runc cont --approve ship-gate; check "--approve without --continue: refused" refusesc "go with --continue"
|
||||
clearlogs; runc cont --continue RUN-1 --resume --note hi; check "--note without --approve: refused" refusesc "--note goes with --approve"
|
||||
clearlogs; runc cont --continue RUN-1 --resume --name other; check "--continue with a setup flag: refused" refusesc "keeps the config"
|
||||
clearlogs; runc cont --continue RUN-1 --approve ship-gate --note $'a\nb'; check "--note with a control character: refused" refusesc "control character"
|
||||
clearlogs; runc cont --continue .. --resume; check "--continue .. is not a run id" refusesc "not a run id"
|
||||
|
||||
[ "$fail" -eq 0 ] && echo "all ok"
|
||||
exit "$fail"
|
||||
Executable
+514
@@ -0,0 +1,514 @@
|
||||
#!/usr/bin/env bash
|
||||
# Set up a proof run (v0-T290): the `software` graph on the real client-gallery-lite, with the realms, three roles
|
||||
# the operator names and caps the operator sets. Every proof run is set up the same way, so that `ygg metrics
|
||||
# --compare` compares two factories and not two setups. IT WILL COST MONEY (without --dry-run):
|
||||
# the script makes the project and the config, runs the factory, grades the run and prints the evidence.
|
||||
# ygg-proof-run.sh --name NAME [--mode key-points|unattended|step-by-step] [--planner AGENT:MODEL]
|
||||
# [--implementer AGENT:MODEL] [--reviewer AGENT:MODEL] [--run-turns N] [--run-tokens N]
|
||||
# [--run-usd N] [--day-turns N] [--day-tokens N] [--day-usd N] [--baseline RUN_ID] [--dry-run]
|
||||
# [--build] [--folder DIR] [--env FILE] [--tasks DIR] [--suites DIR] [--yggdrasil DIR]
|
||||
# [--wall TIME] [--clip]
|
||||
# ygg-proof-run.sh --continue RUN_ID (--approve GATE [--note TEXT] | --resume) [--baseline RUN_ID] [--dry-run]
|
||||
# [--folder DIR] [--env FILE] [--wall TIME] [--clip]
|
||||
# AGENT is claude-code or opencode; an opencode MODEL is openrouter/<model>; a MODEL has only A-Za-z0-9._/:-
|
||||
# because it goes into a YAML line. Defaults: key-points; planner and implementer claude-code:sonnet; reviewer
|
||||
# opencode:openrouter/qwen/qwen3.8-27b; a run 1,500 turns, 12,000,000 tokens, $5; a day 2,000 turns, 24,000,000
|
||||
# tokens, $10; --wall 8h; DIR $HOME/ygg-it; FILE $HOME/.config/ouroboric/factory.env; --tasks ../mimir;
|
||||
# --suites $HOME/ygg-it/hidden-suites; --yggdrasil the umbrella beside this repo. The task is client-gallery-lite
|
||||
# (YGG_PROOF_TASK names another, for tests); its suite is the `suite` key of its task.toml.
|
||||
# In order, it
|
||||
# 1. checks the flags (exit 2 on the first bad one), the engine (ygg-engine.sh; YGG_ENGINE_CHECK=0 skips it)
|
||||
# and the tools (git, python3, timeout, sha256sum);
|
||||
# 2. checks the task: task.toml and instruction.md, and that the SHA-256 of instruction.md is the brief_sha256
|
||||
# of task.toml. A changed brief is another task: a proof run uses the frozen one;
|
||||
# 3. checks the suites folder with `test -d SUITES/TASK/SUITE` and nothing more: it is never listed, read or
|
||||
# searched, and neither it nor the tasks folder is inside the other or inside the project;
|
||||
# 4. checks the credentials file WITHOUT reading it, for the adapters the roles name only: claude-code needs
|
||||
# CLAUDE_CODE_OAUTH_TOKEN, opencode needs OPENROUTER_API_KEY, each on one line with a value after the equals
|
||||
# sign (grep -c counts the lines; no value is printed or kept);
|
||||
# 5. makes DIR/proof-NAME-TIME/project (a git repository, umbrella.yaml with root: true) and
|
||||
# DIR/proof-NAME-TIME/factory.yaml, filled from scripts/testdata/proof/factory.proof.example.yaml (a "usd"
|
||||
# line only when a role is an opencode role, which pays per token);
|
||||
# 6. runs `ygg doctor` on that config and prints the plan: what would run.
|
||||
# --dry-run stops there with exit 0. Without it (v0-T291) the script then
|
||||
# 7. with --build runs `ygg sandbox build --all --config C`; runs `ygg run --project P --config C --graph software
|
||||
# --brief <task>/instruction.md --mode M` under `timeout --signal=INT --kill-after=2m WALL`, with the credentials
|
||||
# loaded in that subshell only (never exported by this shell, never in an argument), its output (credential
|
||||
# values removed) shown as it comes and kept in DIR/proof-NAME-TIME/run.txt. A Ctrl-C, a closed terminal (HUP)
|
||||
# or a TERM stops the run through timeout, once, and the script waits for it to end; after a hang-up what the
|
||||
# script still prints goes to DIR/proof-NAME-TIME/hangup.txt;
|
||||
# 8. prints the exit code (0 done, 1 a node failed, 3 stopped, 4 paused, 5 done with tasks given up, 124 the wall
|
||||
# clock); after an abnormal end (any code but 0, 3, 4 and 5) runs `ygg doctor`, which stops the sandboxes a run
|
||||
# left; after a run that ended 0 or 5, runs `ygg grade RUN`; after any ending runs `ygg metrics RUN`, with
|
||||
# --baseline RUN_ID also `ygg metrics --compare RUN_ID RUN` (printed as the CLI gives it: this script judges
|
||||
# nothing; each proof run has its own project, so the two run folders are linked into DIR/proof-NAME-TIME/compare
|
||||
# for it), `ygg report RUN`, ygg-cost.sh on the ledger and ygg-evidence.sh. Everything goes through a filter that
|
||||
# replaces the credentials' exact values, and is added to DIR/proof-NAME-TIME/report.txt (--clip copies this
|
||||
# report to the Windows clipboard);
|
||||
# 9. for a run that stopped (3) or paused (4), ends with this script's command that continues it. A stopped or
|
||||
# paused run is not graded. The exit code is the run's.
|
||||
# --baseline RUN_ID names a run of an earlier proof run: exactly one DIR/proof-*/project/.factory/runs/RUN_ID must
|
||||
# hold it, or the script refuses before it makes anything.
|
||||
# --continue RUN_ID continues a run that stopped or paused, in the proof folder under DIR that holds it (the setup
|
||||
# flags are refused: the run keeps its config). It checks the credentials and runs `ygg doctor`, then `ygg approve
|
||||
# RUN_ID GATE --project P [--note TEXT]` or `ygg resume RUN_ID --project P` as step 7 runs `ygg run` (the same timeout,
|
||||
# the credentials in that subshell only, the filter, the output added to run.txt), then steps 8 and 9, with the
|
||||
# baseline the proof run recorded unless --baseline names another. The raw `ygg approve` and `ygg resume` commands are
|
||||
# printed too, as a fallback: they run without the wall clock, the filter or the report.
|
||||
# For tests and other machines: YGG_CMD (how ygg runs), YGG_BIND (the bind mount, default /mnt/wsl/ygg-bind).
|
||||
set -euo pipefail
|
||||
shopt -u patsub_replacement 2>/dev/null || true
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
usage() {
|
||||
echo "usage: ygg-proof-run.sh --name NAME [--mode MODE] [--planner|--implementer|--reviewer AGENT:MODEL] [--run-turns|--run-tokens|--run-usd|--day-turns|--day-tokens|--day-usd N] [--baseline RUN_ID] [--dry-run] [--build] [--folder DIR] [--env FILE] [--tasks DIR] [--suites DIR] [--yggdrasil DIR] [--wall TIME] [--clip]" >&2
|
||||
echo " ygg-proof-run.sh --continue RUN_ID (--approve GATE [--note TEXT] | --resume) [--baseline RUN_ID] [--dry-run] [--folder DIR] [--env FILE] [--wall TIME] [--clip]" >&2
|
||||
exit 2
|
||||
}
|
||||
fail() { echo "proof-run: $*" >&2; exit 2; }
|
||||
# Only the script's own files are private (the credentials' values never reach them, but they hold the run's output).
|
||||
# Everything else (the project, its git repository and what ygg makes, which the sandboxes read) keeps the caller's
|
||||
# umask, as in the smoke scripts: no umask is set for the whole script.
|
||||
private() { (umask 077 && : >>"$1") && chmod 600 "$1"; } # FILE: made private; what it holds stays
|
||||
put() { (umask 077 && printf '%s\n' "$2" >"$1"); } # FILE TEXT: a small control file, private
|
||||
name="" mode=key-points dry=0 build=0 clip=0 baseline="" wall=8h
|
||||
cont="" gate_in="" note="" note_set=0 resume=0 setup_flags="" # --continue; the setup flags given (a continued run keeps its config)
|
||||
planner=claude-code:sonnet implementer=claude-code:sonnet reviewer=opencode:openrouter/qwen/qwen3.8-27b
|
||||
run_turns=1500 run_tokens=12000000 run_usd=5 day_turns=2000 day_tokens=24000000 day_usd=10
|
||||
dir="$HOME/ygg-it" envfile="$HOME/.config/ouroboric/factory.env" tasks="$HERE/../../mimir"
|
||||
suites="$HOME/ygg-it/hidden-suites" ygg_dir="$HERE/../../yggdrasil"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--name | --mode | --planner | --implementer | --reviewer | --run-* | --day-* | --build | --tasks | --suites | --yggdrasil) setup_flags+=" $1" ;;
|
||||
esac
|
||||
case "$1" in
|
||||
--dry-run) dry=1; shift ;;
|
||||
--build) build=1; shift ;;
|
||||
--resume) resume=1; shift ;;
|
||||
--continue) [ $# -ge 2 ] || usage; cont="$2"; shift 2 ;;
|
||||
--approve) [ $# -ge 2 ] || usage; gate_in="$2"; shift 2 ;;
|
||||
--note) [ $# -ge 2 ] || usage; note="$2"; note_set=1; shift 2 ;;
|
||||
--clip) clip=1; shift ;;
|
||||
--name) [ $# -ge 2 ] || usage; name="$2"; shift 2 ;;
|
||||
--mode) [ $# -ge 2 ] || usage; mode="$2"; shift 2 ;;
|
||||
--planner) [ $# -ge 2 ] || usage; planner="$2"; shift 2 ;;
|
||||
--implementer) [ $# -ge 2 ] || usage; implementer="$2"; shift 2 ;;
|
||||
--reviewer) [ $# -ge 2 ] || usage; reviewer="$2"; shift 2 ;;
|
||||
--run-turns) [ $# -ge 2 ] || usage; run_turns="$2"; shift 2 ;;
|
||||
--run-tokens) [ $# -ge 2 ] || usage; run_tokens="$2"; shift 2 ;;
|
||||
--run-usd) [ $# -ge 2 ] || usage; run_usd="$2"; shift 2 ;;
|
||||
--day-turns) [ $# -ge 2 ] || usage; day_turns="$2"; shift 2 ;;
|
||||
--day-tokens) [ $# -ge 2 ] || usage; day_tokens="$2"; shift 2 ;;
|
||||
--day-usd) [ $# -ge 2 ] || usage; day_usd="$2"; shift 2 ;;
|
||||
--baseline) [ $# -ge 2 ] || usage; baseline="$2"; shift 2 ;;
|
||||
--folder) [ $# -ge 2 ] || usage; dir="$2"; shift 2 ;;
|
||||
--env) [ $# -ge 2 ] || usage; envfile="$2"; shift 2 ;;
|
||||
--tasks) [ $# -ge 2 ] || usage; tasks="$2"; shift 2 ;;
|
||||
--suites) [ $# -ge 2 ] || usage; suites="$2"; shift 2 ;;
|
||||
--yggdrasil) [ $# -ge 2 ] || usage; ygg_dir="$2"; shift 2 ;;
|
||||
--wall) [ $# -ge 2 ] || usage; wall="$2"; shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# 1. The flags.
|
||||
run_id_re='^[A-Za-z0-9][A-Za-z0-9._-]*$' # a run id is a folder name: never . or ..
|
||||
if [ -n "$cont" ]; then
|
||||
[[ $cont =~ $run_id_re ]] || fail "--continue '$cont' is not a run id"
|
||||
[ -z "$setup_flags" ] || fail "--continue keeps the config of the run's proof folder; it takes none of$setup_flags"
|
||||
if [ -n "$gate_in" ] && [ "$resume" -eq 1 ]; then fail "--continue takes --approve GATE or --resume, not both"; fi
|
||||
if [ -z "$gate_in" ] && [ "$resume" -eq 0 ]; then fail "--continue takes --approve GATE or --resume"; fi
|
||||
if [ -n "$gate_in" ] && ! [[ $gate_in =~ $run_id_re ]]; then fail "--approve '$gate_in' is not a gate name"; fi
|
||||
if [ "$note_set" -eq 1 ] && [ -z "$gate_in" ]; then fail "--note goes with --approve"; fi
|
||||
if [[ $note == *[[:cntrl:]]* ]]; then fail "--note has a control character; a note is one line of text"; fi
|
||||
else
|
||||
if [ -n "$gate_in" ] || [ "$resume" -eq 1 ] || [ "$note_set" -eq 1 ]; then fail "--approve, --resume and --note go with --continue RUN_ID"; fi
|
||||
[ -n "$name" ] || fail "--name is needed: the run's name, such as baseline-1"
|
||||
[[ $name =~ ^[a-z0-9-]+$ ]] || fail "--name '$name' has characters outside a-z0-9-"
|
||||
case "$mode" in key-points | unattended | step-by-step) ;; *) fail "--mode is key-points, unattended or step-by-step, not '$mode'" ;; esac
|
||||
for pair in run-turns:"$run_turns" run-tokens:"$run_tokens" day-turns:"$day_turns" day-tokens:"$day_tokens"; do
|
||||
[[ ${pair#*:} =~ ^[1-9][0-9]*$ ]] || fail "--${pair%%:*} is a whole number above 0, not '${pair#*:}'"
|
||||
done
|
||||
for pair in run-usd:"$run_usd" day-usd:"$day_usd"; do
|
||||
[[ ${pair#*:} =~ ^[0-9]+(\.[0-9]+)?$ ]] || fail "--${pair%%:*} is a number of dollars such as 5 or 2.5, not '${pair#*:}'"
|
||||
done
|
||||
fi
|
||||
[[ $wall =~ ^[1-9][0-9]*[smh]$ ]] || fail "--wall is a number and s, m or h, such as 8h"
|
||||
if [ -n "$baseline" ]; then [[ $baseline =~ $run_id_re ]] || fail "--baseline '$baseline' is not a run id"; fi
|
||||
# Folders go into factory.yaml as they are, so their characters cannot change how a YAML line parses.
|
||||
safe_path() { # FLAG VALUE
|
||||
[[ $2 =~ ^[A-Za-z0-9._/@+-]+$ ]] || fail "$1 '$2' has a character outside A-Za-z0-9._/@+- (no spaces); it goes into factory.yaml"
|
||||
}
|
||||
safe_path --folder "$dir"; safe_path --tasks "$tasks"; safe_path --suites "$suites"; safe_path --yggdrasil "$ygg_dir"
|
||||
safe_path YGG_BIND "${YGG_BIND:-/mnt/wsl/ygg-bind}"
|
||||
# A role goes into a YAML line as it is, so its model has only the characters a model id needs.
|
||||
role_ok() { # FLAG VALUE: sets agent_of and model_of
|
||||
local flag="$1" v="$2" agent model
|
||||
[[ $v == *:* ]] || fail "$flag is AGENT:MODEL, with AGENT claude-code or opencode, such as claude-code:sonnet"
|
||||
agent="${v%%:*}" model="${v#*:}"
|
||||
[[ $model =~ ^[A-Za-z0-9._/:-]+$ && $model != *: ]] || fail "$flag has a MODEL with characters outside A-Za-z0-9._/:- or ends in a colon"
|
||||
case "$agent" in
|
||||
claude-code) ;;
|
||||
opencode) [[ $model == openrouter/?* ]] || fail "$flag: an opencode MODEL is openrouter/<model>, such as opencode:openrouter/qwen/qwen3.8-27b" ;;
|
||||
*) fail "$flag is AGENT:MODEL, with AGENT claude-code or opencode, not '$agent'" ;;
|
||||
esac
|
||||
}
|
||||
roles_ok() { # sets paid, need_claude and need_router
|
||||
role_ok --planner "$planner"; role_ok --implementer "$implementer"; role_ok --reviewer "$reviewer"
|
||||
paid=0 need_claude=0 need_router=0
|
||||
for r in "$planner" "$implementer" "$reviewer"; do
|
||||
case "${r%%:*}" in claude-code) need_claude=1 ;; opencode) need_router=1; paid=1 ;; esac
|
||||
done
|
||||
}
|
||||
[ -n "$cont" ] || roles_ok
|
||||
|
||||
# The engine, the tools.
|
||||
export PATH="$dir/dc/node_modules/.bin:$PATH"
|
||||
if [ "${YGG_ENGINE_CHECK:-1}" = 1 ]; then
|
||||
engine_env="$("$HERE/ygg-engine.sh" env --folder "$dir")" || exit 2
|
||||
eval "$engine_env"
|
||||
fi
|
||||
[ -f "$dir/factory.yaml" ] || fail "$dir/factory.yaml is missing; it names the engine (sandbox.engine)"
|
||||
engine="$(sed -n 's/^[[:space:]]*engine:[[:space:]]*\([^[:space:]#]*\).*/\1/p' "$dir/factory.yaml" | head -n 1)"
|
||||
case "$engine" in unix://*) ;; *) fail "sandbox.engine in $dir/factory.yaml is '${engine:-missing}'; it must be a unix:// URL" ;; esac
|
||||
safe_path sandbox.engine "${engine#unix://}"
|
||||
for tool in git python3 timeout sha256sum; do command -v "$tool" >/dev/null 2>&1 || fail "$tool is not installed"; done
|
||||
ygg="${YGG_CMD:-ygg}"
|
||||
read -r -a ygg_arr <<<"$ygg"
|
||||
command -v -- "${ygg_arr[0]:-ygg}" >/dev/null 2>&1 || fail "no ygg on the path (YGG_CMD names how it runs)"
|
||||
|
||||
# A run of an earlier proof run, by its id: exactly one proof folder under DIR may hold it. Each proof run has its
|
||||
# own project, so a run is found by its folder name and nothing else; no run folder is read here.
|
||||
find_run() { # FLAG RUN_ID: sets hit to that run's folder, or sets why and returns 1
|
||||
local h
|
||||
local -a hits=()
|
||||
for h in "$dir_abs"/proof-*/project/.factory/runs/"$2"; do [ -d "$h" ] && hits+=("$h"); done
|
||||
hit="${hits[0]:-}"
|
||||
case "${#hits[@]}" in
|
||||
1) return 0 ;;
|
||||
0) why="$1: no proof folder under $dir_abs holds a run $2" ;;
|
||||
*) why="$1: ${#hits[@]} proof folders under $dir_abs hold a run $2 (${hits[*]%/project/.factory/runs/*}); a run id must name one run" ;;
|
||||
esac
|
||||
return 1
|
||||
}
|
||||
baseline_recorded=0
|
||||
if [ -z "$cont" ]; then
|
||||
# 2. The task, and that its brief is the frozen one.
|
||||
task="${YGG_PROOF_TASK:-client-gallery-lite}"
|
||||
tasks="$(cd -P -- "$tasks" 2>/dev/null && pwd -P)" || fail "no task folder at $tasks (--tasks)"
|
||||
ygg_dir="$(cd -P -- "$ygg_dir" 2>/dev/null && pwd -P)" || fail "no yggdrasil folder at $ygg_dir (--yggdrasil)"
|
||||
for f in "$tasks/$task/task.toml" "$tasks/$task/instruction.md"; do [ -f "$f" ] || fail "no task: $f is missing"; done
|
||||
suite="$(sed -n 's/^suite[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' "$tasks/$task/task.toml" | head -n 1)"
|
||||
[ -n "$suite" ] || fail "$tasks/$task/task.toml has no suite key"
|
||||
[[ $suite =~ ^[a-z0-9][a-z0-9._-]*$ ]] || fail "the suite '$suite' of $task is not a suite version"
|
||||
frozen="$(sed -n 's/^brief_sha256[[:space:]]*=[[:space:]]*"\([0-9a-f]*\)".*/\1/p' "$tasks/$task/task.toml" | head -n 1)"
|
||||
have="$(sha256sum "$tasks/$task/instruction.md" | cut -d' ' -f1)"
|
||||
[ -n "$frozen" ] && [ "$have" = "$frozen" ] || fail "$tasks/$task/instruction.md is not the frozen brief: its SHA-256 is $have and task.toml says '${frozen:-nothing}'; a changed brief is another task"
|
||||
|
||||
# 3. The suites folder: one test -d, never listed or read.
|
||||
# readlink -f resolves the links of the path without entering or listing the suites folder.
|
||||
suites_raw="$suites"
|
||||
suites="$(readlink -f -- "$suites" 2>/dev/null)" && [ -n "$suites" ] || fail "no suites folder at $suites_raw for $task/$suite (--suites)"
|
||||
test -d "$suites/$task/$suite" || fail "the suites folder $suites has no $task/$suite (--suites)"
|
||||
work="$dir/proof-$name-$(date +%Y%m%d-%H%M%S)"
|
||||
dir_abs="$(cd -P -- "$dir" && pwd -P)"
|
||||
work="$dir_abs/${work##*/}"
|
||||
for pair in tasks:"$tasks" suites:"$suites" yggdrasil:"$ygg_dir" folder:"$dir_abs"; do safe_path "--${pair%%:*} (resolved)" "${pair#*:}"; done
|
||||
inside() { case "$1/" in "$2"/*) return 0 ;; esac; return 1; } # inside A B: A is B or below B
|
||||
if inside "$suites" "$tasks"; then fail "the suites folder $suites is inside the tasks folder $tasks"; fi
|
||||
if inside "$tasks" "$suites"; then fail "the tasks folder $tasks is inside the suites folder $suites"; fi
|
||||
for f in "$tasks" "$suites"; do
|
||||
if inside "$f" "$work"; then fail "$f is inside the project folder $work"; fi
|
||||
done
|
||||
else
|
||||
# 2. and 3. for --continue: the proof folder that holds the run, and what its setup recorded. The task, the suites
|
||||
# and the config were checked when the run was set up; the config stays as it was.
|
||||
dir_abs="$(cd -P -- "$dir" 2>/dev/null && pwd -P)" || fail "no folder at $dir (--folder)"
|
||||
safe_path "--folder (resolved)" "$dir_abs"
|
||||
find_run --continue "$cont" || fail "$why"
|
||||
work="${hit%/project/.factory/runs/*}"
|
||||
[[ ${work##*/} =~ ^proof-[a-z0-9-]+-[0-9]{8}-[0-9]{6}$ ]] || fail "$work is not a proof folder (proof-NAME-TIME)"
|
||||
[ -f "$work/factory.yaml" ] && [ -f "$work/proof.meta" ] || fail "$work has no factory.yaml or proof.meta: this script did not set it up"
|
||||
meta() { sed -n "s/^$1 //p" "$work/proof.meta" | head -n 1; } # a key's value; each is checked below as a flag is
|
||||
name="$(meta name)" mode="$(meta mode)" planner="$(meta planner)" implementer="$(meta implementer)" reviewer="$(meta reviewer)"
|
||||
task="$(meta task)" suite="$(meta suite)" have="$(meta brief)"
|
||||
[[ $name =~ ^[a-z0-9-]+$ ]] || fail "$work/proof.meta has no proof run name"
|
||||
case "$mode" in key-points | unattended | step-by-step) ;; *) fail "$work/proof.meta has no mode" ;; esac
|
||||
roles_ok
|
||||
[[ $task =~ ^[A-Za-z0-9._-]+$ && $suite =~ ^[a-z0-9][a-z0-9._-]*$ && $have =~ ^[0-9a-f]{64}$ ]] || fail "$work/proof.meta has no task, suite or brief SHA-256"
|
||||
if [ -z "$baseline" ]; then
|
||||
baseline="$(meta baseline)" baseline_recorded=1
|
||||
if [ -n "$baseline" ] && ! [[ $baseline =~ $run_id_re ]]; then fail "$work/proof.meta records a baseline that is not a run id"; fi
|
||||
fi
|
||||
if [ "$baseline" = "$cont" ]; then fail "--baseline $baseline is the run being continued; a run is not its own baseline"; fi
|
||||
fi
|
||||
# The baseline, found before anything is made or started: a bad one costs nothing. A baseline the proof run recorded
|
||||
# that is gone now only drops the comparison, so that the run can still be continued.
|
||||
base_dir=""
|
||||
if [ -n "$baseline" ]; then
|
||||
if find_run --baseline "$baseline"; then base_dir="$hit"
|
||||
elif [ "$baseline_recorded" -eq 1 ]; then echo "proof-run: $why; the comparison is left out" >&2; baseline=""
|
||||
else fail "$why"
|
||||
fi
|
||||
fi
|
||||
|
||||
# 4. The credentials, by name. grep -c prints a count and nothing else; no line is shown, stored or compared.
|
||||
[ -f "$envfile" ] || fail "the credentials file $envfile is missing"
|
||||
names=()
|
||||
[ "$need_claude" -eq 0 ] || names+=(CLAUDE_CODE_OAUTH_TOKEN)
|
||||
[ "$need_router" -eq 0 ] || names+=(OPENROUTER_API_KEY)
|
||||
for n_ in "${names[@]}"; do
|
||||
n="$(grep -c -E "^(export[[:space:]]+)?${n_}=[^[:space:]]" "$envfile" || true)"
|
||||
[ "$n" = 1 ] || fail "$envfile has $n line(s) that set $n_ with a value; the roles need exactly 1"
|
||||
done
|
||||
|
||||
if [ -z "$cont" ]; then
|
||||
# 5. The project and the config.
|
||||
project="$work/project"
|
||||
mkdir -p "$project"
|
||||
git -C "$project" init -q -b main
|
||||
printf 'root: true\nowner: ouroboric-dev\nvisibility: private\n' >"$project/umbrella.yaml"
|
||||
git -C "$project" add umbrella.yaml
|
||||
git -C "$project" -c user.name=ygg -c user.email=[email protected] commit -q -m "the proof run project"
|
||||
example="$HERE/testdata/proof/factory.proof.example.yaml"
|
||||
[ -f "$example" ] || fail "$example is missing"
|
||||
usd_run="" usd_day=""
|
||||
if [ "$paid" -eq 1 ]; then usd_run=", usd: $run_usd" usd_day=", usd: $day_usd"; fi
|
||||
text="$(<"$example")"
|
||||
fill() { text="${text//@$1@/$2}"; }
|
||||
fill MODE "$mode"
|
||||
fill PLANNER_AGENT "${planner%%:*}"; fill PLANNER_MODEL "${planner#*:}"
|
||||
fill IMPLEMENTER_AGENT "${implementer%%:*}"; fill IMPLEMENTER_MODEL "${implementer#*:}"
|
||||
fill REVIEWER_AGENT "${reviewer%%:*}"; fill REVIEWER_MODEL "${reviewer#*:}"
|
||||
fill RUN_TOKENS "$run_tokens"; fill RUN_TURNS "$run_turns"; fill RUN_USD "$usd_run"
|
||||
fill DAY_TOKENS "$day_tokens"; fill DAY_TURNS "$day_turns"; fill DAY_USD "$usd_day"
|
||||
fill ENGINE "$engine"; fill FOLDER "$dir_abs"; fill BIND "${YGG_BIND:-/mnt/wsl/ygg-bind}"
|
||||
fill REALMS "$ygg_dir"; fill GRAPHS "$ygg_dir/voluspa"
|
||||
fill TASKS "$tasks"; fill SUITES "$suites"
|
||||
printf '%s\n' "$text" >"$work/factory.yaml"
|
||||
if grep -q '@[A-Z_]*@' "$work/factory.yaml"; then fail "$work/factory.yaml still has a place to fill: $(grep -o '@[A-Z_]*@' "$work/factory.yaml" | sort -u | tr '\n' ' ')"; fi
|
||||
echo "proof-run: made $work"
|
||||
echo "proof-run: the task is $task, suite $suite; the brief is the frozen one (SHA-256 $have)"
|
||||
[ "$paid" -eq 1 ] || echo "proof-run: no role is paid per token, so the config has no usd line"
|
||||
# What --continue needs to know about this proof run, one "key value" a line; every value was checked above.
|
||||
private "$work/proof.meta"
|
||||
printf 'name %s\nmode %s\nplanner %s\nimplementer %s\nreviewer %s\ntask %s\nsuite %s\nbrief %s\nbaseline %s\n' \
|
||||
"$name" "$mode" "$planner" "$implementer" "$reviewer" "$task" "$suite" "$have" "$baseline" >"$work/proof.meta"
|
||||
else
|
||||
project="$work/project"
|
||||
echo "proof-run: continuing run $cont of the proof run $name in $work"
|
||||
echo "proof-run: the task is $task, suite $suite; the brief's SHA-256 was $have"
|
||||
fi
|
||||
|
||||
# 6. doctor, and the plan.
|
||||
echo "proof-run: ygg doctor on that config"
|
||||
if ! "${ygg_arr[@]}" doctor --project "$project" --config "$work/factory.yaml"; then fail "ygg doctor refused the config; nothing was spent"; fi
|
||||
# Commands are printed with %q and never run through eval.
|
||||
q() { printf '%q ' "$@"; }
|
||||
cont_what="" # for --continue: what the continuation does
|
||||
if [ -z "$cont" ]; then
|
||||
run_argv=("${ygg_arr[@]}" run --project "$project" --config "$work/factory.yaml" --graph software --brief "$tasks/$task/instruction.md" --mode "$mode")
|
||||
elif [ -n "$gate_in" ]; then
|
||||
run_argv=("${ygg_arr[@]}" approve "$cont" "$gate_in" --project "$project")
|
||||
[ "$note_set" -eq 0 ] || run_argv+=(--note "$note")
|
||||
cont_what="approve $gate_in"
|
||||
else
|
||||
run_argv=("${ygg_arr[@]}" resume "$cont" --project "$project")
|
||||
cont_what="resume"
|
||||
fi
|
||||
build_argv=("${ygg_arr[@]}" sandbox build --all --config "$work/factory.yaml")
|
||||
run_cmd="$(q "${run_argv[@]}")"
|
||||
build_cmd="$(q "${build_argv[@]}")"
|
||||
timeout_argv=(timeout --signal=INT --kill-after=2m "$wall")
|
||||
echo "proof-run: the plan: mode $mode; planner $planner, implementer $implementer, reviewer $reviewer"
|
||||
echo "proof-run: the caps in $work/factory.yaml:"
|
||||
{ grep -E '^ per_(run|day):' "$work/factory.yaml" || true; } | sed 's/^/ /'
|
||||
[ -z "$cont_what" ] || echo "proof-run: the continuation: $cont_what"
|
||||
[ -z "$baseline" ] || echo "proof-run: after the run: compare with the baseline run $baseline, in ${base_dir%/.factory/runs/*}"
|
||||
[ "$clip" -eq 0 ] || echo "proof-run: after the run: the report is copied to the clipboard"
|
||||
if [ "$dry" -eq 1 ]; then
|
||||
[ "$build" -eq 0 ] || echo "proof-run: would run: $build_cmd"
|
||||
echo "proof-run: would run: $(q "${timeout_argv[@]}")$run_cmd"
|
||||
echo "proof-run: dry run: nothing was started and the credentials were not loaded"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# 7. The run. Only `ygg run` starts model sessions, so only it (and the output filter, which needs the exact values
|
||||
# to remove them) gets the credentials: each is sourced in its own subshell and nothing is exported by this shell.
|
||||
load_env() { # in a subshell only
|
||||
{ set +x; } 2>/dev/null # no trace of the sourced assignments, even under bash -x or SHELLOPTS=xtrace
|
||||
set -a
|
||||
# shellcheck source=/dev/null
|
||||
# The source's own stderr is silenced: on a syntax error bash prints the offending line, which can hold a secret.
|
||||
. "$envfile" 2>/dev/null || { echo "proof-run: the credentials file could not be read" >&2; return 1; }
|
||||
set +a
|
||||
}
|
||||
# The filter works on bytes (any byte survives), replaces the exact values, and if it cannot run it drains its input
|
||||
# so the writer is never killed by SIGPIPE: the output is then withheld, not shown unfiltered.
|
||||
redact() {
|
||||
(
|
||||
# shellcheck disable=SC2015 # the fallback is for both: the load failing and the filter failing
|
||||
load_env && python3 -I -u -c '
|
||||
import os, re, sys
|
||||
name = re.compile(rb"TOKEN|KEY|SECRET|PASSWORD|CREDENTIAL", re.I)
|
||||
vals = sorted({v for k, v in os.environb.items() if name.search(k) and len(v) >= 8}, key=len, reverse=True)
|
||||
if not vals:
|
||||
sys.exit(1) # no value to remove: fail closed
|
||||
out = sys.stdout.buffer
|
||||
for line in sys.stdin.buffer:
|
||||
for v in vals:
|
||||
line = line.replace(v, b"[REDACTED]")
|
||||
out.write(line)
|
||||
out.flush()
|
||||
' || { echo "proof-run: the output filter failed or the credentials could not be loaded; the output is withheld" >&2; cat >/dev/null; }
|
||||
)
|
||||
}
|
||||
# The run alone has the credentials. timeout is not --foreground: it gets its own process group and, at the wall clock
|
||||
# (INT, then KILL after the grace period) or when it is signalled, relays the signal to that whole group: ygg and the
|
||||
# model sessions it starts. The sandboxes are not in that group: they run under the container engine (conmon), and
|
||||
# ygg stops them on a clean stop; after an abnormal end (the wall clock, a kill) step 8 runs `ygg doctor`, which finds
|
||||
# and stops what is left. A terminal's Ctrl-C (INT) or hang-up (HUP) goes to this script's whole process group, not
|
||||
# to timeout's: the pipeline below ignores both, so the filter and tee keep reading until the run ends, and only the
|
||||
# trap's forward reaches ygg, once. timeout runs in the background of its subshell and its pid is kept, so that a
|
||||
# signal to this script (which bash would only handle after the pipeline) is passed on to it. timeout sets its own
|
||||
# handlers, so ygg starts with INT and HUP at their defaults. The trap also leaves a stop flag: a signal that comes
|
||||
# before the pid exists (during the build) is not lost, the build is stopped and no run is started.
|
||||
run_part() {
|
||||
stopped() { [ -f "$work/run.stop" ]; }
|
||||
stop_rc() { case "$(cat "$work/run.stop")" in INT) echo 130 ;; HUP) echo 129 ;; *) echo 143 ;; esac; }
|
||||
if [ "$build" -eq 1 ] && ! stopped; then
|
||||
"${build_argv[@]}" &
|
||||
bp=$!
|
||||
put "$work/build.pid" "$bp"
|
||||
if stopped; then kill -s TERM "$bp" 2>/dev/null; fi
|
||||
wait "$bp" || { stopped || { put "$work/run.rc" 2; return 0; }; }
|
||||
fi
|
||||
if stopped; then
|
||||
echo "proof-run: stopped by a signal; the run was not started"
|
||||
put "$work/run.rc" "$(stop_rc)"
|
||||
return 0
|
||||
fi
|
||||
[ -z "$cont_what" ] || echo "proof-run: continuing run $cont: $cont_what"
|
||||
(
|
||||
load_env || { put "$work/run.rc" 2; exit 0; }
|
||||
"${timeout_argv[@]}" "${run_argv[@]}" &
|
||||
tp=$!
|
||||
put "$work/run.pid" "$tp"
|
||||
if stopped; then kill -s "$(cat "$work/run.stop")" "$tp" 2>/dev/null; fi
|
||||
wait "$tp"
|
||||
put "$work/run.rc" "$?"
|
||||
)
|
||||
}
|
||||
out="$work/run.txt" # a continuation adds to it
|
||||
if [ -z "$cont" ]; then echo "proof-run: starting the run (wall clock $wall); this costs money"
|
||||
else echo "proof-run: continuing the run (wall clock $wall); this costs money"; fi
|
||||
set +e
|
||||
rm -f "$work/run.pid" "$work/run.rc" "$work/run.stop" "$work/run.hup" "$work/build.pid"
|
||||
# shellcheck disable=SC2329 # called by the traps
|
||||
on_signal() { # SIGNAL: leave the stop flag, then pass the signal to the build or to timeout (and so to ygg)
|
||||
local p f
|
||||
[ -f "$work/run.stop" ] || put "$work/run.stop" "$1" # the first signal sets the exit code of a stop
|
||||
[ "$1" != HUP ] || put "$work/run.hup" HUP # a hang-up is kept on its own: it can follow a Ctrl-C
|
||||
# The build starts with SIGINT and SIGHUP ignored (an async job, in the pipeline): it gets TERM for any signal.
|
||||
p="$(cat "$work/build.pid" 2>/dev/null)" || p=""
|
||||
[ -z "$p" ] || kill -s TERM "$p" 2>/dev/null
|
||||
p="$(cat "$work/run.pid" 2>/dev/null)" || p=""
|
||||
[ -z "$p" ] || kill -s "$1" "$p" 2>/dev/null
|
||||
return 0
|
||||
}
|
||||
trap 'on_signal TERM' TERM
|
||||
trap 'on_signal INT' INT
|
||||
trap 'on_signal HUP' HUP
|
||||
private "$out"
|
||||
(trap '' INT HUP; run_part 2>&1 | redact | tee -a "$out") &
|
||||
pl=$!
|
||||
while kill -0 "$pl" 2>/dev/null; do wait "$pl"; done
|
||||
trap - TERM INT
|
||||
trap '' HUP # the run is over: a hang-up from now on does not cut the report short
|
||||
rm -f "$work/run.pid" "$work/build.pid"
|
||||
# After a hang-up (alone, or after a Ctrl-C or TERM) the terminal is gone and every write to it fails: what is still
|
||||
# printed is kept in hangup.txt.
|
||||
if [ -f "$work/run.hup" ]; then private "$work/hangup.txt"; exec >>"$work/hangup.txt" 2>&1; fi
|
||||
set -e
|
||||
rc="$(cat "$work/run.rc" 2>/dev/null || echo 2)"
|
||||
[[ $rc =~ ^[0-9]+$ ]] || rc=2
|
||||
echo "proof-run: the run ended with exit code $rc (0 done, 1 a node failed, 2 refused, 3 stopped, 4 paused, 5 done with tasks given up, 124 the wall clock, 129, 130 or 143 ended by HUP, INT or TERM, 137 killed after the grace period, 141 SIGPIPE)"
|
||||
|
||||
# 8. The grade and the evidence. The run's id is the one continued, or the newest folder in the project's runs. None
|
||||
# of these needs a credential; the filter removes the exact values anyway, from anything a tool prints.
|
||||
if [ -n "$cont" ]; then id="$cont"
|
||||
else id="$(find "$project/.factory/runs" -mindepth 1 -maxdepth 1 -type d -printf '%f\n' 2>/dev/null | sort | tail -n 1 || true)"; fi
|
||||
report="$work/report.txt"
|
||||
if [ -z "$id" ] || ! [[ $id =~ ^[A-Za-z0-9._-]+$ ]]; then
|
||||
echo "proof-run: the run left no run folder in $project/.factory/runs; nothing to grade or report" >&2
|
||||
echo "proof-run: the folder is $work"
|
||||
exit "$rc"
|
||||
fi
|
||||
step() { # TITLE CMD...: a heading, then the command's output; its exit code is noted, never fatal
|
||||
echo "## $1"; shift
|
||||
"$@" || echo "(exited $?)"
|
||||
echo
|
||||
}
|
||||
evidence_part() {
|
||||
echo "# proof run $name: run $id, exit code $rc${cont_what:+ (continued: $cont_what)}"
|
||||
echo "mode $mode; planner $planner, implementer $implementer, reviewer $reviewer; brief SHA-256 $have"
|
||||
echo
|
||||
case "$rc" in
|
||||
0 | 3 | 4 | 5) ;;
|
||||
*) step "ygg doctor (an abnormal end: it finds and stops what the run left)" "${ygg_arr[@]}" doctor --project "$project" --config "$work/factory.yaml" ;;
|
||||
esac
|
||||
case "$rc" in 0 | 5) step "ygg grade" "${ygg_arr[@]}" grade "$id" --project "$project" ;; esac
|
||||
step "ygg metrics" "${ygg_arr[@]}" metrics "$id" --project "$project"
|
||||
if [ -n "$baseline" ] && [ "$baseline" = "$id" ]; then
|
||||
printf '## ygg metrics --compare\n(not run: the baseline is this run)\n\n'
|
||||
elif [ -n "$baseline" ]; then
|
||||
# ygg compares two runs of one project, and each proof run has its own: both run folders are linked into one.
|
||||
(umask 077 && mkdir -p "$work/compare/.factory/runs")
|
||||
ln -sfn "$base_dir" "$work/compare/.factory/runs/$baseline"
|
||||
ln -sfn "$project/.factory/runs/$id" "$work/compare/.factory/runs/$id"
|
||||
step "ygg metrics --compare $baseline $id" "${ygg_arr[@]}" metrics --compare "$baseline" "$id" --project "$work/compare"
|
||||
fi
|
||||
step "ygg report" "${ygg_arr[@]}" report "$id" --project "$project"
|
||||
if [ -f "$project/.factory/runs/$id/ledger.jsonl" ]; then
|
||||
step "cost" "$HERE/ygg-cost.sh" "$project/.factory/runs/$id/ledger.jsonl" "$work/factory.yaml"
|
||||
step "evidence" env YGG_CMD="$ygg" "$HERE/ygg-evidence.sh" --output "$out" "$project" "$id"
|
||||
fi
|
||||
}
|
||||
set +e # nothing after the run may change the run's exit code
|
||||
rep_start=0 # a continuation adds its report to report.txt; --clip copies the new one
|
||||
[ ! -f "$report" ] || rep_start="$(wc -c <"$report")"
|
||||
private "$report"
|
||||
evidence_part 2>&1 | redact | tee -a "$report" || true
|
||||
if [ "$clip" -eq 1 ]; then
|
||||
if command -v clip.exe >/dev/null 2>&1 && [ -f "$report" ]; then tail -c +"$((rep_start + 1))" "$report" | clip.exe; echo "proof-run: the report copied to the clipboard"
|
||||
else echo "proof-run: --clip needs clip.exe (WSL); the report is in $report" >&2; fi
|
||||
fi
|
||||
echo "proof-run: the folder is $work"
|
||||
|
||||
# 9. How to continue a run that stopped or paused: this script's --continue, which keeps the wall clock, the filter,
|
||||
# run.txt and the report. The raw ygg command is printed first, as a fallback. The last line is the --continue command.
|
||||
self=("$HERE/ygg-proof-run.sh" --continue "$id")
|
||||
by_hand="proof-run: or by hand, outside this script (no wall clock, no filter, no run.txt or report; load the credentials in that shell first):"
|
||||
case "$rc" in
|
||||
3)
|
||||
gate="$(python3 -I - "$project/.factory/runs/$id/run.json" 2>/dev/null <<'PY' || true
|
||||
import json, sys
|
||||
try:
|
||||
print(json.load(open(sys.argv[1])).get("pending", ""))
|
||||
except Exception:
|
||||
pass
|
||||
PY
|
||||
)"
|
||||
[[ $gate =~ ^[A-Za-z0-9._-]+$ ]] || gate="<gate>"
|
||||
echo "$by_hand"
|
||||
echo " $(q "$HERE/ygg-engine.sh" run --folder "$dir_abs" -- "${ygg_arr[@]}" approve "$id" "$gate" --project "$project")"
|
||||
echo "proof-run: the run stopped at the approval point $gate; read its gates folder, then continue it (add --note TEXT if you like):"
|
||||
echo " $(q "${self[@]}" --approve "$gate" --folder "$dir_abs" --env "$envfile")"
|
||||
;;
|
||||
4)
|
||||
echo "$by_hand"
|
||||
echo " $(q "$HERE/ygg-engine.sh" run --folder "$dir_abs" -- "${ygg_arr[@]}" resume "$id" --project "$project")"
|
||||
echo "proof-run: the run paused (the provider's quota); once it is back, continue it:"
|
||||
echo " $(q "${self[@]}" --resume --folder "$dir_abs" --env "$envfile")"
|
||||
;;
|
||||
esac
|
||||
exit "$rc"
|
||||
Reference in new issue
Block a user