mirror of
https://github.com/piabrota/mu.git
synced 2026-11-07 06:15:59 -10:00
ygg-proof-run.sh: the reviewer's reasoning effort and --reject (gallery-runtime T28, T29) (#78)
* The proof run sets the reviewer's reasoning effort, medium by default (F2) --reviewer-reasoning low|medium|high|off goes into the reviewer role of the proof config, the plan line and proof.meta; --continue refuses it as a setup flag. (gallery-runtime-T28) Co-Authored-By: Claude Opus 5.5 <[email protected]> * A proof folder from before --reviewer-reasoning can still be continued (gallery-runtime-T28) Without a reviewer_reasoning line in proof.meta, --continue shows "reasoning unset" in the plan; a bad value still fails. Co-Authored-By: Claude Opus 5.5 <[email protected]> * ygg-proof-run.sh --continue --reject GATE --reason TEXT, run as --approve is; re-wrap the header (gallery-runtime-T29) Co-Authored-By: Claude Opus 5.5 <[email protected]> * ygg-proof-run.sh: neutral step 9 wording, report header records reasoning, wrap fixes (gallery-runtime-T29) Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Co-authored-by: Bruno Guilhermo de Barros Valério <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
This commit is contained in:
4 files changed
+136
-36
No files matched your search
@@ -41,7 +41,7 @@ folder.
|
||||
| The grader smoke on a real engine (v0-H16), with the public sample suite | `scripts/ygg-smoke-grade.sh [--build]`. Never a hidden suite by default; `--task ID --suite-dir DIR --reference DIR [--suite-engine DIR]` points the grader at a suite store as it is (never copied or read) and grades the task's reference solution; its output holds numbers only |
|
||||
| The real-agent smoke (v0-H17): costs money; `--dry-run` first; the cost from the ledger; a start from a planted-fault leaf; another implementer; a note before the brief; the repair path from the ledger | `scripts/ygg-smoke-real.sh --dry-run`, then without it; `--from-leaf LEAF`, `--implementer AGENT:MODEL`, `--brief-note FILE` (narrows the implementer's task; the run is graded against a derived task version in its folder; the ygg line says if the build used go.work); `scripts/ygg-cost.sh LEDGER`, `scripts/ygg-repair-path.sh LEDGER`. It never prints the credentials file |
|
||||
| Compare reviewer models: costs money unless `--dry-run`; on the planted-fault leaves in `scripts/testdata/reviewer-bench`; runs the OpenCode version gleipnir pins (installed once with npm; `--opencode BIN`, `--any-opencode`) | `scripts/ygg-reviewer-bench.sh --model MODEL --leaf DIR --sessions N --dry-run`, then without it |
|
||||
| A Phase 6 proof run (v0-T290, v0-T291): costs money without `--dry-run`; `--dry-run` first; the `software` graph on the real task with three roles and caps you set, then the grade and the evidence; never reads the suites folder or prints the credentials | `scripts/ygg-proof-run.sh --name NAME --dry-run [--planner AGENT:MODEL --implementer AGENT:MODEL --reviewer AGENT:MODEL --run-usd N ...]`, then without `--dry-run`. `--build` builds the sandbox images first; `--baseline RUN_ID` compares with a run of an earlier proof run; `--continue RUN_ID --approve GATE` or `--resume` continues a stopped or paused run; `--clip` copies the report. The run's output is in `run.txt` and the evidence in `report.txt`, in `DIR/proof-NAME-TIME`. The example config is `scripts/testdata/proof/factory.proof.example.yaml` |
|
||||
| A Phase 6 proof run (v0-T290, v0-T291): costs money without `--dry-run`; `--dry-run` first; the `software` graph on the real task with three roles and caps you set, then the grade and the evidence; never reads the suites folder or prints the credentials | `scripts/ygg-proof-run.sh --name NAME --dry-run [--planner AGENT:MODEL --implementer AGENT:MODEL --reviewer AGENT:MODEL --run-usd N ...]`, then without `--dry-run`. `--reviewer-reasoning EFFORT` (low, medium, high or off; default `medium`) sets the reviewer's reasoning effort; `--build` builds the sandbox images first; `--baseline RUN_ID` compares with a run of an earlier proof run; `--continue RUN_ID --approve GATE`, `--reject GATE --reason TEXT` or `--resume` continues a stopped or paused run (a rejection starts sessions, so it needs the credentials as an approval does); `--clip` copies the report. The run's output is in `run.txt` and the evidence in `report.txt`, in `DIR/proof-NAME-TIME`. The example config is `scripts/testdata/proof/factory.proof.example.yaml` |
|
||||
| Build a plan from reference commits, or execute one task by task | `scripts/plan-*` (`plan-generate.py`, `plan-replay.py`, `plan-states.sh`, `plan-red.sh`, `plan-defences.py`; `plan-blocks.py`, `plan-task-start.sh`, `plan-task-done.sh`, `plan-task-land.sh`) |
|
||||
| Check that the repos have the required setup, and that the realms' shared files match | `scripts/as-bot.sh --read all scripts/check-repos.sh`; `--check shared` runs the realms check alone |
|
||||
| Scan git history for secrets | `scripts/scan-history.sh` |
|
||||
|
||||
+3
-2
@@ -1,13 +1,14 @@
|
||||
# The factory.yaml of a proof run (v0-H19 to v0-H23), with the places between two at signs that
|
||||
# scripts/ygg-proof-run.sh fills from its flags. It never writes your factory.yaml. The credentials come from the
|
||||
# environment, never from this file. A "usd" line is written only when a role pays per token (an opencode role):
|
||||
# the caps then count what the API billed, not the cost figure of a subscription's sessions.
|
||||
# the caps then count what the API billed, not the cost figure of a subscription's sessions. The reviewer's
|
||||
# "reasoning" is the effort it asks for (a claude-code agent ignores it).
|
||||
mode: @MODE@
|
||||
privacy: strict
|
||||
roles:
|
||||
planner: { agent: @PLANNER_AGENT@, model: @PLANNER_MODEL@ }
|
||||
implementer: { agent: @IMPLEMENTER_AGENT@, model: @IMPLEMENTER_MODEL@ }
|
||||
reviewer: { agent: @REVIEWER_AGENT@, model: @REVIEWER_MODEL@ }
|
||||
reviewer: { agent: @REVIEWER_AGENT@, model: @REVIEWER_MODEL@, reasoning: @REVIEWER_REASONING@ }
|
||||
budget:
|
||||
per_run: { tokens: @RUN_TOKENS@, turns: @RUN_TURNS@@RUN_USD@ }
|
||||
per_day: { tokens: @DAY_TOKENS@, turns: @DAY_TURNS@@DAY_USD@ }
|
||||
|
||||
@@ -57,7 +57,7 @@ cat >"$FB/ygg" <<'EOF3'
|
||||
#!/bin/sh
|
||||
# Fake ygg (v0-T291): records its arguments and the credential NAMES in its environment, answers each subcommand.
|
||||
# FAKE_RUN_RC is the exit code of `run`; FAKE_PLANT is a value it prints from run, metrics, report, grade and status.
|
||||
# FAKE_CONT_RC is the exit code of `approve` and `resume`.
|
||||
# FAKE_CONT_RC is the exit code of `approve`, `resume` and `reject`.
|
||||
echo "$*" >>"$FAKE_DIR/ygg.calls"
|
||||
printf '%s: %s\n' "$1" "$(env | grep -E '^(CLAUDE_CODE_OAUTH_TOKEN|OPENROUTER_API_KEY)=' | cut -d= -f1 | sort | tr '\n' ' ')" >>"$FAKE_DIR/ygg.env"
|
||||
proj="" prev=""
|
||||
@@ -84,7 +84,7 @@ case "$1" in
|
||||
while :; do sleep 0.2; done
|
||||
fi
|
||||
exit "${FAKE_RUN_RC:-0}" ;;
|
||||
approve | resume)
|
||||
approve | resume | reject)
|
||||
echo "$1 runs"
|
||||
[ -z "${FAKE_PLANT:-}" ] || echo "$1 saw $FAKE_PLANT"
|
||||
exit "${FAKE_CONT_RC:-0}" ;;
|
||||
@@ -206,7 +206,7 @@ check "the umbrella.yaml sets root: true" grep -q '^root: true$' "$P/project/umb
|
||||
check "the project folder is DIR/proof-<name>-<time>/project" bash -c '[[ $1 =~ /proof-first-[0-9]{8}-[0-9]{6}$ ]]' _ "$P"
|
||||
check "the config has the default mode" grep -q '^mode: key-points$' "$FY"
|
||||
check "the config has the planner" grep -q 'planner: { agent: claude-code, model: sonnet }' "$FY"
|
||||
check "the config has the reviewer" grep -q 'reviewer: { agent: opencode, model: openrouter/qwen/qwen3.8-27b }' "$FY"
|
||||
check "the config has the reviewer" grep -q 'reviewer: { agent: opencode, model: openrouter/qwen/qwen3.8-27b, reasoning: medium }' "$FY"
|
||||
check "the config has the run caps with usd" grep -q 'per_run: { tokens: 12000000, turns: 1500, usd: 5 }' "$FY"
|
||||
check "the config has the day caps with usd" grep -q 'per_day: { tokens: 24000000, turns: 2000, usd: 10 }' "$FY"
|
||||
check "the config has the tasks folder" grep -q -x " tasks_dir: $TK" "$FY"
|
||||
@@ -225,7 +225,28 @@ check "the mode asked for is in the config" grep -q '^mode: unattended$' "$FY"
|
||||
check "the caps asked for are in the config" grep -q 'per_run: { tokens: 20, turns: 10 }' "$FY"
|
||||
check "the day caps asked for are in the config" grep -q 'per_day: { tokens: 40, turns: 30 }' "$FY"
|
||||
check "no usd line when no role is paid" bash -c '! grep -q "usd:" "$1"' _ "$FY"
|
||||
check "the reviewer asked for is in the config" grep -q 'reviewer: { agent: claude-code, model: opus }' "$FY"
|
||||
check "the reviewer asked for is in the config" grep -q 'reviewer: { agent: claude-code, model: opus, reasoning: medium }' "$FY"
|
||||
|
||||
# --- gallery-runtime T28: the reviewer's reasoning effort (F2)
|
||||
rm -rf "$D"/proof-* "$T/ygg.calls"
|
||||
run proof --name reason --dry-run
|
||||
P="$(made)"; FY="$P/factory.yaml"
|
||||
check "the reviewer role asks for reasoning medium by default" grep -q 'reviewer: { agent: opencode, model: openrouter/qwen/qwen3.8-27b, reasoning: medium }' "$FY"
|
||||
check "the planner and the implementer have no reasoning" bash -c '! grep -E "^ (planner|implementer):" "$1" | grep -q reasoning' _ "$FY"
|
||||
check "the plan says reviewer AGENT:MODEL (reasoning medium)" has "reviewer opencode:openrouter/qwen/qwen3.8-27b (reasoning medium)"
|
||||
check "the config with reasoning passes ygg doctor" grep -q -- "^doctor --project .*/project --config $FY\$" "$T/ygg.calls"
|
||||
check "the config has no place left to fill" bash -c '! grep -q "@[A-Z_]*@" "$1"' _ "$FY"
|
||||
rm -rf "$D"/proof-*
|
||||
run proof --name reasonoff --dry-run --reviewer-reasoning off "${AC[@]}"
|
||||
P="$(made)"
|
||||
check "--reviewer-reasoning off writes off, for a claude-code reviewer too" grep -q 'reviewer: { agent: claude-code, model: opus, reasoning: off }' "$P/factory.yaml"
|
||||
check "--reviewer-reasoning off shows in the plan" has "(reasoning off)"
|
||||
for v in low high; do
|
||||
rm -rf "$D"/proof-*; run proof --name reasonv --dry-run --reviewer-reasoning "$v"
|
||||
check "--reviewer-reasoning $v writes $v" grep -q "reasoning: $v }" "$(made)/factory.yaml"
|
||||
done
|
||||
runq proof --name ok --reviewer-reasoning extreme; check "a reasoning outside low, medium, high and off is refused" refuses "--reviewer-reasoning"
|
||||
runq proof --name ok --reviewer-reasoning 'low }'; check "a reasoning with YAML characters is refused" refuses "--reviewer-reasoning"
|
||||
|
||||
# --- the dry run
|
||||
rm -rf "$D"/proof-* "$T/ygg.calls"
|
||||
@@ -585,13 +606,55 @@ check "--continue --dry-run: exit 0, approve never called" bash -c '[ "$1" -eq 0
|
||||
check "--continue --dry-run: it shows the command under the timeout" bash -c 'grep "would run: timeout --signal=INT --kill-after=2m 8h" "$1" | grep -q -F "approve RUN-1 ship-gate --project"' _ "$T/out"
|
||||
refusesc() { [ "$rc" -eq 2 ] && has "$1" && [ ! -s "$T/ygg.calls" ]; } # exit 2, the reason, nothing started
|
||||
clearlogs; runc cont --continue NOPE-RUN --resume; check "--continue with no such run: refused before anything starts" refusesc "no proof folder under"
|
||||
clearlogs; runc cont --continue RUN-1 --approve ship-gate --resume; check "--continue with --approve and --resume: refused" refusesc "not both"
|
||||
clearlogs; runc cont --continue RUN-1; check "--continue with neither --approve nor --resume: refused" refusesc "--approve GATE or --resume"
|
||||
clearlogs; runc cont --continue RUN-1 --approve ship-gate --resume; check "--continue with --approve and --resume: refused" refusesc "only one"
|
||||
clearlogs; runc cont --continue RUN-1; check "--continue with no action: refused" refusesc "--approve GATE, --reject GATE --reason TEXT or --resume"
|
||||
clearlogs; runc cont --approve ship-gate; check "--approve without --continue: refused" refusesc "go with --continue"
|
||||
clearlogs; runc cont --continue RUN-1 --resume --note hi; check "--note without --approve: refused" refusesc "--note goes with --approve"
|
||||
clearlogs; runc cont --continue RUN-1 --resume --name other; check "--continue with a setup flag: refused" refusesc "keeps the config"
|
||||
clearlogs; runc cont --continue RUN-1 --approve ship-gate --note $'a\nb'; check "--note with a control character: refused" refusesc "control character"
|
||||
clearlogs; runc cont --continue .. --resume; check "--continue .. is not a run id" refusesc "not a run id"
|
||||
clearlogs; runc cont --continue RUN-1 --resume --reviewer-reasoning low; check "--continue refuses --reviewer-reasoning as a setup flag" refusesc "keeps the config"
|
||||
# --- gallery-runtime T29: --reject (F3), run as --approve is run
|
||||
fresh; plant_base; FAKE_RUN_RC=3 run proof --name go --baseline "$OLD"; W="$(RUND)"; clearlogs
|
||||
FAKE_PLANT=fake-claude-VALUE-1 runc cont --continue RUN-1 --reject ship-gate --reason 'the gallery has no empty state'
|
||||
check "--reject after 3: exit 0, the continuation's code" test "$rc" -eq 0
|
||||
check "--reject: doctor, reject, grade, metrics, the compare, report" test "$(calls)" = "doctor,reject,grade,metrics,metrics,report"
|
||||
check "--reject: ygg reject runs once, with the run, the gate, the reason and the project" bash -c '[ "$(grep -c "^reject " "$1")" = 1 ] && grep -q -x -F "reject RUN-1 ship-gate --reason the gallery has no empty state --project $2/project" "$1"' _ "$T/ygg.calls" "$W"
|
||||
check "--reject: it runs under the same timeout" bash -c 'head -n 1 "$1" | grep -q -F -- "--signal=INT --kill-after=2m 8h $2 reject RUN-1 ship-gate"' _ "$T/timeout.args" "$FB/ygg"
|
||||
check "--reject: only reject has the credentials" bash -c 'grep -q -x "reject: CLAUDE_CODE_OAUTH_TOKEN OPENROUTER_API_KEY " "$1" && [ "$(grep -c -v -E "^reject: " "$1")" = "$(grep -c -E "^[a-z]+: $" "$1")" ]' _ "$T/ygg.env"
|
||||
check "--reject: the output is added to the folder's run.txt" bash -c 'grep -q "the run runs" "$1" && grep -q "reject runs" "$1"' _ "$W/run.txt"
|
||||
check "--reject: a planted value is filtered from run.txt and the output" bash -c '! grep -q -F fake-claude-VALUE-1 "$1" && ! grep -q -F fake-claude-VALUE-1 "$2" && grep -q REDACTED "$1"' _ "$W/run.txt" "$T/out"
|
||||
check "--reject: the report header records the reviewer's reasoning" grep -q "reviewer opencode:openrouter/qwen/qwen3.8-27b (reasoning medium); brief SHA-256" "$W/report.txt"
|
||||
check "--reject: the report says what was continued" bash -c 'grep -q "(continued: reject ship-gate)" "$1"' _ "$W/report.txt"
|
||||
clearlogs; FAKE_CONT_RC=3 runc cont --continue RUN-1 --reject ship-gate --reason 'again'
|
||||
check "--reject ending 3: the exit code is 3, with no grade" bash -c '[ "$1" -eq 3 ] && [ "$2" = "doctor,reject,metrics,metrics,report" ]' _ "$rc" "$(calls)"
|
||||
check "--reject ending 3: step 9 prints the --reject form beside the --approve one" bash -c 'grep -F -- "--continue RUN-1 --reject ship-gate --reason \"...\"" "$1" | grep -q -F -- "--folder $2" && tail -n 1 "$1" | grep -q -F -- "--continue RUN-1 --approve ship-gate"' _ "$T/out" "$D"
|
||||
check "--reject ending 3: step 9 words the rejection as conditional" has "If ygg offered a rejection above"
|
||||
check "--reject ending 3: the raw fallback names ygg reject" bash -c 'grep -F "ygg-engine.sh run" "$1" | grep -q -F "reject RUN-1 ship-gate --reason"' _ "$T/out"
|
||||
clearlogs; runc cont --continue RUN-1 --reject ship-gate --reason 'no' --dry-run
|
||||
check "--reject --dry-run: exit 0, reject never called" bash -c '[ "$1" -eq 0 ] && ! grep -q "^reject" "$2"' _ "$rc" "$T/ygg.calls"
|
||||
check "--reject --dry-run: it shows the command under the timeout" bash -c 'grep "would run: timeout --signal=INT --kill-after=2m 8h" "$1" | grep -q -F "reject RUN-1 ship-gate --reason no --project"' _ "$T/out"
|
||||
clearlogs; runc cont --continue RUN-1 --reject ship-gate; check "--reject without --reason: refused" refusesc "--reject needs --reason"
|
||||
clearlogs; runc cont --continue RUN-1 --approve ship-gate --reason why; check "--reason without --reject: refused" refusesc "--reason goes with --reject"
|
||||
clearlogs; runc cont --continue RUN-1 --resume --reason why; check "--reason with --resume: refused" refusesc "--reason goes with --reject"
|
||||
clearlogs; runc cont --continue RUN-1 --reject ship-gate --approve ship-gate --reason why; check "--reject with --approve: refused" refusesc "only one"
|
||||
clearlogs; runc cont --continue RUN-1 --reject ship-gate --resume --reason why; check "--reject with --resume: refused" refusesc "only one"
|
||||
clearlogs; runc cont --continue RUN-1 --reject ship-gate --reason why --note hi; check "--note with --reject: refused" refusesc "--note goes with --approve"
|
||||
clearlogs; runc cont --continue RUN-1 --reject ship-gate --reason ' '; check "--reject with a blank reason: refused" refusesc "--reason is blank"
|
||||
clearlogs; runc cont --continue RUN-1 --reject ship-gate --reason $'a\nb'; check "--reject with a control character in the reason: refused" refusesc "control character"
|
||||
clearlogs; runc cont --continue RUN-1 --reject 'a b' --reason why; check "--reject with a bad gate name: refused" refusesc "not a gate name"
|
||||
clearlogs; runc cont --reject ship-gate --reason why; check "--reject without --continue: refused" refusesc "go with --continue"
|
||||
clearlogs; runc cont --reason why; check "--reason without --continue: refused" refusesc "go with --continue"
|
||||
fresh; FAKE_RUN_RC=4 run proof --name go; W="$(RUND)"; clearlogs
|
||||
cp "$W/proof.meta" "$T/meta.keep"
|
||||
grep -v '^reviewer_reasoning ' "$T/meta.keep" >"$W/proof.meta"
|
||||
clearlogs; runc cont --continue RUN-1 --resume --dry-run
|
||||
check "--continue on a proof.meta without reviewer_reasoning goes on (exit 0)" test "$rc" -eq 0
|
||||
check "--continue on such a folder prints the unset form" has "(reasoning unset: a folder from before --reviewer-reasoning)"
|
||||
sed 's/^reviewer_reasoning .*/reviewer_reasoning extreme/' "$T/meta.keep" >"$W/proof.meta"
|
||||
clearlogs; runc cont --continue RUN-1 --resume --dry-run
|
||||
check "--continue on a proof.meta with a bad reviewer_reasoning is refused" refusesc "reviewer reasoning"
|
||||
cp "$T/meta.keep" "$W/proof.meta"
|
||||
|
||||
[ "$fail" -eq 0 ] && echo "all ok"
|
||||
exit "$fail"
|
||||
+63
-27
@@ -4,18 +4,21 @@
|
||||
# --compare` compares two factories and not two setups. IT WILL COST MONEY (without --dry-run):
|
||||
# the script makes the project and the config, runs the factory, grades the run and prints the evidence.
|
||||
# ygg-proof-run.sh --name NAME [--mode key-points|unattended|step-by-step] [--planner AGENT:MODEL]
|
||||
# [--implementer AGENT:MODEL] [--reviewer AGENT:MODEL] [--run-turns N] [--run-tokens N]
|
||||
# [--run-usd N] [--day-turns N] [--day-tokens N] [--day-usd N] [--baseline RUN_ID] [--dry-run]
|
||||
# [--build] [--folder DIR] [--env FILE] [--tasks DIR] [--suites DIR] [--yggdrasil DIR]
|
||||
# [--wall TIME] [--clip]
|
||||
# ygg-proof-run.sh --continue RUN_ID (--approve GATE [--note TEXT] | --resume) [--baseline RUN_ID] [--dry-run]
|
||||
# [--folder DIR] [--env FILE] [--wall TIME] [--clip]
|
||||
# [--implementer AGENT:MODEL] [--reviewer AGENT:MODEL]
|
||||
# [--reviewer-reasoning low|medium|high|off] [--run-turns N] [--run-tokens N] [--run-usd N]
|
||||
# [--day-turns N] [--day-tokens N] [--day-usd N] [--baseline RUN_ID] [--dry-run] [--build]
|
||||
# [--folder DIR] [--env FILE] [--tasks DIR] [--suites DIR] [--yggdrasil DIR] [--wall TIME] [--clip]
|
||||
# ygg-proof-run.sh --continue RUN_ID (--approve GATE [--note TEXT] | --reject GATE --reason TEXT | --resume)
|
||||
# [--baseline RUN_ID] [--dry-run] [--folder DIR] [--env FILE] [--wall TIME] [--clip]
|
||||
# AGENT is claude-code or opencode; an opencode MODEL is openrouter/<model>; a MODEL has only A-Za-z0-9._/:-
|
||||
# because it goes into a YAML line. Defaults: key-points; planner and implementer claude-code:sonnet; reviewer
|
||||
# opencode:openrouter/qwen/qwen3.8-27b; a run 1,500 turns, 12,000,000 tokens, $5; a day 2,000 turns, 24,000,000
|
||||
# tokens, $10; --wall 8h; DIR $HOME/ygg-it; FILE $HOME/.config/ouroboric/factory.env; --tasks ../mimir;
|
||||
# --suites $HOME/ygg-it/hidden-suites; --yggdrasil the umbrella beside this repo. The task is client-gallery-lite
|
||||
# (YGG_PROOF_TASK names another, for tests); its suite is the `suite` key of its task.toml.
|
||||
# opencode:openrouter/qwen/qwen3.8-27b with --reviewer-reasoning medium (the reviewer role's reasoning effort: a
|
||||
# Qwen reviewer at the default high stalled a whole session in the rehearsal; a claude-code agent ignores the
|
||||
# setting, and the line is written for every agent); a run 1,500 turns, 12,000,000 tokens, $5; a day 2,000
|
||||
# turns, 24,000,000 tokens, $10; --wall 8h; DIR $HOME/ygg-it; FILE $HOME/.config/ouroboric/factory.env;
|
||||
# --tasks ../mimir; --suites $HOME/ygg-it/hidden-suites; --yggdrasil the umbrella beside this repo.
|
||||
# The task is client-gallery-lite (YGG_PROOF_TASK names another, for tests); its suite is the `suite` key of its
|
||||
# task.toml.
|
||||
# In order, it
|
||||
# 1. checks the flags (exit 2 on the first bad one), the engine (ygg-engine.sh; YGG_ENGINE_CHECK=0 skips it)
|
||||
# and the tools (git, python3, timeout, sha256sum);
|
||||
@@ -51,17 +54,20 @@
|
||||
# hold it, or the script refuses before it makes anything.
|
||||
# --continue RUN_ID continues a run that stopped or paused, in the proof folder under DIR that holds it (the setup
|
||||
# flags are refused: the run keeps its config). It checks the credentials and runs `ygg doctor`, then `ygg approve
|
||||
# RUN_ID GATE --project P [--note TEXT]` or `ygg resume RUN_ID --project P` as step 7 runs `ygg run` (the same timeout,
|
||||
# the credentials in that subshell only, the filter, the output added to run.txt), then steps 8 and 9, with the
|
||||
# baseline the proof run recorded unless --baseline names another. The raw `ygg approve` and `ygg resume` commands are
|
||||
# printed too, as a fallback: they run without the wall clock, the filter or the report.
|
||||
# RUN_ID GATE --project P [--note TEXT]`, `ygg reject RUN_ID GATE --reason TEXT --project P` or `ygg resume RUN_ID
|
||||
# --project P` as step 7 runs `ygg run` (the same timeout, the credentials in that subshell only, the filter, the
|
||||
# output added to run.txt; a rejection advances the run at once, so it starts sessions as an approval does), then
|
||||
# steps 8 and 9, with the baseline the proof run recorded unless --baseline names another. The raw `ygg approve`,
|
||||
# `ygg reject` and `ygg resume` commands are printed too, as a fallback: they run without the wall clock, the filter
|
||||
# or the report.
|
||||
# For tests and other machines: YGG_CMD (how ygg runs), YGG_BIND (the bind mount, default /mnt/wsl/ygg-bind).
|
||||
set -euo pipefail
|
||||
shopt -u patsub_replacement 2>/dev/null || true
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
usage() {
|
||||
echo "usage: ygg-proof-run.sh --name NAME [--mode MODE] [--planner|--implementer|--reviewer AGENT:MODEL] [--run-turns|--run-tokens|--run-usd|--day-turns|--day-tokens|--day-usd N] [--baseline RUN_ID] [--dry-run] [--build] [--folder DIR] [--env FILE] [--tasks DIR] [--suites DIR] [--yggdrasil DIR] [--wall TIME] [--clip]" >&2
|
||||
echo " ygg-proof-run.sh --continue RUN_ID (--approve GATE [--note TEXT] | --resume) [--baseline RUN_ID] [--dry-run] [--folder DIR] [--env FILE] [--wall TIME] [--clip]" >&2
|
||||
echo "usage: ygg-proof-run.sh --name NAME [--mode MODE] [--planner|--implementer|--reviewer AGENT:MODEL] [--reviewer-reasoning low|medium|high|off] [--run-turns|--run-tokens|--run-usd|--day-turns|--day-tokens|--day-usd N] [--baseline RUN_ID] [--dry-run] [--build] [--folder DIR] [--env FILE] [--tasks DIR] [--suites DIR] [--yggdrasil DIR] [--wall TIME] [--clip]" >&2
|
||||
echo " ygg-proof-run.sh --continue RUN_ID (--approve GATE [--note TEXT] | --reject GATE --reason TEXT | --resume)" >&2
|
||||
echo " [--baseline RUN_ID] [--dry-run] [--folder DIR] [--env FILE] [--wall TIME] [--clip]" >&2
|
||||
exit 2
|
||||
}
|
||||
fail() { echo "proof-run: $*" >&2; exit 2; }
|
||||
@@ -71,14 +77,16 @@ fail() { echo "proof-run: $*" >&2; exit 2; }
|
||||
private() { (umask 077 && : >>"$1") && chmod 600 "$1"; } # FILE: made private; what it holds stays
|
||||
put() { (umask 077 && printf '%s\n' "$2" >"$1"); } # FILE TEXT: a small control file, private
|
||||
name="" mode=key-points dry=0 build=0 clip=0 baseline="" wall=8h
|
||||
cont="" gate_in="" note="" note_set=0 resume=0 setup_flags="" # --continue; the setup flags given (a continued run keeps its config)
|
||||
# --continue; setup_flags are the setup flags given (a continued run keeps its config)
|
||||
cont="" gate_in="" reject_in="" reason="" reason_set=0 note="" note_set=0 resume=0 setup_flags=""
|
||||
planner=claude-code:sonnet implementer=claude-code:sonnet reviewer=opencode:openrouter/qwen/qwen3.8-27b
|
||||
reviewer_reasoning=medium
|
||||
run_turns=1500 run_tokens=12000000 run_usd=5 day_turns=2000 day_tokens=24000000 day_usd=10
|
||||
dir="$HOME/ygg-it" envfile="$HOME/.config/ouroboric/factory.env" tasks="$HERE/../../mimir"
|
||||
suites="$HOME/ygg-it/hidden-suites" ygg_dir="$HERE/../../yggdrasil"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--name | --mode | --planner | --implementer | --reviewer | --run-* | --day-* | --build | --tasks | --suites | --yggdrasil) setup_flags+=" $1" ;;
|
||||
--name | --mode | --planner | --implementer | --reviewer | --reviewer-reasoning | --run-* | --day-* | --build | --tasks | --suites | --yggdrasil) setup_flags+=" $1" ;;
|
||||
esac
|
||||
case "$1" in
|
||||
--dry-run) dry=1; shift ;;
|
||||
@@ -86,6 +94,8 @@ while [ $# -gt 0 ]; do
|
||||
--resume) resume=1; shift ;;
|
||||
--continue) [ $# -ge 2 ] || usage; cont="$2"; shift 2 ;;
|
||||
--approve) [ $# -ge 2 ] || usage; gate_in="$2"; shift 2 ;;
|
||||
--reject) [ $# -ge 2 ] || usage; reject_in="$2"; shift 2 ;;
|
||||
--reason) [ $# -ge 2 ] || usage; reason="$2"; reason_set=1; shift 2 ;;
|
||||
--note) [ $# -ge 2 ] || usage; note="$2"; note_set=1; shift 2 ;;
|
||||
--clip) clip=1; shift ;;
|
||||
--name) [ $# -ge 2 ] || usage; name="$2"; shift 2 ;;
|
||||
@@ -93,6 +103,7 @@ while [ $# -gt 0 ]; do
|
||||
--planner) [ $# -ge 2 ] || usage; planner="$2"; shift 2 ;;
|
||||
--implementer) [ $# -ge 2 ] || usage; implementer="$2"; shift 2 ;;
|
||||
--reviewer) [ $# -ge 2 ] || usage; reviewer="$2"; shift 2 ;;
|
||||
--reviewer-reasoning) [ $# -ge 2 ] || usage; reviewer_reasoning="$2"; shift 2 ;;
|
||||
--run-turns) [ $# -ge 2 ] || usage; run_turns="$2"; shift 2 ;;
|
||||
--run-tokens) [ $# -ge 2 ] || usage; run_tokens="$2"; shift 2 ;;
|
||||
--run-usd) [ $# -ge 2 ] || usage; run_usd="$2"; shift 2 ;;
|
||||
@@ -115,16 +126,28 @@ run_id_re='^[A-Za-z0-9][A-Za-z0-9._-]*$' # a run id is a folder name: never . or
|
||||
if [ -n "$cont" ]; then
|
||||
[[ $cont =~ $run_id_re ]] || fail "--continue '$cont' is not a run id"
|
||||
[ -z "$setup_flags" ] || fail "--continue keeps the config of the run's proof folder; it takes none of$setup_flags"
|
||||
if [ -n "$gate_in" ] && [ "$resume" -eq 1 ]; then fail "--continue takes --approve GATE or --resume, not both"; fi
|
||||
if [ -z "$gate_in" ] && [ "$resume" -eq 0 ]; then fail "--continue takes --approve GATE or --resume"; fi
|
||||
actions=0
|
||||
[ -z "$gate_in" ] || actions=$((actions + 1))
|
||||
[ -z "$reject_in" ] || actions=$((actions + 1))
|
||||
[ "$resume" -eq 0 ] || actions=$((actions + 1))
|
||||
if [ "$actions" -gt 1 ]; then fail "--continue takes --approve GATE, --reject GATE --reason TEXT or --resume, only one"; fi
|
||||
if [ "$actions" -eq 0 ]; then fail "--continue takes --approve GATE, --reject GATE --reason TEXT or --resume"; fi
|
||||
if [ -n "$gate_in" ] && ! [[ $gate_in =~ $run_id_re ]]; then fail "--approve '$gate_in' is not a gate name"; fi
|
||||
if [ -n "$reject_in" ] && ! [[ $reject_in =~ $run_id_re ]]; then fail "--reject '$reject_in' is not a gate name"; fi
|
||||
if [ "$note_set" -eq 1 ] && [ -z "$gate_in" ]; then fail "--note goes with --approve"; fi
|
||||
if [ "$reason_set" -eq 1 ] && [ -z "$reject_in" ]; then fail "--reason goes with --reject"; fi
|
||||
if [ -n "$reject_in" ] && [ "$reason_set" -eq 0 ]; then fail "--reject needs --reason TEXT: the reason goes to the agents"; fi
|
||||
if [[ $note == *[[:cntrl:]]* ]]; then fail "--note has a control character; a note is one line of text"; fi
|
||||
if [[ $reason == *[[:cntrl:]]* ]]; then fail "--reason has a control character; a reason is one line of text"; fi
|
||||
if [ "$reason_set" -eq 1 ] && [[ $reason =~ ^[[:space:]]*$ ]]; then fail "--reason is blank; say why the gate is rejected"; fi
|
||||
else
|
||||
if [ -n "$gate_in" ] || [ "$resume" -eq 1 ] || [ "$note_set" -eq 1 ]; then fail "--approve, --resume and --note go with --continue RUN_ID"; fi
|
||||
if [ -n "$gate_in" ] || [ -n "$reject_in" ] || [ "$resume" -eq 1 ] || [ "$note_set" -eq 1 ] || [ "$reason_set" -eq 1 ]; then
|
||||
fail "--approve, --reject, --reason, --resume and --note go with --continue RUN_ID"
|
||||
fi
|
||||
[ -n "$name" ] || fail "--name is needed: the run's name, such as baseline-1"
|
||||
[[ $name =~ ^[a-z0-9-]+$ ]] || fail "--name '$name' has characters outside a-z0-9-"
|
||||
case "$mode" in key-points | unattended | step-by-step) ;; *) fail "--mode is key-points, unattended or step-by-step, not '$mode'" ;; esac
|
||||
case "$reviewer_reasoning" in low | medium | high | off) ;; *) fail "--reviewer-reasoning is low, medium, high or off, not '$reviewer_reasoning'" ;; esac
|
||||
for pair in run-turns:"$run_turns" run-tokens:"$run_tokens" day-turns:"$day_turns" day-tokens:"$day_tokens"; do
|
||||
[[ ${pair#*:} =~ ^[1-9][0-9]*$ ]] || fail "--${pair%%:*} is a whole number above 0, not '${pair#*:}'"
|
||||
done
|
||||
@@ -230,9 +253,13 @@ else
|
||||
[ -f "$work/factory.yaml" ] && [ -f "$work/proof.meta" ] || fail "$work has no factory.yaml or proof.meta: this script did not set it up"
|
||||
meta() { sed -n "s/^$1 //p" "$work/proof.meta" | head -n 1; } # a key's value; each is checked below as a flag is
|
||||
name="$(meta name)" mode="$(meta mode)" planner="$(meta planner)" implementer="$(meta implementer)" reviewer="$(meta reviewer)"
|
||||
reviewer_reasoning="$(meta reviewer_reasoning)"
|
||||
task="$(meta task)" suite="$(meta suite)" have="$(meta brief)"
|
||||
[[ $name =~ ^[a-z0-9-]+$ ]] || fail "$work/proof.meta has no proof run name"
|
||||
case "$mode" in key-points | unattended | step-by-step) ;; *) fail "$work/proof.meta has no mode" ;; esac
|
||||
# A folder from before --reviewer-reasoning has no such line: its factory.yaml is written, so the value is only shown.
|
||||
if ! grep -q '^reviewer_reasoning ' "$work/proof.meta"; then reviewer_reasoning="unset"
|
||||
else case "$reviewer_reasoning" in low | medium | high | off) ;; *) fail "$work/proof.meta has no valid reviewer reasoning" ;; esac; fi
|
||||
roles_ok
|
||||
[[ $task =~ ^[A-Za-z0-9._-]+$ && $suite =~ ^[a-z0-9][a-z0-9._-]*$ && $have =~ ^[0-9a-f]{64}$ ]] || fail "$work/proof.meta has no task, suite or brief SHA-256"
|
||||
if [ -z "$baseline" ]; then
|
||||
@@ -278,7 +305,7 @@ if [ -z "$cont" ]; then
|
||||
fill MODE "$mode"
|
||||
fill PLANNER_AGENT "${planner%%:*}"; fill PLANNER_MODEL "${planner#*:}"
|
||||
fill IMPLEMENTER_AGENT "${implementer%%:*}"; fill IMPLEMENTER_MODEL "${implementer#*:}"
|
||||
fill REVIEWER_AGENT "${reviewer%%:*}"; fill REVIEWER_MODEL "${reviewer#*:}"
|
||||
fill REVIEWER_AGENT "${reviewer%%:*}"; fill REVIEWER_MODEL "${reviewer#*:}"; fill REVIEWER_REASONING "$reviewer_reasoning"
|
||||
fill RUN_TOKENS "$run_tokens"; fill RUN_TURNS "$run_turns"; fill RUN_USD "$usd_run"
|
||||
fill DAY_TOKENS "$day_tokens"; fill DAY_TURNS "$day_turns"; fill DAY_USD "$usd_day"
|
||||
fill ENGINE "$engine"; fill FOLDER "$dir_abs"; fill BIND "${YGG_BIND:-/mnt/wsl/ygg-bind}"
|
||||
@@ -291,8 +318,8 @@ if [ -z "$cont" ]; then
|
||||
[ "$paid" -eq 1 ] || echo "proof-run: no role is paid per token, so the config has no usd line"
|
||||
# What --continue needs to know about this proof run, one "key value" a line; every value was checked above.
|
||||
private "$work/proof.meta"
|
||||
printf 'name %s\nmode %s\nplanner %s\nimplementer %s\nreviewer %s\ntask %s\nsuite %s\nbrief %s\nbaseline %s\n' \
|
||||
"$name" "$mode" "$planner" "$implementer" "$reviewer" "$task" "$suite" "$have" "$baseline" >"$work/proof.meta"
|
||||
printf 'name %s\nmode %s\nplanner %s\nimplementer %s\nreviewer %s\nreviewer_reasoning %s\ntask %s\nsuite %s\nbrief %s\nbaseline %s\n' \
|
||||
"$name" "$mode" "$planner" "$implementer" "$reviewer" "$reviewer_reasoning" "$task" "$suite" "$have" "$baseline" >"$work/proof.meta"
|
||||
else
|
||||
project="$work/project"
|
||||
echo "proof-run: continuing run $cont of the proof run $name in $work"
|
||||
@@ -304,6 +331,7 @@ echo "proof-run: ygg doctor on that config"
|
||||
if ! "${ygg_arr[@]}" doctor --project "$project" --config "$work/factory.yaml"; then fail "ygg doctor refused the config; nothing was spent"; fi
|
||||
# Commands are printed with %q and never run through eval.
|
||||
q() { printf '%q ' "$@"; }
|
||||
qt() { local t; t="$(q "$@")"; printf '%s' "${t% }"; } # q without its last space
|
||||
cont_what="" # for --continue: what the continuation does
|
||||
if [ -z "$cont" ]; then
|
||||
run_argv=("${ygg_arr[@]}" run --project "$project" --config "$work/factory.yaml" --graph software --brief "$tasks/$task/instruction.md" --mode "$mode")
|
||||
@@ -311,6 +339,9 @@ elif [ -n "$gate_in" ]; then
|
||||
run_argv=("${ygg_arr[@]}" approve "$cont" "$gate_in" --project "$project")
|
||||
[ "$note_set" -eq 0 ] || run_argv+=(--note "$note")
|
||||
cont_what="approve $gate_in"
|
||||
elif [ -n "$reject_in" ]; then
|
||||
run_argv=("${ygg_arr[@]}" reject "$cont" "$reject_in" --reason "$reason" --project "$project")
|
||||
cont_what="reject $reject_in"
|
||||
else
|
||||
run_argv=("${ygg_arr[@]}" resume "$cont" --project "$project")
|
||||
cont_what="resume"
|
||||
@@ -319,7 +350,9 @@ build_argv=("${ygg_arr[@]}" sandbox build --all --config "$work/factory.yaml")
|
||||
run_cmd="$(q "${run_argv[@]}")"
|
||||
build_cmd="$(q "${build_argv[@]}")"
|
||||
timeout_argv=(timeout --signal=INT --kill-after=2m "$wall")
|
||||
echo "proof-run: the plan: mode $mode; planner $planner, implementer $implementer, reviewer $reviewer"
|
||||
reasoning_note="" # for an old folder, why the reasoning is unset
|
||||
[ "$reviewer_reasoning" != unset ] || reasoning_note=": a folder from before --reviewer-reasoning"
|
||||
echo "proof-run: the plan: mode $mode; planner $planner, implementer $implementer, reviewer $reviewer (reasoning $reviewer_reasoning${reasoning_note})"
|
||||
echo "proof-run: the caps in $work/factory.yaml:"
|
||||
{ grep -E '^ per_(run|day):' "$work/factory.yaml" || true; } | sed 's/^/ /'
|
||||
[ -z "$cont_what" ] || echo "proof-run: the continuation: $cont_what"
|
||||
@@ -450,7 +483,7 @@ step() { # TITLE CMD...: a heading, then the command's output; its exit code is
|
||||
}
|
||||
evidence_part() {
|
||||
echo "# proof run $name: run $id, exit code $rc${cont_what:+ (continued: $cont_what)}"
|
||||
echo "mode $mode; planner $planner, implementer $implementer, reviewer $reviewer; brief SHA-256 $have"
|
||||
echo "mode $mode; planner $planner, implementer $implementer, reviewer $reviewer (reasoning $reviewer_reasoning); brief SHA-256 $have"
|
||||
echo
|
||||
case "$rc" in
|
||||
0 | 3 | 4 | 5) ;;
|
||||
@@ -501,7 +534,10 @@ PY
|
||||
[[ $gate =~ ^[A-Za-z0-9._-]+$ ]] || gate="<gate>"
|
||||
echo "$by_hand"
|
||||
echo " $(q "$HERE/ygg-engine.sh" run --folder "$dir_abs" -- "${ygg_arr[@]}" approve "$id" "$gate" --project "$project")"
|
||||
echo "proof-run: the run stopped at the approval point $gate; read its gates folder, then continue it (add --note TEXT if you like):"
|
||||
echo " $(qt "$HERE/ygg-engine.sh" run --folder "$dir_abs" -- "${ygg_arr[@]}" reject "$id" "$gate") --reason \"...\" $(q --project "$project")"
|
||||
echo "proof-run: the run stopped at $gate; read its gates folder. If ygg offered a rejection above, reject it with a reason:"
|
||||
echo " $(qt "${self[@]}" --reject "$gate") --reason \"...\" $(q --folder "$dir_abs" --env "$envfile")"
|
||||
echo "proof-run: or approve it (add --note TEXT if you like; some stops take no rejection or no approval):"
|
||||
echo " $(q "${self[@]}" --approve "$gate" --folder "$dir_abs" --env "$envfile")"
|
||||
;;
|
||||
4)
|
||||
|
||||
Reference in new issue
Block a user