mirror of
https://github.com/esengine/DeepSeek-Reasonix.git
synced 2026-11-07 10:45:59 -10:00
zumac
7e92388102
The e2e fixtures finish in 5-15 requests, while a real task in this tree took
deepseek-flash 38. A change aimed at cutting round trips cannot show up on
the short suite, so this adds four long tasks whose workdir is a git archive
of commit 42636ddcb (pricing, ablation, agentgraph and five stdlib-only
leaves) plus a trimmed REASONIX.md, a Makefile, a module README and a small
benchctl consumer:
- rt-cache-write-rate: a CacheWrite rate threaded through cost, fingerprint,
card identity and the catalog, plus an Anthropic row that uses it
- rt-stale-skills: Markdown-only repair of three project skills with no
recognised verifier, the shape behind #13293's lingering stale_verification
- rt-rate-lookup-signatures: five lookups moved to (RateCard, bool),
CatalogKey and ModelRef across about 20 call sites, most in _test.go
- rt-plan-ablation-module: a five-step plan-first task signed off per step
snapshot.sh rebuilds every workdir from the pin and --check detects drift.
One pinned test compares CheckedOn with the wall clock and would fail every
workdir from 2026-11-20; base/ anchors it at the snapshot day instead.
Each task ships a solution/; the corpus is registered with the authoring
guards, which check that every seed fails its grader and every solution
passes. A deepseek-flash baseline (2 trials) solved 8/8 at 13-31 requests
once two grader false negatives in rt-stale-skills were fixed; the README
records it, along with a multi_edit prompt arm and a complete_step
description arm that did not move the request count.
53 lines
1.3 KiB
Bash
Executable File
53 lines
1.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Builds every task's workdir/ from one pinned commit of this repository plus
|
|
# the fixture-authored files in base/ and overlay/<task-id>/. With --check it
|
|
# rebuilds into a scratch directory and fails if any committed workdir differs.
|
|
set -euo pipefail
|
|
|
|
PIN=42636ddcb425b7af07abd88397d02c1cfe3adcf5
|
|
PATHS=(
|
|
internal/contract/pricing
|
|
internal/contract/ablation
|
|
internal/contract/agentgraph
|
|
internal/base/retrieval
|
|
internal/base/rulelist
|
|
internal/base/tokencount
|
|
internal/base/neterr
|
|
internal/base/nilutil
|
|
)
|
|
|
|
here="$(cd "$(dirname "$0")" && pwd)"
|
|
repo="$(git -C "$here" rev-parse --show-toplevel)"
|
|
|
|
build() {
|
|
local id="$1" dest="$2"
|
|
rm -rf "$dest"
|
|
mkdir -p "$dest"
|
|
git -C "$repo" archive "$PIN" -- "${PATHS[@]}" | tar -x -C "$dest"
|
|
cp -R "$here/base/." "$dest/"
|
|
if [ -d "$here/overlay/$id" ]; then
|
|
cp -R "$here/overlay/$id/." "$dest/"
|
|
fi
|
|
}
|
|
|
|
check=0
|
|
[ "${1:-}" = "--check" ] && check=1
|
|
|
|
status=0
|
|
for task in "$here"/tasks/*/; do
|
|
id="$(basename "$task")"
|
|
if [ "$check" = 1 ]; then
|
|
scratch="$(mktemp -d)"
|
|
build "$id" "$scratch/workdir"
|
|
if ! diff -r "$scratch/workdir" "$task/workdir" >/dev/null; then
|
|
echo "$id: workdir/ differs from the pinned snapshot; run $0" >&2
|
|
diff -rq "$scratch/workdir" "$task/workdir" >&2 || true
|
|
status=1
|
|
fi
|
|
rm -rf "$scratch"
|
|
else
|
|
build "$id" "$task/workdir"
|
|
fi
|
|
done
|
|
exit "$status"
|