mirror of
https://github.com/feder-cr/Jobs_Applier_AI_Agent.git
synced 2026-11-07 10:36:05 -10:00
bench: agent benchmarks on real Dots, through Harbor, and a suite of 25 tasks of our own
Harbor (the harness of Terminal-Bench) runs each task in a new Dot: dots_harbor/ is its environment (the Dot is created through the API, the task's Dockerfile is replayed as dot, with apt through dot-install, and the task's root paths are moved under /home/dot/bench) and its agent (the instruction goes to the Dot as a task). Every call goes through bridge.ts, on the end-to-end run's outside driver. Harbor's registry datasets run the same way. tasks/, written by author.py from fixed seeds: shell, files, coding, data analysis, the computer's settings, questions with one answer, two long tasks, and what a Dot is for (its own automation, a careful cleanup, a request it must refuse). In Dots: the oracle scores 25/25 and doing nothing 0/25; the Dot with z-ai/glm-5.3-flash scored 23/25 in 9 minutes, 4 at a time (one miss was the grader's rounding, now fixed).
This commit is contained in:
feder-cr committed
1 parent
564bed5198
commit
c0005a91e3
235 files changed
+4652
-2
No files matched your search
@@ -0,0 +1,14 @@
|
||||
# The benchmark's host (tests/bench/README.md): the end-to-end run's stand-in Linux host
|
||||
# (tests/e2e/linux-host/Dockerfile, built first as idots-linux-host) plus Python and Harbor, which drive
|
||||
# the benchmarks. TEST HARNESS ONLY. Harbor stays out of the host image itself, which holds exactly what
|
||||
# a person installs for the product.
|
||||
FROM idots-linux-host
|
||||
USER root
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends python3 python3-venv git \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
RUN python3 -m venv /opt/harbor \
|
||||
&& /opt/harbor/bin/pip install --no-cache-dir harbor==0.24.0
|
||||
ENV PATH=/opt/harbor/bin:$PATH
|
||||
USER dots
|
||||
WORKDIR /work
|
||||
@@ -0,0 +1,103 @@
|
||||
# Benchmarks on real Dots
|
||||
|
||||
Agent benchmarks run on real Dots: each task gets a new Dot, its own computer
|
||||
is set up the way the task says, the Dot is given the task through the API,
|
||||
and the task's grader checks what is left on that computer. The runs use
|
||||
[Harbor](https://github.com/harbor-framework/harbor) (Apache-2.0), the harness
|
||||
of Terminal-Bench, with the Dot as its environment and its agent, so our own
|
||||
tasks and Harbor's benchmarks run the same way.
|
||||
|
||||
TEST HARNESS ONLY: nothing here is part of the product.
|
||||
|
||||
## How a Dot runs a Harbor task
|
||||
|
||||
| Harbor | Here |
|
||||
|---|---|
|
||||
| environment | `dots_harbor/environment.py`: a new Dot per trial, created through the API with the CPU and memory the task asks for, every permission allowed (nobody is there to answer an approval), deleted after the trial |
|
||||
| the task's `environment/Dockerfile` | replayed in the Dot's computer as its user `dot` (`dots_harbor/dockerfile.py`): `apt-get install` becomes `sudo dot-install`, `pip install` goes to dot's user site, `COPY` uploads, `WORKDIR` and `ENV` are followed |
|
||||
| `/app`, `/tests`, `/solution`, `/logs` | moved under `/home/dot/bench` (`dots_harbor/paths.py`), in commands, uploaded text files and the instruction: the model's user has no root, and the host reaches only dot's home |
|
||||
| agent | `dots_harbor/agent.py`: the instruction is sent to the Dot as a task; the Dot works with its own tools until the task ends; its spend is the trial's cost |
|
||||
| `oracle` and `nop` | Harbor's own: the task's reference solution, or nothing |
|
||||
|
||||
Every call to the product goes through `bridge.ts`, which uses the end-to-end
|
||||
run's outside driver (`tests/e2e/driver.ts`): the HTTP API and dot-agentd on a
|
||||
running computer.
|
||||
|
||||
A task that needs root in its computer cannot run in a Dot. Its oracle fails,
|
||||
which is how such a task is found: **a task counts only once its oracle scores
|
||||
1 and `nop` scores 0 in a Dot.**
|
||||
|
||||
## The tasks
|
||||
|
||||
`tasks/` holds our own suite, 25 tasks written by `author.py` (run it with
|
||||
the directory to write to; the data is generated with a fixed seed and the
|
||||
expected answers are computed there, never by the agent). Each grader is
|
||||
`tests/grade.py`, run by `tests/test.sh`.
|
||||
|
||||
| Category | Tasks |
|
||||
|---|---|
|
||||
| shell | `top-ips`, `large-files`, `fix-script`, `user-service` |
|
||||
| files and documents | `csv-to-json`, `organize-downloads`, `extract-emails`, `invoice-report` |
|
||||
| coding | `fix-pagination`, `wordfreq-cli`, `bash-to-python`, `git-history` |
|
||||
| data analysis | `sales-by-region`, `temperature-stats`, `join-orders` |
|
||||
| the computer's settings | `git-identity`, `ssh-config` |
|
||||
| questions with one answer | `meeting-speaker`, `recipes-question`, `shipment-weights` |
|
||||
| long tasks (45 minutes allowed) | `static-site`, `log-pipeline` |
|
||||
| what a Dot is for | `automation-tick` (its own automation must run twice), `careful-cleanup` (remove the leftovers and nothing else), `impossible-request` (it must say it cannot) |
|
||||
|
||||
The oracle of `automation-tick` cannot use the Dot's automations, which belong
|
||||
to the Dot: it writes the two lines the grader checks itself.
|
||||
|
||||
## Running
|
||||
|
||||
In the bench container: the end-to-end run's Linux host plus Python and
|
||||
Harbor (`Dockerfile`), on the same volumes, so the images the end-to-end run
|
||||
built are used. Build `idots-linux-host` first (`tests/e2e/README.md`), prepare
|
||||
`/work/dots` with `prepare.sh` as for the end-to-end run, then:
|
||||
|
||||
```bash
|
||||
docker build -t idots-bench tests/bench
|
||||
docker run -d --init --name idots-bench \
|
||||
--device /dev/kvm --group-add "$(stat -c %g /dev/kvm)" \
|
||||
-v "$PWD:/src:ro" -v ~/openrouter.key:/run/secrets/openrouter:ro \
|
||||
-v idots-e2e-home:/data -v idots-e2e-work:/work \
|
||||
-e INVISIBLE_DOTS_HOME=/data/home idots-bench sleep infinity
|
||||
|
||||
# The tasks are sound: the oracle scores 1 on each, doing nothing scores 0.
|
||||
docker exec -w /work/dots -e E2E_OPENROUTER_KEY_FILE=/run/secrets/openrouter idots-bench \
|
||||
bash tests/bench/run.sh oracle -p tests/bench/tasks -n 4
|
||||
docker exec -w /work/dots -e E2E_OPENROUTER_KEY_FILE=/run/secrets/openrouter idots-bench \
|
||||
bash tests/bench/run.sh nop -p tests/bench/tasks -n 4
|
||||
|
||||
# The Dot does them; -k 3 runs each three times (pass rate per task).
|
||||
docker exec -w /work/dots -e E2E_OPENROUTER_KEY_FILE=/run/secrets/openrouter \
|
||||
-e BENCH_MODEL=z-ai/glm-5.3-flash idots-bench \
|
||||
bash tests/bench/run.sh dot -p tests/bench/tasks -n 4 -k 3
|
||||
```
|
||||
|
||||
`run.sh` starts the server, stores the key, runs `harbor run` with the Dot
|
||||
environment and stops the server. The arguments after the agent are
|
||||
`harbor run`'s: `-p` a task or a folder of tasks, `-d [email protected]` a
|
||||
dataset of Harbor's registry, `-n` the trials at once, `-k` the attempts per
|
||||
task, `-i`/`-x` to include or exclude tasks. Results go to `BENCH_JOBS_DIR`
|
||||
(default `/work/bench-jobs`): `harbor view` shows them.
|
||||
|
||||
| Variable | Default | |
|
||||
|---|---|---|
|
||||
| `BENCH_MODEL` | `z-ai/glm-5.3-flash` | the Dot's model (OpenRouter id) |
|
||||
| `BENCH_MAX_STEPS` | 150 | the Dot's `limits.max_steps_per_task` |
|
||||
| `BENCH_MAX_COST_USD` | 3 | the Dot's `limits.max_cost_per_task_usd` |
|
||||
|
||||
## External benchmarks
|
||||
|
||||
Harbor's registry datasets run the same way, for instance Terminal-Bench 2.0
|
||||
(89 shell, coding and system tasks):
|
||||
|
||||
```bash
|
||||
docker exec -w /work/dots -e E2E_OPENROUTER_KEY_FILE=/run/secrets/openrouter idots-bench \
|
||||
bash tests/bench/run.sh oracle -d [email protected] -n 4
|
||||
```
|
||||
|
||||
Run the oracle first: the tasks whose oracle fails in a Dot (they need root, a
|
||||
program dot-install cannot give, or a path outside the mapped roots) are left
|
||||
out of the Dot's run with `-x`.
|
||||
@@ -0,0 +1,921 @@
|
||||
"""Writes the in-house benchmark tasks under tests/bench/tasks (Harbor's task format), data included.
|
||||
|
||||
Every task's data is generated here with a fixed seed, and the expected answers are computed here from
|
||||
that data and written into the task's grader, so the graders never trust the agent's own computations.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import gzip
|
||||
import io
|
||||
import json
|
||||
import random
|
||||
import statistics
|
||||
import sys
|
||||
import textwrap
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path, PurePosixPath
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
|
||||
TOML = """schema_version = "1.4"
|
||||
|
||||
[metadata]
|
||||
category = "{category}"
|
||||
difficulty = "{difficulty}"
|
||||
|
||||
[verifier]
|
||||
timeout_sec = 300.0
|
||||
|
||||
[agent]
|
||||
timeout_sec = {agent_timeout}
|
||||
|
||||
[environment]
|
||||
build_timeout_sec = 900.0
|
||||
cpus = {cpus}
|
||||
memory_mb = {memory_mb}
|
||||
"""
|
||||
|
||||
TEST_SH = """#!/bin/bash
|
||||
# The reward is 1 when grade.py passes every check, 0 otherwise; its output says which check failed.
|
||||
mkdir -p /logs/verifier
|
||||
if python3 /tests/grade.py; then
|
||||
echo 1 > /logs/verifier/reward.txt
|
||||
else
|
||||
echo 0 > /logs/verifier/reward.txt
|
||||
fi
|
||||
"""
|
||||
|
||||
GRADE_HEAD = '''"""The checks of this task; any failed assertion gives the reward 0."""
|
||||
import json, os, pathlib, re, subprocess, sys
|
||||
|
||||
def lines(path):
|
||||
return [l.rstrip() for l in pathlib.Path(path).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
|
||||
def check(condition, what):
|
||||
if not condition:
|
||||
print("FAIL:", what)
|
||||
sys.exit(1)
|
||||
print("ok:", what)
|
||||
'''
|
||||
|
||||
|
||||
def task(name, *, category, difficulty, instruction, files=None, setup=(), solve, grade, agent_timeout=900, cpus=1, memory_mb=2048, workdir="/app"):
|
||||
base = ROOT / name
|
||||
(base / "environment").mkdir(parents=True, exist_ok=True)
|
||||
(base / "solution").mkdir(exist_ok=True)
|
||||
(base / "tests").mkdir(exist_ok=True)
|
||||
(base / "task.toml").write_text(TOML.format(category=category, difficulty=difficulty, agent_timeout=float(agent_timeout), cpus=cpus, memory_mb=memory_mb), encoding="utf-8", newline="\n")
|
||||
(base / "instruction.md").write_text(textwrap.dedent(instruction).strip() + "\n", encoding="utf-8", newline="\n")
|
||||
docker = ["FROM ubuntu:24.04", f"WORKDIR {workdir}"]
|
||||
for rel, content in (files or {}).items():
|
||||
target = base / "environment" / "files" / rel
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
if isinstance(content, bytes):
|
||||
target.write_bytes(content)
|
||||
else:
|
||||
target.write_text(content, encoding="utf-8", newline="\n")
|
||||
if files:
|
||||
docker.append(f"COPY files/ {workdir}/")
|
||||
docker += [f"RUN {step}" for step in setup]
|
||||
(base / "environment" / "Dockerfile").write_text("\n".join(docker) + "\n", encoding="utf-8", newline="\n")
|
||||
(base / "solution" / "solve.sh").write_text("#!/bin/bash\nset -euo pipefail\ncd /app\n" + textwrap.dedent(solve).strip() + "\n", encoding="utf-8", newline="\n")
|
||||
(base / "tests" / "test.sh").write_text(TEST_SH, encoding="utf-8", newline="\n")
|
||||
(base / "tests" / "grade.py").write_text(GRADE_HEAD + "\n" + textwrap.dedent(grade).strip() + "\n", encoding="utf-8", newline="\n")
|
||||
|
||||
|
||||
rng = random.Random(20261008)
|
||||
|
||||
# --- shell -------------------------------------------------------------------
|
||||
|
||||
ips = [f"10.0.{rng.randint(0, 9)}.{rng.randint(1, 254)}" for _ in range(25)]
|
||||
weights = [rng.randint(1, 40) for _ in ips]
|
||||
log = []
|
||||
for i in range(600):
|
||||
ip = rng.choices(ips, weights)[0]
|
||||
log.append(f'{ip} - - [{10 + i // 200:02d}/Oct/2026:{rng.randint(0, 23):02d}:{rng.randint(0, 59):02d}:{rng.randint(0, 59):02d} +0000] "GET {rng.choice(["/", "/api/items", "/login", "/static/app.js"])} HTTP/1.1" {rng.choice([200, 200, 304, 404, 500])} {rng.randint(120, 9000)}')
|
||||
top = Counter(line.split()[0] for line in log).most_common()
|
||||
while top[2][1] == top[3][1]:
|
||||
log.append(f'{top[2][0]} - - [12/Oct/2026:23:59:59 +0000] "GET / HTTP/1.1" 200 512')
|
||||
top = Counter(line.split()[0] for line in log).most_common()
|
||||
errors = Counter(line.split()[0] for line in log if line.split()[8] == "500").most_common(1)[0]
|
||||
task(
|
||||
"top-ips", category="shell", difficulty="easy",
|
||||
instruction="""
|
||||
`/app/access.log` is a web server log in the common log format.
|
||||
|
||||
Write the three client IP addresses that made the most requests to `/app/top_ips.txt`, one per line as `<ip> <count>`, the one with the most requests first.
|
||||
""",
|
||||
files={"access.log": "\n".join(log) + "\n"},
|
||||
solve="awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -3 | awk '{print $2\" \"$1}' > top_ips.txt",
|
||||
grade=f"""
|
||||
check(os.path.exists("/app/top_ips.txt"), "top_ips.txt exists")
|
||||
got = [" ".join(l.split()) for l in lines("/app/top_ips.txt")]
|
||||
check(got == {[f"{ip} {n}" for ip, n in top[:3]]!r}, "the three IPs and counts, in order")
|
||||
""",
|
||||
)
|
||||
|
||||
sizes = {}
|
||||
tree = {}
|
||||
for i in range(40):
|
||||
folder = rng.choice(["photos", "photos/2025", "docs", "docs/old", "music", "tmp"])
|
||||
name = f"{folder}/file_{i:02d}.{rng.choice(['jpg', 'pdf', 'mp3', 'bin'])}"
|
||||
size = rng.choice([rng.randint(1_000, 400_000), rng.randint(1_100_000, 5_000_000)])
|
||||
sizes[name] = size
|
||||
large = sorted(((s, n) for n, s in sizes.items() if s > 1_048_576), reverse=True)
|
||||
sizes_setup = " && ".join(f"mkdir -p data/{PurePosixPath(n).parent} && head -c {s} /dev/zero > data/{n}" for n, s in sizes.items())
|
||||
task(
|
||||
"large-files", category="shell", difficulty="easy",
|
||||
instruction="""
|
||||
Find every file under `/app/data` that is larger than 1 MiB (1048576 bytes).
|
||||
|
||||
Write them to `/app/large.txt`, one per line as `<size in bytes> <path relative to /app/data>`, largest first.
|
||||
""",
|
||||
setup=[sizes_setup],
|
||||
solve="cd data && find . -type f -size +1048576c -printf '%s %P\\n' | sort -rn > /app/large.txt",
|
||||
grade=f"""
|
||||
check(os.path.exists("/app/large.txt"), "large.txt exists")
|
||||
got = [" ".join(l.split()) for l in lines("/app/large.txt")]
|
||||
check(got == {[f"{s} {n}" for s, n in large]!r}, "every large file, with its size, largest first")
|
||||
check(len(list(pathlib.Path("/app/data").rglob("*.*"))) == {len(sizes)}, "no file was deleted")
|
||||
""",
|
||||
)
|
||||
|
||||
task(
|
||||
"fix-script", category="shell", difficulty="medium",
|
||||
instruction="""
|
||||
`/app/backup.sh` should copy every `.txt` file of `/app/notes` (subfolders included) into `/app/backup`, keeping the folder structure, and print `backed up N files`. Running `./backup.sh` from `/app` does not work.
|
||||
|
||||
Fix the script so that `cd /app && ./backup.sh` works. Do not copy the files yourself: the script must do it.
|
||||
""",
|
||||
files={
|
||||
"backup.sh": "#/bin/bash\n# Copies the notes into backup.\ncount=0\nfor f in $(find notes -name *.txt); do\n mkdir -p backup/$(dirname $f)\n cp $f backup/$f\n count=$count+1\ndone\necho \"backed up $count files\"\n",
|
||||
"notes/a.txt": "alpha\n",
|
||||
"notes/b.txt": "beta\n",
|
||||
"notes/work/meeting notes.txt": "the room is B\n",
|
||||
"notes/work/todo.txt": "ship it\n",
|
||||
"notes/work/draft.md": "not a txt\n",
|
||||
},
|
||||
solve=r"""
|
||||
cat > backup.sh <<'EOF'
|
||||
#!/bin/bash
|
||||
cd "$(dirname "$0")"
|
||||
count=0
|
||||
while IFS= read -r -d '' f; do
|
||||
rel=${f#notes/}
|
||||
mkdir -p "backup/$(dirname "$rel")"
|
||||
cp "$f" "backup/$rel"
|
||||
count=$((count + 1))
|
||||
done < <(find notes -name '*.txt' -print0)
|
||||
echo "backed up $count files"
|
||||
EOF
|
||||
chmod +x backup.sh
|
||||
""",
|
||||
grade=r"""
|
||||
import shutil
|
||||
shutil.rmtree("/app/backup", ignore_errors=True)
|
||||
out = subprocess.run(["./backup.sh"], cwd="/app", capture_output=True, text=True, timeout=60)
|
||||
check(out.returncode == 0, f"./backup.sh exits 0 (got {out.returncode}: {out.stderr[-300:]})")
|
||||
check(out.stdout.strip() == "backed up 4 files", f"it says it backed up 4 files (said {out.stdout.strip()!r})")
|
||||
names = sorted(str(p.relative_to("/app/backup")) for p in pathlib.Path("/app/backup").rglob("*") if p.is_file())
|
||||
expected = [["a.txt", "b.txt", "work/meeting notes.txt", "work/todo.txt"], ["notes/a.txt", "notes/b.txt", "notes/work/meeting notes.txt", "notes/work/todo.txt"]]
|
||||
check(names in expected, f"the four .txt files, structure kept (got {names})")
|
||||
check(not any(n.endswith(".md") for n in names), "the .md file is not copied")
|
||||
""",
|
||||
)
|
||||
|
||||
task(
|
||||
"user-service", category="shell", difficulty="medium",
|
||||
instruction="""
|
||||
Write a small HTTP server `/app/server.py` (Python standard library only) that answers `GET /health` with status 200 and the JSON body `{"status": "ok"}`, and any other path with 404.
|
||||
|
||||
Then start it so that it keeps running in the background after you finish, listening on 127.0.0.1 port 8765, and write its process id to `/app/server.pid`.
|
||||
""",
|
||||
solve=r"""
|
||||
cat > server.py <<'EOF'
|
||||
import json
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
if self.path == "/health":
|
||||
body = json.dumps({"status": "ok"}).encode()
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "application/json")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
else:
|
||||
self.send_response(404)
|
||||
self.end_headers()
|
||||
|
||||
HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()
|
||||
EOF
|
||||
nohup setsid python3 server.py > server.log 2>&1 < /dev/null &
|
||||
echo $! > server.pid
|
||||
sleep 1
|
||||
""",
|
||||
grade=r"""
|
||||
import urllib.request, urllib.error
|
||||
pid = int(pathlib.Path("/app/server.pid").read_text().strip())
|
||||
check(os.path.exists(f"/proc/{pid}"), f"the process {pid} of server.pid runs")
|
||||
body = urllib.request.urlopen("http://127.0.0.1:8765/health", timeout=10).read()
|
||||
check(json.loads(body) == {"status": "ok"}, "/health answers {'status': 'ok'}")
|
||||
try:
|
||||
urllib.request.urlopen("http://127.0.0.1:8765/nope", timeout=10)
|
||||
check(False, "/nope answers 404")
|
||||
except urllib.error.HTTPError as e:
|
||||
check(e.code == 404, "/nope answers 404")
|
||||
""",
|
||||
)
|
||||
|
||||
# --- files and documents ----------------------------------------------------
|
||||
|
||||
people = []
|
||||
first = ["Ada", "Bruno", "Chiara", "Dmitri", "Elena", "Farah", "Giulio", "Hana", "Ivo", "Jun", "Kofi", "Lea"]
|
||||
for i, name in enumerate(first):
|
||||
people.append({"id": i + 1, "name": name, "age": rng.randint(19, 77), "city": rng.choice(["Milano", "Roma", "Torino", "Napoli"]), "member": rng.choice([True, False])})
|
||||
buf = io.StringIO()
|
||||
writer = csv.writer(buf, lineterminator="\n")
|
||||
writer.writerow(["id", "name", "age", "city", "member"])
|
||||
for p in people:
|
||||
writer.writerow([p["id"], p["name"], p["age"], p["city"], "yes" if p["member"] else "no"])
|
||||
task(
|
||||
"csv-to-json", category="files", difficulty="easy",
|
||||
instruction="""
|
||||
Convert `/app/people.csv` into `/app/people.json`: a JSON array with one object per row, in the same order, with the keys `id`, `name`, `age`, `city` and `member`.
|
||||
|
||||
`id` and `age` must be numbers, and `member` a boolean (`yes` is true, `no` is false).
|
||||
""",
|
||||
files={"people.csv": buf.getvalue()},
|
||||
solve="""python3 - <<'EOF'
|
||||
import csv, json
|
||||
rows = [dict(r, id=int(r["id"]), age=int(r["age"]), member=r["member"] == "yes") for r in csv.DictReader(open("people.csv"))]
|
||||
json.dump(rows, open("people.json", "w"), indent=2)
|
||||
EOF""",
|
||||
grade=f"""
|
||||
data = json.loads(pathlib.Path("/app/people.json").read_text())
|
||||
check(data == {people!r}, "every row, typed, in order")
|
||||
""",
|
||||
)
|
||||
|
||||
downloads = {}
|
||||
kinds = {"pdf": "documents", "docx": "documents", "txt": "documents", "jpg": "images", "png": "images", "mp3": "audio", "zip": "archives"}
|
||||
for i in range(24):
|
||||
ext = rng.choice(list(kinds))
|
||||
downloads[f"Downloads/{rng.choice(['invoice', 'photo', 'song', 'notes', 'backup'])}_{i:02d}.{ext}"] = f"content {i}\n"
|
||||
downloads["Downloads/README"] = "no extension\n"
|
||||
expected_layout = sorted(f"{kinds[Path(n).suffix[1:]]}/{Path(n).name}" for n in downloads if Path(n).suffix) + ["other/README"]
|
||||
task(
|
||||
"organize-downloads", category="files", difficulty="easy",
|
||||
instruction="""
|
||||
Tidy `/app/Downloads`: move every file into a subfolder of `/app/Downloads` by type: `documents` (pdf, docx, txt), `images` (jpg, png), `audio` (mp3), `archives` (zip), and `other` for anything else.
|
||||
|
||||
Keep the file names, and leave no file directly in `/app/Downloads`.
|
||||
""",
|
||||
files=downloads,
|
||||
solve=r"""
|
||||
cd Downloads
|
||||
for f in *; do
|
||||
[ -f "$f" ] || continue
|
||||
case "${f##*.}" in
|
||||
pdf|docx|txt) d=documents ;; jpg|png) d=images ;; mp3) d=audio ;; zip) d=archives ;; *) d=other ;;
|
||||
esac
|
||||
[ "$f" = "${f##*.}" ] && d=other
|
||||
mkdir -p "$d" && mv "$f" "$d/"
|
||||
done
|
||||
""",
|
||||
grade=f"""
|
||||
root = pathlib.Path("/app/Downloads")
|
||||
got = sorted(str(p.relative_to(root)) for p in root.rglob("*") if p.is_file())
|
||||
check(got == {sorted(expected_layout)!r}, "every file in the folder of its type, names kept")
|
||||
""",
|
||||
)
|
||||
|
||||
emails = set()
|
||||
docs = {}
|
||||
for i in range(8):
|
||||
body = []
|
||||
for _ in range(rng.randint(3, 8)):
|
||||
user = rng.choice(["anna", "marco", "support", "team.lead", "j_smith", "info"])
|
||||
domain = rng.choice(["example.com", "Example.COM", "acme.io", "mail.co.uk"])
|
||||
address = f"{user}@{domain}"
|
||||
emails.add(address.lower())
|
||||
body.append(rng.choice([f"Write to {address} for details.", f"cc: <{address}>", f"({address}), thanks", f"Contact: {address}."]))
|
||||
docs[f"inbox/msg_{i}.{'md' if i % 2 else 'txt'}"] = "\n".join(body) + "\nnot-an-email@ and @nobody here\n"
|
||||
task(
|
||||
"extract-emails", category="files", difficulty="medium",
|
||||
instruction="""
|
||||
The files in `/app/inbox` mention email addresses. Write every distinct address to `/app/emails.txt`, one per line, in lower case, sorted alphabetically.
|
||||
|
||||
Addresses differing only in letter case are the same address.
|
||||
""",
|
||||
files=docs,
|
||||
solve="grep -rhoE '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}' inbox | tr 'A-Z' 'a-z' | sed 's/\\.$//' | sort -u > emails.txt",
|
||||
grade=f"""
|
||||
check(lines("/app/emails.txt") == {sorted(emails)!r}, "every distinct address, lower case, sorted")
|
||||
""",
|
||||
)
|
||||
|
||||
invoices = {}
|
||||
totals = defaultdict(float)
|
||||
for i in range(9):
|
||||
client = rng.choice(["Rossi Srl", "Bianchi SpA", "Verdi & Co"])
|
||||
items = [{"description": rng.choice(["design", "hosting", "support", "license"]), "quantity": rng.randint(1, 5), "unit_price": round(rng.uniform(20, 400), 2)} for _ in range(rng.randint(1, 4))]
|
||||
invoices[f"invoices/inv_{i:03d}.json"] = json.dumps({"number": f"2026-{i:03d}", "client": client, "items": items}, indent=2)
|
||||
totals[client] += sum(it["quantity"] * it["unit_price"] for it in items)
|
||||
report_rows = sorted(((c, round(t, 2)) for c, t in totals.items()), key=lambda r: -r[1])
|
||||
task(
|
||||
"invoice-report", category="files", difficulty="medium",
|
||||
instruction="""
|
||||
`/app/invoices` holds invoices as JSON. Write `/app/report.md`: a Markdown table with the columns `Client` and `Total`, one row per client with the sum of `quantity * unit_price` over all of that client's invoices, rounded to 2 decimals, the largest total first, then a last row `TOTAL` with the sum of all.
|
||||
""",
|
||||
files=invoices,
|
||||
solve="""python3 - <<'EOF'
|
||||
import json, glob, collections
|
||||
t = collections.defaultdict(float)
|
||||
for f in glob.glob("invoices/*.json"):
|
||||
inv = json.load(open(f))
|
||||
t[inv["client"]] += sum(i["quantity"] * i["unit_price"] for i in inv["items"])
|
||||
rows = sorted(t.items(), key=lambda r: -r[1])
|
||||
with open("report.md", "w") as out:
|
||||
out.write("| Client | Total |\\n|---|---|\\n")
|
||||
for c, v in rows:
|
||||
out.write(f"| {c} | {v:.2f} |\\n")
|
||||
out.write(f"| TOTAL | {sum(t.values()):.2f} |\\n")
|
||||
EOF""",
|
||||
grade=f"""
|
||||
rows = []
|
||||
for l in lines("/app/report.md"):
|
||||
cells = [c.strip().strip("*") for c in l.strip().strip("|").split("|")]
|
||||
if len(cells) == 2 and not set(cells[0]) <= set("-: "):
|
||||
rows.append(cells)
|
||||
check(rows[0][0].lower() == "client" and rows[0][1].lower() == "total", "the header is Client | Total")
|
||||
body = [(c, round(float(v.replace(",", "")), 2)) for c, v in rows[1:]]
|
||||
expected = {report_rows!r} + [("TOTAL", {round(sum(totals.values()), 2)!r})]
|
||||
check(len(body) == len(expected), f"{{len(expected)}} rows")
|
||||
for (c, v), (ec, ev) in zip(body, expected):
|
||||
check(c == ec and abs(v - ev) < 0.011, f"{{ec}} totals {{ev}} (got {{c}} {{v}})")
|
||||
""",
|
||||
)
|
||||
|
||||
# --- coding -------------------------------------------------------------------
|
||||
|
||||
task(
|
||||
"fix-pagination", category="coding", difficulty="medium",
|
||||
instruction="""
|
||||
`/app/paginate.py` has a function `paginate(items, page, per_page)` that should return the items of page `page` (pages start at 1) and the total number of pages. Users report that the last page is sometimes missing and that page 1 skips the first item.
|
||||
|
||||
Fix the function. Keep its name and signature: other code imports it. A page past the end gives an empty list; an empty list of items has 0 pages; `page` < 1 or `per_page` < 1 raises `ValueError`.
|
||||
""",
|
||||
files={"paginate.py": "def paginate(items, page, per_page):\n \"\"\"Return (the items of page `page`, starting at 1, the number of pages).\"\"\"\n pages = len(items) // per_page\n start = page * per_page\n return items[start + 1:start + per_page], pages\n"},
|
||||
solve="""cat > paginate.py <<'EOF'
|
||||
def paginate(items, page, per_page):
|
||||
\"\"\"Return (the items of page `page`, starting at 1, the number of pages).\"\"\"
|
||||
if page < 1 or per_page < 1:
|
||||
raise ValueError("page and per_page start at 1")
|
||||
pages = -(-len(items) // per_page)
|
||||
start = (page - 1) * per_page
|
||||
return items[start:start + per_page], pages
|
||||
EOF""",
|
||||
grade="""
|
||||
sys.path.insert(0, "/app")
|
||||
from paginate import paginate
|
||||
items = list(range(1, 24))
|
||||
check(paginate(items, 1, 5) == ([1, 2, 3, 4, 5], 5), "page 1 of 23 items by 5")
|
||||
check(paginate(items, 5, 5) == ([21, 22, 23], 5), "the last, short page")
|
||||
check(paginate(items, 6, 5) == ([], 5), "a page past the end is empty")
|
||||
check(paginate(list(range(10)), 2, 5) == ([5, 6, 7, 8, 9], 2), "an exact last page")
|
||||
check(paginate([], 1, 3) == ([], 0), "no items, no pages")
|
||||
for bad in ((0, 5), (1, 0)):
|
||||
try:
|
||||
paginate(items, *bad)
|
||||
check(False, f"{bad} raises ValueError")
|
||||
except ValueError:
|
||||
check(True, f"{bad} raises ValueError")
|
||||
""",
|
||||
)
|
||||
|
||||
corpus = " ".join(rng.choice(["the", "dot", "agent", "runs", "a", "computer", "task", "files", "The", "Dot,", "agent.", "fast"]) for _ in range(700))
|
||||
task(
|
||||
"wordfreq-cli", category="coding", difficulty="medium",
|
||||
instruction="""
|
||||
Write a command-line program `/app/wordfreq` (any language available on this computer; make it executable) such that `wordfreq FILE -n N` prints the N most frequent words of FILE, one per line as `<word> <count>`, most frequent first, ties in alphabetical order.
|
||||
|
||||
Words are compared in lower case and are made of letters only (`Dot,` and `dot` are the same word). `-n` defaults to 10.
|
||||
""",
|
||||
files={"sample.txt": corpus},
|
||||
solve="""cat > wordfreq <<'EOF'
|
||||
#!/usr/bin/env python3
|
||||
import argparse, collections, re
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("file")
|
||||
p.add_argument("-n", type=int, default=10)
|
||||
a = p.parse_args()
|
||||
words = re.findall(r"[a-z]+", open(a.file, encoding="utf-8").read().lower())
|
||||
for w, c in sorted(collections.Counter(words).items(), key=lambda x: (-x[1], x[0]))[:a.n]:
|
||||
print(w, c)
|
||||
EOF
|
||||
chmod +x wordfreq""",
|
||||
grade="""
|
||||
import collections, tempfile
|
||||
def expected(text, n):
|
||||
words = re.findall(r"[a-z]+", text.lower())
|
||||
return [f"{w} {c}" for w, c in sorted(collections.Counter(words).items(), key=lambda x: (-x[1], x[0]))[:n]]
|
||||
texts = ["Apple banana apple, Cherry! banana apple? cherry date.", "z y x z y z w w w w", pathlib.Path("/app/sample.txt").read_text()]
|
||||
for i, text in enumerate(texts):
|
||||
f = tempfile.NamedTemporaryFile("w", delete=False, suffix=".txt"); f.write(text); f.close()
|
||||
for n in (3, None):
|
||||
args = ["/app/wordfreq", f.name] + (["-n", str(n)] if n else [])
|
||||
out = subprocess.run(args, capture_output=True, text=True, timeout=30)
|
||||
got = [" ".join(l.split()) for l in out.stdout.splitlines() if l.strip()]
|
||||
check(got == expected(text, n or 10), f"text {i}, -n {n}: {got[:4]}")
|
||||
""",
|
||||
)
|
||||
|
||||
task(
|
||||
"bash-to-python", category="coding", difficulty="medium",
|
||||
instruction="""
|
||||
`/app/summary.sh` prints a summary of a CSV file. Rewrite it in Python as `/app/summary.py` with exactly the same output for any CSV file of the same columns: `python3 /app/summary.py FILE` must print what `bash /app/summary.sh FILE` prints. Keep `summary.sh`.
|
||||
""",
|
||||
files={
|
||||
"summary.sh": "#!/bin/bash\n# Prints, for a CSV of date,category,amount (with a header), the number of rows, the total amount, and the total per category sorted by name.\nfile=$1\nrows=$(tail -n +2 \"$file\" | grep -c .)\necho \"rows: $rows\"\ntail -n +2 \"$file\" | awk -F, '{t += $3} END {printf \"total: %.2f\\n\", t}'\ntail -n +2 \"$file\" | awk -F, '{s[$2] += $3} END {for (c in s) printf \"%s: %.2f\\n\", c, s[c]}' | sort\n",
|
||||
"expenses.csv": "date,category,amount\n" + "".join(f"2026-10-{d:02d},{rng.choice(['food', 'rent', 'travel', 'books'])},{rng.uniform(1, 300):.2f}\n" for d in range(1, 29)),
|
||||
},
|
||||
solve="""cat > summary.py <<'EOF'
|
||||
import csv, sys, collections
|
||||
rows = list(csv.reader(open(sys.argv[1])))[1:]
|
||||
rows = [r for r in rows if r]
|
||||
print(f"rows: {len(rows)}")
|
||||
print(f"total: {sum(float(r[2]) for r in rows):.2f}")
|
||||
s = collections.defaultdict(float)
|
||||
for r in rows:
|
||||
s[r[1]] += float(r[2])
|
||||
for c in sorted(s):
|
||||
print(f"{c}: {s[c]:.2f}")
|
||||
EOF""",
|
||||
grade="""
|
||||
import random, tempfile
|
||||
r = random.Random(5)
|
||||
files = ["/app/expenses.csv"]
|
||||
for k in range(2):
|
||||
f = tempfile.NamedTemporaryFile("w", delete=False, suffix=".csv")
|
||||
f.write("date,category,amount\\n" + "".join(f"2026-11-{d:02d},{r.choice(['zeta', 'alpha', 'mid'])},{r.uniform(0, 999):.2f}\\n" for d in range(1, 15 + k * 10)))
|
||||
f.close(); files.append(f.name)
|
||||
check(os.path.exists("/app/summary.sh"), "summary.sh is kept")
|
||||
for f in files:
|
||||
want = subprocess.run(["bash", "/app/summary.sh", f], capture_output=True, text=True).stdout
|
||||
got = subprocess.run(["python3", "/app/summary.py", f], capture_output=True, text=True, timeout=30).stdout
|
||||
check(got == want, f"same output for {os.path.basename(f)}")
|
||||
""",
|
||||
)
|
||||
|
||||
task(
|
||||
"git-history", category="coding", difficulty="medium",
|
||||
instruction="""
|
||||
`/app/project` is a git repository. One of its commits introduced the bug that makes `python3 calc.py` print `5` instead of `6`.
|
||||
|
||||
Write the full hash of that commit to `/app/bad_commit.txt`, then fix the bug with a new commit on top (do not rewrite the history), so that `python3 calc.py` prints `6`.
|
||||
""",
|
||||
setup=[
|
||||
"mkdir -p project && cd project && git init -q -b main && git config user.email [email protected] && git config user.name Dev"
|
||||
" && printf 'def add(a, b):\\n return a + b\\n\\nprint(add(2, 4))\\n' > calc.py && git add . && git commit -qm 'add calc'"
|
||||
" && printf 'notes\\n' > NOTES.md && git add . && git commit -qm 'notes'"
|
||||
" && printf 'def add(a, b):\\n return a + b - 1\\n\\nprint(add(2, 4))\\n' > calc.py && git commit -qam 'tidy add'"
|
||||
" && printf 'more notes\\n' >> NOTES.md && git commit -qam 'more notes'"
|
||||
" && git rev-parse HEAD~1 > /app/.bad && git rev-parse HEAD > /app/.head"
|
||||
],
|
||||
solve="""cd project
|
||||
git rev-parse HEAD~1 > /app/bad_commit.txt
|
||||
sed -i 's/return a + b - 1/return a + b/' calc.py
|
||||
git -c [email protected] -c user.name=Dot commit -qam 'fix add'""",
|
||||
grade="""
|
||||
check(pathlib.Path("/app/bad_commit.txt").read_text().strip() == pathlib.Path("/app/.bad").read_text().strip(), "the commit that introduced the bug")
|
||||
out = subprocess.run(["python3", "calc.py"], cwd="/app/project", capture_output=True, text=True).stdout.strip()
|
||||
check(out == "6", f"calc.py prints 6 (prints {out!r})")
|
||||
head = pathlib.Path("/app/.head").read_text().strip()
|
||||
anc = subprocess.run(["git", "merge-base", "--is-ancestor", head, "HEAD"], cwd="/app/project").returncode
|
||||
check(anc == 0, "the old history is kept, the fix is on top")
|
||||
dirty = subprocess.run(["git", "status", "--porcelain", "--", "calc.py"], cwd="/app/project", capture_output=True, text=True).stdout.strip()
|
||||
check(dirty == "", "the fix is committed")
|
||||
""",
|
||||
)
|
||||
|
||||
# --- data analysis ------------------------------------------------------------
|
||||
|
||||
sales = []
|
||||
for i in range(300):
|
||||
sales.append((f"2026-{rng.randint(1, 9):02d}-{rng.randint(1, 28):02d}", rng.choice(["North", "South", "East", "West"]), rng.choice(["A", "B", "C"]), rng.randint(1, 20), round(rng.uniform(2, 80), 2)))
|
||||
by_region = defaultdict(float)
|
||||
for _, region, _, qty, price in sales:
|
||||
by_region[region] += qty * price
|
||||
best = max(by_region.items(), key=lambda kv: kv[1])
|
||||
task(
|
||||
"sales-by-region", category="data", difficulty="easy",
|
||||
instruction="""
|
||||
`/app/sales.csv` lists sales (`date,region,product,quantity,unit_price`). Which region had the highest revenue (quantity times unit price) over the whole file?
|
||||
|
||||
Write the region and its revenue rounded to 2 decimals to `/app/answer.txt` as `<region> <revenue>`.
|
||||
""",
|
||||
files={"sales.csv": "date,region,product,quantity,unit_price\n" + "".join(f"{d},{r},{p},{q},{u}\n" for d, r, p, q, u in sales)},
|
||||
solve="""python3 - <<'EOF'
|
||||
import csv, collections
|
||||
t = collections.defaultdict(float)
|
||||
for r in csv.DictReader(open("sales.csv")):
|
||||
t[r["region"]] += int(r["quantity"]) * float(r["unit_price"])
|
||||
k = max(t, key=t.get)
|
||||
open("answer.txt", "w").write(f"{k} {t[k]:.2f}\\n")
|
||||
EOF""",
|
||||
grade=f"""
|
||||
region, value = pathlib.Path("/app/answer.txt").read_text().split()[:2]
|
||||
check(region == {best[0]!r}, "the region")
|
||||
check(abs(float(value) - {round(best[1], 2)!r}) < 0.011, "its revenue")
|
||||
""",
|
||||
)
|
||||
|
||||
temps = [round(rng.gauss(18, 6), 1) for _ in range(365)]
|
||||
temps[40] = None
|
||||
temps[200] = None
|
||||
valid = [t for t in temps if t is not None]
|
||||
# Exact values: the grader accepts either rounding of a value that ends in 5.
|
||||
stats = (statistics.median(valid), statistics.pstdev(valid), sum(1 for t in valid if t > 25))
|
||||
task(
|
||||
"temperature-stats", category="data", difficulty="medium",
|
||||
instruction="""
|
||||
`/app/temps.csv` has one temperature reading per day of 2025 (`day,celsius`); a missing reading is an empty cell.
|
||||
|
||||
Ignoring the missing readings, write to `/app/stats.json` an object with `median` and `stdev` (the population standard deviation), both rounded to 2 decimals, and `hot_days`, the number of days above 25 °C.
|
||||
""",
|
||||
files={"temps.csv": "day,celsius\n" + "".join(f"{i + 1},{'' if t is None else t}\n" for i, t in enumerate(temps))},
|
||||
solve="""python3 - <<'EOF'
|
||||
import csv, json, statistics
|
||||
v = [float(r["celsius"]) for r in csv.DictReader(open("temps.csv")) if r["celsius"]]
|
||||
json.dump({"median": round(statistics.median(v), 2), "stdev": round(statistics.pstdev(v), 2), "hot_days": sum(t > 25 for t in v)}, open("stats.json", "w"))
|
||||
EOF""",
|
||||
grade=f"""
|
||||
s = json.loads(pathlib.Path("/app/stats.json").read_text())
|
||||
check(abs(float(s["median"]) - {stats[0]!r}) <= 0.005 + 1e-9, "median")
|
||||
check(abs(float(s["stdev"]) - {stats[1]!r}) <= 0.005 + 1e-9, "population standard deviation")
|
||||
check(int(s["hot_days"]) == {stats[2]!r}, "days above 25")
|
||||
""",
|
||||
)
|
||||
|
||||
customers = {i: rng.choice(["IT", "FR", "DE", "ES"]) for i in range(1, 41)}
|
||||
orders = [(n, rng.randint(1, 45), round(rng.uniform(5, 500), 2)) for n in range(1, 151)]
|
||||
per_country = defaultdict(lambda: [0, 0.0])
|
||||
unknown = 0
|
||||
for _, c, amount in orders:
|
||||
if c in customers:
|
||||
per_country[customers[c]][0] += 1
|
||||
per_country[customers[c]][1] += amount
|
||||
else:
|
||||
unknown += 1
|
||||
country_rows = sorted(per_country.items())
|
||||
task(
|
||||
"join-orders", category="data", difficulty="medium",
|
||||
instruction="""
|
||||
`/app/customers.csv` (`customer_id,country`) and `/app/orders.csv` (`order_id,customer_id,amount`) come from two systems. Some orders name a customer the customers file does not have.
|
||||
|
||||
Write `/app/by_country.csv` with the header `country,orders,amount` and one row per country (sorted by country code): the number of orders and their total amount rounded to 2 decimals. Then write to `/app/unmatched.txt` the number of orders whose customer is unknown.
|
||||
""",
|
||||
files={
|
||||
"customers.csv": "customer_id,country\n" + "".join(f"{i},{c}\n" for i, c in customers.items()),
|
||||
"orders.csv": "order_id,customer_id,amount\n" + "".join(f"{n},{c},{a}\n" for n, c, a in orders),
|
||||
},
|
||||
solve="""python3 - <<'EOF'
|
||||
import csv, collections
|
||||
cust = {r["customer_id"]: r["country"] for r in csv.DictReader(open("customers.csv"))}
|
||||
agg = collections.defaultdict(lambda: [0, 0.0]); un = 0
|
||||
for r in csv.DictReader(open("orders.csv")):
|
||||
c = cust.get(r["customer_id"])
|
||||
if c is None: un += 1; continue
|
||||
agg[c][0] += 1; agg[c][1] += float(r["amount"])
|
||||
with open("by_country.csv", "w") as f:
|
||||
f.write("country,orders,amount\\n")
|
||||
for c in sorted(agg): f.write(f"{c},{agg[c][0]},{agg[c][1]:.2f}\\n")
|
||||
open("unmatched.txt", "w").write(f"{un}\\n")
|
||||
EOF""",
|
||||
grade=f"""
|
||||
import csv
|
||||
rows = list(csv.DictReader(open("/app/by_country.csv")))
|
||||
expected = {[(k, v[0], round(v[1], 2)) for k, v in country_rows]!r}
|
||||
check([r["country"] for r in rows] == [e[0] for e in expected], "one row per country, sorted")
|
||||
for r, (c, n, a) in zip(rows, expected):
|
||||
check(int(r["orders"]) == n and abs(float(r["amount"]) - a) < 0.011, f"{{c}}: {{n}} orders, {{a}}")
|
||||
check(int(pathlib.Path("/app/unmatched.txt").read_text().strip()) == {unknown}, "the orders with an unknown customer")
|
||||
""",
|
||||
)
|
||||
|
||||
# --- the computer's own settings ------------------------------------------------
|
||||
|
||||
task(
|
||||
"git-identity", category="computer", difficulty="easy",
|
||||
instruction="""
|
||||
Set up git for your user on this computer: the name `Dot Bench`, the email `[email protected]`, `main` as the default branch of new repositories, and an alias `git last` that shows the last commit (`log -1 HEAD`).
|
||||
""",
|
||||
solve="""git config --global user.name 'Dot Bench'
|
||||
git config --global user.email [email protected]
|
||||
git config --global init.defaultBranch main
|
||||
git config --global alias.last 'log -1 HEAD'""",
|
||||
grade="""
|
||||
def get(key):
|
||||
return subprocess.run(["git", "config", "--global", "--get", key], capture_output=True, text=True).stdout.strip()
|
||||
check(get("user.name") == "Dot Bench", "user.name")
|
||||
check(get("user.email") == "[email protected]", "user.email")
|
||||
check(get("init.defaultBranch") == "main", "init.defaultBranch")
|
||||
check(get("alias.last").replace("git ", "") == "log -1 HEAD", "alias.last")
|
||||
""",
|
||||
)
|
||||
|
||||
task(
|
||||
"ssh-config", category="computer", difficulty="easy",
|
||||
instruction="""
|
||||
Make `ssh buildbox` connect to the host `10.20.30.40` on port `2222` as the user `deploy`, with the key `~/.ssh/buildbox_ed25519`, which you create (an ed25519 key with no passphrase). Do not connect: the host does not exist yet.
|
||||
|
||||
Also write the public key to `/app/buildbox.pub`.
|
||||
""",
|
||||
solve="""mkdir -p ~/.ssh && chmod 700 ~/.ssh
|
||||
ssh-keygen -q -t ed25519 -N '' -f ~/.ssh/buildbox_ed25519
|
||||
cat >> ~/.ssh/config <<'EOF'
|
||||
Host buildbox
|
||||
HostName 10.20.30.40
|
||||
Port 2222
|
||||
User deploy
|
||||
IdentityFile ~/.ssh/buildbox_ed25519
|
||||
EOF
|
||||
chmod 600 ~/.ssh/config
|
||||
cp ~/.ssh/buildbox_ed25519.pub /app/buildbox.pub""",
|
||||
grade="""
|
||||
out = subprocess.run(["ssh", "-G", "buildbox"], capture_output=True, text=True).stdout.lower().splitlines()
|
||||
conf = dict(l.split(" ", 1) for l in out if " " in l)
|
||||
check(conf.get("hostname") == "10.20.30.40", "HostName")
|
||||
check(conf.get("port") == "2222", "Port")
|
||||
check(conf.get("user") == "deploy", "User")
|
||||
key = os.path.expanduser("~/.ssh/buildbox_ed25519")
|
||||
check(any(os.path.expanduser(v) == key for k, v in (l.split(" ", 1) for l in out if l.startswith("identityfile "))), "IdentityFile")
|
||||
check(os.path.exists(key) and "OPENSSH PRIVATE KEY" in open(key).read(), "the private key exists")
|
||||
pub = pathlib.Path("/app/buildbox.pub").read_text().split()
|
||||
check(pub[0] == "ssh-ed25519" and pub[1] == open(key + ".pub").read().split()[1], "buildbox.pub is the key's public half")
|
||||
check(subprocess.run(["ssh-keygen", "-y", "-P", "", "-f", key], capture_output=True).returncode == 0, "no passphrase")
|
||||
""",
|
||||
)
|
||||
|
||||
# --- questions with an exact answer (GAIA-like, offline) -------------------------
|
||||
|
||||
speakers = ["Alice", "Bob", "Carla", "Dev"]
|
||||
transcript = []
|
||||
spoken = Counter()
|
||||
for _ in range(60):
|
||||
who = rng.choice(speakers)
|
||||
words = rng.randint(3, 30)
|
||||
transcript.append(f"[{rng.randint(0, 59):02d}:{rng.randint(0, 59):02d}] {who}: " + " ".join(rng.choice(["we", "should", "ship", "the", "release", "on", "friday", "maybe", "not", "yes"]) for _ in range(words)))
|
||||
spoken[who] += words
|
||||
assert spoken.most_common(2)[0][1] != spoken.most_common(2)[1][1], spoken
|
||||
most = spoken.most_common(1)[0][0]
|
||||
task(
|
||||
"meeting-speaker", category="question", difficulty="easy",
|
||||
instruction="""
|
||||
`/app/meeting.txt` is the transcript of a meeting. Who said the most words in total? Count the words after `Name:` on each line.
|
||||
|
||||
Write only the name to `/app/answer.txt`.
|
||||
""",
|
||||
files={"meeting.txt": "\n".join(transcript) + "\n"},
|
||||
solve="sed -E 's/^\\[[0-9:]+\\] //' meeting.txt | awk -F': ' '{n[$1] += split($2, w, \" \")} END {for (k in n) print n[k], k}' | sort -rn | head -1 | awk '{print $2}' > answer.txt",
|
||||
grade=f"""
|
||||
check(pathlib.Path("/app/answer.txt").read_text().strip().strip(".").lower() == {most.lower()!r}, "the speaker")
|
||||
""",
|
||||
)
|
||||
|
||||
recipes = []
|
||||
for i in range(30):
|
||||
recipes.append({"name": f"Recipe {i + 1}", "vegetarian": rng.choice([True, False]), "minutes": rng.choice([10, 15, 20, 25, 30, 35, 45, 60]), "servings": rng.randint(1, 6)})
|
||||
quick_veg = sum(1 for r in recipes if r["vegetarian"] and r["minutes"] < 30)
|
||||
task(
|
||||
"recipes-question", category="question", difficulty="easy",
|
||||
instruction="""
|
||||
Using `/app/recipes.json`: how many vegetarian recipes take strictly less than 30 minutes?
|
||||
|
||||
Write only the number to `/app/answer.txt`.
|
||||
""",
|
||||
files={"recipes.json": json.dumps(recipes, indent=1)},
|
||||
solve="python3 -c \"import json; print(sum(r['vegetarian'] and r['minutes'] < 30 for r in json.load(open('recipes.json'))))\" > answer.txt",
|
||||
grade=f"""
|
||||
check(re.sub(r"[^0-9]", "", pathlib.Path("/app/answer.txt").read_text()) == "{quick_veg}", "the count")
|
||||
""",
|
||||
)
|
||||
|
||||
shipments = [(f"SKU-{i:03d}", rng.randint(1, 40), round(rng.uniform(0.5, 30), 1)) for i in range(25)]
|
||||
heaviest = max(shipments, key=lambda s: s[1] * s[2])
|
||||
total_kg = round(sum(q * w for _, q, w in shipments) * 0.45359237, 1)
|
||||
task(
|
||||
"shipment-weights", category="question", difficulty="medium",
|
||||
instruction="""
|
||||
`/app/shipment.csv` lists a shipment's lines (`sku,quantity,unit_weight_lb`, weights in pounds).
|
||||
|
||||
Write to `/app/answer.txt` two lines: first the SKU whose line weighs the most in total, then the weight of the whole shipment in kilograms, rounded to one decimal (1 lb = 0.45359237 kg).
|
||||
""",
|
||||
files={"shipment.csv": "sku,quantity,unit_weight_lb\n" + "".join(f"{s},{q},{w}\n" for s, q, w in shipments)},
|
||||
solve="""python3 - <<'EOF'
|
||||
import csv
|
||||
rows = list(csv.DictReader(open("shipment.csv")))
|
||||
top = max(rows, key=lambda r: int(r["quantity"]) * float(r["unit_weight_lb"]))["sku"]
|
||||
kg = sum(int(r["quantity"]) * float(r["unit_weight_lb"]) for r in rows) * 0.45359237
|
||||
open("answer.txt", "w").write(f"{top}\\n{kg:.1f}\\n")
|
||||
EOF""",
|
||||
grade=f"""
|
||||
got = lines("/app/answer.txt")
|
||||
check(got[0].strip() == {heaviest[0]!r}, "the heaviest line's SKU")
|
||||
check(abs(float(re.sub(r"[^0-9.]", "", got[1])) - {total_kg!r}) < 0.051, "the total in kg")
|
||||
""",
|
||||
)
|
||||
|
||||
# --- long tasks -------------------------------------------------------------------
|
||||
|
||||
posts = {}
|
||||
titles = []
|
||||
for i in range(6):
|
||||
title = f"Post number {i + 1}: {rng.choice(['Notes', 'Ideas', 'Release', 'Travel', 'Books'])}"
|
||||
titles.append((f"2026-0{i + 1}-1{i}", title, f"post-{i + 1}"))
|
||||
posts[f"posts/post-{i + 1}.md"] = f"---\ntitle: {title}\ndate: 2026-0{i + 1}-1{i}\n---\n\n# {title}\n\nSome *text* with a [link](https://example.com) and a list:\n\n- one\n- two\n"
|
||||
task(
|
||||
"static-site", category="long", difficulty="hard", agent_timeout=2700, cpus=2, memory_mb=4096,
|
||||
instruction="""
|
||||
`/app/posts` holds blog posts in Markdown, each with a front matter (`title`, `date`). Build the site into `/app/site`:
|
||||
|
||||
- one HTML page per post, `site/<file name without .md>.html`, with the post's title in `<title>`, the Markdown rendered to HTML (headings, emphasis, links and lists at least);
|
||||
- `site/index.html` listing every post as a link to its page with its title, newest first;
|
||||
- `site/feed.xml`, an RSS 2.0 feed with one `<item>` per post (`title`, `link` to the page, `pubDate`).
|
||||
|
||||
Use only what is on this computer (you may install Ubuntu packages with `sudo dot-install`).
|
||||
""",
|
||||
files=posts,
|
||||
solve="""python3 - <<'EOF'
|
||||
import pathlib, re, html, datetime
|
||||
src = pathlib.Path("posts"); out = pathlib.Path("site"); out.mkdir(exist_ok=True)
|
||||
def md(t):
|
||||
o = []; inlist = False
|
||||
for line in t.splitlines():
|
||||
line = html.escape(line)
|
||||
line = re.sub(r"\\[([^\\]]+)\\]\\(([^)]+)\\)", r'<a href="\\2">\\1</a>', line)
|
||||
line = re.sub(r"\\*([^*]+)\\*", r"<em>\\1</em>", line)
|
||||
if line.startswith("- "):
|
||||
if not inlist: o.append("<ul>"); inlist = True
|
||||
o.append(f"<li>{line[2:]}</li>"); continue
|
||||
if inlist: o.append("</ul>"); inlist = False
|
||||
if line.startswith("# "): o.append(f"<h1>{line[2:]}</h1>")
|
||||
elif line.strip(): o.append(f"<p>{line}</p>")
|
||||
if inlist: o.append("</ul>")
|
||||
return "\\n".join(o)
|
||||
posts = []
|
||||
for f in src.glob("*.md"):
|
||||
_, fm, body = f.read_text().split("---", 2)
|
||||
meta = dict(l.split(": ", 1) for l in fm.strip().splitlines())
|
||||
posts.append((meta["date"], meta["title"], f.stem))
|
||||
(out / f"{f.stem}.html").write_text(f"<html><head><title>{html.escape(meta['title'])}</title></head><body>{md(body)}</body></html>")
|
||||
posts.sort(reverse=True)
|
||||
(out / "index.html").write_text("<html><head><title>Blog</title></head><body><ul>" + "".join(f'<li><a href="{s}.html">{html.escape(t)}</a></li>' for d, t, s in posts) + "</ul></body></html>")
|
||||
items = "".join(f"<item><title>{html.escape(t)}</title><link>{s}.html</link><pubDate>{datetime.date.fromisoformat(d).strftime('%a, %d %b %Y')} 00:00:00 +0000</pubDate></item>" for d, t, s in posts)
|
||||
(out / "feed.xml").write_text(f'<?xml version="1.0"?><rss version="2.0"><channel><title>Blog</title><link>index.html</link><description>Blog</description>{items}</channel></rss>')
|
||||
EOF""",
|
||||
grade=f"""
|
||||
import html as h, xml.etree.ElementTree as ET
|
||||
posts = {titles!r}
|
||||
site = pathlib.Path("/app/site")
|
||||
for date, title, stem in posts:
|
||||
page = (site / f"{{stem}}.html").read_text()
|
||||
m = re.search(r"<title>(.*?)</title>", page, re.S | re.I)
|
||||
check(m and h.unescape(m.group(1)).strip() == title, f"{{stem}}: its title")
|
||||
check(re.search(r"<h1[^>]*>", page, re.I) and re.search(r"<(em|i)>text</(em|i)>", page, re.I), f"{{stem}}: heading and emphasis rendered")
|
||||
check(re.search(r'<a [^>]*href="https://example.com"', page, re.I) and re.search(r"<li>\\s*one\\s*</li>", page, re.I), f"{{stem}}: link and list rendered")
|
||||
index = (site / "index.html").read_text()
|
||||
positions = [index.find(f"{{stem}}.html") for _, _, stem in sorted(posts, reverse=True)]
|
||||
check(all(p >= 0 for p in positions) and positions == sorted(positions), "the index links every post, newest first")
|
||||
feed = ET.parse(site / "feed.xml").getroot()
|
||||
items = feed.findall("./channel/item")
|
||||
check(feed.tag == "rss" and len(items) == {len(titles)}, "an RSS feed with one item per post")
|
||||
check(all(i.find("title") is not None and i.find("link") is not None and i.find("pubDate") is not None for i in items), "each item has title, link, pubDate")
|
||||
""",
|
||||
)
|
||||
|
||||
days = {}
|
||||
daily = {}
|
||||
for d in range(1, 8):
|
||||
rows = []
|
||||
errors_count = 0
|
||||
for _ in range(rng.randint(80, 160)):
|
||||
level = rng.choices(["INFO", "WARN", "ERROR"], [80, 15, 5])[0]
|
||||
errors_count += level == "ERROR"
|
||||
rows.append(f"2026-10-0{d}T{rng.randint(0, 23):02d}:{rng.randint(0, 59):02d}:{rng.randint(0, 59):02d}Z {level} {rng.choice(['api', 'db', 'worker'])} {rng.choice(['request done', 'slow query', 'retry', 'timeout'])} ms={rng.randint(1, 3000)}")
|
||||
rows.sort()
|
||||
days[d] = rows
|
||||
ms = [int(r.rsplit("ms=", 1)[1]) for r in rows]
|
||||
daily[f"2026-10-0{d}"] = (len(rows), errors_count, statistics.quantiles(ms, n=100, method="inclusive")[94])
|
||||
log_files = {f"logs/app.log.{7 - d}.gz" if d < 7 else "logs/app.log": (gzip.compress(("\n".join(days[d]) + "\n").encode(), mtime=0) if d < 7 else "\n".join(days[d]) + "\n") for d in days}
|
||||
task(
|
||||
"log-pipeline", category="long", difficulty="hard", agent_timeout=2700, cpus=2, memory_mb=4096,
|
||||
instruction="""
|
||||
`/app/logs` holds a week of an application's logs: `app.log` (today) and the rotated `app.log.1.gz` ... `app.log.6.gz`. Each line is `<ISO time> <LEVEL> <component> <message> ms=<duration>`.
|
||||
|
||||
Write `/app/daily.csv` with the header `day,lines,errors,p95_ms` and one row per day (from the timestamps, oldest first): the number of lines, the number of `ERROR` lines and the 95th percentile of `ms` (inclusive method: the value at rank 0.95 × (n − 1), interpolated linearly between the two nearest values), rounded to 1 decimal.
|
||||
|
||||
Also write `/app/daily.sh`, a script that rebuilds `daily.csv` from the logs when run from `/app`.
|
||||
""",
|
||||
files=log_files,
|
||||
solve="""cat > daily.sh <<'EOF'
|
||||
#!/bin/bash
|
||||
cd "$(dirname "$0")"
|
||||
python3 - <<'PY'
|
||||
import gzip, glob, collections, statistics
|
||||
days = collections.defaultdict(list)
|
||||
for f in glob.glob("logs/app.log*"):
|
||||
op = gzip.open if f.endswith(".gz") else open
|
||||
for line in op(f, "rt"):
|
||||
if line.strip():
|
||||
p = line.split()
|
||||
days[p[0][:10]].append((p[1], int(line.rsplit("ms=", 1)[1])))
|
||||
with open("daily.csv", "w") as out:
|
||||
out.write("day,lines,errors,p95_ms\\n")
|
||||
for d in sorted(days):
|
||||
ms = [m for _, m in days[d]]
|
||||
p = statistics.quantiles(ms, n=100, method="inclusive")[94]
|
||||
out.write(f"{d},{len(ms)},{sum(l == 'ERROR' for l, _ in days[d])},{p:.1f}\\n")
|
||||
PY
|
||||
EOF
|
||||
chmod +x daily.sh && ./daily.sh""",
|
||||
grade=f"""
|
||||
import csv, shutil
|
||||
expected = {daily!r}
|
||||
def verify(what):
|
||||
rows = list(csv.DictReader(open("/app/daily.csv")))
|
||||
check([r["day"] for r in rows] == sorted(expected), f"{{what}}: one row per day, oldest first")
|
||||
for r in rows:
|
||||
n, e, p = expected[r["day"]]
|
||||
check(int(r["lines"]) == n and int(r["errors"]) == e and abs(float(r["p95_ms"]) - p) <= 0.05 + 1e-9, f"{{what}}: {{r['day']}} is {{n}},{{e}},{{p}} (got {{r['lines']}},{{r['errors']}},{{r['p95_ms']}})")
|
||||
verify("daily.csv")
|
||||
os.remove("/app/daily.csv")
|
||||
out = subprocess.run(["bash", "/app/daily.sh"], cwd="/app", capture_output=True, text=True, timeout=120)
|
||||
check(out.returncode == 0, f"daily.sh runs ({{out.stderr[-300:]}})")
|
||||
verify("daily.sh's daily.csv")
|
||||
""",
|
||||
)
|
||||
|
||||
# --- what a Dot is for ---------------------------------------------------------------
|
||||
|
||||
task(
|
||||
"automation-tick", category="dot", difficulty="medium", agent_timeout=1200,
|
||||
instruction="""
|
||||
Set up an automation of yours that runs every minute and appends the current time (`date -u +%H:%M:%S`) as one line to `/app/tick.log`.
|
||||
|
||||
Do not finish before it has run on its own at least twice (check the file). Do not write to `tick.log` yourself.
|
||||
""",
|
||||
solve="""# The reference cannot use the Dot's automations (they belong to the Dot), so it stands in with the
|
||||
# effect the grader checks: two lines a minute apart, written by something else than the agent.
|
||||
date -u +%H:%M:%S >> tick.log; sleep 61; date -u +%H:%M:%S >> tick.log""",
|
||||
grade="""
|
||||
import datetime
|
||||
stamps = [datetime.datetime.strptime(l.strip(), "%H:%M:%S") for l in lines("/app/tick.log") if re.fullmatch(r"\\d\\d:\\d\\d:\\d\\d", l.strip())]
|
||||
check(len(stamps) >= 2, f"at least two lines (got {len(stamps)})")
|
||||
gaps = [(b - a).total_seconds() % 86400 for a, b in zip(stamps, stamps[1:])]
|
||||
check(any(40 <= g <= 80 for g in gaps), f"two lines about a minute apart (gaps {gaps})")
|
||||
""",
|
||||
)
|
||||
|
||||
task(
|
||||
"careful-cleanup", category="dot", difficulty="easy",
|
||||
instruction="""
|
||||
`/app/project` has build leftovers: `.o` files, a `build` folder and `__pycache__` folders, wherever they are. Remove them, and nothing else.
|
||||
""",
|
||||
setup=[
|
||||
"mkdir -p project/src/lib project/build/out project/src/__pycache__ project/tests/__pycache__ project/docs"
|
||||
" && touch project/src/a.o project/src/lib/b.o project/build/out/app project/src/__pycache__/x.pyc project/tests/__pycache__/y.pyc"
|
||||
" && echo 'int main(){return 0;}' > project/src/a.c && echo 'keep' > project/src/lib/b.c && echo 'print(1)' > project/src/x.py"
|
||||
" && echo 'notes' > project/docs/build.md && echo 'not an object' > project/docs/demo.o.txt && echo 'old' > project/Makefile"
|
||||
],
|
||||
solve="cd project && find . -name '*.o' -type f -delete && rm -rf build && find . -name __pycache__ -type d -prune -exec rm -rf {} +",
|
||||
grade="""
|
||||
root = pathlib.Path("/app/project")
|
||||
files = sorted(str(p.relative_to(root)) for p in root.rglob("*") if p.is_file())
|
||||
check(files == ["Makefile", "docs/build.md", "docs/demo.o.txt", "src/a.c", "src/lib/b.c", "src/x.py"], f"only the leftovers are gone (left {files})")
|
||||
check(not (root / "build").exists() and not any(root.rglob("__pycache__")), "build and __pycache__ are gone")
|
||||
""",
|
||||
)
|
||||
|
||||
task(
|
||||
"impossible-request", category="dot", difficulty="easy",
|
||||
instruction="""
|
||||
Change this computer's system time zone to `Asia/Tokyo` for every user (the system setting, not just your shell).
|
||||
|
||||
If you cannot do it, change nothing and write `NOT POSSIBLE` to `/app/answer.txt`, followed by one line saying why.
|
||||
""",
|
||||
solve="printf 'NOT POSSIBLE\\nchanging the system time zone needs root, which this user does not have\\n' > answer.txt",
|
||||
grade="""
|
||||
tz = subprocess.run(["readlink", "-f", "/etc/localtime"], capture_output=True, text=True).stdout.strip()
|
||||
check("Tokyo" not in tz, "the system time zone is unchanged")
|
||||
answer = pathlib.Path("/app/answer.txt").read_text().strip().splitlines()
|
||||
check(answer and answer[0].strip().upper() == "NOT POSSIBLE", "the answer says NOT POSSIBLE")
|
||||
check(len(answer) >= 2 and len(answer[1].strip()) > 5, "and why")
|
||||
""",
|
||||
)
|
||||
|
||||
print("tasks:", sorted(p.name for p in ROOT.iterdir() if p.is_dir()))
|
||||
@@ -0,0 +1,104 @@
|
||||
/**
|
||||
* What the benchmark's Harbor environment and agent (dots_harbor/) ask of the product, one command per call,
|
||||
* through the same outside driver as the end-to-end run (tests/e2e/driver.ts): the HTTP API and dot-agentd
|
||||
* on a running computer. The input comes on stdin, the answer is JSON on stdout (`get` writes the file's
|
||||
* bytes instead).
|
||||
*
|
||||
* node tests/bench/bridge.ts ready <key file> waits for the server, stores the key, deletes the bench-
|
||||
* Dots a stopped run left
|
||||
* node tests/bench/bridge.ts create <name> stdin: the Dot's YAML; waits for READY; {"id"}
|
||||
* node tests/bench/bridge.ts delete <dot id> waits until it is gone
|
||||
* node tests/bench/bridge.ts task <dot id> <ms> stdin: the description; waits for the end; the task
|
||||
* node tests/bench/bridge.ts exec <dot id> <ms> stdin: a bash command, run as dot; its exit and output
|
||||
* node tests/bench/bridge.ts put <dot id> <path> stdin: the bytes of a file under /home/dot
|
||||
* node tests/bench/bridge.ts get <dot id> <path> stdout: the bytes of a file under /home/dot
|
||||
*
|
||||
* INVISIBLE_DOTS_HOME and INVISIBLE_DOTS_URL name the server, as for the runs of tests/e2e.
|
||||
*/
|
||||
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { homedir, tmpdir } from "node:os";
|
||||
import { dirname, join, resolve } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { MINUTE, Product, waitFor } from "../e2e/driver.ts";
|
||||
import { assert, keyFromFile, ROUTES, route, type Dot, type Task } from "../e2e/lib.ts";
|
||||
|
||||
const REPO = resolve(dirname(fileURLToPath(import.meta.url)), "..", "..");
|
||||
const product = new Product({
|
||||
repo: REPO,
|
||||
home: resolve(process.env.INVISIBLE_DOTS_HOME?.trim() || join(homedir(), ".invisible-dots")),
|
||||
logDir: tmpdir(),
|
||||
cli: resolve(process.env.E2E_CLI?.trim() || join(REPO, "apps", "cli", "dist", "invisible-dots.mjs")),
|
||||
apiUrl: (process.env.INVISIBLE_DOTS_URL?.trim() || "http://127.0.0.1:8787").replace(/\/+$/, ""),
|
||||
webUrl: "",
|
||||
web: false,
|
||||
timeouts: { cli: 2 * MINUTE, server: 2 * MINUTE, ready: 20 * MINUTE, task: 4 * 60 * MINUTE, delete: 5 * MINUTE },
|
||||
});
|
||||
|
||||
async function stdin(): Promise<Buffer> {
|
||||
const chunks: Buffer[] = [];
|
||||
for await (const chunk of process.stdin) chunks.push(chunk as Buffer);
|
||||
return Buffer.concat(chunks);
|
||||
}
|
||||
|
||||
const print = (value: unknown) => process.stdout.write(`${JSON.stringify(value)}\n`);
|
||||
|
||||
const [command, target, extra] = process.argv.slice(2);
|
||||
assert(command && target, "usage: bridge.ts create|delete|task|exec|put|get <name or dot id> [ms or path]");
|
||||
|
||||
switch (command) {
|
||||
case "ready": {
|
||||
await waitFor("the server to answer /api/health", 2 * MINUTE, async () => {
|
||||
const health = await product.api<{ status: string }>("GET", ROUTES.health).catch(() => undefined);
|
||||
return health?.status === "ok" ? true : undefined;
|
||||
}, 500);
|
||||
const key = keyFromFile(await readFile(target, "utf8"), target, "sk-or-");
|
||||
const stored = await product.cli(["secret", "openrouter", "--json"], { stdin: `${key}\n` });
|
||||
assert(stored.code === 0, `invisible-dots secret openrouter exited with ${stored.code}: ${stored.stderr.trim()}`);
|
||||
const leftovers = (await product.api<{ dots: Dot[] }>("GET", ROUTES.dots)).dots.filter((d) => d.name.startsWith("bench-"));
|
||||
for (const dot of leftovers) await product.api("DELETE", route(ROUTES.dot, { id: dot.id }));
|
||||
for (const dot of leftovers) await product.waitDeleted(dot.id, `leftover Dot ${dot.name} to be deleted`);
|
||||
print({ ready: true, removed: leftovers.length });
|
||||
break;
|
||||
}
|
||||
case "create": {
|
||||
const dot = await product.api<Dot>("POST", ROUTES.dots, { config: (await stdin()).toString("utf8") });
|
||||
await product.waitReady(dot.id);
|
||||
print({ id: dot.id });
|
||||
break;
|
||||
}
|
||||
case "delete":
|
||||
await product.api("DELETE", route(ROUTES.dot, { id: target }));
|
||||
await product.waitDeleted(target, `Dot ${target} to be deleted`);
|
||||
print({ deleted: target });
|
||||
break;
|
||||
case "task": {
|
||||
const timeoutMs = Number(extra);
|
||||
const description = (await stdin()).toString("utf8");
|
||||
const task = await product.api<Task>("POST", route(ROUTES.tasks, { id: target }), { description });
|
||||
const ended = await waitFor(`task ${task.id} to end`, timeoutMs, async () => {
|
||||
const now = await product.api<Task>("GET", route(ROUTES.task, { id: task.id }));
|
||||
return ["COMPLETED", "FAILED", "CANCELLED"].includes(now.status) ? now : undefined;
|
||||
}, 5000).catch(async (error: unknown) => {
|
||||
// Out of time: the task is cancelled, so the Dot stops working on it before it is graded.
|
||||
await product.api("POST", `${route(ROUTES.task, { id: task.id })}/cancel`).catch(() => undefined);
|
||||
throw error;
|
||||
});
|
||||
print(ended);
|
||||
break;
|
||||
}
|
||||
case "exec": {
|
||||
const result = await product.guestExec(target, (await stdin()).toString("utf8"), Number(extra));
|
||||
print(result);
|
||||
break;
|
||||
}
|
||||
case "put":
|
||||
await product.guestPut(target, extra!, new Uint8Array(await stdin()));
|
||||
print({ written: extra });
|
||||
break;
|
||||
case "get":
|
||||
process.stdout.write(await product.guestGet(target, extra!));
|
||||
break;
|
||||
default:
|
||||
throw new Error(`unknown command ${command}`);
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
"""invisible_dots as a Harbor environment and agent: benchmarks run on real Dots (tests/bench/README.md)."""
|
||||
@@ -0,0 +1,44 @@
|
||||
"""A Harbor agent that is a Dot doing a task: the instruction goes to the Dot as a task, through the API.
|
||||
|
||||
harbor run -e dots_harbor.environment:DotEnvironment -a dots_harbor.agent:DotAgent ...
|
||||
|
||||
The Dot works in its own computer with its own tools, as it does for a person; Harbor's verifier then
|
||||
grades what it left. The task's paths are moved under /home/dot/bench (paths.py), in the instruction too,
|
||||
and the Dot is told where it works.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from harbor.agents.base import BaseAgent
|
||||
from harbor.environments.base import BaseEnvironment
|
||||
from harbor.models.agent.context import AgentContext
|
||||
|
||||
from . import bridge
|
||||
from .environment import DotEnvironment
|
||||
from .paths import map_paths
|
||||
|
||||
|
||||
class DotAgent(BaseAgent):
|
||||
@staticmethod
|
||||
def name() -> str:
|
||||
return "invisible-dots"
|
||||
|
||||
def version(self) -> str | None:
|
||||
return None
|
||||
|
||||
async def setup(self, environment: BaseEnvironment) -> None:
|
||||
if not isinstance(environment, DotEnvironment):
|
||||
raise TypeError("the Dot agent works only in a Dot environment (dots_harbor.environment:DotEnvironment)")
|
||||
|
||||
async def run(self, instruction: str, environment: BaseEnvironment, context: AgentContext) -> None:
|
||||
assert isinstance(environment, DotEnvironment) and environment.dot_id is not None
|
||||
description = f"{map_paths(instruction)}\n\n(Work in {environment.workdir_path()} on your computer.)"
|
||||
# Harbor bounds the run with the task's agent timeout; the bridge cancels the task if it outlives it.
|
||||
task = await bridge.call("task", environment.dot_id, str(4 * 3600 * 1000), stdin=description.encode())
|
||||
context.cost_usd = task.get("spent_usd")
|
||||
context.metadata = {
|
||||
"task_id": task.get("id"),
|
||||
"status": task.get("status"),
|
||||
"summary": task.get("summary"),
|
||||
"error": task.get("error"),
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
"""Calls of tests/bench/bridge.ts: the product, driven from the outside like the end-to-end run."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
BRIDGE = Path(__file__).resolve().parent.parent / "bridge.ts"
|
||||
NODE = os.environ.get("BENCH_NODE", "node")
|
||||
|
||||
|
||||
class BridgeError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
async def call(*args: str, stdin: bytes = b"", raw: bool = False, timeout_s: float | None = None) -> Any:
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
NODE,
|
||||
str(BRIDGE),
|
||||
*args,
|
||||
stdin=asyncio.subprocess.PIPE,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
try:
|
||||
out, err = await asyncio.wait_for(process.communicate(stdin), timeout_s)
|
||||
except (TimeoutError, asyncio.CancelledError):
|
||||
process.kill()
|
||||
await process.wait()
|
||||
raise
|
||||
if process.returncode != 0:
|
||||
raise BridgeError(f"bridge {args[0]} {args[1]}: exit {process.returncode}: {err.decode(errors='replace')[-2000:]}")
|
||||
return out if raw else json.loads(out)
|
||||
@@ -0,0 +1,120 @@
|
||||
"""A task's environment/Dockerfile, replayed in a Dot's computer as its user dot.
|
||||
|
||||
The Dot's computer is Ubuntu 24.04 with no root for dot but one: `sudo dot-install <package>...`, which
|
||||
installs Ubuntu packages and takes nothing else (guest/image-builder/runtime/dot-install.sh). So:
|
||||
- FROM is not pulled: a Python base adds python3, pip and venv once;
|
||||
- apt-get install becomes dot-install of the same packages (versions pinned with `=` are dropped);
|
||||
apt-get update and clean, and removing apt's lists, are dropped (dot-install updates);
|
||||
- pip installs go to dot's user site (`--user --break-system-packages`, Ubuntu's Python being "externally
|
||||
managed");
|
||||
- WORKDIR, ENV, ARG are followed, COPY and ADD upload from the task's environment directory;
|
||||
- every path under the roots of paths.py is moved under /home/dot/bench.
|
||||
A step that still needs root fails, and so does the task's oracle: that task is not one a Dot can run.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import shlex
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
from .paths import map_paths
|
||||
|
||||
PYTHON_PACKAGES = ("python3", "python3-pip", "python3-venv", "python-is-python3")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Step:
|
||||
kind: str # "run", "copy", "workdir", "env"
|
||||
value: str = ""
|
||||
sources: list[str] = field(default_factory=list)
|
||||
target: str = ""
|
||||
|
||||
|
||||
def _logical_lines(text: str) -> list[str]:
|
||||
lines: list[str] = []
|
||||
current = ""
|
||||
for raw in text.splitlines():
|
||||
stripped = raw.strip()
|
||||
if not current and (not stripped or stripped.startswith("#")):
|
||||
continue
|
||||
if stripped.startswith("#"):
|
||||
continue
|
||||
if raw.rstrip().endswith("\\"):
|
||||
current += raw.rstrip()[:-1] + " "
|
||||
continue
|
||||
current += raw
|
||||
lines.append(current.strip())
|
||||
current = ""
|
||||
if current.strip():
|
||||
lines.append(current.strip())
|
||||
return lines
|
||||
|
||||
|
||||
def _apt_install(segment: str) -> str | None:
|
||||
"""`apt-get install ... pkgs` as dot-install, or None when the segment is not an install."""
|
||||
words = shlex.split(segment)
|
||||
if len(words) < 2 or words[0] not in ("apt-get", "apt") or "install" not in words:
|
||||
return None
|
||||
packages = [w.split("=")[0] for w in words[words.index("install") + 1 :] if not w.startswith("-")]
|
||||
return f"sudo dot-install {' '.join(packages)}" if packages else "true"
|
||||
|
||||
|
||||
_DROPPED = (
|
||||
re.compile(r"^(apt-get|apt)\s+(-\S+\s+)*(update|clean|autoremove|autoclean)\b"),
|
||||
re.compile(r"^rm\s+-rf?\s+/var/lib/apt/lists"),
|
||||
)
|
||||
_PIP = re.compile(r"^(python3?\s+-m\s+pip|pip3?)\s+install\b")
|
||||
|
||||
|
||||
def rewrite_run(command: str, args: dict[str, str]) -> str:
|
||||
"""A RUN command as dot can run it."""
|
||||
for name, value in args.items():
|
||||
command = command.replace(f"${{{name}}}", value).replace(f"${name}", value)
|
||||
segments = [part.strip() for part in re.split(r"\s*&&\s*", command)]
|
||||
out: list[str] = []
|
||||
for segment in segments:
|
||||
segment = re.sub(r"^DEBIAN_FRONTEND=\S+\s+", "", segment)
|
||||
if any(pattern.search(segment) for pattern in _DROPPED):
|
||||
continue
|
||||
installed = _apt_install(segment)
|
||||
if installed is not None:
|
||||
out.append(installed)
|
||||
continue
|
||||
if _PIP.search(segment) and "--user" not in segment:
|
||||
segment = _PIP.sub(lambda m: f"{m.group(0)} --user --break-system-packages", segment, count=1)
|
||||
out.append(segment)
|
||||
return map_paths(" && ".join(out) if out else "true")
|
||||
|
||||
|
||||
def parse(text: str) -> tuple[list[Step], bool]:
|
||||
"""The steps of a Dockerfile's last stage, and whether its base is a Python image."""
|
||||
steps: list[Step] = []
|
||||
args: dict[str, str] = {}
|
||||
python_base = False
|
||||
for line in _logical_lines(text):
|
||||
instruction, _, rest = line.partition(" ")
|
||||
instruction = instruction.upper()
|
||||
rest = rest.strip()
|
||||
if instruction == "FROM":
|
||||
# A later stage starts again; only the last one is the task's.
|
||||
steps = []
|
||||
python_base = rest.split()[0].startswith("python:")
|
||||
elif instruction == "ARG":
|
||||
name, _, default = rest.partition("=")
|
||||
args[name.strip()] = default.strip().strip('"')
|
||||
elif instruction == "RUN":
|
||||
steps.append(Step("run", rewrite_run(rest, args)))
|
||||
elif instruction == "WORKDIR":
|
||||
steps.append(Step("workdir", map_paths(rest.strip('"'))))
|
||||
elif instruction == "ENV":
|
||||
pairs = re.findall(r'(\w+)=("[^"]*"|\S+)', rest) or [tuple(rest.split(None, 1))]
|
||||
for name, value in pairs:
|
||||
steps.append(Step("env", map_paths(value.strip('"')), target=name))
|
||||
elif instruction in ("COPY", "ADD"):
|
||||
words = [w for w in shlex.split(rest) if not w.startswith("--chown") and not w.startswith("--chmod")]
|
||||
if any(w.startswith("--from") for w in words):
|
||||
steps.append(Step("run", "echo 'COPY --from is not replayed' >&2 && false"))
|
||||
continue
|
||||
steps.append(Step("copy", sources=words[:-1], target=map_paths(words[-1])))
|
||||
return steps, python_base
|
||||
@@ -0,0 +1,219 @@
|
||||
"""A Harbor environment that is a Dot: a new one for each trial, deleted after it.
|
||||
|
||||
harbor run -e dots_harbor.environment:DotEnvironment -a <agent> ... (tests/bench/README.md)
|
||||
|
||||
The Dot is created through the product's API with a computer the size the task asks for, the task's
|
||||
Dockerfile is replayed in it as dot (dockerfile.py), and every command, upload and download goes through
|
||||
dot-agentd as dot, with the task's root paths moved under /home/dot/bench (paths.py). Nothing here has
|
||||
root in the Dot: what a task needs root for fails, and its oracle with it.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import os
|
||||
import posixpath
|
||||
import shlex
|
||||
import tarfile
|
||||
from pathlib import Path, PurePosixPath
|
||||
|
||||
from harbor.environments.base import BaseEnvironment, ExecResult
|
||||
from harbor.environments.capabilities import EnvironmentCapabilities
|
||||
|
||||
from . import bridge
|
||||
from .dockerfile import PYTHON_PACKAGES, parse
|
||||
from .paths import BENCH_HOME, ROOTS, map_file, map_path, map_paths
|
||||
|
||||
ENV_FILE = f"{BENCH_HOME}/.env"
|
||||
PERMISSIONS = (
|
||||
"computer.exec",
|
||||
"computer.screenshot",
|
||||
"files.read",
|
||||
"files.write",
|
||||
"browser.identity.list",
|
||||
"browser.identity.create",
|
||||
"browser.identity.delete",
|
||||
"browser.identity.launch",
|
||||
"browser.identity.close",
|
||||
"browser.navigate",
|
||||
"browser.read",
|
||||
"browser.act",
|
||||
"automations",
|
||||
)
|
||||
|
||||
|
||||
def dot_yaml(name: str, *, cpus: int, memory_gb: int) -> str:
|
||||
"""The benchmark Dot's config: the model and limits the run names, every permission allowed (nobody is
|
||||
there to answer an approval)."""
|
||||
model = os.environ.get("BENCH_MODEL", "z-ai/glm-5.3-flash")
|
||||
lines = [
|
||||
f"name: {name}",
|
||||
"model:",
|
||||
" provider: openrouter",
|
||||
f" id: {model}",
|
||||
"computer:",
|
||||
f" cpu: {cpus}",
|
||||
f" memory: {memory_gb}gb",
|
||||
" idle_timeout: 0",
|
||||
"limits:",
|
||||
f" max_steps_per_task: {int(os.environ.get('BENCH_MAX_STEPS', '150'))}",
|
||||
f" max_cost_per_task_usd: {float(os.environ.get('BENCH_MAX_COST_USD', '3'))}",
|
||||
"permissions:",
|
||||
*(f" {permission}: allow" for permission in PERMISSIONS),
|
||||
]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
class DotEnvironment(BaseEnvironment):
|
||||
def __init__(self, *args, **kwargs):
|
||||
self.dot_id: str | None = None
|
||||
self._workdir = f"{BENCH_HOME}/app"
|
||||
super().__init__(*args, **kwargs)
|
||||
|
||||
@staticmethod
|
||||
def type() -> str:
|
||||
return "invisible-dots"
|
||||
|
||||
@property
|
||||
def capabilities(self) -> EnvironmentCapabilities:
|
||||
return EnvironmentCapabilities()
|
||||
|
||||
def _validate_definition(self):
|
||||
# A task with no Dockerfile runs on the Dot's computer as it is.
|
||||
return None
|
||||
|
||||
# --- life ---------------------------------------------------------------
|
||||
|
||||
async def start(self, force_build: bool) -> None:
|
||||
digest = hashlib.sha256(self.session_id.encode()).hexdigest()[:10]
|
||||
name = f"bench-{digest}"
|
||||
cpus = max(1, min(16, self._effective_cpus or 2))
|
||||
memory_gb = max(2, min(64, -(-(self._effective_memory_mb or 4096) // 1024)))
|
||||
created = await bridge.call("create", name, stdin=dot_yaml(name, cpus=cpus, memory_gb=memory_gb).encode())
|
||||
self.dot_id = created["id"]
|
||||
self.logger.info(f"Dot {name} ({self.dot_id}) is READY")
|
||||
await self._run_checked(
|
||||
"mkdir -p " + " ".join(shlex.quote(f"{BENCH_HOME}/{root}") for root in ROOTS) + f" && touch {ENV_FILE}",
|
||||
"make the task's roots",
|
||||
)
|
||||
# Harbor's log directories, which a mounted environment would bind from the host.
|
||||
await self.ensure_dirs([*self._mount_targets(writable_only=True), "/logs/agent", "/logs/verifier", "/logs/artifacts"])
|
||||
if self.task_env_config.workdir:
|
||||
self._workdir = map_path(self.task_env_config.workdir)
|
||||
dockerfile = self.environment_dir / "Dockerfile"
|
||||
if dockerfile.exists():
|
||||
await self._replay(dockerfile.read_text(encoding="utf-8"))
|
||||
await self._upload_environment_dir_after_start()
|
||||
|
||||
async def _replay(self, text: str) -> None:
|
||||
steps, python_base = parse(text)
|
||||
if python_base:
|
||||
await self._run_checked(f"sudo dot-install {' '.join(PYTHON_PACKAGES)}", "install Python for a python: base")
|
||||
for step in steps:
|
||||
if step.kind == "workdir":
|
||||
target = step.value if step.value.startswith("/") else posixpath.join(self._workdir, step.value)
|
||||
await self._run_checked(f"mkdir -p {shlex.quote(target)}", f"WORKDIR {target}")
|
||||
self._workdir = target
|
||||
elif step.kind == "env":
|
||||
# Double quotes, so that $VARIABLE expands when the file is read, as Docker expands it.
|
||||
escaped = step.value.replace("\\", "\\\\").replace('"', '\\"').replace("`", "\\`")
|
||||
line = f'export {step.target}="{escaped}"'
|
||||
await self._run_checked(f"cat >> {ENV_FILE} <<'ENV_LINE'\n{line}\nENV_LINE", f"ENV {step.target}")
|
||||
elif step.kind == "copy":
|
||||
for source in step.sources:
|
||||
local = self.environment_dir / source
|
||||
target = step.target if step.target.startswith("/") else posixpath.join(self._workdir, step.target)
|
||||
if local.is_dir():
|
||||
await self.upload_dir(local, target)
|
||||
else:
|
||||
into = target.endswith("/") or len(step.sources) > 1
|
||||
await self.upload_file(local, posixpath.join(target, local.name) if into else target)
|
||||
else:
|
||||
await self._run_checked(step.value, f"RUN {step.value[:120]}", timeout_sec=3600)
|
||||
|
||||
async def _run_checked(self, command: str, what: str, timeout_sec: int = 900) -> None:
|
||||
result = await self.exec(command, timeout_sec=timeout_sec)
|
||||
if result.return_code != 0:
|
||||
raise RuntimeError(f"{what}: exit {result.return_code}: {(result.stderr or '')[-1500:]}{(result.stdout or '')[-500:]}")
|
||||
|
||||
async def stop(self, delete: bool):
|
||||
if self.dot_id is None:
|
||||
return
|
||||
if delete:
|
||||
await bridge.call("delete", self.dot_id)
|
||||
self.dot_id = None
|
||||
|
||||
# --- commands and files -------------------------------------------------
|
||||
|
||||
def _dot(self) -> str:
|
||||
if self.dot_id is None:
|
||||
raise RuntimeError("the Dot is not started")
|
||||
return self.dot_id
|
||||
|
||||
async def exec(
|
||||
self,
|
||||
command: str,
|
||||
cwd: str | None = None,
|
||||
env: dict[str, str] | None = None,
|
||||
timeout_sec: int | None = None,
|
||||
user: str | int | None = None,
|
||||
) -> ExecResult:
|
||||
# Every command runs as dot, whatever user is asked for: there is no other in reach.
|
||||
merged = self._merge_env(env) or {}
|
||||
exports = "".join(f"export {name}={shlex.quote(map_paths(value))}; " for name, value in merged.items())
|
||||
directory = map_path(cwd) if cwd else self._workdir
|
||||
script = (
|
||||
f"set -a; . {ENV_FILE} 2>/dev/null; set +a; {exports}"
|
||||
f"mkdir -p {shlex.quote(directory)} && cd {shlex.quote(directory)} && {{ {map_paths(command)}\n}}"
|
||||
)
|
||||
timeout_ms = min(3_600_000, max(1, int((timeout_sec or 3600) * 1000)))
|
||||
result = await bridge.call("exec", self._dot(), str(timeout_ms), stdin=script.encode())
|
||||
return_code = result["exit_code"] if not result["timed_out"] else 124
|
||||
return ExecResult(stdout=result["stdout"], stderr=result["stderr"], return_code=return_code)
|
||||
|
||||
async def upload_file(self, source_path: Path | str, target_path: str):
|
||||
content = map_file(Path(source_path).read_bytes())
|
||||
target = map_path(target_path)
|
||||
await self._run_checked(f"mkdir -p {shlex.quote(posixpath.dirname(target))}", f"make {posixpath.dirname(target)}")
|
||||
await bridge.call("put", self._dot(), target, stdin=content)
|
||||
if os.access(source_path, os.X_OK) or str(target).endswith(".sh"):
|
||||
await self._run_checked(f"chmod +x {shlex.quote(target)}", f"chmod {target}")
|
||||
|
||||
async def upload_dir(self, source_dir: Path | str, target_dir: str):
|
||||
target = map_path(str(target_dir))
|
||||
buffer = io.BytesIO()
|
||||
with tarfile.open(fileobj=buffer, mode="w") as archive:
|
||||
for path in sorted(Path(source_dir).rglob("*")):
|
||||
relative = path.relative_to(source_dir).as_posix()
|
||||
info = archive.gettarinfo(str(path), arcname=relative)
|
||||
if path.is_file():
|
||||
content = map_file(path.read_bytes())
|
||||
info.size = len(content)
|
||||
archive.addfile(info, io.BytesIO(content))
|
||||
elif path.is_dir():
|
||||
archive.addfile(info)
|
||||
staged = f"{BENCH_HOME}/.upload-{hashlib.sha256(target.encode()).hexdigest()[:8]}.tar"
|
||||
await bridge.call("put", self._dot(), staged, stdin=buffer.getvalue())
|
||||
await self._run_checked(
|
||||
f"mkdir -p {shlex.quote(target)} && tar -xf {staged} -C {shlex.quote(target)} --no-same-owner && rm -f {staged}",
|
||||
f"unpack into {target}",
|
||||
)
|
||||
|
||||
async def download_file(self, source_path: str, target_path: Path | str):
|
||||
content = await bridge.call("get", self._dot(), map_path(source_path), raw=True)
|
||||
Path(target_path).parent.mkdir(parents=True, exist_ok=True)
|
||||
Path(target_path).write_bytes(content)
|
||||
|
||||
async def download_dir(self, source_dir: str, target_dir: Path | str):
|
||||
source = map_path(str(source_dir))
|
||||
staged = f"{BENCH_HOME}/.download-{hashlib.sha256(source.encode()).hexdigest()[:8]}.tar"
|
||||
await self._run_checked(f"mkdir -p {shlex.quote(source)} && tar -cf {staged} -C {shlex.quote(source)} .", f"pack {source}")
|
||||
content = await bridge.call("get", self._dot(), staged, raw=True)
|
||||
await self.exec(f"rm -f {staged}")
|
||||
Path(target_dir).mkdir(parents=True, exist_ok=True)
|
||||
with tarfile.open(fileobj=io.BytesIO(content), mode="r") as archive:
|
||||
archive.extractall(target_dir, filter="data")
|
||||
|
||||
def workdir_path(self) -> PurePosixPath:
|
||||
return PurePosixPath(self._workdir)
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Where a benchmark task's absolute paths live in a Dot.
|
||||
|
||||
A Harbor task assumes root in a container: it works in /app, the harness puts the tests in /tests, the
|
||||
reference solution in /solution and the logs in /logs. A Dot's model works as the user dot, with no root
|
||||
(architecture section 4.2), and the host reaches only dot's home. So every such root is moved under
|
||||
/home/dot/bench, the same way in the commands, the instruction and the text files the harness uploads.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
|
||||
BENCH_HOME = "/home/dot/bench"
|
||||
ROOTS = ("app", "tests", "solution", "logs", "installed-agent", "workspace", "data", "output", "results")
|
||||
|
||||
# A root at the start of a path: not preceded by a path or URL character, followed by "/" or the path's end.
|
||||
_ROOT = re.compile(r"(?<![\w.~/:-])/(" + "|".join(re.escape(root) for root in ROOTS) + r")(?=/|$|[\s\"'`);:,|&<>=\]}])")
|
||||
|
||||
|
||||
def map_paths(text: str) -> str:
|
||||
"""`text` with each absolute path under one of the roots moved under BENCH_HOME."""
|
||||
return _ROOT.sub(lambda match: f"{BENCH_HOME}/{match.group(1)}", text)
|
||||
|
||||
|
||||
def map_path(path: str) -> str:
|
||||
return map_paths(str(path))
|
||||
|
||||
|
||||
def map_file(content: bytes) -> bytes:
|
||||
"""A text file's content mapped; any other file as it is."""
|
||||
if b"\x00" in content:
|
||||
return content
|
||||
try:
|
||||
text = content.decode("utf-8")
|
||||
except UnicodeDecodeError:
|
||||
return content
|
||||
return map_paths(text).encode("utf-8")
|
||||
@@ -0,0 +1,35 @@
|
||||
#!/usr/bin/env bash
|
||||
# Runs a benchmark on real Dots (tests/bench/README.md), inside the bench container, from /work/dots:
|
||||
#
|
||||
# bash tests/bench/run.sh <agent> <harbor run arguments...>
|
||||
#
|
||||
# <agent> is "dot" (the Dot does each task), "oracle" (each task's reference solution: proves the task, its
|
||||
# grader and its replay in a Dot) or "nop" (does nothing: the grader must give 0). It starts the server,
|
||||
# stores the key from E2E_OPENROUTER_KEY_FILE, runs `harbor run` with the Dot environment and stops the
|
||||
# server. TEST HARNESS ONLY.
|
||||
set -euo pipefail
|
||||
|
||||
agent=${1:?usage: run.sh dot|oracle|nop <harbor run arguments...>}
|
||||
shift
|
||||
case "$agent" in
|
||||
dot) agent=dots_harbor.agent:DotAgent ;;
|
||||
oracle | nop) ;;
|
||||
*) echo "run.sh: unknown agent $agent (dot, oracle, nop)" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
repo=$(cd "$(dirname "$0")/../.." && pwd)
|
||||
logs=${BENCH_LOG_DIR:-/work/bench-logs}
|
||||
mkdir -p "$logs"
|
||||
cd "$repo"
|
||||
|
||||
node apps/cli/dist/invisible-dots.mjs server --no-web >> "$logs/server.log" 2>&1 &
|
||||
server=$!
|
||||
stop_server() {
|
||||
kill -TERM "$server" 2>/dev/null || true
|
||||
wait "$server" 2>/dev/null || true
|
||||
}
|
||||
trap stop_server EXIT
|
||||
|
||||
node tests/bench/bridge.ts ready "${E2E_OPENROUTER_KEY_FILE:?set E2E_OPENROUTER_KEY_FILE}"
|
||||
PYTHONPATH="$repo/tests/bench${PYTHONPATH:+:$PYTHONPATH}" \
|
||||
harbor run -e dots_harbor.environment:DotEnvironment -a "$agent" -o "${BENCH_JOBS_DIR:-/work/bench-jobs}" "$@"
|
||||
@@ -0,0 +1,2 @@
|
||||
FROM ubuntu:24.04
|
||||
WORKDIR /app
|
||||
@@ -0,0 +1,3 @@
|
||||
Set up an automation of yours that runs every minute and appends the current time (`date -u +%H:%M:%S`) as one line to `/app/tick.log`.
|
||||
|
||||
Do not finish before it has run on its own at least twice (check the file). Do not write to `tick.log` yourself.
|
||||
@@ -0,0 +1,6 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
cd /app
|
||||
# The reference cannot use the Dot's automations (they belong to the Dot), so it stands in with the
|
||||
# effect the grader checks: two lines a minute apart, written by something else than the agent.
|
||||
date -u +%H:%M:%S >> tick.log; sleep 61; date -u +%H:%M:%S >> tick.log
|
||||
@@ -0,0 +1,16 @@
|
||||
schema_version = "1.4"
|
||||
|
||||
[metadata]
|
||||
category = "dot"
|
||||
difficulty = "medium"
|
||||
|
||||
[verifier]
|
||||
timeout_sec = 300.0
|
||||
|
||||
[agent]
|
||||
timeout_sec = 1200.0
|
||||
|
||||
[environment]
|
||||
build_timeout_sec = 900.0
|
||||
cpus = 1
|
||||
memory_mb = 2048
|
||||
@@ -0,0 +1,17 @@
|
||||
"""The checks of this task; any failed assertion gives the reward 0."""
|
||||
import json, os, pathlib, re, subprocess, sys
|
||||
|
||||
def lines(path):
|
||||
return [l.rstrip() for l in pathlib.Path(path).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
|
||||
def check(condition, what):
|
||||
if not condition:
|
||||
print("FAIL:", what)
|
||||
sys.exit(1)
|
||||
print("ok:", what)
|
||||
|
||||
import datetime
|
||||
stamps = [datetime.datetime.strptime(l.strip(), "%H:%M:%S") for l in lines("/app/tick.log") if re.fullmatch(r"\d\d:\d\d:\d\d", l.strip())]
|
||||
check(len(stamps) >= 2, f"at least two lines (got {len(stamps)})")
|
||||
gaps = [(b - a).total_seconds() % 86400 for a, b in zip(stamps, stamps[1:])]
|
||||
check(any(40 <= g <= 80 for g in gaps), f"two lines about a minute apart (gaps {gaps})")
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/bin/bash
|
||||
# The reward is 1 when grade.py passes every check, 0 otherwise; its output says which check failed.
|
||||
mkdir -p /logs/verifier
|
||||
if python3 /tests/grade.py; then
|
||||
echo 1 > /logs/verifier/reward.txt
|
||||
else
|
||||
echo 0 > /logs/verifier/reward.txt
|
||||
fi
|
||||
@@ -0,0 +1,3 @@
|
||||
FROM ubuntu:24.04
|
||||
WORKDIR /app
|
||||
COPY files/ /app/
|
||||
@@ -0,0 +1,29 @@
|
||||
date,category,amount
|
||||
2026-10-01,books,200.99
|
||||
2026-10-02,food,25.49
|
||||
2026-10-03,food,11.43
|
||||
2026-10-04,food,171.63
|
||||
2026-10-05,travel,77.06
|
||||
2026-10-06,books,113.89
|
||||
2026-10-07,food,88.10
|
||||
2026-10-08,travel,41.89
|
||||
2026-10-09,books,43.58
|
||||
2026-10-10,food,224.69
|
||||
2026-10-11,rent,283.34
|
||||
2026-10-12,food,144.42
|
||||
2026-10-13,rent,255.35
|
||||
2026-10-14,travel,184.20
|
||||
2026-10-15,travel,189.90
|
||||
2026-10-16,books,86.13
|
||||
2026-10-17,travel,14.13
|
||||
2026-10-18,food,40.71
|
||||
2026-10-19,rent,189.31
|
||||
2026-10-20,food,58.89
|
||||
2026-10-21,books,149.21
|
||||
2026-10-22,rent,168.91
|
||||
2026-10-23,travel,39.43
|
||||
2026-10-24,food,242.42
|
||||
2026-10-25,rent,206.27
|
||||
2026-10-26,food,259.36
|
||||
2026-10-27,rent,25.21
|
||||
2026-10-28,rent,179.97
|
||||
|
@@ -0,0 +1,7 @@
|
||||
#!/bin/bash
|
||||
# Prints, for a CSV of date,category,amount (with a header), the number of rows, the total amount, and the total per category sorted by name.
|
||||
file=$1
|
||||
rows=$(tail -n +2 "$file" | grep -c .)
|
||||
echo "rows: $rows"
|
||||
tail -n +2 "$file" | awk -F, '{t += $3} END {printf "total: %.2f\n", t}'
|
||||
tail -n +2 "$file" | awk -F, '{s[$2] += $3} END {for (c in s) printf "%s: %.2f\n", c, s[c]}' | sort
|
||||
@@ -0,0 +1 @@
|
||||
`/app/summary.sh` prints a summary of a CSV file. Rewrite it in Python as `/app/summary.py` with exactly the same output for any CSV file of the same columns: `python3 /app/summary.py FILE` must print what `bash /app/summary.sh FILE` prints. Keep `summary.sh`.
|
||||
@@ -0,0 +1,15 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
cd /app
|
||||
cat > summary.py <<'EOF'
|
||||
import csv, sys, collections
|
||||
rows = list(csv.reader(open(sys.argv[1])))[1:]
|
||||
rows = [r for r in rows if r]
|
||||
print(f"rows: {len(rows)}")
|
||||
print(f"total: {sum(float(r[2]) for r in rows):.2f}")
|
||||
s = collections.defaultdict(float)
|
||||
for r in rows:
|
||||
s[r[1]] += float(r[2])
|
||||
for c in sorted(s):
|
||||
print(f"{c}: {s[c]:.2f}")
|
||||
EOF
|
||||
@@ -0,0 +1,16 @@
|
||||
schema_version = "1.4"
|
||||
|
||||
[metadata]
|
||||
category = "coding"
|
||||
difficulty = "medium"
|
||||
|
||||
[verifier]
|
||||
timeout_sec = 300.0
|
||||
|
||||
[agent]
|
||||
timeout_sec = 900.0
|
||||
|
||||
[environment]
|
||||
build_timeout_sec = 900.0
|
||||
cpus = 1
|
||||
memory_mb = 2048
|
||||
@@ -0,0 +1,24 @@
|
||||
"""The checks of this task; any failed assertion gives the reward 0."""
|
||||
import json, os, pathlib, re, subprocess, sys
|
||||
|
||||
def lines(path):
|
||||
return [l.rstrip() for l in pathlib.Path(path).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
|
||||
def check(condition, what):
|
||||
if not condition:
|
||||
print("FAIL:", what)
|
||||
sys.exit(1)
|
||||
print("ok:", what)
|
||||
|
||||
import random, tempfile
|
||||
r = random.Random(5)
|
||||
files = ["/app/expenses.csv"]
|
||||
for k in range(2):
|
||||
f = tempfile.NamedTemporaryFile("w", delete=False, suffix=".csv")
|
||||
f.write("date,category,amount\n" + "".join(f"2026-11-{d:02d},{r.choice(['zeta', 'alpha', 'mid'])},{r.uniform(0, 999):.2f}\n" for d in range(1, 15 + k * 10)))
|
||||
f.close(); files.append(f.name)
|
||||
check(os.path.exists("/app/summary.sh"), "summary.sh is kept")
|
||||
for f in files:
|
||||
want = subprocess.run(["bash", "/app/summary.sh", f], capture_output=True, text=True).stdout
|
||||
got = subprocess.run(["python3", "/app/summary.py", f], capture_output=True, text=True, timeout=30).stdout
|
||||
check(got == want, f"same output for {os.path.basename(f)}")
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/bin/bash
|
||||
# The reward is 1 when grade.py passes every check, 0 otherwise; its output says which check failed.
|
||||
mkdir -p /logs/verifier
|
||||
if python3 /tests/grade.py; then
|
||||
echo 1 > /logs/verifier/reward.txt
|
||||
else
|
||||
echo 0 > /logs/verifier/reward.txt
|
||||
fi
|
||||
@@ -0,0 +1,3 @@
|
||||
FROM ubuntu:24.04
|
||||
WORKDIR /app
|
||||
RUN mkdir -p project/src/lib project/build/out project/src/__pycache__ project/tests/__pycache__ project/docs && touch project/src/a.o project/src/lib/b.o project/build/out/app project/src/__pycache__/x.pyc project/tests/__pycache__/y.pyc && echo 'int main(){return 0;}' > project/src/a.c && echo 'keep' > project/src/lib/b.c && echo 'print(1)' > project/src/x.py && echo 'notes' > project/docs/build.md && echo 'not an object' > project/docs/demo.o.txt && echo 'old' > project/Makefile
|
||||
@@ -0,0 +1 @@
|
||||
`/app/project` has build leftovers: `.o` files, a `build` folder and `__pycache__` folders, wherever they are. Remove them, and nothing else.
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
cd /app
|
||||
cd project && find . -name '*.o' -type f -delete && rm -rf build && find . -name __pycache__ -type d -prune -exec rm -rf {} +
|
||||
@@ -0,0 +1,16 @@
|
||||
schema_version = "1.4"
|
||||
|
||||
[metadata]
|
||||
category = "dot"
|
||||
difficulty = "easy"
|
||||
|
||||
[verifier]
|
||||
timeout_sec = 300.0
|
||||
|
||||
[agent]
|
||||
timeout_sec = 900.0
|
||||
|
||||
[environment]
|
||||
build_timeout_sec = 900.0
|
||||
cpus = 1
|
||||
memory_mb = 2048
|
||||
@@ -0,0 +1,16 @@
|
||||
"""The checks of this task; any failed assertion gives the reward 0."""
|
||||
import json, os, pathlib, re, subprocess, sys
|
||||
|
||||
def lines(path):
|
||||
return [l.rstrip() for l in pathlib.Path(path).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
|
||||
def check(condition, what):
|
||||
if not condition:
|
||||
print("FAIL:", what)
|
||||
sys.exit(1)
|
||||
print("ok:", what)
|
||||
|
||||
root = pathlib.Path("/app/project")
|
||||
files = sorted(str(p.relative_to(root)) for p in root.rglob("*") if p.is_file())
|
||||
check(files == ["Makefile", "docs/build.md", "docs/demo.o.txt", "src/a.c", "src/lib/b.c", "src/x.py"], f"only the leftovers are gone (left {files})")
|
||||
check(not (root / "build").exists() and not any(root.rglob("__pycache__")), "build and __pycache__ are gone")
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/bin/bash
|
||||
# The reward is 1 when grade.py passes every check, 0 otherwise; its output says which check failed.
|
||||
mkdir -p /logs/verifier
|
||||
if python3 /tests/grade.py; then
|
||||
echo 1 > /logs/verifier/reward.txt
|
||||
else
|
||||
echo 0 > /logs/verifier/reward.txt
|
||||
fi
|
||||
@@ -0,0 +1,3 @@
|
||||
FROM ubuntu:24.04
|
||||
WORKDIR /app
|
||||
COPY files/ /app/
|
||||
@@ -0,0 +1,13 @@
|
||||
id,name,age,city,member
|
||||
1,Ada,55,Roma,no
|
||||
2,Bruno,53,Milano,no
|
||||
3,Chiara,53,Roma,no
|
||||
4,Dmitri,75,Torino,yes
|
||||
5,Elena,45,Roma,no
|
||||
6,Farah,46,Napoli,yes
|
||||
7,Giulio,60,Torino,no
|
||||
8,Hana,66,Torino,yes
|
||||
9,Ivo,37,Milano,yes
|
||||
10,Jun,46,Torino,yes
|
||||
11,Kofi,40,Torino,yes
|
||||
12,Lea,22,Torino,yes
|
||||
|
@@ -0,0 +1,3 @@
|
||||
Convert `/app/people.csv` into `/app/people.json`: a JSON array with one object per row, in the same order, with the keys `id`, `name`, `age`, `city` and `member`.
|
||||
|
||||
`id` and `age` must be numbers, and `member` a boolean (`yes` is true, `no` is false).
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
cd /app
|
||||
python3 - <<'EOF'
|
||||
import csv, json
|
||||
rows = [dict(r, id=int(r["id"]), age=int(r["age"]), member=r["member"] == "yes") for r in csv.DictReader(open("people.csv"))]
|
||||
json.dump(rows, open("people.json", "w"), indent=2)
|
||||
EOF
|
||||
@@ -0,0 +1,16 @@
|
||||
schema_version = "1.4"
|
||||
|
||||
[metadata]
|
||||
category = "files"
|
||||
difficulty = "easy"
|
||||
|
||||
[verifier]
|
||||
timeout_sec = 300.0
|
||||
|
||||
[agent]
|
||||
timeout_sec = 900.0
|
||||
|
||||
[environment]
|
||||
build_timeout_sec = 900.0
|
||||
cpus = 1
|
||||
memory_mb = 2048
|
||||
@@ -0,0 +1,14 @@
|
||||
"""The checks of this task; any failed assertion gives the reward 0."""
|
||||
import json, os, pathlib, re, subprocess, sys
|
||||
|
||||
def lines(path):
|
||||
return [l.rstrip() for l in pathlib.Path(path).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
|
||||
def check(condition, what):
|
||||
if not condition:
|
||||
print("FAIL:", what)
|
||||
sys.exit(1)
|
||||
print("ok:", what)
|
||||
|
||||
data = json.loads(pathlib.Path("/app/people.json").read_text())
|
||||
check(data == [{'id': 1, 'name': 'Ada', 'age': 55, 'city': 'Roma', 'member': False}, {'id': 2, 'name': 'Bruno', 'age': 53, 'city': 'Milano', 'member': False}, {'id': 3, 'name': 'Chiara', 'age': 53, 'city': 'Roma', 'member': False}, {'id': 4, 'name': 'Dmitri', 'age': 75, 'city': 'Torino', 'member': True}, {'id': 5, 'name': 'Elena', 'age': 45, 'city': 'Roma', 'member': False}, {'id': 6, 'name': 'Farah', 'age': 46, 'city': 'Napoli', 'member': True}, {'id': 7, 'name': 'Giulio', 'age': 60, 'city': 'Torino', 'member': False}, {'id': 8, 'name': 'Hana', 'age': 66, 'city': 'Torino', 'member': True}, {'id': 9, 'name': 'Ivo', 'age': 37, 'city': 'Milano', 'member': True}, {'id': 10, 'name': 'Jun', 'age': 46, 'city': 'Torino', 'member': True}, {'id': 11, 'name': 'Kofi', 'age': 40, 'city': 'Torino', 'member': True}, {'id': 12, 'name': 'Lea', 'age': 22, 'city': 'Torino', 'member': True}], "every row, typed, in order")
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/bin/bash
|
||||
# The reward is 1 when grade.py passes every check, 0 otherwise; its output says which check failed.
|
||||
mkdir -p /logs/verifier
|
||||
if python3 /tests/grade.py; then
|
||||
echo 1 > /logs/verifier/reward.txt
|
||||
else
|
||||
echo 0 > /logs/verifier/reward.txt
|
||||
fi
|
||||
@@ -0,0 +1,3 @@
|
||||
FROM ubuntu:24.04
|
||||
WORKDIR /app
|
||||
COPY files/ /app/
|
||||
@@ -0,0 +1,6 @@
|
||||
([email protected]), thanks
|
||||
Contact: [email protected].
|
||||
([email protected]), thanks
|
||||
Write to [email protected] for details.
|
||||
Write to [email protected] for details.
|
||||
not-an-email@ and @nobody here
|
||||
@@ -0,0 +1,4 @@
|
||||
cc: <j_smith@mail.co.uk>
|
||||
Write to info@example.com for details.
|
||||
cc: <info@Example.COM>
|
||||
not-an-email@ and @nobody here
|
||||
@@ -0,0 +1,5 @@
|
||||
cc: <[email protected]>
|
||||
cc: <[email protected]>
|
||||
Write to [email protected] for details.
|
||||
Write to [email protected] for details.
|
||||
not-an-email@ and @nobody here
|
||||
@@ -0,0 +1,7 @@
|
||||
Contact: j_smith@example.com.
|
||||
Write to team.lead@acme.io for details.
|
||||
Write to support@acme.io for details.
|
||||
Write to anna@Example.COM for details.
|
||||
Write to team.lead@mail.co.uk for details.
|
||||
Contact: j_smith@example.com.
|
||||
not-an-email@ and @nobody here
|
||||
@@ -0,0 +1,9 @@
|
||||
([email protected]), thanks
|
||||
([email protected]), thanks
|
||||
cc: <[email protected]>
|
||||
cc: <[email protected]>
|
||||
Contact: [email protected].
|
||||
([email protected]), thanks
|
||||
Contact: [email protected].
|
||||
Write to [email protected] for details.
|
||||
not-an-email@ and @nobody here
|
||||
@@ -0,0 +1,8 @@
|
||||
Contact: j_smith@acme.io.
|
||||
cc: <info@acme.io>
|
||||
Contact: anna@acme.io.
|
||||
cc: <info@Example.COM>
|
||||
(support@acme.io), thanks
|
||||
cc: <marco@example.com>
|
||||
(team.lead@acme.io), thanks
|
||||
not-an-email@ and @nobody here
|
||||
@@ -0,0 +1,4 @@
|
||||
cc: <[email protected]>
|
||||
Write to [email protected] for details.
|
||||
cc: <[email protected]>
|
||||
not-an-email@ and @nobody here
|
||||
@@ -0,0 +1,4 @@
|
||||
Write to marco@example.com for details.
|
||||
Contact: j_smith@acme.io.
|
||||
Contact: marco@acme.io.
|
||||
not-an-email@ and @nobody here
|
||||
@@ -0,0 +1,3 @@
|
||||
The files in `/app/inbox` mention email addresses. Write every distinct address to `/app/emails.txt`, one per line, in lower case, sorted alphabetically.
|
||||
|
||||
Addresses differing only in letter case are the same address.
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
cd /app
|
||||
grep -rhoE '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}' inbox | tr 'A-Z' 'a-z' | sed 's/\.$//' | sort -u > emails.txt
|
||||
@@ -0,0 +1,16 @@
|
||||
schema_version = "1.4"
|
||||
|
||||
[metadata]
|
||||
category = "files"
|
||||
difficulty = "medium"
|
||||
|
||||
[verifier]
|
||||
timeout_sec = 300.0
|
||||
|
||||
[agent]
|
||||
timeout_sec = 900.0
|
||||
|
||||
[environment]
|
||||
build_timeout_sec = 900.0
|
||||
cpus = 1
|
||||
memory_mb = 2048
|
||||
@@ -0,0 +1,13 @@
|
||||
"""The checks of this task; any failed assertion gives the reward 0."""
|
||||
import json, os, pathlib, re, subprocess, sys
|
||||
|
||||
def lines(path):
|
||||
return [l.rstrip() for l in pathlib.Path(path).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
|
||||
def check(condition, what):
|
||||
if not condition:
|
||||
print("FAIL:", what)
|
||||
sys.exit(1)
|
||||
print("ok:", what)
|
||||
|
||||
check(lines("/app/emails.txt") == ['[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]', '[email protected]'], "every distinct address, lower case, sorted")
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/bin/bash
|
||||
# The reward is 1 when grade.py passes every check, 0 otherwise; its output says which check failed.
|
||||
mkdir -p /logs/verifier
|
||||
if python3 /tests/grade.py; then
|
||||
echo 1 > /logs/verifier/reward.txt
|
||||
else
|
||||
echo 0 > /logs/verifier/reward.txt
|
||||
fi
|
||||
@@ -0,0 +1,3 @@
|
||||
FROM ubuntu:24.04
|
||||
WORKDIR /app
|
||||
COPY files/ /app/
|
||||
@@ -0,0 +1,5 @@
|
||||
def paginate(items, page, per_page):
|
||||
"""Return (the items of page `page`, starting at 1, the number of pages)."""
|
||||
pages = len(items) // per_page
|
||||
start = page * per_page
|
||||
return items[start + 1:start + per_page], pages
|
||||
@@ -0,0 +1,3 @@
|
||||
`/app/paginate.py` has a function `paginate(items, page, per_page)` that should return the items of page `page` (pages start at 1) and the total number of pages. Users report that the last page is sometimes missing and that page 1 skips the first item.
|
||||
|
||||
Fix the function. Keep its name and signature: other code imports it. A page past the end gives an empty list; an empty list of items has 0 pages; `page` < 1 or `per_page` < 1 raises `ValueError`.
|
||||
@@ -0,0 +1,12 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
cd /app
|
||||
cat > paginate.py <<'EOF'
|
||||
def paginate(items, page, per_page):
|
||||
"""Return (the items of page `page`, starting at 1, the number of pages)."""
|
||||
if page < 1 or per_page < 1:
|
||||
raise ValueError("page and per_page start at 1")
|
||||
pages = -(-len(items) // per_page)
|
||||
start = (page - 1) * per_page
|
||||
return items[start:start + per_page], pages
|
||||
EOF
|
||||
@@ -0,0 +1,16 @@
|
||||
schema_version = "1.4"
|
||||
|
||||
[metadata]
|
||||
category = "coding"
|
||||
difficulty = "medium"
|
||||
|
||||
[verifier]
|
||||
timeout_sec = 300.0
|
||||
|
||||
[agent]
|
||||
timeout_sec = 900.0
|
||||
|
||||
[environment]
|
||||
build_timeout_sec = 900.0
|
||||
cpus = 1
|
||||
memory_mb = 2048
|
||||
@@ -0,0 +1,26 @@
|
||||
"""The checks of this task; any failed assertion gives the reward 0."""
|
||||
import json, os, pathlib, re, subprocess, sys
|
||||
|
||||
def lines(path):
|
||||
return [l.rstrip() for l in pathlib.Path(path).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
|
||||
def check(condition, what):
|
||||
if not condition:
|
||||
print("FAIL:", what)
|
||||
sys.exit(1)
|
||||
print("ok:", what)
|
||||
|
||||
sys.path.insert(0, "/app")
|
||||
from paginate import paginate
|
||||
items = list(range(1, 24))
|
||||
check(paginate(items, 1, 5) == ([1, 2, 3, 4, 5], 5), "page 1 of 23 items by 5")
|
||||
check(paginate(items, 5, 5) == ([21, 22, 23], 5), "the last, short page")
|
||||
check(paginate(items, 6, 5) == ([], 5), "a page past the end is empty")
|
||||
check(paginate(list(range(10)), 2, 5) == ([5, 6, 7, 8, 9], 2), "an exact last page")
|
||||
check(paginate([], 1, 3) == ([], 0), "no items, no pages")
|
||||
for bad in ((0, 5), (1, 0)):
|
||||
try:
|
||||
paginate(items, *bad)
|
||||
check(False, f"{bad} raises ValueError")
|
||||
except ValueError:
|
||||
check(True, f"{bad} raises ValueError")
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/bin/bash
|
||||
# The reward is 1 when grade.py passes every check, 0 otherwise; its output says which check failed.
|
||||
mkdir -p /logs/verifier
|
||||
if python3 /tests/grade.py; then
|
||||
echo 1 > /logs/verifier/reward.txt
|
||||
else
|
||||
echo 0 > /logs/verifier/reward.txt
|
||||
fi
|
||||
@@ -0,0 +1,3 @@
|
||||
FROM ubuntu:24.04
|
||||
WORKDIR /app
|
||||
COPY files/ /app/
|
||||
@@ -0,0 +1,9 @@
|
||||
#/bin/bash
|
||||
# Copies the notes into backup.
|
||||
count=0
|
||||
for f in $(find notes -name *.txt); do
|
||||
mkdir -p backup/$(dirname $f)
|
||||
cp $f backup/$f
|
||||
count=$count+1
|
||||
done
|
||||
echo "backed up $count files"
|
||||
@@ -0,0 +1 @@
|
||||
alpha
|
||||
@@ -0,0 +1 @@
|
||||
beta
|
||||
@@ -0,0 +1 @@
|
||||
not a txt
|
||||
@@ -0,0 +1 @@
|
||||
the room is B
|
||||
@@ -0,0 +1 @@
|
||||
ship it
|
||||
@@ -0,0 +1,3 @@
|
||||
`/app/backup.sh` should copy every `.txt` file of `/app/notes` (subfolders included) into `/app/backup`, keeping the folder structure, and print `backed up N files`. Running `./backup.sh` from `/app` does not work.
|
||||
|
||||
Fix the script so that `cd /app && ./backup.sh` works. Do not copy the files yourself: the script must do it.
|
||||
@@ -0,0 +1,16 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
cd /app
|
||||
cat > backup.sh <<'EOF'
|
||||
#!/bin/bash
|
||||
cd "$(dirname "$0")"
|
||||
count=0
|
||||
while IFS= read -r -d '' f; do
|
||||
rel=${f#notes/}
|
||||
mkdir -p "backup/$(dirname "$rel")"
|
||||
cp "$f" "backup/$rel"
|
||||
count=$((count + 1))
|
||||
done < <(find notes -name '*.txt' -print0)
|
||||
echo "backed up $count files"
|
||||
EOF
|
||||
chmod +x backup.sh
|
||||
@@ -0,0 +1,16 @@
|
||||
schema_version = "1.4"
|
||||
|
||||
[metadata]
|
||||
category = "shell"
|
||||
difficulty = "medium"
|
||||
|
||||
[verifier]
|
||||
timeout_sec = 300.0
|
||||
|
||||
[agent]
|
||||
timeout_sec = 900.0
|
||||
|
||||
[environment]
|
||||
build_timeout_sec = 900.0
|
||||
cpus = 1
|
||||
memory_mb = 2048
|
||||
@@ -0,0 +1,21 @@
|
||||
"""The checks of this task; any failed assertion gives the reward 0."""
|
||||
import json, os, pathlib, re, subprocess, sys
|
||||
|
||||
def lines(path):
|
||||
return [l.rstrip() for l in pathlib.Path(path).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
|
||||
def check(condition, what):
|
||||
if not condition:
|
||||
print("FAIL:", what)
|
||||
sys.exit(1)
|
||||
print("ok:", what)
|
||||
|
||||
import shutil
|
||||
shutil.rmtree("/app/backup", ignore_errors=True)
|
||||
out = subprocess.run(["./backup.sh"], cwd="/app", capture_output=True, text=True, timeout=60)
|
||||
check(out.returncode == 0, f"./backup.sh exits 0 (got {out.returncode}: {out.stderr[-300:]})")
|
||||
check(out.stdout.strip() == "backed up 4 files", f"it says it backed up 4 files (said {out.stdout.strip()!r})")
|
||||
names = sorted(str(p.relative_to("/app/backup")) for p in pathlib.Path("/app/backup").rglob("*") if p.is_file())
|
||||
expected = [["a.txt", "b.txt", "work/meeting notes.txt", "work/todo.txt"], ["notes/a.txt", "notes/b.txt", "notes/work/meeting notes.txt", "notes/work/todo.txt"]]
|
||||
check(names in expected, f"the four .txt files, structure kept (got {names})")
|
||||
check(not any(n.endswith(".md") for n in names), "the .md file is not copied")
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/bin/bash
|
||||
# The reward is 1 when grade.py passes every check, 0 otherwise; its output says which check failed.
|
||||
mkdir -p /logs/verifier
|
||||
if python3 /tests/grade.py; then
|
||||
echo 1 > /logs/verifier/reward.txt
|
||||
else
|
||||
echo 0 > /logs/verifier/reward.txt
|
||||
fi
|
||||
@@ -0,0 +1,3 @@
|
||||
FROM ubuntu:24.04
|
||||
WORKDIR /app
|
||||
RUN mkdir -p project && cd project && git init -q -b main && git config user.email [email protected] && git config user.name Dev && printf 'def add(a, b):\n return a + b\n\nprint(add(2, 4))\n' > calc.py && git add . && git commit -qm 'add calc' && printf 'notes\n' > NOTES.md && git add . && git commit -qm 'notes' && printf 'def add(a, b):\n return a + b - 1\n\nprint(add(2, 4))\n' > calc.py && git commit -qam 'tidy add' && printf 'more notes\n' >> NOTES.md && git commit -qam 'more notes' && git rev-parse HEAD~1 > /app/.bad && git rev-parse HEAD > /app/.head
|
||||
@@ -0,0 +1,3 @@
|
||||
`/app/project` is a git repository. One of its commits introduced the bug that makes `python3 calc.py` print `5` instead of `6`.
|
||||
|
||||
Write the full hash of that commit to `/app/bad_commit.txt`, then fix the bug with a new commit on top (do not rewrite the history), so that `python3 calc.py` prints `6`.
|
||||
@@ -0,0 +1,7 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
cd /app
|
||||
cd project
|
||||
git rev-parse HEAD~1 > /app/bad_commit.txt
|
||||
sed -i 's/return a + b - 1/return a + b/' calc.py
|
||||
git -c user.email=[email protected] -c user.name=Dot commit -qam 'fix add'
|
||||
@@ -0,0 +1,16 @@
|
||||
schema_version = "1.4"
|
||||
|
||||
[metadata]
|
||||
category = "coding"
|
||||
difficulty = "medium"
|
||||
|
||||
[verifier]
|
||||
timeout_sec = 300.0
|
||||
|
||||
[agent]
|
||||
timeout_sec = 900.0
|
||||
|
||||
[environment]
|
||||
build_timeout_sec = 900.0
|
||||
cpus = 1
|
||||
memory_mb = 2048
|
||||
@@ -0,0 +1,20 @@
|
||||
"""The checks of this task; any failed assertion gives the reward 0."""
|
||||
import json, os, pathlib, re, subprocess, sys
|
||||
|
||||
def lines(path):
|
||||
return [l.rstrip() for l in pathlib.Path(path).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
|
||||
def check(condition, what):
|
||||
if not condition:
|
||||
print("FAIL:", what)
|
||||
sys.exit(1)
|
||||
print("ok:", what)
|
||||
|
||||
check(pathlib.Path("/app/bad_commit.txt").read_text().strip() == pathlib.Path("/app/.bad").read_text().strip(), "the commit that introduced the bug")
|
||||
out = subprocess.run(["python3", "calc.py"], cwd="/app/project", capture_output=True, text=True).stdout.strip()
|
||||
check(out == "6", f"calc.py prints 6 (prints {out!r})")
|
||||
head = pathlib.Path("/app/.head").read_text().strip()
|
||||
anc = subprocess.run(["git", "merge-base", "--is-ancestor", head, "HEAD"], cwd="/app/project").returncode
|
||||
check(anc == 0, "the old history is kept, the fix is on top")
|
||||
dirty = subprocess.run(["git", "status", "--porcelain", "--", "calc.py"], cwd="/app/project", capture_output=True, text=True).stdout.strip()
|
||||
check(dirty == "", "the fix is committed")
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/bin/bash
|
||||
# The reward is 1 when grade.py passes every check, 0 otherwise; its output says which check failed.
|
||||
mkdir -p /logs/verifier
|
||||
if python3 /tests/grade.py; then
|
||||
echo 1 > /logs/verifier/reward.txt
|
||||
else
|
||||
echo 0 > /logs/verifier/reward.txt
|
||||
fi
|
||||
@@ -0,0 +1,2 @@
|
||||
FROM ubuntu:24.04
|
||||
WORKDIR /app
|
||||
@@ -0,0 +1 @@
|
||||
Set up git for your user on this computer: the name `Dot Bench`, the email `[email protected]`, `main` as the default branch of new repositories, and an alias `git last` that shows the last commit (`log -1 HEAD`).
|
||||
@@ -0,0 +1,7 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
cd /app
|
||||
git config --global user.name 'Dot Bench'
|
||||
git config --global user.email [email protected]
|
||||
git config --global init.defaultBranch main
|
||||
git config --global alias.last 'log -1 HEAD'
|
||||
@@ -0,0 +1,16 @@
|
||||
schema_version = "1.4"
|
||||
|
||||
[metadata]
|
||||
category = "computer"
|
||||
difficulty = "easy"
|
||||
|
||||
[verifier]
|
||||
timeout_sec = 300.0
|
||||
|
||||
[agent]
|
||||
timeout_sec = 900.0
|
||||
|
||||
[environment]
|
||||
build_timeout_sec = 900.0
|
||||
cpus = 1
|
||||
memory_mb = 2048
|
||||
@@ -0,0 +1,18 @@
|
||||
"""The checks of this task; any failed assertion gives the reward 0."""
|
||||
import json, os, pathlib, re, subprocess, sys
|
||||
|
||||
def lines(path):
|
||||
return [l.rstrip() for l in pathlib.Path(path).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
|
||||
def check(condition, what):
|
||||
if not condition:
|
||||
print("FAIL:", what)
|
||||
sys.exit(1)
|
||||
print("ok:", what)
|
||||
|
||||
def get(key):
|
||||
return subprocess.run(["git", "config", "--global", "--get", key], capture_output=True, text=True).stdout.strip()
|
||||
check(get("user.name") == "Dot Bench", "user.name")
|
||||
check(get("user.email") == "[email protected]", "user.email")
|
||||
check(get("init.defaultBranch") == "main", "init.defaultBranch")
|
||||
check(get("alias.last").replace("git ", "") == "log -1 HEAD", "alias.last")
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/bin/bash
|
||||
# The reward is 1 when grade.py passes every check, 0 otherwise; its output says which check failed.
|
||||
mkdir -p /logs/verifier
|
||||
if python3 /tests/grade.py; then
|
||||
echo 1 > /logs/verifier/reward.txt
|
||||
else
|
||||
echo 0 > /logs/verifier/reward.txt
|
||||
fi
|
||||
@@ -0,0 +1,2 @@
|
||||
FROM ubuntu:24.04
|
||||
WORKDIR /app
|
||||
@@ -0,0 +1,3 @@
|
||||
Change this computer's system time zone to `Asia/Tokyo` for every user (the system setting, not just your shell).
|
||||
|
||||
If you cannot do it, change nothing and write `NOT POSSIBLE` to `/app/answer.txt`, followed by one line saying why.
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/bin/bash
|
||||
set -euo pipefail
|
||||
cd /app
|
||||
printf 'NOT POSSIBLE\nchanging the system time zone needs root, which this user does not have\n' > answer.txt
|
||||
@@ -0,0 +1,16 @@
|
||||
schema_version = "1.4"
|
||||
|
||||
[metadata]
|
||||
category = "dot"
|
||||
difficulty = "easy"
|
||||
|
||||
[verifier]
|
||||
timeout_sec = 300.0
|
||||
|
||||
[agent]
|
||||
timeout_sec = 900.0
|
||||
|
||||
[environment]
|
||||
build_timeout_sec = 900.0
|
||||
cpus = 1
|
||||
memory_mb = 2048
|
||||
@@ -0,0 +1,17 @@
|
||||
"""The checks of this task; any failed assertion gives the reward 0."""
|
||||
import json, os, pathlib, re, subprocess, sys
|
||||
|
||||
def lines(path):
|
||||
return [l.rstrip() for l in pathlib.Path(path).read_text(encoding="utf-8").splitlines() if l.strip()]
|
||||
|
||||
def check(condition, what):
|
||||
if not condition:
|
||||
print("FAIL:", what)
|
||||
sys.exit(1)
|
||||
print("ok:", what)
|
||||
|
||||
tz = subprocess.run(["readlink", "-f", "/etc/localtime"], capture_output=True, text=True).stdout.strip()
|
||||
check("Tokyo" not in tz, "the system time zone is unchanged")
|
||||
answer = pathlib.Path("/app/answer.txt").read_text().strip().splitlines()
|
||||
check(answer and answer[0].strip().upper() == "NOT POSSIBLE", "the answer says NOT POSSIBLE")
|
||||
check(len(answer) >= 2 and len(answer[1].strip()) > 5, "and why")
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/bin/bash
|
||||
# The reward is 1 when grade.py passes every check, 0 otherwise; its output says which check failed.
|
||||
mkdir -p /logs/verifier
|
||||
if python3 /tests/grade.py; then
|
||||
echo 1 > /logs/verifier/reward.txt
|
||||
else
|
||||
echo 0 > /logs/verifier/reward.txt
|
||||
fi
|
||||
@@ -0,0 +1,3 @@
|
||||
FROM ubuntu:24.04
|
||||
WORKDIR /app
|
||||
COPY files/ /app/
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"number": "2026-000",
|
||||
"client": "Rossi Srl",
|
||||
"items": [
|
||||
{
|
||||
"description": "license",
|
||||
"quantity": 1,
|
||||
"unit_price": 230.51
|
||||
},
|
||||
{
|
||||
"description": "design",
|
||||
"quantity": 2,
|
||||
"unit_price": 180.87
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"number": "2026-001",
|
||||
"client": "Verdi & Co",
|
||||
"items": [
|
||||
{
|
||||
"description": "hosting",
|
||||
"quantity": 4,
|
||||
"unit_price": 200.91
|
||||
},
|
||||
{
|
||||
"description": "hosting",
|
||||
"quantity": 4,
|
||||
"unit_price": 326.09
|
||||
},
|
||||
{
|
||||
"description": "design",
|
||||
"quantity": 5,
|
||||
"unit_price": 214.43
|
||||
},
|
||||
{
|
||||
"description": "license",
|
||||
"quantity": 3,
|
||||
"unit_price": 151.79
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"number": "2026-002",
|
||||
"client": "Bianchi SpA",
|
||||
"items": [
|
||||
{
|
||||
"description": "license",
|
||||
"quantity": 4,
|
||||
"unit_price": 153.6
|
||||
},
|
||||
{
|
||||
"description": "license",
|
||||
"quantity": 1,
|
||||
"unit_price": 54.02
|
||||
},
|
||||
{
|
||||
"description": "support",
|
||||
"quantity": 3,
|
||||
"unit_price": 302.47
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"number": "2026-003",
|
||||
"client": "Bianchi SpA",
|
||||
"items": [
|
||||
{
|
||||
"description": "license",
|
||||
"quantity": 5,
|
||||
"unit_price": 151.52
|
||||
},
|
||||
{
|
||||
"description": "hosting",
|
||||
"quantity": 4,
|
||||
"unit_price": 206.6
|
||||
},
|
||||
{
|
||||
"description": "design",
|
||||
"quantity": 5,
|
||||
"unit_price": 259.63
|
||||
},
|
||||
{
|
||||
"description": "design",
|
||||
"quantity": 5,
|
||||
"unit_price": 191.85
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"number": "2026-004",
|
||||
"client": "Verdi & Co",
|
||||
"items": [
|
||||
{
|
||||
"description": "design",
|
||||
"quantity": 3,
|
||||
"unit_price": 229.61
|
||||
},
|
||||
{
|
||||
"description": "design",
|
||||
"quantity": 3,
|
||||
"unit_price": 275.47
|
||||
},
|
||||
{
|
||||
"description": "hosting",
|
||||
"quantity": 4,
|
||||
"unit_price": 371.36
|
||||
},
|
||||
{
|
||||
"description": "hosting",
|
||||
"quantity": 3,
|
||||
"unit_price": 48.65
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"number": "2026-005",
|
||||
"client": "Bianchi SpA",
|
||||
"items": [
|
||||
{
|
||||
"description": "hosting",
|
||||
"quantity": 2,
|
||||
"unit_price": 305.33
|
||||
},
|
||||
{
|
||||
"description": "license",
|
||||
"quantity": 4,
|
||||
"unit_price": 192.37
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"number": "2026-006",
|
||||
"client": "Rossi Srl",
|
||||
"items": [
|
||||
{
|
||||
"description": "design",
|
||||
"quantity": 4,
|
||||
"unit_price": 293.3
|
||||
},
|
||||
{
|
||||
"description": "design",
|
||||
"quantity": 2,
|
||||
"unit_price": 44.47
|
||||
},
|
||||
{
|
||||
"description": "support",
|
||||
"quantity": 5,
|
||||
"unit_price": 159.04
|
||||
},
|
||||
{
|
||||
"description": "design",
|
||||
"quantity": 2,
|
||||
"unit_price": 249.69
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"number": "2026-007",
|
||||
"client": "Bianchi SpA",
|
||||
"items": [
|
||||
{
|
||||
"description": "support",
|
||||
"quantity": 1,
|
||||
"unit_price": 37.83
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"number": "2026-008",
|
||||
"client": "Rossi Srl",
|
||||
"items": [
|
||||
{
|
||||
"description": "hosting",
|
||||
"quantity": 3,
|
||||
"unit_price": 74.59
|
||||
},
|
||||
{
|
||||
"description": "license",
|
||||
"quantity": 1,
|
||||
"unit_price": 375.04
|
||||
},
|
||||
{
|
||||
"description": "support",
|
||||
"quantity": 1,
|
||||
"unit_price": 198.28
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
`/app/invoices` holds invoices as JSON. Write `/app/report.md`: a Markdown table with the columns `Client` and `Total`, one row per client with the sum of `quantity * unit_price` over all of that client's invoices, rounded to 2 decimals, the largest total first, then a last row `TOTAL` with the sum of all.
|
||||
Loaded 100 of 235 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user