mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-07 06:16:06 -10:00
docs: add eval guide, authoring skills, and public history hub (#13535)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its evaluations test both the Runner and complete product workflows. > - The guides and run histories are in separate places. > - The shared Evalbook viewer can make the test boundary unclear. > - This pull request names the two families and adds a guide, authoring skills, and a public hub. > - Contributors can choose the correct test and inspect its history. ## Linked Issues or Issue Description **Issue type** Missing documentation. **Where is the issue?** Runner and Product E2E evaluation guides, case-authoring procedures, and public result navigation. **What's wrong?** There is no single entry point. A report format can be mistaken for an execution boundary. There are no dedicated case-authoring skills for these two families. **Suggested fix** Add a guide and three skills. Link both existing histories from a public hub. Keep existing campaign URLs and grading unchanged. ## What Changed - Add `doc/evals.md` and links from existing guides. - Add the `paperclip-evals`, `add-runner-eval`, and `add-product-e2e-eval` skills. Install copies in `~/paperclipai/.agents/skills`. - Add a static hub builder that reads the existing public history feeds. - Show a dated snapshot for each family. Label partial campaigns and preserve measurement dates across report refreshes. - Document publication and refresh commands for https://pages.paperclip.ing/evals/. ## Verification - Seven Python summary tests pass: `python3 -m unittest discover -s scripts/evals-hub -p 'test_*.py'`. Run these checks directly; this PR does not modify package scripts. - All three skills pass the skill-creator `quick_validate.py` check with `/usr/bin/python3`. - Build tested with saved history fixtures and the live public feeds. - Desktop and mobile browser checks pass. The mobile page has no horizontal overflow. - Published https://pages.paperclip.ing/evals/. Browser check: HTTP 200, no page errors, all eight links return HTTP 200, no mobile overflow. - Independent skill exercises found the existing Notion-decline case and a direct Runner permission-denial case. Roster validation with an explicit run ID passes. - Missing refresh measurement date: regression fails before the fix and passes after it. - `git diff --check` passes. - No paid evals were run for this documentation and reporting change. The preceding head passed typecheck, build, server/workspace tests, runner verification, browser E2E, and the canary dry run. Checks for the latest commit are pending. Local repository-wide typecheck, test, and build were not repeated because no product code changed. ## Risks The hub is a dated static snapshot. It can lag behind the linked histories until an operator refreshes it. A changed history schema stops the build. Existing archives and grades are not modified. The published guide link is pinned to the reviewed commit so branch deletion cannot break it. Later builds can use master. ## Model Used OpenAI gpt-6-astra for implementation and review. OpenAI gpt-5.6-luna for documentation and independent skill checks. Both used repository tools and code execution. Context window sizes are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (latest commit pending) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups (preceding head was 5/5; latest commit pending) - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <[email protected]>
This commit is contained in:
DottaandPaperclip authored and GitHub committed
1 parent
4577d10029
commit
18989a9e73
10 files changed
+698
No files matched your search
@@ -0,0 +1,84 @@
|
||||
---
|
||||
name: add-product-e2e-eval
|
||||
description: Add or extend a Paperclip full-stack runner E2E workflow, fixture, matcher, or report evidence path for local or Daytona execution.
|
||||
---
|
||||
|
||||
# Add a Product E2E Eval
|
||||
|
||||
Use this skill for **Product E2E Evals**: real Chromium, Paperclip server,
|
||||
database, runner, provider, and optionally Daytona. Everyday Workflows are in
|
||||
this family even when their packaged results are imported into Evalbook. Runner
|
||||
protocol cases against the mock control plane belong in
|
||||
[add-runner-eval](../add-runner-eval/SKILL.md).
|
||||
|
||||
Locate the repository using `PAPERCLIP_ROOT` when supplied, or
|
||||
`git rev-parse --show-toplevel` from a checkout. From outside Git, inspect
|
||||
workspace roots such as `~/paperclipai/paperclip`; verify the selected root
|
||||
contains `tests/runner-e2e` and `packages/paperclip-runner`. Run commands from
|
||||
that repository root. The copied skill may live outside the checkout.
|
||||
Read `doc/evals.md`, then the authoritative
|
||||
`tests/runner-e2e/README.md`, `FIXTURES.md`, `SECURITY.md`, and
|
||||
`EVERYDAY-WORKFLOWS.md` for the selected area. Inspect the nearest existing
|
||||
catalog entry, case, harness flow, matcher, evidence writer, and report test
|
||||
before changing anything. Keep the user journey on production browser/API
|
||||
surfaces; do not add private runner hooks or direct fixture database writes.
|
||||
|
||||
Define one bounded workflow with a clear user outcome, durable state
|
||||
assertions, and independent evidence. Declare profile, environment, expected
|
||||
provider turns, timeout, cleanup, screenshots, artifact checks, and billing
|
||||
scope. Keep credentials and secrets out of catalog data, screenshots, fixture
|
||||
metadata, logs, and tracked files. Daytona images must be immutable digest
|
||||
references. Use the existing result validator, failure classifier, screenshot
|
||||
policy, and Product E2E report pipeline rather than duplicating them.
|
||||
|
||||
Register the workflow in `tests/runner-e2e/catalog.ts` and the relevant
|
||||
`fixture-registry.ts` paths. Everyday cases and actions live in
|
||||
`everyday-cases.ts` and `everyday-flow.ts`; match the existing suite's structure.
|
||||
Wire its
|
||||
profile/environment/case IDs through existing selector and matcher tables, and
|
||||
add report/catalog coverage tests where the surrounding suite does so. Calibrate
|
||||
new grading assertions with a valid outcome and a plausible wrong outcome;
|
||||
missing evidence must not produce a pass. Confirm
|
||||
discovery before running it:
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner -- --list --suite <suite-name>
|
||||
pnpm test:e2e:runner -- --list --suite everyday-workflows
|
||||
```
|
||||
|
||||
`--all` intentionally excludes the manual `everyday-workflows` suite and other
|
||||
explicit-only cells. Use the exact suite or execution ID for those. The Product
|
||||
E2E generator is `pnpm test:e2e:runner:report`; an Everyday Workflows result may
|
||||
also be imported into Evalbook with its canonical importer, but it remains a
|
||||
Product E2E run and should use its packaged dashboard/report first.
|
||||
|
||||
Run credential-free checks first:
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner:typecheck
|
||||
pnpm test:e2e:runner:unit
|
||||
pnpm test:e2e:runner -- --list
|
||||
```
|
||||
|
||||
For an authorized live check, use the smallest explicit local `--id` selector.
|
||||
Select Daytona only with the configured immutable image and credentials. A
|
||||
full `--all` campaign is paid and is for the governed workflow. Verify results
|
||||
from the packaged attempt evidence and dashboard; passing model-authored tests
|
||||
cannot override the independent oracle.
|
||||
|
||||
Classify a completed wrong workflow as product or model/provider behavior as
|
||||
the assertions warrant. Startup, transport, and timeout symptoms require
|
||||
evidence-based attribution: they may indicate a Paperclip/Runner product bug,
|
||||
provider behavior, or infrastructure. Preserve the observed failure and cause
|
||||
separately, keep the existing machine grade/classifier unchanged, and retain
|
||||
partial attempts, retries, source SHA, catalog/definition digest, model/profile,
|
||||
environment, grader version, timing, tokens, runtime estimates, and cost
|
||||
coverage. Do not claim qualification from a partial/manual selection.
|
||||
|
||||
Update `README.md`, `FIXTURES.md`, `SECURITY.md`, or
|
||||
`EVERYDAY-WORKFLOWS.md` when their authoritative contract changes, and link
|
||||
from `doc/evals.md`. Do not recreate Evalbook HTML or publish raw trusted
|
||||
traces, videos, archives, databases, workspaces, credentials, SVG, provider
|
||||
session IDs, or hidden reasoning. Public sanitized fixture conversation, marked
|
||||
screenshots, and allowlisted structured evidence are expected when the existing
|
||||
publisher permits them.
|
||||
@@ -0,0 +1,72 @@
|
||||
---
|
||||
name: add-runner-eval
|
||||
description: Add or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.
|
||||
---
|
||||
|
||||
# Add a Runner Eval
|
||||
|
||||
Use this skill for the **Runner Evals** family: a real Runner/provider session
|
||||
against a seeded mock control plane. Product browser/server/database/Daytona
|
||||
coverage belongs in [add-product-e2e-eval](../add-product-e2e-eval/SKILL.md).
|
||||
|
||||
Locate the Paperclip checkout using `PAPERCLIP_ROOT` when supplied, or
|
||||
`git rev-parse --show-toplevel` from a checkout. From outside Git, inspect the
|
||||
workspace roots (for example `~/paperclipai/paperclip`) and verify that the
|
||||
selected root contains `packages/paperclip-runner` and `tests/runner-e2e`.
|
||||
Locate `paperclip-evals` using `PAPERCLIP_EVALS_ROOT` or a discovered sibling;
|
||||
a worktree's parent directory need not contain that repository. Read
|
||||
`doc/evals.md` and `packages/paperclip-runner/docs/runner-protocol-live-evals.md`,
|
||||
then inspect the nearest existing case, roster, schema, and report test before
|
||||
editing. Definitions and authored cases belong in the sibling
|
||||
`paperclip-evals/evals/paperclip-runner`; Runner integration, aggregation,
|
||||
viewer, and publication behavior belongs in `packages/paperclip-runner`.
|
||||
Keep the control-plane boundary explicit in names and documentation.
|
||||
|
||||
The sibling eval README is the concrete map: cases live under `cases/`,
|
||||
company fixtures under `fixtures/`, runtime/model settings under `configs/`,
|
||||
selections under `rosters/`, and maintained campaign membership under
|
||||
`campaigns/live-direct-full.json`. Update inventory/coverage mappings when the
|
||||
program requires them; a new file alone does not join the maintained campaign.
|
||||
From the Evals repository root, adapt these provider-free checks to the case
|
||||
and roster you changed. Run the reliability-plan validator only when that
|
||||
separate plan changes:
|
||||
|
||||
```sh
|
||||
python3 evals/paperclip-runner/tools/eval_program.py validate \
|
||||
--case evals/paperclip-runner/cases/get-task-context.json \
|
||||
--config evals/paperclip-runner/configs/live-codex-pinned.json
|
||||
python3 evals/paperclip-runner/tools/run_live_roster.py validate \
|
||||
--roster evals/paperclip-runner/rosters/live-mini.json --run-id validate-new-case
|
||||
python3 evals/paperclip-runner/tools/run_live_campaign.py validate \
|
||||
--campaign evals/paperclip-runner/campaigns/live-direct-full.json
|
||||
python3 evals/paperclip-runner/tools/reliability_campaign.py validate \
|
||||
--plan evals/paperclip-runner/campaigns/paperclip-runner-reliability.json
|
||||
```
|
||||
|
||||
Use nearby positive and negative grader cases/fixtures to calibrate the new
|
||||
assertion, including malformed or missing evidence where the grader must fail
|
||||
closed. Preserve the existing machine disposition and grade; product,
|
||||
model/provider, grading, and infrastructure labels are analytical annotations,
|
||||
not instructions to rewrite classifiers.
|
||||
|
||||
Author one bounded case with a deterministic semantic assertion and an
|
||||
inspectable result. Declare its expected operation, state effect, provider
|
||||
lane/profile, timeout and retry policy, and any required evidence. Do not grade
|
||||
hidden reasoning, infer success from a provider terminal message, or invent
|
||||
conversation/tool evidence. Public output follows the reviewed projection:
|
||||
sanitized fixture conversation and allowlisted tool outcomes may be published;
|
||||
raw trusted artifacts, credentials, secrets, private references, and hidden
|
||||
reasoning may not.
|
||||
|
||||
Validate without provider calls first using the commands above and the relevant
|
||||
report/render validation documented in the Runner docs. When a live run is
|
||||
authorized, pin the Paperclip commit and exact 40-character
|
||||
`paperclip-evals` commit, select the smallest useful roster, and retain the
|
||||
complete provenance and cost record.
|
||||
|
||||
Update authoritative detailed docs when the contract or command changes, then
|
||||
link from `doc/evals.md` rather than duplicating the Runner runbook. Keep public
|
||||
reports immutable and use the reviewed projection; sanitized fixture
|
||||
conversation and allowlisted tool outcomes may be public, while credentials,
|
||||
secrets, private references, raw trusted payloads, and hidden reasoning must not
|
||||
be exposed.
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
name: paperclip-evals
|
||||
description: Choose, inspect, validate, and report Paperclip Runner or Product E2E evaluations while preserving evidence, provenance, cost, and failure classification.
|
||||
---
|
||||
|
||||
# Paperclip evals
|
||||
|
||||
Use this skill when a request concerns Paperclip evaluation selection,
|
||||
interpretation, evidence, history, or a live run. Read `doc/evals.md` in the
|
||||
Paperclip repository first. It defines the two families and their boundaries.
|
||||
|
||||
## Discover the repository
|
||||
|
||||
Do not assume the skill's installed location is inside a checkout. Locate the
|
||||
repo explicitly with `git rev-parse --show-toplevel` from the current directory,
|
||||
or inspect likely workspace roots and select the checkout containing
|
||||
`package.json`, `tests/runner-e2e`, and `packages/paperclip-runner`. Locate the
|
||||
private sibling `paperclip-evals` only when a Runner Eval needs its definitions;
|
||||
use an explicit `PAPERCLIP_EVALS_ROOT` or a discovered sibling checkout. Never
|
||||
invent a relative path from this copied skill into the repository.
|
||||
|
||||
## Route the request
|
||||
|
||||
Choose **Runner Evals** for real runner/provider protocol behavior against the
|
||||
mock control plane. Authoritative details are in
|
||||
`packages/paperclip-runner/docs/runner-protocol-live-evals.md` and the sibling
|
||||
`paperclip-evals/evals/paperclip-runner` definitions.
|
||||
|
||||
Choose **Product E2E Evals** for real browser/server/database/runner/provider
|
||||
workflows, including local and Daytona environments. Read
|
||||
`tests/runner-e2e/README.md`, then `FIXTURES.md`, `SECURITY.md`, or
|
||||
`EVERYDAY-WORKFLOWS.md` as relevant. Everyday Workflows remain Product E2E
|
||||
even when imported into Evalbook. “Headless” is a browser mode, not a family.
|
||||
|
||||
## Work safely
|
||||
|
||||
Start with read-only catalog inspection and credential-free validation. For
|
||||
Product E2E use `pnpm test:e2e:runner:typecheck`,
|
||||
`pnpm test:e2e:runner:unit`, and `pnpm test:e2e:runner -- --list`; run one
|
||||
explicit cell only when the user has authorized a live/paid run and the needed
|
||||
credentials and immutable Daytona image are configured. For Runner Evals use
|
||||
the pinned eval revision and the documented workflow/CLI. Never use a partial
|
||||
selector as evidence of full coverage.
|
||||
|
||||
Keep source revisions, definition/catalog fingerprints, model/profile,
|
||||
environment, selected cells, retries, timing, usage/cost coverage, and grader
|
||||
version attached to every interpretation. Preserve partial attempts and
|
||||
classify failures as product, model/provider behavior, grading/evidence, or
|
||||
infrastructure from the observed failure and supported cause. A usable
|
||||
completed behavior failure is not infrastructure; missing provider/profile,
|
||||
transport, startup, or evidence requires examining the evidence before choosing
|
||||
the cause.
|
||||
|
||||
Use the existing family generator and viewer. Public projections may contain
|
||||
sanitized fixture conversation and allowlisted tool outcomes/evidence; follow
|
||||
the family's projection and publisher checks. Do not expose raw trusted
|
||||
artifacts, credentials, secrets, private data, provider session IDs, or hidden
|
||||
reasoning. A refresh from retained evidence has zero provider calls and remains
|
||||
the original measurement with a new presentation. Link the public histories
|
||||
and hub from `doc/evals.md` when reporting results.
|
||||
|
||||
For adding a case or fixture, use the narrower [add-runner-eval](../add-runner-eval/SKILL.md)
|
||||
or [add-product-e2e-eval](../add-product-e2e-eval/SKILL.md) skill.
|
||||
+214
@@ -0,0 +1,214 @@
|
||||
# Paperclip evaluation guide
|
||||
|
||||
Paperclip has two live eval families with different questions, owners, and
|
||||
evidence. Choose the family before selecting a model, profile, or case.
|
||||
|
||||
- **Runner Evals:** real Runner/provider behavior against a seeded mock control
|
||||
plane. Definitions live in `paperclip-evals/evals/paperclip-runner`; see the
|
||||
[direct live protocol evals](../packages/paperclip-runner/docs/runner-protocol-live-evals.md).
|
||||
- **Product E2E Evals:** real browser, Paperclip server, database, Runner,
|
||||
provider, and (where selected) Daytona, using an isolated instance and
|
||||
grading oracle. See [`tests/runner-e2e`](../tests/runner-e2e/README.md) and
|
||||
[Everyday Workflows](../tests/runner-e2e/EVERYDAY-WORKFLOWS.md).
|
||||
|
||||
Runner Evals answer whether a real runner/provider can perform a bounded
|
||||
protocol operation against the expected control-plane contract. Product E2E
|
||||
Evals answer whether a person can complete a product workflow through the real
|
||||
Paperclip surfaces and whether the resulting artifact and state are usable.
|
||||
The names describe the system under test; “headless” is an execution option,
|
||||
not an eval category.
|
||||
|
||||
## Selecting a family
|
||||
|
||||
Use **Runner Evals** for a runner protocol, adapter, transport, native session,
|
||||
tool grant, or one-turn provider qualification question. The workflow checks
|
||||
out an exact `paperclip-evals` revision, builds the Runner and viewer, runs a
|
||||
live roster, and renders the canonical Evalbook report. The control plane is a
|
||||
seeded test authority, so a passing result does not prove browser UX, production
|
||||
server behavior, database persistence, Daytona behavior, or a real third-party
|
||||
mutation.
|
||||
|
||||
Use **Product E2E Evals** for browser interaction, issue/task lifecycle,
|
||||
approval and clarification UI, project/repository selection, persistence over a
|
||||
controller restart, artifact delivery, billing/evidence behavior, or runner
|
||||
continuity in local or Daytona environments. The harness creates a fresh
|
||||
Paperclip instance per cell and uses public APIs and the production browser
|
||||
surface. The suite's [Everyday Workflows](../tests/runner-e2e/EVERYDAY-WORKFLOWS.md)
|
||||
are Product E2E even when their results are imported into Evalbook.
|
||||
|
||||
Do not combine a partial Runner campaign and a partial Product E2E campaign into
|
||||
one score. A campaign is comparable when its definition/grader, model/profile,
|
||||
environment, and contract match. The evaluated Paperclip revision may
|
||||
intentionally differ for a before/after fix comparison; record it as a
|
||||
comparison axis.
|
||||
|
||||
## Ownership and codepaths
|
||||
|
||||
Runner Evals are owned by the Runner/evals maintainers. Definitions, rosters,
|
||||
case prompts, and the report program live in the sibling private repository
|
||||
`paperclipai/paperclip-evals`; Runner integration, viewer, aggregation, and
|
||||
publication code live under `packages/paperclip-runner` and the
|
||||
`runner-protocol-live-evals.yml` workflow. The public-facing report uses the
|
||||
same Evalbook renderer and Runner Lab viewer as the trusted report after
|
||||
sanitization.
|
||||
|
||||
Product E2E Evals are owned by the runner E2E maintainers. The catalog and
|
||||
harness are under `tests/runner-e2e`; the package scripts are `test:e2e:runner`,
|
||||
`test:e2e:runner:unit`, `test:e2e:runner:typecheck`, and
|
||||
`test:e2e:runner:report`. `README.md`, `FIXTURES.md`, `SECURITY.md`, and
|
||||
`EVERYDAY-WORKFLOWS.md` are the detailed sources of truth. The harness starts
|
||||
the server and embedded database, creates the company/agent/task through the
|
||||
real APIs, drives Chromium, and invokes the selected local or Daytona runner.
|
||||
|
||||
## Validation ladder
|
||||
|
||||
Start with credential-free checks and a catalog listing. For Product E2E:
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner:typecheck
|
||||
pnpm test:e2e:runner:unit
|
||||
pnpm test:e2e:runner -- --list
|
||||
```
|
||||
|
||||
For one explicitly selected local cell, configure only the credentials named
|
||||
by that cell in `.env.runner-e2e.local`, then run a narrow ID:
|
||||
|
||||
```sh
|
||||
pnpm test:e2e:runner -- --id core-compatibility.runner-codex.local.message-marker
|
||||
```
|
||||
|
||||
Use the selectors documented in the [runner E2E README](../tests/runner-e2e/README.md)
|
||||
for a suite, profile, case, group, or environment. Daytona needs the immutable
|
||||
image digest and `DAYTONA_API_KEY`; follow the README and fixture security guide.
|
||||
`--all` excludes manual suites such as `everyday-workflows`. Select that suite
|
||||
explicitly; use a narrow selector while developing a fixture.
|
||||
|
||||
For Runner Evals, the narrowest useful local validation is the report program's
|
||||
help/validation path and the deterministic Runner checks documented in
|
||||
[`runner-workflow-evals.md`](../packages/paperclip-runner/docs/runner-workflow-evals.md).
|
||||
Hosted direct live runs must use the default-branch workflow, an exact 40
|
||||
character `evals_sha`, an explicitly selected roster (or the maintained
|
||||
enabled `all` campaign), and the protected paid environment. The complete
|
||||
hosted command is intentionally kept in the workflow and
|
||||
[direct live protocol guide](../packages/paperclip-runner/docs/runner-protocol-live-evals.md).
|
||||
Live provider runs can spend money; use the existing workflow authorization and
|
||||
the user's stated scope when selecting them.
|
||||
|
||||
## Failure taxonomy
|
||||
|
||||
Record the primary failure class and preserve the evidence that supports it.
|
||||
|
||||
- **Product failure:** evidence shows Paperclip or Runner behavior violates the
|
||||
authored case or a hard invariant, such as wrong task state, missing approval
|
||||
gate, lost persistence, bad artifact, or incorrect protocol operation.
|
||||
- **Model/provider behavior failure:** the provider turn completed with usable
|
||||
evidence but the model gave the wrong answer, ignored an interaction, failed
|
||||
to complete the authored operation, or violated a semantic assertion. It is
|
||||
scored as behavior, not silently retried as infrastructure.
|
||||
- **Grading/evidence failure:** the case or matcher cannot establish its claim,
|
||||
a required recording/screenshot/result is malformed, or the report contract
|
||||
is invalid. Fix the harness or grader before interpreting the score.
|
||||
- **Infrastructure failure:** the evidence points to provider/profile
|
||||
unavailability, transport admission failure, service startup failure, a
|
||||
missing credential/image, or inability to produce usable evidence. Startup,
|
||||
transport, and timeout symptoms can instead be product defects when evidence
|
||||
implicates Paperclip or Runner; classify from the observed failure and
|
||||
supported cause, rather than the symptom name alone. Preserve the artifact.
|
||||
|
||||
Missing usage or price data means unknown, not free. Keep provider-reported
|
||||
costs separate from estimates, and include retry costs when available.
|
||||
Latency, cleanup, billing coverage, and unpriced usage are dimensions of the
|
||||
result and should remain visible alongside the primary class. A timeout after
|
||||
successful product state reads can be a product behavior failure; a failed
|
||||
server-health read may be infrastructure, but inspect its cause. Use the
|
||||
family-specific classifier and read the attempt evidence before changing an
|
||||
analytical label.
|
||||
|
||||
## Evidence, provenance, and history
|
||||
|
||||
An Evalbook report is a presentation of immutable attempt records, not the
|
||||
source of truth. Keep the campaign ID, Paperclip commit, `paperclip-evals`
|
||||
commit, catalog/roster or definition fingerprint, model/profile, environment,
|
||||
grader version, selected cells, retries, and provider/runtime usage with the
|
||||
report. Public projections follow each family's reviewed allowlist and may
|
||||
include sanitized fixture conversation, named tool outcomes, screenshots, and
|
||||
structured evidence intended for public history. Credentials, secrets, private
|
||||
data, raw unredacted records, and hidden reasoning stay out of public
|
||||
projections.
|
||||
|
||||
Distinguish a complete campaign from a partial campaign. A narrow selector,
|
||||
manual diagnostic, missing cell, or infrastructure retry can be useful evidence
|
||||
without being a qualification run. History should retain both, with explicit
|
||||
coverage and completeness, while trend and latest-green views compare only
|
||||
compatible complete campaigns. Refreshing an existing report from retained
|
||||
evidence has zero provider calls and is a new presentation of the old
|
||||
measurement, not a new model run.
|
||||
|
||||
Existing public histories are available at
|
||||
[Runner protocol history](https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html)
|
||||
and [Runner Product E2E history](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/).
|
||||
The consolidated eval hub is at
|
||||
[pages.paperclip.ing/evals](https://pages.paperclip.ing/evals/).
|
||||
|
||||
For a repeatable workflow, use the matching skill: [paperclip-evals](../.agents/skills/paperclip-evals/SKILL.md),
|
||||
[add-runner-eval](../.agents/skills/add-runner-eval/SKILL.md), or
|
||||
[add-product-e2e-eval](../.agents/skills/add-product-e2e-eval/SKILL.md).
|
||||
|
||||
## Install the authoring skills
|
||||
|
||||
The reviewable sources live in this repository's `.agents/skills`. For a
|
||||
multi-repository workspace, install the three skills at
|
||||
`~/paperclipai/.agents/skills` (not `~/paperclipai/skills`). From the Paperclip
|
||||
checkout, run:
|
||||
|
||||
```sh
|
||||
for skill in paperclip-evals add-runner-eval add-product-e2e-eval; do
|
||||
install -d "$HOME/paperclipai/.agents/skills/$skill"
|
||||
install -m 644 ".agents/skills/$skill/SKILL.md" \
|
||||
"$HOME/paperclipai/.agents/skills/$skill/SKILL.md"
|
||||
done
|
||||
```
|
||||
|
||||
This replaces only the three named skill entrypoints. Run it again after
|
||||
updating their tracked sources. Each skill locates the repository independently
|
||||
of its installation directory.
|
||||
|
||||
## Maintain the public hub
|
||||
|
||||
The hub is a static directory with two links to the existing history systems.
|
||||
It displays a dated snapshot, not a live scoreboard. It does not run models,
|
||||
create another result archive, or change the existing campaign URLs.
|
||||
|
||||
Build from the public history feeds and check its summary logic:
|
||||
|
||||
```sh
|
||||
python3 -m unittest discover -s scripts/evals-hub -p 'test_*.py'
|
||||
python3 scripts/evals-hub/build.py --output .paperclip/evals-hub
|
||||
```
|
||||
|
||||
The hub checks need Python 3 and do not call model providers.
|
||||
|
||||
For offline checks, pass `--history-dir <directory>` containing
|
||||
`runner-protocol-evals-history.json` and `runner-e2e-history.json`.
|
||||
For a pre-merge preview, pass `--docs-ref <branch-or-sha>` to link the guide
|
||||
at that revision. The default guide link uses `master`.
|
||||
|
||||
Publish with the [Paperclip page helper](../.agents/skills/paperclip-page/SKILL.md)
|
||||
and the configured page-uploader credentials. Use Bash 4 or newer; macOS's
|
||||
system Bash 3 cannot run this helper. On macOS with Homebrew Bash installed,
|
||||
put `$(brew --prefix bash)/bin` first in `PATH` before these commands:
|
||||
|
||||
```sh
|
||||
export PAPERCLIP_PAGE_BUCKET=pages.paperclip.ing
|
||||
export PAPERCLIP_PAGE_BASE_URL=https://pages.paperclip.ing
|
||||
export AWS_REGION=us-east-1
|
||||
bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals --dry-run
|
||||
bash .agents/skills/paperclip-page/scripts/publish.sh .paperclip/evals-hub --slug evals
|
||||
```
|
||||
|
||||
For later refreshes, rebuild in the same output directory and publish with
|
||||
`--update`. Keep its ignored `.paperclip-page/state.json` ownership record;
|
||||
without that record, the helper will refuse to overwrite an existing prefix.
|
||||
Verify the public page and its links after publication. This manual refresh
|
||||
does not add a scheduled workflow. Preserve the measurement date when choosing
|
||||
a newer rendering of the same campaign.
|
||||
@@ -1,5 +1,10 @@
|
||||
# Direct live Runner protocol evals
|
||||
|
||||
For choosing between Runner Evals and full-stack Product E2E Evals, and for
|
||||
shared guidance on provenance, history, cost, and failure classification, see
|
||||
the [Paperclip evaluation guide](../../../doc/evals.md). This document remains
|
||||
the authoritative Runner Evals live protocol runbook.
|
||||
|
||||
## One Evalbook presentation
|
||||
|
||||
Every new report uses the canonical Evalbook grid and the existing Runner Lab
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
__pycache__/
|
||||
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build the public eval directory from the two existing public history feeds."""
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import html
|
||||
import json
|
||||
from pathlib import Path
|
||||
from string import Template
|
||||
from urllib.parse import quote, urlsplit
|
||||
from urllib.request import urlopen
|
||||
|
||||
ORIGIN = "https://d1p6rlowie26tp.cloudfront.net"
|
||||
SYSTEMS = {
|
||||
"runner": ("runner-protocol-evals", "paperclip.runner-protocol-eval.history/v1"),
|
||||
"product": ("runner-e2e", "paperclip.runner-e2e.history/v1"),
|
||||
}
|
||||
|
||||
|
||||
def latest(history, schema):
|
||||
if history.get("schema") != schema:
|
||||
raise ValueError("Unsupported history schema")
|
||||
campaign_id = history.get("latestCampaignId")
|
||||
if not campaign_id:
|
||||
raise ValueError("History has no latest campaign")
|
||||
candidates = [c for c in history["campaigns"] if
|
||||
c["campaignId"] == campaign_id or
|
||||
c.get("reportRevision", {}).get("sourceCampaignId") == campaign_id]
|
||||
if not candidates:
|
||||
raise ValueError("Latest campaign is absent from history")
|
||||
# Report refreshes replace presentation, never the measurement date.
|
||||
return max(candidates, key=lambda c: c.get("reportRevision", {}).get("renderedAt", c["generatedAt"]))
|
||||
|
||||
|
||||
def public_report(url, prefix):
|
||||
parsed = urlsplit(url)
|
||||
if (parsed.scheme != "https" or parsed.netloc != urlsplit(ORIGIN).netloc
|
||||
or not parsed.path.startswith(f"/{prefix}/campaigns/")
|
||||
or parsed.query or parsed.fragment):
|
||||
raise ValueError("Unexpected public report URL")
|
||||
return html.escape(url, quote=True)
|
||||
|
||||
|
||||
def display_date(value):
|
||||
date = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
if date.tzinfo is None:
|
||||
raise ValueError("History date must include a timezone")
|
||||
return date.astimezone(timezone.utc).strftime("%d %b %Y · %H:%M UTC")
|
||||
|
||||
|
||||
def summarize(history, kind):
|
||||
prefix, schema = SYSTEMS[kind]
|
||||
campaign = latest(history, schema)
|
||||
totals = campaign["totals"] if kind == "runner" else campaign
|
||||
passed, selected = totals["passed"], totals["selected"]
|
||||
if any(type(n) is not int for n in (passed, selected)) or not 0 <= passed <= selected:
|
||||
raise ValueError("Invalid campaign counts")
|
||||
if type(campaign.get("complete")) is not bool:
|
||||
raise ValueError("Missing campaign coverage flag")
|
||||
revision = campaign.get("reportRevision", {})
|
||||
if revision and not isinstance(revision.get("sourceGeneratedAt"), str):
|
||||
raise ValueError("Report refresh has no reliable measurement date")
|
||||
measured = revision.get("sourceGeneratedAt", campaign["generatedAt"])
|
||||
return {
|
||||
"url": public_report(campaign["publicUrl"], prefix),
|
||||
"date": html.escape(display_date(measured)),
|
||||
"counts": f"{passed} / {selected} selected cases passed",
|
||||
"coverage": "Complete campaign" if campaign["complete"] else "Partial campaign",
|
||||
"id": html.escape(revision.get("sourceCampaignId", campaign["campaignId"])),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--history-dir", type=Path, help="Use saved public history JSON files for offline verification")
|
||||
parser.add_argument("--docs-ref", default="master", help="Git ref containing doc/evals.md")
|
||||
args = parser.parse_args()
|
||||
values = {}
|
||||
for kind, (prefix, _) in SYSTEMS.items():
|
||||
if args.history_dir:
|
||||
history = json.loads((args.history_dir / f"{prefix}-history.json").read_text())
|
||||
else:
|
||||
with urlopen(f"{ORIGIN}/{prefix}/history.json", timeout=30) as response:
|
||||
history = json.load(response)
|
||||
values.update({f"{kind}_{k}": v for k, v in summarize(history, kind).items()})
|
||||
values["refreshed"] = display_date(datetime.now(timezone.utc).isoformat())
|
||||
values["guide_url"] = "https://github.com/paperclipai/paperclip/blob/" + quote(args.docs_ref, safe="/") + "/doc/evals.md"
|
||||
template = Template(Path(__file__).with_name("template.html").read_text())
|
||||
rendered = template.substitute(values)
|
||||
args.output.mkdir(parents=True, exist_ok=True)
|
||||
(args.output / "index.html").write_text(rendered)
|
||||
print(args.output / "index.html")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,75 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<meta name="description" content="Paperclip evaluation results, run histories, and guides for Runner Evals and Product E2E Evals.">
|
||||
<title>Paperclip Evals</title>
|
||||
<style>
|
||||
:root { color-scheme: light dark; --bg:#f8f8f5; --card:#fff; --text:#20221e; --muted:#60645c; --line:#d9ddd3; --accent:#306340; --tint:#edf3ec; }
|
||||
* { box-sizing:border-box; }
|
||||
body { margin:0; background:var(--bg); color:var(--text); font:16px/1.6 -apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif; }
|
||||
a { color:inherit; text-underline-offset:4px; }
|
||||
a:focus-visible { outline:3px solid var(--accent); outline-offset:5px; }
|
||||
.bar { border-bottom:1px solid var(--line); }
|
||||
.bar div { max-width:1160px; margin:auto; padding:20px 28px; display:flex; justify-content:space-between; align-items:center; gap:20px; }
|
||||
.brand { font-weight:750; letter-spacing:-.04em; font-size:21px; text-decoration:none; }
|
||||
.bar span, .eyebrow { color:var(--muted); font-size:12px; text-transform:uppercase; letter-spacing:.13em; }
|
||||
main { max-width:1160px; margin:64px auto; padding:0 28px; }
|
||||
h1 { font-size:clamp(38px,6vw,62px); line-height:1.05; letter-spacing:-.055em; margin:14px 0 20px; font-weight:650; }
|
||||
.intro { max-width:690px; font-size:19px; color:var(--muted); margin-bottom:36px; }
|
||||
.grid { display:grid; grid-template-columns:repeat(2,minmax(0,1fr)); gap:24px; }
|
||||
article { background:var(--card); border:1px solid var(--line); border-radius:12px; padding:30px; display:flex; flex-direction:column; }
|
||||
.number { font:13px/1.4 ui-monospace,monospace; color:var(--accent); }
|
||||
h2 { margin:12px 0 12px; letter-spacing:-.035em; font-size:29px; line-height:1.2; }
|
||||
article p { color:var(--muted); margin:0 0 22px; }
|
||||
.boundary { font-size:14px; padding:14px 16px; background:var(--tint); border-radius:6px; margin-bottom:24px; }
|
||||
.latest { border-top:1px solid var(--line); padding-top:20px; margin-top:auto; }
|
||||
.latest .label { color:var(--muted); font-size:12px; text-transform:uppercase; letter-spacing:.08em; }
|
||||
.counts { font-size:19px; font-weight:600; margin:7px 0; }
|
||||
.meta { font-size:13px; color:var(--muted); }
|
||||
.id { overflow-wrap:anywhere; margin-top:4px; font:11px/1.6 ui-monospace,monospace; color:var(--muted); }
|
||||
.links { display:flex; flex-wrap:wrap; align-items:center; gap:18px; margin:24px 0 18px; }
|
||||
.button { display:inline-block; background:var(--text); color:var(--card); padding:10px 15px; border-radius:6px; text-decoration:none; font-weight:600; font-size:14px; }
|
||||
.guide { font-size:13px; }
|
||||
.note { font-size:13px; color:var(--muted); margin:19px 0 38px; }
|
||||
.start { display:flex; gap:28px; justify-content:space-between; align-items:center; padding:25px 0; border-top:1px solid var(--line); border-bottom:1px solid var(--line); }
|
||||
.start h2 { font-size:22px; margin:0 0 6px; }
|
||||
.start p { margin:0; color:var(--muted); font-size:14px; max-width:650px; }
|
||||
footer { margin-top:25px; color:var(--muted); font-size:12px; }
|
||||
@media (prefers-color-scheme:dark) { :root { --bg:#151714; --card:#1d201b; --text:#f2f4ed; --muted:#aeb6a6; --line:#363d31; --accent:#a0cda5; --tint:#273024; } }
|
||||
@media (max-width:700px) { main { margin:36px auto; padding:0 20px; } .grid { grid-template-columns:1fr; } article { padding:24px; } .start { align-items:flex-start; flex-direction:column; gap:18px; } .bar div { padding:16px 20px; } .bar span { font-size:10px; } }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<header class="bar"><div><a class="brand" href="https://paperclip.ing">Paperclip</a><span>Evaluation directory</span></div></header>
|
||||
<main>
|
||||
<div class="eyebrow">Evidence for better agents</div>
|
||||
<h1>Paperclip Evals</h1>
|
||||
<p class="intro">Two ways to test Paperclip. Follow each program from a test case to its evidence, results, and run history.</p>
|
||||
<div class="grid">
|
||||
<article>
|
||||
<div class="number">01 / RUNNER</div>
|
||||
<h2>Runner Evals</h2>
|
||||
<p>Does the runner use tools correctly, respect permissions, and preserve its execution contract?</p>
|
||||
<div class="boundary">Real runner + model provider<br>Controlled mock Paperclip control plane</div>
|
||||
<div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$runner_counts</div><div class="meta">$runner_date · $runner_coverage</div><div class="id">$runner_id</div></div>
|
||||
<div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-protocol-evals/index.html">Browse run history →</a><a href="$runner_url">Open report</a></div>
|
||||
<a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/packages/paperclip-runner/docs/runner-protocol-live-evals.md">Runner operations guide</a>
|
||||
</article>
|
||||
<article>
|
||||
<div class="number">02 / PRODUCT</div>
|
||||
<h2>Product E2E Evals</h2>
|
||||
<p>Can a user complete work through Paperclip, from the first request to a usable result?</p>
|
||||
<div class="boundary">Real browser + server + database + runner + model provider<br>Local and Daytona environments, as selected</div>
|
||||
<div class="latest"><div class="label">Latest recorded measurement</div><div class="counts">$product_counts</div><div class="meta">$product_date · $product_coverage</div><div class="id">$product_id</div></div>
|
||||
<div class="links"><a class="button" href="https://d1p6rlowie26tp.cloudfront.net/runner-e2e/">Browse run history →</a><a href="$product_url">Open report</a></div>
|
||||
<a class="guide" href="https://github.com/paperclipai/paperclip/blob/master/tests/runner-e2e/README.md">Product E2E operations guide</a>
|
||||
</article>
|
||||
</div>
|
||||
<p class="note">Snapshot refreshed $refreshed. History pages may contain newer runs. A partial campaign covers only its selected cases. These programs measure different boundaries; their pass rates are not combined.</p>
|
||||
<section class="start"><div><h2>Choose a test. Make the result inspectable.</h2><p>The eval guide explains which program to use, how to add a case, and how to distinguish a product defect from model behavior, a grading error, or a setup failure. Evalbook is a report framework used to present evidence; it is not an execution boundary.</p></div><a class="button" href="$guide_url">Read the eval guide →</a></section>
|
||||
<footer>Existing run archives remain the source of truth. Report refreshes do not count as new model runs.</footer>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,82 @@
|
||||
import copy
|
||||
import unittest
|
||||
|
||||
from build import SYSTEMS, summarize
|
||||
|
||||
|
||||
class HistorySummaryTests(unittest.TestCase):
|
||||
def history(self, kind="runner"):
|
||||
prefix, schema = SYSTEMS[kind]
|
||||
campaign = {
|
||||
"campaignId": "run-1", "generatedAt": "2026-09-01T12:00:00Z",
|
||||
"publicUrl": f"https://d1p6rlowie26tp.cloudfront.net/{prefix}/campaigns/run-1/",
|
||||
"complete": False,
|
||||
}
|
||||
counts = {"selected": 3, "passed": 2}
|
||||
campaign.update({"totals": counts} if kind == "runner" else counts)
|
||||
return {"schema": schema, "latestCampaignId": "run-1", "campaigns": [campaign]}
|
||||
|
||||
def test_both_families_keep_partial_coverage(self):
|
||||
for kind in SYSTEMS:
|
||||
with self.subTest(kind=kind):
|
||||
result = summarize(self.history(kind), kind)
|
||||
self.assertEqual(result["coverage"], "Partial campaign")
|
||||
self.assertEqual(result["counts"], "2 / 3 selected cases passed")
|
||||
|
||||
def test_refresh_changes_report_but_not_measurement_date(self):
|
||||
history = self.history()
|
||||
refreshed = copy.deepcopy(history["campaigns"][0])
|
||||
refreshed.update(campaignId="run-1-refresh", generatedAt="2026-09-03T12:00:00Z")
|
||||
refreshed["publicUrl"] += "refresh/"
|
||||
refreshed["reportRevision"] = {
|
||||
"sourceCampaignId": "run-1", "sourceGeneratedAt": "2026-09-01T12:00:00Z",
|
||||
"renderedAt": "2026-09-03T12:00:00Z",
|
||||
}
|
||||
history["campaigns"].append(refreshed)
|
||||
result = summarize(history, "runner")
|
||||
self.assertEqual(result["id"], "run-1")
|
||||
self.assertIn("01 Sep 2026", result["date"])
|
||||
self.assertTrue(result["url"].endswith("/refresh/"))
|
||||
|
||||
def test_missing_latest_does_not_fall_back_to_an_arbitrary_run(self):
|
||||
history = self.history()
|
||||
history["latestCampaignId"] = "absent"
|
||||
with self.assertRaises(ValueError):
|
||||
summarize(history, "runner")
|
||||
|
||||
def test_refresh_without_measurement_date_is_rejected(self):
|
||||
history = self.history()
|
||||
campaign = history["campaigns"][0]
|
||||
campaign["campaignId"] = "run-1-refresh"
|
||||
campaign["reportRevision"] = {
|
||||
"sourceCampaignId": "run-1", "renderedAt": "2026-09-03T12:00:00Z",
|
||||
}
|
||||
with self.assertRaisesRegex(ValueError, "measurement date"):
|
||||
summarize(history, "runner")
|
||||
|
||||
def test_unknown_schema_and_missing_coverage_are_rejected(self):
|
||||
for field in ("schema", "complete"):
|
||||
history = self.history()
|
||||
if field == "schema":
|
||||
history["schema"] = "future-schema"
|
||||
else:
|
||||
del history["campaigns"][0]["complete"]
|
||||
with self.subTest(field=field), self.assertRaises(ValueError):
|
||||
summarize(history, "runner")
|
||||
|
||||
def test_bad_counts_cannot_be_published(self):
|
||||
history = self.history()
|
||||
history["campaigns"][0]["totals"]["passed"] = 4
|
||||
with self.assertRaises(ValueError):
|
||||
summarize(history, "runner")
|
||||
|
||||
def test_external_report_urls_are_rejected(self):
|
||||
history = self.history()
|
||||
for url in ("javascript:alert(1)", "https://example.com/", "https://d1p6rlowie26tp.cloudfront.net/other/"):
|
||||
history["campaigns"][0]["publicUrl"] = url
|
||||
with self.subTest(url=url), self.assertRaises(ValueError):
|
||||
summarize(history, "runner")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,5 +1,11 @@
|
||||
# Paid runner full-stack E2E
|
||||
|
||||
For family selection, ownership, provenance, history, and failure taxonomy,
|
||||
see the [Paperclip evaluation guide](../../doc/evals.md). This README is the
|
||||
authoritative runbook for Product E2E runner cells; the separate Runner Evals
|
||||
protocol guide lives at
|
||||
`packages/paperclip-runner/docs/runner-protocol-live-evals.md`.
|
||||
|
||||
This is the billable browser acceptance campaign system for Paperclip runner
|
||||
profiles. It is deliberately separate from `tests/e2e`: every independently
|
||||
scheduled execution gets
|
||||
|
||||
Reference in new issue
Block a user