pydantic/pydantic-ai
1
0
Fork 0
mirror of https://github.com/pydantic/pydantic-ai.git synced 2026-10-11 21:59:51 +00:00
Code Issues Packages Projects Releases Wiki Activity

Make the harness agent instructions work from the monorepo root (#9658)

Browse Source
This commit is contained in:
David SF
2026-10-02 11:30:43 -05:00
committed by GitHub
parent 69b4105c76
commit 829a1dabab
28 changed files with 185 additions and 1060 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/pydantic_ai_harness/.agents/agents/docs-parity-reviewer.md → .agents/skills/docs-parity-reviewer/SKILL.md
+13 -11
View File
@@ -1,22 +1,23 @@
---
name: docs-parity-reviewer
description: Use as the final documentation gate before a capability PR merges. Verifies that a user-facing change keeps the capability README and its unified-docs page in sync with each other and with the code, that every snippet is runnable, and that links follow repo convention. Reports gaps; does not edit.
tools: Read, Grep, Glob, Bash
description: Use as the final documentation gate before a `pydantic-ai-harness` capability PR merges. Verifies that a user-facing change keeps the capability README under `src/pydantic_ai_harness/pydantic_ai_harness/` and its `docs/harness/` page in sync with each other and with the code, that every snippet is runnable, and that links follow repo convention. Reports gaps; does not edit. Skip it for changes that touch no harness capability or its docs.
context: fork
model: sonnet
disallowed-tools: Edit, Write, NotebookEdit
---
You are the documentation parity gate for `pydantic-ai-harness`. Every released
capability ships two docs that must stay in sync with the code and with each
other:
- **README** -- `pydantic_ai_harness/<capability>/README.md` (or
`pydantic_ai_harness/experimental/acp/README.md` for ACP). Serves GitHub and
- **README** -- `src/pydantic_ai_harness/pydantic_ai_harness/<capability>/README.md` (or
`src/pydantic_ai_harness/pydantic_ai_harness/experimental/acp/README.md` for ACP). Serves GitHub and
PyPI. Keeps absolute links and its badges.
- **Unified doc** -- flat at `docs/<capability>.md`. Renders on the docs site
- **Unified doc** -- flat at `docs/harness/<capability>.md`. Renders on the docs site
(`https://pydantic.dev/docs/ai/harness/`). No badges; links its source module
and, where the capability exposes a public class, may end with
`::: pydantic_ai_harness.<Class>` autodoc blocks. The sidebar is a flat list --
no `capabilities/` or `experimental/` subdirectories.
`::: pydantic_ai_harness.<Class>` autodoc blocks. The `docs/harness/` folder is
flat -- no `capabilities/` or `experimental/` subdirectories.
Both are hand-maintained. A change to one that is not reflected in the other is
the failure mode you exist to catch.
@@ -25,7 +26,8 @@ the failure mode you exist to catch.
The diff or description of a capability change (the touched capability, and what
its user-facing behavior now is). If you are not told which capability changed,
infer it from the changed files under `pydantic_ai_harness/`.
infer it from the files the current branch changes under
`src/pydantic_ai_harness/pydantic_ai_harness/` and `docs/harness/`.
## Checks
@@ -36,7 +38,7 @@ problem as a finding (blocking / warning / nit) with a concrete fix.
class, constructor params, defaults, tool names, extras, safety semantics)
and only one of README / unified doc reflects it, that is blocking. A doc
describing behavior the code no longer has is also blocking.
2. **Snippets parse and run.** Run `uv run pytest tests/test_doc_snippets.py`;
2. **Snippets parse and run.** Run `uv run pytest tests/harness/test_doc_snippets.py`;
this checks parsing and harness imports only. Execute every changed
deterministic snippet unchanged. For snippets that need credentials or a
live service, verify the complete runnable wrapper and require a fake-backed
@@ -64,7 +66,7 @@ problem as a finding (blocking / warning / nit) with a concrete fix.
docstring, not a hand-written table.
6. **Safety caveats preserved.** Where the source carries access, sandbox, or
command-control limits (Shell, CodeMode, FileSystem), both docs state them.
7. **Writing style.** Both follow `AGENTS.md` "Writing style": no em-dashes (use
7. **Writing style.** Both follow `src/pydantic_ai_harness/AGENTS.md` "Writing style": no em-dashes (use
`--`), no hype, plain ASCII punctuation.
8. **Purpose-first lead.** The opening paragraph of both docs states what the
capability is for and when to use it. An internal hook or class name
@@ -78,7 +80,7 @@ problem as a finding (blocking / warning / nit) with a concrete fix.
`HarnessExperimentalWarning` block or "removed in any release" wording. ACP
is the only page that keeps an `!!! warning "Experimental"`.
If a released capability has a README but no `docs/` page (or vice versa), that
If a released capability has a README but no `docs/harness/` page (or vice versa), that
missing file is a blocking finding.
## Output
.claude/skills/docs-parity-reviewer
Symlink
+1
View File
@@ -0,0 +1 @@
../../.agents/skills/docs-parity-reviewer
.gitignore
+4
View File
@@ -47,6 +47,9 @@ AGENTS.override.md
!.agents/skills/pushing-commits-to-the-repo/
.agents/skills/pushing-commits-to-the-repo/*
!.agents/skills/pushing-commits-to-the-repo/SKILL.md
!.agents/skills/docs-parity-reviewer/
.agents/skills/docs-parity-reviewer/*
!.agents/skills/docs-parity-reviewer/SKILL.md
.claude/*
!.claude/plans/
!.claude/skills/
@@ -57,6 +60,7 @@ AGENTS.override.md
!.claude/skills/i-have-adhd
!.claude/skills/poweruser-feature-audit
!.claude/skills/pushing-commits-to-the-repo
!.claude/skills/docs-parity-reviewer
!.claude/skills/address-feedback/
.claude/skills/address-feedback/*
!.claude/skills/address-feedback/SKILL.md
AGENTS.md
+3
View File
@@ -140,6 +140,7 @@ When generating or reviewing code anywhere in this repo, always read [agent_docs
Additionally, always read the directory-specific instructions when working in those directories:
- [docs/AGENTS.md](docs/AGENTS.md)
- [docs/harness/AGENTS.md](docs/harness/AGENTS.md)
- [docs/realtime/AGENTS.md](docs/realtime/AGENTS.md)
- [pydantic_ai_slim/pydantic_ai/AGENTS.md](pydantic_ai_slim/pydantic_ai/AGENTS.md)
- [pydantic_ai_slim/pydantic_ai/capabilities/AGENTS.md](pydantic_ai_slim/pydantic_ai/capabilities/AGENTS.md)
@@ -151,4 +152,6 @@ Additionally, always read the directory-specific instructions when working in th
- [pydantic_ai_slim/pydantic_ai/realtime/AGENTS.md](pydantic_ai_slim/pydantic_ai/realtime/AGENTS.md)
- [pydantic_ai_slim/pydantic_ai/toolsets/AGENTS.md](pydantic_ai_slim/pydantic_ai/toolsets/AGENTS.md)
- [pydantic_ai_slim/pydantic_ai/ui/AGENTS.md](pydantic_ai_slim/pydantic_ai/ui/AGENTS.md)
- [src/pydantic_ai_harness/AGENTS.md](src/pydantic_ai_harness/AGENTS.md)
- [tests/AGENTS.md](tests/AGENTS.md)
- [tests/harness/AGENTS.md](tests/harness/AGENTS.md)
agent_docs/documentation.md
+1 -1
View File
@@ -25,7 +25,7 @@
- Put user-facing features where users naturally look for them, not only in API reference docstrings.
- Use current frontier models in reader-facing examples. Verify the latest supported identifiers instead of copying static examples from this guidance.
- Use Markdown headings for real document sections. Register new published pages in `docs/navigation.yml`.
- Link to rendered Pydantic AI Harness documentation when it exists. Use the Harness repository only when no published page covers the capability.
- Link to the Pydantic AI Harness page under `docs/harness/` when one exists. Link to the source under `src/pydantic_ai_harness/` only when no page covers the capability.
## Docstrings
docs/AGENTS.md
+1 -1
View File
@@ -33,4 +33,4 @@ The docs index and repository README tell the same story on two surfaces. Keep t
- `README.md` uses absolute documentation links, `###` sections instead of tabs, and plain one-line `#` comments instead of annotations.
- Mirrored code examples remain code-identical; only comments, annotations, link forms, and fence attributes may differ.
- README snippets that cannot run in the documentation test environment are excluded by `tests/test_examples.py`, not by fence attributes, so README fences remain compatible with GitHub rendering.
- When the shared tagline or Harness framing changes, check the Harness repository's `docs/index.md` and `README.md` too.
- When the shared tagline or Harness framing changes, check `docs/harness/index.md` and `src/pydantic_ai_harness/README.md` too.
docs/harness/AGENTS.md
+15
View File
@@ -0,0 +1,15 @@
# `docs/harness/` guidelines
Rules for the Pydantic AI Harness pages. They add to [`docs/AGENTS.md`](../AGENTS.md). Where these rules or
`docs-conventions.md` differ from `docs/AGENTS.md`, such as the version-promise blockquote, they apply under
`docs/harness/`.
Read [`docs-conventions.md`](../../src/pydantic_ai_harness/agent_docs/docs-conventions.md) before you add, rename, or
edit a page.
- Each capability page has a twin: the `README.md` in the capability's package under
`src/pydantic_ai_harness/pydantic_ai_harness/`. Update both for every user-facing change.
- Keep this directory flat. Do not add `capabilities/` or `experimental/` subdirectories.
- Follow "Writing style" in [`src/pydantic_ai_harness/AGENTS.md`](../../src/pydantic_ai_harness/AGENTS.md).
- Run the `docs-parity-reviewer` skill on a capability change as the last documentation check before merge. Treat
its blocking findings as merge blockers.
docs/harness/CLAUDE.md
Symlink
+1
View File
@@ -0,0 +1 @@
AGENTS.md
src/pydantic_ai_harness/.agents/settings.json
-10
View File
@@ -1,10 +0,0 @@
{
"permissions": {
"allow": [
"Bash(gh pr list:*)",
"Bash(gh issue list:*)",
"Bash(gh issue view:*)",
"Bash(gh pr view:*)"
]
}
}
src/pydantic_ai_harness/.agents/skills/adopt-pr/SKILL.md
-178
View File
@@ -1,178 +0,0 @@
---
name: adopt-pr
description: Bootstrap branch-context on an existing PR. Writes issue-brief.md from the linked issue(s) + current PR state, and backfills pr-decisions.md with decision-bearing entries from already-resolved review threads. Use when picking up a PR mid-flight (yours or someone else's) without prior local context.
user-invocable: true
allowed-tools: Bash(git:*), Bash(gh:*), Bash(jq:*), Bash(date:*), Bash(ls:*), Bash(cat:*), Bash(cp:*), Bash(.claude/skills/branch-context/*), Read, Write, Edit, Glob, Grep, AskUserQuestion
---
# Adopt PR -- bootstrap branch-context on an existing PR
Populate `issue-brief.md` and `pr-decisions.md` for a worktree whose PR already
exists and has history.
## When to use
- You checked out an existing PR and the branch-context files are empty
templates (or absent).
- You are picking up someone else's PR.
- Your own PR predates the branch-context setup and you want to backfill.
Do not use this for fresh work with no PR yet -- write the brief from the linked
issue directly.
## Startup
Bootstrapping presumes the linked issue describes a real problem. If you have
not confirmed that, skim the issue before adopting -- authorship (a bot,
contributor, or automated sweep) is not proof the problem is real.
1. Read `CLAUDE.md` for project context.
2. Instantiate the branch-context instances if they do not exist:
```bash
cd "$(git rev-parse --show-toplevel)"
D=.claude/skills/branch-context
[ -f "$D/issue-brief.md" ] || cp "$D/issue-brief.template.md" "$D/issue-brief.md"
[ -f "$D/pr-decisions.md" ] || cp "$D/pr-decisions.template.md" "$D/pr-decisions.md"
```
If `issue-brief.md` already has populated `issues:` frontmatter, ask via
`AskUserQuestion` whether to overwrite the brief, re-seed decisions, both, or
neither, before proceeding.
## Step 1 -- Resolve the PR
Parse `$ARGUMENTS`:
- **PR number/URL** -> use it directly.
- **Empty** -> detect from the current branch:
```bash
PR_NUMBER=$(gh pr view --json number -q .number 2>/dev/null)
```
If detection fails, ask via `AskUserQuestion`.
Fetch PR metadata:
```bash
gh pr view "$PR_NUMBER" --json number,title,url,state,headRefName,body,closingIssuesReferences,createdAt,author
```
## Step 2 -- Resolve linked issues
Two sources, in order:
1. `closingIssuesReferences` from the PR metadata (canonical -- set via the PR
sidebar or `Fixes #N` keywords).
2. Fallback: grep the PR body for `(?:Fixes|Closes|Resolves|Relates to) #\d+` if
(1) is empty.
For each linked issue:
```bash
gh issue view "<N>" --json number,url,title,state,body,labels,comments
```
If there is no linked issue and the body has no problem description worth
linking, proceed with `issues: []` (free-text problem) -- ask the user whether to
continue or abort.
## Step 3 -- Study the existing diff
Understand the code before synthesizing the brief:
```bash
git fetch origin main 2>/dev/null || git fetch origin master
MERGE_BASE=$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD origin/master)
git diff --stat "$MERGE_BASE"..HEAD
git log --oneline "$MERGE_BASE"..HEAD
```
Map the changes: which files/modules are touched, which tests exist on the
branch (existing tests are implied success criteria), which public symbols
changed, any notable new abstractions.
## Step 4 -- Write `issue-brief.md`
Use the schema in `issue-brief.template.md`. Adaptations for adoption:
- `related_pr`: the PR URL (not `TBD` -- the PR already exists).
- `branch`: `git rev-parse --abbrev-ref HEAD`.
- **Success criteria** -- derive from the issue text and from tests already on
the branch (each existing test is a de facto criterion; cross-reference them
in the table).
- **Affected surface** -- extract from the diff, not from a planning pass.
- **Constraints** -- include anything reviewers have already emphasized in
comments (e.g. "must preserve backwards-compat" from a maintainer reply).
Write to `.claude/skills/branch-context/issue-brief.md`, overwriting the
template.
## Step 5 -- Backfill `pr-decisions.md` from resolved threads
Fetch resolved review threads via the GraphQL API:
```bash
read -r OWNER REPO < <(gh repo view --json owner,name -q '.owner.login + " " + .name')
gh api graphql -f query='
query($owner:String!,$repo:String!,$pr:Int!){
repository(owner:$owner,name:$repo){
pullRequest(number:$pr){
reviewThreads(first:100){ nodes{
isResolved
comments(first:50){ nodes{ author{login} bodyText url path } }
}}
}
}
}' -F owner="$OWNER" -F repo="$REPO" -F pr="$PR_NUMBER" \
--jq '.data.repository.pullRequest.reviewThreads.nodes | map(select(.isResolved))'
```
For each resolved thread, read the full conversation, then classify:
- **Decision-bearing** -- the thread debated two or more options, or a reviewer
flagged a concern and the author adjusted the code (e.g. "use kwargs over
positional", "renamed the public method", "moved the helper to a different
module").
- **Noise** -- typo fixes, "nit: missing docstring", "good catch thanks",
resolved without a code change. Skip these.
For each decision-bearing thread, append an entry (Source is the root comment's
`url`):
```bash
.claude/skills/branch-context/append-pr-decision.sh \
--title "thread: <short title>" \
--decision "<one-line summary of what was decided>" \
--why "<one-line reason, quoting reviewer or author if concise>" \
--source "<thread URL>" \
--iter "-"
```
Decision budget: aim for <=10 entries. If more resolved threads than that seem
worth logging, you are probably over-including noise -- re-apply the filter.
## Step 6 -- Seed an "adoption" meta-entry
Append one final entry marking the boundary between backfilled decisions
(everything above) and live-logged ones (everything below):
```bash
.claude/skills/branch-context/append-pr-decision.sh \
--title "adopted PR #<N> at <DATE>" \
--decision "Branch-context bootstrapped from existing PR + issue(s). Decisions above are backfilled from resolved threads." \
--why "PR predates the branch-context setup" \
--source "<PR URL>" \
--iter "-"
```
## Step 7 -- Report
Print a concise summary:
```
Adopted PR #<N> -- "<title>"
Issues linked: #<A>, #<B>
Resolved threads seeded: <count> decisions
Unresolved threads (for later triage): <count>
```
## Rules
- Do not open a new PR (it already exists). Do not commit anything to the branch
during adoption -- only read, classify, and write the two branch-context files.
- Do not re-classify unresolved threads during adoption -- just report the count.
- Do not log every resolved thread -- only decision-bearing ones. The decisions
log is meant to reward reading; diluting it with noise defeats the point.
- Every backfilled entry must have a thread URL in `Source:`. If you cannot find
one, you are over-inferring -- skip it.
- If the existing brief/decisions files are already populated (not templates),
the user's Startup confirmation governs whether to overwrite.
src/pydantic_ai_harness/.agents/skills/branch-context/SKILL.md
-167
View File
@@ -1,167 +0,0 @@
---
name: branch-context
description: Branch-local durable PR state -- issue brief, decisions log, and session handoffs. Read the brief, decisions, and latest handoff at session start; append decisions as you work; write a handoff only on user request or explicit session turnover.
allowed-tools: Read, Write, Edit, Bash
---
# Branch Context
This directory is the home for durable PR/branch state that outlives a single
session. Three surfaces:
| File / dir | Role | Lifetime |
|------------|------|----------|
| `issue-brief.md` | Synthesis of the issue(s) this branch addresses | Rewritten only when adopting or re-syncing a branch |
| `pr-decisions.md` | Append-only log of non-obvious PR-shaping decisions | Append forever; supersede, never edit |
| `handoffs/` + `handoffs-index.md` | Append-only session handoffs for the next agent | Never overwrite a handoff file; index points at the latest |
The instances (`issue-brief.md`, `pr-decisions.md`, `handoffs-index.md`,
`handoffs/`) are per-branch and git-ignored. The scaffolding (this file, the
scripts, the `*.template.md` files) is committed. Instantiate the instances from
the templates, or via `/adopt-pr` when a PR already exists.
## Session defaults
**On start, before coding:**
1. Confirm the branch matches the brief's `branch:` field
(`git rev-parse --abbrev-ref HEAD`).
2. Read `issue-brief.md`, `pr-decisions.md`, and `handoffs-index.md`.
3. Read the latest handoff -- the last entry in `handoffs-index.md` points at
its file under `handoffs/`. That is what the previous session left for you.
If the index has no entries, there is no handoff yet; start from the brief.
4. If the brief is still the unfilled template, populate it first: run
`/adopt-pr` when a PR already exists, or write it from the linked issue when
starting fresh.
**While working -- persist without being asked:**
- A non-obvious decision (picking path A over B, a plan deviation, an ambiguous
thread resolution) goes into `pr-decisions.md` via `append-pr-decision.sh` in
the same turn you make it, not "later".
- Disk in this directory is the continuity channel, not chat history. Do not
rely on the conversation surviving a context clear or a new session.
- Do not write a handoff because the context "feels full" -- that misjudges and
primes an early stop. Write a handoff only when the user asks, or when session
turnover is already decided (the user is clearing, switching tools, or ending
the sitting).
## When to write each surface
### `issue-brief.md`
Rewritten only when adopting a branch (`/adopt-pr`) or re-syncing after new
issue activity. Do not freestyle-edit it mid-session.
### `pr-decisions.md`
Append whenever you make a decision the issue did not already spell out:
```bash
.claude/skills/branch-context/append-pr-decision.sh \
--title "<short title>" \
--decision "<one-line decision>" \
--why "<one-line why>" \
--source "<source url -- mandatory>" \
[--iter N] [--supersedes "<earlier title>"]
```
Named flags are preferred (they resist arg-order mistakes). Positional order is
`title decision why source [iter] [supersedes]` -- do not pass `iter` as the
second argument.
Entry shape (the script writes this):
```
## YYYY-MM-DD · <short title> · iter <N or "-">
- Decision: <one line>
- Why: <one line>
- Source: <link -- mandatory>
- Supersedes: <earlier title, if applicable>
```
When the decision restates a modal claim (always / never / only if / must /
unless), quote that clause verbatim instead of paraphrasing -- "always X unless
Y" compressed to "always X" is a different instruction.
### Handoffs
One handoff per session, newest last. Never overwrite another session's handoff.
```bash
.claude/skills/branch-context/append-handoff.sh [--writer <name>] "<one-line summary>" [path-to-body.md]
```
`--writer` tags the index line with the skill that produced the handoff. If you
omit the body path, the script writes a stub you must fill in via Write/Edit
before stopping; prefer writing the full body first and passing its path. To
revise a handoff you already wrote this session, edit the file in place rather
than appending a second index entry.
Handoff body sections (required):
```markdown
# Handoff · YYYY-MM-DD · <summary>
## Done
- ...
## Next
- ... (ordered; first item is what the next agent starts on)
## Commitments & constraints carried forward
- ... (verbatim; or "none")
## Key paths
- `path` -- why
## Open questions
- ... (or "none")
## Branch-context pointers
- Brief: issue-brief.md (still valid? yes/no)
- Decisions appended this session: <titles or "none">
- Related plan file (if any): <path>
```
**Commitments & constraints is a required check, not an optional extra.** Before
writing the handoff, sweep the session for two things and quote them verbatim --
do not paraphrase, the modality is the payload:
- **Constraints the user stated** that are not already in the brief's
Constraints section. "Always X unless Y" and "always X" are different
instructions, and a one-line paraphrase is where the qualifier gets dropped.
- **Promises you made and have not kept** -- to the user ("I'll add the
regression test next"), or on the record in a PR/review comment ("I'll file a
follow-up issue", "I'll re-run this once CI clears"). A promise made to a
reviewer and then dropped across a session boundary is the expensive kind: the
next agent cannot know it exists, and the reviewer is still waiting.
A longer, accurate handoff beats a short lossy one. Do not compress this section
to save space.
## Session turnover
Write the handoff (and any unlogged decisions) when the user turns the session
over. If the harness has a plan mode, capture the remaining work as a concrete
plan (next steps, files, verification), persist the handoff, then exit plan mode
so the fresh session inherits it. If the harness has no plan mode, persist the
handoff and tell the user to start a new session in this worktree whose first
action is to read `handoffs-index.md` and the latest handoff.
## Scope boundary
- Not for research notes or repro scripts -- those belong outside this
directory.
- Not for durable codebase facts that outlive the PR -- those belong in
longer-lived project docs or memory. Rule of thumb: if removing the linked
thread would make this PR's diff confusing, it is a decision; if the fact still
helps after merge, it belongs elsewhere.
## Helpers
```bash
.claude/skills/branch-context/status.sh # JSON: brief/decisions/handoffs state
.claude/skills/branch-context/append-pr-decision.sh ...
.claude/skills/branch-context/append-handoff.sh ...
```
src/pydantic_ai_harness/.agents/skills/branch-context/append-handoff.sh
-113
View File
@@ -1,113 +0,0 @@
#!/bin/bash
# Register one append-only handoff under branch-context.
#
# Usage:
# append-handoff.sh [--writer <name>] <one-line-summary> [body-file]
#
# - Creates handoffs/YYYY-MM-DDTHHMMZ-<slug>.md (from body-file, or a stub to fill)
# - Appends one line to handoffs-index.md, tagged with the writing skill
# - Never overwrites another session's handoff
#
# One handoff per session, newest last. The reader reads the last entry in the
# index. To revise a handoff you already wrote this session, edit the file in
# place rather than calling this again.
#
# Prints the handoff path on stdout (last line) so agents can Write/Edit the body.
set -e
WRITER="handoff"
if [ "${1:-}" = "--writer" ]; then
if [ -z "${2:-}" ]; then
echo "error: --writer needs a name" >&2
exit 1
fi
WRITER="$2"
shift 2
fi
if [ $# -lt 1 ]; then
echo "Usage: $0 [--writer <name>] <one-line-summary> [body-file]" >&2
exit 1
fi
SUMMARY="$1"
BODY_SRC="${2:-}"
DIR=".claude/skills/branch-context"
INDEX="$DIR/handoffs-index.md"
HAND_DIR="$DIR/handoffs"
if [ ! -d "$DIR" ]; then
echo "error: $DIR not found. Are you at the worktree root?" >&2
exit 1
fi
mkdir -p "$HAND_DIR"
if [ ! -f "$INDEX" ]; then
if [ -f "$DIR/handoffs-index.template.md" ]; then
cp "$DIR/handoffs-index.template.md" "$INDEX"
else
printf '# Handoffs index\n\n<!-- entries below, newest at bottom -->\n' > "$INDEX"
fi
fi
TS="$(date -u +%Y-%m-%dT%H%MZ)"
# slug: first ~40 chars of summary, safe filename
SLUG="$(printf '%s' "$SUMMARY" | tr '[:upper:]' '[:lower:]' | sed -E 's/[^a-z0-9]+/-/g; s/^-+//; s/-+$//; s/-+/-/g' | cut -c1-40)"
[ -z "$SLUG" ] && SLUG="handoff"
FNAME="${TS}-${SLUG}.md"
DEST="$HAND_DIR/$FNAME"
if [ -e "$DEST" ]; then
# collision within the same minute -- add seconds
TS="$(date -u +%Y-%m-%dT%H%M%SZ)"
FNAME="${TS}-${SLUG}.md"
DEST="$HAND_DIR/$FNAME"
fi
if [ -n "$BODY_SRC" ]; then
if [ ! -f "$BODY_SRC" ]; then
echo "error: body file not found: $BODY_SRC" >&2
exit 1
fi
cp "$BODY_SRC" "$DEST"
else
cat > "$DEST" <<EOF
# Handoff · ${TS} · ${SUMMARY}
## Done
- TODO
## Next
- TODO (first item = next agent starts here)
## Commitments & constraints carried forward
- TODO -- quote VERBATIM, never paraphrase: user constraints not already in issue-brief.md
(modality matters: "always X unless Y" != "always X"), and promises you made and have not
kept, including ones made on the record in a PR/review comment. Write "none" only after
actually sweeping the session for both.
## Key paths
- TODO
## Open questions
- none
## Branch-context pointers
- Brief: issue-brief.md -- still valid?
- Decisions appended this session: none
- Related plan file (if any): none
EOF
fi
{
echo ""
echo "## $TS · handoffs/$FNAME · [$WRITER] $SUMMARY"
} >> "$INDEX"
echo "Appended handoff index entry → $INDEX" >&2
echo "Handoff file: $DEST" >&2
# stdout: path only (for scripting)
echo "$DEST"
src/pydantic_ai_harness/.agents/skills/branch-context/append-pr-decision.sh
-126
View File
@@ -1,126 +0,0 @@
#!/bin/bash
# Append one decision entry to the adjacent pr-decisions.md.
#
# Preferred (named flags -- resistant to arg-order mistakes):
# append-pr-decision.sh \
# --title "thread #123: use kwargs not positional" \
# --decision "Reviewer suggestion adopted: kwargs-only for new param" \
# --why "Reduces breakage risk at call sites" \
# --source "https://github.com/owner/repo/pull/4567#discussion_r987654" \
# [--iter 3] [--supersedes "..."]
#
# Positional (compat):
# append-pr-decision.sh <title> <decision> <why> <source-url> [iteration] [supersedes]
#
# Field order in the file is always: title, decision, why, source, iter.
# Do NOT pass iteration as the second positional arg -- that mangles the entry.
set -euo pipefail
TITLE=""
DECISION=""
WHY=""
SOURCE=""
ITER="-"
SUPERSEDES=""
usage() {
cat >&2 <<'EOF'
Usage:
append-pr-decision.sh --title T --decision D --why W --source URL [--iter N] [--supersedes S]
append-pr-decision.sh <title> <decision> <why> <source-url> [iteration] [supersedes]
Named flags are preferred. Positional order is title, decision, why, source
(NOT title, iter, decision, ...).
EOF
exit 1
}
if [[ $# -eq 0 ]]; then
usage
fi
# Named-flag path if first arg starts with -
if [[ "${1:-}" == -* ]]; then
while [[ $# -gt 0 ]]; do
case "$1" in
-h|--help) usage ;;
--title) TITLE="${2:?}"; shift 2 ;;
--decision) DECISION="${2:?}"; shift 2 ;;
--why) WHY="${2:?}"; shift 2 ;;
--source) SOURCE="${2:?}"; shift 2 ;;
--iter|--iteration) ITER="${2:?}"; shift 2 ;;
--supersedes) SUPERSEDES="${2:?}"; shift 2 ;;
*)
echo "error: unknown option $1" >&2
usage
;;
esac
done
else
# Named-flag mode triggers only on a leading flag, so `"$title" --decision D ...` parses
# as positional and writes the literal "--decision" into a field. Fail instead.
for arg in "$@"; do
if [[ "$arg" == --* ]]; then
echo "error: '$arg' looks like a named flag, but this call parsed as positional." >&2
echo " Flags are only honoured when the FIRST argument is one." >&2
echo " Pass every field as a flag, or none of them." >&2
usage
fi
done
if [[ $# -lt 4 ]]; then
usage
fi
TITLE="$1"
DECISION="$2"
WHY="$3"
SOURCE="$4"
ITER="${5:--}"
SUPERSEDES="${6:-}"
fi
if [[ -z "$TITLE" || -z "$DECISION" || -z "$WHY" || -z "$SOURCE" ]]; then
echo "error: --title, --decision, --why, and --source are required" >&2
usage
fi
# Source is the load-bearing field -- an entry whose link doesn't resolve can't be traced back,
# and it's the field prose lands in when positional args go in the wrong order.
case "$SOURCE" in
http://*|https://*) ;;
*)
echo "error: source must be a URL, got: $SOURCE" >&2
echo " Prose here usually means the fields landed out of order." >&2
echo " Positional order is: title, decision, why, source." >&2
exit 1
;;
esac
# Static helpers can be reached through any harness skill root. Live state stays
# beside the invoked helper, so resolve the directory rather than assuming cwd.
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd -P)"
FILE="$DIR/pr-decisions.md"
if [ ! -f "$FILE" ]; then
echo "error: $FILE not found. Initialize branch context before appending a decision." >&2
exit 1
fi
DATE="$(date -u +%Y-%m-%d)"
ENTRY="
## $DATE · $TITLE · iter $ITER
- Decision: $DECISION
- Why: $WHY
- Source: $SOURCE"
if [ -n "$SUPERSEDES" ]; then
ENTRY="$ENTRY
- Supersedes: $SUPERSEDES"
fi
if ! printf '%s\n' "$ENTRY" >> "$FILE"; then
echo "error: could not append decision to $FILE" >&2
exit 1
fi
echo "Appended decision to $FILE"
src/pydantic_ai_harness/.agents/skills/branch-context/handoffs-index.template.md
-13
View File
@@ -1,13 +0,0 @@
# Handoffs index
Append-only pointer log. Newest at bottom. Full bodies live under `handoffs/` -- never overwrite a handoff file. The latest handoff is the last entry below.
## Entry format
```
## YYYY-MM-DDTHHMMZ · handoffs/<filename>.md · [<writer>] <one-line summary>
```
---
<!-- entries below, newest at bottom -->
src/pydantic_ai_harness/.agents/skills/branch-context/issue-brief.template.md
-68
View File
@@ -1,68 +0,0 @@
---
last_fetched_at: TODO
last_fetched_comment_count: 0
branch: TODO
related_pr: TBD
issues: []
---
# Issue Brief
_This file has not been populated yet. Run `/adopt-pr` to fill it from an
existing PR, or write it from the linked issue when starting fresh._
<!--
Populated schema below.
## Issues
- [#NNNN -- <title>](<url>) -- type: <bug|improvement|feature> -- role: <primary|related|follow-up>
- ...
## Problem
<1-2 sentences: the user-visible symptom or missing capability>
## Current behavior vs. expected (bugs only)
- Current: <what happens>
- Expected: <what should happen>
## Scope
**In:**
- <specific change 1>
- <specific change 2>
**Out:**
- <adjacent thing we're NOT touching>
- <nice-to-have deferred to a follow-up>
## Success criteria
Each criterion maps to a test. If a test doesn't exist yet, name the file/function we'll add.
| # | Criterion | Test |
|---|-----------|------|
| 1 | <what must be true> | `tests/path.py::test_name` |
| 2 | ... | ... |
## Constraints
- Backwards compatibility: <statement>
- Public API surface: <statement>
- Other codebase rules that apply: <...>
## Affected surface
- `path/to/file.py` -- <role>
- `path/to/module/` -- <role>
## References
- [similar PR #NNNN](<url>) -- <what it taught us>
- [docs](<url>) -- <relevance>
- [maintainer comment](<url>) -- <intent it set>
-->
src/pydantic_ai_harness/.agents/skills/branch-context/pr-decisions.template.md
-35
View File
@@ -1,35 +0,0 @@
# PR Decisions Log
Append-only record of decisions made during this PR's lifecycle that weren't obvious from the linked issue.
## Entry format
```
## YYYY-MM-DD · <short title> · iter <N or "-">
- Decision: <one line>
- Why: <one line>
- Source: <link to comment/thread/issue/commit -- mandatory>
- Supersedes: <earlier entry title, if applicable>
```
No prose. No multi-paragraph rationale. The Source link carries the full context.
**One exception -- modality.** When the Decision restates a modal claim (always / never / only if / must / unless), quote that clause verbatim instead of paraphrasing it. Brevity everywhere else, but "always X unless Y" compressed to "always X" is a different instruction, and the next agent has no way to notice the qualifier went missing.
## When to append
- You made a pick between two valid implementations and the reviewer later asked about it
- A thread resolution chose path A over B
- An ambiguity was settled during the work
- You deviated from the plan while coding
- A CI finding forced a design change
## When NOT to append
- Routine implementation work already captured by the diff
- Decisions already spelled out in the issue or in `issue-brief.md`
- Research notes (those belong outside this directory)
---
<!-- entries below, newest at bottom -->
src/pydantic_ai_harness/.agents/skills/branch-context/status.sh
-95
View File
@@ -1,95 +0,0 @@
#!/bin/bash
# Emit JSON describing the branch-context state of a worktree.
#
# Usage:
# status.sh # use $PWD as the worktree
# status.sh <worktree-path> # explicit worktree path
#
# Output (JSON, single line):
# {"initialized": <bool>,
# "brief_size": <int>,
# "decisions_count": <int>,
# "handoffs_count": <int>,
# "latest_handoff": "<relative path or empty>",
# "last_brief_update": "<ISO ts|>",
# "last_decision_at": "<ISO ts|>",
# "last_handoff_at": "<ISO ts|>",
# "worktree": "<path>"}
set -e
WT="${1:-$PWD}"
WT="$(cd "$WT" && pwd)"
DIR="$WT/.claude/skills/branch-context"
BRIEF="$DIR/issue-brief.md"
DEC="$DIR/pr-decisions.md"
INDEX="$DIR/handoffs-index.md"
HAND_DIR="$DIR/handoffs"
initialized=false
brief_size=0
last_brief_update=""
if [ -f "$BRIEF" ]; then
brief_size=$(wc -c < "$BRIEF" | tr -d ' ')
last_brief_update=$(date -u -r "$BRIEF" +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || echo "")
first_line=$(head -1 "$BRIEF" 2>/dev/null || echo "")
if [ "$brief_size" -gt 200 ] && [[ "$first_line" != *"Issue Brief Template"* ]]; then
initialized=true
fi
fi
# Count only entries after the live marker so template examples in fenced
# blocks above "<!-- entries below" don't inflate the totals.
count_after_marker() {
local file="$1" pattern="$2"
[ -f "$file" ] || { echo 0; return; }
# `grep -c` already prints 0 on no match, but exits 1; `|| echo 0` would
# append a second 0 and the caller's printf then sees "0\n0".
if grep -q '<!-- entries below' "$file" 2>/dev/null; then
sed -n '/<!-- entries below/,$p' "$file" | grep -cE "$pattern" 2>/dev/null || true
else
grep -cE "$pattern" "$file" 2>/dev/null || true
fi
}
decisions_count=0
last_decision_at=""
if [ -f "$DEC" ]; then
decisions_count=$(count_after_marker "$DEC" '^## .* · ')
last_decision_at=$(date -u -r "$DEC" +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || echo "")
fi
handoffs_count=0
latest_handoff=""
last_handoff_at=""
if [ -f "$INDEX" ]; then
handoffs_count=$(count_after_marker "$INDEX" '^## .* · handoffs/')
# last matching header line after the marker → extract path
if grep -q '<!-- entries below' "$INDEX" 2>/dev/null; then
last_line=$(sed -n '/<!-- entries below/,$p' "$INDEX" | grep -E "^## .* · handoffs/" 2>/dev/null | tail -1 || true)
else
last_line=$(grep -E "^## .* · handoffs/" "$INDEX" 2>/dev/null | tail -1 || true)
fi
if [ -n "$last_line" ]; then
# ## TS · handoffs/foo.md · summary
latest_handoff=$(printf '%s' "$last_line" | sed -E 's/^## [^·]+ · (handoffs\/[^ ]+) · .*/\1/')
if [ -f "$DIR/$latest_handoff" ]; then
last_handoff_at=$(date -u -r "$DIR/$latest_handoff" +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || echo "")
fi
fi
elif [ -d "$HAND_DIR" ]; then
# fallback: newest file in handoffs/
newest=$(ls -t "$HAND_DIR"/*.md 2>/dev/null | head -1 || true)
if [ -n "$newest" ]; then
handoffs_count=$(ls "$HAND_DIR"/*.md 2>/dev/null | wc -l | tr -d ' ')
latest_handoff="handoffs/$(basename "$newest")"
last_handoff_at=$(date -u -r "$newest" +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || echo "")
fi
fi
# JSON-escape latest_handoff (minimal: quotes/backslashes)
latest_handoff_esc=$(printf '%s' "$latest_handoff" | sed 's/\\/\\\\/g; s/"/\\"/g')
printf '{"initialized":%s,"brief_size":%d,"decisions_count":%d,"handoffs_count":%d,"latest_handoff":"%s","last_brief_update":"%s","last_decision_at":"%s","last_handoff_at":"%s","worktree":"%s"}\n' \
"$initialized" "$brief_size" "$decisions_count" "$handoffs_count" "$latest_handoff_esc" \
"$last_brief_update" "$last_decision_at" "$last_handoff_at" "$WT"
src/pydantic_ai_harness/.agents/skills/pushing-commits-to-the-repo/SKILL.md
-40
View File
@@ -1,40 +0,0 @@
---
name: pushing-commits-to-the-repo
description: What to do when you open a PR and every time you push -- label the PR, watch CI to
green, triage every review comment to a reply and a reaction, and escalate genuine design
trade-offs to maintainers. Use whenever you open a PR or push a commit to one.
---
# pushing-commits-to-the-repo
Pushing starts a loop; it does not end the task. **Work stops only when CI is green AND no comment
is left unresolved.**
## When you open the PR
Apply a label -- the repo triages and filters by them. Fetch the real list first with
`gh label list --limit 100`, because the set changes and a guessed label silently fails to apply.
Pick the one naming what the PR *is* (`bug`, `enhancement`, `documentation`) and add a topic label
where one fits (`capability`, `primitive`, `agent-feature`, `core-change`, `code-mode`, `media`,
`externalization`, ...): `gh pr edit <number> --add-label <label>`.
Labelling needs triage permission on the repo. If it fails, quote the actual error rather than
concluding you lack permission.
## Before you push
- Attempt the push. If it fails, read the real error — do not preemptively decide you lack
permission from a flag or setting.
- Leave nothing unstaged or uncommitted locally, unless the user's instructions override this.
## After you push — the loop
1. **Watch CI to a terminal state.** Don't idle. If it fails, diagnose: fix if the failure is
yours; if it's a known flake or pre-existing on main, say so with evidence.
2. **Triage every comment** (bots and humans alike). For each one:
- **Valid** → fix it, then reply saying what changed, and react 👍.
- **Invalid** → reply explaining concretely why (with code evidence), and react 👎.
- Never silently ignore a comment, and never resolve a thread without a reply.
3. **Escalate real trade-offs, don't guess.** If a comment needs a maintainer decision (a design
choice, an API trade-off, a behavioral default), leave a comment containing: the background,
your reasoning, the decision that needs making, the trade-offs (pros/cons of each option), and
your recommendation. Then **poll every 30 minutes for a reply** and continue when it lands.
4. Repeat until CI is green and no comment is outstanding.
src/pydantic_ai_harness/AGENTS.md
+27 -109
View File
@@ -1,6 +1,10 @@
# Pydantic AI Harness
## Repository purpose
Rules for the `pydantic-ai-harness` package in `src/pydantic_ai_harness/`. They add to the root
`AGENTS.md`. Test rules are in `tests/harness/AGENTS.md`. Docs page rules are in
`docs/harness/AGENTS.md`. Paths in this file are relative to the repository root.
## Package purpose
`pydantic-ai-harness` is the first-party capability library for Pydantic AI.
@@ -24,15 +28,13 @@ change instead of reimplementing core behavior in harness.
- **Guard**: a type of capability that validates inputs/outputs or controls tool access (e.g. `InputGuard`, `OutputGuard`)
- **Harness**: this package -- a collection of pre-made capabilities for Pydantic AI.
- **AICA**: AI Code Assistant -- the automated agent that implements issues, reviews plans, and handles PR feedback
- **Ralph loop**: the state-machine-based workflow that drives AICA through phases (TRIAGE -> GOALS -> PLAN -> CODE -> VERIFY -> REVIEW -> PUBLISH)
- **DDD+ protocol**: classification system for PR review comments (do, dismiss, discuss, waiting, done)
## AICA preflight
Before implementing or reviewing a capability change:
1. Read `agent_docs/index.md`.
2. Read the linked `agent_docs/` guide for the task.
1. Read `src/pydantic_ai_harness/agent_docs/index.md`.
2. Read the linked `src/pydantic_ai_harness/agent_docs/` guide for the task.
3. Read the public Pydantic AI docs for every integration point you touch:
- capabilities: <https://pydantic.dev/docs/ai/capabilities/overview/>
- hooks: <https://pydantic.dev/docs/ai/core-concepts/hooks/>
@@ -40,29 +42,18 @@ Before implementing or reviewing a capability change:
- advanced tools: <https://pydantic.dev/docs/ai/tools-toolsets/tools-advanced/>
- agents: <https://pydantic.dev/docs/ai/core-concepts/agent/>
- testing: <https://pydantic.dev/docs/ai/guides/testing/>
4. Inspect the installed `pydantic_ai` package source for exact hook/toolset
signatures when needed. Do not assume a contributor's local checkout layout.
4. Read the `pydantic_ai` source in `pydantic_ai_slim/pydantic_ai/` for exact
hook/toolset signatures when needed.
5. Use `pydantic_ai_harness.code_mode` as the exemplar for capability shape,
docs, tests, and public exports until another capability becomes a better
example. Capabilities live in their own top-level submodule
`pydantic_ai_harness/<name>/` (module name = capability name; one module per
capability or strategy) and are re-exported lazily from the root
`__init__.py`, which keeps each one's optional dependencies out of the root
import. The `experimental` tier is
retired; ACP is the sole remaining experimental capability (see
`agent_docs/capability-authoring.md`, "Capability Submodules And Exports").
## Branch context (optional)
`.agents/skills/branch-context/` is opt-in per-branch state for work that spans
sessions: an issue brief, an append-only decisions log, and session handoffs.
The scaffolding (`SKILL.md`, scripts, `*.template.md`) is committed; the
instances are git-ignored and created per branch.
- If `.agents/skills/branch-context/issue-brief.md` exists, read it plus
`pr-decisions.md` and the latest handoff before making design decisions.
- To adopt a branch, instantiate the surfaces from the templates or run
`/adopt-pr`. Contributors who never instantiate them can ignore this section.
`src/pydantic_ai_harness/pydantic_ai_harness/<name>/` (module name = capability
name; one module per capability or strategy) and are re-exported lazily from
`src/pydantic_ai_harness/pydantic_ai_harness/__init__.py`, which keeps each
one's optional dependencies out of the package import. The `experimental` tier
is retired; ACP is the sole remaining experimental capability (see
`src/pydantic_ai_harness/agent_docs/capability-authoring.md`, "Capability
Submodules And Exports").
## Capabilities API reference
@@ -74,12 +65,12 @@ When implementing a new capability, reference these docs:
- <https://pydantic.dev/docs/ai/tools-toolsets/toolsets/> -- toolset abstraction, building tools for capabilities
- <https://pydantic.dev/docs/ai/tools-toolsets/tools-advanced/> -- tool hooks, prepare tools, tool validation
- <https://pydantic.dev/docs/ai/core-concepts/agent/> -- agent configuration, instructions, model settings
- Installed `pydantic_ai.capabilities` source -- `AbstractCapability`, hook signatures, and composition behavior
- Installed `pydantic_ai.toolsets` source -- `AbstractToolset`, `WrapperToolset`, and `ToolsetTool`
- `pydantic_ai_slim/pydantic_ai/capabilities/` -- `AbstractCapability`, hook signatures, and composition behavior
- `pydantic_ai_slim/pydantic_ai/toolsets/` -- `AbstractToolset`, `WrapperToolset`, and `ToolsetTool`
## Capability naming
Follow the naming convention in `agent_docs/capability-authoring.md` ("Naming
Follow the naming convention in `src/pydantic_ai_harness/agent_docs/capability-authoring.md` ("Naming
Capabilities"): a noun when the capability names a thing (a tool or faculty the
model uses, a subsystem, a named strategy); an imperative verb phrase when it
acts on the run and one verb phrase states its entire contract. Never invent a
@@ -97,7 +88,7 @@ Emitting nothing is a valid answer when core's own spans already cover the work.
It is an answer to state in the docs, not a step to skip.
The house pattern (spans on `ctx.tracer`, attribute naming, content behind
`trace_include_content`) is in `agent_docs/capability-authoring.md`
`trace_include_content`) is in `src/pydantic_ai_harness/agent_docs/capability-authoring.md`
"Telemetry".
## Coding standards
@@ -135,8 +126,6 @@ Applies to docs, READMEs, docstrings, comments, commit messages, and PR text.
## Package management
- Change dependencies only when required. Use `uv` and link an issue.
- PRs touching `pyproject.toml` or `uv.lock` require the
`dependencies:approved` label; pushes clear approval.
## Local verification
@@ -145,102 +134,31 @@ Run Ruff across the repository. Run Pyright and pytest for the paths that you mo
```bash
uv run --no-sync ruff format --check .
uv run --no-sync ruff check .
PYRIGHT_PYTHON_IGNORE_WARNINGS=1 uv run --no-sync pyright pydantic_ai_harness/<module>.py tests/<module>.py
uv run --no-sync pytest -p no:cacheprovider tests/<capability>
PYRIGHT_PYTHON_IGNORE_WARNINGS=1 uv run --no-sync pyright src/pydantic_ai_harness/pydantic_ai_harness/<capability>/ tests/harness/<capability>/
uv run --no-sync pytest -p no:cacheprovider tests/harness/<capability>
```
CI runs the repository-wide typecheck, test, and combined coverage gates.
Do not run repository-wide Pyright, pytest, or coverage locally.
If CI reports a coverage gap, run coverage only for the flagged file or focused test.
### Workspace And Python 3.10
```bash
uv sync --locked --all-packages --all-extras --group lint
```
Harness and `pydantic-clai2` share the root `uv.lock`, `.venv`, and Pyright
configuration. Workspace commands require Python 3.11+ because CLAI depends on
Termflow. Harness's package metadata and Pyright target remain Python 3.10+.
Tracking issue: https://github.com/pydantic/pydantic-ai-harness/issues/875.
```bash
uv venv /tmp/harness-py310 --python 3.10
uv pip install --python /tmp/harness-py310/bin/python --resolution lowest-direct --group dev \
--editable . --requirements pyproject.toml --all-extras
/tmp/harness-py310/bin/python -m pytest -p no:cacheprovider tests/code_mode
```
Use `uv pip` for Python 3.10 to resolve Harness without CLAI's Python requirement.
CI retains Python 3.10 slim, all-extras, and lowest-versions jobs. The slim and
all-extras installs use the shared lock's versions as ceilings, allowing older
releases when a locked dependency requires Python 3.11+. Run the environment's
Python directly so `uv run` does not select the workspace interpreter.
## File structure
The tree is discoverable by listing it; only the conventions that are not are
recorded here.
Each released capability is a self-contained package under
`pydantic_ai_harness/<capability>/` (naming and exports are covered in the
preflight above), with tests under `tests/<capability>/`. It ships **two**
hand-maintained docs that must stay in sync: the `README.md` next to the code
(GitHub/PyPI) and the `docs/<capability>.md` page (the docs site at
pydantic.dev/docs/ai/harness). The `docs/` folder is flat -- there are no
`capabilities/` or `experimental/` subdirectories. A user-facing change updates
both; `agent_docs/review-checklist.md` "Docs" and the `docs-parity-reviewer`
subagent enforce the parity before merge.
`src/pydantic_ai_harness/pydantic_ai_harness/<capability>/` (naming and exports are covered in the
preflight above), with tests under `tests/harness/<capability>/`. It ships **two**
hand-maintained docs: the `README.md` next to the code (GitHub/PyPI) and the
`docs/harness/<capability>.md` page (the docs site at pydantic.dev/docs/ai/harness).
`docs/harness/AGENTS.md` covers keeping them in sync.
Do not add placeholder template files for new capabilities. Start from the
existing `CodeMode` package shape, then delete what the new capability does not
need.
## Testing patterns
- Import `TestModel` from `pydantic_ai.models.test` for model behavior.
- Keep real provider calls out of tests.
- `ALLOW_MODEL_REQUESTS = False` is set globally in `conftest.py`
- Tests use `pytest-anyio` for async support
- Each capability test class follows: `TestCapabilityName` with methods `test_<scenario>`
- Prefer tests through `Agent(..., capabilities=[...])` when that is the public
behavior. Use direct `Toolset`/`RunContext` tests for lower-level lifecycle,
schema, retry, or wrapper behavior that is hard to isolate through `Agent`.
- Don't import private (`_`-prefixed) helpers into tests. Exercise them through
the capability's public surface so tests survive internal refactors: drive the
behavior through `Agent(..., capabilities=[...])`, or import the public class
re-exported from the capability package's `__init__.py` (e.g.
`from pydantic_ai_harness.filesystem import FileSystemToolset`, not
`from pydantic_ai_harness.filesystem._toolset import _content_hash`). When a
branch is only reachable by calling a private helper directly, mark it
`# pragma: no cover` rather than reaching into the helper from a test.
- Test async behavior directly, following `agent_docs/concurrency.md`: assert
the cancellation, the ordering, or the cleanup itself rather than the output
it happens to produce; order steps with `Event`s instead of sleeps; and reach
the real trigger (a real outer `anyio` cancel scope, the Trio parametrization
where the suite has it) rather than a stand-in.
## Contributing rules for AICAs
- Always link sources for any claims made during research
- Commit messages should summarize the "why", not the "what"
- PR titles feed the release notes verbatim -- GitHub builds "What's Changed" from them. Write an
imperative sentence naming the change, and wrap every code identifier (class names, keyword
arguments, module paths, CLI flags, env vars, file paths) in backticks. No `feat:` / `fix:` /
`docs:` prefix -- that belongs on the commit subject, not the title. Merged titles predating this
rule carry prefixes; follow the rule, not the back catalogue.
## Pushing changes
**A restriction is a conclusion you earn from a real failure, not a field you read.** Never report an
operation as blocked, unavailable, or not-permitted based on a metadata flag, a config field, or a
docs claim — attempt it and quote the actual error. (`maintainerCanModify: false` on a PR does *not*
mean you cannot push: it governs the upstream-maintainer auto-grant, not your own access to the
fork.) If you genuinely cannot attempt it, say "not attempted", never "we can't".
**Pushing is not the end of the task.** After you push, do not go idle. The work is done when
**CI is green and there are no unresolved comments** — see the `pushing-commits-to-the-repo` skill
for the full loop.
**Do not leave work uncommitted.** Don't end a turn with unstaged or uncommitted local changes
unless the user's own instructions say otherwise.
src/pydantic_ai_harness/agent_docs/capability-authoring.md
+16 -15
View File
@@ -26,7 +26,7 @@ Each capability package should normally have:
- `_capability.py` for the public capability class
- `_toolset.py` only if the capability needs toolset behavior
- `README.md` with focused usage docs (serves GitHub and PyPI)
- a unified-docs page at `docs/<capability>.md` (the `docs/` folder is flat --
- a unified-docs page at `docs/harness/<capability>.md` (the `docs/harness/` folder is flat --
no `capabilities/` or `experimental/` subdirectories). It mirrors the README
for the docs site, drops badges, links other harness pages and Pydantic AI
docs with relative `.md` links (`../toolsets.md`), links its
@@ -34,28 +34,28 @@ Each capability package should normally have:
with a `::: pydantic_ai_harness.<Class>` autodoc block. The README and this
page are kept in sync (see `review-checklist.md` "Docs").
- an entry in the `pydantic-ai-harness` agent skill
(`pydantic_ai_harness/.agents/skills/pydantic-ai-harness/`): a row in the
(`src/pydantic_ai_harness/pydantic_ai_harness/.agents/skills/pydantic-ai-harness/`): a row in the
`SKILL.md` routing tables and a section in the matching `references/` file
- mirrored tests under `tests/<capability>/`
- mirrored tests under `tests/harness/<capability>/`
The root `pydantic_ai_harness/__init__.py` should re-export stable public
The root `src/pydantic_ai_harness/pydantic_ai_harness/__init__.py` should re-export stable public
capabilities. Keep implementation helpers private unless users need them.
### Capability Submodules And Exports
The `experimental` tier is retired. ACP is the sole remaining experimental
capability (`pydantic_ai_harness/experimental/acp/`); do not add new capabilities
capability (`src/pydantic_ai_harness/pydantic_ai_harness/experimental/acp/`); do not add new capabilities
there.
New capabilities land as a top-level submodule `pydantic_ai_harness/<name>/`
AND are re-exported **lazily** from the root `pydantic_ai_harness/__init__.py` —
New capabilities land as a top-level submodule `src/pydantic_ai_harness/pydantic_ai_harness/<name>/`
AND are re-exported **lazily** from the root `src/pydantic_ai_harness/pydantic_ai_harness/__init__.py` —
top-level importability is the norm (ruled 8/13): every public capability class
(and public constants like `DEFAULT_ALLOWED_COMMANDS`) is added to `__all__`,
to the `TYPE_CHECKING` import block, and to the `__getattr__` dispatch. The lazy
`__getattr__` pattern is what makes this safe with optional dependencies:
importing the root package pulls in nothing, and accessing an extra-gated name
without its extra installed raises the standard install-hint `ImportError`.
`tests/test_placeholder.py::test_all_exports_are_importable` enforces the
`tests/harness/test_placeholder.py::test_all_exports_are_importable` enforces the
contract — a new-capability PR that skips the root export fails it. The sole
exception is `experimental` (ACP), which stays submodule-only. Docs and examples
prefer the top-level import (`from pydantic_ai_harness import <Name>`).
@@ -105,9 +105,9 @@ tool) rather than guessing -- a name is a public commitment once shipped.
When a class or module is renamed, keep the old name working for at least one
release: a renamed module keeps a shim package at its old path, and a renamed
class keeps a module-level `__getattr__` alias, both emitting
`HarnessDeprecationWarning` via the helpers in `pydantic_ai_harness/_warn.py`.
`HarnessDeprecationWarning` via the helpers in `src/pydantic_ai_harness/pydantic_ai_harness/_warn.py`.
Top-level re-exports in `pydantic_ai_harness/__init__.py` (`CodeMode`,
Top-level re-exports in `src/pydantic_ai_harness/pydantic_ai_harness/__init__.py` (`CodeMode`,
`FileSystem`, `Shell`, `ManagedPrompt`) are the exception, not the rule. Once an
export has shipped in a published release it is a backward-compatibility
commitment: do not move, rename, or break it. Do not add new top-level
@@ -146,7 +146,7 @@ the capability name lets a reader trace spend straight back to the
capability without knowing its internals.
Test it through `instrument_all_agents` + `agent_run_names(capfire)` from
`tests/conftest.py`; per-agent `instrument=` on the outer agent does not reach
`tests/harness/conftest.py`; per-agent `instrument=` on the outer agent does not reach
the inner one.
### Policy Lives In The Pluggable Component
@@ -257,7 +257,7 @@ agent = Agent(
The model then sees `hr_bamboohr_list_employees` and
`crm_bamboohr_list_employees`, and each routes to its own account.
Record the choice in `tests/test_capability_combine.py`;
Record the choice in `tests/harness/test_capability_combine.py`;
`test_every_capability_declares_a_combine_policy` fails until you do. Write the
reason from what the capability actually does, not from what would be
convenient: a reason like "one per rooted directory" is wrong if the toolset's
@@ -332,9 +332,10 @@ When a capability needs machinery of that weight:
to a CI `environment` that holds the secret, so checkout and setup steps never
see it.
The `localstack` capability's `localstack-integration` job is the reference for
this shape. Whether the heavy job blocks merges (listed in `check`'s `needs`) or
only signals is the capability owner's call; state which in the PR.
The `harness-integration-changes` and `harness-mongodb-integration` jobs in
`.github/workflows/ci.yml` show the path scoping and the `check` wiring. Whether
the heavy job blocks merges (listed in `check`'s `needs`) or only signals is the
capability owner's call; state which in the PR.
## External-Service Assumptions And Refresh
src/pydantic_ai_harness/agent_docs/concurrency.md
+7 -6
View File
@@ -2,7 +2,7 @@
Rules for `asyncio`/`anyio` code, ported from Pydantic AI core's guide
(pydantic/pydantic-ai#8097, `agent_docs/concurrency.md`) and re-anchored to
this repository. Most of the core rules were paid for by a real bug there; the
the harness. Most of the core rules were paid for by a real bug in core; the
harness inherits the same runtime, so it inherits the same rules.
When to check: whenever you write or review code that spawns a task, opens a
@@ -15,8 +15,9 @@ anchor is judgment, not evidence. Check the anchor before you argue with a rule
and before you extend one: the usual way to get this wrong is to state a true
general mechanism more broadly than the code supports, or to describe a design
that was proposed but never shipped. If the code has moved, update the rule.
Anchors marked "core" live in `pydantic_ai`; read the installed package source,
not a contributor's checkout.
Anchors marked "core" are under `pydantic_ai_slim/pydantic_ai/`. Paths that
start with `agent_docs/`, `tests/`, or `pydantic_ai_slim/` are relative to the
repository root; other bare paths are under `src/pydantic_ai_harness/pydantic_ai_harness/`.
Before adding any of this, name the scope that guarantees teardown for every
task, scope, lock, stream, subprocess, span, and connection you create. "The
@@ -35,7 +36,7 @@ garbage collector" or "the caller remembers" is a bug, not a design.
- Prefer a task group, whose `async with` encloses everything the children
touch, over loose tasks. Core's `LocalWorkspaceBackend.run` reads stdout and
stderr as two children of one group inside one `move_on_after`, so a timeout
cancels both readers together (`pydantic_ai/workspaces/local.py`). Avoid
cancels both readers together (`pydantic_ai_slim/pydantic_ai/workspaces/local.py`). Avoid
`asyncio.gather(..., return_exceptions=False)` when one failure should stop
the batch: it propagates the first failure while siblings keep running.
`return_exceptions=True` is fine for a cleanup-only drain.
@@ -175,8 +176,8 @@ before writing cleanup.
tasks.
- Order steps with `Event`s, not sleeps, and wait on them with a module-level
readiness timeout, not a one-second literal: short waits flake under `xdist`
(pydantic/pydantic-ai#5399). `tests/shell/test_shell.py` and
`tests/guardrails/test_input_guardrail.py` already carry sleeps; when you touch
(pydantic/pydantic-ai#5399). `tests/harness/shell/test_shell.py` and
`tests/harness/guardrails/test_input_guardrail.py` already carry sleeps; when you touch
one, convert it.
- Prove ownership directly: diff `asyncio.all_tasks()` for ordinary leak
checks; for a subprocess, assert the process group is gone after the timeout
src/pydantic_ai_harness/agent_docs/docs-conventions.md
+33 -28
View File
@@ -1,7 +1,7 @@
# Docs Conventions
How the harness's user-facing docs stay correct, consistent, and discoverable. Read this before
touching `README.md`, `docs/`, or a capability README — and before adding a
touching `src/pydantic_ai_harness/README.md`, `docs/harness/`, or a capability README — and before adding a
capability, since every capability lands with docs or fails CI.
## Writing Style
@@ -12,24 +12,27 @@ mechanism, not significance; no throat-clearing, no "powerful"/"seamless"-class
"not just X, but Y" reveals; plain verbs over industry metaphors; lead with the fact; one idea
per sentence.
## The Parity Contract (enforced by `tests/test_docs_parity.py`)
## The Parity Contract (enforced by `tests/harness/test_docs_parity.py`)
Every capability package must, or CI fails:
1. Have its own `README.md` (purpose-first lead, source-module link, spaced-words H1).
2. Be linked from the top-level `README.md` capability tables.
3. Have a `docs/<slug>.md` page registered in `_CAPABILITY_PAGE_META` (source module + exact H1).
4. Have a partner PR that adds or updates the page in pydantic-ai's
`docs/navigation.yml`.
2. Be linked from the `src/pydantic_ai_harness/README.md` capability tables.
3. Have a `docs/harness/<slug>.md` page registered in `_CAPABILITY_PAGE_META` (source module + exact H1).
## Sidebar Source Of Truth Lives In pydantic-ai (since 2026-08-12)
Also add the page to `docs/navigation.yml`, in its category subsection under "Capabilities". A
durable-execution backend goes in the "Durable Execution" section instead (like
`harness/aws-lambda.md`). An interface goes in the "Interfaces" section instead (like
`harness/acp.md`). No test checks the `docs/navigation.yml` entry.
Since unified-docs #179, the harness sidebar renders from **pydantic-ai's
`docs/navigation.yml`** (the "Pydantic AI Harness" section; entries carry
`source: "harness"`). This repository has no local navigation file. Adding,
renaming, removing, or regrouping a harness docs page therefore requires a partner
PR that updates pydantic-ai's `docs/navigation.yml`. Keep slugs
`harness/<page>` and preserve existing `aliases:` when renaming.
## Sidebar Source Of Truth
The harness sidebar renders from **`docs/navigation.yml`**: mostly the category
subsections under "Capabilities", shared with core capability pages, plus a few
pages under "Durable Execution" and "Interfaces". Harness entries carry
`path: "harness/<page>.md"`. Adding, renaming, removing, or regrouping a harness
docs page updates `docs/navigation.yml` in the same PR. Keep each harness page's
slug as `harness/<page>`. When renaming a page, keep its existing `aliases:`.
## Page Conventions
@@ -60,18 +63,18 @@ PR that updates pydantic-ai's `docs/navigation.yml`. Keep slugs
2026-08 that means `claude-fable-5` / `gpt-5.6-sol`, not older defaults) unless a page
intentionally pins an older one.
## The Capability Index Spans Two Repos
## The Capability Index Spans Core And Harness
`README.md` and `docs/index.md` present **one capability index across core and Harness** (the
"Ships in" column): core capabilities (Web Search, MCP, Tool Search, Thinking, tool approval, …)
appear here linked to the pydantic-ai docs, and pydantic-ai's docs mirror harness entries in the
other direction.
`src/pydantic_ai_harness/README.md` and `docs/harness/index.md` present **one capability index across core and
Harness** (the "Package" column): core capabilities (Web Search, MCP, Tool Search, Thinking, tool
approval, …) appear there linked to the Pydantic AI docs, and the Pydantic AI docs mirror harness
entries in the other direction.
**This mirroring is a maintenance contract.** If you add, rename, re-categorize, or remove a
capability here, check whether pydantic-ai's docs (its capabilities overview and front pages)
reference it — and open a companion PR there when they do. The same applies in reverse when core
capabilities change. Until an automated docs-review workflow enforces this, the reminder lives
here: a rename PR is not done until both repos agree.
harness capability, check whether the Pydantic AI docs (the capabilities overview and the front
pages) reference it, and update them in the same PR when they do. The same applies in reverse
when core capabilities change. Until an automated docs-review workflow enforces this, the
reminder lives here: a rename PR is not done until both indexes agree.
## Categories
@@ -82,9 +85,10 @@ workspace the agent acts in) · **Tools & native abilities** · **Web & research
window — Code Mode lives here, not under execution: it changes *how* the agent executes, not
*where*) · **Knowledge & memory** · **Control & safety** · **Self-extension** ·
**Execution runtime** (durable execution, persistence, observability plumbing). The same scheme
orders the sidebar in pydantic-ai's `docs/navigation.yml`. A new capability goes in the category
matching its user-facing benefit; if none fits, raise it in the PR rather than inventing an
eleventh silently. Keep every table's "Package" column and one-line description style intact.
orders the subsections under "Capabilities" in `docs/navigation.yml`. That file adds a core-only
"Loop Customization" subsection after them. A new capability goes in the category matching its
user-facing benefit. If no category fits, say so in the PR instead of silently adding a new
category. Keep every table's "Package" column and one-line description style intact.
## Harness Pages Specifically
@@ -93,14 +97,15 @@ capability composing other capabilities** and show the blown-out equivalent `cap
list — the transparency promise is part of the product. Harness classes are named like the agent
they create (`Coder`), not like capabilities; the word "preset" is not used.
The blown-out equivalent — on the docs page and in the harness's `examples/` counterpart — writes
The blown-out equivalent — on the docs page and in the harness's `src/pydantic_ai_harness/examples/` counterpart — writes
the harness's defaults out literally (instructions, command allowlists) instead of importing the
constants: the reader is meant to see the entire picture and copy-tweak it. It mirrors the
exported agent (`coder_agent`, …) exactly, including its `name=` and identity `instructions=`.
The Coder blown-out block lives on **five surfaces** that must stay identical:
`docs/coder.md`, `docs/index.md`, `README.md`, `pydantic_ai_harness/coder/README.md`, and (as a
parameterized module) `examples/coding_agent.py`. `tests/test_docs_parity.py` compares the four
`docs/harness/coder.md`, `docs/harness/index.md`, `src/pydantic_ai_harness/README.md`,
`src/pydantic_ai_harness/pydantic_ai_harness/coder/README.md`, and (as a parameterized module)
`src/pydantic_ai_harness/examples/coding_agent.py`. `tests/harness/test_docs_parity.py` compares the four
markdown copies byte-for-byte and checks the written-out allowlist against
`DEFAULT_ALLOWED_COMMANDS`, so a change to any one of them fails CI until all move together —
change the implementation, the pages, and the example in the same PR. Each block carries a
src/pydantic_ai_harness/agent_docs/index.md
+5 -5
View File
@@ -6,7 +6,7 @@ Use these guides when building or reviewing `pydantic-ai-harness` changes.
For any code change:
1. `AGENTS.md`
1. `src/pydantic_ai_harness/AGENTS.md`
2. This file
3. The guide below that matches the task
4. The public Pydantic AI docs for the integration points you touch
@@ -26,7 +26,7 @@ For any code change:
Component"
- Deciding what a capability emits to OpenTelemetry, or documenting that it
emits nothing: `capability-authoring.md` "Telemetry"
- README, `docs/` pages, or capability READMEs: `docs-conventions.md`
- `src/pydantic_ai_harness/README.md`, `docs/harness/` pages, or capability READMEs: `docs-conventions.md`
- Review, pre-PR check, or final self-check: `review-checklist.md`
- Commands/parsers, processes/containers, network endpoints, resource cleanup,
output limits, or CI trust boundaries: `review-checklist.md` "Executable
@@ -39,12 +39,12 @@ For any code change:
Use `pydantic_ai_harness.code_mode` as the current exemplar for capability
shape:
- public re-export from `pydantic_ai_harness/__init__.py`
- package-level re-export from `pydantic_ai_harness/code_mode/__init__.py`
- public re-export from `src/pydantic_ai_harness/pydantic_ai_harness/__init__.py`
- package-level re-export from `src/pydantic_ai_harness/pydantic_ai_harness/code_mode/__init__.py`
- public capability class in `_capability.py`
- implementation toolset in `_toolset.py`
- capability README next to implementation
- mirrored tests under `tests/code_mode/`
- mirrored tests under `tests/harness/code_mode/`
Do not copy `CodeMode` mechanically. Use it to understand package shape,
testing depth, docs placement, and how a harness capability composes with
src/pydantic_ai_harness/agent_docs/review-checklist.md
+20 -26
View File
@@ -26,9 +26,7 @@ Use this before opening a PR or reviewing a capability change.
`ctx.tracer`; emitting nothing is documented with its reason. See
`capability-authoring.md` "Telemetry".
- Capability ordering is justified when present.
- Dependency changes are required, linked to an issue, and made through `uv`;
every PR touching `pyproject.toml` or `uv.lock` carries
`dependencies:approved` for the current head.
- Dependency changes are required, linked to an issue, and made through `uv`.
- A capability that adds heavy CI machinery (a Docker image, an external service
with a secret, a large system binary, live network calls) scopes its expensive
job to its own paths and keeps the aggregate check green when that job is
@@ -65,22 +63,19 @@ retain the command and result as review evidence.
## Stale Or Pre-Merge PRs
Run these checks when adopting, rebasing, or re-reviewing a PR that was opened
well before now, or that was built against unreleased Pydantic AI changes.
well before now, or that is stacked on another open PR.
- Temporary `[tool.uv.sources]` pins to a branch or git ref are removed once the
upstream change they waited on has landed in a released `pydantic-ai-slim`.
- Each upstream Pydantic AI PR or branch the change rode on has merged. Link the
upstream PR and its merge state.
- Each Pydantic AI PR the change is stacked on has merged. Link it and its merge
state.
- The touched surface has not drifted: re-check the capability, hook, and toolset
signatures it depends on against current main, not against the state at fork
time.
- Behavior the PR worked around because a primitive was missing is reconsidered
if that primitive now exists in core.
- A flood of pyright or import errors right after merging main or rebasing is
usually uninstalled extras, not a real regression. Re-sync (`make install`, or
`uv sync --frozen --all-extras --group lint`) before treating the merge as
broken; errors that name third-party types (`modal`, `openai`, ...) as unknown
in files the PR did not touch are the tell.
usually uninstalled extras, not a real regression. Re-sync with `make install`
before treating the merge as broken; errors that name third-party types
(`modal`, `openai`, ...) as unknown in files the PR did not touch are the tell.
## Issue References
@@ -111,26 +106,25 @@ README, or source code.
explicit, tested incompatibility. Mocked lifecycle tests alone do not
establish state continuity across activity, process, or replay boundaries.
- Relevant protocol-shaped output is snapshotted.
- **Harness cassettes are re-recorded on composition change.** Each packaged
harness has a recorded end-to-end integration test (e.g.
`tests/coder/test_coder_integration.py`) that runs it against a real task.
Any change to that harness's composition, defaults, or instructions
- **Harness cassettes are re-recorded on composition change.** When a packaged
harness has a recorded end-to-end integration test that runs it against a real
task (e.g. `tests/harness/researcher/test_researcher_integration.py`), any
change to that harness's composition, defaults, or instructions
re-records the cassette in the same PR
(`uv run --env-file .env --no-sync pytest -p no:cacheprovider <test> --record-mode=rewrite`) —
a green replay of a stale cassette proves nothing about the new definition.
- Run the local verification commands in `AGENTS.md` before handoff.
- Run the local verification commands in `src/pydantic_ai_harness/AGENTS.md` before handoff.
## Docs
Every released capability ships two hand-maintained docs that must stay in sync
with the code and with each other:
- the **README** next to the implementation (`pydantic_ai_harness/<capability>/README.md`,
or `pydantic_ai_harness/experimental/<capability>/README.md` for ACP), which
- the **README** next to the implementation (`src/pydantic_ai_harness/pydantic_ai_harness/<capability>/README.md`,
or `src/pydantic_ai_harness/pydantic_ai_harness/experimental/<capability>/README.md` for ACP), which
serves GitHub and PyPI, and
- the **unified doc** on the docs site, flat under `docs/<capability>.md`. The
sidebar is a flat list under "Pydantic AI Harness" -- no `capabilities/` or
`experimental/` subdirectories.
- the **unified doc** on the docs site, flat under `docs/harness/<capability>.md`. The
`docs/harness/` folder has no `capabilities/` or `experimental/` subdirectories.
Checks:
@@ -138,14 +132,14 @@ Checks:
(public class, params, defaults, tool names, extras, safety semantics). A
change reflected in only one of them is a defect, not a follow-up.
- The `pydantic-ai-harness` agent skill
(`pydantic_ai_harness/.agents/skills/pydantic-ai-harness/`) reflects the change:
(`src/pydantic_ai_harness/pydantic_ai_harness/.agents/skills/pydantic-ai-harness/`) reflects the change:
a new capability is listed in its routing table and covered in the matching
`references/` file, and a changed parameter, default, tool name, or extra is
updated there too.
- **Harness blown-out parity.** A packaged harness (`Coder`, `Researcher`, ...)
has its composition written out in full — default instructions and allowlists
included, not imported — in its docs page's "Blown-out equivalent" AND in its
`examples/` counterpart. Any change to a harness's composition or defaults
`src/pydantic_ai_harness/examples/` counterpart. Any change to a harness's composition or defaults
updates all three together (implementation, docs page, example) in the same
PR; drift here is a defect, not a follow-up.
- The two do not contradict each other or the source on extras, option names,
@@ -182,10 +176,10 @@ Checks:
The mechanical half of these checks (README present + linked, flat page present,
source link present, name matches, no experimental strings on non-ACP pages, no
hook name in the lead) is enforced by `tests/test_docs_parity.py`. The semantic
hook name in the lead) is enforced by `tests/harness/test_docs_parity.py`. The semantic
half (does the prose match the code, are snippets truly runnable) is what the
reviewer below is for.
This is the last documentation gate before merge. Run the `docs-parity-reviewer`
subagent (`.agents/agents/docs-parity-reviewer.md`) on the change as the final
skill (`.agents/skills/docs-parity-reviewer/SKILL.md`) on the change as the final
review step; treat its blocking findings as merge blockers.
src/pydantic_ai_harness/agent_docs/testing-capabilities.md
+4 -4
View File
@@ -8,8 +8,8 @@ helpers.
- Import `TestModel` from `pydantic_ai.models.test` for model behavior.
- Keep real provider calls out of tests.
- Prefer `Agent(..., capabilities=[...])` tests for public behavior.
- Mirror source packages under `tests/<capability>/`.
- Use `pytest-anyio` for async capability/toolset behavior.
- Mirror source packages under `tests/harness/<capability>/`.
- Use anyio's pytest plugin for async capability/toolset behavior.
## Lower-Level Tests
@@ -64,8 +64,8 @@ Run Ruff across the repository. Run pytest and Pyright for the changed paths:
```bash
uv run --no-sync ruff format --check .
uv run --no-sync ruff check .
uv run --no-sync pytest -p no:cacheprovider tests/<capability>
PYRIGHT_PYTHON_IGNORE_WARNINGS=1 uv run --no-sync pyright pydantic_ai_harness/<module>.py tests/<module>.py
uv run --no-sync pytest -p no:cacheprovider tests/harness/<capability>
PYRIGHT_PYTHON_IGNORE_WARNINGS=1 uv run --no-sync pyright src/pydantic_ai_harness/pydantic_ai_harness/<capability>/ tests/harness/<capability>/
```
CI runs the repository-wide typecheck, test, and combined coverage gates.
tests/harness/AGENTS.md
+20
View File
@@ -0,0 +1,20 @@
# `tests/harness/` guidelines
Rules for the `pydantic-ai-harness` test suite. They add to [`tests/AGENTS.md`](../AGENTS.md) and to the package
rules in [`src/pydantic_ai_harness/AGENTS.md`](../../src/pydantic_ai_harness/AGENTS.md). Where these rules or the
harness guides they link differ from `tests/AGENTS.md`, they apply under `tests/harness/`.
Read [`testing-capabilities.md`](../../src/pydantic_ai_harness/agent_docs/testing-capabilities.md) before you add or
change a capability's tests. For async behavior, follow the harness
[`concurrency.md`](../../src/pydantic_ai_harness/agent_docs/concurrency.md).
- `tests/harness/conftest.py` sets `ALLOW_MODEL_REQUESTS = False` for the suite.
- Name a capability test class `TestCapabilityName`, with `test_<scenario>` methods.
- Don't import private (`_`-prefixed) helpers into tests. Exercise them through
the capability's public surface so tests survive internal refactors: drive the
behavior through `Agent(..., capabilities=[...])`, or import the public class
re-exported from the capability package's `__init__.py` (e.g.
`from pydantic_ai_harness.filesystem import FileSystemToolset`, not
`from pydantic_ai_harness.filesystem._toolset import _content_hash`). When a
branch is only reachable by calling a private helper directly, mark it
`# pragma: no cover` rather than reaching into the helper from a test.
tests/harness/CLAUDE.md
Symlink
+1
View File
@@ -0,0 +1 @@
AGENTS.md
tests/harness/test_docs_parity.py
+13 -9
View File
@@ -1,8 +1,8 @@
"""Keep the README honest about what ships.
Every capability package must document itself with a `README.md` and be linked
from the top-level `README.md`. A capability cannot land without showing up in
the docs, so the "what's available today" tables cannot silently fall behind the
from `src/pydantic_ai_harness/README.md`. A capability cannot land without showing up in
the docs, so the capability tables cannot silently fall behind the
code. This is the mechanical half of docs parity; the semantic half (does the
prose match the code as written) is a review-time concern, not a unit test.
"""
@@ -80,15 +80,15 @@ def test_capability_linked_from_top_readme(package: Path) -> None:
# anywhere (prose or an unrelated URL would otherwise satisfy the check).
linked = any(t.startswith(link_target) for t in _markdown_link_targets(top_readme))
assert linked, (
f'{package.relative_to(_ROOT)} is not linked from the top-level README.md. '
f'Add a row for it (linking `{link_target}`) to the "What\'s available today" or "Roadmap" tables '
f'{package.relative_to(_ROOT)} is not linked from src/pydantic_ai_harness/README.md. '
f'Add a row for it (linking `{link_target}`) to the matching category table under "Capabilities" '
'so the README stays in step with the code.'
)
# --- Unified-docs page checks (docs/*.md) -----------------------------------
# --- Unified-docs page checks (docs/harness/*.md) ---------------------------
#
# The flat pages under `docs/` render on the unified site. These mechanical
# The flat pages under `docs/harness/` render on the unified site. These mechanical
# checks encode the capability-authoring rules agreed in the 2026-07-10 team
# sync: purpose-first leads, a source link on every page, names that match the
# capability, and no leftover "experimental" framing on graduated capabilities.
@@ -97,8 +97,11 @@ def test_capability_linked_from_top_readme(package: Path) -> None:
_DOCS_DIR = _ROOT / '../../docs/harness'
# `media.md` documents Step Persistence's storage plumbing (see `_NOT_A_CAPABILITY` above),
# `gh-aw.md` walks through the gh-aw engine definition under `gh-aw/`, and `durable-execution.md` is a
# guide across capabilities, so the capability-page checks do not apply to them.
# guide across capabilities, so the capability-page checks do not apply to them. `AGENTS.md` (and its
# `CLAUDE.md` link) holds the directory's agent instructions.
_NON_CAPABILITY_PAGES = {
'AGENTS.md',
'CLAUDE.md',
'clai2.md',
'durable-execution.md',
'examples.md',
@@ -365,8 +368,9 @@ def test_capability_readme_links_source(package: Path) -> None:
# The Coder blown-out example is repeated on four markdown surfaces (plus, parameterized,
# in examples/coding_agent.py). They must stay byte-identical so no page drifts from what
# `coder_agent` actually is; see agent_docs/docs-conventions.md.
# in src/pydantic_ai_harness/examples/coding_agent.py). They must stay byte-identical so no
# page drifts from what `coder_agent` actually is; see
# src/pydantic_ai_harness/agent_docs/docs-conventions.md.
_BLOWN_OUT_MARKER = '<!-- Keep this blown-out example in sync across'
_BLOWN_OUT_SURFACES = (
'../../docs/harness/coder.md',