Release PRs currently verify that dependencies resolve against PyPI, but
they do not show when a declared minimum has fallen behind the newest
published version. That makes dependency reconciliation easy to miss at
the point maintainers are preparing a release.
This adds an advisory release-PR check for the changed package
manifests. It compares required and optional dependency minimums with
PyPI, reports whether the newest version still fits the current ceiling,
and maintains one non-blocking PR comment. The existing `release-deps:
acknowledged` label records that maintainers reviewed the result without
requiring a metadata change.
Stable releases are compared by default. Pre-releases participate only
when the declared minimum is itself a pre-release, while an environment
setting keeps that policy explicit and adjustable. Local path and
workspace sources are excluded, so sibling development installs do not
create noise while external LangChain dependencies remain covered.
PyPI requests use bounded concurrency, timeouts, and retries. Query
failures remain advisory and preserve any previous finding rather than
incorrectly declaring the dependency set current. Unit coverage
exercises requirement parsing, release selection, upper bounds, source
exclusions, retries, output formatting, and canonical-name request
deduplication.
## What
Replace the serialized per-provider / per-category eval jobs with one
dynamically-drained **per-model pool spanning all three categories**. A
shard is a 1-task unit carrying its own `(category, dataset, agent_impl,
task)`; `unified_prep` emits a per-model flat matrix and one `eval` job
drains it. `max_parallel`/`model_parallel` are **derived** from
`concurrency`+`rollouts`; `MAX_SHARDS` 64→200 with proportional
per-category packing above the cap. Per-category aggregation keeps the
tier-2 scorecard (`aggregate_unified.py`) unchanged. `harbor.yml`'s
single-dataset path decouples `shard_parallel` from `n_shards` for
dynamic dispatch (no-op at the default `n_shards=10`).
## Validated (CI, docker sandbox, bare harness)
- **Lite single-model** (`gpt-5.6-terra`): complete scorecard across all
3 categories, macro pass@3 0.479 (context 0.0 → 0.625 after the fixes
below).
- **Multi-model** (`gpt-5.6-terra` + `anthropic:claude-sonnet-5`): both
per-model pools run concurrently under `model_parallel`, total-job guard
holds (68 shards), and the non-OpenAI agent runs with the OpenAI judge.
## Fixes folded in (found during the dry-runs)
- Context corpus populate + judge `OPENAI_API_KEY` now read the
per-shard `matrix.*` values (they were gated on the leaf inputs, which
the flat design leaves empty).
- LangSmith experiment name scoped per category (fixes the cross-dataset
run double-count).
## Dependency
Needs the harbor build-lock fix
([harbor-framework/harbor#2340](https://github.com/harbor-framework/harbor/pull/2340))
via `harbor_package_override` until it's released — pin
`nick-hollon-lc/harbor@27a6eac8` meanwhile.
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
## Summary
Hardens the unified multi-model Harbor evals workflow
(`unified_evals.yml`) and adds a model scorecard. Validated end-to-end
on `openai:gpt-5.6-luna` (docker sandbox, bare harness).
### Workflow / harness
- **`profile=lite`**: a frozen high-signal task subset (`lite_tasks.py`,
difficulty-frontier tasks; full rollouts, fewer tasks) selectable via
the `profile` input. Sharded one task per shard so GitHub's matrix
drains a dynamic queue (long pole is the slowest single task, and heavy
task images never concentrate on one runner). Roughly 8x faster and 6x
cheaper than the full run.
- **`agent_impl` input** (`dcode` | `bare`) to pick the harness for the
code categories; conversation stays on tau3.
- **Multi-spec `harbor_package_override`** (one pip spec per line) so
`harbor` and `harbor-langsmith` can be pinned to the same commit.
- **LangSmith sandbox credential isolation**: the sandbox authenticates
with a dedicated `LANGSMITH_SANDBOX_API_KEY`, passed via harbor's
`--environment-kwarg` (resolved from the host env at construction, so it
never lands on argv or in the serialized job/trial config), while
tracing/datasets/results stay on `LANGSMITH_API_KEY`. Requires a harbor
build with `--environment-kwarg ${VAR}` support
(harbor-framework/harbor#2344); langsmith-sandbox runs must therefore
set `harbor_package_override` until that releases.
- Experiments land on the **canonical shared datasets** (one dataset per
category, model-as-experiment) instead of per-model datasets.
### Fixes
- Use the **OpenAI Responses API** in the harbor langgraph agent, fixing
the `reasoning_effort` + tools 400 on Responses-only models.
- Supply **`JUDGE_*` config + `OPENAI_API_KEY`** to the harbor-index
verifier env for LLM-judge tasks, fixing `RewardFileNotFoundError`.
- Fix **`resolve_langsmith_api_key`** preferring
`LANGSMITH_SANDBOX_API_KEY` for tracing/dataset/results calls (which
routed them to the wrong tenant); those now use `LANGSMITH_API_KEY`.
### Docs
- `libs/evals/UNIFIED_SCORECARD.md`: GH aggregate pass@k / avg@k for
luna, with **Full (default)** and **Lite** tables plus per-run
conditions.
Release PR checks can start before the automatic curated-note draft is
published, immediately marking the PR red even though the draft is still
being generated. Applying those notes can also report a stale-head
failure after successfully updating the release branch because the
pull-request API may lag behind the branch ref.
This makes the release flow tolerate those expected ordering delays
without weakening its concurrency checks:
- The required check waits for the initial automatic draft on PR events,
distinguishes a valid but unapplied draft from missing or stale
metadata, and continues through transient GitHub read failures.
- Interrupted refresh runs explicitly close their in-progress check
instead of leaving it spinning indefinitely.
- Post-apply validation reads the canonical release branch ref while
continuing to reject PR-body edits, curated-draft edits, and concurrent
branch movement.
- The prerelease SDK-pin warning points maintainers to `release-deps:
acknowledged` when an intentional cross-package release sequence has not
reached PyPI yet.
Regression coverage exercises delayed draft publication, state changes
during polling, stale pull-request heads after apply, and concurrent
release-branch updates.
`deepagents-code` release PRs currently expose release-please's
generated changelog directly. Maintainers need a durable place to turn
those generated entries into concise, user-facing notes without losing
their edits the next time release-please refreshes the PR.
When a `deepagents-code` release PR moves from draft to ready for
review, the release bot posts a marked comment containing a polished
draft. Maintainers can edit that comment while preserving the generated
version heading, then run `@dcode-release-bot apply` to copy the
finalized section into both `libs/code/CHANGELOG.md` and the release PR
preview.
For a ready, non-bypassed release PR, the `curated release notes` check
binds the latest valid bot-authored draft to its applied metadata,
requires the current version section in `libs/code/CHANGELOG.md` and the
PR-body preview to match that draft, and verifies that the recorded
draft and apply commits remain in the current release branch's ancestry.
Unrelated descendant commits are allowed. If release-please regenerates
the curated section or adds generated entries, the check fails until a
maintainer runs `@dcode-release-bot draft` and then `@dcode-release-bot
apply` again. While a release PR is still a draft, the check passes
without validating curation and the manual commands are disabled. The
exact `release: dangerously skip curated notes` label remains an
explicit escape hatch for exceptional releases.
Drafting makes one non-agentic request to one of three fixed provider
API endpoints, using a repository-configured model. The model receives
isolated release-note input and no callable tools. The trusted helper
reads that prepared input, writes the draft output, and sends the
request using only the selected provider key; the output is structurally
validated and the release snapshot is rechecked before the draft is
published. Privileged curation changes use a short-lived installation
token minted from the existing organization-membership GitHub App, while
workflow feedback and refreshed check runs use the scoped
`GITHUB_TOKEN`. Manual commands require repository write access.
## Required repository setup
- Ensure the installed organization-membership GitHub App credentials
are available as repository secrets `ORG_MEMBERSHIP_APP_CLIENT_ID` and
`ORG_MEMBERSHIP_APP_PRIVATE_KEY`, and that the App grants read/write
access to contents, issues, and pull requests.
- Add repository variables `DCODE_RELEASE_BOT_LOGIN` and
`DCODE_RELEASE_BOT_ID` for the installed App's `<app-slug>[bot]`
identity.
- Configure the `release-dcode` environment without required reviewers
or other deployment approval rules. Set `DCODE_RELEASE_MODEL` to an
`openai:<model>`, `anthropic:<model>`, or `google_genai:<model>` value,
and add the matching provider API key as an environment secret
(`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, or `GOOGLE_API_KEY`).
- Add the literal `curated release notes` job name to `main`'s required
status checks. Without it, failures are visible but do not block a stale
or unapplied release PR from merging.
The release guide includes the exact identity lookup and setup
instructions.
Adds a README for the unified evals CI job and makes the deep-agents
harness configurable. The full rationale (benchmarks, categories, task
selection, and when to use this vs. the pytest eval suite) lives in
`libs/evals/UNIFIED_EVALS.md`.
## Code changes
- `unified_prep.py`: autonomous and context now default to the `bare`
`create_deep_agent` harness (conversation stays `tau3`). A new
`agent_impl` override selects `dcode` for the deep-agents categories
only, validated against `{bare, dcode}`.
- `unified_evals.yml`: new `agent_impl` choice input (default `bare`,
options `bare`/`dcode`) threaded to prep, plus a prep-job step that
writes a run-configuration summary of every dispatch input (with the
resolved model list and effective `shard_parallel`) to the run summary
for debuggability.
- `test_unified_prep.py`, `test_unified_workflow.py`: cover the bare
defaults, the `dcode` override applying only to the deep-agents
categories, invalid-harness rejection, and the new input / prep-env /
summary wiring.
- `libs/evals/UNIFIED_EVALS.md`: new doc.
Tests: 31 passed (`test_unified_prep.py`, `test_unified_workflow.py`).
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
Co-authored-by: Mason Daugherty <mason@langchain.dev>
Alpha releases are cut from throwaway branches, so the previous alpha
tag is a sibling of the next release commit rather than its ancestor.
The release workflow therefore rejected `deepagents==0.7.0a6` while
preparing `0.7.0a7` and fell back to the latest stable tag, producing
“Git log since deepagents==0.6.12”.
This makes prerelease resolution prefer the highest earlier prerelease
tag with the same base version. Sibling tags are accepted when they
share history with the release commit and are not ahead of it; the first
prerelease still falls back to the base-version or latest stable tag.
For example, `deepagents==0.7.0a7` now produces “Git log since
deepagents==0.7.0a6”.
The release setup summary also distinguishes an omitted `release-sha`
input from the canonical commit selected by the workflow, so alpha runs
show the exact commit they publish instead of only `(empty)`.
The workflow shell coverage exercises sibling branches, future-tag
rejection, prerelease phase and serial ordering, stable fallbacks,
malformed prereleases, and canonical release-target summaries.
A release-log boundary test creates 100 commits through separate `git
commit` subprocesses. A hosted runner failed partway through that setup
with exit 128, while the same job passed on adjacent runs.
Override the extracted workflow's `MAX_COMMITS` value to 3 in the two
boundary tests. They still cover exactly-at-limit and one-past-limit
behavior, and the helper asserts that the production setting remains
100, but setup now creates seven commits instead of 205.
The full workflow-helper suite passes with 434 tests, and the affected
tests passed 20 consecutive runs.
Open, non-draft pull requests are warned after fourteen days and
automatically closed after thirty days — measured from when the PR was
opened, not last activity — unless they carry the `do-not-close` label.
---
The existing cleanup only covered PRs opened by `app/open-swe`, warning
after seven inactive days and closing after fourteen. This broadens the
policy to all open, non-draft PRs and bases it on age since opening
rather than recent activity.
**Closing behavior**
- A PR is only ever closed once it already carries a warning comment
from this workflow, so every PR gets at least one warning cycle (~16
days) of notice. A PR already past thirty days that was never warned is
warned now and becomes eligible to close on a later run.
- The marker comment must be authored by `github-actions[bot]`; a marker
forged by a PR participant is ignored and can neither trigger nor block
a closure. Warnings are deduplicated across daily runs and upgraded in
place to the closing notice.
**Failure handling**
- An incomplete PR search, a non-transient error (auth/validation), or
every processed PR failing makes the run fail visibly.
- Isolated transient errors (rate limits / 5xx on a single PR) are
logged but don't fail the run, since they self-heal on the next daily
run.
Thresholds, the bypass label, and the max-PRs cap are configurable;
hitting the cap is surfaced with a warning.
Harbor evaluations currently require a separate workflow dispatch for
each model and benchmark. This makes cross-model comparisons cumbersome
and leaves Harbor without the multi-model orchestration already
available in `evals.yml`.
This PR adds a unified Harbor entry point that accepts a comma-separated
list of `provider:model` IDs and any combination of the autonomous,
conversation, and context categories.
| Before | After |
|---|---|
| Each dispatch evaluated exactly one model. | One dispatch evaluates a
list of models. |
| The dataset and agent implementation were selected manually for each
run. | Each category selects the appropriate benchmark and agent
configuration automatically. |
| Evaluating three categories across several models required a separate
run for every pair. | The workflow coordinates the complete model ×
category matrix. |
| Each run produced an independent result. | Results are combined into a
leaderboard, JSON summary, and radar chart. |
| Harbor execution was embedded in the single-model workflow. |
Single-model and unified dispatches share the same reusable Harbor
execution workflow. |
The categories map to the existing Harbor-backed evaluations:
- **Autonomous:** Harbor Index with the dcode harness
- **Conversation:** the tau3 subset with the conversational agent
- **Context:** the context-retrieval evaluation with the dcode harness
Models are grouped by provider so calls to a single provider are
serialized while different providers can run in parallel. Per-model and
global concurrency limits are enforced, and shard counts remain
configurable by category. Each `(model × category)` pair is tracked in
its own LangSmith dataset.
The final report includes pass@K and avg@K for each pair, a macro
overall that weights categories equally, and a micro overall weighted by
task count. The expected model × category grid is retained so missing or
incomplete runs are reported rather than silently omitted.
The existing single-model Harbor dispatch keeps its current interface.
Extracting its execution into a reusable workflow is an enabling
refactor; multi-model, cross-category orchestration and combined
reporting are the net-new functionality.
---------
Co-authored-by: Mason Daugherty <mason@langchain.dev>
Co-authored-by: Mason Daugherty <github@mdrxy.com>
GitHub release descriptions now include a collapsible, package-scoped
commit history.
---
GitHub releases currently use package changelog entries when available.
That leaves out commit-level package history.
This change appends a collapsible, package-scoped Git log to every
release body. The log uses the latest stable package tag that is
reachable from the release commit, so releases from version branches do
not accidentally compare against a newer tag from another line. Initial
and divergent histories are identified separately, and commit subjects,
log length, and the final release body are bounded to stay within
GitHub's limits.
## Summary
Reworks #4630: instead of a separate 783-line workflow fork, make Harbor
Index a dataset in the shared `harbor.yml` and add general per-dispatch
shard aggregation.
- **Dataset option** — adds `harbor-index/harbor-index-1.0` to the
`dataset` dropdown. It runs through the existing model matrix / sharding
/ LangSmith tracing / concurrency like any other dataset (no fork).
- **`aggregate` job** — merges the per-shard result artifacts
(`shard-*`) via `.github/scripts/aggregate_shards.py` into:
- `summary.json` + `per_task.jsonl` reporting **`pass@K`** (fraction of
tasks that passed ≥1 of K rollouts) and **`avg@K`** (passing rollouts /
total rollouts), where K = `rollouts_per_task` (keys are serialized
dynamically, e.g. `pass@3` / `avg@3`);
- **one combined zip** per run (`harbor-combined-<dataset-slug>`), after
which the per-shard intermediates are deleted.
- **Eval-a-model defaults + caps** — defaults set to rollouts 3 /
concurrency 4 / n_shards 10; a `prep` step caps n_shards ≤ 10,
concurrency ≤ 4, and models × shards × concurrency ≤ 40. Per-shard
uploads renamed `harbor-*` → `shard-*` so the aggregate job can glob
them.
This is the reusable building block for a future "measure a model across
3 categories" harness (autonomous / conversation / retrieval): each
category is a `harbor.yml` run emitting its own `summary.json`, which an
orchestrator can roll into an overall score.
## Testing
- Unit tests `.github/scripts/test_aggregate_shards.py` (run by CI's
`check-release-options` job): 8/8 pass.
- End-to-end smoke run on this branch (haiku / langsmith / 2 shards /
`n_tasks=2`): `prep` → 2 shards → `aggregate` all green; produced a
correct `summary.json` (`pass@2` / `avg@2`) and a single combined zip,
with the per-shard intermediates deleted. Workflow plumbing validated;
the two sampled tasks errored for the reasons in **Out of scope** below.
## Out of scope / follow-ups
- Aggregation is **single-model** for now (errors on >1 model);
multi-model grouping and the caps/defaults interaction are deferred.
- The `harbor-index/bix-ordinal-logit-covid` task errors with
`EnvironmentStartTimeoutError` on a **cold** build — its 7 GB
`chenzizhao/bixbench` base image plus a build-time HuggingFace capsule
download exceed the task's own 1800s env-start budget. That's a
**separate dataset/infra issue** (mirror + digest-pin the base image,
reuse/pre-warm the snapshot, or pre-bake per-task images) and is not
addressed here.
Supersedes #4630.
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
Only include the model integrations that the model list needs to improve
the performance of Harbor jobs.
Also fixes an issue where non-Fireworks jobs failed to install because
of the Fireworks prerelease dependency: `langchain-fireworks` was being
installed every time but without `prerelease=allow`.
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
The Deep Agents Code GitHub Action now supports headless options for
model configuration, skills, startup commands, sandboxes, MCP,
interpreter tools, turn and task limits, rubric grading, stdin, quiet
output, buffered output, and JSON output.
---
The root GitHub Action exposed only a small subset of headless `dcode`
options, so workflows could not configure skills, sandboxes, MCP,
interpreter tools, rubric grading, model parameters, or output modes
without bypassing the Action.
This expands the Action input surface to match supported headless
`dcode` behavior while deliberately keeping interactive-only
`--auto-approve` out of workflows. Inputs are assembled as shell
argument arrays, typed values are validated before launch, stdin and
timeout handling preserve the agent's actual exit code, and memory
caching now includes SQLite WAL files. `ACTION.md` documents the common
inputs, outputs, memory behavior, and the need to treat agent output as
untrusted.
The Action tests compare forwarded flags with the CLI parser and execute
the real command-assembly shell against stubbed processes, covering
validation, timeout arithmetic, stdin behavior, and exit-code
propagation.
SDK pin warning comments now make the pinned `deepagents` version
clickable, pointing reviewers directly at the matching GitHub release
tag. That keeps prerelease/stale-pin review context one click away from
the advisory comment.
## Changes
- Build a GitHub release-tag URL for the currently pinned
`deepagents==<version>` value in the SDK pin warning workflow.
- Render the pin as a Markdown link in stale-pin and prerelease-pin
warning comments.
- Add coverage that checks the workflow’s embedded `github-script`
contract links the pin to the encoded release tag and interpolates that
link into warning bodies.
- **`agent_timeout_multiplier`** — scale the agent EXECUTION timeout
(e.g. `0.1` to bound per-rollout wall-clock for cheap concurrency stress
tests; setup/env-build budgets unaffected).
- **`disable_verification`** — skip the task verifier (tests) for
setup/concurrency stress runs.
- **`n_shards`** — split the dataset's tasks across N parallel shard
jobs per model
- **`--reinstall --refresh`** on the `harbor_package_override` install,
so re-running the same git ref picks up a freshly-built wheel instead of
a stale cached one
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
Release-please PRs can touch generated/version artifacts across multiple
package directories, which does not fit the normal “title scope covers
touched package dirs” rule. The scope/file gate now stands down only for
release-shaped PRs whose entire changeset is limited to known
release-please artifacts, so ordinary source edits still trigger the
mismatch check.
Reduces the "green-local / red-CI" friction for `libs/code`
contributors. Adds a single `make bootstrap` (sync deps + install git
hooks) and `make check` (run the CI gauntlet locally: lint, types,
imports, tests, plus the cross-cutting repo gates). Also closes two
pre-commit gaps where the `extras-sync` hook only covered `libs/cli` and
the `deepagents` SDK pin had no local check at all (the new check is
advisory, matching the existing workflow's nature).
Made by [Open SWE](https://openswe.vercel.app)
---------
Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
Follow-up hardening for the two scope-check merge gates
(`release_please_scope_check.yml`, `pr_scope_file_check.yml`), applying
review findings from the trusted-base work in #4175 / #4171.
- **Make a silently-disabled gate visible.** The detector-absent branch
(bootstrapping PR or a branch cut before the gate existed) is a
deliberate fail-open. If the detector is ever renamed or moved on `base`
without updating the hard-coded path, that same branch would silently
stop enforcing. Escalating its `::notice::` to `::warning::` surfaces
the desync in the Checks UI instead of only the fold-out log. The gate's
pass/fail logic is unchanged.
- **Scope the trust-model comment.** The base checkout closes only the
detector/config edit vector; under `pull_request` the workflow file
itself comes from the PR head, so integrity also depends on branch
protection plus review of `.github/workflows/` edits. The comment now
says so rather than implying the gate is un-bypassable.
- **Document the base-config staleness tradeoff.** Running the detector
and config from `base` means a PR that itself adds a new package (or
labeler rule) is not checked against it until it lands on base. Reading
head config would reopen the bypass the base checkout closes, so this is
intentional, not a regression.
- **Better diagnostics on swallowed errors.** The three `core.warning`
fallbacks now include `[status=…]`, matching the already-merged
`release_please_initial_baseline_check.yml`.
- **Lock the wire contract.** Added a test asserting the detector prints
exactly `[]` on a clean PR — the contract the workflow's fail-closed
JSON parse and the detector-absent sentinel both rely on.
Adds a pre-merge CI check that compares the package scopes declared by a
PR title with the package directories touched by the PR. The check uses
the existing PR labeler config as its source of truth, so scope aliases
like `code`/`deepagents-code` and directory mappings like `libs/code/`
stay aligned with the rest of the repo automation.
When a PR declares one package scope but touches a different package
directory, the workflow posts a sticky warning comment and fails. If the
mismatch is intentional, maintainers can apply `allow-scope-mismatch` to
downgrade the block to an informational note while keeping the mismatch
visible.
The detector is covered by unit tests for matching scope/file pairs, the
`fix(cli)` + `libs/code/**` mismatch, multi-scope titles, unscoped
scopes/paths, and config-read fail-closed behavior.
## Summary
- Adds `fireworks:accounts/fireworks/models/glm-5p2` (GLM-5.2) to the
`open-fireworks` and `fireworks` model groups
- Adds `openrouter:z-ai/glm-5.2` (GLM-5.2) to the `open` and
`openrouter` model groups
- Updates the `models:` dropdown in both `evals.yml` and `harbor.yml`
- Regenerates `MODEL_GROUPS.md` via `make model-groups`
## Test plan
- [ ] All 27 `test_models.py` tests pass locally (verified)
- [ ] Dispatch **📊 Evals** with `models_override:
fireworks:accounts/fireworks/models/glm-5p2` and
`eval_categories_exclude: memory`
- [ ] Dispatch **📊 Evals - Harbor** (Terminal Bench) with the same model
override
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Switches terminal-bench Harbor runs to Harbor's LangGraph agent using
the Deep Agents Code harness, wires the LangSmith plugin and runtime env
propagation, removes the custom wrapper/backend, and points Harbor at
the integration fork for plugin validation.
---------
Co-authored-by: Mason Daugherty <mason@langchain.dev>
New packages get added to `.release-please-manifest.json`, but that file
stores each package's *last released* version — not its desired first
release. Entering a brand-new package at `0.0.1` makes release-please
treat `0.0.1` as already shipped and open the first release PR at
`0.0.2`, silently skipping the package's actual first release. This adds
a pre-merge gate that catches that mistake and documents the correct
convention.
Add first-party support for Vercel Sandbox as a sandbox provider, mostly
following the Daytona and Modal providers.
Release-As: 0.0.1
---------
Co-authored-by: Mason Daugherty <mason@langchain.dev>
Co-authored-by: Mason Daugherty <github@mdrxy.com>
Because `release-please-config.json` sets `separate-pull-requests:
true`, a release PR only ever bumps a single package, so the
same-PR-bump detection and dependency-pin stripping never fired (the
bump set was always just the released package, which can't depend on
itself). This removes that dead machinery (`detect_package_bumps`,
`PackageBump`, the requirement/pin filtering) and resolves each changed
release manifest directly against real PyPI after stripping local
`[tool.uv.sources]`. Supersedes #3885.
Note: the `.github/workflows/check_release_deps.yml` header comment
still describes the old behavior; it couldn't be updated here because
the bot lacks `workflows` push permission.
## Test Plan
- [ ] `build_resolver_manifest` keeps all deps and drops
`[tool.uv.sources]` while preserving resolver `[tool.uv]` keys
- [ ] `check_release_dependencies` resolves each changed manifest and is
a no-op when none changed
Made by [Open SWE](https://openswe.vercel.app)
---------
Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
Lockfile validation now uses a single command builder for both the `uv
lock --check` invocation and the recovery command shown to contributors.
Failures point at the stale `uv.lock` with a GitHub annotation and the
exact repo-root command needed to regenerate it.
Lockfile validation now follows the same package boundary as the PR diff
instead of treating every package in the repo as in scope. That keeps
unrelated stale lockfiles from blocking scoped work or forcing extra
path changes that release-please may interpret as release-relevant.
Adds a release dependency check for release-please PRs that resolves
each changed package manifest against real PyPI while accounting for
coordinated same-PR package bumps. This prevents false failures when a
release PR pins another package version that is being published by the
same merge, while still catching unpublished or unsatisfiable runtime
dependency pins.
The check also supports a maintainer acknowledgement label
(`release-deps: acknowledged`), preserves resolver-affecting `[tool.uv]`
settings, drops `[tool.uv.sources]`, and prints resolver logs with a
transient network/index warning when appropriate.
`_touches_talon()` excluded `libs/talon/uv.lock`, so a PR changing only
that lockfile skipped Talon's `uv lock --check`, allowing a
stale/malformed lockfile to pass pre-commit. Now any `libs/talon` path
(including `uv.lock`) counts as touching Talon, while unrelated changes
still skip Talon to preserve the optimization.
Made by [Open SWE](https://openswe.vercel.app)
Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
Cached-session MCP tools now match `langchain-mcp-adapters` 0.3 behavior
for MCP execution errors: `CallToolResult(isError=True)` is returned to
the model as a failed tool result instead of escaping as a raised
`ToolException`. Transport, session, retry, and reauth failures keep
their existing exception paths, while the generic recovery middleware
remains documented as a compatibility fallback for
non-MCP/tool-originated `ToolException`s.
Add `deepagents-talon`, a local single-operator runtime host for
long-running Deep Agents over channels and scheduled jobs. The package
wires a Deep Agents runtime to channel adapters, cron tools, MCP
loading, optional voice transcription, LangSmith trace context, and a
runnable WhatsApp Docker Compose example.
This first PR focused is on the local host, WhatsApp adapter, scheduler,
and CI/release plumbing while reusing existing `deepagents-code` web and
MCP utilities (where practical).
Areas that need careful review:
- Whether the generic channel abstractions should stay in the first PR
or be narrowed to WhatsApp-only.
- WhatsApp exposure defaults, session storage, downloaded media
retention, and open-mode acknowledgement.
- The dependency boundary with `deepagents-code`, especially MCP helpers
and web tools.
---------
Co-authored-by: Deep Agent <agent@deepagents.dev>
Co-authored-by: Mason Daugherty <github@mdrxy.com>
Adds a pull-request gate for release-please fan-out caused by
regenerated package lockfiles. The check catches bump-worthy PRs that
would accidentally create release PRs for managed packages whose only
change is `uv.lock`, while still allowing intentional lockfile-only
releases through an explicit bypass label.
The extras-sync check previously ran against two hardcoded
`pyproject.toml` paths (`libs/cli` and `libs/code`). This broadens it to
auto-discover every `pyproject.toml` under `libs/` via a `find` loop, so
new partner packages are covered automatically without editing the
workflow. The script already no-ops on packages without
`[project.optional-dependencies]`, so the wider sweep is safe.
The same PR opportunistically hardens several other CI workflows that
were touched during investigation.
Synchronize the PR labeler and package auto-labeler configurations to
assign the `dcode` label (instead of `deepagents-code`) to the
`deepagents-code` package.
Adds a scheduled workflow that reminds on and eventually closes stale
PRs opened by `app/open-swe`. Maintainers can opt a PR out of cleanup by
adding the `do-not-close` label.
## Changes
- **Scheduled workflow** (`close_open_swe_prs.yml`) runs daily, posting
a reminder comment after 7 days of inactivity and closing the PR after
14 days total
- **Cleanup script** (`close-open-swe-prs.js`) with idempotent comment
handling: detects existing marker comments and updates them rather than
creating duplicates, and refreshes the reminder timer when new
maintainer activity occurs after the initial warning
- **Bypass mechanism** via a configurable `do-not-close` label that is
auto-created if missing; drafts and already-closed PRs are skipped
automatically
- **Resilient error handling**: individual PR failures are logged and
the workflow fails at the end so one bad PR doesn't block cleanup of the
rest; search pagination timeouts are handled gracefully
Pin upper bounds on all `langchain-*`, `harbor`, `langsmith`, and
`deepagents` dependencies in `libs/evals/pyproject.toml` to prevent
unexpected breakage from major-version bumps. Also reorder the Fireworks
model entries in the eval registry and workflow dispatch lists, and drop
`--locked` from `uv sync` calls in CI workflows since `UV_FROZEN=true`
already enforces lockfile fidelity and uv 0.11 rejects combining both
flags.
Add human-readable display labels (`display_name`, `provider_label`) to
every `Model` in the registry so that chart legends, doc headings, and
workflow dropdowns render curated names instead of raw spec slugs like
`anthropic:claude-sonnet-4-6`. Also prune outdated/removed models across
Anthropic, Ollama, OpenAI, and Fireworks, and sync the
`evals.yml`/`harbor.yml` dropdowns to match.
Add an exclude filter to the eval runner so specific categories can be
skipped without having to enumerate every other category. Previously,
running "all categories except memory" required listing every category
individually in the include filter.
Drop the numeric index prefix from `_artifact_key` and enforce
uniqueness through dedupe + collision detection instead. The old
zero-padded index (`000-`, `001-`, ...) was a blunt disambiguator that
made artifact names harder to scan — since `_resolve_models` already
controls what specs enter the matrix, uniqueness is better guaranteed at
the source.
## Changes
- Remove the `index` parameter from `_artifact_key` and `_matrix_entry`,
dropping the `{index:03d}-` prefix so artifact keys are just the
slugified spec (e.g., `anthropic-claude-sonnet-4-6` instead of
`000-anthropic-claude-sonnet-4-6`)
- Add order-preserving dedupe (`dict.fromkeys`) to `_resolve_models`
across both the preset and manual code paths, preventing duplicate specs
from ever reaching `_matrix_outputs`
- Add a collision-detection tripwire in `_matrix_outputs` that raises
`ValueError` with the offending slug and specs if two distinct specs
slugify to the same key — defense-in-depth in case dedupe is ever
bypassed
Fix a falsy-string bug in `analyze_eval_failures.py` where an empty
`ANALYSIS_MODEL` env var (common when a GitHub Actions workflow input is
unset) passed straight through to `init_chat_model("")`, producing a
delayed crash instead of falling back to the default model.
Replace the single `eval` matrix job — which used GHA `concurrency` to
serialize calls per provider but silently dropped queued runs beyond the
first — with one `workflow_call` job per provider, each running its own
matrix at `max-parallel: 1`. The new shape gives every provider a real
serial queue while preserving cross-provider parallelism, so big
matrices no longer lose work to `concurrency` cancellation.
## Changes
- Fan out the old `eval` job into eleven `eval-<provider>` jobs in
`evals.yml`, each calling a new reusable `_eval.yml` workflow at
`max-parallel: 1`. Cross-provider runs stay parallel.
- Extract the per-model eval body (pytest, results-to-summary heredoc,
optional `analyze_eval_failures.py`, artifact uploads) into
`.github/workflows/_eval.yml` as a `workflow_call` workflow. Inputs are
validated upstream by `_resolve_models`/`_SAFE_SPEC_RE`.
- Add `_matrix_outputs`, `_matrix_entry`, `_provider`, `_artifact_key`
helpers to `.github/scripts/models.py`. `_matrix_outputs` partitions
selected models into `<provider>_matrix`/`<provider>_has_models` outputs
plus a catch-all `other` bucket for `models_override` specs whose
provider isn't enumerated in `_EVAL_PROVIDER_OUTPUTS`. `_artifact_key`
produces stable, globally unique upload names
(`000-anthropic-claude-sonnet-4-6`) so partitioned matrices don't
collide.
- Broaden the helper-script validation job in `ci.yml` from
`test_release_options.py` to `.github/scripts/test_*.py`, so future
helpers are picked up automatically.
- Replace the fragile `EVAL_OUTCOME: contains(toJSON(needs),
'"failure"')` substring scan in the aggregate step with explicit
`needs.eval-*.result` enumeration, and add a `continue-on-error` +
warning fallback to `Generate radar chart` so a chart-generation crash
no longer silently skips uploads.
- Pass `MODEL` into `make evals` via an env var instead of direct
expression interpolation, narrow the `except Exception` in the summary
heredoc to specific exception types with a `traceback.format_exc()`
warning, and gate `Upload failure analysis` on `inputs.analyze_failures`
to drop the spurious `if-no-files-found: warn` on every non-opt-in run.
Add a `docs` model preset to the eval/harbor workflow registry that
mirrors the 7 models showcased in the Deep Agents documentation's "Model
evaluations" table. Lets contributors reproduce the table with a single
dispatch (`gh workflow run evals.yml -f models=docs`) instead of
copy-pasting a `models_override` string.
Bundle three CI improvements onto one branch: a new pre-merge gate that
catches release-please parser failures (the class of bug that silently
dropped #2892 from its release PR), branch-name-based PR labeling, and a
guard so the issue-link workflow stops re-firing on merged PRs.
## Changes
**Release-please parse check** (new workflow
`release_please_parse_check.yml`)
- Run `@conventional-commits/parser@0.4.1` on the would-be squash-merge
message (`<title> (#<num>)\n\n<body>`) for every PR. Mirror
release-please's `preprocessCommitMessage` (override-block extraction)
and `splitMessages` (split on nested-commit markers and on
`\n\n(?=type(scope)?:)`), then parse each sub-message independently —
first failure wins.
- On parse failure: post a sticky PR comment with the parser error, the
offending line text, and a paste-ready override-block scaffold derived
from the PR title. On success: delete any stale failure comment.
Comment-post failures (fork PRs, rate limits) fall back to
`core.summary` so the remediation block is still recoverable.
- Hard-fail when the body contains more than one literal override-begin
marker. release-please uses only the first occurrence; if the first one
is a prose mention rather than a real block, the extracted "override" is
garbage and the commit is silently dropped from the changelog. The check
refuses to guess which marker is real and asks the author to dedupe
(rephrase prose mentions, or break the literal with `<wbr>`).
- Parser version is exact-pinned (not a `^` range) and must move in
lock-step with `release-please/package.json`.
**Branch-name PR labeling** (`pr-labeler.js`, both labeler workflows,
config)
- Add `branchRules` to `pr-labeler-config.json` and a
`matchBranchLabels` matcher in `pr-labeler.js` supporting `prefix` /
`suffix` / `exact` / `pattern` clauses. Wire it into both
`pr_labeler.yml` (online) and `pr_labeler_backfill.yml` (manual
backfill).
- Seed with one rule: branches starting with `open-swe/` get the
`open-swe` label. Branch ref is immutable for an existing PR, so this
matcher only adds — no removal logic needed.
**Issue-link enforcement** (`require_issue_link.yml`)
- Skip the workflow when `github.event.pull_request.merged == true`.
Editing a merged PR's title/body or shuffling labels was re-firing the
enforcement (label add, comment, no-op close, failed check).
Closed-but-not-merged PRs still flow through so a contributor editing
the body to add `Fixes #N` triggers a reopen.
**Docs** (`RELEASING.md`, `AGENTS.md`)
- Move the override-block workflow to `RELEASING.md` under
Troubleshooting and leave a one-line pointer in `AGENTS.md`. Document
both reasons to use the override (wrong type/scope inferred, or
parser-hostile body). Note the squash-merge-only constraint, the
multi-message form requirements, and the `gh api -f` flag for multi-line
bodies. List the new parse check under "PR labeling and linting".
Mason Daugherty