## Summary
Reworks #4630: instead of a separate 783-line workflow fork, make Harbor
Index a dataset in the shared `harbor.yml` and add general per-dispatch
shard aggregation.
- **Dataset option** — adds `harbor-index/harbor-index-1.0` to the
`dataset` dropdown. It runs through the existing model matrix / sharding
/ LangSmith tracing / concurrency like any other dataset (no fork).
- **`aggregate` job** — merges the per-shard result artifacts
(`shard-*`) via `.github/scripts/aggregate_shards.py` into:
- `summary.json` + `per_task.jsonl` reporting **`pass@K`** (fraction of
tasks that passed ≥1 of K rollouts) and **`avg@K`** (passing rollouts /
total rollouts), where K = `rollouts_per_task` (keys are serialized
dynamically, e.g. `pass@3` / `avg@3`);
- **one combined zip** per run (`harbor-combined-<dataset-slug>`), after
which the per-shard intermediates are deleted.
- **Eval-a-model defaults + caps** — defaults set to rollouts 3 /
concurrency 4 / n_shards 10; a `prep` step caps n_shards ≤ 10,
concurrency ≤ 4, and models × shards × concurrency ≤ 40. Per-shard
uploads renamed `harbor-*` → `shard-*` so the aggregate job can glob
them.
This is the reusable building block for a future "measure a model across
3 categories" harness (autonomous / conversation / retrieval): each
category is a `harbor.yml` run emitting its own `summary.json`, which an
orchestrator can roll into an overall score.
## Testing
- Unit tests `.github/scripts/test_aggregate_shards.py` (run by CI's
`check-release-options` job): 8/8 pass.
- End-to-end smoke run on this branch (haiku / langsmith / 2 shards /
`n_tasks=2`): `prep` → 2 shards → `aggregate` all green; produced a
correct `summary.json` (`pass@2` / `avg@2`) and a single combined zip,
with the per-shard intermediates deleted. Workflow plumbing validated;
the two sampled tasks errored for the reasons in **Out of scope** below.
## Out of scope / follow-ups
- Aggregation is **single-model** for now (errors on >1 model);
multi-model grouping and the caps/defaults interaction are deferred.
- The `harbor-index/bix-ordinal-logit-covid` task errors with
`EnvironmentStartTimeoutError` on a **cold** build — its 7 GB
`chenzizhao/bixbench` base image plus a build-time HuggingFace capsule
download exceed the task's own 1800s env-start budget. That's a
**separate dataset/infra issue** (mirror + digest-pin the base image,
reuse/pre-warm the snapshot, or pre-bake per-task images) and is not
addressed here.
Supersedes #4630.
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
Add human-readable display labels (`display_name`, `provider_label`) to
every `Model` in the registry so that chart legends, doc headings, and
workflow dropdowns render curated names instead of raw spec slugs like
`anthropic:claude-sonnet-4-6`. Also prune outdated/removed models across
Anthropic, Ollama, OpenAI, and Fireworks, and sync the
`evals.yml`/`harbor.yml` dropdowns to match.
Drop the numeric index prefix from `_artifact_key` and enforce
uniqueness through dedupe + collision detection instead. The old
zero-padded index (`000-`, `001-`, ...) was a blunt disambiguator that
made artifact names harder to scan — since `_resolve_models` already
controls what specs enter the matrix, uniqueness is better guaranteed at
the source.
## Changes
- Remove the `index` parameter from `_artifact_key` and `_matrix_entry`,
dropping the `{index:03d}-` prefix so artifact keys are just the
slugified spec (e.g., `anthropic-claude-sonnet-4-6` instead of
`000-anthropic-claude-sonnet-4-6`)
- Add order-preserving dedupe (`dict.fromkeys`) to `_resolve_models`
across both the preset and manual code paths, preventing duplicate specs
from ever reaching `_matrix_outputs`
- Add a collision-detection tripwire in `_matrix_outputs` that raises
`ValueError` with the offending slug and specs if two distinct specs
slugify to the same key — defense-in-depth in case dedupe is ever
bypassed
Replace the single `eval` matrix job — which used GHA `concurrency` to
serialize calls per provider but silently dropped queued runs beyond the
first — with one `workflow_call` job per provider, each running its own
matrix at `max-parallel: 1`. The new shape gives every provider a real
serial queue while preserving cross-provider parallelism, so big
matrices no longer lose work to `concurrency` cancellation.
## Changes
- Fan out the old `eval` job into eleven `eval-<provider>` jobs in
`evals.yml`, each calling a new reusable `_eval.yml` workflow at
`max-parallel: 1`. Cross-provider runs stay parallel.
- Extract the per-model eval body (pytest, results-to-summary heredoc,
optional `analyze_eval_failures.py`, artifact uploads) into
`.github/workflows/_eval.yml` as a `workflow_call` workflow. Inputs are
validated upstream by `_resolve_models`/`_SAFE_SPEC_RE`.
- Add `_matrix_outputs`, `_matrix_entry`, `_provider`, `_artifact_key`
helpers to `.github/scripts/models.py`. `_matrix_outputs` partitions
selected models into `<provider>_matrix`/`<provider>_has_models` outputs
plus a catch-all `other` bucket for `models_override` specs whose
provider isn't enumerated in `_EVAL_PROVIDER_OUTPUTS`. `_artifact_key`
produces stable, globally unique upload names
(`000-anthropic-claude-sonnet-4-6`) so partitioned matrices don't
collide.
- Broaden the helper-script validation job in `ci.yml` from
`test_release_options.py` to `.github/scripts/test_*.py`, so future
helpers are picked up automatically.
- Replace the fragile `EVAL_OUTCOME: contains(toJSON(needs),
'"failure"')` substring scan in the aggregate step with explicit
`needs.eval-*.result` enumeration, and add a `continue-on-error` +
warning fallback to `Generate radar chart` so a chart-generation crash
no longer silently skips uploads.
- Pass `MODEL` into `make evals` via an env var instead of direct
expression interpolation, narrow the `except Exception` in the summary
heredoc to specific exception types with a `traceback.format_exc()`
warning, and gate `Upload failure analysis` on `inputs.analyze_failures`
to drop the spurious `if-no-files-found: warn` on every non-opt-in run.
Nick Hollon