mirror of
https://github.com/langchain-ai/deepagents.git
synced 2026-07-21 17:25:26 -04:00
Nick Hollon
671c578006
## What Replace the serialized per-provider / per-category eval jobs with one dynamically-drained **per-model pool spanning all three categories**. A shard is a 1-task unit carrying its own `(category, dataset, agent_impl, task)`; `unified_prep` emits a per-model flat matrix and one `eval` job drains it. `max_parallel`/`model_parallel` are **derived** from `concurrency`+`rollouts`; `MAX_SHARDS` 64→200 with proportional per-category packing above the cap. Per-category aggregation keeps the tier-2 scorecard (`aggregate_unified.py`) unchanged. `harbor.yml`'s single-dataset path decouples `shard_parallel` from `n_shards` for dynamic dispatch (no-op at the default `n_shards=10`). ## Validated (CI, docker sandbox, bare harness) - **Lite single-model** (`gpt-5.6-terra`): complete scorecard across all 3 categories, macro pass@3 0.479 (context 0.0 → 0.625 after the fixes below). - **Multi-model** (`gpt-5.6-terra` + `anthropic:claude-sonnet-5`): both per-model pools run concurrently under `model_parallel`, total-job guard holds (68 shards), and the non-OpenAI agent runs with the OpenAI judge. ## Fixes folded in (found during the dry-runs) - Context corpus populate + judge `OPENAI_API_KEY` now read the per-shard `matrix.*` values (they were gated on the leaf inputs, which the flat design leaves empty). - LangSmith experiment name scoped per category (fixes the cross-dataset run double-count). ## Dependency Needs the harbor build-lock fix ([harbor-framework/harbor#2340](https://github.com/harbor-framework/harbor/pull/2340)) via `harbor_package_override` until it's released — pin `nick-hollon-lc/harbor@27a6eac8` meanwhile. --------- Co-authored-by: Mason Daugherty <github@mdrxy.com>
43 lines
1.6 KiB
Python
43 lines
1.6 KiB
Python
import os, sys
|
|
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
|
import enumerate_tasks as et # noqa: E402
|
|
|
|
|
|
def test_local_task_names_lists_dirs_with_task_toml(tmp_path):
|
|
(tmp_path / "cb-cloud-1").mkdir()
|
|
(tmp_path / "cb-cloud-1" / "task.toml").write_text("")
|
|
(tmp_path / "cb-cloud-0").mkdir()
|
|
(tmp_path / "cb-cloud-0" / "task.toml").write_text("")
|
|
(tmp_path / "not-a-task").mkdir() # no task.toml -> excluded
|
|
(tmp_path / "dataset.toml").write_text("") # file, not a task dir
|
|
assert et.local_task_names(str(tmp_path)) == ["cb-cloud-0", "cb-cloud-1"]
|
|
|
|
|
|
def test_local_task_names_empty_raises(tmp_path):
|
|
import pytest
|
|
with pytest.raises(SystemExit, match="No local Harbor tasks"):
|
|
et.local_task_names(str(tmp_path))
|
|
|
|
|
|
def test_tau3_subset_is_registered_synthetic():
|
|
# "tau3-subset" is not an org/name registry package, so it must resolve via
|
|
# a synthetic resolver rather than the registry client.
|
|
assert "tau3-subset" in et.SYNTHETIC_DATASETS
|
|
|
|
|
|
def test_main_routes_synthetic_dataset_not_registry(tmp_path, monkeypatch):
|
|
import pytest
|
|
|
|
out = tmp_path / "names.txt"
|
|
monkeypatch.setitem(et.SYNTHETIC_DATASETS, "tau3-subset", lambda: ["a", "b", "c"])
|
|
monkeypatch.setenv("ENUM_DATASET", "tau3-subset")
|
|
monkeypatch.setenv("ENUM_OUTPUT", str(out))
|
|
monkeypatch.delenv("ENUM_DATASET_PATH", raising=False)
|
|
monkeypatch.setattr(
|
|
et,
|
|
"registry_task_names",
|
|
lambda _ref: pytest.fail("registry must not be queried for a synthetic dataset"),
|
|
)
|
|
assert et.main() == 0
|
|
assert out.read_text().split() == ["a", "b", "c"]
|