Douwe Maan 1c96491fe8 Speed up CI by fixing the slowest tests, balancing shards by measured duration, and guarding test cost (#10054)

2026-10-09 14:05:31 -06:00

..

cassettes/test_typesafe_judges

Add supports_text_output to ModelProfile, and run LLMJudge and GEval on a model without it (#8480)

2026-09-18 15:18:17 -06:00

__init__.py

Add pydantic-evals package (#935)

2025-03-28 14:55:29 +00:00

conftest.py

feat: online evaluation infrastructure for pydantic-evals (#4782)

2026-03-30 20:04:27 -06:00

test_agentic_evaluators.py

Target py311 in Ruff and modernize compatible syntax (#9854)

2026-10-06 07:07:09 -04:00

test_dataset.py

Target py311 in Ruff and modernize compatible syntax (#9854)

2026-10-06 07:07:09 -04:00

test_evaluator_base.py

Bump anyio dev floor to 4.11 and run async tests via anyio_mode = "auto" (#8874)

2026-09-28 08:45:29 +00:00

test_evaluator_common.py

Bump anyio dev floor to 4.11 and run async tests via anyio_mode = "auto" (#8874)

2026-09-28 08:45:29 +00:00

test_evaluator_context.py

Target py311 in Ruff and modernize compatible syntax (#9854)

2026-10-06 07:07:09 -04:00

test_evaluator_spec.py

Bump anyio dev floor to 4.11 and run async tests via anyio_mode = "auto" (#8874)

2026-09-28 08:45:29 +00:00

test_evaluators.py

Bump anyio dev floor to 4.11 and run async tests via anyio_mode = "auto" (#8874)

2026-09-28 08:45:29 +00:00

test_llm_as_a_judge.py

Bump anyio dev floor to 4.11 and run async tests via anyio_mode = "auto" (#8874)

2026-09-28 08:45:29 +00:00

test_multi_run.py

Bump anyio dev floor to 4.11 and run async tests via anyio_mode = "auto" (#8874)

2026-09-28 08:45:29 +00:00

test_online_capability.py

Populate span_tree and metrics for run_on_errors evaluators when a task raises (#7070)

2026-10-06 16:02:15 -05:00

test_online.py

Speed up CI by fixing the slowest tests, balancing shards by measured duration, and guarding test cost (#10054)

2026-10-09 14:05:31 -06:00

test_otel_emit.py

Online evaluation via OpenTelemetry events (#5125)

2026-04-21 08:34:14 -06:00

test_otel.py

Populate span_tree and metrics for run_on_errors evaluators when a task raises (#7070)

2026-10-06 16:02:15 -05:00

test_render_numbers.py

Render non-finite eval metrics as inf/-inf/nan (#8852)

2026-09-30 21:12:55 -05:00

test_report_evaluators.py

Use lightweight snapshot imports in evaluator tests (#9966)

2026-10-07 11:46:00 -04:00

test_reporting.py

Stop EvaluationReport.render() and print() from parsing case, task and evaluator text as Rich markup (#9544)

2026-10-09 09:56:04 -06:00

test_reports.py

Bump anyio dev floor to 4.11 and run async tests via anyio_mode = "auto" (#8874)

2026-09-28 08:45:29 +00:00

test_typesafe_judges.py

Bump anyio dev floor to 4.11 and run async tests via anyio_mode = "auto" (#8874)

2026-09-28 08:45:29 +00:00

test_utils.py

Target py311 in Ruff and modernize compatible syntax (#9854)

2026-10-06 07:07:09 -04:00

utils.py

Fall back to ASCII in EvaluationReport.print() on non-UTF-8 consoles (#7290)

2026-08-13 09:35:57 -05:00