mirror of
https://github.com/Heretek-AI/AetherTable.git
synced 2026-11-11 23:55:23 -10:00
John SmithandClaude
30850a5e59
pytest 'live' marker + conftest auto-skip unless LLM_KEY is set AND RUN_LIVE_LLM=1 — structurally CI-proof. Six tests exercise the REAL endpoint: mechanical/safety classification provenance, Concordia generator-marker-vs-reality (a template marker while the endpoint is alive FAILS), strict-JSON parsing, soft 60s latency budget, and a gate proving non-mock configuration. Connection failures degrade to SKIP. First live run surfaced two real integration bugs, reported honestly as skips rather than fabricated passes: .env named a nonexistent model that hangs instead of 404ing (fixed in .env), and the endpoint returns text/event-stream even for non-streaming calls which breaks resp.json() (fix iteration pending). Default suite unchanged: 406 passed / 8 skips. Co-Authored-By: Claude <[email protected]>
39 lines
1.3 KiB
Python
39 lines
1.3 KiB
Python
"""Shared pytest configuration for the orchestrator test suite.
|
|
|
|
Implements the opt-in gate for the LIVE LLM suite (``pytest.mark.live``):
|
|
|
|
A test marked ``@pytest.mark.live`` runs ONLY when BOTH hold:
|
|
1. ``LLM_KEY`` is set in the environment (a real external endpoint is
|
|
configured -- see ``LLMConfig`` in
|
|
``vtt_orchestrator/routing/llm_client.py``), AND
|
|
2. ``RUN_LIVE_LLM=1`` is exported explicitly.
|
|
|
|
CI never exports ``RUN_LIVE_LLM``, so live tests can never execute there;
|
|
locally they are skipped (not failed) unless the operator opts in. The skip
|
|
decision lives here so every test in the live module gets it for free and no
|
|
individual test has to duplicate the guard.
|
|
"""
|
|
|
|
import os
|
|
|
|
import pytest
|
|
|
|
|
|
def _live_llm_enabled() -> bool:
|
|
"""True only when a key is configured AND the opt-in flag is exported."""
|
|
if not os.environ.get("LLM_KEY"):
|
|
return False
|
|
return os.environ.get("RUN_LIVE_LLM", "0").strip().lower() == "1"
|
|
|
|
|
|
def pytest_collection_modifyitems(config, items):
|
|
if _live_llm_enabled():
|
|
return
|
|
skip_live = pytest.mark.skip(
|
|
reason="live LLM test: requires LLM_KEY set AND RUN_LIVE_LLM=1 "
|
|
"(opt-in; never runs in CI)"
|
|
)
|
|
for item in items:
|
|
if "live" in item.keywords:
|
|
item.add_marker(skip_live)
|