mlx-knife/docs/ADR/appendix/ADR-009-test-plan.md

ADR-009 Appendix: Test Plan

Status: Active – authoritative live-test blueprint for ADR-009
Related: ADR-009 Stop Token Detection Fix
Purpose: Real-model validation strategy for Beta.6

---

Test Models

Representative Models (Initial Validation)

Model	ID	Expected Issue	Purpose
MXFP4	mlx-community/gpt-oss-20b-MXFP4-Q8	`<	end
Qwen 2.5	mlx-community/Qwen2.5-0.5B-Instruct-4bit	Self-conversation (?)	Validate chat template handling
Llama 3.2	mlx-community/Llama-3.2-3B-Instruct-4bit	None (control)	Regression testing

Note: These 3 models serve as initial validation. Full portfolio testing (below) extends coverage to all MLX models in user cache.

Portfolio Discovery (Production Validation)

Status: ✅ Implemented (2.0.1)

Instead of hard-coded models, tests iterate over all MLX-compatible models in user cache.

Implementation:

• Location: test_stop_tokens_live.py (Category 2: Live Tests, read-only user cache)
• Function: discover_mlx_models_in_user_cache() (~40 LOC)
• Fixture: portfolio_models with fallback to TEST_MODELS for backward compatibility

Discovery Algorithm:

def discover_mlx_models_in_user_cache() -> List[Dict[str, Any]]:
    """Scan HF_HOME/hub/models--*/snapshots/* for MLX models.

    Filters:
    - MLX-compatible: Has safetensors + config.json
    - RAM-aware: Estimates model size, skips if exceeds budget

    Returns: List of models with: model_id, ram_needed_gb, snapshot_path
    """

RAM Gating (already implemented):

• Progressive budget: 40% (16GB), 50% (32GB), 60% (64GB), 70% (96GB+)
• Auto-skip models exceeding available RAM
• See get_safe_ram_budget_gb(), should_skip_model() helpers

Safety:

• Read-only cache access (no pull/rm)
• No isolated cache needed (Category 2 pattern)
• No CoW required (CoW only for Clone/ADR-007, not inference)

---

Test Phases

Phase 1: Baseline Measurement

Goal: Document current broken behavior

Test Case:

prompt = "Write one sentence about cats."
output = runner.generate_streaming(prompt, max_tokens=50)

Collect:

• Full generated text
• Token IDs (if accessible)
• Stop condition (why stopped?)
• Visible stop tokens

Expected Baseline Results:

• MXFP4: <|end|> appears in output ✗
• Qwen: TBD (may self-converse) ?
• Llama: Clean output ✓

Phase 2: Fix Validation

After implementing fix, same test case

Expected After-Fix Results:

• MXFP4: No stop tokens visible ✓
• Qwen: No self-conversation ✓
• Llama: Still works (no regression) ✓

Phase 3: Empirical Mapping

Document tokenizer configs:

{
  "model": "gpt-oss",
  "configured_eos": ["<|return|>"],     # From tokenizer
  "generated_tokens": ["<|end|>", ...], # Empirically observed
  "workaround_needed": True/False
}

---

Test Implementation

File: tests_2.0/test_stop_tokens_live.py

Markers:

@pytest.mark.live_stop_tokens  # Requires models downloaded
@pytest.mark.slow              # >1 min per model

Run:

# Baseline
pytest tests_2.0/test_stop_tokens_live.py::test_baseline -v -m live_stop_tokens

# After fix
pytest tests_2.0/test_stop_tokens_live.py::test_validation -v -m live_stop_tokens

---

Success Criteria

Initial Validation (3 Models): ✅ Phase 1 Complete: Baseline measurements documented ✅ Phase 2 Complete: All 3 models pass validation tests ✅ Phase 3 Complete: Empirical mapping generated (test artifact: stop_token_config_report.json)

Portfolio Validation (All Models in Cache): ✅ Portfolio Discovery: Implemented (2.0.1) - discover_mlx_models_in_user_cache() in test_stop_tokens_live.py ✅ Cache Iterator: Implemented - Auto-scans HF_HOME/hub/models--*/ for MLX-compatible models ✅ Dynamic Validation: Validated - Scales to all models in user cache, RAM-aware skipping, empirical report generation

---

Related Documentation

• ADR-009 Main: Implementation details, 2-LOC fix, add_eos_token() fallback
• ADR-011: E2E Live Test Architecture (Server/HTTP/CLI validation, reuses portfolio discovery)
• TESTING.md: Live test execution, markers, environment setup