Test Reports

This directory contains JSONL test reports from mlx-knife's E2E test suite.

File Naming Convention

YYYY-MM-DD-vX.Y.Z.jsonl

Example: 2025-11-16-v2.0.3.jsonl

Format

One JSON object per line (JSONL)
Schema version: Each report has schema_version field
Appending: New releases append new files (never edit old ones)

Historical Data

Reports are git-tracked to preserve historical trends:

Performance changes over releases
Model compatibility evolution
Stop token workaround stability

Analysis Examples

# Count tests by outcome
jq -r '.outcome' benchmarks/reports/*.jsonl | sort | uniq -c

# Average tokens/sec by model family
jq -r 'select(.performance) | "\(.model.family) \(.performance.tokens_per_sec)"' \
  benchmarks/reports/*.jsonl | \
  awk '{sum[$1]+=$2; count[$1]++} END {for (f in sum) print f, sum[f]/count[f]}'

# List models with workarounds
jq -r 'select(.stop_tokens.workaround != "none") | "\(.model.id): \(.stop_tokens.workaround)"' \
  benchmarks/reports/*.jsonl | sort -u

# Performance regression detection
jq -r 'select(.performance) | "\(.timestamp) \(.model.id) \(.performance.tokens_per_sec)"' \
  benchmarks/reports/*.jsonl | sort

Schema Version

Current: 0.1.0 (Phase 0 - Experimental)

See ../schemas/report-v0.1.schema.json for details.