mirror of https://github.com/cloudstack-llc/mlx-knife.git synced 2026-07-21 10:05:26 -04:00

Files

T

The BROKE Cluster Team 05f1c30486 Release 2.0.3: Foundation for pipes

Foundation release for Unix pipe integration with stderr separation,
benchmark infrastructure, and reasoning control improvements.

Breaking Changes:
- stdout/stderr separation (Issue #43) - errors to stderr in human mode
- JSON mode unchanged (all output to stdout)

Features:
- Benchmark reporting infrastructure (ADR-013 Phase 0)
- --no-reasoning flag (Issue #40 partial - GPT-OSS/QwQ only)
- Interactive mode reasoning control (review_report.md fixes)

Bug Fixes:
- huggingface-hub 1.x incompatibility (critical dependency fix)
- Streaming parity tests refactored (Portfolio Discovery)

Testing:
- 308 tests passing (Python 3.9-3.13)
- 35 skipped (opt-in live tests)
- 79/91 E2E tests passing with HF_HOME

See CHANGELOG.md for complete details and migration guide.

2025-11-17 22:54:06 +01:00

1.3 KiB

Raw Permalink Blame History

Test Reports

This directory contains JSONL test reports from mlx-knife's E2E test suite.

File Naming Convention

YYYY-MM-DD-vX.Y.Z.jsonl

Example: 2025-11-16-v2.0.3.jsonl

Format

One JSON object per line (JSONL)
Schema version: Each report has schema_version field
Appending: New releases append new files (never edit old ones)

Historical Data

Reports are git-tracked to preserve historical trends:

Performance changes over releases
Model compatibility evolution
Stop token workaround stability

Analysis Examples

# Count tests by outcome
jq -r '.outcome' benchmarks/reports/*.jsonl | sort | uniq -c

# Average tokens/sec by model family
jq -r 'select(.performance) | "\(.model.family) \(.performance.tokens_per_sec)"' \
  benchmarks/reports/*.jsonl | \
  awk '{sum[$1]+=$2; count[$1]++} END {for (f in sum) print f, sum[f]/count[f]}'

# List models with workarounds
jq -r 'select(.stop_tokens.workaround != "none") | "\(.model.id): \(.stop_tokens.workaround)"' \
  benchmarks/reports/*.jsonl | sort -u

# Performance regression detection
jq -r 'select(.performance) | "\(.timestamp) \(.model.id) \(.performance.tokens_per_sec)"' \
  benchmarks/reports/*.jsonl | sort

Schema Version

Current: 0.1.0 (Phase 0 - Experimental)

See ../schemas/report-v0.1.schema.json for details.

1.3 KiB Raw Permalink Blame History