mirror of
https://github.com/cloudstack-llc/mlx-knife.git
synced 2026-07-18 20:54:26 -04:00
The BROKE Cluster Team
d3f7d091bc
Bugfixes and compatibility improvements. No new features. Core fixes: - Framework detection for web API models (Issue #48) - Video-only model filtering from vision capability - Page size detection for memory metrics (macOS) - Model switch log timing (after load completion) Compatibility: - hub 1.x + transformers 5.0 support - Python 3.9-3.14 verified (494 tests passing) Testing infrastructure: - Benchmark schema v0.2.0 (hardware profiling, system health) - Benchmark template v1.0 (automated JSONL→Markdown reports) - Memory timeline visualization (memplot.py) - Unified model filter (build_model_object single source) Documentation: - Multi-Modal Support section in README (Vision subsection) - JSON API 0.1.5-0.1.6 marked Stable - Vision promoted from alpha to beta status - Removed conceptual drift and outdated references See CHANGELOG.md for complete details.
Test Reports
This directory contains JSONL test reports from mlx-knife's E2E test suite.
File Naming Convention
YYYY-MM-DD-vX.Y.Z.jsonl
Example: 2025-11-16-v2.0.3.jsonl
Format
- One JSON object per line (JSONL)
- Schema version: Each report has
schema_versionfield - Appending: New releases append new files (never edit old ones)
Historical Data
Reports are git-tracked to preserve historical trends:
- Performance changes over releases
- Model compatibility evolution
- Stop token workaround stability
Analysis Examples
# Count tests by outcome
jq -r '.outcome' benchmarks/reports/*.jsonl | sort | uniq -c
# Average tokens/sec by model family
jq -r 'select(.performance) | "\(.model.family) \(.performance.tokens_per_sec)"' \
benchmarks/reports/*.jsonl | \
awk '{sum[$1]+=$2; count[$1]++} END {for (f in sum) print f, sum[f]/count[f]}'
# List models with workarounds
jq -r 'select(.stop_tokens.workaround != "none") | "\(.model.id): \(.stop_tokens.workaround)"' \
benchmarks/reports/*.jsonl | sort -u
# Performance regression detection
jq -r 'select(.performance) | "\(.timestamp) \(.model.id) \(.performance.tokens_per_sec)"' \
benchmarks/reports/*.jsonl | sort
Schema Version
Current: 0.1.0 (Phase 0 - Experimental)
See ../schemas/report-v0.1.schema.json for details.