mirror of
https://github.com/cloudstack-llc/mlx-knife.git
synced 2026-07-21 18:15:37 -04:00
The BROKE Cluster Team
296 lines
10 KiB
Markdown
296 lines
10 KiB
Markdown
# MLX Knife Benchmarks
|
|
|
|
**Status:** Phase 0 - Organic Data Collection (WIP)
|
|
|
|
## What's Here?
|
|
|
|
This directory contains benchmark infrastructure for mlx-knife:
|
|
- Empirical performance and compatibility data from E2E tests
|
|
- Tools for analysis and visualization
|
|
- Schema definitions for structured reports
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
benchmarks/
|
|
├── reports/ # JSONL test reports + Markdown analyses
|
|
│ ├── 2025-12-20-v2.0.4b3.jsonl # Raw data (one file per test run)
|
|
│ └── BENCHMARK-*.md # Generated analysis reports
|
|
├── schemas/ # JSON Schema definitions
|
|
│ ├── report-v0.1.schema.json # legacy schema
|
|
│ ├── report-v0.2.2.schema.json # Current schema
|
|
│ └── report-current.schema.json # Symlink → current schema
|
|
├── tools/ # Standalone tools
|
|
│ ├── memmon.py # Memory monitor (background sampling)
|
|
│ └── memplot.py # Memory timeline visualizer
|
|
├── generate_benchmark_report.py # Report generator (Template v1.1)
|
|
├── validate_reports.py # Schema validation
|
|
├── README.md # ← You are here
|
|
└── TESTING.md # Benchmark handbook (How-To)
|
|
```
|
|
|
|
## Tools
|
|
|
|
| Tool | Purpose |
|
|
|------|---------|
|
|
| `generate_benchmark_report.py` | JSONL → Markdown report (Template v1.1) |
|
|
| `validate_reports.py` | Schema validation of JSONL files |
|
|
| `tools/memmon.py` | Memory + CPU + GPU monitoring (200ms sampling) |
|
|
| `tools/memplot.py` | Interactive 3-row timeline (Memory/CPU/GPU, HTML) |
|
|
|
|
## Schema
|
|
|
|
**Current:** v0.2.2 (Phase 0 - Test Infrastructure)
|
|
|
|
| Version | Release | Content |
|
|
|---------|---------|---------|
|
|
| v0.1.0 | 2.0.3 | Minimal: test, outcome, duration, model |
|
|
| v0.2.0 | 2.0.4-beta.3 | + hardware_profile, system_health, quality_flags |
|
|
| v0.2.1 | 2.0.4-beta.7 | + inference_modality (vision/text/audio) |
|
|
| v0.2.2 | 2.0.4-beta.9 | + test_start_ts, test_end_ts (precise timing) |
|
|
| v1.0.0 | Future | Model benchmarks (mlxk-benchmark package) |
|
|
|
|
**Schema Strategy:** No v0.3.x planned. v0.2.x → v1.0.0 directly.
|
|
- v0.x = Test infrastructure ("Was the test run clean?")
|
|
- v1.x = Model benchmarks ("How good is the model?")
|
|
|
|
See `schemas/LEARNINGS-FOR-v1.0.md` for details.
|
|
|
|
## Recent Reports
|
|
|
|
Latest baseline reports are in `reports/` directory:
|
|
- Pattern: `BENCHMARK-<template>-<version>-<date>-*.md`
|
|
- Hardware: Mac14,13 (M2 Max, 64 GB)
|
|
- Test suite: ~167 tests (Vision + Text + Audio E2E)
|
|
- Quality target: 100% clean (0 MB swap, 0 zombies)
|
|
|
|
## Phase 0 Goals
|
|
|
|
1. **Collect data organically** from E2E tests
|
|
2. **No perfect schema** - schema evolves with data
|
|
3. **Git-tracked reports** - historical trends
|
|
4. **Foundation for Phase 1** - mlxk-benchmark package
|
|
|
|
## Memory Timeline Visualization
|
|
|
|
**Tool:** `tools/memplot.py` - 3-row interactive plot (Memory / CPU / GPU)
|
|
|
|
### Quick Start
|
|
|
|
```bash
|
|
# Collect data (memmon runs in background)
|
|
python benchmarks/tools/memmon.py --output memory.jsonl -- \
|
|
pytest -m live_e2e tests_2.0/live/ --report-output benchmark.jsonl
|
|
|
|
# Generate interactive HTML with test + model markers
|
|
python benchmarks/tools/memplot.py memory.jsonl benchmark.jsonl -o timeline.html
|
|
```
|
|
|
|
**Note:** `benchmark.jsonl` adds test markers showing test name + model name - essential for plot navigation!
|
|
|
|
### Visual Legend
|
|
|
|
#### Row 1: Memory (RAM Free GB)
|
|
|
|
**Blue line:** RAM free over time (GB)
|
|
|
|
**RAM line marker colors** (per-point coloring based on available RAM):
|
|
- 🟢 **Green:** ≥32 GB free - healthy
|
|
- 🟠 **Orange:** 16-32 GB free - warning zone
|
|
- 🔴 **Red:** <16 GB free - critical
|
|
|
|
**Memory pressure background** (semi-transparent overlays):
|
|
- 🟡 **Yellow `rgba(255, 204, 0, 0.15)`:** WARN level - system preparing to swap
|
|
- 🔴 **Red `rgba(255, 59, 48, 0.15)`:** CRITICAL level - system actively swapping
|
|
|
|
**Red line (right axis):** Swap Used (MB) - only visible when > 0
|
|
|
|
#### Row 2: CPU Load
|
|
|
|
**Load Average (purple):** 1-minute load average
|
|
**User (green fill):** User space CPU %
|
|
**System (red fill):** Kernel CPU % (stacked on User)
|
|
|
|
#### Row 3: GPU Utilization (Apple Silicon)
|
|
|
|
**Device (orange solid):** Overall GPU busy %
|
|
**Renderer (green fill):** 3D rendering cores %
|
|
**Tiler (purple dashed):** Geometry processing %
|
|
|
|
**Source:** `ioreg` PerformanceStatistics (no sudo required)
|
|
|
|
#### Background Rectangles: Test Regions (All Rows)
|
|
|
|
**Gray (rgba(200, 200, 200, 0.3)):**
|
|
- Model tests that load an LLM model
|
|
- Example: `test_run_command[text_00]`, `test_chat_completion[vision_01]`
|
|
- **Meaning:** Model is loaded in RAM during this time
|
|
|
|
**Light Blue (rgba(173, 216, 230, 0.2)):**
|
|
- Infrastructure tests without model
|
|
- Example: `test_portfolio_discovery`, `test_health_check`
|
|
- **Meaning:** No model loaded, only test infrastructure active
|
|
|
|
⚠️ **Known limitation (v0.2.2):** Server tests appear as "light blue" even when loading models (LocalServer fixture doesn't record model metadata). Recognizable by: high RAM usage + long duration in blue region. Example: `test_text_request_still_works_on_vision_model` (57 GB used, 16s duration).
|
|
|
|
#### Labels (All Rows)
|
|
|
|
**Top (90° rotated, black):**
|
|
- Model names at each model switch
|
|
- Example: `DeepHermes-3-Mistral`, `pixtral-12b-8bit`
|
|
- Position: Left-aligned with test start
|
|
|
|
**Bottom (90° rotated, gray):**
|
|
- Test names for each test (model + infrastructure)
|
|
- Example: `test_run_command`, `test_chat_completion`
|
|
- Position: Left-aligned with test start
|
|
|
|
**Vertical helper lines:**
|
|
- Thin gray lines at each test start
|
|
- Help correlate labels with timeline
|
|
|
|
#### Secondary Y-Axis: Swap Used (MB)
|
|
|
|
**Red line (right axis):**
|
|
- Only visible when swap > 0 MB
|
|
- **Meaning:** System paging RAM to SSD → performance loss
|
|
- **Normal:** 0 MB
|
|
- **Problematic:** >100 MB
|
|
|
|
### Interpretation Patterns
|
|
|
|
**Typical model load:**
|
|
```
|
|
Pattern: RAM Free drops suddenly (e.g., 52 GB → 28 GB)
|
|
Duration: 2-5 seconds
|
|
Color: Gray rectangle begins
|
|
Label: Model name appears at top
|
|
→ Model loaded into RAM (24 GB)
|
|
```
|
|
|
|
**Typical model unload:**
|
|
```
|
|
Pattern: RAM Free rises suddenly (e.g., 28 GB → 52 GB)
|
|
Duration: <1 second
|
|
Color: Gray rectangle ends (or switches to next)
|
|
Label: New model name (or none)
|
|
→ Model removed from RAM
|
|
```
|
|
|
|
**Memory pressure without swap:**
|
|
```
|
|
Pattern: Yellow/Red background WITHOUT swap line
|
|
RAM Free: Still >10 GB
|
|
→ macOS preparing to swap, not yet active
|
|
→ Often during large model loads (temporary)
|
|
```
|
|
|
|
**Memory pressure with swap:**
|
|
```
|
|
Pattern: Red background + Red swap line rises
|
|
RAM Free: <10 GB
|
|
Swap: >100 MB
|
|
→ System actually at limit
|
|
→ Performance significantly worse
|
|
→ Typical: Multiple large models in short time
|
|
```
|
|
|
|
**Infrastructure test with high RAM usage:**
|
|
```
|
|
Pattern: Light blue rectangle + RAM drops significantly (>20 GB)
|
|
Duration: >10 seconds
|
|
Example: 57 GB used in test_text_request_still_works_on_vision_model
|
|
→ ⚠️ Schema bug: Server test loads model but "model": null
|
|
→ Should be gray, not light blue
|
|
→ Fix: v1.0 schema with log parsing
|
|
```
|
|
|
|
### Data Sources
|
|
|
|
**RAM Free:**
|
|
- Source: `vm_stat` (macOS native)
|
|
- Calculation: `(free + inactive + purgeable + speculative) * page_size / 1e9`
|
|
- Sample rate: 200ms (5 samples/second)
|
|
|
|
**Memory Pressure:**
|
|
- Source: `sysctl kern.memorystatus_vm_pressure_level`
|
|
- Values: 1=NORMAL, 2=WARN, 4=CRITICAL
|
|
- Sample rate: 200ms (synchronized with RAM)
|
|
|
|
**Swap Used:**
|
|
- Source: `sysctl vm.swapusage`
|
|
- Unit: MB
|
|
- Sample rate: 200ms
|
|
|
|
**Test Metadata:**
|
|
- Source: Benchmark JSONL (pytest-json-report format)
|
|
- Fields: `timestamp`, `duration`, `test`, `model` (optional), `outcome`
|
|
- Correlation: ISO timestamp → Unix timestamp → elapsed seconds
|
|
|
|
### Known Limitations (v0.2.0)
|
|
|
|
1. **Model load/unload events missing**
|
|
- Gray regions show "test with model", not "model is loaded"
|
|
- Pytest runs through ALL models 4x → each model loaded/unloaded 4x
|
|
- Regions overlap visually though sequential
|
|
- **Fix planned:** v1.0 schema with explicit events
|
|
|
|
2. **Server tests without model attribution**
|
|
- Server tests (LocalServer fixture) load models internally
|
|
- Appear as "infrastructure" (light blue) instead of "model" (gray)
|
|
- Recognizable: High RAM + long duration in blue region
|
|
- **Fix planned:** Log parsing in v0.3.0/v1.0
|
|
|
|
3. **Dense test sequences**
|
|
- Tests shorter than 200ms sample rate → no coloring
|
|
- Typical: Fast infrastructure tests (<100ms)
|
|
- **Workaround:** Test labels show all tests
|
|
|
|
4. **Label overlap**
|
|
- Many tests in short time (>10 tests/min)
|
|
- Labels may overlap (90° rotated)
|
|
- **Mitigation:** Zoom for detailed view
|
|
- **Future:** Adaptive label density or collapsing
|
|
|
|
### Interactive Features
|
|
|
|
- **Zoom & Pan:** Mouse wheel (vertical), Shift+wheel (horizontal), click+drag
|
|
- **Range Slider:** Quick navigation in long (>20 min) timelines
|
|
- **Hover:** X-axis unified mode shows all values at same time
|
|
|
|
### Future Extensions (Ideas)
|
|
|
|
**For plot:**
|
|
- [ ] Embedded legend in plot (not external file)
|
|
- [ ] Toggle show/hide infrastructure tests
|
|
- [ ] Hover shows full test names (not truncated)
|
|
- [ ] Color-blind mode (alternative palette)
|
|
|
|
**For schema v1.0:**
|
|
- [ ] Model load/unload events → precise "in RAM" regions
|
|
- [ ] Log parsing for server tests → correct attribution
|
|
- [ ] GPU activity (Metal performance)
|
|
- [ ] Net T/S (tokens/second, pure inference)
|
|
|
|
**For analysis:**
|
|
- [ ] Automatic anomaly detection (memory leaks, zombies)
|
|
- [ ] Per-model memory profiling (min/max/avg RAM)
|
|
- [ ] Scheduling optimization (avoid model-switch overlap)
|
|
|
|
---
|
|
|
|
## Roadmap
|
|
|
|
| Phase | Release | Description |
|
|
|-------|---------|-------------|
|
|
| **Phase 0** | 2.0.3-2.0.4 | Organic Data Collection ✅ |
|
|
| Phase 1 | 2.1+ | `mlxk-benchmark` package (separate tool) |
|
|
| Phase 2 | 2.2+ | Report aggregation, hardware correlation |
|
|
| Phase 3 | 2.3+ | Public database, community contributions |
|
|
|
|
## Further Documentation
|
|
|
|
- **[TESTING.md](TESTING.md)** - Benchmark handbook (How-To)
|
|
- **[schemas/LEARNINGS-FOR-v1.0.md](schemas/LEARNINGS-FOR-v1.0.md)** - Learnings for Phase 1
|
|
- **[docs/ADR/ADR-013-Community-Model-Quality-Database.md](../docs/ADR/ADR-013-Community-Model-Quality-Database.md)** - Architecture vision
|