From e8b10ea10b2e08d589707ec6cd6bf534aa282cdf Mon Sep 17 00:00:00 2001
From: The BROKE Cluster Team
Date: Fri, 23 Jan 2026 20:20:59 +0100
Subject: [PATCH] Release 2.0.4-beta.8: Audio transcription support
(experimental)
Audio input via --audio flag (CLI) and input_audio content type (Server API).
Uses mlx-vlm native audio processing. ~30s duration limit (model constraint).
Currently only Gemma-3n tested (requires --repair-index fix).
Also includes:
- SERVER-HANDBOOK compliance (image limits, validation error envelopes)
- Dependency updates: mlx>=0.30.0, mlx-lm>=0.30.0, huggingface-hub>=1.0.0
- Audio E2E test suite + ADR-019
---
.gitignore | 5 +-
CHANGELOG.md | 34 ++
README.md | 77 +++-
TESTING-DETAILS.md | 70 +++-
docs/ADR/ADR-019-Audio-Input-Support.md | 256 ++++++++++++
docs/SERVER-HANDBOOK.md | 381 +++++++++++++++---
docs/json-api-schema.json | 7 +-
docs/json-api-specification.md | 34 +-
mlxk2/cli.py | 50 ++-
mlxk2/core/capabilities.py | 42 +-
mlxk2/core/server_base.py | 151 +++++--
mlxk2/core/vision_runner.py | 40 +-
mlxk2/errors.py | 2 +
mlxk2/operations/common.py | 48 ++-
mlxk2/operations/run.py | 35 +-
mlxk2/output/human.py | 7 +
mlxk2/spec.py | 2 +-
mlxk2/tools/vision_adapter.py | 164 ++++++--
pyproject.toml | 8 +-
...A MAN SAID TO THE UNIVERSE SIR I EXIST.mp3 | Bin 0 -> 29672 bytes
...A MAN SAID TO THE UNIVERSE SIR I EXIST.wav | Bin 0 -> 148844 bytes
.../HAVING RETURNED TO THE ROYAL CAVERN.mp3 | Bin 0 -> 90764 bytes
.../HAVING RETURNED TO THE ROYAL CAVERN.txt | 1 +
.../HAVING RETURNED TO THE ROYAL CAVERN.wav | Bin 0 -> 463084 bytes
tests_2.0/assets/audio/sources.txt | 11 +
tests_2.0/live/conftest.py | 91 +++++
tests_2.0/live/test_audio_e2e_live.py | 223 ++++++++++
tests_2.0/live/test_utils.py | 83 ++++
tests_2.0/test_audio_cli.py | 120 ++++++
tests_2.0/test_capabilities.py | 54 +++
tests_2.0/test_server_audio.py | 318 +++++++++++++++
tests_2.0/test_server_models_and_errors.py | 38 ++
tests_2.0/test_vision_adapter.py | 191 ++++++++-
33 files changed, 2346 insertions(+), 197 deletions(-)
create mode 100644 docs/ADR/ADR-019-Audio-Input-Support.md
create mode 100644 tests_2.0/assets/audio/A MAN SAID TO THE UNIVERSE SIR I EXIST.mp3
create mode 100644 tests_2.0/assets/audio/A MAN SAID TO THE UNIVERSE SIR I EXIST.wav
create mode 100644 tests_2.0/assets/audio/HAVING RETURNED TO THE ROYAL CAVERN.mp3
create mode 100644 tests_2.0/assets/audio/HAVING RETURNED TO THE ROYAL CAVERN.txt
create mode 100644 tests_2.0/assets/audio/HAVING RETURNED TO THE ROYAL CAVERN.wav
create mode 100644 tests_2.0/assets/audio/sources.txt
create mode 100644 tests_2.0/live/test_audio_e2e_live.py
create mode 100644 tests_2.0/test_audio_cli.py
create mode 100644 tests_2.0/test_server_audio.py
diff --git a/.gitignore b/.gitignore
index 924e523..187933f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -23,13 +23,16 @@ server.log
install_*.log
.gitignore
docs/ISSUES/
+docs/reviews/
# Test artifacts (generated reports)
*_report.json
test-img-collection/
small-img-collection
benchmarks/reports/*.html
+benchmarks/reports/2025-12*.jsonl
+benchmarks/reports/2026-0[12]*.jsonl
# Benchmark reports (ADR-013 Phase 0)
# These reports ARE tracked in git for historical data
-!benchmarks/reports/*.jsonl
\ No newline at end of file
+#!benchmarks/reports/*.jsonl
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2f4e294..cbf82e4 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,39 @@
# Changelog
+## [2.0.4-beta.8] - 2026-01-23
+
+### Highlights
+
+**Audio Input Support (Experimental):** Full audio transcription pipeline for audio-capable models. CLI `--audio` flag and OpenAI-compatible Server API (`input_audio` format). Uses mlx-vlm's native audio processing with ~30 second duration limit (model architecture constraint). Currently only Gemma-3n tested; requires `--repair-index` fix (mlx-vlm #624).
+
+### Added
+
+- **Audio Capability Detection:** `list` and `show` display `[audio]` tag for audio-capable models
+- **CLI `--audio` Parameter:** `mlxk run model "prompt" --audio file.wav` (WAV/MP3 supported)
+- **Server Audio API:** OpenAI-compatible `input_audio` content type for `/v1/chat/completions`
+ - Format: `{"type": "input_audio", "input_audio": {"data": "", "format": "wav"}}`
+ - Supported formats: WAV, MP3
+ - Temperature default: 0.0 for audio (greedy sampling reduces hallucination)
+- **Audio E2E Tests:** Portfolio-based transcription tests with deterministic audio clips (ADR-019)
+
+### Changed
+
+- **Dependency Updates:**
+ - `mlx>=0.30.0` (was 0.29.0) - MXFP4/NVFP4 support
+ - `mlx-lm>=0.30.0` (was 0.28.4)
+ - `huggingface-hub>=1.0.0` (was <1.0) - transformers 5.x compatibility
+ - `mlx-vlm` upstream update: fc8c92e → 5812270 (audio support)
+
+- **SERVER-HANDBOOK Compliance:**
+ - Image limits enforced: max 5 images per request, max 50MB total
+ - Multi-audio blocked with clear error (mlx-vlm limitation)
+ - Audio+Vision combined: Audio silently ignored (single modality per request)
+ - `chunk=1` explicitly respected (was defaulting to env var)
+ - 422 validation errors now return ADR-004 envelope format
+ - 501 NOT_IMPLEMENTED error type added for unsupported features
+
+---
+
## [2.0.4-beta.7] - 2026-01-18
### Highlights
diff --git a/README.md b/README.md
index 84fe160..1acef80 100644
--- a/README.md
+++ b/README.md
@@ -4,9 +4,9 @@
-**Current Version: 2.0.4-beta.7** (Stable: 2.0.3)
+**Current Version: 2.0.4-beta.8** (Stable: 2.0.3)
-[](https://github.com/mzau/mlx-knife/releases)
+[](https://github.com/mzau/mlx-knife/releases)
[](https://www.apache.org/licenses/LICENSE-2.0)
[-blue.svg)](https://www.python.org/downloads/)
[](https://support.apple.com/en-us/HT211814)
@@ -18,6 +18,7 @@
## Features
### What's New in 2.0.4 (Coming Soon - Currently Beta)
+- **Audio Transcription (Experimental)** - Speech-to-text via `--audio` flag (CLI + Server API)
- **Vision Models with EXIF Metadata** - Image analysis + automatic GPS/date/camera extraction visible to the model
- **Unix Pipe Integration** - Chain models without temp files (`vision → text` workflows)
- **Local Development Workflow** - Clone → Repair → Test models without HuggingFace round-trips
@@ -76,7 +77,7 @@ This license applies **only** to the `mlx-knife` code and **does not extend** to
MLX Knife has been comprehensively tested and verified on:
✅ **Python 3.9.6 - 3.14** - Text LLMs fully supported (mlx-lm 0.28.4+)
-✅ **Python 3.10 - 3.14** - Vision models supported (mlx-vlm 0.3.9+; beta.7 uses commit fc8c92e31983a52761f37d503f903ec40bebbd62 with MXFP4 support)
+✅ **Python 3.10 - 3.14** - Vision models supported (mlx-vlm 0.3.9+; beta.8 uses commit 5812270 with audio + MXFP4 support)
**Note:** Vision features require Python 3.10+. Native macOS Python 3.9.6 users need to upgrade (e.g., via Homebrew).
@@ -106,17 +107,17 @@ mlxk --version # → mlxk 2.0.3 (latest stable on PyPI)
### Via GitHub (Latest Beta)
```bash
-# Install 2.0.4-beta.7 (Workspace discovery + Server robustness)
-pip install "git+https://github.com/mzau/mlx-knife.git@v2.0.4-beta.7"
+# Install 2.0.4-beta.8 (Audio transcription + Server enhancements)
+pip install "git+https://github.com/mzau/mlx-knife.git@v2.0.4-beta.8"
# With Vision support (Python 3.10+ required)
-pip install "git+https://github.com/mzau/mlx-knife.git@v2.0.4-beta.7#egg=mlx-knife[vision]"
+pip install "git+https://github.com/mzau/mlx-knife.git@v2.0.4-beta.8#egg=mlx-knife[vision]"
# Verify installation
-mlxk --version # → mlxk 2.0.4b7
+mlxk --version # → mlxk 2.0.4b8
```
-**Beta.7 note:** Uses mlx-vlm commit fc8c92e (includes MXFP4 quantization support). The `[vision]` extra automatically installs the correct version.
+**Beta.8 note:** Uses mlx-vlm commit 5812270 (includes audio support + MXFP4 quantization). The `[vision]` extra automatically installs the correct version.
**For production use:** Wait for 2.0.4 stable on PyPI (requires mlx-vlm 0.3.10 release).
@@ -134,7 +135,7 @@ pip install -e ".[dev,test]"
pip install -e ".[dev,test,vision]"
# Verify installation
-mlxk --version # → mlxk 2.0.4b7
+mlxk --version # → mlxk 2.0.4b8
# Run tests and quality checks (before committing)
pytest -v
@@ -210,7 +211,7 @@ mlxk run "Phi-4" "Hello" # Fuzzy match
mlxk show "Qwen3@e96" --json # Specific version
```
-### Local Paths (2.0.4-beta.7+)
+### Local Paths (2.0.4-beta.6+)
| Format | Example |
|--------|---------|
@@ -240,7 +241,7 @@ mlxk run ./fixed "Test"
---
-## Workspace Development Workflow (2.0.4-beta.7+)
+## Workspace Development Workflow (2.0.4-beta.6+)
**Complete local development cycle** for model experimentation, repair, and testing without HuggingFace round-trips:
@@ -310,8 +311,8 @@ Image analysis via the `--image` flag (CLI and server). Requires Python 3.10+.
- **Python 3.10+** (mlx-vlm dependency)
- **Installation:** `pip install mlx-knife[vision]`
-- **Backend:** mlx-vlm commit c536165df2b3b4aece3a795b2e414349f935e750 (auto-installed)
-- **Beta.4 note:** The `[vision]` extra automatically installs mlx-vlm from git with the Pixtral pad_token fix. Will switch to PyPI v0.3.10 when released.
+- **Backend:** mlx-vlm commit 5812270 (audio + MXFP4 support, auto-installed)
+- **Beta.8 note:** The `[vision]` extra automatically installs mlx-vlm from git with audio support. Will switch to PyPI v0.3.10 when released.
#### Usage
@@ -475,6 +476,54 @@ mlxk convert