diff --git a/.gitignore b/.gitignore
index f410c73..69a69b4 100644
--- a/.gitignore
+++ b/.gitignore
@@ -21,8 +21,8 @@ CLAUDE.md
TODO_REAL_TESTS.md
server.log
install_*.log
-openwebui311/bin/
.gitignore
+docs/ISSUES/
# Test artifacts (generated reports)
*_report.json
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 719c911..3c7f50e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,243 @@
# Changelog
+## [2.0.4-beta.6] - 2026-01-07
+
+### Highlights
+
+**Complete Local Development Workflow:** Beta.6 completes the workspace story started in beta.5. Full local development cycle without HuggingFace round-trips:
+1. Clone a model to a local workspace (`mlxk clone`)
+2. Repair or modify it locally (`mlxk convert --repair-index` fixes index/shard mismatches)
+3. Use it directly without pushing:
+ - Run inference: `mlxk run ./workspace "prompt"`
+ - Inspect metadata: `mlxk show ./workspace`
+ - Start a dev server: `mlxk server --model ./workspace`
+
+Enables model experimentation, repair (mlx-vlm #624 affected models), and testing before publishing to HuggingFace.
+
+**Vision Batch Processing:** Production-ready vision processing with automatic chunking for large image collections. Default: process one image at a time for maximum stability (prevents Metal OOM + model-specific hallucination). Server now accepts unlimited images with safe chunking. Complete Vision→Text pipe workflow support. Metadata now appears before model output (vision models can reference filenames, GPS, dates in their analysis).
+
+**Enhanced Reliability:** Resumable clone operations with `--force-resume`, deterministic temp cache naming for better debugging, and improved test architecture (umbrella marker convention) for multi-Python compatibility (3.9-3.14). 161 Wet Umbrella tests passing (includes new Vision→Geo pipe integration tests).
+
+---
+
+### Added
+
+- **Workspace Operation Support (ADR-018 Phase 0c):**
+ - Complete local workflow: `clone → convert → run/show/server` without HF push
+ - `mlxk run ./workspace "prompt"` - Direct inference on workspace paths
+ - `mlxk show ./workspace` - Metadata inspection for workspaces
+ - `mlxk server --model ./workspace` - Local dev server with workspace models
+ - Server `/v1/models` endpoint lists workspaces with `"owned_by": "workspace"`
+ - MLXRunner + VisionRunner now support workspace paths
+ - Central implementation via `resolve_model_for_operation()` + `is_workspace_path()`
+ - Files changed: 11 files (~250 LOC)
+ - See ADR-018.md Phase 0c for details
+
+- **Path Resolution Convention (README.md):**
+ - `model-name` → Cache resolution (expands to `mlx-community/...`)
+ - `./model-name` → Workspace path (local directory)
+ - `/abs/path` → Workspace path (absolute)
+ - `.` or `..` → Workspace path (current/parent directory)
+ - Explicit path prefixes required to treat local directories as workspaces
+ - Prevents ambiguity when workspace name matches cache model name
+
+- **Health command workspace support:**
+ - `mlxk health ./workspace` - Integrity checks for workspace directories
+ - Returns minimal format per JSON API Schema 0.1.5: `{name, status, reason}`
+ - Uses same integrity checks as list/show commands (consistency)
+
+- **Resumable Clone with --force-resume (ADR-018 Phase 0b):**
+ - `mlxk clone --force-resume` - Skip confirmation prompt for partial downloads
+ - Deterministic temp cache naming: `.mlxk2_temp_{hash(model+target)}` (SHA256-based, replaces PID random names)
+ - Resume detection: Verifies health + download completion marker before resuming
+ - Conditional cleanup: Keeps complete downloads on failure (debugging/retry), cleanup incomplete
+ - Helper functions: `_get_deterministic_temp_cache_name()`, `_create_temp_cache_same_volume()`
+ - Files: `operations/clone.py` (~100 LOC)
+
+- **Portfolio Discovery Blacklist (Testing Infrastructure):**
+ - `KNOWN_BROKEN_MODELS` config in `tests_2.0/live/test_utils.py`
+ - Filters models with upstream runtime bugs (pass static health checks, fail at runtime)
+ - Policy: Add only when static health ✅ but runtime initialization ❌ due to verified upstream bug
+ - First entry: `mlx-community/MiMo-VL-7B-RL-bf16` (mlx-vlm NoneType iteration bug)
+ - Portfolio Discovery now filters against blacklist (27 → 26 models)
+ - Files: `tests_2.0/live/test_utils.py` (+29 LOC blacklist config)
+
+- **Vision Batch Processing (ADR-012 Phase 1c, #47):**
+ - Automatic chunking for large image collections (prevents Metal OOM crashes)
+ - CLI: `mlxk run model --image *.jpg --chunk N` (N = images per batch)
+ - Server: `POST /v1/chat/completions {"chunk": N, ...}` + `mlxk serve --chunk N`
+ - Model loaded once per chunk (isolation guarantees no state leakage between batches)
+ - Global image numbering preserved (Image 1..N across all chunks)
+ - Batch context markers in prompt: `[Processing batch 1/2: Images 1-4 (chunk_size=4, 8 total)]`
+ - Collapsible batch info in output: `📸 Batch 1/2: Images 1-4` (WebUI-friendly)
+ - Environment variable: `MLXK2_VISION_BATCH_SIZE` sets default chunk size
+ - Server unlimited images: Removed MAX_IMAGES_PER_REQUEST limit, added MAX_SAFE_CHUNK_SIZE=5 validation
+ - Chunk size validation: CLI + Server validate against MAX_SAFE_CHUNK_SIZE (HTTP 400 if chunk>5)
+ - Files: `cli.py`, `operations/run.py`, `operations/serve.py`, `core/server_base.py`, `core/vision_runner.py`, `tools/vision_adapter.py` (~200 LOC)
+
+- **Vision Chunk Isolation (Prevents State Leakage):**
+ - Fresh VisionRunner created per chunk (no KV-cache or model state persistence)
+ - Fixes chunk hallucination bug where model "remembered" images from previous batches
+ - Trade-off: Model load overhead (~2-3s per chunk) vs guaranteed isolation
+ - Alternative approach deferred: Model reuse + cache clearing (2.0.5-beta, requires mlx-vlm API support)
+
+- **Flexible Prompt Argument Order (UX Improvement):**
+ - Added `--prompt` flag as alternative to positional argument
+ - Use case: Test different prompts with same parameters: `mlxk run model --image *.jpg --chunk 5 --prompt "prompt"`
+ - Solves argparse limitation where `--image` with `nargs='+'` consumes following arguments
+ - Both forms supported: `mlxk run model "prompt" --image file.jpg` OR `mlxk run model --image file.jpg --prompt "prompt"`
+ - Backward compatible: Positional prompt still works when placed before flags
+ - Pipe mode integration: `mlx-run pixtral ... | mlx-run qwen - --prompt "..."` combines stdin with additional prompt
+ - File: `cli.py` (+10 LOC)
+
+- **Vision→Geo Pipe Integration Tests (Wet Umbrella Phase 4):**
+ - Comprehensive smoke tests for Vision→Text pipe workflows
+ - Test suite: `tests_2.0/live/test_pipe_vision_geo.py` (3 tests, marker: `live_vision_pipe`)
+ - **Test 1:** Vision batch processing with `--chunk 1` (validates ADR-012 Phase 1c)
+ - **Test 2:** Complete Vision→Geo pipe (validates stdin + --prompt fix)
+ - **Test 3:** Chunk isolation (validates no state leakage between chunks)
+ - Test assets: `tests_2.0/assets/geo-test/` (9 publishable Stockholm photos with EXIF)
+ - Model selection: Hardcoded pixtral (vision) + Portfolio discovery for text model (largest eligible)
+ - Wet Umbrella integration: 161 total tests passed (Phase 1: 152, Phase 2: 3, Phase 3: 3, Phase 4: 3)
+ - Graceful skip: Tests skip cleanly if vision/text models unavailable
+ - **NOT a quality benchmark:** Pure workflow validation (pass = exit 0, no GOLD reference)
+
+- **Test Infrastructure Improvements (Umbrella Marker Convention):**
+ - Added `pytest.mark.live` umbrella marker for scalable test exclusion
+ - All 11 live test files decorated with umbrella + specific markers
+ - `test-multi-python.sh` simplified: `pytest -m "not live"` (future-proof)
+ - Documentation: TESTING-DETAILS.md "Writing New Live Tests: Umbrella Marker Convention"
+ - Rationale: Blacklist pattern (excluding individual markers) doesn't scale with new test categories
+
+### Changed
+
+- **BREAKING: Vision processing now defaults to processing one image at a time (ADR-012 Phase 1c, #47)**
+ - Previous: All images processed in single batch (could OOM crash with many images)
+ - New: Process one image at a time by default (maximum safety on all systems)
+ - Override with `--chunk N` (CLI) or `"chunk": N` (API) for faster batching when system can handle it
+ - Rationale: Safety over speed - prevents Metal OOM + model-specific hallucination with large batches
+ - **Important:** Chunk isolation (fresh VisionRunner per chunk) prevents state leakage but adds ~2-3s model load overhead per chunk
+ - Migration: Users with large image workflows should use `--chunk 5` or higher for old behavior
+ - Environment variable: `MLXK2_VISION_BATCH_SIZE=N` sets default (e.g., export `MLXK2_VISION_BATCH_SIZE=5`)
+ - **Model-specific hallucination triggers:**
+ - Larger chunks with global context hints (e.g., "8 total images" when batch only has 4)
+ - Plural prompts mismatched to actual count (e.g., "describe these images" with chunk=1)
+ - Default chunk=1 with singular prompts provides maximum robustness
+
+- **Model metadata `cached` field:** Now `false` for workspace paths, `true` for cache models
+ - Workspace paths are NOT in HuggingFace cache → `cached: false`
+ - Cache models remain `cached: true`
+ - Semantic distinction: cache-managed vs user-managed models
+
+### Fixed
+
+- **Vision model detection (False Negatives):**
+ - Fixed: Models with `vision_config` dict but `skip_vision: true` incorrectly marked as non-vision
+ - Root cause: `skip_vision` flag misinterpreted as "no vision support" (actually means "optional for text-only")
+ - Solution: Presence of `vision_config` dict now reliably indicates vision capability
+ - Impact: Mistral-Small 3.1, DeepSeek-OCR now correctly detected as vision models
+ - File: `operations/common.py:detect_vision_capability()`
+
+- **Health check improvements:**
+ - **processor_config.json support:** Vision models can use EITHER `preprocessor_config.json` OR `processor_config.json`
+ - Different models use different naming conventions (e.g., DeepSeek-OCR)
+ - Previously: Only `preprocessor_config.json` accepted → false "unhealthy" status
+ - Impact: More vision models pass health checks
+ - **mlx-vlm #624 specific error message:** Index/shard mismatch now distinguished from incomplete downloads
+ - Message: "Index/shard mismatch (mlx-vlm #624). Index references N shards but found M different files. Fix: mlxk convert
-**Current Version: 2.0.4-beta.5** (Stable: 2.0.3)
+**Current Version: 2.0.4-beta.6** (Stable: 2.0.3)
-[](https://github.com/mzau/mlx-knife/releases)
+[](https://github.com/mzau/mlx-knife/releases)
[](https://www.apache.org/licenses/LICENSE-2.0)
-[](https://www.python.org/downloads/)
+[-blue.svg)](https://www.python.org/downloads/)
[](https://support.apple.com/en-us/HT211814)
[](https://github.com/ml-explore/mlx)
@@ -17,13 +17,20 @@
## Features
+### What's New in 2.0.4 (Coming Soon - Currently Beta)
+- **Vision Models with EXIF Metadata** - Image analysis + automatic GPS/date/camera extraction visible to the model
+- **Unix Pipe Integration** - Chain models without temp files (`vision → text` workflows)
+- **Local Development Workflow** - Clone → Repair → Test models without HuggingFace round-trips
+- **Community Model Repair Tool** - Fix broken mlx-vlm models with `--repair-index`
+- **Resumable Downloads** - Interrupted clone/pull operations continue automatically
+- **Safe Vision Batch Processing** - Automatic chunking prevents Metal OOM crashes
+- **Workspace Path Support** - Run/show/server commands work with local directories
+
### Core Functionality
- **List & Manage Models**: Browse your HuggingFace cache with MLX-specific filtering
- **Model Information**: Detailed model metadata including quantization info
- **Download Models**: Pull models from HuggingFace with progress tracking
- **Run Models**: Native MLX execution with streaming and chat modes
-- **Vision Models**: Image analysis (Python 3.10+, beta)
-- **Unix Pipes**: Chain models via stdin/stdout - no temp files (beta)
- **Health Checks**: Verify model integrity and MLX runtime compatibility
- **Cache Management**: Clean up and organize your model storage
- **Privacy & Network**: No background network or telemetry; only explicit Hugging Face interactions when you run pull or the experimental push.
@@ -99,17 +106,17 @@ mlxk --version # → mlxk 2.0.3 (latest stable on PyPI)
### Via GitHub (Latest Beta)
```bash
-# Install 2.0.4-beta.5 (Community repair tools + BPE fix)
-pip install "git+https://github.com/mzau/mlx-knife.git@v2.0.4-beta.5"
+# Install 2.0.4-beta.6 (Workspace operations + Resumable clone)
+pip install "git+https://github.com/mzau/mlx-knife.git@v2.0.4-beta.6"
# With Vision support (Python 3.10+ required)
-pip install "git+https://github.com/mzau/mlx-knife.git@v2.0.4-beta.5#egg=mlx-knife[vision]"
+pip install "git+https://github.com/mzau/mlx-knife.git@v2.0.4-beta.6#egg=mlx-knife[vision]"
# Verify installation
-mlxk --version # → mlxk 2.0.4b5
+mlxk --version # → mlxk 2.0.4b6
```
-**Beta.5 note:** Uses mlx-vlm commit c536165df2b3b4aece3a795b2e414349f935e750 (includes Pixtral text-only fix). The `[vision]` extra automatically installs the correct version.
+**Beta.6 note:** Uses mlx-vlm commit c536165df2b3b4aece3a795b2e414349f935e750 (includes Pixtral text-only fix). The `[vision]` extra automatically installs the correct version.
**For production use:** Wait for 2.0.4 stable on PyPI (requires mlx-vlm 0.3.10 release).
@@ -169,6 +176,83 @@ mlxk run "Phi-3-mini" -c
mlxk serve --port 8080
```
+## Model References
+
+MLX-Knife supports multiple ways to reference models:
+
+### HuggingFace Models (Cache)
+
+| Format | Example | Description |
+|--------|---------|-------------|
+| Full name | `mlx-community/Phi-4-4bit` | Exact HuggingFace repo ID |
+| Short name | `Phi-4` | Fuzzy match against cache |
+| With hash | `Phi-4@e96f3b2` | Specific commit/version |
+
+```bash
+mlxk run "mlx-community/Phi-4-4bit" "Hello"
+mlxk run "Phi-4" "Hello" # Fuzzy match
+mlxk show "Qwen3@e96" --json # Specific version
+```
+
+### Local Paths (2.0.4-beta.6+)
+
+| Format | Example |
+|--------|---------|
+| Relative | `./my-workspace` |
+| Absolute | `/Volumes/External/model` |
+| Home | `~/models/fine-tuned` |
+
+```bash
+# Clone → Run
+mlxk clone org/model ./workspace
+mlxk run ./workspace "Hello"
+
+# Convert → Run
+mlxk convert ./broken ./fixed --repair-index
+mlxk run ./fixed "Test"
+```
+
+**Disambiguating paths vs cache names:** When a local directory exists with the same name as a cached model, use `./` prefix to force workspace resolution. Otherwise, cache lookup is attempted first.
+
+---
+
+## Workspace Development Workflow (2.0.4-beta.6+)
+
+**Complete local development cycle** for model experimentation, repair, and testing without HuggingFace round-trips:
+
+```bash
+export MLXK2_ENABLE_ALPHA_FEATURES=1
+
+# Clone → Repair → Test → Publish (optional)
+mlxk clone "model" ./workspace
+mlxk convert ./workspace ./fixed --repair-index # Fix mlx-vlm #624 models
+mlxk run ./fixed "test prompt" # Local inference
+mlxk server --model ./fixed # Dev server
+mlxk push ./fixed "your-org/model" # Optional publish
+```
+
+**Key capabilities:**
+- **Model repair:** Fix index/shard mismatches from mlx-vlm #624
+- **Local testing:** Run/server/show without pushing to HuggingFace
+- **Side-by-side comparison:** Multiple workspaces with parallel servers
+- **Rapid iteration:** Clone → Modify → Test loop
+
+### Workspace Commands Reference
+
+| Command | Workspace Support | Example |
+|---------|-------------------|---------|
+| `run` | ✅ Yes | `mlxk run ./workspace "prompt"` |
+| `show` | ✅ Yes | `mlxk show ./workspace --files` |
+| `health` | ✅ Yes | `mlxk health ./workspace` |
+| `server` | ✅ Yes | `mlxk server --model ./workspace` |
+| `clone` | ✅ Creates | `mlxk clone model ./workspace` |
+| `convert` | ✅ Yes | `mlxk convert ./in ./out --repair-index` |
+| `push` | ✅ Yes | `mlxk push ./workspace "org/name"` |
+| `pull` | ❌ Cache only | Downloads to HuggingFace cache |
+| `list` | ❌ Cache only | Lists cached models only |
+
+---
+
## Web Interface
For a web-based chat UI, use **[nChat](https://github.com/mzau/broke-nchat)** - a lightweight web interface for the BROKE ecosystem:
@@ -182,7 +266,7 @@ cd broke-nchat
mlxk serve
# Open web UI:
-open index.html
+open webui/index.html
```
**On-Prem:** Pure HTML/CSS/JS - runs entirely locally, zero dependencies.
@@ -240,24 +324,73 @@ mlxk run vision-model --image cat.jpg
mlxk run "mlx-community/Llama-3.2-11B-Vision-Instruct-4bit" "What is 2+2?"
```
+#### Batch Processing
+
+**Breaking Change (2.0.4-beta.6):** Vision processing now defaults to processing **one image at a time** for maximum stability on all systems. Use `--chunk N` to process multiple images per batch when your system can handle it.
+
+```bash
+# Default: one image at a time (most robust, automatic chunking)
+mlxk run pixtral "Describe image" --image photos/*.jpg
+
+# Faster: 5 images per batch (requires more RAM, may trigger model-specific issues)
+mlxk run pixtral "Describe images" --chunk 5 --image photos/*.jpg
+
+# Alternative: Use --prompt flag (useful when experimenting with different prompts)
+mlxk run pixtral --chunk 5 --image photos/*.jpg --prompt "Describe images"
+
+# Set default batch size via environment variable
+export MLXK2_VISION_BATCH_SIZE=3
+mlxk run pixtral "Describe images" --image photos/*.jpg
+```
+
+**Why chunking?**
+- **Safety:** Prevents Metal OOM crashes with large image batches
+- **Isolation:** Fresh model load per chunk prevents state leakage between batches
+- **Trade-off:** ~2-3s model load overhead per chunk vs guaranteed isolation
+
+**⚠️ Important:** Some vision models may hallucinate details about non-existent images when:
+- Processing larger chunks (model sees global context like "8 total images" but only has 4 in current batch)
+- Prompts use plural forms that don't match actual image count (e.g., "describe these images" when chunk=1)
+
+Default chunk=1 with singular prompts provides maximum robustness.
+
+**Server API:**
+```bash
+curl -X POST http://localhost:8000/v1/chat/completions \
+ -H "Content-Type: application/json" \
+ -d '{"model": "pixtral", "chunk": 3, "messages": [...]}'
+```
+
#### Metadata Output Format
-When processing images, MLX Knife automatically appends metadata in a **collapsible table** (collapsed by default):
+When processing images, MLX Knife automatically prepends metadata in a **collapsible table** (collapsed by default) **before** the model output:
```
-A beach with palm trees and clear blue water.
-
-📸 Image Metadata (2 images)
+📸 Batch 1/3: Images 1-4
| Image | Filename | Original | Location | Date | Camera |
|-------|----------|----------|----------|------|--------|
| 1 | image_abc123.jpeg | beach.jpg | 📍 32.79°N, 16.92°W | 📅 2023-12-06 12:19 | 📷 Apple iPhone SE |
| 2 | image_def456.jpeg | mountain.jpg | 📍 32.87°N, 17.17°W | 📅 2023-12-10 15:42 | 📷 Apple iPhone SE |
+| 3 | image_xyz789.jpeg | sunset.jpg | 📍 32.82°N, 17.05°W | 📅 2023-12-08 18:30 | 📷 Apple iPhone SE |
+| 4 | image_uvw456.jpeg | forest.jpg | 📍 32.88°N, 17.12°W | 📅 2023-12-09 10:15 | 📷 Apple iPhone SE |
+
+A beach with palm trees and clear blue water. A mountain landscape with snow-capped peaks...
```
+**Batch information in summary:**
+- Shows current batch and total batches (e.g., "Batch 1/3")
+- Shows image range in current batch (e.g., "Images 1-4")
+- Helps track progress in WebUI and prevents confusion about which images are being described
+- **Important:** Batch context prevents hallucination by making scope clear to both model and user
+
+**Why metadata comes first:**
+- Vision models can **reference metadata in their analysis** (filename, GPS coordinates, date visible in prompt context)
+- Clearer association with output when processing multiple chunks
+
**Metadata includes:**
- **Image ID** → **Filename mapping** (identify which description belongs to which file)
- **GPS coordinates** (latitude/longitude, if available in EXIF)
@@ -278,7 +411,11 @@ mlxk run vision-model --image photo.jpg "describe"
#### Limitations
- **Non-streaming:** Vision runs always use batch mode (no streaming output)
-- **Image limits:** 5 images max per request, 20 MB per image, 50 MB total
+- **Image limits:** Model-dependent due to Metal buffer constraints (~41.7GB on Apple Silicon)
+ - **pixtral-12b-8bit:** Up to 5 images tested on M2 Max 64GB (multi-image capable)
+ - **Llama-3.2-11B / Other models:** Single-image only
+ - **Larger models (24B+):** Limited to 1-2 images on 64GB RAM
+ - **Per-image:** 20 MB max, 50 MB total per request
#### Server API
@@ -297,6 +434,36 @@ curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/jso
}'
```
+#### Model Compatibility
+
+**⚠️ Important:** Vision support relies on mlx-vlm (upstream), which has known stability issues. While `mlxk health` verifies file integrity, **runtime failures may occur** with certain model architectures due to upstream bugs.
+
+**✅ Tested & Working Models** (mlx-knife v2.0.4-beta.6):
+
+| Model | Size | Notes |
+|-------|------|-------|
+| `mlx-community/pixtral-12b-8bit` | ~13.5GB | **⭐ Recommended:** Excellent text recognition, multi-image support (5+ images on M2 Max 64GB); beta.4+ for text-only |
+| `mlx-community/Llama-3.2-11B-Vision-Instruct-4bit` | ~6.5GB | Reliable, good quality; single-image only |
+| `Devstral-Small-2-24B-Instruct-2512-6bit` | ~14GB | Single-image only on 64GB RAM; requires `--repair-index` (mlx-vlm #624); better for Mac Studio/Ultra |
+
+**❌ Known Issues** (as of 2026-01-03):
+
+| Model | Issue | Workaround |
+|-------|-------|------------|
+| `Mistral-Small-3.1-24B-Instruct-2503-4bit` | Vision feature mismatch (5476 positions ≠ 1369 features). **Note:** `--repair-index` fixes mlx-vlm #624 but NOT this bug | Use alternative models (pixtral-12b-8bit, Llama-3.2-11B) |
+| `MiMo-VL-7B-RL-bf16` | NoneType iteration error | mlx-vlm processor bug, no workaround |
+| `DeepSeek-OCR-8bit` | Runs but hallucinates details | Quality issue, not recommended |
+
+**Models affected by mlx-vlm #624** (index/shard mismatch) can be repaired:
+
+```bash
+mlxk convert