Replace tokenizers-cpp with our own Rust tokenizer FFI and add zero-shot classification
ort-server
A generic, self-contained ONNX Runtime model server for Lemonade. It loads one exported ONNX model + a small manifest and serves it over HTTP. Lemonade launches it as a subprocess (like moonshine-server) and forwards requests to it.
Consumer: lemonade's onnxruntime backend (/v1/classify, #2592). The router's classifier condition type will consume that endpoint once #2384 wires it. Classification is the first capability; the server is intentionally generic so embeddings and reranking can follow.
Status: experimental — released and consumed by Lemonade's
onnxruntimebackend. See #2592.
Scope
ort-server serves text-modality ONNX graphs — classification today, embeddings
and reranking next. Because it tokenizes from the model's own tokenizer.json,
any HuggingFace text classifier runs from a standard ONNX export with no per-model
work. Vision and audio models are out of scope by design: those modalities are
served by other Lemonade backends (images → stable-diffusion.cpp; audio →
whisper / moonshine / kokoro).
Design
The server is thin, and a model is easy to bring: a stock Optimum export runs
as-is — optimum-cli export onnx --model <hf_id> <dir> and point ort-server
at the directory.
Supported architectures. ort-server implements the single-sequence encoder
convention: an all-ones attention mask, all-zero token_type_ids, and
trailing-token truncation. That is correct for BERT, DistilBERT, RoBERTa,
XLM-RoBERTa, DeBERTa (v1/v2), ELECTRA, ALBERT and CamemBERT, and not for
architectures with different segment/special-token conventions (XLNet, for
instance, puts its classifier token last with a distinct segment id). The
supported set is an explicit allowlist checked against config.json's
model_type at startup — an unsupported model is refused, not served with
silently wrong scores. The allowlist is per task: zero-shot routing uses a
different input convention entirely, so it has its own list and the two do not
mix in either direction.
model.onnxis a plain export (input_ids/attention_mask[/token_type_ids] → logits) — the ordinaryoptimumexport, no in-graph baking, no custom ops.- The server tokenizes at runtime by loading the model's
tokenizer.jsonthroughthird_party/tok_ffi, a small C ABI over the HuggingFacetokenizerscrate — the same Rust tokenizertransformersuses.
What "parity" means here, precisely. The tokenizer's ids are exact, but the
attention mask, segment ids, padding handling and special-token filtering are
reconstructed by this server to match the HuggingFace pipeline. That reconstruction
is verified against HF-computed reference scores on every release
(test/fixtures/*/golden.json), but it is only valid for the single-sequence
encoder families in the allowlist below — it is not a general guarantee for "any
HuggingFace classifier". The tokenizer half of that claim has its own harness,
test/parity_tokenizer.py, which compares ids and character offsets against
HuggingFace case by case; run it whenever the tokenizer moves.
- The output contract (task, labels, normalization, token budget) is inferred
from the export's own
config.json+tokenizer_config.json.problem_typeis only a training-time hint, so manifest-less inference assumes single-label softmax: a multi-label model must declareproblem_type: multi_label_classificationor ship a manifest with"score_normalization": "sigmoid". Regression heads are rejected (no label scores in [0,1]). - An optional
manifest.jsonoverrides the inference; when present it is the contract and is validated strictly.
model-dir/
model.onnx # plain export: input_ids/attention_mask -> logits
tokenizer.json # the model's HuggingFace tokenizer
config.json # stock HF config (id2label / problem_type / architectures)
tokenizer_config.json # model_max_length
manifest.json # OPTIONAL explicit override of the inferred contract
manifest.json override (validated at startup — unknown values are a startup
error, and the model's output dimension must match id2label at inference time):
{
"task": "text-classification", // or "token-classification"
"id2label": {"0": "SAFE", "1": "INJECTION"},
"score_normalization": "softmax", // "softmax" (default) | "sigmoid"
"token_aggregation": null, // token-classification: "max" (default) | "mean"
"max_length": 512 // optional token budget; longer inputs are truncated
}
Zero-shot routing
task: "zero-shot-classification" takes its labels from each request
instead of from a manifest, and scores them against the text on the fly:
POST /classify
{"text": "Write a Python function that merges two sorted lists.",
"labels": ["code generation", "math reasoning", "creative writing"]}
Such a model reads the labels as part of its input text and needs two extra pooling inputs, built here from the tokenizer's character offsets. It has its own architecture allowlist, its own token budget, its own request limits and its own truncation rule, all described in docs/ZERO-SHOT-ROUTER.md.
A ready-to-serve model directory is
lemonade-sdk/LFM2.5-Encoder-350M-Prompt-Router-ONNX,
converted from
LiquidAI/LFM2.5-Encoder-350M-Prompt-Router
with tools/zero_shot_router/export.py.
HTTP contract
| Method | Path | Body | Response |
|---|---|---|---|
| GET | /health |
— | 200 when the model is loaded and ready: {"status": "ok", "version": "<ort-server>", "onnxruntime": "<ORT>"} |
| POST | /classify |
{"text": "...", "top_k": N?, "labels": [...]?} |
{"labels": {"<label>": <score in [0,1]>, ...}, "truncated": <bool>} — top_k omitted or 0 returns all labels |
labels is required for a zero-shot-classification model and rejected for any
other; both mismatches are a 400 rather than a silently ignored field.
truncated is on every response, for any task: true when the text did not fit
the token window and was cut, false when it fitted. What the cut drops
depends on the path: the encoder tasks keep the head and drop the end, while
zero-shot-classification keeps the head and the tail and drops the middle.
Servers before 0.4.0 cut without reporting it: they send no truncated, and
their /health carries no version. A missing field therefore means "unknown",
never "fitted".
Future capabilities (same server, new endpoints): POST /embed, POST /rerank.
Model catalog tooling
tools/classifier_catalog/ holds the Python tooling that produces the
lemonade-sdk HF catalog for this server: export.py (Optimum export +
manifest + parity validation vs the PyTorch reference) and publish.py
(fail-closed license allowlist + parity gates + model cards). It lives in
this repo so the manifest contract's producer and consumer version together.
Execution providers
v1 ships CPU EP only. The roadmap adds providers as build variants without changing the code:
cpu → dml (Windows GPU/NPU) → rocm (AMD GPU) → vitisai (Ryzen NPU).
Build
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release
./build/ort-server --model-path <model-dir> --port 8100
CMake fetches ONNX Runtime, cpp-httplib, and nlohmann/json (see CMakeLists.txt) and builds third_party/tok_ffi in-tree. tok_ffi is a Rust static lib, so cargo/rustup must be on PATH to build ort-server from source (build-time only — users of the prebuilt binary need nothing).
Releases
CPU-only, so release bundles build on GitHub-hosted runners (windows-latest, ubuntu-latest, macos-latest) — no self-hosted/GPU runners for v1. Pushing a v* tag builds and attaches per-platform archives named to match Lemonade's downloader:
ort-server-<version>-windows-x64.zip
ort-server-<version>-linux-x64.tar.gz
ort-server-<version>-linux-arm64.tar.gz
ort-server-<version>-macos-arm64.tar.gz
License
Apache-2.0. Release archives bundle the ONNX Runtime shared library and statically link the tokenizer stack; see THIRD_PARTY_NOTICES.md.