ort-server

A generic, self-contained ONNX Runtime model server for Lemonade. It loads one exported ONNX model + a small manifest and serves it over HTTP. Lemonade launches it as a subprocess (like moonshine-server) and forwards requests to it.

Consumer: lemonade's onnxruntime backend (/v1/classify, #2592). The router's classifier condition type will consume that endpoint once #2384 wires it. Classification is the first capability; the server is intentionally generic so embeddings and reranking can follow.

Status: experimental — released and consumed by Lemonade's onnxruntime backend. See #2592.

Scope

ort-server serves text-modality ONNX graphs — classification today, embeddings and reranking next. Because it tokenizes from the model's own tokenizer.json, any HuggingFace text classifier runs from a standard ONNX export with no per-model work. Vision and audio models are out of scope by design: those modalities are served by other Lemonade backends (images → stable-diffusion.cpp; audio → whisper / moonshine / kokoro).

Design

The server is thin, and a model is easy to bring: a stock Optimum export runs as-is — optimum-cli export onnx --model <hf_id> <dir> and point ort-server at the directory.

Supported architectures. ort-server implements the single-sequence encoder convention: an all-ones attention mask, all-zero token_type_ids, and trailing-token truncation. That is correct for BERT, DistilBERT, RoBERTa, XLM-RoBERTa, DeBERTa (v1/v2), ELECTRA, ALBERT and CamemBERT, and not for architectures with different segment/special-token conventions (XLNet, for instance, puts its classifier token last with a distinct segment id). The supported set is an explicit allowlist checked against config.json's model_type at startup — an unsupported model is refused, not served with silently wrong scores. The allowlist is per task: zero-shot routing uses a different input convention entirely, so it has its own list and the two do not mix in either direction.

  • model.onnx is a plain export (input_ids/attention_mask[/token_type_ids] → logits) — the ordinary optimum export, no in-graph baking, no custom ops.
  • The server tokenizes at runtime by loading the model's tokenizer.json through third_party/tok_ffi, a small C ABI over the HuggingFace tokenizers crate — the same Rust tokenizer transformers uses.

What "parity" means here, precisely. The tokenizer's ids are exact, but the attention mask, segment ids, padding handling and special-token filtering are reconstructed by this server to match the HuggingFace pipeline. That reconstruction is verified against HF-computed reference scores on every release (test/fixtures/*/golden.json), but it is only valid for the single-sequence encoder families in the allowlist below — it is not a general guarantee for "any HuggingFace classifier". The tokenizer half of that claim has its own harness, test/parity_tokenizer.py, which compares ids and character offsets against HuggingFace case by case; run it whenever the tokenizer moves.

  • The output contract (task, labels, normalization, token budget) is inferred from the export's own config.json + tokenizer_config.json. problem_type is only a training-time hint, so manifest-less inference assumes single-label softmax: a multi-label model must declare problem_type: multi_label_classification or ship a manifest with "score_normalization": "sigmoid". Regression heads are rejected (no label scores in [0,1]).
  • An optional manifest.json overrides the inference; when present it is the contract and is validated strictly.
model-dir/
  model.onnx             # plain export: input_ids/attention_mask -> logits
  tokenizer.json         # the model's HuggingFace tokenizer
  config.json            # stock HF config (id2label / problem_type / architectures)
  tokenizer_config.json  # model_max_length
  manifest.json          # OPTIONAL explicit override of the inferred contract

manifest.json override (validated at startup — unknown values are a startup error, and the model's output dimension must match id2label at inference time):

{
  "task": "text-classification",        // or "token-classification"
  "id2label": {"0": "SAFE", "1": "INJECTION"},
  "score_normalization": "softmax",     // "softmax" (default) | "sigmoid"
  "token_aggregation": null,            // token-classification: "max" (default) | "mean"
  "max_length": 512                     // optional token budget; longer inputs are truncated
}

Zero-shot routing

task: "zero-shot-classification" takes its labels from each request instead of from a manifest, and scores them against the text on the fly:

POST /classify
{"text": "Write a Python function that merges two sorted lists.",
 "labels": ["code generation", "math reasoning", "creative writing"]}

Such a model reads the labels as part of its input text and needs two extra pooling inputs, built here from the tokenizer's character offsets. It has its own architecture allowlist, its own token budget, its own request limits and its own truncation rule, all described in docs/ZERO-SHOT-ROUTER.md.

A ready-to-serve model directory is lemonade-sdk/LFM2.5-Encoder-350M-Prompt-Router-ONNX, converted from LiquidAI/LFM2.5-Encoder-350M-Prompt-Router with tools/zero_shot_router/export.py.

HTTP contract

Method Path Body Response
GET /health — 200 when the model is loaded and ready: {"status": "ok", "version": "<ort-server>", "onnxruntime": "<ORT>"}
POST /classify {"text": "...", "top_k": N?, "labels": [...]?} {"labels": {"<label>": <score in [0,1]>, ...}, "truncated": <bool>} — top_k omitted or 0 returns all labels

labels is required for a zero-shot-classification model and rejected for any other; both mismatches are a 400 rather than a silently ignored field.

truncated is on every response, for any task: true when the text did not fit the token window and was cut, false when it fitted. What the cut drops depends on the path: the encoder tasks keep the head and drop the end, while zero-shot-classification keeps the head and the tail and drops the middle.

Servers before 0.4.0 cut without reporting it: they send no truncated, and their /health carries no version. A missing field therefore means "unknown", never "fitted".

Future capabilities (same server, new endpoints): POST /embed, POST /rerank.

Model catalog tooling

tools/classifier_catalog/ holds the Python tooling that produces the lemonade-sdk HF catalog for this server: export.py (Optimum export + manifest + parity validation vs the PyTorch reference) and publish.py (fail-closed license allowlist + parity gates + model cards). It lives in this repo so the manifest contract's producer and consumer version together.

Execution providers

v1 ships CPU EP only. The roadmap adds providers as build variants without changing the code:

cpu → dml (Windows GPU/NPU) → rocm (AMD GPU) → vitisai (Ryzen NPU).

Build

cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release
./build/ort-server --model-path <model-dir> --port 8100

CMake fetches ONNX Runtime, cpp-httplib, and nlohmann/json (see CMakeLists.txt) and builds third_party/tok_ffi in-tree. tok_ffi is a Rust static lib, so cargo/rustup must be on PATH to build ort-server from source (build-time only — users of the prebuilt binary need nothing).

Releases

CPU-only, so release bundles build on GitHub-hosted runners (windows-latest, ubuntu-latest, macos-latest) — no self-hosted/GPU runners for v1. Pushing a v* tag builds and attaches per-platform archives named to match Lemonade's downloader:

ort-server-<version>-windows-x64.zip
ort-server-<version>-linux-x64.tar.gz
ort-server-<version>-linux-arm64.tar.gz
ort-server-<version>-macos-arm64.tar.gz

License

Apache-2.0. Release archives bundle the ONNX Runtime shared library and statically link the tokenizer stack; see THIRD_PARTY_NOTICES.md.

S
Description
Generic C++ ONNX Runtime model server for Lemonade (classification first; multi-EP roadmap). Consumer: lemonade #2592.
Readme Apache-2.0
730 KiB
0 Stars 1 Watchers 0 Forks
0.4.0
Latest
2026-11-21 05:02:48 -10:00
Languages
Python 60.4%
C++ 35.3%
CMake 4.3%