Skip to main content

ADR-0002: Rust for all runtime components; ONNX for local inference

Status​

Accepted

Context​

routeD sits in the request path and must add well under 30 ms p95 with local classifiers on CPU, while running small transformer models (task, complexity, sensitivity, injection, embedding). The original plan used Go; Go's ONNX Runtime bindings rely on dlopen through purego, produce dynamically linked binaries that break distroless/static, and there is no maintained pure-Go Hugging Face tokenizer. The project owner decided to build in Rust.

Decision​

  • All runtime components (router, operator, CLI, artifact fetcher) are Rust (toolchain pinned in rust-toolchain.toml, edition 2024). Python 3.12 with uv is used only in trainer/ for offline training and ONNX export.
  • Local inference uses ONNX Runtime through the ort crate, statically linked where the platform allows, behind the onnx cargo feature so the core builds and tests without it. Tokenization uses the native tokenizers crate.
  • The classifier, embedder, and learned router are traits (routed-classify, routed-embed, routed-router); ONNX implementations are one option, type: http external services are another. The engine never sees the implementation.
  • The Kubernetes operator uses kube/kube-runtime. CRD schemas are derived (KubeSchema + schemars) and the CRD YAML is generated by routedctl crd gen and committed under config/crd/, with CI drift checks.
  • Fallback ladder if static ort is unworkable on a platform: (1) ort load-dynamic with a Microsoft-published libonnxruntime.so layer in the image, (2) type: http sidecar running ONNX Runtime in Python, (3) heuristic classifiers (degraded mode, always available).

Consequences​

  • No GC pauses in the hot path; CPU-bound inference runs on a bounded blocking pool with strict timeouts.
  • Runtime images are gcr.io/distroless/cc-debian12:nonroot (glibc + libstdc++ for ONNX Runtime); the builder and runtime base must stay on the same Debian release.
  • Compared to kubebuilder we give up scaffolding generators and envtest; we compensate with fake-client unit tests, an apiserver-in-a-box harness, and kind e2e (a dedicated test-pyramid ADR may follow).
  • Contributors need podman (or a matching host toolchain); see ADR-0005.

Alternatives considered​

  • Go (previous plan): rejected for the ONNX/tokenizer gaps above.
  • Python data plane: rejected on latency and memory grounds.
  • Hybrid Go operator + Rust router: rejected to avoid two toolchains and a duplicated policy compiler for routedctl.