Skip to main content

ADR-0016: Artifact resolution and the ONNX model contract

Status​

Accepted

Context​

Phase 4 brings local inference: RouterProfile references model artifacts (classifier.uri, classifier.tokenizerUri, embedder.uri, security.piiDetector, security.injectionDetector) that the router must fetch, verify and load, and the ONNX classifier needs a defined tensor contract so the trainer (phase 6) and any externally produced model agree with the runtime. Models are supply-chain sensitive: a swapped classifier can silently loosen routing, so artifact integrity is part of the security posture, not a convenience.

Decision​

Artifact URIs and verification (crates/artifact)​

  • https://<host>/<path>@sha256:<64 hex>: fetched over TLS, digest required; the fetched bytes must hash to the pinned digest or resolution fails. http:// is accepted with the same mandatory digest (the pin makes the transport tamper-evident; useful for in-cluster registries), but logs a warning.
  • file:///<path>: loaded as-is, no digest required (air-gapped installs and tests own their filesystem). An optional @sha256: suffix is verified when present.
  • oci://: reserved, rejected with a clear error for now. Pulling from OCI registries lands with the artifact-signing work (phase 7); the URI syntax in the CRDs already anticipates it.
  • Cache: content-addressed under $ROUTED_MODEL_CACHE (default ~/.cache/routed/models, /var/cache/routed/models when $HOME is unset): sha256/<hex> for pinned artifacts. A cache hit re-verifies the digest before reuse; downloads go to a temp file and are renamed into place only after verification. routedctl models pull <uri> pre-warms the same cache the router uses.

ONNX classifier contract (feature onnx)​

One multi-head encoder classifies each request. Inputs, produced by the tokenizers crate from classifier.tokenizerUri (a HuggingFace tokenizer.json), truncated to the model's sequence budget:

  • input_ids: int64 [1, seq]
  • attention_mask: int64 [1, seq]

Outputs, all optional except risk (ADR-0006: a classifier must always report a risk score):

  • task_logits: f32 [1, len(labels.task)], argmax indexes classifier.labels.task
  • complexity_logits: f32 [1, 3] for low / medium / high
  • sensitivity_logits: f32 [1, len(labels.sensitivity)], argmax indexes classifier.labels.sensitivity (a DataClass name)
  • risk: f32 [1] or [1, 1], already a probability in [0, 1]

The classified text is the same selection every implementation uses (ADR-0006): truncated system prompt, last user message, tool outputs.

The ONNX classifier composes with the heuristics rather than replacing them: PII entities always come from routed-security's span detectors (a sequence classifier cannot produce them), and the reported risk score is max(model risk, heuristic injection score) - the model can only tighten, never loosen, what the heuristics already catch. Dedicated PII / injection detector models (security.piiDetector / injectionDetector) ride the same session-and-artifact plumbing in a later increment; until then those references are accepted and unused.

Build and packaging​

ort (ONNX Runtime) and tokenizers sit behind the onnx cargo feature (ADR-0002); the default build, make ci, and the default container images stay ONNX-free. The pyke-published static binaries need a newer libstdc++ than the pinned bookworm toolchain ships, so the feature uses ADR-0002's fallback (1): ort in load-dynamic mode, loading the Microsoft-published libonnxruntime.so named by ORT_DYLIB_PATH at runtime. make onnx downloads that library into .cache/, then builds and tests the feature (including the p95 < 30 ms gate under ROUTED_PERF=1, docs/performance.md); an ONNX-enabled image must ship the same library and set ORT_DYLIB_PATH. Test fixtures (a tiny valid model and tokenizer) are generated by trainer/scripts/make_classifier_fixture.py and committed, so the feature tests run without network or Python.

Consequences​

  • A profile with type: onnx fails at load (not per request) when the binary lacks the feature, the artifacts don't resolve, or the digest does not match - the router then refuses readiness rather than serving with a silently missing classifier.
  • Every artifact fetch is reproducible: same URI, same bytes, or an error.
  • The trainer has a fixed export target; changing head names or dtypes is a breaking change to this ADR.

Alternatives considered​

  • Bundling models into the container image: rejected as the only mechanism (image rebuilds per model change); it still works via file:// for those who want it.
  • A tarball artifact holding model + tokenizer under one digest: rejected for now; two pinned URIs are simpler than unpacking semantics, and OCI artifacts will subsume the bundle case.
  • Making the ONNX model emit PII spans: rejected; span-level detection is a different model family, and the heuristic detectors already meet the conformance contract.