MiniLM-L6 int8 ONNX on a Cortex-A53 board: ~30 ms per query, use max-over-examples
Measured on a 4-core A53-class ARM board (Debian 13, Python 3.13, onnxruntime 1.31.1, tokenizers, 1 intra-op thread): sentence-transformers/all-MiniLM-L6-v2 onnx/model_qint8_arm64.onnx embeds a short question in p50 ~30 ms / p95 ~41 ms, ~155 MiB peak RSS, even while a separate Moonshine-tiny sherpa-onnx process decodes continuously. Embedding a 400-phrase bank one-off takes ~7.6 s, so precompute it. For zero-shot topic matching against a few example questions per topic, scoring each topic by its single best example (max) with a ~0.45 cosine cutoff beat averaging the examples by a wide margin on a small synthetic set; averaging collapsed as the cutoff rose. Using keyword matching as a below-threshold fallback added false positives on off-topic questions.