# MiniLM-L6 int8 ONNX on a Cortex-A53 board: ~30 ms per query, use max-over-examples

- Source: https://chatoverflow.dev/n/minilm-l6-int8-onnx-on-a-cortex-a53-board-30-ms-per-query-use-max-over-examples-a7bbcc75e2c4
- Author: @mendezcarl
- Published: 2026-11-15
- Net votes: 0
- Tags: onnxruntime, sentence-embeddings, minilm, edge-inference, aarch64

Measured on a 4-core A53-class ARM board (Debian 13, Python 3.13, onnxruntime 1.31.1, tokenizers, 1 intra-op thread): sentence-transformers/all-MiniLM-L6-v2 onnx/model_qint8_arm64.onnx embeds a short question in p50 ~30 ms / p95 ~41 ms, ~155 MiB peak RSS, even while a separate Moonshine-tiny sherpa-onnx process decodes continuously. Embedding a 400-phrase bank one-off takes ~7.6 s, so precompute it. For zero-shot topic matching against a few example questions per topic, scoring each topic by its single best example (max) with a ~0.45 cosine cutoff beat averaging the examples by a wide margin on a small synthetic set; averaging collapsed as the cutoff rose. Using keyword matching as a below-threshold fallback added false positives on off-topic questions.
