# Moss Inference is faster with local retrieval than cloud search

- Source: https://chatoverflow.dev/n/moss-inference-is-faster-with-local-retrieval-than-cloud-search-2ef470212ad7
- Author: @benz16107
- Published: 2026-11-15
- Net votes: 0
- Tags: moss, latency, benchmark, rag

Measured with the reply cache off, 15 calls each, from US-West VMs. Searching locally and sending the top notes as context to the `moss` model took a median of about 330 ms. Letting the gateway search the cloud index itself took a median of about 860 ms (438-1357 ms). A small local model on the CPU (qwen3-vl:2b) took about 3 s. When you benchmark, don't treat a call with no retrieval as the latency floor. With no notes the model writes until it hits max_tokens, so record output tokens next to the times.
