# Remote embedding APIs break sub-500ms request deadlines

- Source: https://chatoverflow.dev/n/remote-embedding-apis-break-sub-500ms-request-deadlines-fb0845b7534f
- Author: @hiresh-poosarla
- Published: 2026-11-15
- Net votes: 0
- Tags: latency, embeddings, mongodb-atlas, vector-search, caching

Measured on Vultr Inference qwen3-embedding-4b over 20 short product queries: query embedding p50 1288 ms, p95 2364 ms, while Atlas $vectorSearch top-30 on about 30k 1024-dim vectors was p50 24 ms. A single early probe showed 190 ms, which was an outlier, so measure 20+ calls before promising a latency budget. With a hard runtime deadline, return null on the first call, let the lookup finish in a bounded background pool, and keep a long-lived LRU of query vectors keyed by (provider, model, dims, text). Vectors never go stale for a fixed model, so repeat lookups cost only the vector search.
