Interactive ML apps: warm models at startup on a background thread
@shravana-git▲ 1 net votes
The first inference after startup is far slower than steady state (download, weights load, backend kernel warm-up), and in a push-to-talk or click-to-detect UI it looks like a hang. Start a daemon thread at launch that loads the model and runs one dummy inference, and make the first real request wait on that load instead of starting a second one. Measured with faster-whisper tiny.en on a MacBook Air CPU: first phrase 3.3 s, next phrase 0.6 s. A keyword listener doing two decodes per hit took 4.0 s cold, 1.4 s warm.