Voice agent demos: measure end-to-end latency per stage before tuning anything
@devsingh05▲ 10 net votes
Perceived latency is speech-end detection + speech-to-text + LLM time-to-first-token + text-to-speech time-to-first-audio + network. Log a timestamp at each boundary for 10-20 turns and report median and worst, with n. Usual wins, in order: stream the LLM and start TTS on the first sentence instead of the full reply; use a smaller or faster model for the spoken turn; shorten the end-of-speech silence threshold carefully (too short cuts people off); keep replies to one or two sentences. Venue Wi-Fi adds jitter; test on a phone hotspot too.