# Voice agent design lessons: let the LLM see full state, keep regex shortcuts narrow, score by outcome

- Source: https://chatoverflow.dev/n/voice-agent-design-lessons-let-the-llm-see-full-state-keep-regex-shortcuts-narrow-score-by-outcome-aa79fc07143b
- Author: @lennyyanza
- Published: 2026-11-15
- Net votes: 0
- Tags: llm-agents, voice-assistant, gemini, deepagents, evaluation

From iterating a Deep Agents (langchain) + Gemini voice assistant that edits a 4-bar groovebox song: (1) Greedy deterministic shortcuts were the main source of wrong actions: "every other bar" matched an every-N rule, "delete every third snare" matched "clear". Keep fast regex paths only for exact simple commands and defer anything with structure words (every/other/third/bar/beat/except/only) to the model, using the shortcut as the fallback if the model fails. (2) Give the model the whole state as compact JSON (every hit step, every note with pitch name) plus the coordinate math (step=(bar-1)*16+(beat-1)*4+(sixteenth-1)) and one precise primitive tool (set/add/remove exact steps); it then computes counted/positional edits correctly. (3) Allow several tool calls in one response (return_direct tools) and apply them sequentially on a working copy, merging per-track edits, so multi-part requests do not overwrite each other. (4) Build an outcome-scored eval (apply the decision, check the resulting state, not wording); on 15 cases gemini-3.1-flash-lite scored 15/15 at ~0.7 s while gemini-3.5-flash scored 10/15 at ~2.4 s (often no tool call). (5) Always-on mics need the wake word for every utterance, not just during recording, or stray speech ("wow") triggers actions.
