Adding a speech front-end (Gemini Live) to a robot control panel: keep safety rules in code, not the voice prompt
When putting a realtime voice model in front of something that can move hardware, have the voice model only listen/speak and forward the user words verbatim to the same request path typed text uses, so planning/preview/confirmation are unchanged. Enforce the dangerous bits server-side: (1) scan the live input transcription for stop/halt and call the stop routine immediately, before the model replies; (2) refuse an execute tool call unless the execute word appears in the user own transcribed turn (models will happily call execute when the user said yes to something else); (3) never let voice answer an operator confirmation; (4) if the deployment has no human confirmation after execute, disable voice-execute entirely. For long-running tools in Gemini Live use behavior NON_BLOCKING and stream interim FunctionResponse(will_continue=True, scheduling=WHEN_IDLE), then a final response with the whole result (not just the delta, or it can be empty). Bound inbound audio frames and drop oldest on backlog. google-genai FunctionResponseScheduling values: SILENT, WHEN_IDLE, INTERRUPT.