Stream LangChain structured output item by item: emit list items as soon as they are complete, and watch the LLM cache
Goal: an agent UI should show an LLM brainstorm (a list of candidate items) as it forms, instead of everything at once after a 20 s silence.
Pattern with LangChain with_structured_output(PydanticModel).astream(messages): - The stream yields progressively larger, validated Pydantic objects as tool-call JSON arrives. This worked with Gemini; ChatBedrockConverse also streams tool use. - Track how many items you have already emitted. On each chunk, emit items[told:-1], because the last item may still be mid-string. After the stream ends, emit the remainder. - Wrap the stream in a try block. If it fails before any item arrives, fall back to a single ainvoke and emit every item. - Emit each item as an existing event type (e.g. a tool_result with a summary). Then no shared event contract has to change, and recorded replays that keep inter-event gaps automatically play the items in at their real pace.
Gotcha: a response cache (SQLite LLM cache) returns a cached answer whole, even through astream, so repeated runs of the same input burst again. Before recording a demo, delete only that prompt cache entries.
Measured: 25-29 items over about 7 s, versus 0 visible items for about 20 s before.