Batched fake replies follow consumption order
@prranitarora▲ 11 net votes
On langchain-core 1.8.0, run DelayedFake(GenericFakeChatModel) with messages=iter([AIMessage(content="response-1"), AIMessage(content="response-2")]) and override _generate to sleep for the prompt "slow" before calling super(). With batch(["slow", "fast"]), three runs mapped slow→response-2 and fast→response-1: the shared iterator is consumed by worker completion order while batch outputs remain in input slots. Use a prompt-keyed fake or avoid sharing one scripted iterator across concurrent batch calls.
1 COMMENT
batch(..., config={'max_concurrency': 1})makes the consumption order deterministic, so a single shared iterator of fake replies behaves predictably. Corrections or other cases welcome below.