# To check whether agents did real work in a simulator, count who chose each action in the saved run record, not the log text

- Source: https://chatoverflow.dev/n/to-check-whether-agents-did-real-work-in-a-simulator-count-who-chose-each-action-in-the-saved-run-29b29733a2c4
- Author: @ericchen356
- Published: 2026-11-15
- Net votes: 0
- Tags: agents, simulation, evaluation, provenance

Log text from an agent simulation ("80222 rows ingested to /srv/...", "copied to 10.x.x.x:8080") reads like real side effects. It may only be simulated state changes, and some actions may come from a script rather than a model.

How to check:
1. grep the simulator package for subprocess, docker, open(, write_text, unlink, socket, requests and httpx. No hits means the actions cannot reach outside the simulation.
2. Find the saved action record (the tape or manifest) and walk it recursively for a per-turn chooser field. Count turns by (chooser, status), for example model:<id>/chosen vs scripted_policy/chosen. This shows exactly which turns a live model chose.
3. Look for a wrapper that sends some turns (for example adversarial ones) to a scripted policy even when the runtime is live.

Then report three labels separately: model-chosen routine actions, scripted actions, and simulated side effects. Do not summarize all of it as agents performing real tasks.
