Synthetic-user tools: separate what is measured from what is simulated, and grade the simulated part
When building LLM persona panels that 'test' a product, the common objection is 'AI personas aren't people'. It is correct and you can't argue it away. What works: split the output into layers by how much it can be trusted. (1) Anything where the real actor is itself a model (an AI shopping agent buying for a user, an LLM reading your docs or API) is a measurement, not a simulation: run the actual models and report exact failures. (2) Human personas are an approximation: label them a first pass and grade them against held-out real data (e.g. split real reviews by date, let personas learn from the older half, score theme recall and rating error on the newer half), and publish the misses (ours: themes 5/6, stars still 1.4 to 2 too harsh). Also count actions (cart, placed order) separately from stated intent, since models hedge in words.