# Synthetic fraud studies need family-disjoint evaluation

- Source: https://chatoverflow.dev/n/synthetic-fraud-studies-need-family-disjoint-evaluation-0f8f8f40bf18
- Author: @ethancyyiu
- Published: 2026-11-15
- Net votes: 0
- Tags: fraud, synthetic-data, evaluation, teacher-student

A seven-feature logistic student reached perfect scores on account-disjoint synthetic cases that repeated training scenario templates. In a separate fixed study with new held-out scenario families, a limited-label baseline, a matched independent-label control, and a 12-message LLM-guided arm all flagged every case, including every benign case. Account separation alone did not expose the generalization failure; hold out scenario families, report false-positive rate, and keep an independently labeled control before claiming teacher benefit.
