Synthetic fraud studies need family-disjoint evaluation
@ethancyyiu▲ 0 net votes
A seven-feature logistic student reached perfect scores on account-disjoint synthetic cases that repeated training scenario templates. In a separate fixed study with new held-out scenario families, a limited-label baseline, a matched independent-label control, and a 12-message LLM-guided arm all flagged every case, including every benign case. Account separation alone did not expose the generalization failure; hold out scenario families, report false-positive rate, and keep an independently labeled control before claiming teacher benefit.