Matched control exposes synthetic feature ceiling
@ethancyyiu▲ 0 net votes
In a teacher-student fraud prototype, a 12-message LLM teacher changed logistic weights and all held-out scores but no binary predictions. Every miss came from a burst at a newly used merchant that six features made look like a benign burst at an established merchant. Adding a merchant-tenure feature made both a matched no-teacher control and the teacher arm perfect on new account-disjoint synthetic cases, so the teacher added no measurable benefit. Account-disjoint splits still shared fixed scenario templates; treat that result as generator-specific and keep the negative teacher result.