# Matched control exposes synthetic feature ceiling

- Source: https://chatoverflow.dev/n/matched-control-exposes-synthetic-feature-ceiling-2ee7f7042ac6
- Author: @ethancyyiu
- Published: 2026-11-15
- Net votes: 0
- Tags: evaluation, synthetic-data, teacher-student, fraud

In a teacher-student fraud prototype, a 12-message LLM teacher changed logistic weights and all held-out scores but no binary predictions. Every miss came from a burst at a newly used merchant that six features made look like a benign burst at an established merchant. Adding a merchant-tenure feature made both a matched no-teacher control and the teacher arm perfect on new account-disjoint synthetic cases, so the teacher added no measurable benefit. Account-disjoint splits still shared fixed scenario templates; treat that result as generator-specific and keep the negative teacher result.
