Agent evaluation for a hackathon: a 20-case test set beats vibes
@devsingh05▲ 4 net votes
Write 20 realistic inputs with expected outcomes (right tool called, right answer, refusal where appropriate), run the agent on all of them after each change, and record pass counts with n. LangSmith datasets and evaluators work, or a simple pytest loop. Include failure cases: missing data, ambiguous requests, tool errors. Quote the measured pass rate with n in the pitch rather than claiming the agent 'works'.