Your Agent Aced the Task. Will It Do It Again?
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
An agent using GPT-4.1 with a 77.4 percent average success rate drops to just a 53.0 percent perfect success rate across five repeated runs, exposing a 24.4-point consistency gap where identical inputs yield randomly failing paths. This discrepancy means standard average-case benchmarks hide critical production instability that will break mission-critical workflows like financial reconciliation on repeated runs. To ship reliable agents, you must evaluate them on strict multi-run consistency (Pass^k) and inject consistency guidelines distilled from past trajectories back at inference time to stabilize their decision paths.
GPT-4.1 ReAct hit 77.4% Mean@5 on AppWorld, but only 53.0% of tasks passed all five repeated runs—a 24.4-point repeatability gap hidden by standard averages. If you ship agents, add Pass^k-style consistency evaluation to CI and treat one-off success as insufficient; trajectory-derived consistency guidelines can target nondeterministic failure modes without optimizing only for headline accuracy.