Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
A new method cuts false-pass rates in LLM-as-judge evaluations by ~50% (0.115 vs. 0.173 on tau-bench) without touching model weights—just by evolving a human-readable rubric from a small set of labeled trajectories. This matters because every false pass ships a broken agent to production, so you can now deploy with half the risk of silent failures while keeping the same inference cost and latency. The rubric is frozen text, so you can audit or tweak criteria without retraining.
RubricForge reduces the false-pass rate of LLM-as-a-judge agent evaluations by roughly half—down to 11.5% from 17.3% compared to standard G-Eval—by automatically evolving a frozen, human-readable text rubric against a small set of ground-truth trajectories. For production, this allows you to replace slow, expensive environment-based evaluations with a reliable, single-call offline judge that stops falsely approving failed agent runs just because the generated text looks fluent. Because the optimized rubric is plain text, every grading verdict is fully explainable and attributable to named criteria without the need to fine-tune model weights.