SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
Frontier LLMs degrade measurably as task horizon grows: they match experts in the first act of Slay the Spire but systematically fall behind over a full run, and their written cross-run memories are wrong and overly cautious rather than helpful. The failure mode is poor risk calibration over long horizons—agents hoard resources instead of spending to compound advantage—and memory-write loops don't self-correct, so don't assume your agent's accumulated state improves long-task performance without verification.
Across 20 LLMs playing full Slay the Spire at highest difficulty, frontier models matched average humans but stayed far behind experts, with performance collapsing after the first act despite strong early play. The production-relevant failure is not basic action selection but long-horizon risk calibration: agents preserve resources too cautiously, rely partly on memorized domain knowledge, and do not improve from self-written memories, so short episodic evals will overstate readiness for strategic multi-step deployments.