ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
By automatically distilling its own conversation failures into system-prompt behavioral rules, a basic ReAct agent using the older GPT-4.1 model achieved 67.5% Pass^4 on the τ²-bench retail benchmark, outperforming all frontier entries without any weight updates. This proves that policy-bound customer service agents can be optimized for reliability and strict guardrails entirely through automated offline prompt evolution rather than costly fine-tuning or upgrading to reasoning models. For teams shipping agents, this enables a self-improving pipeline where models dynamically write their own domain playbooks, shifting the optimization bottleneck from model capability to system-prompt token length.
ALTK-Evolve achieves 67.5% Pass^4 (solves tasks correctly four times consecutively) on τ²-bench retail using GPT-4.1 by distilling past attempts into behavioral rules, proving lightweight, in-context learning can match frontier-model performance for policy-bound tasks. This lets teams upgrade existing agents without costly fine-tuning or model swaps by dynamically generating and applying domain-specific guidelines—cutting deployment time and compute costs while maintaining strict compliance.