Jose Hernandez-Orallo proposes LLM-referenced C-test for supraintelligence
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
A new C-test generated and certified items with difficulty measured in bits using LLMs as the reference machine, then administered a first certified battery to twelve LLMs. The practical shift is that benchmark items can be produced beyond current human/frontier-model solvability instead of saturating, giving production teams a way to track capability headroom and model-to-model progress where today’s evals flatten out.
This proposes a benchmark that generates inductive-inference items whose difficulty is measured on an absolute bit scale (via Levin's Kt complexity) using an LLM as the reference machine, so item difficulty no longer saturates against human or frontier-model ability — you can certify problems harder than any current model can solve before they can solve them. Practically, it offers a non-circular, forward-compatible way to rank and track model capability as a continuous ratio-scale quantity, meaning your eval harness wouldn't need re-baselining each time a stronger model ships, though the twelve-model agent-characteristic curves here are a research artifact, not yet a drop-in production evaluation tool.