Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
A single formatting change in prompt wrappers can swing LLM accuracy by up to 30x—even when token count is held constant. This means your production agents may silently fail schema compliance or drop performance when switching templates, wrappers, or even minor formatting tweaks, forcing you to either lock in brittle prompt structures or add costly compliance checks and variance reporting to every eval.
The Format Sensitivity Index varies by over 30x across models, indicating that minor formatting changes in prompt wrappers can drastically alter model scores and leaderboard conclusions. This variance primarily stems from compliance failures, making reported accuracy statistically fragile without considering wrapper variance and schema compliance. For production deployments, this means that robust benchmarking and strict schema compliance are now crucial to avoid accuracy fluctuations.