Using DSPy to evaluate and improve Datasette Agent's SQL system prompts
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
DSPy prompt optimization surfaced a concrete failure mode: telling an agent to skip describe_table when it "already has" schema info triggered column-name hallucination (guessing page_count, o.order_id) and error-retry loops, because the schema listing only exposed table names, not columns. The fix is trivial—include column names in the prompt or drop the premature-optimization advice—but the lesson is that "don't re-fetch what you have" instructions backfire when your context never actually contained the detail the model needs.
Datasette Agent’s SQL prompt was causing models to guess nonexistent column names because the schema context listed only table names while discouraging extra describe_table calls. For production text-to-SQL agents, include column names in the initial schema context or relax tool-use constraints, otherwise you’ll pay for avoidable failed queries, retries, and lower answer reliability.