Using DSPy to evaluate and improve Datasette Agent's SQL system prompts
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
Datasette Agent’s SQL prompt was causing models to guess nonexistent column names because the schema context listed only table names while discouraging extra describe_table calls. For production text-to-SQL agents, include column names in the initial schema context or relax tool-use constraints, otherwise you’ll pay for avoidable failed queries, retries, and lower answer reliability.
DSPy prompt optimization surfaced a concrete failure mode: telling an agent to skip describe_table when it "already has" schema info triggered column-name hallucination (guessing page_count, o.order_id) and error-retry loops, because the schema listing only exposed table names, not columns. The fix is trivial—include column names in the prompt or drop the premature-optimization advice—but the lesson is that "don't re-fetch what you have" instructions backfire when your context never actually contained the detail the model needs.