AI-101 Day 44: Function calling and structured output, making a model emit something a…
🎓 AI-101 ｜ Day 44: Function calling and structured output, making a model emit something a machine can read
A model's output is prose. Prose is for people. If the next thing downstream is a program, prose is a disaster: a program needs a fixed shape, which fields, which types, nothing missing.
😬 The common approach, and the fragile one: add "please reply in JSON" to the prompt. It works nine times out of ten. In production, nine times out of ten *is broken*.
You get a preamble ("Sure! Here's the JSON you asked for:"), or ```json fences, or a trailing comma, or occasionally just prose. So you write string-wrangling code to guess which one you got today.
I hit exactly this. One of my own morning scripts asks a model for a JSON array and parses it directly. One morning it couldn't parse, and the system only knew how to say "search failed". Search hadn't failed. I had treated a polite request as a guarantee.
🔧 Proper approach 1, function calling. Instead of asking, you hand over a spec: the tool's name, what it does, and the shape of its parameters as JSON Schema. The model then emits a structured request, `search_flights` with `{from:"LHR", to:"HKG", date:"2026-09-01"}`, rather than a sentence.
Same point as Day 40 and Day 43: the model executes nothing. It only ever emits text saying what it would like called. Something outside it does the calling.
🔒 Proper approach 2, constrained decoding. This is the part most people haven't met, and it's the one that matters.
One morning it couldn't parse, and the system only knew how to say "search failed".
At each step the model picks the next token from its whole vocabulary. Constrained decoding sets the probability of every token that would make the output invalid to zero. Not "follow the schema and hope", it *cannot* deviate. The JSON that comes out will parse, by construction.
(In practice: JSON mode, structured outputs; in open source, GBNF grammars and Outlines. Same idea, the schema clamps the decoder.)
⚠️ The easy misreading: valid shape ≠ correct content. Constrained decoding guarantees you receive well-formed JSON. It guarantees nothing about whether the numbers are true. A perfectly valid `{"invoice_no": "INV-2024-0831", "amount": 12500}` still passes the schema with an invented invoice number. It's Day 41's benchmark lesson from the other side: a green tick isn't a right answer. After parsing, you still have to validate the *values*.
🧩 Three practical ones:
• 🎯 Tool descriptions are written for the model, not for your docs, it chooses mostly on the description, not the name. Vague description, wrong tool
• 📉 Keep the tool list short, every schema rides in the context on every call, costing tokens and degrading selection. Few and non-overlapping beats comprehensive
• ✅ Make required fields actually required, anything optional will be filled optionally
📌 Takeaway: "please reply in JSON" is a request. A schema plus constrained decoding is a guarantee. The difference isn't convenience, it's whether your system breaks one random morning.
