Most of the demos are the easy 80%. You wire an LLM to a handful of tools, give it a schema to fill in, and watch it walk through a legal-entity migration that used to take a human analyst most of an afternoon. It works, and it works fast — four times faster, in our case, on the cases that fit the pattern.
The interesting failures show up in the last 20%: the schema mapping that’s almost right, the field that maps cleanly nine times out of ten and silently does the wrong thing on the tenth. That’s not a prompting problem. It’s an evaluation problem.
What we actually monitor
We stopped asking “did the model get it right” and started asking “how would we know if it didn’t.” In practice that meant building confidence scoring directly into the pipeline, so low-certainty outputs get routed to a human before they touch anything that matters.
def route_output(result, threshold=0.92):
if result.confidence < threshold:
return queue_for_review(result)
return commit(result)The model doesn’t need to be perfect. The pipeline needs to know when the model isn’t.
That distinction changed how we shipped the whole system — less time tuning prompts, more time building the scaffolding that catches what the prompts miss.