$ cat choices/llm-apis.md
the call
I reach for an LLM API when the task is genuinely fuzzy: language, ambiguity, judgment-ish classification a plain function can't express. When a deterministic function would do the job, calling a model is just slower, costlier, and non-deterministic for no reason.
A model earns its keep on the problems code is bad at: natural language, ambiguity, and the kind of soft classification where the rules don’t fit in an if-statement. “Is this support ticket angry?” “Summarize this in plain English.” “Which of these twelve buckets does this messy input belong in?” You can’t write that as a function. Or you can, and it’s a brittle pile of regexes that breaks on the first input you didn’t anticipate. That’s where the API buys you something: it absorbs the fuzziness instead of forcing you to enumerate it.
The hype says put a model on everything. The discipline is the opposite: if a plain function can express the task, write the function. A deterministic call is faster, cheaper, testable, and gives the same answer every time. The moment you swap it for a model, you’ve signed up for latency, token cost, and an output you have to validate because it might be different on the next call. Reaching for an LLM where if/switch/a lookup table would do isn’t being modern. It’s adding a network dependency and a coin flip to something that used to be a unit test.
The honest version of “in production” here is a decision rule, not a war story. Before any model goes in, the question is the same: can a deterministic function express this? If yes (a validation, a routing rule, a status check) it stays a function, and stays fast, cheap, and testable. The model only earns the call when the input is genuinely fuzzy and the rules won’t close: free-form language, real ambiguity, judgment that doesn’t fit in a branch. Build on the model where it’s the right tool, and around it everywhere else. Let deterministic code handle the deterministic parts and only hand the model what actually needs a model.— see: choices / the-agent-fleet
Pick the tool by the shape of the problem, not the temperature of the trend. A model is the right answer to fuzziness and the wrong answer to everything you can already pin down in code. The skill isn’t knowing how to call the API. That’s the easy part. It’s the restraint to keep the deterministic 90% deterministic and spend the model only on the 10% that genuinely needs judgment.
the gaps — what it costs even when it’s right
Non-determinism is the price of admission. The same input can give you a different answer, so anything downstream needs validation, fallbacks, and a plan for when the output is wrong. You’re not calling a function anymore. You’re calling something that’s usually right.
Latency and cost scale with usage, not with code. A function is free to run a million times. Every model call is tokens and round-trip time, so volume that’s trivial for deterministic code becomes a real bill and a real wait.
The vendor owns the floor under you. Models get deprecated, prices change, behavior shifts between versions. Building on the model means living with someone else’s roadmap. Fine for the fuzzy parts, a bad bet for anything load-bearing that didn’t need a model in the first place.
used at
choices / the-agent-fleet