Scale Is a Design, Not a Dial

The autoscaler throws hardware at latency and latency gets worse. The ceiling wasn't set in the console. It was set in a design meeting, by whoever decided where the state lives. That's the decision a fleet can't make for you.

By Jason Waldrip, with Claude Opus 4.8

The dashboard says you’re running forty instances of the service. This morning it was twelve. The autoscaler did exactly its job: it saw latency climb and threw hardware at the problem, which is what you built it to do. And latency is worse. Not flat. Worse. You’re paying for three times the compute to serve a slower product, and every instinct you’ve got says add more, because more has always been the answer. It won’t work this time. Somewhere under all forty of those boxes is a single thing they’re all waiting in line for, and every instance you add makes the line longer.

That’s the part that feels like a betrayal. Adding capacity is supposed to help, or at worst do nothing. Here it actively hurts, and once you see why, you can’t unsee it. The autoscaler multiplied the part of the system that was never the bottleneck. It spun up forty copies of the stateless shell, forty more contenders for the one place they all have to converge: the hot row, the lock, the partition, the queue everything drains through. The console has a knob for CPU and a knob for replica count. It has no knob for the thing that’s actually saturated.

the dial has a ceiling nobody drew on purpose

Horizontal scaling does one trick well. It multiplies work that doesn’t have to coordinate. Spread a million independent requests across a hundred workers and you get something close to a hundred times the throughput, because none of them need to talk to each other. The trick fails the instant the work does have to coordinate, and almost every real system has a piece that does.

Amdahl wrote this down in 1967, and it hasn’t stopped being true: the fraction of a job that has to happen in sequence sets a hard ceiling on how much faster you can go, no matter how many processors you throw at the parallel part. If a tenth of the work is serial, you top out at ten times, forever, even with infinite hardware. Neil Gunther’s Universal Scalability Law made the picture honest in the way your dashboard just did: past a certain point, the cost of all those nodes coordinating with each other doesn’t merely flatten the curve, it bends it back down. Add capacity, get less throughput. Retrograde scaling. That downward bend is the curve your forty instances are living on, and it has a name and a formula that predicted your morning before you had it.

Here’s the part that matters for anyone who leads a team. That ceiling was not set by the autoscaler, and it will not be moved by the autoscaler. It was set a long time before this morning, in a room, by whoever decided where the state lives and who has to touch it at the same instant. The console is downstream of a whiteboard, and the whiteboard already spent your scaling budget.

scale is a decision, and it’s an early one

The industry files scale under operations. Bigger box. More replicas. A beefier managed database. A cache bolted in front of the thing that’s slow. Those are all real levers, and I’ve pulled every one of them. Every single one is downstream of a decision made before the first line of code: where does state live, and which requests contend on it under load.

I learned that framing at iTriage, running a consumer health app at real scale. Scale showed up as a systems-design problem long before it was ever a code problem. You could write every function correctly, ship a clean diff, pass every test, and still hit a wall, because a wall like that grows out of how the pieces are arranged, not out of any one piece. Where things lived and what had to talk to what set the limit. The most durable calls I made there happened at the whiteboard, in words, before anyone opened an editor.

At GigSmart the same truth wore a product costume. A real-time marketplace is a contention problem before it’s anything else. Two apps, workers on one side and requesters on the other, both converging on the same handful of moments: who’s available right now, who just got claimed, whose request is still open a half-second later. I reached for Elixir because fault isolation being the floor and not a bolt-on is exactly what a marketplace needs. But the runtime was the second decision. The first one was deciding where that hot, shared state was going to live and what would serialize on it, because that call sets the ceiling, and a framework can’t raise a ceiling the design already poured.

the diff looks perfectly scalable

Now hand this to a fleet, and watch the failure get quieter.

An agent will write you a service that looks built to scale. Stateless handlers, a tidy repo, horizontally-ready by every surface signal. It deploys clean. The tests are green, because a unit test exercises one request at a time and one request never contends with itself. The canary bakes fine at 1%, because 1% of traffic almost never has two requests fighting over the same row in the same millisecond. Every gate you trust says ship it. And the bottleneck is sitting right there in the design, in the spot where concurrent load lands on the one thing everything funnels through, invisible to all of it.

It’s invisible because the mistake isn’t in the lines. It’s in the shape, in where the state was placed and what got to touch it at once, and you cannot catch a shape problem by reading a diff. I’ve argued that reviewing an agent means reading the decisions, not the characters. This is the most expensive decision of exactly that invisible kind. A fleet has no model of your load shape, your hot keys, your fan-out, the one endpoint that gets hammered on the hour. It will put the state somewhere reasonable-looking and be confidently wrong in a way no reviewer catches by reading code, because there’s nothing wrong with the code. The wrongness is architectural, and architecture doesn’t render in a diff.

That’s the trap cheap generation sets under this specific problem. When output was scarce, a senior engineer sketched the state layout on a whiteboard first, because typing was expensive enough to make you think before you spent it. When a fleet will hand you the whole service in a minute, the whiteboard step is the one that quietly gets skipped, and it was carrying the entire scaling decision. You multiply judgment by a fleet or you multiply its absence, and the absence hides best right here, where the system runs green all the way to the day it meets real concurrency.

what you decide before the editor opens

The work reorders around the decision, in priority order, because the sequence is the whole point:

1. Name the hot state before the framework. Where does the contended state live, and which requests touch it at the same instant? Answer that out loud, in plain words, before anyone picks a language, a database, or a queue. The tool is downstream of that answer, every time.
2. Split stateless from stateful on purpose. The part you can multiply and the part you can’t are different animals with different rules. Draw the line between them deliberately, so the autoscaler is pointed at work it can actually help and kept away from work it can only crowd.
3. Find the serial fraction and go straight at it. Amdahl’s ceiling is set by the piece that has to happen in order. Shrinking that piece is worth more than any number of instances. Partition the hot key, move the contention off the critical path, make the shared thing append-only instead of read-modify-write. One point off the serial fraction beats forty boxes.
4. Load-test the collision, not the endpoint. A benchmark that fires independent requests at one handler proves nothing about the moment two of them want the same row. Simulate the contention on purpose, because the contention is the product under load, and it’s the only test that would have warned you this morning.

So go back to the forty instances. You can keep turning the dial, and the curve will keep bending the wrong way, because the number on the console was never the variable that mattered. The fix lives back in the room where somebody decided where the state would go, and the honest move is to reopen that decision rather than buy more hardware to outrun it.

Which leaves one question worth sitting with before you touch the autoscaler again. Where does your state actually live, who contends on it under real load, and did anyone ever decide that on purpose? If the honest answer is that it just happened, that nobody drew it, then that’s your ceiling, and there’s no dial in the building that reaches it.