I Ran Your Demo

The vendor demo is a guided tour with the broken doors locked. I know, because I used to draw the route. Here's how I evaluate now that I'm on the other side of the table.

By Jason Waldrip, with Claude Opus 4.8

There’s a calendar invite in your inbox for a 45-minute demo of a platform you might buy. The vendor will share a screen. Someone smooth and likeable will walk you through a flow that works perfectly, every click landing exactly where it should, the sample data clean, the latency invisible, the one feature you actually care about saved for the end so you leave on a high note. You’ll nod. It’ll look great. And you will have learned almost nothing about whether the thing works, because that wasn’t the point of the meeting. It was never going to be.

I can say that with some confidence, because for a year that smooth likeable person was me.

the edge

My first real job in this industry was sales engineer at a CDN. I was the technical half of the sales motion, the one who got pulled in when the prospect had questions the account rep couldn’t answer. I helped land Blizzard, YouTube, Disney, Rockstar. Big logos, hard rooms, smart buyers. And my job, stripped of the title, was to make our product look like the obvious answer in the time we had on the call.

I was good at it. Being good at it meant knowing exactly where the product was strong and steering every demo straight down that road. The features that lagged, the configuration that was a nightmare, the edge case where throughput fell off a cliff: those weren’t on the tour. Not because I was lying. I never lied. I just built a route through the building where every door I opened led somewhere we’d cleaned up first. A good demo is a magic trick, and the trick is misdirection, not deception. You’re looking exactly where I want you looking.

That’s the part buyers don’t internalize. The demo is a piece of software in its own right, and the thing it’s engineered to do is produce a yes. It is the single least representative way to experience a product, optimized end to end by people whose income depends on you not noticing the gaps until after the contract is signed.

what the demo is built to hide

Every product has a shape. There’s the part it does beautifully, the part it does adequately, and the part where it quietly falls apart under real conditions. The honest version of evaluating a tool is finding that third part as fast as possible, because that’s where you’ll actually live once it’s in production carrying your load instead of demo load.

The demo is built to keep you out of that third part. The data is seeded so the slow query never fires. The flow avoids the integration that breaks half the time. The scale is a toy, so the thing that dies at real volume never gets the chance. I used to spend more prep time deciding what not to show than what to show. The skill wasn’t presenting the strengths. Anyone can do that. The skill was routing cleanly around the weaknesses so the prospect never tripped over one and started asking the question that ends the deal.

Now I sit on the other side of that table, and I know the route exists, so I refuse to walk it.

evaluate on the gaps, not the highlight reel

Here’s the move. The demo shows you the highlight reel. Your job is to evaluate on the gaps, because the gaps are what you’re actually buying. A tool’s strengths are table stakes, they’re why it’s even on your list. The decision lives entirely in what it’s bad at and whether you can live there.

So I don’t take the guided tour. I ask for the docs and a trial account, and I go straight at the thing the demo would have avoided. I throw production-shaped data at it, not the clean sample set. I wire up the ugly integration first, the one that touches our actual auth and our actual edge cases, instead of the toy one in the tutorial. I find the limits page and read it like a contract, because the rate caps and the unsupported configurations are where the truth lives. And I go looking for someone who ran this thing at scale and hit the wall, because a buyer who churned will tell you more in ten minutes than the vendor will in ten meetings.

The tell I trust most is simple. Ask a vendor to let you click the thing they didn’t show you. Watch what happens. The good ones say “sure, here’s a sandbox, break it however you want.” The ones to walk away from get a little tighter, suggest a follow-up call, offer to “set that up properly for you.” That hesitation is the whole answer. They know where their third part is too, and they’re managing you away from it, exactly like I used to.

the buyer’s job is the judgment

This connects to the thing I keep landing on from every direction. Volume is cheap. Pitches are cheap. Polished surfaces are cheaper than they’ve ever been, and a good demo has never been easier to produce. The scarce thing, the only thing that protects you, is the judgment to look past the surface to the shape underneath. To ask what this is bad at, and whether bad-at-that is survivable for the actual workload you have, not the impressive one on the slide.

I’m not anti-vendor. I’ve sold to some of the sharpest engineering orgs on the planet, and the good ones were a pleasure precisely because they were impossible to misdirect. They’d thank me for the overview, decline the rest of the tour, and go find the wall themselves. They bought on the gaps. They were right more often than the ones who bought on the demo, and they got there faster, because they spent their time where the answer actually was.

So when the invite lands, I send the same reply every time. Skip the call. Send me the docs and a sandbox with no guardrails, and I’ll have your answer by Friday. If that makes someone nervous, I already have it.