The label that gets the sequence backwards
"AI-first" has become a branding exercise. Startups put it in their pitch decks. Enterprise teams put it in their strategy documents. Innovation labs put it on their slide templates.
The problem isn't ambition. The problem is sequence. "AI-first" starts with the technology and goes looking for problems to apply it to. That's backwards, and it leads to a specific, predictable pattern of failure.
What happens when you start with the technology
When the starting question is "how can we use AI here?" the team works backwards from the solution to find a problem worth solving. This sounds productive. In practice, it creates three failure modes that show up consistently.
Over-engineered systems. A team builds an LLM-powered workflow to classify support tickets. The model requires prompt engineering, embedding infrastructure, a vector database, and ongoing fine-tuning. The accuracy settles around 91%. A senior support agent looks at the same problem and says "we could do this with five keyword rules and a regex." The keyword approach would have been 95% accurate, cost nothing to run, and taken a day to build. But nobody asked whether AI was the right tool because the mandate was AI-first.
Expensive fragility. AI systems have operational overhead that traditional software doesn't. They hallucinate. They drift. They require monitoring for quality degradation. They cost real money per inference. When AI is the right tool, that overhead is justified by the value it creates. When AI was forced into a workflow where a simpler solution would have worked, you're paying the operational tax of an AI system for a problem that didn't require one.
Maintenance burden nobody scoped. The prototype works in the demo. The team ships it. Six months later, the model's performance has degraded because the data distribution shifted. The prompt that worked in March produces different results in September. The embedding index needs rebuilding. The vector database needs scaling. None of this was budgeted because the initial estimate was based on getting it working, not on keeping it working.
Start with the friction instead
The teams I've seen build the most effective AI systems don't start with AI. They start with a specific operational friction and then evaluate whether AI is the simplest tool to resolve it.
Where is the operational bottleneck? Not "where could AI add value" but "what is actually slow, expensive, or error-prone right now." The bottleneck is the starting point because it's concrete, measurable, and has a clear before/after.
What is causing the delay in decision-making? Sometimes the delay is that a human needs to process information that arrives faster than they can read it. AI is excellent for that. Sometimes the delay is that two departments use different ticket systems and nobody can see the full picture. AI won't fix that. A shared dashboard will.
Is AI actually the simplest tool for this? This is the question that gets skipped most often. A database query that runs in milliseconds. A rule engine that handles the known cases. A spreadsheet formula that does the calculation. A cron job that sends the reminder. These aren't exciting. They're also not fragile, don't hallucinate, don't require monitoring for drift, and don't cost per inference.
The most impressive engineering is the code you didn't write
There's a cultural problem underneath the "AI-first" label. It implies that using AI is inherently more sophisticated than not using it. That's wrong. Using AI where it's not needed is over-engineering, which is the opposite of sophistication.
The most effective engineering decisions are often the ones where someone looked at a problem and said "we don't need AI for this." Not because they're against AI, but because they evaluated the problem honestly and chose the tool that matched the complexity of the problem, not the complexity of the tool.
AI is powerful when applied to problems that actually require it: ambiguous inputs, unstructured data, pattern recognition across large datasets, natural language understanding, tasks where the decision space is too large for rules. For everything else, the simpler tool is the better tool.
Have you seen "AI-first" mandates lead to over-engineered solutions on your team? What's the best example of a problem where AI was proposed but a simpler solution turned out to be better?
Top comments (14)
Strong piece, and the friction-first framing is right. I would push it one level deeper. Most AI-first failures I get called to clean up are not "should have been a rule engine." They are one monolithic LLM call doing a job that was 90 percent deterministic parsing and 10 percent real ambiguity. The fix is not AI or no AI, it is drawing the line inside the problem so the model only owns the slice that genuinely needs judgment, and everything it returns gets checked against something deterministic.
The 91 versus 95 example lands for a second reason too. A rule engine that is wrong 5 percent of the time knows its boundaries. A model that is confidently wrong 9 percent of the time, with no signal for which 9 percent, is worse, because you cannot even route around it. Calibration, the model knowing when it does not know, is usually worth more than a few points of raw accuracy.
The "draw the line inside the problem" framing is more precise than what I wrote. You're right that the real decision isn't AI vs no AI, it's identifying which 10% of the problem actually needs judgment and isolating the model to just that slice. The other 90% should be deterministic because it can be. That architecture is also way easier to debug because when something goes wrong you know immediately whether the failure was in the deterministic layer (your bug) or the model layer (ambiguity the model mishandled). The calibration point is the sharper version of the same argument. A rule engine with known failure boundaries lets you build routing around those boundaries. A model that's confidently wrong with no uncertainty signal gives you no information about when to trust it, which means you either trust it always or verify it always, and both of those defeat the purpose.
On the trust-always-or-verify-always fork, there is a third door, and it is the one that made calibration practical for me. The model rarely volunteers a usable uncertainty signal, self-reported confidence is just more sampled text from the distribution that produced the answer. So stop asking it, and build the signal outside the model from properties you can check without trusting it: does the output satisfy the schema, is every claim traceable to the source you handed it, does it survive a few resamples. None of those ask the model to grade itself.
That also softens "verify always defeats the purpose," since the checks are not one cost. Schema and grounding are cheap and deterministic, so run them on every call; resampling is the expensive one, so spend it only on the slice the cheap checks flagged. And set the threshold on the cost of a false accept, not raw accuracy, since a wrong value that reads clean and ships is a different kind of expensive from one you caught.
The "stop asking the model to grade itself" point is important and I should have been clearer about that in my response. Self-reported confidence is just more generated text, it has no privileged relationship to actual correctness. Building the uncertainty signal externally from schema validation, source traceability, and resampling is the right architecture because those checks are independent of the thing being checked.
The tiered cost model is practical and fixes the objection I raised about verification defeating the purpose. Schema and grounding checks on every call is essentially free compared to inference cost, so it's not "verify everything" in the expensive sense, it's "run the cheap deterministic filter on everything and spend the expensive resampling budget only where the cheap filter can't give you a clear answer." That's the same pattern as the 90/10 split from your first comment applied to the verification layer instead of the generation layer.
Setting the threshold on cost of false accept rather than raw accuracy is the framing that ties it all together. A wrong answer that looks right and ships undetected is a completely different cost from a wrong answer that got caught at the schema check. Same error rate, completely different business impact. Most teams set accuracy targets without weighting for how expensive each type of failure actually is.
Exactly, and one step further: once you price the false accept instead of accuracy, the threshold stops being hand-tuned and falls out of the numbers. Put a cost on a wrong answer that ships and a cost on a human review, and the optimal cut is where the marginal item's expected false-accept cost crosses the review cost. The routing coordinate becomes arithmetic.
The reason teams still default to accuracy is not that they missed this. Accuracy needs no such number; this needs the price of a wrong answer nobody caught, and that number lives with the business, not engineering, so it never gets set. The math was never the hard half.
"The math was never the hard half" is the perfect closing line for this entire thread. The engineering for tiered verification, external calibration, cost-weighted thresholds, all of that is buildable the moment someone provides the input number. The input number is "what does it cost us when a wrong answer ships undetected" and that's a business question that requires a product manager or a domain owner to sit down and actually quantify it. Most never do because it forces them to put a dollar amount on a failure scenario they'd rather not think about concretely. So engineering defaults to accuracy because accuracy is self-contained, no business input required, and the system stays unoptimized because the optimization function was never fully specified. This thread basically mapped the entire architecture from "AI vs no AI" down to "the reason production AI systems underperform is that nobody asked the business to price their own failure modes." That's a good place to land.
You put the why better than I did, and that is the right place to stop. One last piece to close it: refusing to price the failure is not ducking the decision, it is making it, and the price the business sets by omission is zero. Which is exactly why the wrong answer ships. Good exchange.
"The price the business sets by omission is zero" is the sharpest line in the entire thread. That's the mechanism nobody names. Not pricing the failure isn't neutral, it's setting the price to zero, which means every wrong answer that ships was implicitly approved by the absence of a number. That reframe should be uncomfortable for every product owner reading this. Great exchange, one of the best threads I've had on here.
That "implicitly approved by the absence of a number" is the part that should sting, and the fix is cheaper than it sounds. Nobody needs a precise cost of failure here, they need any number above zero. A rough number everyone knows is imperfect still beats the implicit zero, because the moment failure carries a nonzero price it has an owner and a review attached to it. Zero has neither. Moving it off zero is the whole move. Good thread, you pushed it further than I left it.
Very true, and AI is just the latest instantiation of this recurring behavior pattern. Every new technology wave comes with it. Part of the problem is that we, as engineers, are forced to stay up to date, and our current project is often the only chance to gain real-world exposure to a new technology. The same is true for managers. If you want to climb the corporate ladder, you better make sure your team uses the latest tech. This problem will remain as long as we have careers, but maybe AI will "solve" that one day 😆
The career incentive angle is the part nobody wants to say out loud but it drives more technical decisions than anyone admits. "We used AI" looks better on a resume than "we solved it with a cron job and three SQL queries" even if the second one was the better engineering decision. Same goes for managers, "led AI transformation initiative" gets you promoted, "prevented team from over-engineering with AI" doesn't even get mentioned. Until "chose the right tool for the problem" carries the same career weight as "used the exciting tool," the pattern will keep repeating with every new technology wave. And yeah, maybe AI will eventually solve the problem of humans adopting AI for the wrong reasons. That would be appropriately recursive.
the 'prompt that worked in March produces different results in September' line is the maintenance cost nobody puts in the estimate. we've seen this exact pattern — got budget for the build, none for the drift.
the friction first framing works for individual features, tbh. the part that gets harder at org scale is that 'what is slow or error prone right now' often has 6 different answers across 6 teams, and AI projects need a sponsor who can pick one and defend it. without that, friction first becomes friction wherever the loudest stakeholder points.
what's your heuristic for distinguishing friction that AI actually resolves vs friction that needs a process fix instead?
Really great read! I think the core issue is a disconnect between strategy planners and the teams who actually build things. Also, there’s a natural pattern for tech evolution: quantity always comes before quality—you can’t nail refined, well-fit AI solutions without running through lots of messy attempts first.
The strategy-builder disconnect is real and it's usually the root cause of the over-engineering pattern. The people deciding "we should use AI" aren't the ones who have to maintain the pipeline when it drifts. Your point about quantity before quality is interesting and probably true at the industry level. We're still in the phase where teams need to build a lot of bad AI implementations before the patterns for good ones become obvious. The problem is that each of those messy attempts costs real money and engineering time, so the faster teams can learn which problems actually need AI and which don't, the less they waste on the learning curve. Thanks for reading.