The AI Feature Is the Easy Part

Two weeks is usually enough. An engineer wires a model into the product, puts a text box in front of it, and the demo lands well in the review. Somebody asks when it can ship.

The answer is that the thing in the demo is not the feature. It is the cheapest part of it, and the only part that gets easier every year.

The rest does not. Somebody has to decide what a correct answer looks like, who may see the data behind it, what happens when the model is slow or confidently wrong, what it costs at a thousand requests an hour, and who gets woken when it starts failing quietly. None of that is model work. All of it is product work, and it decides whether the feature survives its first quarter.

The demo proves it can work once

A demo is one successful attempt. Production is the claim that the same thing happens for every user, on every input, at a price you can afford — and that when it does not, you find out before the customer tells you.

That gap is not new. What is new is that the prototype is now so convincing that nobody treats it as a prototype. It has no error states, no permissions and no bad day, so nobody budgets for those.

Most of the production layer is ordinary engineering

We have taken an enterprise platform from “works in the demo environment” to “an organisation runs its operations on this”. The work that made the difference was unremarkable: a quality function that grew alongside the product instead of arriving after it, a release process with a named owner, and a rule that nothing shipped unless somebody could say how we would know if it broke.

That inventory is the AI production layer too. An AI feature needs failure containment, because a model call is a call to somebody else’s system that can be slow or unavailable. It needs analytics, because “do people use this, and do they keep what it produces” is the only real measure of whether it works. It needs a release process, because you will change the wording of the prompt far more often than the code. A team that cannot do those things for its ordinary features will not learn them from a model.

Speed deserves naming. The thresholds have not changed since Nielsen described them: a tenth of a second feels instant, one second keeps a train of thought intact, ten seconds is the limit of anyone’s patience. A model that thinks for eight seconds is not a slow feature. It is a different interaction, and it needs the answer to appear as it is written, or a design where the person does something else while it works. That is design work, and it has to happen before the interface is built.

Deciding what “good” means is a product decision

The most useful thing an AI project can own is a small, boring, versioned set of real inputs with agreed good answers — a test set for judgment rather than code. It is also the thing most often skipped, because writing it forces someone to say out loud what good means, and that is a decision nobody wants to sign.

Microsoft’s guidance splits this into three stages: choosing the model, testing against your own data before launch, and monitoring afterwards — including re-testing samples of real traffic and deliberately trying to break the thing on a schedule. The third stage is the one that matters. This is not a gate you pass once. Inputs drift, and the model underneath you gets replaced.

Our position is that this set belongs to product rather than engineering, with a named owner. It is the only thing that lets you answer “did that change make it better” with something other than an opinion.

You cannot debug what you never recorded

Standard monitoring for this now exists. The OpenTelemetry project has published conventions for recording model calls — which model was asked, how long it took, how much it cost. The project describes them as still moving, so treat the details as provisional.

There is a trade-off teams discover late. By default none of the actual prompt or response text is recorded, because it carries personal data. The thing you most need in order to diagnose a bad answer is the thing you are least entitled to keep. What to log, for how long, redacted how and visible to whom is a decision to make at design time. Retrofitting it is far harder.

Security here is a permissions problem wearing a costume

The OWASP security project ranks prompt injection first among risks to AI applications — instructions smuggled into content the model reads, which it then follows. It ranks first for a structural reason: instructions and data arrive through the same channel, with nothing reliably separating them. You cannot filter your way out of that.

So assume the model can be talked into attempting anything within its reach, and make its reach small. It should hold exactly the permissions of the person it is acting for, and never more. Whatever it produces should be treated as untrusted input by everything downstream. Its spending should have a ceiling. That is engineering work at the boundary, and it is where most of the security budget belongs.

The feature has an expiry date you do not control

This is the obligation teams most consistently fail to plan for. The model underneath your feature will be retired. Anthropic publishes at least sixty days’ notice, and its own records show several models moving from active to retired during 2026.

You get a choice. Pin your feature to a specific dated version of a model and you get predictable behaviour with a known end date. Point it at whatever the current version happens to be and your feature changes underneath you without anyone deploying anything.

Either way an AI feature carries a maintenance liability a conventional feature does not: a scheduled migration, a re-run of your test set against the replacement, and budget for what breaks. Whoever signs off the feature is signing up for that, and it belongs in a care agreement with a named engineer rather than in a backlog.

A narrow internal tool needs almost none of this

A smart reader will object that this is how a two-week feature becomes a two-quarter programme. They are right to object. Plenty of use cases deserve a light touch. An internal tool that drafts a summary for a colleague who reads it before anyone else sees it has a human check built into the workflow, a blast radius of one person and a worst case of a wasted minute. Building an evaluation pipeline for that is not diligence. It is theatre, and it spends the credibility you will need when something genuinely risky comes along.

The mistake is not doing too much or too little. It is scaling the work to how impressive the model is, rather than to what happens when it is wrong.

Three questions, priced separately

Readiness is not a ladder you climb. It is three independent questions, and a feature can sit high on one and low on the others. Price each on its own and you stop paying for apparatus the feature does not need.

Who sees the output. Internal and named, and two dozen real examples in a spreadsheet are a legitimate test set. Customer-facing, and you need that set versioned and owned, a check before each release, and a clear statement in the interface that this is AI — a legal requirement in Europe since August 2026, which our architecture piece takes apart properly.

What the system does with the output. Shows it to someone, and logging, a spending cap, a named owner and a one-click way to report a bad answer will cover you. Acts on it — writes, sends, orders, schedules — and the model must inherit the permissions of the person it acts for, with a human approving anything consequential.

How reversible a wrong answer is. Cheap to undo, and you can afford to learn in production. Expensive or hard to reverse, and you need continuous checking against real traffic, deliberate attempts to break it on a schedule, a way to switch it off without a deployment, and a plan for the day the model is retired underneath you.

Most features move on one axis only, which is the point: treating a display-only internal tool as though it triggered payments is as much a failure as the reverse. Arguing where a feature actually sits is a conversation worth having before the demo gets a ship date. And if nobody will own the answer after launch, the honest decision is not to ship.