AI Needed a Place in the Product Architecture

Most teams arrive at an AI feature with the model already chosen. Credentials are provisioned, someone has a prototype running, and the rest looks like plumbing. Then somebody asks where the call should actually live.

That is not plumbing. That is the architecture decision, and on most projects it gets made by accident — by whoever built the prototype, in whichever part of the system they happened to be working in that week. The model gets a formal evaluation. Its address does not.

The address matters more. Where you put the call decides what data it can reach, what it can do when it is wrong, how long the user waits, and whether anyone can reconstruct what happened afterwards. Choose badly and no amount of prompt work recovers it.

The model inherits whatever the caller already has

An AI call has no permissions of its own. It borrows the permissions of whatever process makes it, and the borrowing is silent.

Put the call in the browser and it reaches what the browser holds — the logged-in session, whatever is on the page, any key you shipped to make it work. That last one is the obvious failure, and teams mostly avoid it now. The quieter one matters more. Put the call in a backend service and it reaches whatever that service’s account can reach, which in most mature systems is far more than any single customer may see. The service was built to serve many users with a permission check in front of it. Insert a model that writes its own queries and that check now sits behind something unpredictable.

The published guidance is direct about this. The OWASP security project says permissions should be enforced by the systems downstream, not by whatever the model outputs, and that anything done for a customer should run with that customer’s access and nothing more. The Model Context Protocol goes further: never pass a borrowed credential further up the chain. Both are rules about placement.

The practical consequence is duller than the diagram. Before choosing where the call lives, write down what that part of the system can already read without asking anyone. That list, not the prompt, is the feature’s reach.

Latency changes the shape of the request, not just its speed

A normal web request has a budget measured in fractions of a second. A model call has one measured in seconds, and the variation is worse than the average.

The vendor documentation is specific about where the time goes. OpenAI notes that the length of the answer dominates — halving it can halve the wait, while halving the question buys almost nothing — and recommends showing the answer as it is written. Anthropic advises that anything running beyond ten minutes should run as a background job, because networks quietly drop idle connections and the request can simply vanish.

Read those as architecture constraints, not tuning tips. If a call cannot reliably finish inside a normal request, it does not belong inside one. It belongs in a background job with a saved state and a way to resume — which means the frontend now needs to show partial progress and let someone return to a result after navigating away. One backend decision has just rewritten three screens.

We have met the surface-specific version of this. Setting frontend architecture jointly with backend and infrastructure teams means accepting that two consumers of the same system rarely share a tolerance for waiting. An operator standing in front of equipment does not wait for a spinner; someone at a desk will. If a feature cannot degrade to something useful under the tightest budget, that context gets different treatment — or does not get the feature.

Failure containment is chosen in advance or not at all

Ordinary dependencies fail in ways you can list: timeout, error code, empty result. A model fails in more ways than that — slow, unavailable, rate limited, confidently wrong, subtly malformed, or fine but expensive. Only some of those look like errors to your code.

The design response is one we have used to make conventional platforms enterprise-ready. Adding failure boundaries taught us that their value was never the error handling. It was being forced to decide in advance which part of a screen may fail on its own, and what the user sees in its place. AI makes that decision compulsory rather than tidy, because the failure is often invisible to the software.

The containment test is blunt: name the path that does not involve AI. If a customer cannot complete the underlying task when the model is unavailable, the feature is not an addition to the product. It is a rewrite of it, and it should be scoped, staffed and released as one.

Plenty of AI features should stay thin

Not every AI feature deserves this. Turning a small internal convenience into an architecture programme is its own kind of failure, and a team that treats every model call as a platform decision will ship nothing. Some features genuinely are one call, a timeout and a fallback.

What we would resist is deciding that by feel. The scale below is our judgment rather than a published standard, and it grades reach and consequence, not how impressive the model is.

Thin is correct when the feature reads nothing private, writes nothing, triggers nothing, and failure is cosmetic. Drafting alt text for an editor who approves it before publication sits here. Call the service from wherever needs it, set a timeout, move on.

A backend home is required as soon as the feature reads anything belonging to a specific customer. What it may see has to be enforced by the query and that customer’s own access, never by an instruction in the prompt.

A workflow home is required when the output writes, acts, or is consumed by another system with no human in between. This is where human approval for consequential actions stops being optional, and where you need a record of what was done.

The pattern we would expect to see most often is the third kind of feature living in the first kind of place, because it started as a demo and nobody moved it when the scope grew.

When we meet that, we do not walk away. We do insist the placement moves before anything else is built — an argument better had in scoping than in an incident.

Whatever you choose, the trace has to survive it

Every model response comes back with an identifier, and it is worthless unless you join it to your own logs at the moment of the call. AWS makes the same point structurally in its guidance: visibility across every layer, tracing that follows a request through the whole workflow, and prompts managed as controlled assets rather than strings in a file.

Placement decides how much of this you keep. Route the capability to an outside service and you have traded the trace for speed of delivery. Sometimes that is right. It should be deliberate, made with the platform care arrangement in view, because the team answering the 2am question lives with whatever the boundary threw away.

There is a regulatory edge to the same decision. Since August 2026 the EU AI Act has required that people are told, clearly and at first contact, when they are dealing with an AI system — in a form that is itself accessible. That is an interface obligation produced by an infrastructure choice. If the model sits three services deep, the front end still has to say so.

How to decide where it goes

Five candidate homes, each answering a different question. Take them in this order.

The frontend — only when the feature needs no secret, no private data and no server-side identity, and its failure is cosmetic.

A backend service — when the feature needs customer-specific data and your own code must enforce who sees what.

The workflow layer — when the call outlives a single request, or its output causes something to happen.

The search layer — when the real problem is finding, ranking or matching. OpenAI’s own guidance includes the instruction not to reach for a language model by default, and a great many “AI features” are search problems with a model bolted on.

An outside service — when the capability is not what makes you different, and you accept losing the trace.

Then three questions, asked before the first commit rather than after the first incident. What can this call see that the current user cannot? What is the slowest acceptable version of this, and what does someone look at while they wait? And if we switched it off this afternoon, what would still work?

A team that can answer those three has made an architecture decision. One that cannot has made an integration and is hoping it holds. Which of the five a feature belongs in is where our applied AI work starts — and once placement is settled, the harder half begins.