AI orchestration is the layer that sits between language models and the rest of your organisation. It decides which model handles a request, what tools and data that model may reach, how state is carried across steps, whose permissions apply, and what gets recorded. Without it you have a model. With it you have a system that can do work and be held accountable for it.
The reason this layer exists is the same reason application servers exist. Anyone can call a database directly. Nobody runs an enterprise that way, because connection management, transactions, authorisation and audit are real problems that do not go away by being ignored. Orchestration is that layer for models.
The market is learning this expensively. Gartner expects over 40% of agentic AI projects to be cancelled by the end of 2027 on cost, unclear value and inadequate risk controls, and a subsequent forecast expects 40% of enterprises to demote or decommission autonomous agents by 2027 once governance failures surface in production. Cancellation is a governance event before it is a technology one, and governance lives in this layer.
Why calling an LLM API directly stops working
Every enterprise AI project starts with a script that calls a model. It works, and it produces the demo that gets the project funded. Then it meets requirements, in roughly this order.
"It needs to look things up in our systems." Now the model needs tools, and something must decide when to call them, validate the arguments, handle failures, and prevent the model from calling a tool it should not.
"Different questions need different models." A classification task does not need your largest model. A complex analysis does. Something has to route, and routing rules that live in application code get duplicated across every team that builds something.
"It must only see what the user is allowed to see." This is the requirement that breaks naive implementations completely. If the retrieval layer runs under a service account, every user effectively has the union of all permissions. Enforcement has to happen at retrieval time, per request, under the requesting user's identity.
"We need to know why it did that." A regulator, an auditor or an incident review asks about a specific output from three months ago. You need the prompt, the retrieved context, the model version, the tools called and the response — reconstructable.
"Legal says nothing can leave the perimeter." Now the model, the embeddings and the vector store are all local, and something has to manage that infrastructure.
"Three departments have built their own." Three data connections, three security reviews, three audit gaps, no reuse.
Each requirement is individually reasonable. Together they are a platform, and the choice is whether to build it deliberately or accumulate it accidentally across six codebases.
What the layer actually contains
Six responsibilities. A production orchestration layer implements all of them.
Routing
Selecting the right model for each request — by task type, sensitivity, cost, latency or language.
This matters more than it sounds. Inference cost varies by an order of magnitude across model sizes, and most enterprise traffic is simple. Current benchmarks illustrate the spread: a self-hosted 70B-class model on an H100 costs around $0.73 per million output tokens at batch 1 and about $0.18 at batch 8, against dollars per million for frontier hosted models. Routing straightforward requests to a small model and reserving the large one for genuinely hard work is frequently the difference between an economic deployment and one that gets switched off.
Routing also handles fallback. Models become unavailable, get deprecated, and behave differently after upgrades. When routing is centralised, swapping a model is a configuration change. When it is embedded in application code across six teams, it is a project.
Tool access
Giving models the ability to act — query a database, call an internal API, file a ticket, send a notification.
The orchestration layer owns the tool registry, the schemas, argument validation, execution, error handling and the results returned to the model. Critically, it owns which tools are available in which context. A model answering an HR policy question has no business holding a tool that can issue a payment, and the mechanism preventing that must be structural rather than a line in a prompt.
The Model Context Protocol (MCP) has become the common standard for exposing tools and data sources to models. Launched in November 2024 and donated by Anthropic to the Linux Foundation's Agentic AI Foundation in December 2025 — with AWS, Google, Microsoft, OpenAI, Bloomberg and Cloudflare backing it — it moved from vendor project to open infrastructure in about a year. Adoption followed: SDK downloads went from roughly 100,000 in the first month to 97 million per month by March 2026, and surveys in 2026 put around 28% of Fortune 500 companies running MCP servers, with roughly 41% of surveyed software organisations in limited or broad production.
Its practical value is that a tool implemented once is usable by any compatible model — which removes a whole category of rewriting when models change.
State and memory
Multi-step work requires state: what has been done, what was learned, what remains. Conversations require history. Long-running processes require durable checkpoints so a failure at step seven does not restart at step one.
This is ordinary distributed-systems work, and it is where naive implementations are weakest. A prototype keeps state in memory and loses it on restart. Production needs persistence, expiry and a clear policy about what is retained — because conversation history containing customer data is itself regulated data.
Permissions and identity
The most important control, and the one most often deferred.
Every action an AI system takes must occur under the identity of the person on whose behalf it acts. Not a service account. Not an application identity with broad scope.
When retrieval runs under a shared account, the vector store becomes a permission-flattening machine: documents from across the organisation are embedded into one index, and any user who can phrase a question can reach any of it. This has been the most serious real-world failure in enterprise AI deployments, and it is entirely preventable by enforcing permissions at retrieval time against the requesting user.
The orchestration layer integrates with the existing directory, propagates identity through every step, filters retrieval by entitlement and gates tools by role.
Audit and observability
Every request recorded: who asked, what was retrieved, which model and version responded, which tools were called with which arguments, what came back, how long it took, what it cost.
In a regulated institution this is not an operational nicety. It is what makes the system defensible when someone asks about a specific decision, and it is what an ISO/IEC 42001 audit will ask to see.
It is also how you improve the system. Without request-level observability you cannot tell whether a change helped, and you will be making changes constantly.
Guardrails and evaluation
Input and output checks — sensitive data detection, prompt-injection defence, policy filtering — plus the evaluation harness that tells you whether behaviour has regressed.
Evaluation is the part organisations skip and then wish they had not. Without a test set built from your own data and your own criteria, a model upgrade is a gamble taken in production. Build the harness before the first production use case.
Orchestration vs workflow automation vs agent frameworks
Three overlapping categories, routinely conflated.
Workflow automation (traditional BPM, RPA) executes predetermined steps. Deterministic, auditable, brittle when reality deviates from the diagram. Excellent for processes that genuinely are fixed.
Agent frameworks are developer libraries for building agent loops. Useful for building; they are not infrastructure. They do not provide identity integration, audit, multi-tenancy or operational controls, and they are not intended to.
AI orchestration is the runtime and control plane. Deterministic where determinism is required, model-driven where judgement is required, with the governance controls that let it operate on real data under real accountability.
The useful distinction: a framework helps one team build one thing. Orchestration is what lets an organisation run twenty things built by six teams under one set of controls.
Where orchestration decisions go wrong
Building it implicitly. The layer exists in every deployment. The only question is whether it was designed or accumulated across application codebases. Accumulated versions have inconsistent permissions, partial audit and no shared evaluation — and consolidating them later is a rewrite.
Deferring permissions. Almost every serious incident traces back to identity being handled loosely early on. Retrofitting per-user entitlement into a system that indexed everything under a service account means re-architecting retrieval.
Hard-wiring one model. The landscape changes fast. Models must be swappable behind a stable interface, and this is nearly free at design time and expensive later.
No evaluation harness. Without it, every change is unmeasurable and every regression is discovered by a user.
Over-engineering the first deployment. The opposite failure. A twelve-month platform programme before any user has been helped. Build the thin version — routing, permissioned retrieval, audit — get one use case live, and let real requirements drive the rest.
Making it sovereign
For regulated institutions, every component must be inside the perimeter, and the orchestration layer is where this is enforced.
The most common failure: self-hosting the language model while calling a hosted embedding API or a managed vector database. The prompt stays inside; the document content leaves during indexing. The sovereignty claim does not survive a serious review.
The orchestration layer is the correct place to enforce this structurally — a policy that no component may reach an external endpoint, enforced by network egress control rather than by configuration. See sovereign AI for the full treatment.
What to build first
The thin version, in order:
- A single entry point for AI requests. One place where policy is applied.
- Identity propagation from the start. Retrieval filtered by the requesting user's entitlements, always.
- Request-level audit logging. Cheap now, impossible to reconstruct later.
- A model abstraction so the model is configuration, not code.
- One retrieval pipeline, permission-aware, shared across use cases.
- An evaluation harness with fifty to a hundred real cases from your domain.
That is enough to run a first production use case safely and to add the second one cheaply — which is the entire economic argument for the layer.
Key points
- Orchestration is the control plane between models and the organisation: routing, tools, state, identity, audit, evaluation.
- Every deployment has this layer. The choice is whether it was designed or accumulated.
- MCP is now the de facto tool interface — open-governed since December 2025, at 97 million SDK downloads a month by March 2026 — which makes tools portable across models.
- Per-user permission enforcement at retrieval time is the single most important control. A shared service account flattens entitlements across the whole organisation.
- Route simple traffic to small models. At batch, self-hosted inference is cents per million tokens; this is often what makes a deployment economic.
- Build the evaluation harness before the first production use case, not after the first incident.
HAVAA is Yukon Labs' orchestration platform, built to run entirely inside the customer's perimeter — models, retrieval and embeddings included.