AI Orchestration Architecture: Control Plane, Tools, Memory and State

An AI orchestration layer owns six things a model does not: which model handles a request, what tools it may call, what it remembers between turns, what policy applies before an action executes, what gets logged, and what happens when something fails. The model generates text. Everything that makes it an enterprise system lives in the layer around it.

Teams that treat orchestration as glue code discover the components below anyway — one incident at a time, in production.

What does the orchestration layer actually own?

The clearest way to see the boundary is to ask what breaks if the layer is absent.

Without routing, every request goes to one model, so trivial classification costs the same as complex reasoning and a model outage takes down everything. Without a tool boundary, the model's access is whatever credentials the application holds. Without state management, each turn starts blind. Without policy gates, an agent that decides to issue a refund issues one. Without observability, an incorrect output is unexplainable after the fact.

Each of those is a component, and each is described below. The higher-level framing — why an API call to a model is not an enterprise AI system — is in what is AI orchestration.

The control plane

The component that receives a request and decides what happens to it. Four responsibilities.

Identity resolution. Who is asking, in which role, on behalf of which entity. This determines everything downstream — which data is retrievable, which tools are available, which actions require approval. An orchestration layer that does not know who the user is cannot enforce anything, and retrofitting identity is close to a rewrite.

Model routing. Which model handles this request. Routing by task class is the pattern that survives: cheap fast models for classification, extraction and routing itself; larger models for reasoning, synthesis and generation; specialised models where the domain warrants it.

The practical benefit beyond cost is substitutability. When routing is a control-plane decision rather than a hardcoded client, swapping a model is a configuration change. Given how quickly the model landscape moves, an architecture that cannot swap models is an architecture with an expiry date.

Request lifecycle. Timeouts, retries, cancellation, and what happens when a downstream component is unavailable. Mundane and the difference between a demo and a system.

Budget and rate control. Per user, per tenant, per workflow. On-premise this is GPU capacity rather than spend, but the control is the same: a runaway agent loop must not consume the cluster.

Tools and the tool boundary

Tools are what let a model do something rather than only say something — query a database, read a document, create a ticket, send a message. The tool boundary is the most security-relevant part of the architecture.

Tools are the trust boundary, not the model. The model proposes; the tool layer decides whether to execute. Every parameter arriving from a model is untrusted input, validated exactly as input from a web form would be. A tool that interpolates a model-supplied string into a query is the same vulnerability class as SQL injection, with a less predictable attacker.

Tools execute under the user's identity, not the system's. If the tool layer holds broad credentials, then any successful prompt injection inherits them. Scoping each tool call to the requesting user's entitlements is what keeps an assistant from becoming a privilege escalation path — the mechanics are in building secure AI workflows.

Standardised tool interfaces are worth adopting. The Model Context Protocol has become the common way to expose tools and data sources to models, with adoption across the major platforms and a growing connector ecosystem. The practical value is that an integration written once is reusable across models and applications, rather than being rewritten per framework.

Read and write tools deserve different treatment. Reads are governed by retrieval scope. Writes change the world and belong behind policy gates, discussed below.

Memory and state

The component teams most often underestimate, because a demo does not need it.

Three distinct kinds of state, frequently conflated:

Conversation state — the current session's turns. Bounded, ephemeral, and mostly a context window management problem: what to keep verbatim, what to summarise, what to drop.

Working state — a multi-step task's progress. Which steps completed, what they returned, what remains. This is what makes a long-running task resumable after a failure, and it must be durable rather than held in memory, or a restart loses the work.

Long-term memory — what persists across sessions: user preferences, prior decisions, accumulated context. The most valuable and the most dangerous, for two reasons.

First, memory is a data protection surface. Anything remembered about a person is personal data, subject to the same retention, access and erasure requirements as any other record. A memory store that cannot answer "what do you hold about this individual" and delete it is a compliance gap, not a feature.

Second, memory poisoning is a real attack. If content the system ingests can write into long-term memory, an attacker can plant instructions that persist across sessions. Memory writes need the same scrutiny as tool calls.

The design rule that avoids most of this: memory is scoped and attributed. Every stored item records who it belongs to, when it was written, from which interaction, and when it expires.

Policy enforcement and approval gates

Where an orchestration layer stops being infrastructure and starts being governance.

Classify actions by reversibility, not by importance. A read is free. A reversible write — a draft, an internal note — can proceed. An irreversible or externally-visible action — sending a message to a customer, moving money, changing a configuration — requires an approval gate. This classification is the single most useful design artefact in an agent system, and it should exist before any agent is given a write tool.

Approval is a workflow, not a dialog box. The approver needs the proposed action, its inputs, the reasoning, and enough context to judge — and the approval must be recorded. In an institution this record is the evidence an examiner asks for.

Policy evaluates at execution time, on the actual parameters. Not at design time on the tool definition. An agent permitted to issue refunds under a threshold needs the threshold checked against the actual amount.

Fail closed. When the policy service is unavailable, actions do not execute. The opposite default is how incidents happen.

Observability and the audit trail

Two different requirements that share a pipeline.

Operational observability — latency, error rates, token or GPU consumption, tool failure rates, retrieval hit rates. What you need to run the system.

The audit trail — for a given output: which model and version, which prompt, which documents were retrieved, which tools ran with which parameters, which policies evaluated, who approved what, and who acted on the result. What you need to explain the system.

The audit trail is an architectural requirement, not a logging feature, and it has to be designed in. It is what an ISO/IEC 42001 audit asks to see, and it is the difference between "the AI made a mistake" and a specific, correctable defect.

One design note learned in production: log the retrieved context, not just the prompt. When an answer is wrong, the question is almost always what the model was given, and a log with the prompt template but not the retrieved documents cannot answer it.

How does this fail?

Five failure modes, in the order they appear.

Loops. An agent retries a failing tool indefinitely, or two agents hand work back and forth. Requires hard limits on steps, wall-clock time and cost per task — not soft guidance in a prompt.

Silent degradation. Retrieval starts returning worse results after a source change, and the system keeps answering confidently. Only monitoring retrieval quality catches this; user complaints arrive much later.

Context overflow. A long conversation or a large retrieval set exceeds the window, and the earliest instructions — often the safety-relevant ones — are the first to be truncated. Instruction placement and explicit context budgets are the mitigation.

Tool errors interpreted as data. A tool returns an error string, the model treats it as a result and reasons from it. Tool responses need typed success and failure paths.

Cascading failure across agents. In multi-agent designs one agent's wrong output becomes another's input, and errors compound rather than cancel. This is the main argument for keeping agent graphs shallow and verifying at each handoff.

The industry-level version of these problems is visible in analyst expectations: Gartner has predicted that over 40% of agentic AI projects will be cancelled by the end of 2027, citing cost, unclear value and inadequate risk controls. The controls above are what separate the projects that survive.

What does this look like on-premise?

For supervised institutions here, every component runs inside the perimeter: models, orchestration layer, tool implementations, memory store, vector index and logs.

Two practical consequences.

Model routing matters more, not less. With finite GPU capacity rather than elastic cloud, routing cheap tasks to small models is a capacity decision. Sizing and cost reasoning is in deploying LLMs on-premise.

The logs stay inside too. A system that keeps inference local but ships observability data to a foreign SaaS platform has moved the problem rather than solved it, since prompts and retrieved context routinely contain personal data — a point developed in data residency and personal data law in Azerbaijan.

HAVAA is the orchestration layer Yukon Labs deploys for this, running entirely within the customer's perimeter with the control plane, tool boundary, policy gates and audit trail as first-class components rather than application code.

Key points

• The orchestration layer owns routing, tools, state, policy, observability and failure handling. The model only generates.
• The control plane must resolve identity first; retrieval scope, tool availability and approval requirements all derive from it.
• Route by task class. Substitutability matters as much as cost, given how fast models change.
• Tools are the trust boundary. Validate every model-supplied parameter and execute under the user's identity, never the system's.
• Separate conversation state, working state and long-term memory. Long-term memory is personal data and an attack surface.
• Classify actions by reversibility and put approval gates on irreversible ones. Evaluate policy at execution time and fail closed.
• Log retrieved context, not just prompts. The audit trail is an architectural requirement, not a logging feature.
• Guard against loops, silent retrieval degradation, context overflow, tool errors read as data, and cascading multi-agent failure.

Yukon Labs deploys HAVAA on-premise in Azerbaijan, with retrieval grounded on governed data through OvalEdge and Starburst. For the concepts underneath this architecture, start with what is AI orchestration.