Running a large language model on your own infrastructure is now an ordinary engineering problem rather than a research exercise. The serving stack is mature, open-weight models are good enough for most enterprise work, and the hardware requirement is smaller than most people assume — a single high-memory GPU server handles a substantial internal deployment. The difficulty is not the model. It is sizing honestly, keeping the entire data path local, and operating the thing after launch.
This article covers what you actually need to know before committing capital: how to choose a model size, how to size hardware against real concurrency, what it costs against API pricing, and the specific mistakes that recur.
Decide the workload before the hardware
The most expensive error in on-premise AI is buying infrastructure for an imagined workload. Three questions determine everything downstream.
What is the task? Retrieval-grounded question answering, classification, extraction and summarisation are the bulk of enterprise work, and they are well within the reach of mid-sized open-weight models. Complex multi-step reasoning and code generation at a high standard are where model size still matters materially.
The distinction is practical: if the model's job is to read retrieved context and answer from it, capability requirements are modest, because the hard part — knowing the answer — is being done by retrieval. If the model must reason from its own knowledge across several steps, you need more.
What is the real concurrency? Not registered users. Simultaneous in-flight requests at peak. An internal knowledge assistant for two thousand employees typically runs at single-digit to low-double-digit concurrency, because usage is bursty and thinly spread. Organisations routinely size for hundreds and run at 5% utilisation for a year.
What latency is actually required? Interactive chat needs first tokens within a second or two and a generation rate above reading speed. Batch document processing has no latency requirement at all and can be scheduled overnight on the same hardware, which is the cheapest capacity you will ever find.
Write these three numbers down before talking to a hardware vendor. They determine the answer.
Choosing a model size
A rough map of what different scales are good for in enterprise deployments.
Small models (roughly 7–9B parameters). Classification, routing, extraction, simple summarisation, and retrieval-grounded answering on straightforward content. Fast and cheap. Frequently the right choice for the highest-volume path in a deployment, and consistently underrated.
Mid-sized models (roughly 24–40B). The practical sweet spot for most enterprise assistants. Good instruction-following, competent retrieval-grounded reasoning, acceptable multilingual performance. Runs on a single high-memory GPU when quantised.
Large models (70B and above). Noticeably better on complex reasoning and on lower-resource languages. Requires multiple GPUs or aggressive quantisation. Justified when the task genuinely needs it — and in a well-designed system, only a minority of requests do.
Mixture-of-experts models. Large total parameter counts with only a fraction active per token. Excellent capability-per-compute, at the cost of needing memory for all the weights. Worth evaluating specifically; the economics can be very favourable.
The architectural conclusion matters more than the model choice: route by task. Send the high-volume simple traffic to a small model and reserve the large one for requests that need it. This is frequently the difference between a deployment that pays for itself and one that gets switched off after the first infrastructure review. See what is AI orchestration.
Sizing the hardware
Memory is the binding constraint
GPU memory determines what you can run. The approximate requirement:
Model weights. At 16-bit precision, roughly 2 GB per billion parameters. At 8-bit, about 1 GB. At 4-bit, roughly 0.5 GB. In concrete terms, a 70B model needs about 140 GB at full precision and roughly 35 GB at 4-bit — the difference between a multi-GPU node and a single card. A 70B model in FP16 fills essentially all of an H200's 141 GB; in FP8 or INT8 it fits on two H100s or one H200.
KV cache. The per-request memory holding attention state, proportional to context length × concurrent requests. This is what people forget, and it is often larger than the weights in a high-concurrency deployment with long contexts. Long-context retrieval applications are particularly affected.
Overhead. Activations, fragmentation, the serving runtime. Budget 10–20% headroom.
A practical rule: total weights plus expected peak KV cache plus 20%. If that exceeds your GPU memory, quantise further, shorten context, cap concurrency, or add GPUs.
Quantisation
Reducing weight precision. 8-bit quantisation is generally free in quality terms for enterprise tasks — FP8 serving with TensorRT-LLM on H100 or H200 hardware is reported to deliver around 50% memory savings at roughly 99.9% quality retention. 4-bit is usually acceptable with modern methods and shows measurable but often tolerable degradation. Below 4-bit, quality falls off sharply.
Quantisation is the single highest-leverage decision in on-premise deployment, because it moves models between hardware tiers. A model that requires two GPUs at 16-bit may run comfortably on one at 8-bit, which halves the capital cost.
Evaluate the quantised model on your own tasks, not on published benchmarks. Degradation is uneven — it affects some capabilities more than others, and it tends to hit lower-resource languages harder than English. For Azerbaijani-language workloads specifically, test this rather than assuming.
Batching is the other half of the economics
Throughput per GPU is dominated by batch size, and the cost figures move accordingly. On an H100, a 70B-class model costs roughly $0.73 per million output tokens at batch 1 and about $0.18 at batch 8 — a fourfold reduction from a serving-configuration change, no hardware involved.
The practical consequence: a deployment with poor batching is not a hardware problem, it is a configuration problem, and buying more GPUs to fix it is the expensive way round.
A realistic starting configuration
For a first production deployment serving an internal assistant to a large organisation:
- One server, one or two high-memory data-centre GPUs (80 GB class or above). This runs a quantised mid-sized model at good speed with room for meaningful concurrency.
- Ample system RAM — at least equal to total GPU memory, for model loading and caching.
- Fast local NVMe for weights. Loading a large model from network storage is slow enough to be annoying at every restart.
- A separate, much smaller machine for the vector database, orchestration and application layer. These are CPU-bound and should not compete for GPU-host resources.
This configuration handles a genuinely useful deployment. Organisations consistently overbuy at this stage; the correct move is to start here, measure, and scale on evidence.
Scaling
When you outgrow it, in order of cost-effectiveness:
- Quantise further, if quality permits. Free.
- Tune the serving engine — batching parameters, KV cache configuration, context limits. Often yields large gains for no capital.
- Route more traffic to a smaller model. Usually the biggest win available.
- Add GPUs to the existing host.
- Add hosts behind a load balancer.
Work down that list rather than starting at the bottom.
The serving stack
Inference engine. vLLM is the common production choice, with continuous batching and efficient KV cache management — the difference between it and a naive serving loop is severalfold in throughput. SGLang and TensorRT-LLM are strong alternatives. Do not serve a production workload from a research script.
Model gateway. An OpenAI-compatible API in front of the engines, so applications are decoupled from the specific model. This is what makes the model swappable, and it costs nothing to put in place at the start.
Vector store and embeddings — both local. This is where sovereignty claims most often fail. Self-hosting the language model while calling a hosted embedding API means every document you index is transmitted externally. The embedding model must run locally too.
Orchestration. Routing, tools, state, identity, audit. Covered in what is AI orchestration.
Observability. Request-level logging, latency, token throughput, GPU utilisation, cost per request. Without this you cannot size, tune or justify anything.
What it actually costs
Against API pricing, the comparison hinges on volume, and the published break-even points are consistent enough to plan against.
The crossover. Industry analysis in 2026 puts the break-even for self-hosting at roughly 2 million tokens per day for most configurations, or 2–5 million tokens per day on reserved capacity over a twelve-month window. A simpler version of the same rule: start modelling self-hosting seriously once monthly API spend passes about $12,000–$19,000.
Marginal cost after the crossover. At batch, a self-hosted 70B-class model runs at roughly $0.18 per million output tokens, against dollars per million for frontier hosted models. That gap is why sustained, high-volume workloads move in-house and stay there.
Capital. A single-server reference configuration is a substantial but not extraordinary enterprise purchase, plus power, cooling and rack space. Get a current quote for your own market rather than working from a published figure — GPU pricing moves quickly and regional availability moves it further.
Operational. Power draw is meaningful and continuous. Add the engineering time to operate it, which is the cost most often omitted from comparisons.
But for regulated institutions this comparison is frequently irrelevant, because the API option is not legally available for the data in question. Then the question is not whether on-premise is cheaper than an API — it is whether the use case is worth the infrastructure, against a baseline of not doing it at all. See sovereign AI for that argument in full.
Where deployments go wrong
Sizing for imagined peak. The most expensive error, made before any measurement exists. Start small, instrument, grow on data.
Ignoring the KV cache. Sizing on weights alone, then discovering that concurrency is a third of the plan because cache memory was not budgeted.
Serving at batch 1. Leaving a fourfold throughput improvement unclaimed and then buying hardware to compensate.
Hosted embeddings behind a local model. The sovereignty leak that survives right up until a legal review finds it.
No evaluation harness. You will change models — new releases arrive constantly. Without an eval set built from your own data in your own languages, every change is unmeasurable. Build it before the first production use case.
Hard-wiring the model. Put a gateway in front from day one so the model is configuration.
Underestimating operations. Uptime, upgrades, driver compatibility, capacity, incidents. This is a standing capability. Deployments that treat it as a project decay quietly.
Skipping Azerbaijani-language evaluation. Multilingual claims in model cards do not tell you how a specific quantised model performs on your domain terminology in Azerbaijani. Test it. The results are sometimes surprising in both directions.
A sensible first deployment
Weeks 1–2. Define workload, concurrency and latency. Build an evaluation set of 50–100 real cases from your domain, in the languages you will actually serve.
Weeks 3–4. Evaluate two or three candidate models at two quantisation levels against that set. Choose on measured results, not benchmarks.
Weeks 5–6. Procure and install. Deploy the serving stack, gateway, vector store, local embeddings.
Weeks 7–9. Build the first use case — internal knowledge retrieval is the right choice. Identity-aware retrieval and audit logging from the start.
Weeks 10–12. Pilot with a real user group. Measure latency, throughput, quality against the eval set, and utilisation. Tune.
Then scale on evidence. At this point you know your real concurrency, and the second use case costs a fraction of the first — which is the entire argument for building the platform.
Key points
- Decide workload, concurrency and latency before hardware. Most organisations overbuy by a wide margin.
- Memory is the binding constraint: a 70B model is ~140 GB at FP16 and ~35 GB at 4-bit. The KV cache is the part people forget.
- FP8 gives about half the memory at near-identical quality on current hardware — the highest-leverage single decision.
- Batching moves cost per million output tokens from ~$0.73 to ~$0.18 on the same GPU. Tune before you buy.
- Break-even against APIs sits around 2 million tokens a day, or roughly $12K–$19K of monthly API spend.
- Embeddings and the vector store must be local, or the sovereignty claim fails.
- For regulated institutions the API cost comparison is often irrelevant, because the API is not a legal option.
HAVAA is deployed this way by default — models, retrieval and embeddings inside the customer's perimeter. Yukon Labs handles the sizing, deployment and operation.