Private AI

On-Premise LLM Deployment: What You Actually Need

What an on-premise LLM looks like as a service: inference, identity, storage, concurrency and the operational work that vendors normally hide.

On-premise LLM deployment means the weights, the inference process and the data it sees sit on machines your organisation operates: a rack in your datacentre, a tower in a locked room, a blade you already own. It is not the same as “we installed a chat app on a laptop”, and it is not the same as a private GPU instance at a cloud provider. Those are neighbouring options. This article is about the on-site shape.

Teams ask for it when a public API is the wrong default and a workstation is not enough: several people, working hours, a corpus that should not travel, and an IT group that already runs other internal services. The commercial version of that work is self-hosted AI deployment. What follows is the architecture you are actually signing up for.

Deployment architecture

A usable on-premise system is a small set of services, not a single binary.

Typical layout:

Users (browser / internal app)
        ↓
Reverse proxy + TLS + identity
        ↓
Application (chat UI, job API, permissions)
        ↓
Inference server (OpenAI-compatible HTTP)
        ↓
GPU or CPU worker holding the weights

Beside that sit object storage or a fileshare for source documents and audio, a database for metadata and chat, a vector index if you retrieve, and a worker queue if you transcribe or ingest in batch.

Keep the inference server dumb. It should load a model, accept a completion request, and return tokens. Prompt construction, retrieval, permissions and audit belong in the application. If the model process also talks to your file server and your identity provider, you will not be able to reason about a leak.

Network: the inference port should not be on the guest Wi-Fi. Prefer an internal hostname, mutual TLS or a mesh, and no default route to the public internet from the GPU host. Outbound access for model downloads can exist on a jump path; it should not exist on the process that sees prompts.

Inference server

You need an HTTP API that existing tools can call. The de facto shape is OpenAI-compatible: /v1/chat/completions, a model name, a token stream. That lets you swap Ollama, vLLM, llama.cpp’s server, or a commercial inference box without rewriting the UI.

Choose the engine for the load, not for the README:

• Ollama is convenient on a workstation and acceptable for a small team if you accept its scheduling limits.
• llama.cpp (and servers built on it) is the right default when you care about quantized GGUF models and CPU or mixed inference.
• vLLM is the right default when a GPU must serve many concurrent sequences and you are willing to operate a Python service.

Do not run three of them “for flexibility”. Run one, put a model name in config, and keep a second engine as a documented fallback, not as a parallel production path.

Timeouts, max context, and max concurrent sequences are production settings. Leaving them at demo defaults is how one long RAG prompt blocks transcription for twenty minutes.

Hardware

On-premise hardware is a capital decision dressed up as an AI decision. Buy for the workload you measured in a pilot, with headroom for a larger quantized model and for one extra concurrent user than you think you have.

Minimum thinking, not a shopping list:

• Disk: weights are large. A 70B-class quantized model plus a couple of alternatives plus Whisper plus embeddings is hundreds of gigabytes before documents. Fast disk matters for loading, not only for storage.
• RAM: CPU inference and some engines want system RAM in the same order of magnitude as the model. Do not pair a big GPU with a starved motherboard.
• GPU / VRAM: the hard limit for interactive quality. See the hardware requirements article for how VRAM, quantization and context interact.
• Power and cooling: a workstation GPU in a cupboard will thermal-throttle and then become a helpdesk ticket.
• Physical access: the box holds the data. Treat it like a database server, not like a lab toy.

Renting a dedicated box in a colocation facility is still on-premise from a data-control point of view if you control the machine. Sharing a GPU with an unrelated VM is not.

CPU vs GPU

CPU inference works. It is how a lot of honest pilots start. It is also how a lot of team rollouts fail.

Use CPU when the model is small, concurrency is one or two, and users will tolerate several seconds per answer. Use GPU when people expect a conversation, when context is long, or when speech-to-text shares the machine.

A split is often cleaner than a compromise: CPU or a small GPU for embeddings and light extraction; a dedicated GPU for the generator and for Whisper. Two smaller boxes beat one overloaded box that does everything poorly.

VRAM

VRAM is the unit that ends arguments. Weights must fit. KV cache for the context must fit. Concurrent sequences multiply the cache. If it does not fit, the engine spills, slows, or refuses the request.

Quantization is how you trade quality for fit. Q4-class weights are the usual production compromise for business text; Q8 and full precision are for when you have measured a quality problem and can pay for it. Do not pick a 70B model because a chart said it was smart, then quietly quantize it into sludge to make it boot.

Measure: load the model, send the longest prompt you actually use, send two of them at once, watch VRAM. That afternoon saves a purchase order.

Quantization

Quantization is not a cheat code. It is lossy compression of weights. For internal summaries and retrieval-grounded answers, a good Q4 or Q5 model is often indistinguishable from a heavier quant in a blind review. For extraction into a strict schema, test it: some models get sloppy about JSON when squeezed.

Keep the quantisation recipe in version control next to the model name. “Whatever Ollama pulled” is not a recipe. When you change quant, you are changing the product. Treat it like a dependency bump: test the golden prompts, then promote.

Model storage and selection

Store weights on a volume you back up and inventory. Record licence, source URL, checksum, and the tasks they are approved for. Legal will ask. You should be able to answer without opening Discord.

Use different models for different jobs: a speech model, an embedding model, a generator, maybe a smaller extractor. A single “company model” is a marketing idea. It is a poor operations idea.

Have a promotion path: staging host, a fixed evaluation set (real documents, real questions, judged by a person), then production. Pulling a new tag on Friday because the internet liked it is how you spend Monday explaining why the assistant started inventing policy clauses.

Authentication

If more than one person can reach the UI, it needs real identity. SSO against the directory you already have is the least-bad option. A shared password in a wiki is not authentication.

Map identity to data. The legal team should not retrieve HR files because the index was built from a world-readable share. Retrieval that ignores ACLs is the failure mode that gets these projects killed, correctly.

Service-to-service: the application should call inference with a credential. The GPU host should not be an open port on the LAN. API keys in a frontend bundle are a joke that will be copied.

Concurrency

Concurrency is where laptop demos lie.

One user, one short prompt: almost anything looks fine. Eight users, RAG contexts of 20k tokens, a transcription batch, and a manager who pasted a 200-page PDF: the GPU is a queue.

Design for it:

• separate interactive and batch queues;
• cap context and attachments in the UI;
• show a position in the queue rather than a spinner with no end;
• autoscale only if you are not actually on-premise (on-prem, you buy another card or you wait).

vLLM-style continuous batching helps on a single GPU. It does not create VRAM that is not there.

Observability

You need the same kind of telemetry you would put on an internal API: request rate, latency, error rate, queue depth, GPU utilisation, VRAM, tokens in and out.

You do not need the full prompt in the log. You almost never want it. Log a request id, user id, model name, token counts, retrieval hit count, and a hash or a redacted preview if you must debug. Full-prompt logging is how private deployments leak through the back door.

Traces that include the prompt body should be a timed debug mode on staging, off by default, not an OpenTelemetry export into a shared tenant.

Updates

Three clocks: OS and containers, inference engine, weights.

OS and engine follow your normal patch window. Weights follow a slower, judged window. Users should not see the model change under them in the middle of a matter. Pin versions. Write down how to roll back to the previous weights in one command.

Internet-connected auto-update on an on-premise inference host is a gift to whoever owns the registry. Pull on a jump box, scan, copy in.

Backup

Back up configuration, identity mappings, the document index, and any chat history you are allowed to keep. Weights can be re-downloaded if you stored the checksum and the licence; still keep a copy if you are offline.

Test restore. An index you cannot rebuild is a liability. Re-embedding a large corpus is a weekend, not a button.

Do not back up prompt debug dumps to a public bucket. That sentence should not need to exist. It does.

Security boundaries

Draw three lines and keep them:

1. Data plane: files, transcripts, embeddings, chat. Stays on named volumes and named databases.
2. Inference plane: the GPU process. Receives prompts, returns tokens, talks to nothing else.
3. Control plane: CI, model registry, admin UI. Separate credentials.

People, not models, cross those lines. The model should not have filesystem rights, shell tools, or production database credentials. If you want a tool that creates a ticket, the application calls the ticket API after the model proposes a payload you validate.

On-premise is not a security control by itself. It is a place you can build controls. Without the lines above, it is just a GPU in a better postcode.

Operating cost

Budget for:

• hardware amortisation or lease;
• power and cooling;
• one named operator (fractional is fine; unnamed is not);
• evaluation time when weights change;
• disk growth of the corpus and the index.

Compare that, honestly, with local versus cloud. On-premise wins when utilisation is high and data control is non-negotiable. It loses when the GPU sits at 4% and the only user left is the person who requested the project.

If you want this designed against a real corpus rather than a diagram, that is the architecture track of private AI consulting: audit, then a single workflow on your metal, then production if the numbers hold.