One Box Can Run a Useful Local Model. It Cannot Run Every Fantasy
One company server is a valid local LLM contour for a narrow workflow and a small audience. VRAM, backups and honesty about what will not fit matter more than the UI.
A single company server is a legitimate way to run a local LLM. It is also the shape most demos accidentally promise: “we will put the model on a box in the rack and everyone will have ChatGPT”. The first sentence can be true. The second is a fantasy about VRAM, concurrency, and operations. This article is the contour that actually works on one machine, who it is for, and the point at which you should stop adding users and models to that machine.
I design private AI on infrastructure the company controls. One server is often where a real workflow should start. It is not a cluster, not a high-availability story, and not a substitute for a frontier API. If you need the buying decision rather than the box layout, use when a local LLM makes sense first.
What a one-server contour actually is
One hostname. One (or two) GPUs in that chassis, or CPU-only for a small model. One inference process. A small application in front: identity, a UI or an internal API, maybe a job queue on the same disk. Documents and audio either live on that box or on a fileshare the box can reach. Prompts do not go to a public chatbot as the default path.
Sketch:
Internal users
↓
Reverse proxy + SSO
↓
App (prompts, permissions, audit ids)
↓
Inference engine (Ollama, llama.cpp server, or one vLLM)
↓
GPU / CPU on the same machineOptional on the same machine, if you are honest about RAM and disk:
- PostgreSQL or SQLite for chat metadata;
- a vector index for a modest corpus;
- Whisper for a low volume of files, not for “all meetings, live”.
Optional not on the same machine: the company’s primary database, a second GPU host, object storage with the only copy of source files. Collocating inference with the only copy of the CRM is how an AI pilot becomes a disaster-recovery incident.
Network: no guest Wi-Fi, no inbound from the internet, no daily habit of ngrok. Outbound for pulling weights can exist on a controlled path. The process that sees prompts should not need a default route. That is ordinary service hygiene, expanded in on-premise LLM deployment. The one-server version is the same diagram with fewer boxes.
Who it is for
This contour fits when most of the following are true:
- One named workflow. Ticket summaries, internal document Q&A with citations, offline transcription of a defined audio set. Not “an assistant for the whole company”.
- A small concurrent audience. A handful of people in a working day, not a town-hall demo with fifty tabs open.
- Data that should not go to a public AI vendor as the default processor, or a site that cannot depend on outbound APIs.
- Someone who already owns Linux boxes. Drivers, disk, backups, “why is it swapping”. If that person does not exist, you do not have a server. You have a science fair.
- A quality bar that a mid-size local model can meet with retrieval or a template, reviewed by a human. If the bar is “indistinguishable from the best hosted chat”, one box will not save you.
It is a poor fit when procurement already decided on “GPU cluster for AI”, when legal requires failover across rooms, or when the first user story is a consumer-style chat with plugins, browsing, and image generation. Those are different products.
Pilots belong here. A Private AI Readiness Audit from €1,500 is for deciding whether the workflow belongs on a box at all. A single-workflow pilot at €300/day is the first time the box should see real files. Implementation stays at €300/day after the pilot earned another week. The catalogue page is private AI.
VRAM is the wall, not the branding
The model that “runs” in a screenshot at 2k context with one user is not the model you can serve at 32k context with three users and Whisper in the background. VRAM has to hold weights plus KV cache for every active sequence, plus whatever else you loaded and forgot to unload. The arithmetic and the workload profiles are in local LLM hardware requirements. This section is the one-server consequences.
On a typical single workstation-class GPU (illustrative, not a shopping list):
- a 7B–9B-class quantised model can be an interactive tool for one or two users;
- a 14B–32B-class quantised model is the usual honest internal assistant, still not a crowd;
- a 70B-class model may load at an aggressive quant and then crawl, or not load next to any other model;
- speech-to-text plus a generator plus embeddings is three memory budgets. They will fight. Stagger jobs or accept a queue.
CPU-only is valid for a personal or two-person tool on a small quantised model. It is how a lot of honest tests start. It is also how a “team rollout” dies: tokens per second fall off a cliff, people go back to a public chatbot, and you have a silent server.
Concurrency is a first-class input. Two long RAG prompts are not “the same load as one”. If the engine spills to system RAM, the box looks busy and the user is waiting. Cap max context and max parallel sequences in the engine. Put a queue in the app. A one-server contour without a queue is a demo that works until the second person arrives.
Do not run three generators “for choice” on one GPU. Run one default model, keep a second GGUF on disk, and switch in a change window. Choice is VRAM fragmentation with a settings page.
Ollama is a convenience, not an architecture
Ollama is a reasonable inference wrapper on a workstation and on a small internal server: pull a model, OpenAI-shaped HTTP, less ceremony than wiring vLLM on day one. Use it when the audience is small and you accept its scheduling limits. It is not a multi-tenant control plane. It is not a reason to skip SSO, disk encryption, or backup policy.
When you outgrow it on one box, the next step is still often one box: llama.cpp’s server, or vLLM if the GPU and the concurrency justify a Python service. Do not install all three. The on-prem deployment notes on keeping the inference server dumb apply: the engine loads weights and returns tokens. Permissions, retrieval, and audit live in the application.
Pin versions. Record model id, quant, checksum, and a tokens/s number on a fixed prompt in the runbook. “We use Llama” is not an inventory. Weights change. A pull that you did not intend is a production change.
The UI that comes with an engine is fine for an admin test. It is not a company product: no per-workflow policy, no classification, often too much logging, no tenant thinking. Put your own thin front door in front, even on one server.
Backups, logs, and the rest of the machine
The GPU does not get backed up in any useful sense. The things that do:
- Configuration: compose files, nginx, systemd, model pins, prompt templates you are allowed to keep.
- Application database: users, audit ids, any chat history policy says you may retain.
- Indexes: embeddings and metadata. Rebuilding from source is possible if ingest is documented; it is not free.
- Weights: either restore from an internal artefact store with checksums, or download again on a jump path. Do not treat Hugging Face as your only copy if you need to rebuild offline.
- Source documents and audio: preferably not only on the GPU box. The inference host should be replaceable.
Prompt logs are not a backup target to be proud of. If policy forbids keeping bodies, your backup must not resurrect them from /var/log. Debug flags that print prompts will copy those prompts into every snapshot. Turn them off.
Disk: weights plus Whisper plus a couple of alternatives plus documents is hundreds of gigabytes without trying. Fast disk helps loads. Full disks pause inference in stupid ways. Monitor them.
The rest of the machine is a server. OS patches, NVIDIA or AMD driver discipline, thermal reality in a cupboard, UPS if the room blinks, physical access because the disks hold the data. Treat it like a database host. A “lab” label does not change the contents.
Restore test: once. Rebuild inference from backup onto a scratch disk or a VM without the original GPU if you can, and onto the GPU when you must. A backup you have never restored is a story.
Not a cluster, and when to stop
One server means:
- maintenance is downtime, or at best a drained queue;
- a failed disk, a failed GPU, or a failed PSU is an outage;
- you cannot roll weights with a blue/green pair unless you bought a second box and did not mention it;
- you cannot isolate transcription from chat without time-slicing the same VRAM.
Stop adding load to this contour when any of these show up:
- Interactive latency is unacceptable at the real concurrency, after you have capped context and queued jobs.
- Two workflows need the GPU at the same hours and neither can wait.
- You need failover because the workflow is not allowed to be down for a hardware swap.
- The corpus or the audio volume makes ingest or Whisper a second full-time job on the same silicon.
- People start using a public chatbot again because the box is slow or full. That is a product failure, not a moral one. Either add capacity, shrink the workflow, or send the allowed class to a contracted API.
The next contour is still not “Kubernetes for LLMs” by default. It is often a second machine (embeddings and batch on one, interactive chat on the other), or a private cloud GPU with the same application, or a hybrid where confidential jobs stay on the box and public drafting uses a vendor. Hybrid is a classification problem, not a way to pretend one 24 GB card is infinite.
One box can run a useful local model if the workflow is named, the audience is small, VRAM is planned from real context lengths, Ollama or its replacement is treated as a process rather than a miracle, and backups know what not to keep. It cannot run every fantasy: frontier quality, the whole company, zero operations, and no queue. When the fantasy starts winning meetings, write the numbers on the chassis and stop.