Private AI

Local LLMs for Business: When Running AI Yourself Makes Sense

Running a local model is easy. Running one that reliably improves a real workflow is not. This is the decision framework before anyone buys a GPU.

Running an LLM locally is technically easy.

Running one that solves a real business problem reliably is a different problem.

Before buying GPUs or installing models, the first question should be what workflow the system is supposed to improve.

A local large language model is a piece of software that generates and transforms text on hardware you control. That hardware might be a workstation under a desk, a server in your rack, or a virtual machine in an account that only your company can reach. The important part is not the brand of the inference engine. The important part is the path the data takes: what comes in, what leaves, and who can see the prompt.

This article is for the people who have to make that call: CTOs, engineering managers, IT leads, operations leads. It is not a tutorial for installing a chat UI. If you already know you want a deployment, the commercial shape of that work is private AI deployment. What follows is the decision that should happen first.

What a local LLM actually means

“Local LLM” is used loosely. People mix up four different things:

1. A model file on a laptop, talked to through a terminal or a small web UI.
2. An inference server on a company machine, reached by several employees.
3. The same server in a private cloud account, still not a public chatbot.
4. A hybrid: some prompts stay inside, some go to a vendor API.

All four can be legitimate. Only the first two are local in the strict sense. The third is private without being on-premise. The fourth is usually what production systems look like after the slogans have worn off.

A local LLM is also not a clone of a consumer chatbot, and it is not automatically more secure. It is a process that loads weights into RAM or VRAM and predicts tokens. If you log the full prompt, store chats forever, or mount the disk on a backup that leaves the building, you have rebuilt a data-handling problem with extra steps.

Treat it as infrastructure: a service with a network boundary, an identity, a resource budget and an owner.

Why businesses consider local AI

The trigger is almost never “we want to run Llama”. The trigger is a workflow that already exists.

Someone transcribes interviews by hand. Someone summarises the same class of report every Friday. Someone searches a share of PDFs by filename. Someone pastes a contract into a public chatbot because there is no official tool, and security finds out later. Someone’s API bill for speech-to-text or embeddings has become a line item that finance asks about.

Those are process problems with a possible model-shaped component. The model is justified when:

• the input is sensitive enough that a public API is the wrong default;
• the work repeats often enough that a person doing it all day is the expensive part;
• the output has a shape you can check (transcript, summary, JSON, draft);
• someone will own the system after it is deployed.

If the work is rare, public, or undefined, a local model is a hobby. There is no shame in that; it is just not a business system.

Privacy

Privacy is the most common reason to even have this conversation, and it is also the easiest to overclaim.

What local inference actually changes: the prompt does not have to go to a multi-tenant AI vendor as the default path. Weights sit on disks you can inventory. You can deny egress from the inference host. You can keep embeddings, transcripts and chat history in the same database you already trust for other internal tools.

What it does not change: laptops, screenshots, email forwards, overly verbose logs, admin access, and backups. A local model with debug=true writing prompts into a shared log aggregator is not a privacy win. Neither is a “temporary” ngrok tunnel to a GPU at home.

Be precise in the policy. Write down, for each class of data:

• may it leave the building;
• may it go to a named vendor under a DPA;
• must it stay on named hosts;
• how long may derived text (transcripts, embeddings, chat) be kept;
• who may query it.

Then build the system so the default path matches that list. Privacy here is an architecture and an operations habit, not a checkbox next to the model name.

If you want the threat model of putting the model next to an existing API, the older internal notes on isolating prompts and logs still apply in spirit: treat the model as another backend service, not as a magic vault.

Cost

Cost is the second argument, and it is where spreadsheets go wrong in both directions.

Vendor APIs look cheap until volume is high and continuous: every meeting transcribed, every ticket summarised, every document re-embedded on change. They look expensive until you add a GPU server, electricity or cloud rent, engineering time, and the hours people spend correcting answers that a stronger hosted model would have got right.

The one cost that disappears on a local LLM is the per-token (or per-minute) usage fee. Extra drafts and summaries do not generate an OpenAI-style invoice. That is worth stating plainly. It is also not a claim that local is free: the machine, power and the person who operates them still show up in the budget.

A fair comparison uses the same unit of work. For transcription, that is hours of audio. For search, that is questions per day plus the cost of keeping the index fresh. For drafting, that is documents produced and then edited. Convert those to tokens or to GPU-hours, then add the people.

Rules of thumb that survive contact with actual invoices:

• A few hundred chat prompts a month almost never justifies owned hardware.
• All-day speech-to-text on a private corpus often does, especially if the alternative is a hosted speech API plus a hosted LLM.
• Search over a stable internal corpus is usually dominated by engineering time, not by GPU rent, until the corpus and the query rate both get large.
• The first year of a local system is mostly people cost. Hardware is visible; operations is not.

There is a longer side-by-side in local LLM vs cloud AI. Use it. Do not skip to the conclusion.

Latency

Local inference can be faster than a round trip to a public API. It can also be slower. The difference is not ideology; it is queueing.

On a quiet workstation, a small quantized model answers in a second or two. On a shared GPU with four people chatting, a transcription job in the queue, and a 32k context, you wait. Public APIs hide that queue behind a fleet. You pay for that hiding.

Latency matters when a person is staring at a cursor. It matters less when a batch of recordings is processed overnight. Design the workflow around which of those you actually have. Interactive chat on CPU-only hardware is how local AI projects lose political support in week two.

Offline environments

Some sites cannot assume internet, or cannot allow it for this class of data: plants, hospitals, legal war rooms, air-gapped engineering networks. Local models are one of the few ways to put language tools there at all.

Offline is a constraint, not a feature. You need a process to bring weights in (signed media, an update window), a process to get logs out if you need them, and no silent fallback to a vendor when the local model is weak. If the workflow only works with a frontier closed model, offline local AI will not save it. Pick a different workflow, or accept that this site does not get that tool.

Common business workflows

Five workflows absorb most of the demand I see. They are also the ones that survive a pilot.

Transcription. Audio in, text out, then a summary and action items. This is the most operationally boring and the most useful. See local AI transcription.

Private document search. Ask questions against PDFs, policies and wikis, with citations. This is retrieval plus a generator, not a brain. See private AI document search.

Drafting and rewriting. Templates and approved context in, a draft out, a person in the loop. Do not connect this to automatic publishing.

Internal assistant. A bounded chat UI on top of the two items above. The boundary is the point: which corpora, which tools, which identities.

Extraction. Unstructured text to JSON that a database will accept. The schema and the validator do more work than the model.

If a proposed project is not one of these, write the input and the output on one line. If you cannot, you do not have a workflow yet. You have a wish.

Hardware considerations

Hardware follows the workflow. A personal drafting tool can live on a recent laptop with enough RAM, often without a dedicated GPU, if the model is small and quantized. A team assistant, long context, or continuous transcription usually wants a GPU and a machine that is not also someone’s workstation.

The variables that actually move the needle:

• parameter count and quantization (what has to fit in VRAM or RAM);
• context length (which grows memory during a request);
• concurrency (how many requests at once);
• whether speech-to-text shares the same GPU.

Do not start from a GPU shopping list. Start from “one person, drafts, 8k context” versus “twelve people, search, 32k context, plus overnight transcription”. The worked examples live in local LLM hardware requirements. Those numbers will go stale; the structure of the question will not.

Local vs cloud

Local and cloud are not moral categories. They are deployment options with different failure modes.

Use local or private infrastructure when the data is sensitive, the volume is high and predictable, offline is required, or you need to control the model lifecycle.

Use a public API when quality at the frontier is the product, usage is bursty or low, you have no one to operate a GPU, or the text would be fine in email.

Use hybrid when both sentences are true for different workloads: meeting audio stays inside; marketing copy goes to a vendor.

The longer argument, including a decision tree, is in the local versus cloud comparison. If someone in the room is arguing from a vendor blog or from a privacy panic, send them there before you buy anything.

Maintenance requirements

A local model is not a PDF you drop on a share. Someone has to:

• apply OS and container updates;
• pull new weights when you choose to, and test them;
• watch GPU memory, queue depth and error rates;
• keep backups of indexes, prompts you are allowed to keep, and configuration;
• rotate credentials on the UI and the API;
• document how to restart it at 2 a.m.

If that someone does not exist, you do not have a local AI programme. You have a demo that will rot. Paying a vendor to hide this is rational. Paying an engineer to own it is also rational. Pretending it is free is how these systems embarrass people six months later.

Decision checklist

Work through this in order. Stop when a “no” is load-bearing.

1. Name the workflow. Input, output, who uses it, how often.
2. Name the data. What must not leave, what may, under whose policy.
3. Name the quality bar. What does a wrong answer cost? Is a smaller local model good enough if a person reviews it?
4. Name the volume. Per day, not per demo.
5. Name the hardware you already have. Then the gap.
6. Name the operator. A team, a person, a contractor. Not “IT in general”.
7. Run one path on real files. A day of audio, a folder of policies, a set of templates. Not a marketing PDF.
8. Compare to the API on the same files. Quality, time, cost, and how the data would have travelled.
9. Decide local, private cloud, hybrid, or stay on the vendor. Write it down, including what you rejected.

If you want help with that sequence as an engagement, that is the local LLM setup for business offer: an audit first, then a single-workflow pilot, then production only if the pilot earned it.

Local models are a tool. They are the right tool when the workflow is real, the data should stay put, and someone will run the machine. They are the wrong tool when the actual need is “ChatGPT, but inside the firewall, and as good, and cheaper, and nobody has to maintain it”. That product does not exist. The useful versions of it do, and they look like the workflows above.