Most Honest Company AI Architectures Are Hybrid, Not Pure Local
Hybrid AI is two routes after you classify the data: a private path you operate, and a cloud path you have contracted. It is also easy to overbuild.
Pure-local slides are easy to draw. Pure-cloud slides are easy to buy. Production systems that have to survive a DPO, a finance review, and a Tuesday on-call tend to be hybrid: some prompts never leave infrastructure the company controls, some prompts go to a named vendor under a contract, and some work never meets a model at all.
Hybrid private and cloud AI is not a compromise for people who could not decide. It is a classification exercise with two implementations. It is also heavier than a single path. If you cannot say which text takes which route, you do not have a hybrid architecture. You have two ways to be surprised.
I design private AI on kit you operate, and I am not going to pretend a local 8B model is a hosted frontier model with the bill removed. The hybrid design exists because those two tools are good at different jobs.
Hybrid means two routes, not two slogans
A route is a concrete path:
- who may initiate it;
- what data class is allowed in the prompt and in retrieval;
- where inference runs;
- what is logged;
- where the result is stored;
- who can see the result.
Private route. Weights and prompts on a machine you operate (on-premise or a private cloud account you treat as yours). No public AI vendor in the default path. Quality capped by what you can run and license. You patch it. Details of the on-site shape live in on-premise LLM deployment.
Cloud route. A named API, company account, DPA, retention settings, application front door. Text leaves. You usually get better general language quality and no GPU on-call. You accept a processor.
No-model route. Search, templates, macros, humans. Still a route. Write it down so people stop inventing a fourth path in a personal chatbot.
Hybrid is the statement “workflow A uses private, workflow B uses cloud, workflow C uses neither”, plus a gateway that enforces that statement. It is not “we have Ollama and OpenAI keys on the same laptop and we will be careful”. Careful is not a control.
The comparison of the two inference styles, without the hybrid wiring, is local LLM vs cloud AI. Read that if you still think one column of a vendor table is a strategy. This article is what happens when both columns are true for different tickets on the same day.
Classify the data before you classify the vendor
Start with information classes, not with GPUs.
A usable scheme for this problem is small. If you have seventeen labels, nobody will apply them.
Example, not a universal taxonomy:
| Class | Examples | Default route |
|---|---|---|
| Public | Marketing copy already on the website, published docs | Cloud API is fine; local is optional |
| Internal | Sprint notes, non-sensitive ops | Cloud under DPA, or private if you already run it |
| Confidential | Customer tickets, unpublished financials, HR | Private, or cloud only if legal named that processor for that class |
| Restricted | Health, payment secrets, M&A, data your MSA forbids sharing | Private or no model; often no model |
The table is useless until someone owns labelling at the workflow boundary. The agent, the app, or the intake form must pick a class before the prompt is built. A model that “detects sensitivity” after the user has already pasted the merger memo into a cloud field is a detective, not a lock.
Classify workflows as well as documents. Meeting audio, ticket text, and “rewrite this paragraph” are different systems even if they share a UI. Audio and documents often want to stay; a public blog draft often does not. The business framework is the right place to name input and output before you draw boxes.
Do not let “confidential” become a synonym for “everything we produce”. If everything is confidential, you will either block all useful cloud tools or you will ignore the label. Both happen.
European B2B specifics that actually change routing: GDPR roles (who is controller), whether the vendor is a processor you appointed, transfer mechanisms if the vendor is outside the EEA, and customer contracts that are stricter than the law. None of that is solved by a local model with debug logging.
Two routes in the application, not in a slide
The application should expose one user-facing job and pick the backend.
User / internal app
↓
Policy (class, workflow, user role)
↓
├── private inference (local or private GPU)
└── cloud API (named vendor)
↓
Result store with the same class as the inputImplementation notes that save pain:
- One policy module. Not a flag in the UI plus a different flag in the worker plus a third in a notebook. If the classifier says restricted, the cloud client is not in the process’s import graph for that request, or it is hard-failed. Soft “should not” will be bypassed on a deadline.
- Separate credentials and networks. The private host should not hold the cloud API key “in case we fallback”. Fallback is how confidential text travels during an outage. If the private path is down, the job waits or fails, unless the class allows cloud.
- Retrieval stays with the class. Do not retrieve confidential passages into a prompt that then goes to a public API. If you need cloud quality on a public summary, retrieve only public chunks. Mixing indexes without permissions is how hybrid becomes a leak with extra steps.
- Outputs inherit the input class. A summary of a restricted ticket is restricted. Putting it in a “company knowledge bot” index readable by everyone is a promotion of data.
Concurrency and operations split too. A local GPU that also handles the cloud-prep jobs will queue the confidential work behind a marketing rewrite. Separate queues. Separate timeouts. The on-prem article’s warning about one box doing everything poorly applies here even when one of the models is in someone else’s datacentre.
Masking and redaction are not a third route
People invent a magical middle: send the text to a frontier model after “removing PII”. Sometimes that is valid. Often it is a way to avoid classification.
What masking can do:
- strip obvious emails, card numbers, IBAN-shaped strings, phone numbers;
- replace customer names with tokens if you have a reliable named-entity step and a reversible map stored privately;
- drop attachments and screenshots entirely.
What masking cannot do:
- remove identifying stories (“the only importer of X in port Y”);
- read screenshots;
- guarantee that the remaining text is allowed to leave;
- undo a log on the vendor side if you were wrong.
If the unmasked text must not leave, do not send the masked text unless a human who owns the class has agreed that the residue is a different class. That agreement is rare for tickets and contracts. It is common for “here is a stack trace with hostnames replaced”.
Tokenisation maps are sensitive. The dictionary that maps CUSTOMER_17 back to a real name is a personal-data store. Put it on the private side. Do not ship it as a prompt prefix to the cloud model.
Measure masking like you would measure a parser: precision and recall on your own samples, not a vendor blog. A missed IBAN is an incident. An over-redacted prompt that makes the model guess is also an incident, just a quieter one.
When hybrid is heavier than it is worth
Two stacks means two failure domains, two bills, two skill sets, and a policy engine you will be tempted to underspecify.
Hybrid is usually too heavy when:
- One class dominates. If 95% of the work is confidential, run private (or no model) and buy a better internal tool. The remaining 5% of marketing rewrites can use a separate, clearly labelled cloud app with no access to the confidential index.
- One class is public. If nothing sensitive is in scope, a contracted API plus application hygiene is enough. A GPU “for optionality” is a hobby.
- There is no operator for the private path. Then you do not have hybrid. You have a cloud path and a demo that will be used incorrectly.
- Volume is tiny. Two routes for three prompts a week is ceremony. Use the stricter route for all of them, or none.
- The organisation cannot label. If intake cannot choose a class, the gateway cannot route. Fix intake. Do not add a model that “figures it out”.
- You wanted one UI that feels like a consumer chatbot. Hybrid with real controls feels like a form with constraints. If that is unacceptable, you are not buying architecture. You are buying a toy, and the toy will take the confidential route into the cloud.
Cost goes the wrong way when both sides are idle. You pay GPU depreciation or instance hours and token invoices, plus the engineering of the switch. Hybrid pays off when each route is actually busy on the work it is good at: private on a steady confidential workflow (transcription, RAG on internal files, ticket drafts), cloud on bursty public or internal-but-approved language work.
If the private side exists only to “have local AI” while every real prompt is flagged as allowed-for-cloud, shut the private side down. Monuments are not a control.
Operating two paths without lying to yourself
Runbooks must include both routes. Who rotates the API key. Who updates weights. What happens when VRAM is full. What happens when the vendor has an outage. What happens when legal revokes a vendor. The last one is a routing change, not a blog post.
Evaluation has to be per route and per workflow. A cloud model winning a public writing eval does not justify sending HR files there. A local model winning “did not leave the building” does not justify using it as the only writer for customer-facing legal letters.
Audit ids should show which route ran. If you cannot answer “did this prompt leave last Tuesday”, you cannot operate hybrid. Store route, model id, policy version, user, workflow, class. Do not store the prompt body unless the class and retention allow it.
People will press for a silent fallback. Document the refusal. Confidential jobs do not fail open to the frontier model. That sentence will be unpopular during an outage. It is the point of having a private path.
When you want this as an engagement rather than a diagram: private AI overview, a Private AI Readiness Audit from €1,500 that is allowed to recommend a single route, pilots and implementation from €300/day. The audit that concludes “cloud only” or “no model” is a successful audit.
Most honest company AI architectures are hybrid because the data is mixed and the models are unequal. Classify first, build two routes that cannot see each other’s forbidden text, treat masking as a limited helper, and drop hybrid when it is two idle systems and a story. Pure local is a valid special case. Pure cloud is a valid special case. Unclassified paste is not a case. It is the problem hybrid is supposed to end.