The question lands in Slack about once a month on compliance-sensitive teams: "Could we just run the model ourselves?" The instinct behind it is sound. In finance, healthcare, legal, and government-adjacent industries, more enterprises are asking hard questions about where their data goes - and "we send it to OpenAI" is no longer a frictionless answer in many procurement conversations. But "run it ourselves" is doing a lot of work in that sentence, and the answer is more qualified than either the privacy advocates or the cost skeptics usually admit.
What self-hosting actually protects
Self-hosting means running an open-weight model on hardware you control - your own servers, a private cloud VPC, or a rented GPU cluster where you hold the keys. An open-weight model has downloadable parameters you can run on your own hardware, inspect, fine-tune, and control end to end. A closed model is accessed only through a vendor API: you send prompts and get responses, but the model itself stays on the provider's servers. Open weights give you control and privacy; closed APIs give you convenience and zero infrastructure to run.
The privacy guarantee is real and specific. Your prompts stay on your servers. Your customer data does not leave your VPC. Your source code does not touch a third-party inference endpoint. For a team handling medical records, legal discovery documents, or ITAR-controlled technical data, that is not a minor advantage - it is the only compliant path. CMMC requires that CUI be processed only in authorized environments. HIPAA requires Business Associate Agreements and appropriate safeguards for PHI. ITAR restricts defense-related technical data from being accessed by foreign persons, which includes processing on cloud infrastructure in regions with foreign staff. Private AI running on your own servers in your own facility eliminates these geographic and access control concerns.
What self-hosting does not do is make your AI deployment automatically compliant with everything. Every agentic deployment touches four independent data surfaces: the model weights, the inference API, the prompt and context pipeline, and the logs.
Prompt and response logging pipelines can inadvertently persist sensitive user data or proprietary context
- and those logs live on your infrastructure, which means they are your audit burden, not your vendor's. You moved the surface; you did not eliminate it.
The cost math most teams get wrong
The economics of self-hosting look attractive in headlines and misleading in spreadsheets. The summary version:
Self-hosting is not automatically cheaper. Because its cost is mostly fixed, a lightly used self-hosted model is expensive per token, while a heavily used one is cheap per token. The API is the mirror image: cheap when usage is low, expensive when usage is high and constant. That is the entire basis of the break-even calculation.
The numbers behind that shape matter. A 7B model on an A100 80G SXM4 running at roughly 3,300 tokens per second - consistent with vLLM's continuous batching on that hardware - works out to about $0.15 per million tokens. That clears frontier API rates easily. But a 70B model tells a different story: a single H100 running a 70B model in FP8 with vLLM's continuous batching delivers roughly 400 tokens per second, about 1.44 million tokens per hour, which works out to a ceiling of roughly 34.5 million tokens per day before you need a second GPU.
The engineering overhead is where most projections fall apart. A self-hosted LLM deployment conservatively requires 10-20 hours per month of engineering time for maintenance, monitoring, and troubleshooting. At $75-$150 per hour for a senior DevOps or ML engineer, that is $750-$3,000 per month in labor cost alone.
The non-obvious conclusion: against a budget open-weight API at roughly $0.14-$0.50 per million tokens, self-hosting almost never wins on cost alone. It beats a frontier API at roughly a few million tokens per day - provided you keep the GPU at healthy utilization of 60% or higher. Against the budget tier, it rarely wins on cost at all.
If cost savings is your primary argument for self-hosting, the math is fragile. If data residency is your argument, the math becomes secondary - which is exactly the right order of priority.
| Decision factor | Cloud API | Self-hosted open-weight |
|---|---|---|
| Data leaves your environment | Yes | No |
| Variable usage cost | Low | N/A (fixed) |
| High-volume cost | Scales linearly | Fixed until GPU ceiling |
| Engineering burden | Near zero | 10-20 hrs/month ongoing |
| Compliance audit control | Vendor-dependent | Full ownership |
| Model update path | Automatic | Manual |
| Time to first working prototype | Minutes | Days to weeks |
Ollama is a starting point, not the destination
Ollama hit 52 million monthly downloads in Q1 2026 - a 520x increase from 100K in Q1 2023. HuggingFace hosts 135,000 GGUF-formatted models optimized for local inference, up from 200 three years ago. The tool that powers most of those downloads has earned its place as the fastest path from "curious" to "running locally."
Ollama is first a practical model runtime and integration tool for local or private machines. It is not, by itself, a complete enterprise AI platform with organization management, permissions, audit workflows, capacity governance, and multi-tenant controls. For a team of two engineers trying a 7B model before committing to infrastructure, it is the right tool. Even an RTX 3060 12GB can run 7B-13B models. Let a small team experiment for two to four weeks to validate the use cases.
For a production deployment handling 50 concurrent users, audit logging, and SSO integration, if the target system needs hundreds of concurrent users, centralized model governance, multi-tenant permissions, cost allocation, elastic scheduling, and strict audit controls, Ollama can still be a useful validation layer, but it should not be treated as the whole enterprise inference platform. The gap between "it runs" and "it runs reliably at scale" is where most self-hosting pilots stall.
Beagle in action#legal-ops, 11:02amWhat the EU AI Act deadline changes for self-hosters
There is a common misconception that running a model on your own servers puts you outside EU AI Act scope. It does not. The Digital Omnibus on AI entered into force, making the high-risk AI delay to December 2027 legally binding, while chatbot disclosure and GPAI penalty enforcement proceed on schedule for August 2.
Article 50 transparency obligations - requiring chatbot disclosure, synthetic content marking, and deepfake labeling - became enforceable for any AI system deployed in the EU's single market of 450 million people.
Under the Omnibus agreement, high-risk AI system obligations are postponed: stand-alone Annex III systems covering recruitment, credit scoring, law enforcement, education, and border control tools will need to comply by 2 December 2027, and AI embedded in regulated products by 2 August 2028. That buys time on the heavy documentation and conformity assessment requirements. It does not buy time on transparency disclosures.
The practical implication: teams who self-host to avoid compliance obligations are solving the wrong problem. Self-hosting solves data residency. Compliance is a separate layer that applies regardless of your inference stack.
Self-hosted AI: common questions
What does self-hosting an AI model actually protect?
Self-hosting keeps your prompts, context, and outputs on hardware you control. Your data never reaches a third-party inference endpoint, which satisfies data residency requirements under GDPR, HIPAA, CMMC, and ITAR. It does not automatically satisfy disclosure, logging, or risk-management obligations - those apply based on how and where you deploy, not where the model weights live.
When does self-hosting an LLM beat an API on cost?
Self-hosting breaks even at roughly 5-10 million tokens per month for premium models. Below that threshold, the fixed cost of GPU rental plus engineering time makes APIs cheaper on a per-token basis. The break-even shifts dramatically with model size: a 7B model on modest hardware clears the line much earlier than a 70B model requiring multi-GPU infrastructure.
Is Ollama production-ready for enterprise teams?
Ollama is an excellent proof-of-concept and development runtime. Ollama Cloud publishes no uptime guarantee - services are "as is" - versus OpenAI's 99.9% and Anthropic's 99.5% tiers. For production workloads with high concurrency, audit requirements, or SLA commitments, teams typically graduate to vLLM or a managed private inference service behind their own API gateway.
Does the EU AI Act apply if you host the model yourself?
Yes. The EU AI Act's scope is determined by where the AI system is used - specifically, whether it affects people in the EU market - not where the model weights are stored. Any provider placing AI systems on the EU market, or whose AI outputs are used in the EU, falls within scope regardless of where the provider is established. Self-hosting changes your data processing story; it does not change your regulatory scope.
Which open-weight models are actually ready for private deployment?
Models in the 20-70 billion active-parameter range - accessible either via API from open-weight providers like Together AI, Fireworks, or Groq, or self-hosted on reasonable GPU infrastructure - handle the bulk of production workloads. DeepSeek V4 Flash, Qwen 3.6, Gemma 4 26B, and Mistral Small 4 are the workhorses of a hybrid architecture: cheaper than frontier closed APIs, capable enough for most tasks. For the permissive-license requirement most enterprise legal teams have, the most permissive licenses - Apache 2.0, MIT - sit with Mistral, much of Qwen, DeepSeek, and Microsoft Phi.