A program manager wants AI help summarizing case notes. The executive director likes the idea but won’t send client records to an outside company. Someone on staff suggests running a model in-house: “It’s free, and the data never leaves the building.” The director’s caution is sound. The staff member’s pitch is only half true, and the gap between “free and private” and “running reliably and securely for twenty people” is where most self-hosting projects stall.

We regularly run open-weight models on our own hardware through Ollama and vLLM and score their outputs against rubrics in our benchmarking lab. That work informs what we recommend to clients. This guide covers when self-hosting is the right call, what the main options look like, how to keep a self-hosted endpoint from becoming a liability, and how to choose a model before you buy anything.

When Self-Hosting Makes Sense

An open-weight model is one whose trained weights (the files that make up the model) are published for anyone to download and run. Self-hosting means running one on hardware you control, whether a machine in your office or a server you rent. Three situations make it worth the effort.

Sensitive data you can’t or won’t send out. Case notes, health information, legal files, donor records, internal investigations. If your obligations to clients, funders, or regulators rule out a third-party processor, or if nobody can get comfortable with a vendor’s terms, a local model keeps the data on systems you control.

Predictable, high-volume work. If you plan to run thousands of documents through a model every week (classifying intake forms, extracting fields from invoices, tagging records), per-token API costs add up and are hard to forecast. Hardware is an upfront cost plus power and upkeep, and the marginal cost of one more document is close to nothing. That math only works if the volume is real and sustained.

Offline or restricted environments. Field sites with unreliable internet, air-gapped networks, or locations where sending data abroad is a problem.

When an API Is the Better Call

For many organizations, a reputable commercial API with good data terms is the better choice, and we often recommend one. Choose an API when:

  • Quality on hard tasks matters most. The best open-weight models are capable, but on complex reasoning, long documents, and nuanced writing they can trail the frontier models from Anthropic, OpenAI, and Google. Whether that gap matters depends on your task, which is why you test (more on that below).
  • Volume is low or unpredictable. Paying per use beats buying hardware that sits idle.
  • Nobody owns the system. A self-hosted model is a server. Someone has to patch it, monitor it, and fix it when it stops responding on a Monday morning.
  • The vendor’s terms are acceptable. Many API providers offer business plans that exclude your data from training, limit retention, and sign data processing agreements. Our AI vendor due-diligence checklist walks through what to verify.

A hybrid is common and often the best answer: a local model for the sensitive workflow, an API for everything else.

Ollama vs. vLLM in Plain Terms

Both tools load a model and expose it over an API that other software can call. They’re built for different jobs.

Ollama is the simplest path. It installs like a normal application, downloads models with one command, and runs well on a single workstation or a Mac with plenty of memory. It suits one person or a handful of light users, prototyping, and testing which models fit your work. If you have never run a local model, start with our beginner’s guide to Ollama.

vLLM is a serving engine designed for throughput. It handles many simultaneous requests efficiently by batching them together on the GPU, and it exposes an OpenAI-compatible API, the format many existing AI tools already speak. It suits a shared internal service that a whole team or an application depends on. The trade-off is setup: vLLM expects a Linux server with a supported GPU and someone comfortable with drivers, containers, and configuration.

Our usual path: prototype and evaluate with Ollama, then move to vLLM only if real usage shows you need concurrency.

Hardware Without the Hype

We won’t quote specs or prices here because they change too fast to stay accurate. The principles hold steady.

GPU memory decides what fits. A model’s weights have to sit in memory to run quickly. Larger models (measured in billions of parameters) need more memory. At standard 16-bit precision, each parameter takes two bytes, so memory needs scale directly with model size.

Quantization trades some quality for size. Quantization stores weights at lower precision, for example 8-bit or 4-bit instead of 16-bit. A 4-bit version of a model needs roughly a quarter of the memory of the 16-bit original. Quality usually drops a little, sometimes more on precise tasks like extraction or math. Test the quantized version you will actually run, not the full-precision one from a leaderboard.

Context and concurrency cost memory too. Long documents and multiple simultaneous users each consume memory beyond the weights themselves. A model that fits comfortably for one user reading short emails may not fit for five users reading long reports.

Apple Silicon is a legitimate option for small deployments. Its unified memory lets larger models run on a single desktop machine, which works well for Ollama and a few users. For heavy concurrent serving, dedicated GPUs are the standard.

Securing a Self-Hosted Endpoint

Running a model locally solves the third-party problem and creates a new one: you now operate a service that holds or processes sensitive data. Our work in incident response shapes how we think about this. Exposed, forgotten services are a common way attackers get in.

  • Never expose it to the internet unauthenticated. Ollama ships without built-in authentication and listens only on the local machine by default. Changing that setting to make it reachable from other computers is the moment to add protection. An open model endpoint lets anyone use your hardware, and if it’s connected to documents or tools, it may let them reach your data.
  • Put it behind a private network. Keep the server on an internal network segment and have remote staff reach it through a VPN or similar private-access tool, not a forwarded port.
  • Require authentication in front of it. Use a reverse proxy or gateway that checks credentials, ideally tied to your existing staff accounts so offboarding happens in one place. If you issue API keys to applications, give each its own key and keep them in a password manager, not in a shared document.
  • Log access, and decide what to log. Record who called the service and when. Think carefully before logging full prompts and outputs: those logs become a new copy of the sensitive data you were trying to protect, and they need the same access control and retention limits.
  • Limit who can do what. Separate the people who can query the model from the people who can change models or configuration.
  • Get model files from trusted sources. Download weights from the original publisher or a well-known repository, verify you have the official release, and prefer safetensors (or GGUF, the format Ollama uses) over older pickle-based formats that can execute code when loaded. Check the license too. Open-weight licenses vary, and some restrict commercial use or certain applications.
  • Patch the stack. The serving software, GPU drivers, container images, and operating system all need updates.

If the model connects to email, files, or other business systems, the risks multiply. Our guide to connecting AI to business tools safely covers least privilege for those integrations.

Evaluate Before You Commit

One of the most expensive mistakes is buying hardware for a model nobody tested on real work. Public benchmarks measure general ability, not whether a model summarizes your case notes accurately or follows your intake format.

Before spending money:

  1. Collect 20 to 50 real examples of the task, with sensitive details removed or replaced.
  2. Write a short rubric: what counts as correct, complete, and usable, and which failures are unacceptable (invented facts, dropped dates or deadlines, wrong names).
  3. Run two or three candidate open-weight models through Ollama on a machine you already own, plus one commercial API model as a baseline.
  4. Score outputs without knowing which model produced them.
  5. Compare quality against the operational cost of each option.

Our rubric-based evaluation guide explains this process in detail, including a starter rubric. In our experience, this step often changes the decision. Sometimes a small model is good enough and the hardware bill shrinks. Sometimes the open models fail the task outright, and an API with strong data terms is the honest recommendation.

The Honest Trade-Offs

Self-hosting gives you control. It also hands you every responsibility a vendor would otherwise carry.

  • Operational burden is ongoing. Updates, monitoring, outages, and model upgrades don’t stop after launch. Budget staff or consultant time for them, not just hardware.
  • Quality may lag on hard tasks. Open models improve quickly, but on the most demanding work they can trail frontier models. Plan to re-test as both sides change.
  • Privacy depends on your own security. A self-hosted model on a poorly secured server can be less private than a well-run API. Local doesn’t mean safe by default.
  • Hardware ages. Newer models may outgrow the memory you bought for today’s.

None of this rules self-hosting out. It means the decision should rest on your data, your volume, and your capacity to run a service, not on the appeal of “free.”

Getting Help

We help small businesses and nonprofits decide whether to self-host, test candidate models against their real work, and deploy private model servers with authentication, network isolation, and logging set up from day one. If you’re weighing a private AI deployment, learn about our AI services for organizations.