How to Self-Host an Open Source LLM: A Practical Setup Guide
Self-hosting an open-source LLM is all about control—until the GPU bill lands. This no-nonsense guide walks through VRAM sizing, quantisation trade-offs, serving stacks.
Learning how to self host an open source LLM means trading API convenience for control—and a set of infrastructure decisions you can’t afford to get wrong. Under-provision VRAM and your latency spikes; ignore quantifying and you burn GPU memory; skip cost auditing and you’ll wonder why the cloud bill doubled. This guide walks through the stack from GPU sizing to the serving engine, with a hard look at the expense most teams forget.
How to Self-Host an Open Source LLM: Start with VRAM, Not Dreams
Every self-hosting plan lives or dies by memory. Before choosing a model or a quantisation level, map out how many gigabytes you’ll need. The rule is simple but unforgiving: parameters × bytes-per-parameter + KV cache overhead. Adding 20–30% headroom keeps generation smooth.
Model size (B params) | 16-bit (2 B/param) | 8-bit (1 B/param) | 4-bit (0.5 B/param) | Minimal GPU example |
|---|---|---|---|---|
7 B | ~14 GB | ~7 GB | ~3.5 GB | 12 GB RTX 4070 |
13 B | ~26 GB | ~13 GB | ~6.5 GB | 16 GB A4000 |
34 B | ~68 GB | ~34 GB | ~17 GB | 24 GB RTX 4090 (4-bit) |
70 B | ~140 GB | ~70 GB | ~35 GB | 80 GB A100/H100 (4-bit) |
Mixtral 8×7 B (MoE) | ~46.7 GB active, ~93 GB total | ~23 GB active, ~46.7 GB total | ~11.7 GB active, ~23 GB total | 48 GB A6000 (4-bit) |
Active vs total memory matters for MoE models: during inference only a fraction of experts run, so you can often size for active param count if you’re careful with memory management.
All of this assumes a single GPU. If you need to shard across multiple cards, factor in inter-GPU bandwidth—a bottleneck that nullifies the savings from cheaper, smaller cards. For teams building a product around an LLM, this is exactly where generative AI development engagements help you avoid expensive wrong turns.
Quantisation: What You Trade for Smaller Models
Quantisation reduces the number of bits used to store each weight, cutting memory use almost linearly. But the accuracy hit isn’t uniform; some benchmarks show <2% degradation for 8-bit on many comprehension tasks, while aggressive 4-bit can widen that gap significantly on code generation. The only safe play is to evaluate against a sample of your real prompts.
Quantisation level | VRAM saving vs 16-bit | Typical quality impact | Latency change | Best use case |
|---|---|---|---|---|
8-bit (int8) | ~50 % | Negligible on standard QA, small regression on math-heavy prompts | Minimal | First step down from full precision |
4-bit (GPTQ/AWQ) | ~75 % | Generally acceptable for chat, noticeable on precise coding or numerical reasoning | Faster prefill (fewer bytes to move) | Self-hosted chatbots, RAG pipelines |
2-bit | ~87.5 % | Noticeable drop; can become incoherent on long context | Faster but may require more speculation | Experimentation, not production |
Why quantise? Because an 80 GB GPU costs $2–4/hour on demand. Halving your VRAM need might let you run on a 48 GB card, slashing run-rate by 40% or more. Over a month of 24/7 use, that’s thousands in savings. The cost dimension alone makes quantisation the first dial you turn when you learn how to self host an open source LLM.
The Serving Stack: Assembly Required
Your serving framework touches every request: how fast it begins, how many tokens you get per second, and whether the GPU sits idle between calls. Don’t default to a raw Python script.
Quick comparison
Framework | Batching | Quantisation support | Multi-GPU | Production notes |
|---|---|---|---|---|
vLLM | Continuous (paged attention) | AWQ, GPTQ, FP8 | Tensor parallelism | The production default; high throughput, good docs. |
TGI (Hugging Face) | Continuous | GPTQ, bitsandbytes | Sharding & tensor parallel | Solid ecosystem but heavier dependencies. |
llama.cpp | No native continuous batching (batch via queue) | GGUF | Offload layers to GPU | Excellent for single-user, low-resource setups. |
Ollama (wraps llama.cpp) | No continuous batching | GGUF | Single-card focused | Simplest path for local experimentation. |
Recommendation: If you’re serving more than one user or want to keep latency low under load, start with vLLM. The extra throughput from paged attention pays for itself quickly.
Setting up vLLM for a self-hosted LLM
pip install vllm
# Serve a 70B model with full tensor parallelism across 4 GPUs
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Meta-Llama-3-70B-Instruct \
--tensor-parallel-size 4 \
--dtype auto \
--max-model-len 8192 \
--gpu-memory-utilization 0.95Adjust --gpu-memory-utilization to leave room for KV cache overhead. For a 4-bit AWQ quantised version, just point --model at the quantised weights; vLLM autodetects the format.
If you’re still deciding whether to build or buy the infrastructure, our AI services team can benchmark your expected workload and recommend a sizing plan.
The Ongoing Cost Nobody Budgets: GPU Idling
Cloud GPU instances charge by the second whether they’re generating tokens or waiting for a prompt. Even with auto-scaling, cold-start delays force teams to keep at least one instance warm. That idle server burns roughly $1,500–5,000/month (depending on region and card) before serving a single request.
Strategies to cut the waste:
- Spot/preemptible instances: Can be 60–80% cheaper, but require graceful shutdown handling when reclaimed.
- Model multiplexing: Serve several fine-tuned LoRA adapters from the same base model, splitting the one warm GPU across multiple use cases.
- Request batching: Gather requests on the client side and submit them in bulk, reducing the duty cycle needed.
- Sleep-on-idle: Spin down to zero overnight if acceptable latency on first request isn’t critical.
These aren’t afterthoughts; they’re part of the answer to how to self host an open source LLM without your CFO demanding you switch back to an API.
When Self-Hosting Actually Makes Sense
Self-hosting isn’t about beating API per-request pricing on paper. It’s about data sovereignty, predictable latency under load, and vertical integration when the model itself is your product. If you’re shipping tens of millions of tokens daily, or you need custom inference pipelines (constrained decoding, repeated tree-of-thought) that an API can’t provide, self-hosting becomes a lever, not a cost center.
But the operational burden is real. Monitoring GPU memory leaks, handling CUDA version drift, and debugging OOM errors at 3 a.m. are part of the package. Many teams find the sweet spot is a hybrid: develop and experiment on self-hosted hardware, then offload production to a managed service that knows the LLM optimisation game. If that’s the path you’re exploring, get in touch and we’ll help you map out the trade-offs.
FAQ
How much VRAM do I need to self-host a 70B open-source LLM?
A rule of thumb is to multiply the parameter count (in billions) by the bytes per parameter (2 for float16, 1 for 8-bit, 0.5 for 4-bit) and add 20–30% for KV cache overhead. For example, a 70B model at 4-bit needs roughly 35 GB plus overhead, so an 80 GB A100 or H100 is sensible.
Does quantisation ruin model quality?
Not necessarily. Quantisation (4-bit or 8-bit) often preserves downstream accuracy on practical tasks while cutting VRAM requirements and boosting throughput. The key is to benchmark on your prompt distribution before committing.
Which serving stack should I use for an open-source LLM?
For production, vLLM—with its paged attention and continuous batching—is the current standard. For lighter experimentation or CPU-only setups, Ollama with llama.cpp is simpler.