20 Sep 2026 · 4 min read
Building with an LLM without an API: running a model locally
A model running on your own machine ignores country lists, international cards and VPNs — and keeps working when the connection drops. Hardware needed, clean Django integration, and honest limits.
When an AI API isn't available in your country, the reflex is to look for a way around the restriction. There is another path, and a sturdier one: don't depend on an API at all.
A language model running on your own machine knows nothing about supported-country lists, international payment cards or VPNs. It also keeps working when the connection drops — which, in Kinshasa, is not a theoretical detail.
Here is how I actually use it, and above all what it does not replace.
What hardware you need
That's the first question, and the answer has changed a lot in two years. Models in the 3–8 billion parameter range now reach a quality that took 30 billion in 2024.
In practice, with Q4_K_M quantization — the standard quality/memory compromise:
| Available RAM | Realistic model size |
|---|---|
| 8 GB | 7B (viable minimum, no GPU) |
| 16 GB | 13B–14B |
| 24 GB and above | 32B |
A quantized 7B model takes roughly 4–5 GB, plus 1–2 GB for the context cache and system overhead. A GPU is not required: a recent 64-bit CPU is enough to get started, just slower. With 16 GB of RAM and an 8–12 GB GPU you reach comfortable speeds for daily use.
In other words: a decent developer laptop is enough. No need for a multi-thousand-dollar workstation.
Install and run
Ollama is the shortest path. Install it, then:
ollama pull qwen3.5:4b # light, good quality-to-memory ratio
ollama run qwen3.5:4b # immediate interactive test
The server then listens on http://localhost:11434.
Model names move fast — check the current library rather than copying last year's tutorial. The families worth looking at first today: Qwen, Llama, Mistral Small and Phi.
Integrating it into Django properly
This is where most tutorials stop, and where the real problems start. A local generation takes from a few seconds to several tens of seconds: calling the model directly in a view blocks a Gunicorn worker for that whole time.
An isolated service, with an explicit timeout:
# ai/services.py
import requests
from django.conf import settings
OLLAMA_URL = getattr(settings, "OLLAMA_URL", "http://localhost:11434")
MODEL = getattr(settings, "OLLAMA_MODEL", "qwen3.5:4b")
class ModelUnavailable(Exception):
"""The local server is not responding: treat it like a network outage."""
def generate(prompt, *, system="", timeout=120):
try:
response = requests.post(
f"{OLLAMA_URL}/api/chat",
json={
"model": MODEL,
"stream": False,
"messages": (
([{"role": "system", "content": system}] if system else [])
+ [{"role": "user", "content": prompt}]
),
},
timeout=timeout,
)
response.raise_for_status()
except requests.RequestException as exc:
raise ModelUnavailable(str(exc)) from exc
return response.json()["message"]["content"]
Then a Celery task, so the user isn't staring at a frozen page:
# ai/tasks.py
from celery import shared_task
from .services import ModelUnavailable, generate
@shared_task(bind=True, max_retries=3, default_retry_delay=10)
def summarize(self, document_id):
from .models import Document
document = Document.objects.get(pk=document_id)
try:
document.summary = generate(
document.body[:8000],
system="Summarize this text as five factual bullet points.",
)
except ModelUnavailable as exc:
raise self.retry(exc=exc)
document.save(update_fields=["summary"])
Three habits that make the difference:
Truncate the input. An 8,000-character context processed locally is already generous. Sending a whole document multiplies generation time without improving the result.
Treat unavailability as a network outage. The local server can be stopped, saturated, or still loading a model. ModelUnavailable plus a Celery retry — exactly like a payment webhook.
Cache deterministic results. If the same input comes back, don't regenerate: the cost here isn't money, it's CPU time and battery.
What local does not replace
Let's be honest, otherwise the article is worthless.
A local 7B model does not compete with a frontier model on complex reasoning, long code or fine-grained analysis. It is not the same tool.
Where it does excel — and this covers most of the real needs of a business application:
- classification and labelling (tickets, messages, documents)
- structured extraction from free text
- summaries and rewrites
- draft replies for a human to review
- semantic search, through embeddings
For these uses the quality gap with a hosted model is small, and the independence is total.
The real argument
People talk a lot about privacy and cost, which are good arguments. But the one that matters most here is elsewhere: it is infrastructure nobody can take away from you. No country list, no terms of service that change, no payment method to get validated abroad.
When you develop from a country providers haven't put on their lists yet, that property is worth more than a few points of benchmark performance.
The background to this question is covered in my article on AI tool access from the DRC.
Written by
Josué Dikoma — Full-stack engineer and versatile DevSecOps specialist