Skip to content
Josué Dikoma©

20 Sep 2026 · 4 min read

Building with an LLM without an API: running a model locally

A model running on your own machine ignores country lists, international cards and VPNs — and keeps working when the connection drops. Hardware needed, clean Django integration, and honest limits.

When an AI API isn't available in your country, the reflex is to look for a way around the restriction. There is another path, and a sturdier one: don't depend on an API at all.

A language model running on your own machine knows nothing about supported-country lists, international payment cards or VPNs. It also keeps working when the connection drops — which, in Kinshasa, is not a theoretical detail.

Here is how I actually use it, and above all what it does not replace.

What hardware you need

That's the first question, and the answer has changed a lot in two years. Models in the 3–8 billion parameter range now reach a quality that took 30 billion in 2024.

In practice, with Q4_K_M quantization — the standard quality/memory compromise:

Available RAM Realistic model size
8 GB 7B (viable minimum, no GPU)
16 GB 13B–14B
24 GB and above 32B

A quantized 7B model takes roughly 4–5 GB, plus 1–2 GB for the context cache and system overhead. A GPU is not required: a recent 64-bit CPU is enough to get started, just slower. With 16 GB of RAM and an 8–12 GB GPU you reach comfortable speeds for daily use.

In other words: a decent developer laptop is enough. No need for a multi-thousand-dollar workstation.

Install and run

Ollama is the shortest path. Install it, then:

ollama pull qwen3.5:4b     # light, good quality-to-memory ratio
ollama run qwen3.5:4b      # immediate interactive test

The server then listens on http://localhost:11434.

Model names move fast — check the current library rather than copying last year's tutorial. The families worth looking at first today: Qwen, Llama, Mistral Small and Phi.

Integrating it into Django properly

This is where most tutorials stop, and where the real problems start. A local generation takes from a few seconds to several tens of seconds: calling the model directly in a view blocks a Gunicorn worker for that whole time.

An isolated service, with an explicit timeout:

# ai/services.py
import requests
from django.conf import settings

OLLAMA_URL = getattr(settings, "OLLAMA_URL", "http://localhost:11434")
MODEL = getattr(settings, "OLLAMA_MODEL", "qwen3.5:4b")


class ModelUnavailable(Exception):
    """The local server is not responding: treat it like a network outage."""


def generate(prompt, *, system="", timeout=120):
    try:
        response = requests.post(
            f"{OLLAMA_URL}/api/chat",
            json={
                "model": MODEL,
                "stream": False,
                "messages": (
                    ([{"role": "system", "content": system}] if system else [])
                    + [{"role": "user", "content": prompt}]
                ),
            },
            timeout=timeout,
        )
        response.raise_for_status()
    except requests.RequestException as exc:
        raise ModelUnavailable(str(exc)) from exc
    return response.json()["message"]["content"]

Then a Celery task, so the user isn't staring at a frozen page:

# ai/tasks.py
from celery import shared_task

from .services import ModelUnavailable, generate


@shared_task(bind=True, max_retries=3, default_retry_delay=10)
def summarize(self, document_id):
    from .models import Document

    document = Document.objects.get(pk=document_id)
    try:
        document.summary = generate(
            document.body[:8000],
            system="Summarize this text as five factual bullet points.",
        )
    except ModelUnavailable as exc:
        raise self.retry(exc=exc)
    document.save(update_fields=["summary"])

Three habits that make the difference:

Truncate the input. An 8,000-character context processed locally is already generous. Sending a whole document multiplies generation time without improving the result.

Treat unavailability as a network outage. The local server can be stopped, saturated, or still loading a model. ModelUnavailable plus a Celery retry — exactly like a payment webhook.

Cache deterministic results. If the same input comes back, don't regenerate: the cost here isn't money, it's CPU time and battery.

What local does not replace

Let's be honest, otherwise the article is worthless.

A local 7B model does not compete with a frontier model on complex reasoning, long code or fine-grained analysis. It is not the same tool.

Where it does excel — and this covers most of the real needs of a business application:

  • classification and labelling (tickets, messages, documents)
  • structured extraction from free text
  • summaries and rewrites
  • draft replies for a human to review
  • semantic search, through embeddings

For these uses the quality gap with a hosted model is small, and the independence is total.

The real argument

People talk a lot about privacy and cost, which are good arguments. But the one that matters most here is elsewhere: it is infrastructure nobody can take away from you. No country list, no terms of service that change, no payment method to get validated abroad.

When you develop from a country providers haven't put on their lists yet, that property is worth more than a few points of benchmark performance.

The background to this question is covered in my article on AI tool access from the DRC.

IAOllamaDjangoAuto-hébergement

Written by

Josué Dikoma — Full-stack engineer and versatile DevSecOps specialist

Next step

A project in mind?

Let's talk about it over a free 30-minute call — no strings attached.