Running a local LLM on a laptop: Ollama, Mistral and Open WebUI in Docker

“Can we use AI on data that must not leave the company?” One answer is a model that runs entirely on your own hardware. I wanted to know what that looks like in practice - not on a GPU workstation, but on the kind of laptop a finance or operations team actually has.

What I found, in short

• It works. A 12-billion-parameter model, a chat interface and an API, all on one Windows laptop, nothing sent to the cloud.
• It is slow on a CPU. 3-8 tokens per second. Fine for testing and batch jobs, not for a fluent conversation.
• It is confidently wrong when it lacks data. A made-up answer looks exactly like a correct one. Without sources, don’t trust it - not even with the weather.
• “Local” ends the moment you switch on web search. Your query then goes to a search engine.

If those trade-offs are acceptable, here is how to build it. Tested in September 2026 on Windows 11 with 32 GB RAM and no dedicated GPU.

The set-up

┌──────────────┐     HTTP :3000     ┌────────────────────┐    HTTP :11434    ┌──────────────────┐
│  Browser     │ ─────────────────▶ │  Open WebUI        │ ────────────────▶ │  Ollama          │
│  (you)       │                    │  (container)       │                   │  (container)     │
└──────────────┘                    │  • chat, history   │                   │  • runs models   │
                                    │  • users           │                   │  • mistral-nemo  │
                                    │  • web search ─────┼──▶ DuckDuckGo     └──────────────────┘
                                    └────────────────────┘

Three layers, each doing one job:

Layer	What it is	Analogy
Mistral (mistral-nemo)	The model itself - a file of weights (~7 GB)	Brain
Ollama	The runtime that loads the model into RAM and computes answers. Exposes an API on port 11434.	Engine
Open WebUI	The web interface: chat, history, settings, web search	Dashboard

The model knows nothing about the web or the interface. It takes text in and gives text out. Everything else comes from the layers around it.

Prerequisites

Docker Desktop on Windows, running on WSL2:

# PowerShell as administrator
wsl --install --no-distribution
wsl --update
winget install -e --id Docker.DockerDesktop
# restart the PC, start Docker Desktop, accept the terms

Docker Desktop is free for personal use and small businesses (under 250 employees and under USD 10 million revenue). Larger companies need a paid licence - check the current terms on docker.com.

Hardware:

• RAM. Through WSL2, Docker gets 50% of the host’s RAM by default - 16 GB on a 32 GB laptop, shared by all containers.
• GPU. Ollama in Docker on Windows only accelerates on NVIDIA/CUDA. Integrated Intel graphics are not used, so everything runs on the CPU.
• Disk. About 15 GB (Ollama image ~2 GB, model ~7 GB, Open WebUI ~4 GB).

Optional: raise the memory limit - only needed for larger models or when other heavy containers run alongside. Create or edit %UserProfile%\.wslconfig:

[wsl2]
memory=24GB

Then restart WSL and Docker Desktop:

wsl --shutdown
# then start Docker Desktop again

Step 1: Check that Docker is running

docker version

You should see both a Client and a Server section. If Server is missing, Docker Desktop is not running - start it and wait until the tray icon stops animating.

All containers share the same RAM, so check what else is running:

docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.CPUPerc}}"

If another container takes more than 6-8 GB, there won’t be room for the model. Stop it while you work:

docker stop <container_name>      # graceful stop, not a kill
docker start <container_name>

Step 2: Run Ollama

Ollama doesn’t need a separate installation - it comes as a Docker image.

docker pull ollama/ollama
docker run -d --name ollama --restart unless-stopped -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 ollama/ollama

Parameter	Meaning
-d	Detached - runs in the background
--name ollama	Name used to refer to the container instead of its ID
--restart unless-stopped	Starts again after a reboot, unless you stopped it yourself
-v ollama:/root/.ollama	Volume - downloaded models live outside the container and survive deletion and updates
-p 127.0.0.1:11434:11434	Port 11434 reachable from this computer only. Critical: Ollama has no authentication.

Check with docker ps - the ollama container must show Up.

Step 3: Download the model

docker exec -it ollama ollama pull mistral-nemo

docker exec -it ollama runs a command inside the running container; ollama pull downloads the model (~7 GB).

Why mistral-nemo and not mistral:

Tag	Size (Q4)	RAM needed	Context	Non-English languages
mistral (7B)	~4 GB	8 GB	32k	weak
mistral-nemo (12B)	~7 GB	16 GB	128k	decent
mistral-small3.2 (24B)	~15 GB	32 GB	128k	good, very slow on CPU

mistral-nemo is a reasonable compromise between quality (multilingual, long context) and hardware. The models are quantised (Q4) - shrunk at the cost of some precision, so they fit into ordinary RAM.

docker exec -it ollama ollama list               # downloaded models
docker exec -it ollama ollama rm <model_name>    # remove one

Step 4: Test from the command line

Chat:

docker exec -it ollama ollama run mistral-nemo "Explain in one sentence what an ERP system is."

Without the quoted text you get an interactive chat; leave it with /bye.

Speed:

docker exec -it ollama ollama run mistral-nemo --verbose "Explain the difference between an invoice and a credit note in three sentences."

--verbose prints statistics at the end. The number that matters is eval rate (tokens per second):

eval rate	Verdict
< 3 tok/s	For the patient
3-8 tok/s	Usable for testing
> 8 tok/s	Comfortable

The first question after start is slower - the model is still loading from disk into RAM. After 5 minutes of inactivity it is unloaded again.

API:

Invoke-RestMethod http://127.0.0.1:11434/api/generate -Method Post -ContentType "application/json" -Body '{"model":"mistral-nemo","prompt":"Hello","stream":false}' | Select-Object -ExpandProperty response

In PowerShell, curl is an alias for Invoke-WebRequest and mangles JSON in arguments - hence Invoke-RestMethod.

There is also an OpenAI-compatible endpoint at http://127.0.0.1:11434/v1. Any tool that speaks the OpenAI API can be pointed at the local model instead.

Step 5: Run Open WebUI

docker run -d --name open-webui --restart unless-stopped -p 127.0.0.1:3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

Parameter	Meaning
-p 127.0.0.1:3000:8080	The app listens on 8080 inside the container, reachable on localhost:3000 - again, locally only
-e OLLAMA_BASE_URL=…	This is what connects Open WebUI to Ollama. host.docker.internal is a special address through which a container sees the host’s ports.
-v open-webui:/app/backend/data	Volume for accounts, chat history and settings

How does Open WebUI know about Mistral? It doesn’t, in advance. On start it asks Ollama (GET /api/tags) which models are downloaded and lists them all. Every model you ollama pull later appears automatically.

The first start takes 1-3 minutes while helper models download:

docker logs -f --tail 20 open-webui

When you see Uvicorn running on http://0.0.0.0:8080, it’s ready. Ctrl+C stops following the log.

Step 6: First login

1. Open http://localhost:3000.
2. Sign up. The account is purely local, stored in the open-webui volume on your disk. The e-mail isn’t verified and nothing is sent anywhere.
3. The first account becomes administrator. Any later sign-up waits for admin approval.

Want no login at all? Add -e WEBUI_AUTH=False to the docker run command. It only works on a fresh install - once the first account exists, you can’t switch it on without deleting the volume and all its data.

Step 7: Settings that matter

Connection to Ollama. If mistral-nemo doesn’t appear in the model menu (top left): Admin Panel → Settings → Connections → Ollama API http://host.docker.internal:11434 → verify (it turns green) → Save.

Context window. Admin Panel → Settings → Models → mistral-nemo → Advanced Params:

Parameter	Value	Why
num_ctx	8192	Ollama’s default context window is small. Longer input (web results, documents) is silently truncated and the model answers from incomplete data. More than 8192 makes every answer much slower on a CPU.
Function Calling	Default	In Native mode, small models often ignore tools - including web search

“Arena Model” in the model list is not a model. It’s a feature for blind comparison of several models and is pointless with just one. Switch it off in Admin Panel → Settings → Evaluations.

Step 8: Web search

The model can’t see the web. Open WebUI searches for it, downloads the pages and puts their content into the prompt.

Admin Panel → Settings → Web Search:

Setting	Value	Why
Web Search	ON	Main switch
Web Search Engine	DDGS (DuckDuckGo)	No API key needed
Search Result Count	3	Fewer pages = faster, less risk of overflowing the context
Concurrent Requests	1	DuckDuckGo rate-limits parallel queries
Fetch URL Content Length Limit	4000	Otherwise one long page fills the whole context
Bypass Embedding and Retrieval	ON	Skips embedding the pages - the biggest time saver on a CPU
Timeout	15	One slow page won’t block the whole answer

Then Save (bottom right). Nothing applies until you do.

In the chat:

1. Start a new chat - an old one with wrong answers confuses the model with its own history.
2. Click the globe icon next to the message box so it turns blue. It usually resets to off in every new chat.
3. Ask, for example: “What is the temperature in Paris today? Cite your source.”

To use one specific page without searching, type #https://address.of.page - Open WebUI downloads it and attaches it as context.

How to tell it really searched:

Signal	Web search happened	Hallucination
Progress above the answer	“Searching the web…” / “Searched 3 sites”	nothing
Under the answer	clickable sources	none
docker logs --tail 30 open-webui	lines with web_search / ddgs	only POST /api/chat/completions

The main risk of local models: when they don’t have the data, they make it up with confidence, and the invented answer looks exactly like a true one. No sources, no trust.

A single question with web search on a CPU means generating a search query, searching, downloading pages and generating the answer. Expect 30 seconds to several minutes.

Daily operation

docker stop open-webui ollama        # frees RAM and CPU
docker start ollama open-webui
docker ps
docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.CPUPerc}}"

Idle, Ollama uses almost nothing. With mistral-nemo loaded it takes ~8 GB RAM.

Updating. Data lives in volumes, so an update means: pull the new image, remove the old container, create a new one with the same command.

docker pull ollama/ollama
docker rm -f ollama
docker run -d --name ollama --restart unless-stopped -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 ollama/ollama

docker pull ghcr.io/open-webui/open-webui:main
docker rm -f open-webui
docker run -d --name open-webui --restart unless-stopped -p 127.0.0.1:3000:8080 -e OLLAMA_BASE_URL=http://host.docker.internal:11434 -v open-webui:/app/backend/data ghcr.io/open-webui/open-webui:main

Models, accounts, chats and settings are kept.

Troubleshooting

Symptom	Likely cause	Fix
docker version shows no Server	Docker Desktop not running	Start Docker Desktop
localhost:3000 shows nothing	Open WebUI not running or still starting	docker ps -a --filter name=open-webui, then docker logs --tail 30 open-webui
Container in state Exited	Not enough RAM	docker stats, stop other containers, raise the limit in .wslconfig
Model missing from the menu	Wrong connection to Ollama	Step 7, Connection to Ollama
Answers take very long	CPU inference, first model load, web search	Wait. Test without the globe. Turn on Bypass Embedding.
No sources despite the globe	Settings not saved / globe grey / Function Calling = Native	Steps 7 and 8
ratelimit in the log	DuckDuckGo throttling	Concurrent Requests = 1, wait a moment
Answers in English when asked in another language	Small models lean towards English	Add “Answer in …” to the question, or set a System Prompt for the model

Security

• Ports are bound to 127.0.0.1 - not reachable from other machines on the network. Don’t change that to 0.0.0.0 without a reverse proxy with authentication.
• Ollama has no authentication. Whoever reaches port 11434 can run, download and delete models.
• Web search sends queries out. A local model is local only until you click the globe. Don’t put client or other sensitive data into searched questions.
• Docker Desktop is not a production platform. For a server, use Docker Engine on Linux.

Removing everything

docker rm -f open-webui ollama                                 # containers
docker volume rm open-webui ollama                             # data: models, accounts, chats - irreversible
docker rmi ghcr.io/open-webui/open-webui:main ollama/ollama    # images

Appendix: the same with Docker Compose

Instead of two long docker run commands, one docker-compose.yml:

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=10m        # how long to keep the model in RAM after the last request

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434   # in Compose, containers see each other by service name
    volumes:
      - open-webui:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama:
    name: ollama          # without "name", Compose adds the folder prefix and creates new, empty volumes
  open-webui:
    name: open-webui

docker compose up -d          # start everything
docker compose pull           # fetch new images
docker compose up -d          # restart with new versions
docker compose down           # stop (data in volumes stays)

If ollama and open-webui already exist from the manual docker run commands, remove them first (docker rm -f ollama open-webui). The volumes stay and Compose reuses them.