Ollama vs. llama.cpp on Ubuntu: Local LLM Setup and Performance

Ollama vs. llama.cpp on Ubuntu: Local LLM Setup and Performance

📋 Abstract: Running large language models locally on Ubuntu typically comes down to choosing between Ollama and llama.cpp. They operate at different layers: llama.cpp provides the underlying inference engine, while Ollama packages that capability with model management, APIs, and a service-oriented workflow.

The practical choice is straightforward: Ollama prioritizes convenience and automation, while llama.cpp provides finer control over inference and hardware utilization.

This guide covers both approaches, explains realistic performance expectations, and highlights an often-overlooked issue with local LLM deployments: context-window configuration. A model may advertise a large context capacity while the runtime uses a much smaller allocation, causing long prompts or documents to be truncated or rejected.

📋 Setting Realistic Expectations

#

Local LLM inference continues to improve, but first-time users may find that token generation is slower than the experience offered by hosted services such as ChatGPT or Claude.

That tradeoff comes with several important advantages:

1. Data privacy: Prompts and generated responses remain on the local machine.
2. Offline operation: Inference does not depend on an external network connection.
3. Predictable costs: There are no per-token API charges once the required hardware and software are in place.

For developers handling sensitive source code, private documents, internal data, or workloads that need to remain offline, these benefits can outweigh lower inference throughput.

⚙️ Engine vs. Wrapper: How Ollama and llama.cpp Relate

#

Ollama and llama.cpp are better understood as different layers of the same local-inference stack rather than direct alternatives.

+-------------------------------------------------------+
|                       Ollama                          |
|  User Interface, CLI, Model Management, REST API      |
+-------------------------------------------------------+
                         |
                         v
+-------------------------------------------------------+
|                      llama.cpp                        |
|  C/C++ Inference Engine, CPU/GPU Acceleration         |
+-------------------------------------------------------+

llama.cpp: The Inference Engine

#

llama.cpp handles the low-level work required to load supported model formats, execute inference, perform matrix operations, and distribute workloads across CPU and GPU resources.

It is designed for direct control rather than ease of installation. Users generally work with command-line binaries and explicitly configure parameters such as context length and GPU-layer offloading.

This makes it useful when squeezing maximum performance from a particular hardware configuration or testing new upstream inference capabilities.

Ollama: The Management Layer

#

Ollama packages local model inference into a more convenient service-oriented workflow.

It handles tasks such as:

• Downloading and managing models
• Starting and managing the inference service
• Providing a command-line interface
• Exposing HTTP APIs
• Simplifying model configuration
• Integrating with local applications and frontends

By default, Ollama exposes its service on port 11434, making it easy to connect local applications to a running model.

🦙 Getting Started with Ollama

#

1. Install Ollama

#

On Ubuntu, the standard installation command is:

curl -fsSL https://ollama.com/install.sh | sh

The installer configures Ollama as a system service, allowing the daemon to run in the background.

After installation, verify that the service is available:

ollama --version

2. Download and Run a Model

#

For example:

ollama run qwen3:8b

If the model is not already installed, Ollama downloads the required weights before starting an interactive session.

Use:

/bye

to exit the interactive session.

3. Use the OpenAI-Compatible API

#

Ollama can also be accessed through an OpenAI-compatible API endpoint:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3:8b","messages":[{"role":"user","content":"Hello!"}]}'

This makes Ollama particularly convenient for local AI frontends and applications that already support OpenAI-style APIs.

🧠 The Often-Overlooked Bottleneck: Context Windows

#

Model context length is separate from model size and GPU memory capacity.

A model may support a large context window, but the runtime still needs to allocate enough memory for the KV cache to actually process that amount of context.

Ollama uses conservative context settings in some configurations to reduce memory pressure and avoid out-of-memory failures. If the configured context is smaller than expected, long prompts and documents can be limited by the runtime rather than by the model itself.

Check the Active Context Allocation

#

With a model loaded, run:

ollama ps

Inspect the CONTEXT column to determine the context allocation currently being used.

Configure a Larger Global Context

#

For a system-wide Ollama configuration, create a systemd override:

sudo systemctl edit ollama.service

Add:

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=32768"

Then reload systemd and restart Ollama:

sudo systemctl daemon-reload
sudo systemctl restart ollama

A larger context should be selected according to available VRAM rather than simply set to the largest value supported by the model.

Configure Context Per Model

#

A Modelfile can define a model-specific context size:

FROM qwen3:8b
PARAMETER num_ctx 32768

Create the custom model:

ollama create qwen3-32k -f Modelfile
ollama run qwen3-32k

This approach is useful when different models or workloads require different context configurations.

Configure Context Per Request

#

For applications that need dynamic control, the API can specify num_ctx for an individual request:

curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [
    {
      "role": "user",
      "content": "Summarize this paper..."
    }
  ],
  "options": {
    "num_ctx": 16384
  }
}'

This allows applications to trade context capacity against memory consumption on a request-by-request basis.

Understand the VRAM Cost

#

The KV cache can become a significant memory consumer as context length increases.

For an 8B-class model, the approximate overhead can scale substantially:

Actual memory usage varies with the model architecture, precision, KV-cache configuration, and runtime implementation.

The key point is that increasing context length is not free. A model that fits comfortably at 8K tokens may require substantially more VRAM at 32K or 128K.

🚀 Squeezing Maximum Performance With llama.cpp

#

For users who need more control over inference, compiling llama.cpp directly exposes lower-level runtime options and GPU offloading controls.

It can also provide access to upstream performance improvements and newly supported features without waiting for those capabilities to appear in a higher-level package.

1. Clone and Compile llama.cpp

#

Clone the repository:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

For NVIDIA GPUs with CUDA:

cmake -B build -DGGML_CUDA=ON

For AMD GPUs using HIP/ROCm:

cmake -B build -DGGML_HIP=ON

Build the project:

cmake --build build --config Release -j

2. Start the Inference Server

#

A typical server configuration looks like this:

./build/bin/llama-server \
  -m models/model.gguf \
  -ngl 99 \
  -c 16384 \
  --port 8080

The exact command should be adjusted to match the model, GPU, and available system memory.

Important llama.cpp Flags

#

-ngl 99 — GPU layers

Controls how many model layers are offloaded to the GPU. The appropriate value depends on available VRAM; using an excessively high value does not guarantee that the entire model will fit.

-c 16384 — Context length

Sets the runtime context size to 16,384 tokens. Increasing this value increases KV-cache memory requirements.

The important distinction is that llama.cpp exposes these parameters directly, giving experienced users considerably more control over the inference configuration.

💻 Hardware Allocation and Performance Expectations

#

Local LLM performance depends on several variables, including model architecture, quantization, GPU compute capability, memory bandwidth, context length, and how much of the model can be kept in VRAM.

A rough hardware planning guide is:

These are planning ranges rather than hard limits. Quantization, architecture, context size, and partial CPU offloading can significantly change the practical result.

VRAM Capacity vs. Memory Bandwidth

#

A useful rule when evaluating hardware is:

VRAM capacity determines how large a model and context configuration can fit; memory bandwidth strongly influences token-generation throughput.

A GPU with enough VRAM but comparatively low memory bandwidth may fit a large model while still generating tokens relatively slowly.

Conversely, a high-bandwidth GPU can deliver substantially better generation performance when the workload remains predominantly GPU-resident.

Choosing a Quantization

#

For many GGUF-based deployments, Q4_K_M is a practical starting point because it provides a useful balance between model size, memory consumption, and output quality.

However, there is no universally optimal quantization. Higher-bit variants can preserve more quality at the cost of additional memory, while more aggressive quantization can make larger models fit within constrained VRAM.

📊 Ollama vs. llama.cpp: Which Workflow Fits?

#

Choose Ollama for Convenience

#

Ollama is a natural fit when the priority is getting a local model running quickly with minimal configuration.

It is particularly convenient for:

• Local AI applications
• Open WebUI-style frontends
• API-based development
• Rapid model experimentation
• Users who prefer automated model management

After installation, verify the active context configuration rather than assuming the model is using its maximum supported context length.

Choose llama.cpp for Fine-Grained Control

#

llama.cpp is better suited to workflows where runtime configuration and hardware efficiency are central concerns.

It provides direct control over parameters such as:

• GPU-layer offloading
• Context length
• Batch configuration
• CPU/GPU execution
• Model loading behavior
• Server configuration
• Quantized GGUF models

This makes it especially useful for developers tuning inference on dedicated GPUs, experimenting with new GGUF capabilities, or deploying models on headless systems where available VRAM must be managed carefully.

🔧 Practical Configuration Strategy

#

For most Ubuntu systems, a sensible progression is:

1. Start with Ollama to validate the model, hardware, and expected workload.
2. Check ollama ps to confirm the actual context allocation.
3. Increase context gradually rather than immediately selecting an extremely large value.
4. Monitor VRAM usage as context length increases.
5. Move to llama.cpp when you need more detailed control over GPU offloading or inference parameters.
6. Benchmark the exact workload instead of relying exclusively on theoretical model or GPU specifications.

The biggest mistake is treating model size as the only hardware constraint. A local LLM workload must account for model weights, KV cache, runtime overhead, GPU memory capacity, and memory bandwidth simultaneously.

🏁 Final Takeaway

#

Ollama and llama.cpp solve different parts of the local-LLM problem.

Ollama provides the streamlined experience: model management, a background service, CLI commands, and straightforward APIs. It is a practical starting point for developers who want local inference without manually managing every runtime component.

llama.cpp exposes the underlying inference machinery and gives experienced users substantially more control over GPU offloading, context configuration, and performance tuning.

Whichever runtime you choose, context length deserves the same attention as model size. A model can fit comfortably in VRAM while a larger context configuration pushes the KV cache beyond available memory. Understanding that relationship is essential for predictable local LLM performance on Ubuntu.