vLLM vs Ollama: Production Serving Throughput and VRAM Benchmarks on Linux

Why Serving Architecture Matters for Local LLMs

While Ollama has popularized local AI models with its intuitive CLI interface, production workloads requiring concurrent request handling demand specialized serving runtimes. vLLM uses PagedAttention and dynamic batching to deliver up to 3x higher throughput under multi-client loads.

Test Setup & Methodology

Our benchmark was executed on a clean installation of Ubuntu 24.04 LTS equipped with an Intel Core i7-14700K, 64GB DDR5 RAM, and an NVIDIA GeForce RTX 4070 (12GB VRAM). We measured time-to-first-token (TTFT) and sustained tokens per second across concurrency levels ranging from 1 to 20 simultaneous connections.

Installing vLLM with CUDA on Ubuntu

Deploy vLLM in an isolated Python virtual environment:

python3 -m venv ~/vllm-env
source ~/vllm-env/bin/activate
pip install uv
uv pip install vllm

Running the Benchmark Service

Launch the OpenAI-compatible vLLM server with optimal GPU memory utilization:

vllm serve meta-llama/Meta-Llama-3-8B-Instruct --gpu-memory-utilization 0.90 --max-model-len 4096 --port 8000

Benchmark Results Summary

• Single-User Sequential: Ollama: 48 tokens/sec | vLLM: 52 tokens/sec
• 10 Concurrent Clients: Ollama: 48 tokens/sec (queued) | vLLM: 142 tokens/sec (batched)
• VRAM Overhead: Ollama: 5.8 GB | vLLM: 10.8 GB (allocated for KV cache)