Local AI on a VPS without a GPU: which models work and how fast (we measured it)

On an ordinary VPS with no graphics card, you can run language models of up to about 4 billion parameters at a usable speed: on ours, with 3 vCPUs from a Ryzen 9 5900X and 12 GB of RAM, six models generated between 6.7 and 24.9 tokens per second. They’re good for classifying, extracting data or summarizing short texts. They’re not good for reasoning, for writing code you don’t review, or for replacing a large paid model: of the five we asked, only one wrote a simple Python function correctly on the first try, and the smallest one got a basic concept wrong.

We measured this on October 17, 2026 with llama.cpp, build b11714, on the same Windows Server 2022 VPS that runs our assistant. Here are the numbers, the answers exactly as they came out, how to reproduce it and which tasks make it worthwhile.

The machine and how we measured#

• CPU: AMD Ryzen 9 5900X, but the VPS only sees 3 vCPUs. llama.cpp loaded its CPU backend for AVX2 instructions (“haswell”).
• RAM: 12 GB, of which about 6.5 GB were free with the rest of the services running.
• GPU: none. The display adapter is the generic Windows one.
• Free disk: between 1.4 and 8 GB depending on the moment. This dictated what we could test: one model at a time, download it, measure it and delete it.

For each model we did two things:

1. llama-bench -t 3 -p 512 -n 128 -r 3: processes a 512-token prompt (pp512) and generates 128 tokens (tg128), three times, and reports the mean with its deviation.
2. Two real questions with llama-cli, low temperature and reasoning mode turned off (-rea off): an explanation in Spanish for a non-technical person and a Python function.

All models in Q4_K_M quantization, the usual sweet spot: according to llama.cpp’s official table for Llama 3 8B, it worsens perplexity by 2.8% compared to the uncompressed model and takes up less than a third of the space.

The results#

Gemma 4 E2B’s high generation deviation is because another process was using the CPU during that measurement; in its two answers with llama-cli, with that process gone, it generated at 11.3 and 11.8 tokens per second.

Desliza para ver la figura completa

To put those numbers in tangible terms: a 300-word answer in Spanish is about 400 tokens. With the 2B it takes around half a minute to write out; with Qwen3.5 4B, close to a minute. Add the time it takes to read your question, which you do notice with long prompts: a 4,000-token document means more than a minute and a half of processing with a 3-4B model before the first word comes out.

A naming detail that trips people up: Gemma 4 E2B is not a 2-billion-parameter model. The “E” stands for “effective”: llama.cpp counts 4.65 billion, and the file is bigger than Qwen3.5 4B’s. Always look at the file size, not the name.

What they answered#

We asked: “Explain in three sentences, in Spanish and for someone non-technical, what quantizing a language model is and what you lose with it.”

• Qwen3.5 0.8B: “Quantization is a process where a language model reduces the number of words it can generate…”. That’s false. Quantization reduces the precision of the numbers that make up the model, not its vocabulary. It sounds right and it’s wrong, which is the worst kind of error.
• Qwen3.5 2B: “…reduces the number of bits of information in which a language model’s calculations are stored, making it run faster and use less memory. In doing so, the model loses a small amount of mathematical precision…”. Correct and clear.
• Llama 3.2 3B: talked about “reducing complexity”, “discrete values” and losing “the ability to capture continuous information”. Vague, more jargon than explanation.
• Ministral 3 3B: explained the part about reducing the precision of the numbers well, but added that the model can become “slower”. It’s the opposite: quantizing makes it faster.
• Gemma 4 E2B: “…it’s done by reducing the precision of the numbers the model uses to process language. As a result, some of the subtlety is lost…”. Correct.
• Qwen3.5 4B: “Quantization is like going from a high-definition photo to a compressed, lower-quality version so it takes up less space and loads faster. In this process, the numbers the model uses are rounded…”. The best of the six: accurate, with a comparison anyone can follow.

And the Python function (take a list of paths and return a dictionary with the total size per extension, ignoring paths that don’t exist):

• Qwen3.5 0.8B wrote a function that takes a single path and returns its size, miscalculated. Useless.
• Qwen3.5 4B imported os and summed correctly, but wrote _, _, extensión = os.path.splitext(ruta): it unpacks three values from a function that returns two, so it fails on the first file that exists.
• Ministral 3 3B delivered a function that works, but skips files without an extension and, in that case, looks for an “extension” in the folder name. Nobody asked for that.
• Gemma 4 E2B was the only one that got it right on the first try: it imports os, checks that the path exists, lowercases the extension and sums by extension. It also added a usage example we didn’t ask for, which got cut off when it hit the 400-token limit.

The 2B case is the most instructive, because the code looks correct at first glance:

def calcular_tamano_rutas(rutas):
    diccionario = {}
    for ruta in rutas:
        try:
            if os.path.exists(ruta):
                tamaño = os.path.getsize(ruta)
                ext = os.path.splitext(ruta)[1]
                diccionario[ext] = tamaño
        except (OSError, IOError):
            pass
    return diccionario

It looks correct and has two bugs: import os is missing, so it doesn’t run, and diccionario[ext] = tamaño overwrites instead of adding, so with two .txt files it only counts the last one. We ran it twice separately and it made the same two mistakes both times.

Why it runs at this speed#

There are two phases, and they depend on different things:

• Reading the prompt (pp) is pure computation: it multiplies matrices with many tokens at once. It depends on how many cores you have and their vector instructions. That’s why you get dozens of tokens per second here.
• Generating (tg) produces one token at a time and, for each one, has to read all of the model’s weights from memory. The limit is memory bandwidth, not the CPU. The rough rule: tokens per second ≈ bandwidth ÷ model size.

That’s why going from a 0.53 GB file to a 2.02 GB one cuts generation speed by almost a factor of three (from 24.9 to 8.9 tokens per second), and why on a VPS you share both the cores and the bandwidth with the other customers on the physical server. For reference, Corsair estimates that a desktop PC with dual-channel DDR5-5600 gets around 90 GB/s, which leaves a theoretical ceiling of about 18 tokens per second with an 8-billion-parameter model in Q4_K_M.

Other public CPU-only measurements, to put ours in context:

Our VPS is on par with a mini PC with an Intel N150 running the same model. Three virtual cores from a desktop processor are not a modern laptop with all its cores to itself.

The exception: MoE models if you have lots of RAM#

The last row of that table is the interesting one for a server without a GPU. Mixture-of-experts (MoE) models, like Qwen3-30B-A3B or gpt-oss-20b, have many parameters but only activate part of them for each token: about 3 billion in Qwen3-30B-A3B and 3.6 billion in gpt-oss-20b. The whole thing has to fit in memory, but generation speed is similar to that of a small dense model.

On the Radxa board, CPU-only, Qwen3-30B-A3B generates at 15-16.8 tokens per second, more than a 14-billion-parameter dense model on a faster laptop. With a model like that you get the quality of a mid-size model at the speed of a small one, as long as the VPS has the RAM:

• Qwen3-30B-A3B in Q4_K_M is an 18.63 GB file.
• gpt-oss-20b needs 14.9 GB in total with 8,000 tokens of context, according to llama.cpp’s measurements.

They don’t fit on our 12 GB VPS, so we haven’t been able to verify this. On a 32 GB one, it’s the first test we’d run.

How much RAM you need#

The memory you need is roughly the size of the model file, plus the KV cache (which grows with context), plus some headroom for the program and the system. Hugging Face gives the reference for Llama 3.1 8B: the cache takes about 0.125 GB with 1,000 tokens of context and almost 2 GB with 16,000.

Always leave headroom for whatever is already running on the server. If the VPS starts using the swap file, speed collapses.

How to reproduce it on your VPS#

On Linux it’s the same as on Windows, just with a different binary.

1. Download prebuilt llama.cpp from the GitHub releases page (on Windows, the CPU x64 zip; on Linux, the equivalent or brew install llama.cpp). No installation needed.
2. Check your free disk space before downloading anything. A 3B in Q4_K_M is 2 GB.
3. Download the model in GGUF from Hugging Face:

   curl -L -o gemma-4-e2b.gguf \
     https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/resolve/main/gemma-4-E2B-it-Q4_K_M.gguf

4. Measure, with as many threads as you have vCPUs:

   llama-bench -m gemma-4-e2b.gguf -t 3 -p 512 -n 128 -r 3

5. Test it with your real tasks, not generic questions (-rea off turns off reasoning on models that have it):

   llama-cli -m gemma-4-e2b.gguf -t 3 -c 4096 -st -rea off --temp 0.2 \
     -p "Classify this email as invoice, support or spam: ..."

6. Serve it as an API on the local interface only:

   llama-server -m gemma-4-e2b.gguf -t 3 -c 4096 --host 127.0.0.1 --port 8080

   From your computer, reach it through an SSH tunnel (ssh -L 8080:127.0.0.1:8080 tu-vps). Don’t open that port to the internet: without --api-key, anyone who can reach the port can use it.

What it’s actually good for (and what it isn’t)#

Tasks where a small model on CPU makes sense:

• Classifying emails, tickets or comments into a closed list of categories.
• Extracting fields from text into JSON (name, date, amount) when you validate the result.
• Summarizing short texts for internal use.
• Data that can’t leave your server for privacy or contractual reasons.
• Transcribing audio with whisper.cpp, which also runs on CPU. We didn’t measure it on this VPS; its documentation says about 3.9 GB of RAM for the large model.

Where it’s not worth it:

• Coding: you saw it with the Python function. We compare the coding models that are worth it (and that need a GPU or a Mac with lots of memory) in local models for coding vs. Claude.
• Assistants and agents that chain tools, need long context and must rarely make mistakes. Our server assistant uses Claude for that reason; we explain it in how we automate servers with AI.
• Anything customer-facing without human review.

What about cost?#

If you’re already paying for the VPS, running the model costs you no extra money, but it does cost CPU: during generation it takes every core you give it. Compare that with Anthropic’s cheapest API, Claude Haiku 5.5, at $0.10 per million input tokens and $0.50 per million output tokens (with prompts of up to 100,000 tokens). Classifying 10,000 emails of 300 tokens with a 10-token answer is about 3 million input tokens: about $0.35 with the API, versus 20 to 25 hours of VPS CPU at 100% just to read those texts with a 3-4B model, based on our prompt-processing speeds. For low volume and data that can leave the server, the API wins. Local AI on a VPS without a GPU pays off when the data must not leave or when you already have the machine sitting idle.

What to do now#

1. Check how much free RAM your VPS really has, with its services running.
2. Try Gemma 4 E2B or Qwen3.5 4B with the steps above and your own tasks (if the RAM isn’t enough, Qwen3.5 2B): ten real examples tell you more than any benchmark.
3. If quality falls short and you have 32 GB or more, try an MoE model before thinking about a GPU.
4. If you’re torn between tools, read Ollama, llama.cpp or LM Studio. For a server without a GPU, we use llama.cpp.
5. To pick a model for a different kind of machine, see the best local AI models for your hardware.