On-premise LLM hardware: how many staff can one GPU serve?
How to size an on-premise LLM server: GPU memory for the model and the KV cache, active vs total users, and a vLLM load test you can run on one rented GPU.
Sizing on-premise LLM hardware starts with two numbers: how many requests run at the same moment, and how much GPU memory each one needs.
In short:
- Size the server by the number of requests running at the same moment. If 50 staff each send a request every 5 minutes in the busiest hour, and each request takes 30 seconds, about 5 run at once on average and 11 or fewer run at once 99% of the time.
- GPU memory holds two things: the model’s weights and the KV cache, which is the working memory for every open conversation. Qwen3-8B’s weights take 16.4 GB, and each 8,192-token conversation adds about 1.2 GB.
- On one 48 GB card, Qwen3-8B leaves room for about 22 full-length 8,192-token conversations. The larger Qwen3-32B, stored at 8 bits, fits about 4 on the same card and about 26 on a 96 GB card.
- Long documents cut those numbers by four, because a 32,768-token document needs four times the KV cache of an 8,192-token chat.
- Memory sets how many conversations fit. How long people wait depends on the GPU’s speed, which you have to measure. Before you buy, rent the same GPU for an afternoon and run
vllm bench serveat rising concurrency. An NVIDIA L40S costs $1.09 an hour on RunPod’s Secure Cloud.
How many people use it at once?
A company with 50 staff doesn’t have 50 people pressing enter in the same second. The number that sizes a server is requests in flight: how many questions it is working on at one moment.
Little’s law, a basic rule from queueing theory, gives a first estimate:
requests in flight = requests per second × seconds each request takes
The assumptions in this example are ours, so change them to fit your team. In the busiest hour, each person sends one request every 5 minutes (12 an hour). Each request keeps the server busy for 30 seconds, long enough to read a long email or a few pages and write several paragraphs back.
| Staff | Requests in the busiest hour | In flight on average | 99% of the time, at most |
|---|---|---|---|
| 10 | 120 | 1 | 4 |
| 25 | 300 | 2.5 | 7 |
| 50 | 600 | 5 | 11 |
The last column assumes requests arrive at random (a Poisson distribution). Real offices are burstier. After a Monday meeting, everyone may ask for the same summary within a minute. Treat the table as a starting point, and once a pilot server is running, read the real numbers from its logs.
Total users matter elsewhere. Accounts, permissions and chat history live in the chat front end your staff log in to. We compared two of those in Open WebUI vs LibreChat.
What uses GPU memory: the model and the KV cache
Two things fill the card’s memory. The first is the weights, which are the model itself. They have a fixed size and load once. File sizes come from the model pages on Hugging Face:
| Model | Parameters | Weights at 16 bits | Weights at 8 bits (FP8) |
|---|---|---|---|
| Qwen3-8B | 8.2 billion | 16.4 GB | 9.4 GB |
| Qwen3-32B | 32.8 billion | 65.5 GB | 34.3 GB |
We use Qwen3 for the examples because both sizes use the same standard attention design, which keeps the arithmetic simple, and the Apache 2.0 licence allows commercial use.
The second is the KV cache. For every token in a conversation, the model keeps a “key” and a “value” at every layer so it doesn’t have to recompute them for each new word. Per token, that is:
2 (key and value) × layers × KV heads × head size × 2 bytes
The numbers come from each model’s config.json. Qwen3-8B has 36 layers, 8 KV heads and a head size of 128, which gives 147,456 bytes per token. Qwen3-32B has 64 layers with the same heads, which gives 262,144 bytes.
| Model | Per token | One 8,192-token conversation | One 32,768-token document |
|---|---|---|---|
| Qwen3-8B | 0.15 MB | 1.2 GB | 4.8 GB |
| Qwen3-32B | 0.26 MB | 2.1 GB | 8.6 GB |
These are worst cases. vLLM’s PagedAttention design hands out KV cache memory in small blocks as a conversation grows, so a two-line question doesn’t reserve room for 8,192 tokens.
How many conversations fit on one GPU?
The KV cache gets what is left after the weights and vLLM’s own working memory. For the table below we set aside 5 GB of working memory. That figure is our assumption. When vLLM starts, it prints the real numbers, in this format (example from vLLM’s docs):
GPU KV cache size: 15,728,640 tokens
Maximum concurrency for 8,192 tokens per request: 1920The table covers two cards: the NVIDIA L40S with 48 GB and the RTX PRO 6000 Blackwell with 96 GB.
| GPU | Model | Left for KV cache | Full 8,192-token conversations | Full 32,768-token documents |
|---|---|---|---|---|
| 48 GB | Qwen3-8B, 16-bit | 26.6 GB | 22 | 5 |
| 48 GB | Qwen3-32B, FP8 | 8.7 GB | 4 | 1 |
| 96 GB | Qwen3-8B, 16-bit | 74.6 GB | 61 | 15 |
| 96 GB | Qwen3-32B, FP8 | 56.7 GB | 26 | 6 |
| 96 GB | Qwen3-32B, 16-bit | 25.5 GB | 11 | 2 |
vLLM can also store the KV cache itself at 8 bits with --kv-cache-dtype fp8. That halves the size per token and roughly doubles the counts above. Check answer quality on your own tasks with it switched on.
How do you measure wait times before you buy?
Memory decides how many conversations fit. How long each person waits depends on the card’s compute and memory speed: 864 GB/s on the L40S and 1,792 GB/s on the RTX PRO 6000. You can’t calculate that reliably, so measure it. vLLM’s benchmark tool reports the two numbers your staff will notice:
- Time to first token (TTFT), the wait from pressing enter to the first word appearing.
- Time per output token (TPOT), which sets how fast the rest of the answer streams for each person.
Rent the exact card you plan to buy. On RunPod’s pricing page, checked on 12 November 2026, an L40S costs $1.09 an hour and an RTX PRO 6000 costs $2.49 an hour on Secure Cloud, billed by the second. Start the model:
vllm serve Qwen/Qwen3-8B --max-model-len 16384Then, from a second terminal, step up the number of simultaneous requests:
for c in 1 2 4 8 16; do
vllm bench serve --model Qwen/Qwen3-8B \
--dataset-name random --random-input-len 1500 --random-output-len 300 \
--num-prompts $((c * 20)) --max-concurrency $c --seed $c \
--percentile-metrics ttft,tpot,e2el --metric-percentiles 50,99 \
--save-result --result-filename "qwen3-8b-c$c.json"
done1,500 input tokens is roughly a long email thread or a couple of pages, and 300 output tokens is a few paragraphs. Each run gets its own --seed because vLLM’s docs warn that repeat runs can reuse prompts left in the cache and look faster than real use.
Before you start, decide what wait is acceptable, for example the first word within two seconds for 99 requests out of 100. The highest concurrency level where P99 TTFT stays under your limit is the capacity of that card for that model. Repeat the loop for the second model and with --random-input-len 30000 for long documents. Two models at two prompt lengths fit in an afternoon. Four hours costs $4.36 on the L40S or $9.96 on the RTX PRO 6000 at those rates.
Long documents: context length, prefix caching and preemption
Every token a person pastes in sits in the KV cache until the answer is finished, so a 30-page contract costs about four times the memory of a normal chat. It also raises time to first token, because the model reads the whole document before it writes anything.
Three settings help:
- Cap the context length. Qwen3 handles 32,768 tokens natively and up to 131,072 with an extension called YaRN, according to its model card. If nobody pastes more than 16,000 tokens, set
--max-model-len 16384. vLLM’s memory guide lists a lower context length as a way to cut memory use. - Reuse shared text. When many questions share the same long instructions or the same document, automatic prefix caching lets vLLM reuse the part it has already read.
- Watch for preemption. When the KV cache fills, vLLM pauses some requests and recomputes them later, and its tuning guide notes that this hurts latency. Repeated preemption warnings in the log mean the card is short of KV cache memory for the load.
On-premise LLM hardware for 10, 25 and 50 staff
Put the “99% of the time” estimate from the first section next to the memory figures, with 16-bit Qwen3-8B and 8-bit Qwen3-32B, and you get the smallest single card where that many full-length conversations fit.
| Staff | Plan for | Qwen3-8B, everyday chats | Qwen3-8B, long documents | Qwen3-32B, everyday chats | Qwen3-32B, long documents |
|---|---|---|---|---|---|
| 10 | 4 at once | 48 GB | 48 GB | 48 GB, no headroom | 96 GB |
| 25 | 7 at once | 48 GB | 96 GB, or 48 GB with FP8 KV cache | 96 GB | 96 GB with FP8 KV cache |
| 50 | 11 at once | 48 GB | 96 GB | 96 GB | 96 GB with FP8 KV cache, or two cards |
“Everyday chats” means up to 8,192 tokens, and “long documents” means up to 32,768. Read the table as a shortlist of cards to rent and test. The benchmark then tells you whether the wait at that concurrency is acceptable.
What a rented GPU does not tell you: cooling, power and noise in an office
A card in a data centre has the building’s cooling and power behind it. A card under a desk doesn’t, and a rented benchmark can’t show the difference.
NVIDIA lists the L40S as passively cooled. It has no fans of its own and relies on a server chassis to push air through it, so it belongs in a rack server. The RTX PRO 6000 Workstation Edition has its own fans, and so does the Max-Q version, which has the same 96 GB.
Power draw goes up to 350 W for the L40S, 600 W for the RTX PRO 6000 Workstation Edition and 300 W for the Max-Q. Size the power supply, the UPS and the circuit for the whole machine under full load.
We haven’t measured noise or heat for an office build. A machine running flat out all day warms a small room and is audible, so decide where it will live before you order.
Then there is the work a cloud provider usually does for you: physical security, backups, driver and model updates, and someone to restart the service when it hangs.
Ollama or vLLM for a team server?
Ollama is a simple way to run a model on one machine. By default it processes one request per model at a time (OLLAMA_NUM_PARALLEL is 1), and it queues the rest, up to 512, before it starts returning errors. You can raise the parallel setting, but Ollama’s docs say the memory needed grows with the parallel count times the context length.
vLLM is built to serve many users from one GPU. It batches incoming requests continuously and manages the KV cache in blocks, which is why the sums above work. It takes more setup.
For one or two people, Ollama is fine, and at that scale a CPU can be enough: we ran a 4B model on a 2019 Core i9. For a team sharing one GPU, use vLLM.
Before you buy hardware
- Estimate requests in flight for your busiest hour, or read them from a pilot’s logs.
- Find the longest document staff will paste in, and set the context length from that.
- Pick the model on answer quality first, using 20 to 50 of your own real tasks.
- Add up the memory: weights, plus KV cache for your “plan for” number, plus working memory.
- Rent that exact card, run
vllm bench serve, and note the highest concurrency that meets your wait limit. - Check power, cooling, noise and where the machine will sit.
- Decide who patches it, monitors it and restarts it.
Our self-hosted AI pilot follows these steps for one workflow. If you’re not sure a private model is worth it yet, start with the free workflow check.