One RTX 3090 Is Reportedly Serving 64 Qwen3.8 Sessions at 1,035 Tokens/s
An NVIDIA RTX 3090 with 24GB of VRAM is reportedly delivering 1,035 tokens per second across 64 concurrent Qwen3.8-27B sessions.
Even more interesting, the GPU is operating with a power limit of just 250W.
The experiment uses HyperQwen, an open-source inference-serving stack built around vLLM, to turn a consumer graphics card into a multi-session AI server.
But the most important part isn’t simply the generation speed.
It’s the number of simultaneous sessions the system can reportedly handle.
Most local inference benchmarks focus on how quickly a model responds to one user. This experiment explores a different question:
How much combined generation capacity can a single consumer GPU deliver when dozens of users share the same model?
Let’s examine the setup.
1. The Hardware Behind the Experiment
The reported configuration is:
GPU: NVIDIA RTX 3090
VRAM: 24GB
Power limit: 250W
Model: Qwen3.8-27B
Inference stack: HyperQwen + vLLM
Concurrent sessions: 64
Reported single-session decode speed: Approximately 127 tokens/s
Aggregate output throughput: Approximately 1,035 tokens/s
Configured maximum context: Up to 150K tokens
Two figures immediately stand out: 127 tokens/s and 1,035 tokens/s.
However, they measure different things.
The approximately 127 tokens/s figure represents reported generation speed in a single-session test.
The 1,035 tokens/s figure represents the combined output throughput across multiple active sessions.
It does not mean every user receives 1,035 tokens per second. Nor does it mean all 64 users independently generate at 127 tokens/s.
The GPU’s compute resources and memory bandwidth are shared across requests. Individual performance depends on scheduling, prompt length, context size, output length and the number of competing sessions.
The achievement here is aggregate throughput, not individual user speed.
2. How Does One RTX 3090 Handle 64 Sessions?
Serving one AI conversation is relatively straightforward.
Serving 64 simultaneously is a much more complicated engineering problem.
The inference engine has to coordinate incoming requests, manage memory for active conversations, schedule generation and keep the GPU busy without exceeding its available resources
This is where vLLM becomes important
Two of its key technologies are particularly relevant: Continuous Batching and PagedAttention
Continuous Batching: Keeping the GPU Busy
Imagine 64 users sending requests to the same AI server
They won’t all receive responses of equal length
One user might ask a simple question that requires 30 tokens. Another might request a detailed coding solution requiring hundreds or thousands of tokens
With traditional static batching, requests are grouped together and processed as a fixed batch. When some requests finish earlier than others, processing capacity can go unused while the remaining requests continue
Continuous batching takes a more dynamic approach
Instead of waiting for every request in the batch to finish, the inference engine updates the active batch as requests complete
When one conversation finishes generating, another waiting request can take its place
This keeps more requests moving through the GPU and reduces the idle time associated with waiting for longer generations to finish
For a server handling dozens of users, this can make a substantial difference to overall throughput
PagedAttention: Making Better Use of 24GB VRAM
The second challenge is memory management
Every active transformer session requires a Key-Value cache, commonly called the KV cache
The cache stores attention information from previously processed tokens. This allows the model to reuse information from earlier parts of a conversation instead of repeatedly recomputing everything
As the context grows, the KV cache consumes more memory
Now consider managing 64 active conversations on a GPU with only 24GB of VRAM