Qwen3.8–27B, 177 Tokens Per Second, One RTX 3090
A few days ago I published a result I was pretty proud of: Qwen3.8–27B running at a max of 100 tokens per second on a single RTX 3090 Ti. That number felt like the ceiling of what one consumer card could give me for this model.
Today I got something even crazier.
177 tokens per second. One RTX 3090. No Ti — just a 3090. And it’s power-capped at 300 watts.
| Config | Context | Parallel requests | My measured decode speed |
|------------|-----------|-------------------|--------------------------|
| CTX=fast | 65,536 | 8 | ~177 tok/s |
| CTX=long | 131,072 | 4 | ~122 tok/s || Config | Context | Parallel requests | My measured decode speed |
|------------|-----------|-------------------|--------------------------|
| CTX=fast | 65,536 | 8 | ~177 tok/s |
| CTX=long | 131,072 | 4 | ~122 tok/s |I wired it into DeepSeek Harness as my local agent backend, and it runs perfect. Same OpenAI-compatible API the harness already speaks — http://my-box:18020/v1, a bearer key, done. Subagents fire off requests in parallel, the model answers at speeds that used to need a datacenter.
Here is how I got it working on my Linux box with one RTX 3090, why it’s so fast, and — because honesty sells better than hype — exactly where it falls down.
The repo: syv-ai/qwen38–27b-rtx3090
The whole thing is built on a GitHub repo called qwen38–27b-rtx3090. It’s not a new model and it’s not a new engine — it’s a pinned fork of vLLM (version 0.28.0) plus a stack of patches, requantization scripts, and launch profiles that squeeze this specific hybrid model onto one 24 GB card.
The repo ships a Docker image, and if you love Docker, great — docker compose --profile single up -d really is the whole install. But I don’t want another container eating my disk and hiding its logs. This article does it natively, in a plain Python venv. The repo supports this (“or by hand in a venv” — same steps, no containers).
One honest note before we start. Speed numbers on this stack depend heavily on what you ask the model to do. Code generation runs fast; free-form prose runs slower; quoting a document back verbatim runs absurdly fast (I’ll explain why later). The repo’s own documentation puts it perfectly: “A tok/s figure for this stack is meaningless without the prompt that produced it.”
My 177 is real — measured on my box, with my agent workloads. Your mileage will vary by workload, not by luck.
Setting it up (no Docker required)
What you need before starting:
- A Linux box (or WSL2) with an RTX 3090 — any trim, 24 GB is the magic number
- A recent NVIDIA driver (
nvidia-smishould just work) - Python 3.12 or newer, plus its dev headers
- A CUDA 13 toolkit — at minimum its headers (there’s a gotcha here, covered in Step 2)
- Roughly 60 GB of free disk for models and caches
And here’s my real advice on prerequisites: don’t memorize apt package lists — hand the install to an AI agent. Claude Code, Codex, OpenCode, anything with shell access on your machine. No copy-pasting involved: point it at the repo and say “get this running on my 3090.” When a step fails, the agent sees the error itself and fixes it right there — whether that means apt install python3.12-dev, a newer driver, libssl, or a different CUDA toolkit. Missing packages are no longer a research project; they’re one line of work for your agent. I did exactly that myself, and it saved me from an afternoon of forum archaeology.
The steps — exactly as I ran them on my box:
Step 1 — Clone the repo.
git clone https://github.com/syv-ai/qwen38-27b-rtx3090
cd qwen38-27b-rtx3090git clone https://github.com/syv-ai/qwen38-27b-rtx3090
cd qwen38-27b-rtx3090Step 2 — Create a clean venv and install the pinned vLLM. The repo pins vLLM 0.28.0 because that’s the version where DFlash2 speculation became native upstream (it landed as PR #52816). Install exactly this:
python3.12 -m venv venv
venv/bin/pip install -U pip
venv/bin/pip install vllm==0.28.0 huggingface_hub hf_transfer ninja
flashinfer-python flashinfer-cubin==0.6.13 pandaspython3.12 -m venv venv
venv/bin/pip install -U pip
venv/bin/pip install vllm==0.28.0 huggingface_hub hf_transfer ninja
flashinfer-python flashinfer-cubin==0.6.13 pandasTwo gotchas here. First, don’t let pip “fix” flashinfer-python to a different version — the launcher exports FLASHINFER_DISABLE_VERSION_CHECK=1 and the pinned pair is what the patch stack was tested against. Second, and this one bit me: vLLM’s DFlash2 sampling path JIT-compiles a FlashInfer kernel that #includes curand.h. If your CUDA toolkit doesn’t ship the curand headers, that kernel build fails at first boot and the server silently falls back to a slower path — correct output, ~5% slower, nothing in the logs. On Ubuntu with CUDA 13 it’s one package: sudo apt install libcurand-dev-13-0.
Step 3 — Download the model. The base checkpoint is dbirks/Qwen3.8-27B-W4A16-AutoRound (~19.5 GB):
HF_XET_HIGH_PERFORMANCE=1 venv/bin/hf download
dbirks/Qwen3.8-27B-W4A16-AutoRound
--local-dir models/Qwen3.8-27B-W4A16-AutoRoundHF_XET_HIGH_PERFORMANCE=1 venv/bin/hf download
dbirks/Qwen3.8-27B-W4A16-AutoRound
--local-dir models/Qwen3.8-27B-W4A16-AutoRoundStep 4 — Prep the model. The prepare/ scripts do surgery: requantizing the two embedding matrices nobody else quantized, rebuilding the MTP draft head over a smarter vocabulary, then fetching the fast variant and the DFlash2 drafter (1.2 GB). All CPU-only, a few minutes each:
M=models/Qwen3.8-27B-W4A16-AutoRound
venv/bin/python prepare/quant_lm_head.py $M
venv/bin/python prepare/quant_embed.py $M
venv/bin/python prepare/quant_mtp.py $M
venv/bin/python prepare/build_draft_vocab.py $M --ids prepare/draft_vocab_ids.json
venv/bin/python prepare/fetch_fast_variant.py
venv/bin/python prepare/fetch_dflash2.pyM=models/Qwen3.8-27B-W4A16-AutoRound
venv/bin/python prepare/quant_lm_head.py $M
venv/bin/python prepare/quant_embed.py $M
venv/bin/python prepare/quant_mtp.py $M
venv/bin/python prepare/build_draft_vocab.py $M --ids prepare/draft_vocab_ids.json
venv/bin/python prepare/fetch_fast_variant.py
venv/bin/python prepare/fetch_dflash2.pyStep 5 — Apply the patch stack. The patches/ directory holds about fifteen .patch files — embedding-quant wiring, split-KV verify attention, Marlin int8 fixes, DFlash2 lookup drafting, and more. patches/series lists them in order; apply the whole list into the venv’s vLLM, skipping dflash2-backport.patch (DFlash2 is native in 0.28.0 — that patch exists for older vLLMs):
<button data-code="sed -e 's/#.*//' -e 's/^[[:space:]]*//;s/[[:space:]]*$//' -e '/^$/d' patches/series |
while IFS= read -r name; do
[ "$name" = "dflash2-backport.patch" ] && continue
patch -p1 -d venv/lib/python3.12/site-packages/vllm
sed -e 's/#.*//' -e 's/^[[:space:]]*//;s/[[:space:]]*$//' -e '/^$/d' patches/series |
while IFS= read -r name; do
[ "$name" = "dflash2-backport.patch" ] && continue
patch -p1 -d venv/lib/python3.12/site-packages/vllm < "patches/$name"
donesed -e 's/#.*//' -e 's/^[[:space:]]*//;s/[[:space:]]*$//' -e '/^$/d' patches/series |
while IFS= read -r name; do
[ "$name" = "dflash2-backport.patch" ] && continue
patch -p1 -d venv/lib/python3.12/site-packages/vllm < "patches/$name"
doneIf a patch refuses to apply, that’s almost always a version mismatch — check you’re on exactly 0.28.0 with venv/bin/pip show vllm.
Step 6 — Verify before you launch. The repo gates everything behind one script:
api_key.txt
bash verify.sh –no-server # checks all patches applied + model shape correct” data-toggle-ms=”1200″ aria-label=”Copy code” data-nosnippet=”” class=”code-copy-btn absolute right-3 top-1/2 -translate-y-1/2 size-8 p-1.5 flex items-center justify-center bg-black/50 text-white rounded-md transition-colors duration-200 cursor-pointer hover:bg-black/70″>
openssl rand -hex 24 > api_key.txt
bash verify.sh --no-server # checks all patches applied + model shape correctopenssl rand -hex 24 > api_key.txt
bash verify.sh --no-server # checks all patches applied + model shape correctIf it comes back clean, your install is byte-for-byte what the maintainers benchmark. That’s a rare luxury in self-hosted AI.
Step 7 — Launch. The whole configuration surface is environment variables:
CUDA_VISIBLE_DEVICES=0 SPEC=dflash2 PREFIX_CACHE=1 bash single-user/start_qwen.sh
# add CTX=long for ~131k context (4 slots), or CTX=huge after kvarn/install.sh for 245kCUDA_VISIBLE_DEVICES=0 SPEC=dflash2 PREFIX_CACHE=1 bash single-user/start_qwen.sh
# add CTX=long for ~131k context (4 slots), or CTX=huge after kvarn/install.sh for 245kPin CUDA_VISIBLE_DEVICES — vLLM happily picks the emptiest GPU on the box, which on a multi-card machine might not be the one you meant. The server reads the bearer key from api_key.txt (or VLLM_API_KEY) before the port is reachable by anything but your own machine — without it, the server has no authentication at all. Everything else reads from .env. On my box: same command, and within a couple of minutes I had an OpenAI-compatible endpoint speaking for a 27-billion-parameter model on one card that costs less than a used bicycle.
Why is this fork so fast?
Stock vLLM on stock quantizations leaves speed on the table in places you wouldn’t guess. The repo’s optimizations doc lists nine of them. Here are the four that matter most, in plain language:
1. They quantized the two matrices everyone else left behind. Qwen3.8–27B has untied embeddings — an input table and an output table, each 2.5 GB sitting in full bf16 precision inside every public “quantized” checkpoint, because quantizing them is fiddly. The repo’s prepare/ scripts requantize both to int8. That’s 2.6 GB of VRAM back — roughly a whole extra person’s worth of context — with no measurable quality loss.
2. The model is mostly not what you think it is. Qwen3.8–27B is a hybrid: only 16 of its 64 layers are classic attention that grows memory with context length. The other 48 are Gated DeltaNet layers — their state per conversation is a fixed-size blob, whether the chat is 100 tokens or 100,000. Stock vLLM asked for those blobs in fp32 (about 150 MB per request!) and choked at 37 concurrent requests even when configured for 64. The fork runs them in fp16 — perplexity unchanged to three decimals — and suddenly all slots run.
3. A drafter that guesses seven tokens at once. This is the big one, and it’s why your single prompt flies instead of crawling. Speculative decoding works like this: a small “drafter” model proposes several future tokens, then the big target model checks them all in one pass. Approve what’s right, redo from the first wrong one — and because checking is nearly free compared to generating, you get several tokens for the price of one step. Qwen’s built-in MTP head chains four guesses. The fork swaps in DFlash2 — five layers that predict a whole seven-token block in one non-autoregressive pass, reading the target model’s hidden states from five different depths. Then they GPTQ-quantized the drafter itself from 3.85 GB down to 1.19 GB, so it fits on the same card without stealing bandwidth. Net effect: roughly 3 tokens per step instead of 1, losslessly — the output distribution is provably identical to no-speculation at all. GSM8K scores read 96–96.5% across every configuration in the repo.
4. A draft vocabulary counted from the model’s own mouth. Here my inner engineer laughed out loud. Every token the drafter proposes must exist in its small output vocabulary — a token outside it is a guaranteed rejection. The original vocabulary was built from web text and covered 92% of what this model generates. So the maintainers counted 5.4 million tokens of the model’s own outputs and rebuilt the list around them: 97.5% coverage. Same hardware, same model — just a dictionary tuned to how this model actually talks. That alone was worth 10% of throughput.
One more trick deserves a mention because it explains the “crazy fast” moments: lookup drafting. When the answer is quoting something from its own prompt — your codebase, a document you pasted — the fork skips the drafter entirely and proposes tokens straight from your context. Reproducing a 25k-token document runs at 381 tok/s on reference hardware. My agents spend their lives rewriting code they were just shown, which is exactly this workload. That’s part of why my number reads high: harness work quotes its own context constantly.
Why CTX=long doesn’t blow up your VRAM
By default, the repo’s single-user launcher spins up the API with a 65k context. That was too small for what I throw at it, so I wanted something like 131k — and I set CTX=long in the launch command expecting the 24 GB card to choke. A bit to my surprise: no OOM, no error, just slightly slower than the 65k instance.
Intuition says: doubling the context window should double memory use and crash a 24 GB card. Here’s why it doesn’t.
The weights are fixed. W4A16 means the model body is about 15 GB, full stop. Context length doesn’t touch that number.
The KV pool is pinned in bytes at startup — not allocated per request. This is the key design choice. The fork reserves a fixed budget (~5.2 GiB here) before serving anything, and vLLM prints exactly what it got: “GPU KV cache size: 136,429 tokens.” A pinned pool either fits at boot or refuses at boot. There is no sneaky mid-request allocation that OOMs you while your agent is halfway through a task.
CTX=long doesn’t add VRAM — it spends the same budget on cheaper tokens. The default profile stores attention caches in bf16; CTX=long switches them to int8, roughly halving the cost per token. Same 5.2 GiB, twice as many tokens: the pool grows from 68,605 to 136,429, and max context doubles to 131k. Perplexity moves by 0.2% between the two profiles — the repo measured it teacher-forced on an identical passage.
And remember the hybrid architecture. Only 16 of 64 layers grow with context at all. The other 48 keep their fixed-size state no matter how long the conversation gets. Long context on this model is far cheaper than on a classic transformer — that’s half the reason it fits on one card in the first place.
So why does the context cap at 131k when the pool holds 136,429 tokens? Because not every byte of that pool is free for one request’s attention cache. Those 48 linear layers keep a fixed-size state blob per request, and with DFlash2 each resident request holds eight speculative state slots. The long profile seats four concurrent requests, so the launcher caps max context at 131,072 — roughly 4% under pool capacity — reserving the difference for those state pages plus block alignment. That margin is what makes the “no OOM” promise real: the worst case — one request at full length with every other seat occupied — is checked at boot. The pool either fits at startup or the server refuses to start; there is no mid-task allocation that can fail.
What does doubling context cost you? Not a crash — a trade. Each token now occupies int8 instead of bf16, and each resident request reserves its state pages against a pool that must be split among fewer seats: 8 parallel slots become 4, and decode drops from ~177 to ~122 tok/s. The card makes the same number of bytes do more work; it just charges you in speed instead of throwing an exception.
The honest part: deep context still bites
I promised you the fall-down spots, so here they are.
This stack’s superpower is short-and-medium contexts — exactly what agent harnesses feed it. Push a single request toward 100k tokens and the picture changes. On my box today I measured the gradient directly: about 107 tok/s at short context, 78 tok/s at ~10k, and 38 tok/s by ~43k. At around 100k it can sag to roughly 31 tok/s — the drafter’s acceptance rate decays with depth (the repo measures the same thing: acceptance settles near 0.29 regardless of cache dtype, so this is model-and-depth physics, not a bug they forgot to fix).
Meanwhile my llama.cpp fork instance on the same class of card keeps serving a steady 65–80 tok/s at 150k context length — no drafter to starve, no verify block to fall out of favor. It’s never fast; it’s just never slow.
<button data-code="| Context depth | vLLM fork (DFlash2) | llama.cpp ATX fork |
|—————–|———————|——————–|
| short (
| Context depth | vLLM fork (DFlash2) | llama.cpp ATX fork |
|-----------------|---------------------|--------------------|
| short (<4k) | 107–177 tok/s | 65–80 tok/s |
| ~10k | ~78 tok/s | 65–80 tok/s |
| ~43k | ~38 tok/s | 65–80 tok/s |
| ~100k | ~31 tok/s | 65–80 tok/s || Context depth | vLLM fork (DFlash2) | llama.cpp ATX fork |
|-----------------|---------------------|--------------------|
| short (<4k) | 107–177 tok/s | 65–80 tok/s |
| ~10k | ~78 tok/s | 65–80 tok/s |
| ~43k | ~38 tok/s | 65–80 tok/s |
| ~100k | ~31 tok/s | 65–80 tok/s |So, if your day is one giant document and slow reading, llama.cpp remains the calmer companion. If your day is a thousand coding conversations — which, not coincidentally, is what every agent harness actually generates — this fork runs circles around it.
Why this is the right engine for agent harnesses
Here’s the thing about running Pi, Hermes, or DeepSeek Harness with subagents: you don’t have one conversation. You have a context storm — five, eight, twelve parallel threads, each medium-length, all alive at once, most of them sharing the same system prompt and codebase context.
That workload is precisely what this fork was tuned for:
- Aggregate throughput scales. Four concurrent streams deliver ~300+ tok/s combined on my box; the repo’s reference benches read 279–335 at C4 and up to ~400 aggregate at C8 with MTP. One card, one engine, a whole agent team fed.
- Prefix caching turns shared context into free lunch. With
PREFIX_CACHE=1, sixty-four requests sharing a 5.8k system prompt went from 222 seconds to 17 in the repo’s benchmark — median latency dropping from 95 s to 8 s. Every subagent after the first pays nothing for your shared instructions. On hybrid models this caches the recurrent state too, not just attention KV — which is why follow-up turns on a 24k document cost ~1 second instead of 23. - Lookup drafting loves agent-shaped output. Rewrites, refactors, and edits quote their input constantly — the workload where this stack hits its 380+ tok/s peaks.
The limits are honest too: past four or so resident long-context streams you’re in pool geometry, not horsepower (the repo is blunt that C8 on long contexts is strictly worse than C4 — “not a tradeoff, a loss”). Design your harness for ~4 parallel workers at medium depth and this thing sings.
Give it a try
If you have an RTX 3090 collecting dust while you pay monthly rent to a chatbot API, this is the most alive that card has ever felt. A 27-billion-parameter model, 131k context, agent-grade parallelism, on hardware I bought for gaming — and it cost me an evening and an AI agent’s worth of apt packages.
The GPU didn’t get faster. The math got smaller. Sometimes that’s the whole trick.
- Repo: github.com/syv-ai/qwen38–27b-rtx3090
- Why-it’s-fast deep dive: docs/optimizations.md
- Native 3090 reproduction notes: docs/reproductions/native-3090.md
- My previous result (100 tok/s on a 3090 Ti): linked here
