Everyone runs vLLM on NVIDIA. I run it on two AMD R9700s (RDNA4, 32GB each) under ROCm, and the experience is... different. Here's what I've learned over three months of running a 27B model in production.
The setup
- GPU: 2x AMD R9700 (RDNA4, 32GB VRAM each), PCIe 4.0 x16
- ROCm: 6.4.x (the version that actually ships RDNA4 support)
- vLLM: 0.28.0 (RC build:
capicua25x/vllm-rocm-rdna4:0.28.0-rdna4-rc13) - Model: Qwen3.8-27B Uncensored (FP8, ~29GB)
- Tensor parallel: TP=2 (one GPU per weight shard)
- Context: 262K tokens max
- MTP: speculative decoding, n=5 tokens
This is not a "just works" configuration. There were roughly four separate things that needed to be wrong before it ran well.
Attention backend: pick TRITON_ATTN
This is the one that costs you the most time if you get it wrong. vLLM on ROCm supports several attention backends:
| Backend | Median latency (256 tok) | Notes |
|---|---|---|
ROCM_ATTN | 3,369 ms | Native HIP kernel. Sounds right. Isn't. |
TRITON_ATTN | 2,527 ms | Triton-compiled. Faster by ~25%. |
FLASH_ATTENTION_TRITON | ~same as TRITON_ATTN | Only affects ViT, not LM decode |
TORCH_SDPA | Slowest | PyTorch's fallback. Don't. |
The counterintuitive part: ROCM_ATTN is the "native" backend, the one AMD writes for. But on RDNA4 specifically, the Triton-compiled version wins. I A/B tested both with identical configs (10 runs, 1024 tokens, concurrency=1, temp=0) and the gap was consistent across all runs.
Why: RDNA4's native paged-attention fast path requires head_size=128 and auto KV dtype. My config uses head_dim=256 with FP8 KV cache, so the "native" path falls back to a generic kernel anyway. The Triton compiler, on the other hand, generates code for the actual shape I'm running.
This means the "right" answer depends on your model's head dimension. If you're running a model with head_dim=128 and standard FP16/BF16 KV, ROCM_ATTN might actually win. I haven't tested that. I test what I run.
The MTP trap
Multi-Token Prediction (speculative decoding) is where the real speedup lives. My numbers:
- Without MTP: ~35 tok/s decode
- With MTP (n=5, 68% acceptance): ~63 tok/s decode
That's a 1.8x speedup. But here's the trap that almost made me disable it entirely:
When I benchmarked MTP with random input tokens, I got a 49% acceptance rate. With real conversation prompts (actual agent history + system prompt), it jumped to 68%. A 19-point gap from the input distribution alone.
If you benchmark with random tokens and see 49%, you'll think MTP is barely worth it. It's not barely worth it — it's your benchmark that's wrong.
Rule: Always benchmark with inputs that match your actual workload. For a conversational agent, that means real conversation history. For a code completion tool, that means real code context. Random tokens measure a different model than the one you're serving.
Cold start: 9 seconds you shouldn't have to pay
First request after idle: 9.1 seconds (vs 2.5s steady state). This is the GPU warming up + vLLM compiling attention kernels for the first sequence.
I solved this with a backend gate: a canary request that fires before real traffic hits the model. The canary absorbs the 9-second cold start. Users see 2.5s. The canary is a short prompt (50 tokens) that runs every 30 seconds during the day, every 2 minutes at night.
The canary also serves as a health check: if it takes longer than 10 seconds, something's wrong (GPU thermal throttling, VRAM leak, zombie process eating memory).
VRAM: the number that matters
With TP=2 and a 29GB model, you'd expect ~14.5GB per GPU. In practice:
- Weights: ~14.5 GB per GPU (FP8, TP=2)
- KV cache: ~8-10 GB per GPU (262K context, FP8)
- Activations + overhead: ~2-3 GB per GPU
- Total: ~25-27 GB per GPU out of 32GB
That leaves 5-7 GB of headroom. It's enough for concurrent requests up to about 4-5 sequences before you start OOMing. If you're serving a single agent conversation (which is my use case), you'll never hit the limit.
The number to watch: gpu_memory_utilization from the vLLM metrics endpoint. I alert at 90%. If you're running multiple models or longer contexts, this is where you'll find out you've run out of room.
What I'd do differently
- More RAM on the host. I have 39GB. The model needs ~30GB for weights + KV. Tight. I want 64GB to run a 70B model or serve longer contexts without swapping.
- Separate vision GPU. The same 27B model handles both text and image encoding. When a vision request comes in, it competes with text decoding for the same VRAM. A smaller dedicated vision model on one GPU would eliminate the contention.
- A real monitoring stack. I scrape GPU metrics every 60 seconds with a bash script + curl. I want Prometheus + Grafana. That's a project for another day.
The unglamorous 80%
Most of "running vLLM on AMD" is:
- Keeping the container alive (systemd restart on crash)
- Making sure the MTP config survives restarts (it doesn't, by default — you have to pass it as a CLI flag every time)
- Watching VRAM usage and killing runaway contexts
- Updating the ROCm build when a new one comes out and praying nothing breaks
- Figuring out which
HIP_environment variables are still required and which were removed in the last version
It's not glamorous. But it's the difference between a model that works and a model that's a toy. And 63 tok/s on consumer AMD hardware is not a toy number.