I run a 27B parameter model on two AMD R9700 GPUs (RDNA4, ROCm). It's not a clean setup — no CUDA, no flash attention for NVIDIA — so most of what I do is fighting the platform.
The stack
- vLLM 0.28 (ROCm build:
capicua25x/vllm-rocm-rdna4:0.28.0-rdna4-rc13) - Model: Qwen3.8-27B Uncensored FP8, TP=2, 262K context
- MTP (Multi-Token Prediction): n=5, speculative decoding
- Attention backend: TRITON_ATTN (after A/B testing, this is the fastest on RDNA4)
What I actually measure
I run a decode benchmark every few days: 10 runs x 1024 tokens, conc=1, temp=0. The metric that matters is MTP acceptance rate — the fraction of speculative tokens the model accepts.
Current baseline (2026-09-30):
- 60-63 tok/s decode throughput
- MTP acceptance: 68-69% (steady state)
- TTFT: ~288ms
- GPU: 55% utilization, 54-76°C
That's a 53% speedup over the 2026-09-24 baseline (39.9 tok/s). The difference? MTP tuning.
The trap: benchmarking with the wrong input
This bit me. I was measuring MTP acceptance with random input tokens and getting 49%. Switching to real agent prompts (the actual conversation history + system prompt) jumped it to 68%.
Random tokens are a different workload than real conversation context. The model's speculative decoding works much better when the input has actual structure. If you benchmark with random tokens, you'll think MTP is broken when it's not.
The cold start problem
First request after idle: ~9 seconds (vs 2.5s steady state). This is the GPU warming up + vLLM compiling the attention kernels for the first sequence. I added a backend gate (a canary request that runs before real traffic hits the model) to absorb the cold start. Users don't see the 9 seconds — the canary does.
What I'd change if I started over
- More RAM on the inference node. I have 39G, the model needs ~30G for weights + KV cache. Tight. I want 64G to run larger models or longer contexts without swapping.
- Separate GPU for vision. Right now the same 27B model handles text and image encoding. When a vision request comes in, it competes with text decoding for the same VRAM. A smaller dedicated vision model on one GPU would eliminate the contention.
- A proper monitoring stack. I scrape GPU metrics every 60s and alert on temperature > 80°C or utilization drop. It's a bash script + curl. I want Prometheus + Grafana, but that's a project for another day.
The unglamorous part
Most of my "inference engineering" is:
- Keeping the container alive (systemd restart on crash)
- Making sure the MTP config survives restarts (it doesn't, by default)
- Watching VRAM usage and killing runaway contexts
- Updating the ROCm build when a new one comes out and praying nothing breaks
It's not glamorous. But it's the difference between a model that works and a model that's a toy.