I'm running Qwen3.8 27B (FP8 quantized) on two AMD R9700 32GB (RDNA4 architecture), served with vLLM. This post documents the pitfalls I hit and my final configuration.
Why local inference
- Privacy: my conversations, memory, and documents never pass through OpenAI/Anthropic's servers
- Cost: download the 27B model once, then it's free (vs. API billed per token)
- Controllable latency: no waiting on a network round-trip, TTFT 280ms
- Always on: no worries about API downtime, rate limits, or regional restrictions
Model and sources
| Item | Source |
|---|---|
| Model | Qwen3.8-27B-Instruct-FP8 (HuggingFace) |
| vLLM | vLLM 0.28.0 (RDNA4 community image) |
| Docker image | capicua25x/vllm-rocm-rdna4:0.28.0-rdna4-rc13 |
| ROCm | ROCm 7.0 (AMD GPU compute platform) |
| GPU | AMD Radeon R9700 32GB (RDNA4) |
Hardware
- 2× AMD Radeon R9700 32GB (RDNA4, PCIe 4.0 x16)
- 48GB RAM (allocated to the LXC)
- ROCm 7.0 (AMD's equivalent of CUDA)
Why vLLM instead of llama.cpp
| vLLM | llama.cpp | |
|---|---|---|
| throughput | Paged attention + continuous batching, fast under high concurrency | Fast for a single request, multiple requests queue up |
| API | Native OpenAI-compatible (/v1/chat/completions) | Has it too, but a bit less stable |
| Long context | 262K tokens (my config) | Limited by RAM/GPU allocation |
| AMD support | ROCm build (there's a community RDNA4 image) | Vulkan / HIP, stable but slow |
| speculative decoding | MTP (Multi-Token Prediction) | Supported too, but less integration |
My agent has multiple auxiliary tasks running at once (vision, STT, memory extraction), so it needs high concurrency. vLLM wins.
My vLLM configuration
docker run -d --name qwen38-vllm \
--gpus all \
--ipc=host \
-p 8080:8080 \
-v /models:/models \
capicua25x/vllm-rocm-rdna4:0.28.0-rdna4-rc13 \
--model /models/vision \
--tensor-parallel-size 2 \
--max-model-len 262144 \
--kv-cache-dtype fp8 \
--attention-backend TRITON_ATTN \
--speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
--max-num-seqs 8
Explanation of the key parameters:
- TP=2: the two cards each run half the model (tensor parallel)
- max-model-len 262144: 262K context (long enough to throw an entire codebase in at once)
- kv-cache-dtype fp8: saves 50% VRAM (at the cost of a tiny precision loss)
- TRITON_ATTN: fastest on RDNA4 (I A/B tested ROCM_ATTN and FLASH_ATTENTION)
- MTP n=5: predicts 5 tokens per step, actual acceptance rate ~68% (decode speedup ~50%)
Pitfalls I hit
1. ROCm version mismatch
AMD's ROCm is tightly bound to the GPU architecture. RDNA4 needs ROCm 7.0+, and the official vLLM image has no RDNA4 build. I use a community image (capicua25x/vllm-rocm-rdna4).
2. FP8 KV cache needs alignment on RDNA4
head_dim=256 (Qwen3.8) + FP8 KV takes the Triton kernel path rather than the native fast path. Speed is still acceptable, but it's not the theoretical optimum.
3. Cold-start garbling
vLLM's first decode produces 2-3 garbled tokens (the GPU isn't warmed up yet). I added a canary gate: on a request, first send an 8-token canary, and only let real traffic through once decode is confirmed clean.
4. Uneven temps between the two cards
GPU 0 runs more of the compute-heavy layers, so it runs 5-8°C hotter than GPU 1. --tensor-parallel-size 2 balances it automatically, but physically GPU 0 sits in a worse-ventilated spot. Fix: flip the fan direction.
Performance (measured)
- decode: 60-64 tok/s (1024 token output, concurrency=1)
- TTFT: 288ms (1024 token input)
- VRAM usage: ~30.7GB / 32GB per card
- MTP acceptance rate: 67-69% (stable)
Next step
The model's running, but the agent has no "memory." Next post: how to wire up Hindsight so it can remember things across sessions.