intel-arc-pro-b70-inference-cookbook

Pinned Intel Arc Pro B60/B70 inference recipes, engine patches and measurement records: Qwen3.8-27B EXL3 on one card (262,144 context, 63 tok/s; MTP3 measured lossy on 27% of a fixed 329-item set), Nemotron Bolt 74B, Flash-Next, dual-card routes, and a machine-checked catalog where each speculative row states whether t

Automatically checked · GitHub checked 2026-11-17

intel-arc-pro-b70-inference-cookbook

Overview

Intel Arc Pro B70 Inference Cookbook

Pinned vLLM XPU, llama.cpp SYCL and EXL3-XPU recipes, failure notes, and measurement records for Intel Arc Pro B60/B70. Every published benchmark row carries its exact workload, a commit-pinned evidence link, and — for routes that decode with a drafter — whether the drafter changes the model's greedy answer. Start with what worked, what did not, and what to test next. For a first local run, use START.md.

Pick one path

Your goalStart hereStatus
Choose a model for a jobModel comparison94-task quality screen of Bolt 74B-A9B (two cards), Qwen3.8-27B and Qwen3.6-35B-A3B (one card each) with power and heat; one sample per task, Qwen3.6 partial.
Serve on one B70Qwen3.8-27B EXL3 routePrimary route: trellis 4bpw + MTP3 + fp8 KV on the exl3xpu engine; the only measured route at full 262,144 context and clean on the exact-answer battery. MTP3 changes the greedy answer on 27 % of a fixed 329-item set — serve --set vllm.speculative_config=null when one-shot code must match, and see spec parity on every speculative row. The GPTQ/W4A16 route is legacy — that class fails the canary on this model.
Fastest lossless 27B at short contextQwen3.8-27B TensorFold routeScreen 2026-11-08: TensorFold + DFlash2 at 128 tok/s with output identical to plain greedy; EXL3+MTP3 wins from ~16K up. Includes a +48 % long-context patch and, from 2026-11-13, a prefix attention kernel on our TensorFold tag: 90 tok/s at 32K, 51 at 131K (screen; text differs from stock TensorFold, not upstream).
Long prompts on llama.cppKV type at depthScreen 2026-11-08: on llama.cpp after 2026-10-29, q8_0 or q4_0 KV decodes 1.8–2.1× faster at 64K than before; q5_0/q4_1 and f16 do not gain.
Run GGUF or visionMuse-Glimmer llama.cpp routeMeasured on its own engine and workload; no cross-engine speed ranking without matched timing.
Run MoE or long contextQwen3.6-35B-A3B recipeHistorical MTP speed record is stack-specific; a later matched comparison did not reproduce it.
Serve on two B70sIndependent-card setupDefault: one server per card. FP8 TP2 is a research route, not an automatic single-request win.
Diagnose or reproduce a numberImage and patch matrix → family recipe → benchmark catalogMatch image, patch, model, context, cache, power, and timing definition.

Read the decision guide before tuning. It separates measured improvements, failed attempts, unpublished lab screens, and stop/go gates. A passing smoke canary does not establish model quality. No public performance claim from a different model, engine, host, or result generation transfers to this setup.

Reproduce and operate

  1. Check your card and host with scripts/doctor.py, then use the quick start for a pinned single-card run.
  2. Check the image and ordered patch matrix for the exact family before launching. Do not mix family patch stacks.
  3. Connect a client using client setup; use the reliability guide if the server hangs.
  4. Confirm a recipe with the reproduction procedure and benchmark format. The numeric authority is data/benchmarks.v1.json; docs/BENCHMARK-CATALOG.md is generated.

More routes: Ripple-65B on two cards (new, -sm tensor 13.3 tok/s), Nemotron DFlash, Nemotron Bolt 74B (one card on llama.cpp, or two cards on vLLM), Flash-Next, Ornith-1.5-122B GGUF, Windows, and family-specific research. Research routes are not interchangeable with the first-run recipe.

Quantize a model yourself: how the Bolt 74B AutoRound W4A16 quant was made and checked on two B70s, with every command, fix and script.

Request or add a recipe

Need a model or configuration not yet covered? Submit a recipe request with your target Hugging Face model and hardware setup.

To contribute a recipe directly to the B70 Launcher catalog, follow the recipe format specification to edit recipes.json, verify with python3 -m unittest tests.test_recipes, and open a pull request.

For contributors and agents

Use data/system-map.v1.json to select an intent. docs/AGENT-SYSTEM.md explains the authority layers; docs/ADDING-A-RECIPE.md explains publication. Keep historical evidence intact. Validate changes with python3 scripts/check-repo.py.

Code: MIT. Measurement reports and prose: CC BY 4.0. See LICENSE.