llamAmpere

llama.cpp fork for significantly improved performance on Ampere generation Nvidia cards (especially RTX 3090 / 3090 Ti): example: 110+ tok/s over a 100K-token generation at temperature 1 for qwen3.8, supporting full 262K ctx. Supports low memory (12gb) configs for 200k+ ctx. 120+ TPS, 3300+ prefill for QFN 3.8.

Automatically checked · GitHub checked 2026-11-17

llamAmpere

Overview

llamAmpere

llamAmpere is based on mainline llama.cpp (v1.0), tuned for NVIDIA RTX 30-series Ampere cards. With 1.0 we prioritize keeping up with mainline. It includes TheTom TurboQuant work, credited below.

It runs Qwen3.8 models, including Qwen3.8-27B with its built-in MTP draft head (a small helper that guesses several words ahead so the model writes faster) and Qwen3.8 Flash-Next.

What's new in v0.7

  • Built on mainline llama.cpp v1.0, with card presets for Swift 1.5 Qwen3.8-27B: --preset 24dense, 12dense, 10dense and 8dense pick the context length, memory settings and sampling for each card size. The 24 GB, 12 GB and 10 GB models ship with this release; the 8 GB model file comes in v0.7.1. Release notes.

What's new in v0.6.4

  • Fix: Flash-Next ran out of memory or stalled on machines with two or more NVIDIA cards, or on a card that another program was already using (#96). v0.6.4 runs Flash-Next on one card, the one with the most free memory (--device picks another); splitting it across two cards is not supported yet. On a busy card with less than 8 GiB free, the launcher uses the memory planner, which runs but is slow. One idle card: unchanged. Release notes.

What's new in v0.6.3.11

  • Fix: faster first token on short follow-up messages (up to 256 new tokens) after a long first message, on Flash-Next. Longer follow-ups are unchanged. Release notes.

What's new in v0.6.3.10

  • Fix: slow first token on short follow-up messages after a long conversation. Release notes.

What's new in v0.6.3.9

  • Flash-Next on native Windows builds (MSVC + CUDA) can use the fast card-side expert tables, which were Linux-only. The contributor measured 0.26 -> 60 tokens/s on code on an RTX 3090 with 64 GB RAM. Thanks to huhichao (#98). We have no Windows machine: built and tested under MinGW/Wine, a CPU-only build that cannot compile the CUDA and MSVC parts; Windows speed figures are from the contributor. Linux is unchanged. Release notes.

What's new in v0.6.3.8

  • The DFlash2 drafter on dense Swift 1.5 27B writes about 3.4 times faster (27 -> 93 tokens/s at temperature 1.0 on an RTX 3090 Ti, 100K prompt). A mixing step in the drafter ran twice per layer; it now runs once. The big model still checks every guess, so output quality is unchanged. MTP and Flash-Next are not affected. Release notes.

What's new in v0.6.3.6

  • Flash-Next runs AI4, a four-bit AI+ model file, qwen3.8-flash-next-Swift1.5-ATX-AI4 (84.97 GiB, closer to the Swift-Q8 reference than UD-IQ4_XS). Not recommended on 64 GB RAM machines unless you pin a large share of experts on the GPU and move the backup copies out of VRAM to cut RAM use.
  • Fixes the Linux build failure with Clang and some other compilers (#98). Saved prompt caches are keyed by expert-routing and GGML_ settings. Release notes.

What's new in v0.6.3.5

What's new in v0.6.3.3

  • Small cards use memory-bounded prompt passes and shorter prompt-tail chunks by default. Normally free 24 GB cards keep their previous plan.
  • Fixes Windows private model mappings and MSVC atomics (#94, #93, #82). Linux launchers count physical performance cores on hybrid Intel CPUs.
  • Adds CPU expert-split diagnostics and --moe-cpu-share for troubleshooting (#88). Release notes and hardware limits.

What's new in v0.6.3.2

  • Fixes a Flash-Next crash when a follow-up request reuses a cached prompt with shared experts on a side stream (#88).
  • Fixes Windows model loading, Windows build errors, and reads from large expert overlays (#82). Release notes.

What's new in v0.6.3.1

  • Flash-Next leaves more temporary card memory on 24 GB cards and handles --parallel startup. Unsupported Flash-Next tensor split now gives a clear error.
  • Small cards keep 32K prompt passes, with a 4096-token microbatch to avoid a measured prompt fallback. Adaptive prompt passes remain opt-in. Release notes.

What's new in v0.6.3

  • New Flash-Next model file, qwen3.8-flash-next-Swift1.5-ATX-AI3: 80.11 GB (74.61 GiB), compared with 75.97 GB for IQ3_XXS. On an RTX 3090 with ample system RAM, this file wrote 121–125 tokens/s across coding, agentic and RAG tasks. These are observations on the new file, not a same-engine comparison with the standard file. Measured results and limits.
  • The expert kernels our speed tests use are now on by default; whole-file page read-ahead when the model fits in RAM.

What's new in v0.6.1

  • Faster Flash-Next: about 3% faster writing and prompt reading than v0.6 on an RTX 3090 Ti (113.15 vs 109.65 writing tokens/s, 2,751 vs 2,672 prompt tokens/s, 100K prompt; one-seed observation), and room for more experts on the card. Fixes the 12 GB startup crash. v0.6.1 release notes.

What's new in v0.6

  • mb-MTP is on by default with the MTP draft head: more guessed words are kept, so writing is faster (--no-spec-mb-mtp gives exact sampling).
  • Faster prompt reading on Swift 1.5 27B (+5.3% at 100K tokens) with the same output.
  • AI+ model file for Swift 1.5 27B, our own weight format for Ampere and newer cards: 14.4 GB, 10.4% lower KL (a measure of drift from the full-precision model) than IQ4_XS-M. Older AI+ files no longer load.
  • Qwen3.8 Flash-Next support: experts split between the card and RAM, with --kv-gpu-tokens N to keep only part of the context memory on the card.

Details, speed tables and everything older: v0.6 release notes.

Quick start

Quick start: pick your card

Card presets assume a display is attached and reserve 1,024 MiB of GPU memory, except 8dense, which is headless-only and reserves 500 MiB. 8dense headed prints a note and uses the same headless settings. Append headless (--preset 12dense headless) or use 12dense-headless for a 500 MiB margin on 10dense, 12dense, and 24dense. Headed defaults are 180,224 tokens for 10dense and 196,608 for 12dense, with CUDA graphs off for headed 12dense. Headless defaults remain 212,992 for 10dense, 262,144 for 12dense, and 204,800 for 8dense. Headless 12dense at the full 262,144 tokens is tested on an RTX 3080 Ti 12 GB: a 262,142-token fill completed with 11,642 MiB peak (500 MiB margin line 11,788), 1,009 prompt tokens/s and 70.8 writing tokens/s at a 100K prompt. Fitting is on: the dynamic microbatch ladder remains enabled, and fitting may reduce a preset context to honor the margin. An explicit -c, -fitt, or -fit wins in either order. Startup reports the chosen context. Small cards still support one user only; multi-user support is future work. Re-check / update every preset against final 0.7 measurements before ship.

Card profiles target dense Swift 1.5 Qwen 27B. All four set sampler defaults to temperature 1.0, top-k 20, top-p 0.95, and min-p 0; explicit CLI flags and per-request sampler fields override them. The 8dense profile disables mb-MTP by default; add --spec-mb-mtp to enable it. The other three profiles keep mb-MTP on with MTP. Choose the matching model file and use one profile:

llama-server --preset 24dense -m /path/to/P76-AI4.gguf
llama-server --preset 24dense -m /path/to/P76-AI4.gguf -np 3 -c 352256 --slot-ctx 262144,0,0
llama-server --preset 12dense -m /path/to/L0.gguf
llama-server --preset 12dense headless -m /path/to/L0.gguf   # no display on the card: full 262,144 context
llama-server --preset 10dense -m /path/to/public-AI2.gguf
llama-server --preset 8dense -m /path/to/healed-a2p-infile-mtp-r123-drafter-iq4xs.gguf

Bare names work too: llama-server 24dense -m MODEL. llama-cli -p TEXT still takes a prompt; the server does not accept -p. Explicit flags override profile values in either order. Card profiles override config.ini defaults; environment values and CLI flags override the card profile. For adaptive MTP, lowering the maximum draft depth also lowers the profile's minimum depth unless you set both. Raising the minimum also raises the profile's maximum unless you set both. Changing a K/V cache type or SJ body type drops the profile's related KV/LSA defaults; supply the full cache configuration you want. A smaller batch caps the default microbatch. Profiles do not download models. Place lsa-qwen38-27b-d2048.gguf beside the 8/10/12 GB model, or set LLAMA_LSA_MAP / --attn-sparse-map to its path. For 8dense also place mtp-dm.gguf, head-ait2.gguf, and head-exact-f16.gguf beside the model; LLAMA_MODEL_OVERLAY_PATCH and LLAMA_D1_HEAD override these paths. Shell environment values override profile environment defaults. Startup prints the profile version and expanded defaults; a different model family gets a warning.

All four profiles are enabled and need final 0.7 validation; see the preset checks. These are not full-context fit guarantees. The 24 GB profile keeps f32 recurrent state and supports one to four slots with asymmetric context reservations; 0 gives a slot a share of the remaining pool. The 8/10/12 GB profiles require -np 1. TODO: multi-user for small cards (future). All four profiles enable dynamic microbatch sizing, with checks at most every 2 seconds of prompt processing and 10 seconds of decode. Set LLAMA_DEEPCTX_LADDER=0 for static sizing. The microbatch ceilings stay at 1024/512/512/256 for 24/12/10/8dense; an explicit -ub overrides the ceiling. The 8 GB profile uses the accepted F2 settings: reserve 7 rows, stage strips on, CUDA graphs off, 1% buffer growth, and mean chunks on. This build includes the reserve reader. Copy defaults off; earlier 8 GB measurements used copy on, so final measurements must be repeated with copy off. Packed masks remain opt-in. Eligible window-0 paths select DSJ/RM24H automatically, with --no-spec-draft-sjkvarn as the opt-out; the 24dense profile sets window 4096 and therefore does not select that automatic path. Context and memory fit depend on the model and overrides.

Build and run (v0.6.3.11, Qwen3.8 Flash-Next, Linux). For one RTX 30-series card with enough system RAM for the model and prompt workspace. Small-card tests used a 3090 limited to 12 GB with about 90 GiB of system RAM; native 12/16 GB cards remain unmeasured, and 64 GB of system RAM is not a fit promise. Small cards use memory-bounded prompt passes; see the hardware limits. The launcher keeps the tested --fit off setting; its own card planner still chooses the memory plan and keeps at least 5 GiB of RAM free. Automatic --fit on has not been measured on this file. Needs the CUDA 12 or newer toolkit (nvcc), CMake, git, a C++ compiler, the hf download tool (pip install -U huggingface_hub) and 81.63 GB for the model and drafter, plus space for the build, prompt scratch and saved prompts. The launcher uses a 5 GiB free-RAM floor. This one paste clones or updates llamAmpere at v0.6.3.11, builds llama-server, downloads the model and its drafter and starts the server:

if [ -d llamAmpere ]; then
  git -C llamAmpere fetch --tags --force && git -C llamAmpere checkout v0.6.3.11
else
  git clone -b v0.6.3.11 https://github.com/JakeATX/llamAmpere.git
fi && cd llamAmpere && \
cmake -S . -B build-sm86 -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DLLAMA_BUILD_BORINGSSL=ON && \
cmake --build build-sm86 -j8 --target llama-server && \
hf download jakeatx/qwen3.8-flash-next-Swift1.5-ATX-AI3 --include '*.gguf' --local-dir models/flash-next && \
hf download jakeatx/Swift-1.5-Qwen3.8-Flash-Next-MTP-Drafter-GGUF mtp-Swift-Qwen3.8-Flash-Next-GSQ-RCO-Q2S-B.gguf --local-dir models/flash-next && \
LLAMA_SERVER_BIN=build-sm86/bin/llama-server \
FLASH_NEXT_DRAFTER=models/flash-next/mtp-Swift-Qwen3.8-Flash-Next-GSQ-RCO-Q2S-B.gguf \
scripts/flash-next-server.sh -m models/flash-next/qwen3.8-flash-next-Swift1.5-ATX-AI3-00001-of-00002.gguf

To start it again later, run the last three lines from the llamAmpere folder. The server listens on http://127.0.0.1:8080 (OpenAI-compatible API).

Build and run (v0.6.3.11, Swift 1.5 27B at 2.3 bpw, 200K context, one 24 GB RTX 30-series card, Linux). Needs the CUDA 12 or newer toolkit (nvcc), CMake, git and a C++ compiler. The model (about 9.1 GB) downloads on first start. This one paste clones or updates llamAmpere at v0.6.3.11, builds llama-server and starts it:

if [ -d llamAmpere ]; then
  git -C llamAmpere fetch --tags --force && git -C llamAmpere checkout v0.6.3.11
else
  git clone -b v0.6.3.11 https://github.com/JakeATX/llamAmpere.git
fi && cd llamAmpere && \
cmake -S . -B build-sm86 -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DLLAMA_BUILD_BORINGSSL=ON && \
cmake --build build-sm86 -j8 --target llama-server && \
./build-sm86/bin/llama-server -hf jakeatx/qwen3.8-27b-Swift1.5-Uncensored-ATX-AI2 \
  -c 204800 -ngl 99 -fa on -b 4096 -ub 1024 --parallel 1 --jinja --no-context-shift \
  -ctk sjkvarn3 -ctv sjkvarn2 --sjkvarn-body-type auto \
  --spec-type draft-mtp-adaptive --spec-draft-n-min-adaptive 3 --spec-draft-n-max 4 \
  --spec-draft-p-min 0 --spec-mb-mtp

Commands for 12 and 10 GB cards (not yet run on this model) are in the v0.6.2 release notes.

The server listens on http://127.0.0.1:8080 (OpenAI-compatible API). Add --no-spec-mb-mtp for exact drafting. To use the IQ4_XS-M file (15.6 GB, also runs in mainline llama.cpp) instead, use --hf-repo jakeatx/ATX-Swift-1.5-Qwen3.8-27B-Uncensored-IQ4_XS-M-GGUF --hf-file ATX-Swift-1.5-Qwen3.8-27B-Uncensored-IQ4_XS-M.gguf.

Headline results

Numbers for this release (Swift 1.5 27B, v0.6 against v0.5) are in the v0.6 release notes.

Links

License and credits

llamAmpere is MIT licensed (see LICENSE) and is based on mainline llama.cpp by the ggml authors. It includes TheTom's TurboQuant KV-cache work and contributions from that fork. Original authors and license headers are retained. It is the successor of llama-cpp-qwen-ampere. See CREDITS.md for contribution links. Contributing: CONTRIBUTING.md.

Our attention selector is derived from Distilled Latent Sparse Attention (D-LSA), introduced by Ales Kubicek and colleagues in Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention. The paper projects queries and keys into a learned index space, groups keys by similarity, searches group summaries, and applies attention to the selected original keys plus a local window and sink tokens. We use the authors' released Qwen d2048 projection map. Our implementation adapts this approach with fixed blocks of 128 or 16 tokens, finer selection, and host KV offload; these are our adaptations, rather than a reproduction of the paper's semantic clustering or tree search. Attention on selected keys uses the stored KV representation; selection, KV quantization, and compensation still introduce approximation. The map's actual index width is 64; d2048 is its release name.

See the published evaluation and measured quality and limits.

Third-party libraries (from upstream llama.cpp):

  • yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
  • nothings/stb - Single-header image format decoder, used by multimodal subsystem - Public domain
  • nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
  • mackron/miniaudio - Single-header audio format decoder, used by multimodal subsystem - Public domain
  • sheredom/subprocess.h - Single-header process launching solution for C and C++ - Public domain