emmy

Optimized GPU compiler for LLM inference. Choose from a list of optimized recipes or optimize your own model via kernel fusion, autotuning, and advanced scheduling. Run benchmarks across different GPU types and configurations, track results and share experiments with the community.

Automatically checked · GitHub checked 2026-11-14

Overview

PyPI Tests Discord

Compile → Benchmark → Deploy. Emmy compiles PyTorch models into CUDA kernels, benchmarks LLM inference, and deploys models on local or cloud GPUs with vLLM and SGLang.

Compile · Tune · Deploy · Benchmark · Serve · Recipes

See Emmy in action: Outperforming vLLM on Gemma4-12B.

Install

Requires Python 3.12 or newer.

pip install emmy-ml
emmy --version

The package includes ready-to-use recipes. Remote deployment and benchmarking do not require a local GPU or CUDA installation. For gated Hugging Face models, accept the model's license and set HF_TOKEN.

Compile

Compile a PyTorch operation, compare its speed with PyTorch, or inspect the generated CUDA:

pip install "emmy-ml[compile]"

# Compile an operation
emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))"

# Benchmark against eager PyTorch and torch.compile
emmy run --bench -c "nn.Softmax(dim=-1)(torch.randn(1,28,2048,2048))"

# Inspect the generated PTX
emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir ptx

Running compiled kernels requires an NVIDIA GPU and the CUDA toolkit's PTX assembler (ptxas); source builds also need Rust. For GPUs older than Turing, use CUDA 12 and a PyTorch build that supports your GPU. Fast math is enabled by default; set EMMY_FAST_MATH=0 to disable it.

You can also compile a custom module:

emmy compile -c "
class LN(nn.Module):
    def forward(self, x):
        m = x.mean(-1, keepdim=True)
        v = ((x - m) ** 2).mean(-1, keepdim=True)
        return (x - m) * torch.rsqrt(v + 1e-6)
LN()(torch.randn(64, 2048))"

For Python integration, use torch.compile(model, backend="emmy", dynamic=False) under torch.no_grad(). Read the compiler guide or the compiler walkthrough for more.

Lowered kernel example

Readable Schedule: emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir tile

kernel k_b70c36714207  inputs: rms_norm_mean_count, rms_norm_eps, x, p_weight  outputs: rms_norm
    in0 = load rms_norm_mean_count[0]
    in1 = load rms_norm_eps[0]
    Tile(axes=(a0:256=THREAD, a1:32=BLOCK)):
        x_smem = Stage(x, origin=(0, a1, 0), slab=(a2:2048@2)) async
        p_weight_smem = Stage(p_weight, origin=(0), slab=(a3:2048@0)) async
        StridedLoop(a2 = a0; < 2048; += 256):  # reduce
            in2 = load x_smem[a2]
            v0 = multiply(in2, in2)
            acc0 <- add(acc0, v0)
        v1 = divide(acc0, in0)
        v2 = add(v1, in1)
        v3 = rsqrt(v2)
        StridedLoop(a3 = a0; < 2048; += 256):  # free
            in3 = load x_smem[a3]
            in4 = load p_weight_smem[a3]
            v4 = multiply(in3, v3)
            v5 = multiply(v4, in4)
            rms_norm[0, a1, a3] = v5

Optimized kernel, as PTX: emmy compile -c "nn.RMSNorm(2048)(torch.randn(1,32,2048))" --ir ptx (an excerpt; each Kernel IR statement stands as a comment above its instructions)

.visible .entry k_b70c36714207(
	.param .u64 .ptr .align 16 k_b70c36714207_param_0,
	.param .u64 .ptr .align 16 k_b70c36714207_param_1,
	.param .u64 .ptr .align 16 k_b70c36714207_param_2
)
.maxntid 256, 1, 1
{
	...
	// Grid(axes=(a0 in 0..32, a1_co in 0..256), decode=(block_idx.x, thread_idx.x), block_threads=256):
	mov.u32 %r0, %ctaid.x;
	mov.u32 %r1, %tid.x;
	// init f32 acc0 = 0
	mov.f32 %f0, 0f00000000;
	// f32 in0__u0 = load buf0[2048 * a0 + a1_co]
	mul.wide.s32 %rd3, %r0, 8192;
	add.s64 %rd4, %rd0, %rd3;
	mul.wide.s32 %rd5, %r1, 4;
	add.s64 %rd6, %rd4, %rd5;
	ld.global.nc.f32 %f1, [%rd6];
	// f32 v0__u0 = in0__u0 * in0__u0
	// f32 acc0 <- add(acc0, v0__u0) over a1_co
	fma.rn.ftz.f32 %f0, %f1, %f1, %f0;
	// f32 in0__u1 = load buf0[2048 * a0 + a1_co + 256]
	ld.global.nc.f32 %f2, [%rd6+1024];
	...

Tune

Try 20 schedules for a single kernel on your GPU, with cold-cache measurements:

emmy run --bench --cold-cache --tune 20 -c "nn.Linear(4096,4096,bias=False)(torch.randn(1,4096))"

Successful measurements are saved so later compiles can reuse them when the kernel and measurement settings match.

Fit the priors

The schedule prior ranks kernel schedules; the placement prior chooses how to split work into kernels. Both guide compilation when no compatible measurement is available. You can refit them from the repository's golden files in a source checkout, without a GPU:

# Load the repository goldens and the measurement freeze into a separate database
emmy db import --db _data/dataset.db --fresh --repository

# Export a dataset for each prior
emmy db export --db _data/dataset.db _data/schedule --space schedule
emmy db export --db _data/dataset.db _data/placement --space placement

# Refit the shipped weights
emmy fit _data/schedule emmy/compiler/pipeline/search/prior/weights/schedule.json
emmy fit _data/placement emmy/compiler/pipeline/search/prior/weights/placement.json

# Evaluate both priors
emmy eval prior _data/schedule
emmy eval prior _data/placement

To experiment without replacing the shipped weights, fit a separate candidate and compare it on the same dataset:

emmy fit _data/schedule _data/schedule-candidate.json --folds 0
emmy eval prior _data/schedule \
  --offline-file emmy/compiler/pipeline/search/prior/weights/schedule.json \
  --compare-to _data/schedule-candidate.json --json _data/schedule-comparison.json

Adjust --iterations, --depth, or --learning-rate to try different fits. --folds 0 skips cross-validation; omit it to use the default five folds. Each fit writes metrics under _tune/fits/, and the comparison command writes a JSON report. See the CLI reference for all tuning and fitting options.

Keep what a tune measured

emmy run --bench --tune N writes every clean row it measures into the tune DB of the machine it ran on, and freezes the same rows into one file beside the card's under emmy/compiler/pipeline/search/freezes/, the measurement freeze the repository keeps, one golden file per card in git LFS: --repository reads it beside the goldens, and the nightly refresh reports the prior's regret over its pools. Commit the tune's file with your change (run git lfs install once on the machine that commits); the nightly refresh folds the files into each card's file on main, then compiles every row of the freeze under its own schedule and drops the rows the compiler no longer realizes (emmy golden realize --drop). A tune on a rented card writes the file into the copy there: copy it back next to the card's file, as you do the working golden. EMMY_FREEZE_DIR names another directory or, set empty, none.

Candidate files

A hardware golden holds the best known schedule of each kernel. A candidate file, one per hardware golden under emmy/compiler/pipeline/search/candidates/records/, holds a few other measured schedules of the same kernels: 3 where the kernel's schedule space is small, up to 10 where it is large, the slow ones beside the near misses. emmy candidates build fills the files from the measurement freeze and lists the groups still under their target; a tune of such a kernel on its card (emmy run --golden PATH --kernel REF --bench --tune N) freezes the rows the next build takes. emmy candidates check, restamp and realize --drop keep the files current, as the emmy golden commands keep a golden. No compile reads a candidate file.

A tune DB from a machine with no checkout is merged by hand (~/.cache/emmy/autotune.db, or the file EMMY_TUNE_DB names, copied over):

emmy db import --db _data/freeze.db --fresh emmy/compiler/pipeline/search/freezes autotune.db
emmy db freeze --db _data/freeze.db --out emmy/compiler/pipeline/search/freezes

A schedule measured on both sides takes the tune DB's row, faster or slower, a kernel the compiler no longer lowers drops out, and merging the same tune DB again changes nothing. Commit the rewritten files.

Deploy

Choose a recipe that matches your GPU. This Gemma recipe targets one RTX 5090:

emmy recipe list

# Local Docker Compose
emmy deploy local --recipe gemma-4-12B-it

# An existing server
emmy deploy ssh --recipe gemma-4-12B-it --ssh user@host

# Rent a cloud VM and deploy
emmy deploy cloud --recipe gemma-4-12B-it

A recipe name creates an editable copy in the current directory. You can also pass a recipe directory with --recipe path/to/recipe. Deployment leaves the server running and prints its endpoint.

Local deployment needs Docker Compose and NVIDIA container support. Cloud deployment needs provider credentials: CLOUDRIFT_API_KEY for CloudRift, or an authenticated gcloud CLI for GCP. See the deployment reference for teardown, provider options, and multi-model plans.

Benchmark

From a source checkout, run an experiment on cloud GPUs, your machine, or existing servers:

# Rent GPUs, run one experiment, and tear down
emmy bench experiments/gemma-4-12B/gsm8k_mtp_rtx5090

# Select RTX 5090 variants across Gemma experiments
emmy bench experiments/gemma-4-12B/* --filter "deploy.gpu=*5090*"

# Use this machine or an existing server
emmy bench experiments/gemma-4-12B/gsm8k_mtp_rtx5090 --local
emmy bench experiments/gemma-4-12B/gsm8k_mtp_rtx5090 --ssh user@host

Results and logs are saved in a timestamped directory beside the experiment recipe. Cloud VMs are removed after the run; existing hosts are kept. Local benchmarks use SSH to localhost, so local SSH access must be configured. Browse experiments and results or read the benchmark reference.

Serve

Serve an embedding model through vLLM with Emmy-compiled kernels:

pip install "emmy-ml[compile,serving]"
emmy serve Qwen/Qwen3-Embedding-0.6B

Then send a request from another terminal:

curl localhost:8000/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-Embedding-0.6B","input":"Hello"}'

For a one-shot benchmark and a stock-vLLM comparison:

emmy serve Qwen/Qwen3-Embedding-0.6B --bench --random-input-len 32
emmy serve Qwen/Qwen3-Embedding-0.6B --bench --random-input-len 32 --stock

See the serving guide for text generation and the native generation guide for the experimental Rust server.

Recipes

A recipe sets the model, serving engine, GPU, and optional benchmark workload. Save this as my-benchmark/recipe.yaml and run emmy bench my-benchmark:

model:
  huggingface: "Qwen/Qwen3-0.6B"

engine:
  llm:
    tensor_parallel_size: 1
    context_length: 4096
    vllm:
      image: "vllm/vllm-openai:v0.23.0"

benchmark:
  num_prompts: 256
  random_input_len: 1024
  random_output_len: 128

# Cross-product: 2 GPUs × 2 concurrency settings = 4 variants
matrices:
  cross:
    deploy.gpu_count: 1
    deploy.gpu: ["NVIDIA GeForce RTX 5090", "NVIDIA H100 80GB"]
    zip:
      engine.llm.max_concurrent_requests: [32, 128]
      benchmark.max_concurrency: [32, 128]

Browse serving recipes for existing configurations. The recipe reference covers fields, parameter sweeps, LoRA adapters, and custom commands.

Virtual Machine Management

# Rent a GPU server and save its lease for cleanup
emmy vm create gpu --gpu "NVIDIA GeForce RTX 5090" --gpu-count 1 --exact-gpu-count \
  --lease /tmp/emmy-vm.json --owner local-run --json

# Delete that server
emmy vm delete lease /tmp/emmy-vm.json --owner local-run

See the VM reference for provider-specific commands and availability queries.

Documentation

  • CLI reference — command options, tuning, refitting priors, and image publication
  • Compiler tutorials — tracing, optimization, and measuring kernels
  • Glossary — compiler and serving terminology

Related Projects

These repositories provide the platform, serving runtimes, and deployment services that Emmy integrates with. The CloudRift repositories are typically checked out beside Emmy, so use ../<repository-name> for local code inspection when that sibling exists, and verify the remote source when current compatibility matters. Some are internal and require CloudRift organization access.

ProjectRelationship to Emmy
CloudRiftGPU cloud platform and API/protocol implementation behind Emmy's CloudRift provisioning and deployments.
1Cat-vLLMVolta/SM70 vLLM fork and source for docker/1cat-vllm-sm70; consult it when stock vLLM lacks target-GPU support.
Rift ProvisioningAnsible automation for CloudRift provider nodes and the CI and agent runner hosts used by Emmy workflows.
Rift RelayOpenAI-compatible inference proxy and model-offering service that consumes Emmy recipes and invokes Emmy deployments.
vLLMPrimary serving engine upstream and authoritative model-support, release, and image source.
SGLangAlternative serving engine upstream and model-support source.

Project Structure

Contributing

Source setup requires Python 3.12+, a Rust toolchain and git-lfs. make setup builds the Rust extensions into the package and runs again on its own after a Rust or pyproject.toml change:

git clone https://github.com/cloudrift-ai/emmy.git
cd emmy
make setup

Follow STYLE.md and the contribution instructions for development, testing, and PRs. Release procedures live in the workflow documentation.

License

Licensed under the Apache License 2.0.