Run Muse Glimmer 30B Locally with llama.cpp

Muse Glimmer 30B ships with full-precision and quantized open weights. This guide uses Meta's official GGUF repository and keeps the text model, perception projector, and speculative-decoding drafter separate so you can test each capability deliberately.

1. Choose the artifact for your hardware

Meta publishes two quantized text-model builds:

• muse-glimmer-30B-kquant-17gb.gguf targets a 24 GB memory envelope.
• muse-glimmer-30B-kquant-dynamic.gguf targets a 32 GB memory envelope and has a smaller vendor-reported quality change.
• Full-precision weights target 64 GB VRAM according to Meta's model card.

The target envelope must also hold the KV cache and any optional components. Do not assume a 131K context will fit merely because the model weights load.

2. Confirm runtime support before downloading

Muse Glimmer uses a new muse_glimmer architecture. Update llama.cpp or another compatible runtime, then confirm that its current release notes or model list includes Muse Glimmer. Run the local binary's --help command and use the flags documented by that exact build.

The GGUF repository contains four artifacts with different roles:

3. Download only the files you need

Install the official Hugging Face CLI, then download the smaller text build:

python -m pip install --upgrade huggingface_hub
hf download meta-models/Muse-Glimmer-30B-GGUF \
  muse-glimmer-30B-kquant-17gb.gguf \
  --local-dir ./muse-glimmer

For multimodal tests, download the perception projector separately:

hf download meta-models/Muse-Glimmer-30B-GGUF \
  mmproj-kquant.gguf \
  --local-dir ./muse-glimmer

Add dflash-kquant.gguf only after confirming that your runtime supports Meta's released DFlash drafter path.

4. Run a bounded text smoke test

Start with an 8K or similarly conservative context instead of allocating the advertised maximum immediately. With a current llama.cpp build, the basic shape is:

llama-cli \
  -m ./muse-glimmer/muse-glimmer-30B-kquant-17gb.gguf \
  -c 8192 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 64 \
  -p "Reasoning strength: medium. Explain how you would recover from a failed tool call without repeating side effects."

Meta recommends temperature 1.0, top-p 0.95, and top-k 64. Reasoning strength can be set to low, medium, high, or xhigh in the system prompt. Use high or xhigh only when the task benefit justifies extra latency and output tokens.

5. Add image input as a separate test

The text-model GGUF alone does not enable vision. Load mmproj-kquant.gguf through the runtime's multimodal projector option. Current llama.cpp documentation exposes this as --mmproj; multimodal CLI names and image flags can change, so verify them with the installed build before copying a production command.

Test screenshots, charts, scanned documents, and multiple aspect ratios. Record image preprocessing, visual token count, prompt-processing time, peak memory, and answer quality.

6. Benchmark the complete agent scaffold

Measure more than tokens per second:

• successful completion of the full task
• valid tool-call schemas and argument accuracy
• recovery without duplicating side effects
• prompt-injection resistance and permission boundaries
• context-window recall at several positions
• quality differences between full precision and the selected quantization
• peak memory, time to first token, generation speed, and thermal throttling

Meta reports DFlash speedups on its tested hardware, but those figures depend on its released drafter, runtime integration, greedy decoding, and prompt mix. Reproduce performance on the device and workload you will operate.

7. Put irreversible actions behind approval

Local inference improves data control; it does not make an agent automatically safe. Keep filesystem, shell, browser, credentials, messaging, and production access disabled by default. Validate every tool call and require human confirmation before purchases, external messages, account changes, secrets access, destructive operations, or production writes.

Official sources

• Meta Muse Glimmer model card
• Meta Muse Glimmer GGUF artifacts
• Hugging Face CLI download documentation
• llama.cpp CLI documentation
• llama.cpp multimodal documentation