Glimmer HD Vision

Automatic high-resolution vision for Muse Glimmer 30B, served through an OpenAI-compatible API.

Glimmer HD Vision keeps each image inside Muse Glimmer's trained 4,096 visual-token limit. For an image above 3,211,264 pixels, it automatically sends one full overview plus four overlapping detail tiles. A representative 3840x2160 image therefore gives the model 16,680 estimated visual tokens across five views without changing the model or projector.

What it adds

• Automatic HD mode: 1080p remains one image; 4K becomes overview + overlapping 2x2 tiles.
• OCR/layout mode: labels tiles, injects an exact-transcription/layout preset, and guarantees at least 4,096 completion tokens.
• OpenAI compatibility: transforms /v1/chat/completions and transparently forwards other routes.
• Streaming: forwards server-sent events line by line through [DONE].
• Bounded expansion: at most two high-resolution source images are tiled per request.
• Pinned remote fetch: validates public DNS once, connects to that IP, preserves Host/TLS SNI, and refuses redirects.
• Process supervision: starts the model server on a private ephemeral port, propagates graceful shutdown, and uses Linux parent-death signaling to avoid orphan GPU workers.
• Observable contract: response headers report the applied mode, source count, expanded count, and estimated visual tokens.

No model weights, private runtime state, or patched llama.cpp source are included.

How it works

flowchart LR
    C[OpenAI client] -->|chat request| P[Glimmer HD Vision]
    P --> D{image above
3,211,264 px?}
    D -->|no| S[single image]
    D -->|yes| T[overview + 4 overlap tiles]
    S --> L[private llama-server child]
    T --> L
    L -->|JSON or SSE| P
    P -->|response + X-Glimmer headers| C

Loading

The proxy preserves the original overview and inserts four 10%-overlap crops labeled top-left, top-right, bottom-left, and bottom-right. Muse's projector independently resizes every view to at most 4,096 merged visual tokens. See Architecture.

Recommended profile

The validated 24 GB profile is:

Component	Setting
Target	Muse Glimmer 30B UD-Q4_K_XL
Projector	Q8_0
Context	65,536
Batch / ubatch	8,192 / 1,024
K/V cache	Q8_0 / Q8_0
Visual cap	4,096 merged tokens per image
Speculative decoding	Off for the vision-first profile
Default media policy	Auto overview + tiles

Q8 projector selection is precision insurance, not a claimed quality uplift. In the five-row local comparison it matched k-quant quality, used 618 MiB more peak VRAM, and retained 4,274 MiB minimum free VRAM. BF16 added another 1,842 MiB without quality gain. Full results and caveats are in Evidence.

Quick start

1. Install

git clone https://github.com/HawgAuto/Glimmer-HD-Vision.git
cd Glimmer-HD-Vision
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e .

2. Get Muse Glimmer and the Q8 projector

Install the Hugging Face CLI, then download only the required artifacts:

python -m pip install -U huggingface_hub
hf download unsloth/Muse-Glimmer-30B-GGUF \
  --local-dir models/Muse-Glimmer-30B-GGUF \
  --include 'Muse-Glimmer-30B-UD-Q4_K_XL.gguf' \
  --include 'mmproj-Muse-Glimmer-30B-Q8_0.gguf'

The model and its artifacts are distributed separately under Apache 2.0. Review the model card and usage policy before deployment.

3. Launch

Build a current Muse-capable llama.cpp, then set three paths:

export LLAMA_SERVER=/path/to/llama-server
export MODEL_PATH=$PWD/models/Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-UD-Q4_K_XL.gguf
export MMPROJ_PATH=$PWD/models/Muse-Glimmer-30B-GGUF/mmproj-Muse-Glimmer-30B-Q8_0.gguf
./examples/run-muse-glimmer.sh 8001

The public listener defaults to 127.0.0.1:8001; the child is bound to a random private loopback port. Backend-specific library paths and GPU selection remain the operator's responsibility. The profile was exercised against llama.cpp commit dd1ea524.

4. Send an image

python examples/request.py screenshot.png 'Describe every visible UI element and read all text.'
python examples/request.py scan.png 'Transcribe this page exactly.' --mode ocr_layout

Request modes

Set the root-level glimmer_vision_mode field:

Mode	Behavior
auto	Tile images above the threshold; keep smaller images single. Default.
single	Disable tiling.
detail	Apply the detail policy to over-threshold images.
ocr_layout	Detail tiling plus OCR/layout instructions and max_tokens >= 4096.

A request may lower the expansion cap with glimmer_max_tiled_images, but cannot exceed the server's --max-tiled-images value.

{
  "model": "muse-glimmer-30b",
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "Transcribe this page exactly."},
      {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
    ]
  }],
  "glimmer_vision_mode": "ocr_layout",
  "max_tokens": 512
}

The proxy consumes its custom fields before forwarding the request. In this example, the child receives max_tokens: 4096.

Response headers

• X-Glimmer-Vision-Mode
• X-Glimmer-Source-Images
• X-Glimmer-Expanded-Images
• X-Glimmer-Visual-Tokens-Estimated

For a typical 4K auto request, the contract is auto / 1 / 5 / 16680.

Integrations

• Hermes Agent
• Any OpenAI-compatible client that sends image_url content blocks
• Service managers and model routers that can launch a child command containing exactly one {CHILD_PORT} placeholder

Tests

python -m unittest discover -s tests -v
bash -n examples/run-muse-glimmer.sh

The suite covers geometry, mode handling, request expansion, limits, OCR budget repair, custom-field removal, real HTTP proxying, SSE forwarding, child readiness, graceful shutdown, and orphan prevention.

Security

This is an unauthenticated local inference proxy. Keep it on loopback or a trusted network. Do not expose it directly to the public internet. Read SECURITY.md, especially the distinction between the proxy's public-URL prefetch checks and URL handling by the downstream model server.

Project status

The media geometry, request transformation, streaming, and process lifecycle were production-exercised on Linux/CUDA. The public release additionally hardens remote fetches against redirects and DNS rebinding. Other operating systems and backends may work, but Linux process supervision is the tested contract.

License and attribution

Glimmer HD Vision is licensed under Apache 2.0. Muse Glimmer, Unsloth GGUF artifacts, Pillow, and llama.cpp are independent projects with their own licenses and policies. See NOTICE.