Open quantization tooling for TurboQuant-style low-bit LLM releases, stock GGUF deployment, and Apple Silicon runtime experiments.

OpenTQ

OpenTQ is an open quantization lab for low-bit weight formats. It turns quantization from an opaque "one setting for the whole model" step into an auditable release process: plan tensor allocations, generate artifacts, run runtime gates, publish evidence, and keep the claims tied to reproducible files.

The current flagship public artifact is the stock-compatible Qwen3.6-27B-OTQ-GGUF release. It uses standard GGUF tensor types, so users can run it with stock llama.cpp, while OpenTQ controls the tensor-family allocation policy and validation harness.

---

Same-task practical mini-subset for Qwen/Qwen3.6-27B: BF16 157/232, Q3_K_M 154/232, Q4_K_M 155/232, Q5_K_M 155/232. This is a release regression signal, not a leaderboard replacement.

At A Glance

• Stock GGUF track: Q3_K_M, Q4_K_M, and Q5_K_M for Qwen/Qwen3.6-27B; no custom runtime required.
• Transparent allocation: norms/state remain high precision; projection-heavy families absorb most compression.
• Practical quality checks: paired BF16-vs-GGUF mini-subsets with pinned task IDs and public reproducibility data.
• Runtime gates: local Apple Silicon checks with llama.cpp/Metal, bounded generation, release evals, and 8K prefill/decode measurements.
• Custom policies: users can define their own dynamic allocation policy with YAML/JSON via --policy-file.
• Run monitor: long quantization jobs can be watched through the terminal dashboard and machine-readable status output.
• Research runtime tracks: native OpenTQ payloads and compressed-domain runtime work are tracked separately from stock GGUF releases.

Practical Workflows

Workflow	Command / Doc	What It Gives You
List built-in allocation profiles	uv run opentq dynamic-gguf-profiles	discover the bundled Q3/Q4/Q5 dynamic GGUF policies
Generate a stock-compatible plan	uv run opentq dynamic-gguf-plan --profile OTQ-DYN-Q4_K_M ...	plan.json, tensor-types.txt, annotated tensor map, runnable quantize.sh
Use a custom allocation policy	uv run opentq dynamic-gguf-plan --policy-file policies/qwen36-custom-dyn-q4.yaml ...	define where precision is spent without editing OpenTQ source
Watch a long run	uv run opentq monitor --root artifacts/qwen3.6-27b --watch	terminal dashboard for active profile, tensor, category progress, and release state
Read the full guide	docs/cookbook.md	end-to-end examples for profiles, policies, monitoring, validation, and release evidence

OpenTQ's stock-compatible path supports custom allocation policies, not arbitrary new GGUF kernels. You can choose which tensor families use F16, Q3_K, Q4_K, Q5_K, Q6_K, Q8_0, etc.; the resulting GGUF still relies on tensor types supported by llama.cpp.

Public Release Matrix

Track	Repo / Artifact	Runtime	Status	Use It For
Stock GGUF	zlaabsi/Qwen3.6-27B-OTQ-GGUF	stock llama.cpp	public	local text inference with standard GGUF loaders
Reproducibility dataset	zlaabsi/Qwen3.6-27B-OTQ-GGUF-benchmarks	JSON/CSV assets	public	pinned BF16-vs-GGUF samples, raw outputs, reports
BF16 sidecar	zlaabsi/opentq-qwen36-bf16-sidecar	HF Jobs H200 output	public	matching BF16 baseline for the practical subset
Packed .otq	local Qwen3.6-27B-OTQ-Packed artifacts	OpenTQ runtime probes	gated locally	custom-runtime integration and parity testing
Custom OpenTQ GGUF	local Qwen3.6-27B-OTQ-TQ3_SB4-Metal.gguf	patched llama.cpp Metal path	gated locally	native Metal runtime experiments

Qwen3.6-27B GGUF Variants

File	Quant	Size	Recommended Target	Role
Qwen3.6-27B-OTQ-DYN-Q3_K_M.gguf	Q3_K_M	13.48 GiB	32 GB Apple Silicon first pick	compact local run
Qwen3.6-27B-OTQ-DYN-Q4_K_M.gguf	Q4_K_M	16.82 GiB	32 GB with care; 48 GB+ preferred	balanced default
Qwen3.6-27B-OTQ-DYN-Q5_K_M.gguf	Q5_K_M	19.92 GiB	48 GB+ preferred; measured on M1 Max 32 GB with tight headroom	quality-first local run

Quick download:

hf download zlaabsi/Qwen3.6-27B-OTQ-GGUF \
  Qwen3.6-27B-OTQ-DYN-Q4_K_M.gguf \
  --local-dir models/Qwen3.6-27B-OTQ-GGUF

Run with stock llama.cpp:

llama-cli \
  -m models/Qwen3.6-27B-OTQ-GGUF/Qwen3.6-27B-OTQ-DYN-Q4_K_M.gguf \
  -p "Explain OpenTQ in three concise bullet points." \
  -ngl 999

Allocation Transparency

OpenTQ does not apply a flat quantization recipe. It assigns standard GGUF tensor types per tensor family, then publishes the allocation map so users can inspect where precision was spent.

The compact profile spends fewer bits on bulk projection tensors, while normalization, state, embeddings, self-attention anchors, and output-sensitive tensors stay higher precision. The goal is not just a smaller file; it is a quantization policy that can be audited.

Variant	Mapped Tensors	F16	Q3_K	Q4_K	Q5_K	Q6_K	Q8_0
Q3_K_M	851	353	180	252	65	1	0
Q4_K_M	851	353	0	180	237	80	1
Q5_K_M	851	353	0	0	180	237	81

Quality Signal

The paired subset uses the same pinned task IDs, prompt format qwen3-no-think, deterministic decoding, and local scoring rules for BF16 and the GGUF artifacts.

Metric	BF16	Q3_K_M	Q4_K_M	Q5_K_M
Practical mini-subset	157/232	154/232	155/232	155/232
Aggregate score	67.7%	66.4%	66.8%	66.8%
Delta vs BF16	baseline	-1.3 pp	-0.9 pp	-0.9 pp

Important boundary: this is a paired release-regression signal. Official Qwen full-harness scores remain the capability baseline; the mini-subset should not be used as a leaderboard claim.

Runtime And Release Gates

Variant	Prefill Gate	Decode Gate	Release Eval	Measured Hardware
Q3_K_M	107.09 tok/s	10.19 tok/s	passed	M1 Max 32 GB
Q4_K_M	106.98 tok/s	9.62 tok/s	passed	M1 Max 32 GB
Q5_K_M	93.94 tok/s	8.87 tok/s	passed	M1 Max 32 GB, tight headroom

Use Q3_K_M first on 32 GB Macs. Use Q4_K_M for the best balance. Use Q5_K_M when quality is worth the extra memory pressure.

What This Repo Contains

Area	Files	Purpose
Quantizer core	src/opentq/	codebooks, rotations, tensor quantization, packing, CLI
Release scripts	scripts/	GGUF staging, HF release reports, runtime checks
Benchmarks	benchmarks/	pinned benchmark matrix, paired BF16-vs-GGUF summaries
Docs	docs/	architecture, format notes, benchmark methodology
Tests	tests/	unit tests and release tooling checks

Format Family

Variant	Intent	Notes
TQ1_0	minimum footprint	ternary / near-ternary baseline
TQ2_0	very low memory	2-bit scalar path
TQ3_SB4	compact general-purpose	3-bit WHT with four sub-block scales
TQ4_SB2	balanced 16 GiB path	4-bit WHT with two sub-block scales
TQ4_SB4	daily driver	4-bit WHT with four sub-block scales
TQ4R2	quality-first 6-bit total	4+2 residual quantization
TQ4R4	near-lossless 8-bit total	4+4 residual quantization
TQ4_BAL_V2	dense-hybrid mixed profile	model-aware flagship recipe built around TQ4_SB2 + TQ4R2
TQ_MIX_MOE	MoE-aware release profile	tensor-role-aware mixed precision

SB4 means "4 sub-block scales per block". The naming is deliberate: it describes the format instead of inheriting an opaque revision suffix.

Developer Quick Start

uv sync
uv run opentq variants
uv run opentq recipe qwen3.6-27b --format markdown
uv run opentq inventory --model-id Qwen/Qwen3.6-27B
uv run opentq dynamic-gguf-profiles

Create a stock-compatible GGUF allocation plan:

uv run opentq dynamic-gguf-plan \
  --profile OTQ-DYN-Q4_K_M \
  --output artifacts/qwen36-otq-dyn-q4-k-m \
  --llama-cpp /path/to/llama.cpp

Create a custom allocation plan from an external policy file:

uv run opentq dynamic-gguf-plan \
  --policy-file policies/qwen36-custom-dyn-q4.yaml \
  --output artifacts/qwen36-custom-q4 \
  --llama-cpp /path/to/llama.cpp

Build release reports and status pages:

uv run python scripts/build_qwen36_release_report.py
./scripts/status_qwen36_dynamic_ggufs.sh

Release Scope

OpenTQ separates stock-compatible GGUF releases from native runtime research. The public Qwen3.6-27B OTQ GGUF release uses standard llama.cpp tensor types and requires no custom OpenTQ runtime. Native packed payloads and custom Metal/runtime work are separate research tracks.

Practical mini-subsets are reported as quantization-regression signals for the released artifacts. They are not replacements for official full-harness benchmark results.

Further Reading

• Cookbook
• Architecture
• Variant naming
• Dynamic-compatible GGUF path
• Native OpenTQ runtime track
• Benchmark methodology