Skip to content
BranchesTags

Repository files navigation

Homeassistant Voice LLMs

Docker setup for experimenting with various TTS and STT models. Each container is setup to run their respective models with a small Python shim on top exposing them via the Wyoming protocol so Homeassistant can talk to them.

This setup is designed to run on AMD GPUs (ROCm). Specifically the gfx1151 (Strix Halo) series, but it can easily be modified to run on most other modern AMD GPUs.

Currently daily driving: Parakeet v3 for STT and Kokoro for TTS

Note

Great primer on everything voice AI: https://github.com/mahimairaja/voiceai

Features

  • Multiple STT engines - Whisper, Moonshine, Parakeet, Granite, Gemma, and Voxtral
  • Multiple TTS engines - Qwen3, Chatterbox Turbo, Pocket, and Kokoro
  • ROCm GPU acceleration for AMD GPUs (where applicable)
  • Wyoming Protocol for easy Home Assistant integration

Services

Speech-to-Text (STT)

  • wyoming-whisper - Speech-to-Text on port 10300 (CTranslate2 + Whisper)
  • wyoming-moonshine - Real-time STT on port 10302 (Moonshine ONNX, CPU-only, ultra-low latency)
  • wyoming-parakeet - STT on port 10303 (NVIDIA NeMo parakeet-tdt-0.6b-v3, GPU-accelerated)
  • wyoming-granite - STT on port 10304 (IBM granite-speech-4.1-2b-nar, GPU-accelerated)
  • wyoming-gemma-asr - STT on port 10305 (lightweight Wyoming shim for external llama.cpp Gemma audio server)
  • wyoming-voxtral (Not working yet) - Real-time STT on port 10301 (vLLM + Mistral Voxtral, <500ms latency)

Text-to-Speech (TTS)

  • wyoming-qwen-tts - Qwen3 TTS on port 10200 (GPU-accelerated, voice instructions)
  • wyoming-chatterbox-turbo - Chatterbox Turbo on port 10201 (GPU-accelerated, sub-200ms latency)
  • wyoming-pocket-tts - Pocket TTS on port 10202 (CPU-only, ultra-low latency)
  • kokoro-fastapi-rocm - Kokoro-FastAPI on port 8880 (ROCm model server, OpenAI-compatible TTS API, and web UI)
  • wyoming-kokoro-tts - Kokoro TTS on port 10203 (Wyoming shim that proxies to kokoro-fastapi-rocm for Home Assistant)

Prerequisites

  • AMD GPU (i.e. gfx1151 - Radeon 8060S from Ryzen AI Max 395+)
  • ROCm drivers installed on host (version 7.1.1)
  • Docker and Docker Compose
  • ~5GB VRAM for Whisper medium model
  • ~20GB disk space for Docker images

Installation

1. Find Your GPU Architecture

rocminfo | grep "gfx"

You'll see output like gfx1100, gfx1030, gfx906, etc.

2. Configure GPU Architecture

Create the .env file based on the .env.example and set your GPU architecture override:

The default is gfx1151 corresponding to 11.5.1 (for RDNA 3.5 / Radeon 8060S).

# For RDNA 3 (RX 7000 series): gfx1100, gfx1101, gfx1102
HSA_OVERRIDE_GFX_VERSION=11.0.0
# For RDNA 2 (RX 6000 series): gfx1030, gfx1031, gfx1032
HSA_OVERRIDE_GFX_VERSION=10.3.0
# For RDNA (RX 5000 series): gfx1010, gfx1012
HSA_OVERRIDE_GFX_VERSION=10.1.0
# For Vega: gfx900, gfx906
HSA_OVERRIDE_GFX_VERSION=9.0.0

3. Build and Run

docker compose up -d

STT Configuration

  Parakeet (Recommended)

Available environment variables:

  • PARAKEET_MODEL - HuggingFace model ID (default: nvidia/parakeet-tdt-0.6b-v3)
  • PARAKEET_DEVICE - cuda:0 (GPU) or cpu
  • PARAKEET_DEBUG - true/false

Features:

  • NVIDIA NeMo TDT (Token-and-Duration Transducer) architecture
  • GPU-accelerated via ROCm/PyTorch
  • 0.6B parameter model, good accuracy with moderate VRAM usage

  Gemma Audio

Gemma uses a lightweight Wyoming shim that sends captured WAV audio to an existing llama.cpp/OpenAI-compatible server via /v1/audio/transcriptions.

Available environment variables:

  • GEMMA_API_URL - OpenAI-compatible llama.cpp base URL (default: https://llama-dash.puff.lan/v1)
  • GEMMA_API_KEY - Optional bearer token for the llama.cpp proxy
  • GEMMA_MODEL - Model ID exposed by /v1/models (default: gemma-4-12B-fast)
  • GEMMA_PROMPT - Transcription instruction prompt
  • GEMMA_TIMEOUT - API timeout in seconds (default: 600)
  • GEMMA_TEMPERATURE - Sampling temperature (default: 0)
  • GEMMA_MAX_TOKENS - Maximum transcript tokens (default: 128)
  • GEMMA_VERIFY_SSL - true/false TLS verification (default: false for self-signed LAN certs)
  • GEMMA_DEBUG - true/false

  Whisper

Available environment variables:

  • WHISPER_MODEL - Model size: tiny, base, small, medium (default), large
  • WHISPER_COMPUTE_TYPE - float16 (default), int8
  • WHISPER_BEAM_SIZE - 1-10, default 5 (higher = better quality, slower)
  • WHISPER_DEBUG - true/false

Model sizes and VRAM requirements:

  • tiny: ~1GB VRAM, fastest, good for simple commands
  • base: ~1.5GB VRAM, balanced
  • small: ~2GB VRAM, better accuracy
  • medium: ~5GB VRAM, high accuracy (default)
  • large: ~10GB VRAM, best accuracy, slower

  Moonshine

Available environment variables:

  • MOONSHINE_MODEL - Model name: moonshine/tiny (default, 27M params) or moonshine/base (62M params)
  • MOONSHINE_DEBUG - true/false

Features:

  • Designed for live speech recognition with ultra-low latency
  • CPU-only - no GPU required, lightweight Docker image
  • Supports 8 languages: en, ar, zh, ja, ko, es, uk, vi

  Voxtral

Not fully working yet

Available environment variables:

  • VOXTRAL_MODEL - Model ID (default: mistralai/Voxtral-Mini-4B-Realtime-2602)
  • VOXTRAL_LANGUAGE - Default language: en, es, fr, de, it, pt, ru, zh, ja, ko, ar, hi, nl
  • VOXTRAL_GPU_MEMORY - GPU memory utilization 0.0-1.0 (default: 0.9)
  • VOXTRAL_DEBUG - true/false

Features:

  • Real-time streaming transcription with <500ms latency
  • Supports 13 languages with automatic language detection
  • Powered by vLLM for efficient inference
  • Requires ≥16GB GPU memory

Requirements:

  • Minimum VRAM: 16GB
  • Model Size: ~4B parameters (BF16)
  • Throughput: >12.5 tokens/second

TTS Configuration

  Kokoro TTS (Recommended)

Kokoro uses two containers:

  • kokoro-fastapi-rocm runs Kokoro-FastAPI with ROCm and serves the actual Kokoro model on port 8880.
  • wyoming-kokoro-tts is a lightweight Wyoming protocol shim on port 10203 so Home Assistant can use the same Kokoro model.

The FastAPI container is useful outside Home Assistant too. It exposes Kokoro-FastAPI's web UI and OpenAI-compatible TTS endpoint, so other tools like OpenWebUI can call http://your-docker-host:8880/v1/audio/speech directly. The Wyoming shim talks to the same API internally via http://kokoro-fastapi-rocm:8880/v1.

  • KOKORO_API_URL - Kokoro-FastAPI endpoint used by the Wyoming shim (default: http://kokoro-fastapi-rocm:8880/v1)
  • KOKORO_FASTAPI_PORT - Host port for the Kokoro-FastAPI web UI and OpenAI-compatible API (default: 8880)
  • KOKORO_MIOPEN_FIND_MODE - MIOpen tuning mode for Kokoro-FastAPI (default: 2; use 3 temporarily for first-run tuning)
  • KOKORO_VOICE - Voice selection (see options below)
  • KOKORO_SPEED - Speech speed, 0.5-2.0 (default: 1.0)
  • KOKORO_TIMEOUT - API request timeout in seconds (default: 30)
  • KOKORO_DEBUG - true/false

Voice Options:

  • Female American: af_bella, af_sarah, af_sky
  • Male American: am_adam, am_michael
  • Female British: bf_emma, bf_isabella
  • Male British: bm_george, bm_lewis

Voice Mixing:

  • Simple: af_bella+af_sky (equal mix)
  • Weighted: af_bella(2)+af_sky(1) (2:1 ratio)

Features:

  • Multi-language support (en, ja, zh, ko, fr, es)
  • Direct OpenAI-compatible TTS API and web UI on kokoro-fastapi-rocm
  • Home Assistant integration through the separate wyoming-kokoro-tts shim
  • Voice changes require container restart

   Qwen3-TTS

  • QWEN_MODEL - Model choice (default: Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign)
  • QWEN_VOICE_INSTRUCT - Text description of desired voice
  • QWEN_LANGUAGE - Language selection (Auto, Chinese, English, Japanese, etc.)
  • QWEN_DEVICE - cuda:0 (GPU) or cpu
  • QWEN_DTYPE - bfloat16 (default), float16, float32, int8, int4
  • QWEN_FLASH_ATTENTION - true/false
  • QWEN_DEBUG - true/false

   Chatterbox Turbo

  • CHATTERBOX_DEVICE - cuda:0 (GPU) or cpu
  • CHATTERBOX_SAMPLES_PER_CHUNK - Audio streaming chunk size (default: 1024)
  • CHATTERBOX_DEBUG - true/false

Requires HF_TOKEN for gated model access.

  Pocket TTS

  • POCKET_VOICE - Built-in voices: alba, marius, javert, jean, fantine, cosette, eponine, azelma
  • POCKET_DEBUG - true/false

CPU-only, ultra-low latency (~200ms to first audio chunk).

Home Assistant Integration

  1. Go to Settings → Devices & Services → Add Integration
  2. Search for "Wyoming Protocol"
  3. Add each service separately:
    • Whisper: Host = your-docker-host, Port = 10300
    • Moonshine: Host = your-docker-host, Port = 10302
    • Parakeet: Host = your-docker-host, Port = 10303
    • Gemma Audio: Host = your-docker-host, Port = 10305
    • Voxtral: Host = your-docker-host, Port = 10301
    • Qwen3-TTS: Host = your-docker-host, Port = 10200
    • Chatterbox Turbo: Host = your-docker-host, Port = 10201
    • Pocket TTS: Host = your-docker-host, Port = 10202
    • Kokoro TTS: Host = your-docker-host, Port = 10203
  4. Configure your voice assistant pipeline in Settings → Voice Assistants

Resources

Wyoming & STT

TTS Engines

ROCm

Licenses

  • Whisper: MIT License
  • CTranslate2: MIT License
  • Wyoming: MIT License
  • faster-whisper: MIT License
  • Moonshine: MIT License
  • Parakeet TDT: Apache 2.0 License
  • NVIDIA NeMo: Apache 2.0 License
  • Voxtral: Apache 2.0 License
  • vLLM: Apache 2.0 License
  • Qwen3-TTS: Apache 2.0 License
  • Chatterbox Turbo: Apache 2.0 License
  • Pocket TTS: Apache 2.0 License
  • Kokoro-82M: Apache 2.0 License

About

Experimental support for many TTS/STT LLMs wrapped in a Wyoming API for consumption via Homeassistant

Topics

homeassistantpiperrocmstrix-halowhisper

Resources

Readme
Activity

Stars

37 stars

Watchers

0 watching

Forks

5 forks
Report repository

Contributors

Languages