feat: Add XTTS v2 support, refactor Docker/GPU infra, and improve Piper engine
- Add XTTS v2 configuration to .env.example - Refactor Dockerfile to multi-stage build with CUDA 12.1 support - Update Makefile with Kokoro and XTTS test environment targets - Refactor Piper engine (app/engines/piper.py) to use python module execution - Add comprehensive documentation for Kokoro and XTTS plans - Add helper scripts and patches for build process
This commit is contained in:
72
CLAUDE.md
72
CLAUDE.md
@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
|
||||
|
||||
## Project Overview
|
||||
|
||||
AudioEngineHub is a local-first, modular, multi-engine Text-to-Speech (TTS) server built with FastAPI. It provides a unified API to interact with various TTS engines (Piper, StyleTTS, ChatTTS, F5-TTS) with caching, audio format conversion, and Docker deployment support.
|
||||
AudioEngineHub is a local-first, modular, multi-engine Text-to-Speech (TTS) server built with FastAPI. It provides a unified API to interact with various TTS engines (Piper, Kokoro, StyleTTS, ChatTTS, F5-TTS) with caching, audio format conversion, and Docker deployment support.
|
||||
|
||||
## Architecture
|
||||
|
||||
@ -14,7 +14,7 @@ AudioEngineHub is a local-first, modular, multi-engine Text-to-Speech (TTS) serv
|
||||
- Uses `create_app()` factory pattern to dynamically load engines based on configuration
|
||||
- Maintains `app.ENGINE_REGISTRY` dict mapping engine names to instantiated engine objects
|
||||
- Engines are loaded from `settings.ACTIVE_ENGINES` at startup
|
||||
- All available engines defined in `ALL_ENGINES` dict (lines 34-39)
|
||||
- All available engines defined in `ALL_ENGINES` dict (lines 35-41)
|
||||
- Startup event checks for models directory availability with retry logic (lines 192-213)
|
||||
|
||||
**Engine Architecture (`app/engines/`)**
|
||||
@ -206,12 +206,80 @@ Engine implementations use absolute path `/models/piper/` not relative paths. Th
|
||||
|
||||
**Currently Working:**
|
||||
- `piper`: Functional, uses real Piper TTS executable with ONNX models
|
||||
- `kokoro`: Fully functional, uses Kokoro-82M TTS library with 54 voices across 8 languages
|
||||
|
||||
**Dummy Implementations (for testing only):**
|
||||
- `styletts`: Returns hardcoded model/voice lists, generates dummy audio
|
||||
- `chattts`: Dummy implementation
|
||||
- `f5-tts`: Implemented but may be inactive by default
|
||||
|
||||
### Kokoro Engine (`app/engines/kokoro.py`)
|
||||
|
||||
**Overview:**
|
||||
- Uses `kokoro` Python library (KPipeline) for high-performance TTS synthesis
|
||||
- 82M parameter model delivering ~90× real-time performance on RTX 3090 Ti
|
||||
- 54 voices across 8 languages: EN-US, EN-GB, FR, ES, JA, ZH, IT, PT, HI, KO
|
||||
- Outputs 24kHz audio natively (WAV), converts to OGG/MP3 via ffmpeg
|
||||
- Model size: ~200MB per language, auto-downloaded from Hugging Face on first use
|
||||
- Voice metadata: `app/engines/kokoro_voices.py` contains all 54 voices with metadata
|
||||
|
||||
**Architecture:**
|
||||
```python
|
||||
class KokoroEngine(TTSEngineBase):
|
||||
def __init__(self):
|
||||
# GPU/CPU detection via settings.KOKORO_DEVICE
|
||||
# Pipeline instances cached per language code
|
||||
|
||||
async def synthesize(text, speaker, model, fmt):
|
||||
# Uses KPipeline for synthesis (24kHz native output)
|
||||
# Format conversion via ffmpeg (_run_ffmpeg_blocking)
|
||||
# Returns temp file path to generated audio
|
||||
|
||||
def list_models(self):
|
||||
# Returns 10 language models:
|
||||
# kokoro-en-us, kokoro-en-gb, kokoro-fr, kokoro-es,
|
||||
# kokoro-ja, kokoro-zh, kokoro-it, kokoro-pt, kokoro-hi, kokoro-ko
|
||||
|
||||
def list_voices(self, model=None):
|
||||
# Returns all 54 voices or filtered by language
|
||||
# Uses get_voices_for_model() from kokoro_voices.py
|
||||
```
|
||||
|
||||
**Language Code Mapping:**
|
||||
The engine maps model names to Kokoro's internal language codes:
|
||||
- `kokoro-en-us` → `'a'` (American English)
|
||||
- `kokoro-en-gb` → `'b'` (British English)
|
||||
- `kokoro-fr` → `'fr'` (French)
|
||||
- `kokoro-es` → `'es'` (Spanish)
|
||||
- `kokoro-ja` → `'ja'` (Japanese)
|
||||
- `kokoro-zh` → `'zh'` (Chinese)
|
||||
- `kokoro-it` → `'it'` (Italian)
|
||||
- `kokoro-pt` → `'pt'` (Portuguese)
|
||||
- `kokoro-hi` → `'hi'` (Hindi)
|
||||
- `kokoro-ko` → `'ko'` (Korean)
|
||||
|
||||
**Voice Organization (`app/engines/kokoro_voices.py`):**
|
||||
- All 54 voices documented with metadata (gender, language, description)
|
||||
- Naming convention: `{language}{gender}_{name}` (e.g., `af_bella`, `am_adam`)
|
||||
- Popular voices: `af_bella`, `af_sarah`, `af_sky`, `am_adam`, `am_michael`
|
||||
- Helper functions: `get_voices_for_model()`, `get_voice_info()`
|
||||
|
||||
**Model Download & Caching:**
|
||||
- Models auto-download from Hugging Face on first synthesis
|
||||
- Cached in `~/.cache/huggingface/` (inside container)
|
||||
- First synthesis may take 30-60s due to download + compilation
|
||||
- Subsequent syntheses are fast (~90× real-time on GPU)
|
||||
|
||||
**Configuration (`app/config.py`):**
|
||||
- `KOKORO_DEVICE`: "cuda" or "cpu" (default: "cuda")
|
||||
- `KOKORO_TIMEOUT_SECONDS`: Synthesis timeout (default: 30)
|
||||
|
||||
**Error Handling:**
|
||||
- Applies all bug fixes from Piper engine (timeouts, temp file cleanup, logging)
|
||||
- Graceful GPU fallback if CUDA unavailable
|
||||
- Voice validation before synthesis
|
||||
- Comprehensive error logging with context
|
||||
|
||||
### Debugging Tips
|
||||
|
||||
**Port Conflicts**: The Makefile automatically finds free ports starting from 8000. Run with `sudo make dev-up` for most reliable port detection.
|
||||
|
||||
Reference in New Issue
Block a user