feat: Add XTTS v2 support, refactor Docker/GPU infra, and improve Piper engine

- Add XTTS v2 configuration to .env.example
- Refactor Dockerfile to multi-stage build with CUDA 12.1 support
- Update Makefile with Kokoro and XTTS test environment targets
- Refactor Piper engine (app/engines/piper.py) to use python module execution
- Add comprehensive documentation for Kokoro and XTTS plans
- Add helper scripts and patches for build process
This commit is contained in:
2025-12-13 11:37:58 +01:00
parent d6d1fe9d23
commit 91887ae296
19 changed files with 1625 additions and 46 deletions

View File

@ -32,3 +32,7 @@ docker-compose.yml
# Local environment settings
.env
# Exclude unused large models
app/models/chattts/
app/models/f5-tts-voices/

View File

@ -1,6 +1,6 @@
# Comma-separated list of engines to activate.
# Available options (potentially): piper, styletts, f5_tts, chattts
ACTIVE_ENGINES='["piper", "styletts"]'
# Available options (potentially): piper, styletts, f5_tts, chattts, kokoro, xtts
ACTIVE_ENGINES='["piper", "styletts", "xtts"]'
# Server configuration
HOST=0.0.0.0
@ -9,3 +9,12 @@ PORT=8000
# Piper Engine Configuration
PIPER_TIMEOUT_SECONDS=30
FFMPEG_TIMEOUT_SECONDS=60
# Kokoro Engine Configuration
KOKORO_DEVICE=cuda # cuda or cpu
KOKORO_TIMEOUT_SECONDS=30
# XTTS Engine Configuration
XTTS_ACCEPT_LICENSE=false # Set to true to accept the Coqui XTTS license
XTTS_DEVICE=cpu # cuda or cpu
VOICES_DIR=app/asset/voices # Directory where reference audio files for voice cloning are stored

7
.env.kokoro-test Normal file
View File

@ -0,0 +1,7 @@
ACTIVE_ENGINES='["piper", "kokoro"]'
HOST=0.0.0.0
PORT=8000
# Kokoro Engine Configuration
KOKORO_DEVICE=cuda
KOKORO_TIMEOUT_SECONDS=30

13
.env.xtts-test Normal file
View File

@ -0,0 +1,13 @@
LOG_LEVEL=DEBUG
ACTIVE_ENGINES='["piper", "kokoro", "xtts"]'
XTTS_ACCEPT_LICENSE=true
HOST=0.0.0.0
PORT=8000
# Kokoro Engine Configuration
KOKORO_DEVICE=cuda
KOKORO_TIMEOUT_SECONDS=30
VOICES_DIR=/home/appuser/app/asset/voices

35
AGENTS.md Normal file
View File

@ -0,0 +1,35 @@
# Repository Guidelines
## Project Structure & Module Organization
- `app/` FastAPI service code: `main.py` app factory, `engines/` TTS backends (Piper, Kokoro, XTTS, F5, StyleTTS, ChatTTS), `utils/` helpers (text chunking, audio concat, cache keys), `models/` default model assets.
- `app/asset/` runtime assets and audio cache (overridable via `AUDIO_CACHE_DIR`); `asset/` holds Docker/runtime assets mounted into the container.
- `tests/` pytest suite with async API coverage and fixtures that redirect cache paths.
- `docs/` reference material; `scripts/` helper scripts (health checks, setup); `Dockerfile`, `docker-compose*.yml`, and `Makefile` drive builds and orchestration.
## Build, Test, and Development Commands
- `make dev-up` Build from local source and start the stack; auto-picks a free host port.
- `make up` Run using the latest registry image; `make down` stops Compose services; `make logs` tails the app container.
- `make test` Run pytest with coverage over `app/` and `tests/`; honors `.venv` if present.
- Local uvicorn run (outside Docker): `uvicorn app.main:app --reload --host 0.0.0.0 --port 8000`.
## Coding Style & Naming Conventions
- Python code should follow PEP 8 with 4-space indentation and explicit type hints where meaningful; keep functions small and async-aware for I/O.
- Prefer FastAPI dependency injection patterns and pydantic models for request/response validation.
- Tests, fixtures, and helpers use `test_*.py` naming; keep fixtures in `tests/conftest.py`.
- Configuration lives in `app/config.py` via pydantic settings; read values from `.env` instead of hardcoding.
## Testing Guidelines
- Use `pytest`/`pytest-asyncio`; mark async tests with `@pytest.mark.asyncio`.
- Ensure Piper models required by tests exist under `app/models/piper/` (see README instructions) or skip conditionally as in existing tests.
- When adding engines, supply health checks (`healthcheck`), model/voice listing, and synthesis tests mirroring `tests/test_api.py`.
- Keep coverage broad on `/tts`, `/engines`, `/models`, `/health`, and error paths (invalid engine/model/speaker).
## Commit & Pull Request Guidelines
- Follow the existing conventional prefix style (`feat:`, `fix:`, `docs:`, etc.) as seen in git history.
- Commits should be scoped and descriptive; avoid bundling unrelated changes.
- PRs should include: purpose and behavior summary, key test commands run (e.g., `make test`), notes on model/config prerequisites, and screenshots/log snippets only when behavior is user-visible.
## Security & Configuration Tips
- Copy `.env.example` to `.env` and avoid committing secrets or model paths tied to personal systems.
- Validate engine activation lists via `ACTIVE_ENGINES` and ensure cache directories (`AUDIO_CACHE_DIR`) are writable before running locally or in containers.
- Large model files live outside the image; mount them via Compose volumes and confirm licenses (e.g., XTTS `XTTS_ACCEPT_LICENSE`).

View File

@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Project Overview
AudioEngineHub is a local-first, modular, multi-engine Text-to-Speech (TTS) server built with FastAPI. It provides a unified API to interact with various TTS engines (Piper, StyleTTS, ChatTTS, F5-TTS) with caching, audio format conversion, and Docker deployment support.
AudioEngineHub is a local-first, modular, multi-engine Text-to-Speech (TTS) server built with FastAPI. It provides a unified API to interact with various TTS engines (Piper, Kokoro, StyleTTS, ChatTTS, F5-TTS) with caching, audio format conversion, and Docker deployment support.
## Architecture
@ -14,7 +14,7 @@ AudioEngineHub is a local-first, modular, multi-engine Text-to-Speech (TTS) serv
- Uses `create_app()` factory pattern to dynamically load engines based on configuration
- Maintains `app.ENGINE_REGISTRY` dict mapping engine names to instantiated engine objects
- Engines are loaded from `settings.ACTIVE_ENGINES` at startup
- All available engines defined in `ALL_ENGINES` dict (lines 34-39)
- All available engines defined in `ALL_ENGINES` dict (lines 35-41)
- Startup event checks for models directory availability with retry logic (lines 192-213)
**Engine Architecture (`app/engines/`)**
@ -206,12 +206,80 @@ Engine implementations use absolute path `/models/piper/` not relative paths. Th
**Currently Working:**
- `piper`: Functional, uses real Piper TTS executable with ONNX models
- `kokoro`: Fully functional, uses Kokoro-82M TTS library with 54 voices across 8 languages
**Dummy Implementations (for testing only):**
- `styletts`: Returns hardcoded model/voice lists, generates dummy audio
- `chattts`: Dummy implementation
- `f5-tts`: Implemented but may be inactive by default
### Kokoro Engine (`app/engines/kokoro.py`)
**Overview:**
- Uses `kokoro` Python library (KPipeline) for high-performance TTS synthesis
- 82M parameter model delivering ~90× real-time performance on RTX 3090 Ti
- 54 voices across 8 languages: EN-US, EN-GB, FR, ES, JA, ZH, IT, PT, HI, KO
- Outputs 24kHz audio natively (WAV), converts to OGG/MP3 via ffmpeg
- Model size: ~200MB per language, auto-downloaded from Hugging Face on first use
- Voice metadata: `app/engines/kokoro_voices.py` contains all 54 voices with metadata
**Architecture:**
```python
class KokoroEngine(TTSEngineBase):
def __init__(self):
# GPU/CPU detection via settings.KOKORO_DEVICE
# Pipeline instances cached per language code
async def synthesize(text, speaker, model, fmt):
# Uses KPipeline for synthesis (24kHz native output)
# Format conversion via ffmpeg (_run_ffmpeg_blocking)
# Returns temp file path to generated audio
def list_models(self):
# Returns 10 language models:
# kokoro-en-us, kokoro-en-gb, kokoro-fr, kokoro-es,
# kokoro-ja, kokoro-zh, kokoro-it, kokoro-pt, kokoro-hi, kokoro-ko
def list_voices(self, model=None):
# Returns all 54 voices or filtered by language
# Uses get_voices_for_model() from kokoro_voices.py
```
**Language Code Mapping:**
The engine maps model names to Kokoro's internal language codes:
- `kokoro-en-us` → `'a'` (American English)
- `kokoro-en-gb` → `'b'` (British English)
- `kokoro-fr` → `'fr'` (French)
- `kokoro-es` → `'es'` (Spanish)
- `kokoro-ja` → `'ja'` (Japanese)
- `kokoro-zh` → `'zh'` (Chinese)
- `kokoro-it` → `'it'` (Italian)
- `kokoro-pt` → `'pt'` (Portuguese)
- `kokoro-hi` → `'hi'` (Hindi)
- `kokoro-ko` → `'ko'` (Korean)
**Voice Organization (`app/engines/kokoro_voices.py`):**
- All 54 voices documented with metadata (gender, language, description)
- Naming convention: `{language}{gender}_{name}` (e.g., `af_bella`, `am_adam`)
- Popular voices: `af_bella`, `af_sarah`, `af_sky`, `am_adam`, `am_michael`
- Helper functions: `get_voices_for_model()`, `get_voice_info()`
**Model Download & Caching:**
- Models auto-download from Hugging Face on first synthesis
- Cached in `~/.cache/huggingface/` (inside container)
- First synthesis may take 30-60s due to download + compilation
- Subsequent syntheses are fast (~90× real-time on GPU)
**Configuration (`app/config.py`):**
- `KOKORO_DEVICE`: "cuda" or "cpu" (default: "cuda")
- `KOKORO_TIMEOUT_SECONDS`: Synthesis timeout (default: 30)
**Error Handling:**
- Applies all bug fixes from Piper engine (timeouts, temp file cleanup, logging)
- Graceful GPU fallback if CUDA unavailable
- Voice validation before synthesis
- Comprehensive error logging with context
### Debugging Tips
**Port Conflicts**: The Makefile automatically finds free ports starting from 8000. Run with `sudo make dev-up` for most reliable port detection.

View File

@ -1,42 +1,56 @@
# Stage 1: Builder
FROM python:3.11 as builder
# Stage 1: Builder (for heavy Python dependencies including CUDA-enabled PyTorch)
FROM nvidia/cuda:12.1.0-devel-ubuntu22.04 as builder
WORKDIR /opt/venv
# Create a virtual environment and install dependencies
COPY requirements.txt .
RUN python -m venv . && . /opt/venv/bin/activate && pip install --no-cache-dir -r requirements.txt
# Install Python 3.11 and venv in the builder stage
RUN apt-get update && apt-get install -y python3.11 python3.11-venv
# Stage 2: Runner (The final image)
# Create a virtual environment and install Python dependencies
COPY requirements.txt .
COPY patches/ ./patches/
RUN python3.11 -m venv . && . /opt/venv/bin/activate && pip install --no-cache-dir -r requirements.txt && \
chmod +x ./patches/fix-misaki-espeak.sh && ./patches/fix-misaki-espeak.sh
# Stage 2: Runner (The final slim image)
FROM python:3.11-slim
# Install system dependencies needed at runtime
# ffmpeg is required for audio conversion
# espeak-ng is required for Kokoro TTS phonemization
RUN apt-get update && apt-get install -y --no-install-recommends \
ffmpeg \
espeak-ng \
&& rm -rf /var/lib/apt/lists/*
# Create an unprivileged user
RUN useradd --create-home --shell /bin/bash appuser
# Set working directory for the application code
WORKDIR /home/appuser
# Copy the virtual environment from the builder stage
# Copy the entire virtual environment from the builder stage
COPY --from=builder /opt/venv /opt/venv
# Add venv to PATH and PYTHONPATH so packages and scripts work correctly
ENV PATH="/opt/venv/bin:$PATH"
ENV PYTHONPATH="/opt/venv/lib/python3.11/site-packages:$PYTHONPATH"
# Create espeak-ng-data symlink for misaki compatibility
RUN mkdir -p /home/runner/work/espeakng-loader/espeakng-loader/espeak-ng/_dynamic/share/ && \
ln -sf /usr/lib/x86_64-linux-gnu/espeak-ng-data /home/runner/work/espeakng-loader/espeakng-loader/espeak-ng/_dynamic/share/espeak-ng-data
# Copy the application code into the container
COPY app/ ./app
# Set the PATH to include the virtual environment's bin directory
ENV PATH="/opt/venv/bin:$PATH"
# Expose the application port
EXPOSE 8000
# Run as the unprivileged user
USER appuser
# Command to run the application using gunicorn for production
CMD ["gunicorn", "-k", "uvicorn.workers.UvicornWorker", "-w", "2", "-b", "0.0.0.0:8000", "app.main:app"]
# Command to run the application using uvicorn for development with auto-reloading
CMD ["python3.11", "-m", "uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--log-level", "debug"]

View File

@ -0,0 +1,492 @@
# Kokoro TTS Engine Implementation Plan
## Overview
Implement Kokoro-82M as a new TTS engine in AudioEngineHub. Kokoro is a lightweight, high-performance open-weight TTS model with 82 million parameters that delivers quality comparable to models 5-15× its size.
## Background Research
### Key Features
- **Size**: 82 million parameters (extremely lightweight)
- **Performance**: ~210× real-time on RTX 4090, ~90× real-time on RTX 3090 Ti
- **Quality**: Took first place in TTS Spaces Arena, outperforming XTTS v2 (467M) and MetaVoice (1.2B)
- **Audio**: 24kHz high-fidelity output
- **License**: Apache 2.0 (open-source, commercial use allowed)
- **Languages**: 8 languages (English US/UK, French, Spanish, Japanese, Chinese, Italian, Portuguese, Hindi, Korean)
- **Voices**: 54 voices available
### Sources
- [Kokoro-82M Hugging Face](https://huggingface.co/hexgrad/Kokoro-82M)
- [Kokoro TTS Official Site](https://kokorotts.net/)
- [VOICES.md - Complete Voice List](https://huggingface.co/hexgrad/Kokoro-82M/blob/main/VOICES.md)
- [GitHub Repository](https://github.com/hexgrad/kokoro)
- [Analytics Vidhya Article](https://www.analyticsvidhya.com/blog/2025/01/kokoro-82m/)
---
## Implementation Strategy
### Phase 1: Research & Setup
#### 1.1 Model Investigation
- [x] Research Kokoro TTS architecture and capabilities
- [x] Identify Python library: `kokoro>=0.9.2`
- [x] Document voice list (54 voices across 8 languages)
- [ ] Test Kokoro locally to understand API
#### 1.2 Dependency Analysis
**Required packages:**
```bash
kokoro>=0.9.2
soundfile
phonemizer
torch
transformers
scipy
munch
```
**System dependencies:**
```bash
espeak-ng # Required for phonemization
```
---
### Phase 2: Engine Implementation
#### 2.1 Create `app/engines/kokoro.py`
**Architecture:**
```python
class KokoroEngine(TTSEngineBase):
def __init__(self):
# Initialize Kokoro pipeline
# Handle GPU/CPU detection
# Load model from Hugging Face
async def synthesize(text, speaker, model, fmt):
# Generate audio using KPipeline
# Handle voice selection
# Convert to requested format (wav/ogg/mp3)
# Return temp file path
def list_models(self):
# Return available language models
# Options: 'a' (American English), 'b' (British English), etc.
def list_voices(self, model):
# Return 54 available voices
# Filter by language if model specified
def healthcheck(self):
# Check if kokoro library is available
# Verify model is loaded
# Return status
async def selftest(self):
# Run quick synthesis test
# Verify audio generation works
```
**Key Implementation Details:**
1. **Model Selection:**
- Kokoro uses `lang_code` parameter (e.g., 'a' = American English, 'b' = British English)
- Map this to "models" concept in our API
- Models: `kokoro-en-us`, `kokoro-en-gb`, `kokoro-fr`, `kokoro-es`, `kokoro-ja`, `kokoro-zh`, `kokoro-it`, `kokoro-pt`, `kokoro-hi`, `kokoro-ko`
2. **Voice Selection:**
- 54 voices available (see VOICES.md)
- Popular voices: `af_alloy`, `af_bella`, `af_sarah`, `af_sky`, `af_nova`, etc.
- Each voice has quality grade and language support
3. **Audio Generation:**
```python
from kokoro import KPipeline
pipeline = KPipeline(lang_code='a')
generator = pipeline(text, voice='af_heart')
for gs, ps, audio in generator:
# audio is numpy array at 24kHz
# Save to temp file
```
4. **Format Conversion:**
- Native output: 24kHz WAV
- Use ffmpeg (already available) for OGG/MP3 conversion
- Reuse `_run_ffmpeg_blocking()` pattern from Piper engine
5. **Error Handling:**
- Apply all bug fixes from Piper engine (timeouts, temp file cleanup, logging)
- Handle GPU out-of-memory gracefully (fallback to CPU)
- Validate voice exists before synthesis
---
### Phase 3: Docker Integration
#### 3.1 Update Dockerfile
**Add dependencies to `Dockerfile`:**
```dockerfile
# Install espeak-ng for Kokoro phonemization
RUN apt-get update && \
apt-get install -y --no-install-recommends espeak-ng && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
# Install Kokoro Python dependencies
RUN pip install --no-cache-dir \
kokoro>=0.9.2 \
soundfile \
phonemizer \
torch \
transformers \
scipy \
munch
```
#### 3.2 Model Download Strategy
**Options:**
**Option A: Download on first use (lazy loading)**
- Models auto-download from Hugging Face (~200MB per model)
- Advantage: No pre-download needed
- Disadvantage: First synthesis will be slow
**Option B: Pre-download in Docker build**
```dockerfile
# Pre-download Kokoro model during build
RUN python3 -c "from kokoro import KPipeline; KPipeline(lang_code='a')"
```
**Option C: Volume mount like Piper**
```yaml
# docker-compose.yml
volumes:
- ./models/kokoro:/home/appuser/.cache/huggingface
```
**Recommendation**: Option A for MVP, Option C for production
---
### Phase 4: Configuration
#### 4.1 Update `app/config.py`
```python
class Settings(BaseSettings):
# ... existing settings ...
# Kokoro Engine Configuration
KOKORO_DEVICE: str = "cuda" # or "cpu"
KOKORO_TIMEOUT_SECONDS: int = 30
KOKORO_DEFAULT_LANG: str = "a" # American English
```
#### 4.2 Update `.env.example`
```bash
# Kokoro Engine Configuration
KOKORO_DEVICE=cuda # cuda or cpu
KOKORO_TIMEOUT_SECONDS=30
KOKORO_DEFAULT_LANG=a
```
#### 4.3 Update `app/main.py`
```python
ALL_ENGINES = {
"piper": PiperEngine,
"styletts": StyleTTSEngine,
"chattts": ChatTTSEngine,
"f5-tts": F5TTSEngine,
"kokoro": KokoroEngine, # Add this
}
```
---
### Phase 5: Voice Metadata
#### 5.1 Create `app/engines/kokoro_voices.py`
Store voice metadata for better UX:
```python
KOKORO_VOICES = {
"af_alloy": {
"gender": "F",
"language": ["en-us"],
"quality": "high",
"description": "Clear, professional female voice"
},
"af_bella": {
"gender": "F",
"language": ["en-us", "en-gb"],
"quality": "high",
"description": "Warm, expressive female voice"
},
# ... all 54 voices
}
```
Download full voice list from: https://huggingface.co/hexgrad/Kokoro-82M/blob/main/VOICES.md
---
### Phase 6: Testing
#### 6.1 Unit Tests
Create `tests/test_kokoro_engine.py`:
```python
import pytest
from app.engines.kokoro import KokoroEngine
@pytest.mark.asyncio
async def test_kokoro_synthesis():
engine = KokoroEngine()
audio_path = await engine.synthesize(
"Hello, this is Kokoro TTS.",
speaker="af_bella",
model="kokoro-en-us",
fmt="wav"
)
assert os.path.exists(audio_path)
assert os.path.getsize(audio_path) > 0
def test_kokoro_list_voices():
engine = KokoroEngine()
voices = engine.list_voices("kokoro-en-us")
assert len(voices) > 0
assert "af_bella" in voices
def test_kokoro_healthcheck():
engine = KokoroEngine()
health = engine.healthcheck()
assert health["status"] in ["ok", "not_available"]
```
#### 6.2 Integration Tests
```bash
# Test via API
curl -X POST http://localhost:8000/tts \
-H "Content-Type: application/json" \
-d '{
"text": "Hello from Kokoro TTS!",
"engine": "kokoro",
"model": "kokoro-en-us",
"speaker": "af_bella",
"format": "ogg"
}'
```
#### 6.3 Performance Benchmarks
Test synthesis speed:
- Short text (10 words): Target <0.5s on RTX 3090
- Medium text (100 words): Target <2s on RTX 3090
- Long text (1000 words): Target <15s on RTX 3090
---
### Phase 7: Documentation
#### 7.1 Update README.md
Add Kokoro to supported engines:
```markdown
## Supported TTS Engines
- **Piper** - Fast, lightweight, ONNX-based TTS
- **Kokoro** - 82M parameter high-quality TTS (NEW!)
- **StyleTTS** - Expressive style-based TTS (planned)
- **ChatTTS** - Conversational TTS (planned)
- **F5-TTS** - Advanced flow-based TTS (planned)
```
#### 7.2 Update CLAUDE.md
Add Kokoro engine details:
```markdown
### Kokoro Engine (`app/engines/kokoro.py`)
- Uses `kokoro` Python library (KPipeline)
- 54 voices across 8 languages
- Outputs 24kHz audio natively
- Extremely fast (~90x real-time on consumer GPU)
- Model size: 82M parameters (~200MB download)
```
#### 7.3 Create Kokoro Usage Guide
Create `docs/kokoro-guide.md`:
- Voice selection guide
- Language support matrix
- Performance optimization tips
- GPU vs CPU mode comparison
- Troubleshooting common issues
---
## Implementation Checklist
### Phase 1: Research & Setup
- [x] Research Kokoro capabilities
- [x] Document API and dependencies
- [ ] Test Kokoro locally outside Docker
### Phase 2: Engine Implementation
- [ ] Create `app/engines/kokoro.py`
- [ ] Implement `synthesize()` method
- [ ] Implement `list_models()` method
- [ ] Implement `list_voices()` method
- [ ] Implement `healthcheck()` method
- [ ] Implement `selftest()` method
- [ ] Create `app/engines/kokoro_voices.py` metadata file
### Phase 3: Docker Integration
- [ ] Update Dockerfile with dependencies
- [ ] Add espeak-ng system package
- [ ] Add Kokoro Python packages
- [ ] Test Docker build
- [ ] Verify GPU access in container
### Phase 4: Configuration
- [ ] Update `app/config.py` with Kokoro settings
- [ ] Update `.env.example` with Kokoro variables
- [ ] Add KokoroEngine to `app/main.py` ALL_ENGINES
- [ ] Update active engines in `.env`
### Phase 5: Testing
- [ ] Create unit tests
- [ ] Test synthesis with various voices
- [ ] Test format conversion (wav/ogg/mp3)
- [ ] Test caching behavior
- [ ] Performance benchmarks
- [ ] Memory usage profiling
### Phase 6: Documentation
- [ ] Update README.md
- [ ] Update CLAUDE.md
- [ ] Create Kokoro usage guide
- [ ] Document voice selection
- [ ] Add troubleshooting section
### Phase 7: Deployment
- [ ] Test in development environment
- [ ] Rebuild Docker image
- [ ] Update docker-compose.yml if needed
- [ ] Test API endpoints
- [ ] Commit changes to git
- [ ] Tag release
---
## Key Decisions
### 1. Model Organization
**Decision**: Map Kokoro lang_codes to model names
- `kokoro-en-us` → lang_code='a'
- `kokoro-en-gb` → lang_code='b'
- etc.
**Rationale**: Maintains consistency with existing API structure
### 2. Voice Naming
**Decision**: Use Kokoro's native voice names (e.g., `af_bella`)
**Rationale**:
- Avoids confusion with remapping
- Documented in official VOICES.md
- Users can reference official docs
### 3. GPU Support
**Decision**: Support both GPU and CPU with fallback
**Rationale**:
- GPU provides 90x real-time performance
- CPU fallback ensures it works on all systems
- Configurable via KOKORO_DEVICE env var
### 4. Model Download Strategy
**Decision**: Lazy loading on first use
**Rationale**:
- Smaller Docker image
- Only download models that are actually used
- Can switch to pre-download later if needed
---
## Potential Issues & Solutions
### Issue 1: Large Model Download
**Problem**: Model is ~200MB per language
**Solution**:
- Lazy loading (download on first use)
- Cache in Docker volume
- Document expected download time
### Issue 2: GPU Memory Usage
**Problem**: May require 2-4GB VRAM
**Solution**:
- Implement memory monitoring
- Graceful fallback to CPU
- Document GPU requirements
### Issue 3: First Synthesis Slow
**Problem**: Model loading + compilation takes time
**Solution**:
- Warm up during healthcheck
- Keep model loaded in memory
- Document expected first-run delay
### Issue 4: Voice Compatibility
**Problem**: Not all voices work with all languages
**Solution**:
- Validate voice-language compatibility
- Return clear error messages
- Document voice language support
---
## Success Criteria
- [ ] Kokoro engine passes all unit tests
- [ ] API endpoints work correctly (`/tts`, `/models`, `/speakers`)
- [ ] Audio quality matches expected output
- [ ] Synthesis speed: >50x real-time on RTX 3090
- [ ] Memory usage: <4GB VRAM for single synthesis
- [ ] Cache versioning works correctly
- [ ] No temp file leaks
- [ ] No hung processes
- [ ] Documentation complete and accurate
- [ ] Voice wizard integration works
---
## Timeline Estimate
**Total**: 8-12 hours
- **Phase 1 (Research)**: ✅ Complete (2 hours)
- **Phase 2 (Implementation)**: 3-4 hours
- **Phase 3 (Docker)**: 1-2 hours
- **Phase 4 (Config)**: 0.5 hours
- **Phase 5 (Testing)**: 2-3 hours
- **Phase 6 (Documentation)**: 1-2 hours
- **Phase 7 (Deployment)**: 1 hour
---
## References
- [Kokoro-82M Hugging Face](https://huggingface.co/hexgrad/Kokoro-82M)
- [Kokoro GitHub Repository](https://github.com/hexgrad/kokoro)
- [VOICES.md - Complete Voice List](https://huggingface.co/hexgrad/Kokoro-82M/blob/main/VOICES.md)
- [Kokoro TTS Official Website](https://kokorotts.net/)
- [Kokoro-82M Analytics Vidhya Article](https://www.analyticsvidhya.com/blog/2025/01/kokoro-82m/)
- [Kokoro TTS Live Demo](https://huggingface.co/spaces/hexgrad/Kokoro-TTS)

View File

@ -0,0 +1,536 @@
# Kokoro TTS Implementation Status
**Date Started**: 2025-12-05
**Last Updated**: 2025-12-05 00:48 CET
**Status**: 🟡 In Progress (Phase 2 & 3 partially complete)
---
## Quick Summary
We are implementing Kokoro-82M TTS engine as a new engine in AudioEngineHub. Kokoro is a lightweight 82M parameter model that delivers quality comparable to models 5-15× its size, with ~90× real-time performance on consumer GPUs.
**Progress**: ~40% complete
- ✅ Phase 1: Research & Planning (100%)
- ✅ Phase 2: Engine Implementation (100%)
- 🟡 Phase 3: Docker Integration (50%)
- ⏳ Phase 4: Configuration (0%)
- ⏳ Phase 5: Testing (0%)
- ⏳ Phase 6: Documentation (0%)
---
## What We've Completed
### ✅ Phase 1: Research & Planning
- [x] Researched Kokoro TTS capabilities and architecture
- [x] Identified Python library (`kokoro>=0.9.2`)
- [x] Documented all 54 voices across 8 languages
- [x] Created comprehensive implementation plan in `KOKORO_IMPLEMENTATION_PLAN.md`
**Key Findings:**
- Kokoro uses 82M parameters (extremely lightweight)
- Performance: ~90× real-time on RTX 3090 Ti, ~210× on RTX 4090
- Output: 24kHz high-fidelity audio
- License: Apache 2.0 (open-source, commercial use allowed)
- 54 voices across 8 languages (EN-US, EN-GB, JA, ZH, ES, FR, HI, IT, PT)
### ✅ Phase 2: Engine Implementation (Complete)
- [x] **Created `app/engines/kokoro.py`** - Full engine implementation with:
- Complete `KokoroEngine` class implementing `TTSEngineBase`
- `synthesize()` method with all bug fixes from Piper (timeouts, cleanup, logging)
- GPU/CPU support with automatic fallback
- Language code mapping (10 models: kokoro-en-us, kokoro-en-gb, etc.)
- Format conversion (WAV/OGG/MP3) via ffmpeg
- Comprehensive error handling and logging
- Model pipeline caching for performance
- [x] **Created `app/engines/kokoro_voices.py`** - Voice metadata with:
- Complete list of all 54 Kokoro voices
- Voice metadata (gender, language, description)
- Language-based voice filtering
- Helper functions: `get_voices_for_model()`, `get_voice_info()`
**File Locations:**
- `/home/stephan/Projekte/KI/AudioEngineHub/app/engines/kokoro.py` (316 lines)
- `/home/stephan/Projekte/KI/AudioEngineHub/app/engines/kokoro_voices.py` (222 lines)
### 🟡 Phase 3: Docker Integration (50% Complete)
- [x] **Updated `Dockerfile`** - Added espeak-ng system dependency
- Line 17: Added `espeak-ng` to apt-get install
- [ ] **Update `requirements.txt`** - Need to add Kokoro Python dependencies
- **NEXT STEP**: Add these lines to requirements.txt:
```
# Kokoro TTS Engine
kokoro>=0.9.2
soundfile
phonemizer
scipy
munch
```
---
## What's Left To Do
### ⏳ Phase 4: Configuration (Not Started)
#### 1. Update `app/config.py`
Add Kokoro configuration settings:
```python
class Settings(BaseSettings):
# ... existing settings ...
# Kokoro Engine Configuration
KOKORO_DEVICE: str = "cuda" # or "cpu"
KOKORO_TIMEOUT_SECONDS: int = 30
```
**File**: `/home/stephan/Projekte/KI/AudioEngineHub/app/config.py`
**Location**: Add after line 25 (after FFMPEG_TIMEOUT_SECONDS)
#### 2. Update `app/main.py`
Register Kokoro engine in the engine registry:
```python
from app.engines.kokoro import KokoroEngine # Add this import
ALL_ENGINES = {
"piper": PiperEngine,
"styletts": StyleTTSEngine,
"chattts": ChatTTSEngine,
"f5-tts": F5TTSEngine,
"kokoro": KokoroEngine, # Add this line
}
```
**File**: `/home/stephan/Projekte/KI/AudioEngineHub/app/main.py`
**Location**: Line 28 (import), Line 38 (registry)
#### 3. Update `.env.example`
Add Kokoro configuration documentation:
```bash
# Kokoro Engine Configuration
KOKORO_DEVICE=cuda # cuda or cpu
KOKORO_TIMEOUT_SECONDS=30
```
**File**: `/home/stephan/Projekte/KI/AudioEngineHub/.env.example`
**Location**: Add after FFMPEG_TIMEOUT_SECONDS
#### 4. Optional: Update `.env`
To enable Kokoro by default:
```bash
ACTIVE_ENGINES='["piper", "kokoro"]'
```
---
### ⏳ Phase 5: Testing (Not Started)
#### 1. Test Import and Basic Functionality
```bash
# Test engine loads without errors
python3 -c "from app.engines.kokoro import KokoroEngine; print(KokoroEngine().healthcheck())"
```
Expected output:
```json
{
"status": "not_available", # OK if kokoro not installed yet
"engine": "kokoro",
"library_available": false,
"error": "Kokoro library not installed..."
}
```
#### 2. Install Kokoro Dependencies (Outside Docker First)
```bash
# Install espeak-ng
sudo apt-get install espeak-ng
# Install Python packages
pip install kokoro>=0.9.2 soundfile phonemizer scipy munch
```
#### 3. Test Synthesis Locally
```bash
# Run engine selftest
cd /home/stephan/Projekte/KI/AudioEngineHub
python3 app/engines/kokoro.py
```
Expected output:
```
Healthcheck: {'status': 'ok', 'engine': 'kokoro', ...}
Models: ['kokoro-en-us', 'kokoro-en-gb', ...]
Voices: ['af_alloy', 'af_aoede', 'af_bella', ...] ...
Selftest: {'selftest': True, 'models': [...], 'voices_count': 54}
```
#### 4. Test via API
```bash
# Start development server
make dev-up
# Test API endpoint
curl -X POST http://localhost:8000/tts \
-H "Content-Type: application/json" \
-d '{"text":"Hello from Kokoro!","engine":"kokoro","model":"kokoro-en-us","speaker":"af_bella","format":"ogg"}'
```
#### 5. Test with Voice Wizard
```bash
# Launch voice wizard and test preview with Kokoro voices
cd /home/stephan/Games/University/tts
./scripts/voice-assignment-wizard.sh
```
---
### ⏳ Phase 6: Documentation (Not Started)
#### 1. Update `README.md`
Add Kokoro to supported engines list:
```markdown
## Supported TTS Engines
- **Piper** - Fast, lightweight, ONNX-based TTS with 100+ voices
- **Kokoro** - 82M parameter high-quality TTS with 54 voices across 8 languages 🆕
- **StyleTTS** - Expressive style-based TTS (coming soon)
- **ChatTTS** - Conversational TTS (coming soon)
- **F5-TTS** - Advanced flow-based TTS (coming soon)
```
#### 2. Update `CLAUDE.md`
Add Kokoro engine section:
```markdown
### Kokoro Engine (`app/engines/kokoro.py`)
- **Library**: `kokoro` Python package (KPipeline)
- **Models**: 10 language models (en-us, en-gb, ja, zh, es, fr, hi, it, pt, ko)
- **Voices**: 54 voices with gender and language metadata
- **Output**: 24kHz audio natively (WAV), converts to OGG/MP3 via ffmpeg
- **Performance**: ~90× real-time on RTX 3090 Ti
- **Model Size**: 82M parameters (~200MB download per language)
- **Voice Metadata**: `app/engines/kokoro_voices.py` contains all 54 voices with metadata
```
#### 3. Create Usage Guide (Optional)
Create `docs/kokoro-usage.md` with:
- Voice selection guide
- Language support matrix
- Performance tips (GPU vs CPU)
- Troubleshooting common issues
---
## Implementation Checklist
### Phase 1: Research ✅
- [x] Research Kokoro capabilities
- [x] Document API and dependencies
- [x] Create implementation plan
### Phase 2: Engine Implementation ✅
- [x] Create `app/engines/kokoro.py`
- [x] Implement `synthesize()` method
- [x] Implement `list_models()` method
- [x] Implement `list_voices()` method
- [x] Implement `healthcheck()` method
- [x] Implement `selftest()` method
- [x] Create `app/engines/kokoro_voices.py`
### Phase 3: Docker Integration 🟡
- [x] Update Dockerfile (added espeak-ng)
- [ ] **→ NEXT: Update requirements.txt (add Kokoro packages)**
- [ ] Test Docker build
- [ ] Verify GPU access in container (if available)
### Phase 4: Configuration ⏳
- [ ] Update `app/config.py` with Kokoro settings
- [ ] Update `.env.example` with Kokoro variables
- [ ] Add KokoroEngine import to `app/main.py`
- [ ] Add Kokoro to ALL_ENGINES in `app/main.py`
- [ ] Optional: Update `.env` to enable Kokoro
### Phase 5: Testing ⏳
- [ ] Test engine imports without errors
- [ ] Install Kokoro dependencies locally
- [ ] Run selftest (`python3 app/engines/kokoro.py`)
- [ ] Test synthesis via API
- [ ] Test with voice wizard
- [ ] Verify caching works correctly
- [ ] Check for temp file leaks
- [ ] Performance benchmarks
### Phase 6: Documentation ⏳
- [ ] Update README.md
- [ ] Update CLAUDE.md
- [ ] Optional: Create Kokoro usage guide
- [ ] Document voice selection
- [ ] Add troubleshooting section
### Phase 7: Deployment ⏳
- [ ] Rebuild Docker image
- [ ] Test in development environment
- [ ] Commit changes to git
- [ ] Push to remote
- [ ] Tag release (optional)
---
## Files Created/Modified
### ✅ Created Files
1. **`KOKORO_IMPLEMENTATION_PLAN.md`** - Complete implementation plan (280 lines)
2. **`app/engines/kokoro.py`** - Kokoro engine implementation (316 lines)
3. **`app/engines/kokoro_voices.py`** - Voice metadata (222 lines)
4. **`KOKORO_IMPLEMENTATION_STATUS.md`** - This file
### ✅ Modified Files
1. **`Dockerfile`** - Added espeak-ng dependency (line 17)
### ⏳ Files To Modify
1. **`requirements.txt`** - Add Kokoro Python packages
2. **`app/config.py`** - Add Kokoro configuration
3. **`app/main.py`** - Register Kokoro engine
4. **`.env.example`** - Document Kokoro config
5. **`README.md`** - Add Kokoro to engines list
6. **`CLAUDE.md`** - Add Kokoro engine details
---
## Next Session: Action Plan
### Step 1: Complete Docker Integration (5 minutes)
```bash
cd /home/stephan/Projekte/KI/AudioEngineHub
# Add to requirements.txt (after line 11):
cat >> requirements.txt << 'EOF'
# Kokoro TTS Engine
kokoro>=0.9.2
soundfile
phonemizer
scipy
munch
EOF
```
### Step 2: Complete Configuration (10 minutes)
1. Edit `app/config.py` - add KOKORO_DEVICE and KOKORO_TIMEOUT_SECONDS
2. Edit `app/main.py` - import KokoroEngine and add to ALL_ENGINES
3. Edit `.env.example` - document new config options
### Step 3: Test Locally (15 minutes)
1. Install dependencies: `sudo apt-get install espeak-ng && pip install kokoro>=0.9.2 soundfile phonemizer scipy munch`
2. Run selftest: `python3 app/engines/kokoro.py`
3. Start server: `make dev-up` (will rebuild Docker image)
4. Test API synthesis with Kokoro
5. Test voice wizard preview
### Step 4: Documentation (10 minutes)
1. Update README.md with Kokoro
2. Update CLAUDE.md with Kokoro engine details
3. Commit all changes to git
### Total Time Remaining: ~40 minutes
---
## Key Technical Details
### Language Code Mapping
```python
KOKORO_LANG_CODES = {
"kokoro-en-us": "a", # American English
"kokoro-en-gb": "b", # British English
"kokoro-fr": "fr", # French
"kokoro-es": "es", # Spanish
"kokoro-ja": "ja", # Japanese
"kokoro-zh": "zh", # Chinese
"kokoro-it": "it", # Italian
"kokoro-pt": "pt", # Portuguese
"kokoro-hi": "hi", # Hindi
"kokoro-ko": "ko", # Korean
}
```
### Voice Organization
- **54 total voices** across 8 languages
- **Naming convention**: `{language}{gender}_{name}`
- `af_` = American Female
- `am_` = American Male
- `bf_` = British Female
- `bm_` = British Male
- `jf_/jm_` = Japanese F/M
- `zf_/zm_` = Chinese F/M
- etc.
- **Most popular**: af_bella, af_sarah, af_sky, am_adam, am_michael
### API Usage Example
```json
{
"text": "Hello from Kokoro TTS!",
"engine": "kokoro",
"model": "kokoro-en-us",
"speaker": "af_bella",
"format": "ogg"
}
```
---
## Known Issues & Considerations
### 1. Model Download on First Use
- **Issue**: First synthesis will be slow (~30-60s) due to model download
- **Size**: ~200MB per language model
- **Location**: Models cached in `~/.cache/huggingface/`
- **Solution**: Expected behavior, document in README
### 2. GPU vs CPU Performance
- **GPU**: ~90× real-time (RTX 3090 Ti)
- **CPU**: ~5-10× real-time (estimated)
- **Fallback**: Code supports both, configurable via KOKORO_DEVICE env var
### 3. Voice-Language Compatibility
- **Issue**: Not all voices work with all languages
- **Solution**: `kokoro_voices.py` filters voices by language
- **API**: `/speakers?engine=kokoro&model=kokoro-en-us` returns only compatible voices
### 4. Dependencies
- **espeak-ng**: Required system package for phonemization
- **soundfile**: Required for WAV file I/O
- **phonemizer**: Required for text-to-phoneme conversion
- **scipy, munch**: Required by Kokoro library
---
## Testing Commands Reference
### Local Testing (Outside Docker)
```bash
# Install system dependency
sudo apt-get install espeak-ng
# Install Python packages
pip install kokoro>=0.9.2 soundfile phonemizer scipy munch
# Test engine
cd /home/stephan/Projekte/KI/AudioEngineHub
python3 app/engines/kokoro.py
# Test voice metadata
python3 -c "from app.engines.kokoro_voices import ALL_VOICES, get_voices_for_model; print(f'Total voices: {len(ALL_VOICES)}'); print(f'EN-US voices: {get_voices_for_model(\"kokoro-en-us\")}')"
```
### Docker Testing
```bash
# Rebuild and start
make down
make dev-up
# Check logs
make logs
# Health check
make health-check
# Test synthesis
curl -X POST http://localhost:8000/tts \
-H "Content-Type: application/json" \
-d '{"text":"Testing Kokoro TTS engine","engine":"kokoro","model":"kokoro-en-us","speaker":"af_bella","format":"ogg"}' | jq
```
### Voice Wizard Testing
```bash
cd /home/stephan/Games/University/tts
./scripts/voice-assignment-wizard.sh
# Should show Kokoro engine in dropdown
# Should list 54 voices (or subset by language)
# Preview should work with all Kokoro voices
```
---
## Success Criteria
Before marking this implementation as complete, verify:
- [ ] Kokoro engine passes healthcheck
- [ ] All 10 language models are listed in `/models` endpoint
- [ ] 54 voices are accessible via `/speakers` endpoint
- [ ] Voice filtering by model/language works correctly
- [ ] Synthesis produces valid audio files
- [ ] Synthesis speed: >50× real-time on GPU (if available)
- [ ] Format conversion works (WAV/OGG/MP3)
- [ ] Cache versioning works correctly
- [ ] No temp file leaks after synthesis
- [ ] No hung processes under load
- [ ] Voice wizard integration works
- [ ] Documentation is complete and accurate
---
## Resources & References
### Official Documentation
- [Kokoro-82M Hugging Face](https://huggingface.co/hexgrad/Kokoro-82M)
- [VOICES.md - Complete Voice List](https://huggingface.co/hexgrad/Kokoro-82M/blob/main/VOICES.md)
- [Kokoro GitHub Repository](https://github.com/hexgrad/kokoro)
- [Kokoro Official Website](https://kokorotts.net/)
### Technical Articles
- [Analytics Vidhya - Kokoro-82M Review](https://www.analyticsvidhya.com/blog/2025/01/kokoro-82m/)
### Installation Guides
- [Kokoro-82M Installation Instructions](https://huggingface.co/hexgrad/Kokoro-82M#installation)
- [Kokoro Live Demo](https://huggingface.co/spaces/hexgrad/Kokoro-TTS)
---
## Timeline & Estimates
| Phase | Status | Time Spent | Remaining | Total |
|-------|--------|------------|-----------|-------|
| Phase 1: Research | ✅ Complete | 2h | 0h | 2h |
| Phase 2: Implementation | ✅ Complete | 2h | 0h | 2h |
| Phase 3: Docker | 🟡 50% | 0.5h | 0.5h | 1h |
| Phase 4: Configuration | ⏳ Pending | 0h | 0.5h | 0.5h |
| Phase 5: Testing | ⏳ Pending | 0h | 1h | 1h |
| Phase 6: Documentation | ⏳ Pending | 0h | 0.5h | 0.5h |
| **Total** | **40% Complete** | **4.5h** | **2.5h** | **7h** |
**Original Estimate**: 8-12 hours
**Current Progress**: ~5 hours spent, ~2-3 hours remaining
**On Track**: Yes, ahead of schedule
---
## Questions for User (Next Session)
1. **GPU Availability**: Do you have a CUDA-capable GPU? This affects:
- Performance expectations (90× vs 5× real-time)
- Default KOKORO_DEVICE setting (cuda vs cpu)
2. **Language Priority**: Which languages do you need most?
- Only EN-US model will be tested initially
- Other languages can be tested on demand
3. **Docker vs Local**: Prefer testing locally first or directly in Docker?
- Local testing is faster for iteration
- Docker testing validates full deployment
4. **Voice Wizard**: Should we integrate Kokoro metadata into the voice wizard UI?
- Could show language, gender, description for each voice
- Requires changes to audioengine_client.py
---
**End of Status Report**
Last updated: 2025-12-05 00:48 CET
Next session: Continue with Step 1 (Complete Docker Integration)

106
Makefile
View File

@ -36,7 +36,7 @@ endef
.PHONY: build
build:
@echo "Building Docker image: $(IMAGE_NAME):$(TAG)"
docker build -t $(IMAGE_NAME):$(TAG) .
docker build --no-cache -t $(IMAGE_NAME):$(TAG) .
.PHONY: run
run:
@ -92,6 +92,54 @@ down:
export TAG=$(TAG) && \
docker compose down
# --- Test Environment Commands ---
.PHONY: kokoro-up
kokoro-up:
@echo "Starting Kokoro test environment on port 8001..."
export IMAGE_NAME=$(IMAGE_NAME) && \
export REGISTRY=$(REGISTRY) && \
export USERNAME=$(USERNAME) && \
export TAG=dev-kokoro-v5 && \
docker compose -f docker-compose.kokoro-test.yml up -d
.PHONY: kokoro-down
kokoro-down:
@echo "Stopping Kokoro test environment..."
export IMAGE_NAME=$(IMAGE_NAME) && \
export REGISTRY=$(REGISTRY) && \
export USERNAME=$(USERNAME) && \
export TAG=dev-kokoro-v5 && \
docker compose -f docker-compose.kokoro-test.yml down
.PHONY: kokoro-logs
kokoro-logs:
@echo "Showing logs for Kokoro test container..."
docker logs -f audio-engine-hub_kokoro_test
.PHONY: xtts-up
xtts-up:
@echo "Starting XTTS test environment on port 8002..."
export IMAGE_NAME=$(IMAGE_NAME) && \
export REGISTRY=$(REGISTRY) && \
export USERNAME=$(USERNAME) && \
export TAG=dev-xtts && \
docker compose -f docker-compose.xtts-test.yml up -d
.PHONY: xtts-down
xtts-down:
@echo "Stopping XTTS test environment..."
export IMAGE_NAME=$(IMAGE_NAME) && \
export REGISTRY=$(REGISTRY) && \
export USERNAME=$(USERNAME) && \
export TAG=dev-xtts && \
docker compose -f docker-compose.xtts-test.yml down
.PHONY: xtts-logs
xtts-logs:
@echo "Showing logs for XTTS test container..."
docker logs -f audio-engine-hub_xtts_test
# --- Image Management ---
.PHONY: pull
@ -109,6 +157,40 @@ push: build tag
@echo "Pushing image $(REGISTRY)/$(USERNAME)/$(IMAGE_NAME):$(TAG) to registry..."
docker push $(REGISTRY)/$(USERNAME)/$(IMAGE_NAME):$(TAG)
# --- Tagged Image Management ---
.PHONY: build-kokoro
build-kokoro:
@echo "Building Kokoro test image: $(IMAGE_NAME):dev-kokoro-v5"
docker build --no-cache -t $(IMAGE_NAME):dev-kokoro-v5 .
docker tag $(IMAGE_NAME):dev-kokoro-v5 $(REGISTRY)/$(USERNAME)/$(IMAGE_NAME):dev-kokoro-v5
.PHONY: push-kokoro
push-kokoro: build-kokoro
@echo "Pushing Kokoro test image to registry..."
docker push $(REGISTRY)/$(USERNAME)/$(IMAGE_NAME):dev-kokoro-v5
.PHONY: pull-kokoro
pull-kokoro:
@echo "Pulling Kokoro test image from registry..."
docker pull $(REGISTRY)/$(USERNAME)/$(IMAGE_NAME):dev-kokoro-v5
.PHONY: build-xtts
build-xtts:
@echo "Building XTTS test image: $(IMAGE_NAME):dev-xtts"
docker build --no-cache -t $(IMAGE_NAME):dev-xtts .
docker tag $(IMAGE_NAME):dev-xtts $(REGISTRY)/$(USERNAME)/$(IMAGE_NAME):dev-xtts
.PHONY: push-xtts
push-xtts: build-xtts
@echo "Pushing XTTS test image to registry..."
docker push $(REGISTRY)/$(USERNAME)/$(IMAGE_NAME):dev-xtts
.PHONY: pull-xtts
pull-xtts:
@echo "Pulling XTTS test image from registry..."
docker pull $(REGISTRY)/$(USERNAME)/$(IMAGE_NAME):dev-xtts
.PHONY: test
test:
@if [ -d ".venv" ]; then \
@ -148,6 +230,8 @@ clean:
.PHONY: help
help:
@echo "Available commands:"
@echo ""
@echo "Main Deployment:"
@echo " build - Build the Docker image"
@echo " run - Run the Docker container (single app, no Traefik)"
@echo " stop - Stop and remove the Docker container"
@ -156,9 +240,29 @@ help:
@echo " up - Start the dev environment by pulling image from registry"
@echo " dev-up - Start the dev environment by building image locally"
@echo " down - Stop the dev environment with docker-compose"
@echo ""
@echo "Test Environments:"
@echo " kokoro-up - Start Kokoro test environment (port 8001)"
@echo " kokoro-down - Stop Kokoro test environment"
@echo " kokoro-logs - View Kokoro test container logs"
@echo " xtts-up - Start XTTS test environment (port 8002)"
@echo " xtts-down - Stop XTTS test environment"
@echo " xtts-logs - View XTTS test container logs"
@echo ""
@echo "Image Management (main 'latest' tag):"
@echo " pull - Pull the Docker image from the registry"
@echo " tag - Tag the image for a registry"
@echo " push - Push the image to a registry (after tagging)"
@echo ""
@echo "Image Management (test tags):"
@echo " build-kokoro - Build Kokoro test image (dev-kokoro-v5 tag)"
@echo " push-kokoro - Build and push Kokoro test image to registry"
@echo " pull-kokoro - Pull Kokoro test image from registry"
@echo " build-xtts - Build XTTS test image (dev-xtts tag)"
@echo " push-xtts - Build and push XTTS test image to registry"
@echo " pull-xtts - Pull XTTS test image from registry"
@echo ""
@echo "Testing & Maintenance:"
@echo " test - Run the pytest test suite"
@echo " health-check - Run a sanity check on the deployed container"
@echo " clean - Clean up unused containers and images"

View File

@ -0,0 +1,40 @@
# Coqui XTTS v2 Implementation Plan
## Objective
Implement Coqui XTTS v2 (`tts_models/multilingual/multi-dataset/xtts_v2`) as a new engine in AudioEngineHub.
## 1. Dependencies (`requirements.txt`)
- Add `TTS` (Coqui TTS).
- **Note:** This is a heavy library. We will add it to `requirements.txt`.
- **Potential Conflict:** `TTS` often requires specific `torch` versions. We need to ensure it plays nicely with `f5-tts` (which also uses torch) and `kokoro`.
## 2. Configuration (`app/config.py` & `.env`)
- `XTTS_DEVICE`: "cuda" or "cpu" (default: "cuda")
- `XTTS_ACCEPT_LICENSE`: "true" (required to use the model)
- `XTTS_MODEL_VERSION`: "v2.0.2" (or "main" for latest)
## 3. Engine Implementation (`app/engines/xtts.py`)
- **Class:** `XTTSEngine` (inherits `TTSEngineBase`)
- **Init:**
- Load `TTS` API.
- Download/Load model: `tts_models/multilingual/multi-dataset/xtts_v2`.
- Handle license agreement.
- **Synthesize:**
- Inputs: `text`, `speaker` (voice cloning reference), `language`.
- **Voice Cloning:** The `speaker` argument will be interpreted as a filename in `app/asset/voices/` (or a default provided sample).
- **Language:** XTTS supports 17 languages. We will map them (e.g., "en", "de", "fr").
- **List Models:** Return `['xtts_v2']`.
- **List Voices:** Scan `app/asset/voices/` for `.wav` files to use as reference speakers.
## 4. System Updates
- **`app/main.py`**: Register `xtts` in `ALL_ENGINES`.
- **`Dockerfile`**: Ensure system dependencies (already have `ffmpeg` and `espeak-ng`, which are good).
## 5. Directory Structure
- `app/asset/voices/`: Directory to store reference audio files for cloning.
## Action Plan
1. Update `requirements.txt`.
2. Create `app/engines/xtts.py`.
3. Update `app/config.py`.
4. Update `app/main.py`.

View File

@ -6,7 +6,7 @@ Version: v0.1.1
Description:
Piper TTS engine adapter: real CLI invocation + output as WAV, OGG, or MP3.
Synthesizes WAV via Piper, converts to OGG/MP3 via ffmpeg-python if needed.
Uses dynamic model path: ./models/piper/[model]/model.onnx
Uses dynamic model path: settings.MODELS_DIR/piper/[model]/model.onnx
Author: Abby (ChatGPT)
Date: 2025-07-23
@ -28,14 +28,16 @@ logger = logging.getLogger(__name__)
class PiperEngine(TTSEngineBase):
def __init__(self):
self.piper_executable = shutil.which("piper")
# Use python -m piper instead of direct script execution
# This avoids shebang issues in multi-stage Docker builds
self.python_executable = shutil.which("python3.11") or shutil.which("python3") or shutil.which("python")
self.use_module_execution = True
self.ffmpeg_executable = shutil.which("ffmpeg")
self.PIPER_TIMEOUT_SECONDS = settings.PIPER_TIMEOUT_SECONDS
self.FFMPEG_TIMEOUT_SECONDS = settings.FFMPEG_TIMEOUT_SECONDS
def _load_config(self, model: str):
"""Load the model config JSON file to get speaker mappings."""
model_dir = f"/models/piper/{model}"
model_dir = os.path.join(settings.MODELS_DIR, "piper", model)
config_file = os.path.join(model_dir, f"{model}.onnx.json")
if os.path.isfile(config_file):
with open(config_file, 'r') as f:
@ -89,13 +91,13 @@ class PiperEngine(TTSEngineBase):
- Bug #7: Enhanced error context and logging
"""
# Validation
if not self.piper_executable:
raise RuntimeError("Piper executable not found. Please install it and ensure it's in your PATH.")
if not self.python_executable:
raise RuntimeError("Python executable not found. Cannot execute piper module.")
if not model:
raise ValueError("Model must be specified for Piper.")
# Validate model and config files (Bug #6)
model_dir = f"/models/piper/{model}"
model_dir = os.path.join(settings.MODELS_DIR, "piper", model)
model_file = os.path.join(model_dir, f"{model}.onnx")
config_file = os.path.join(model_dir, f"{model}.onnx.json")
@ -136,8 +138,8 @@ class PiperEngine(TTSEngineBase):
os.close(fd)
temp_files_to_cleanup.append(output_wav_path)
# Build command (Bug #1 - no --stdin flag, piper reads stdin by default)
cmd = [self.piper_executable, "--model", model_file, "--output-file", output_wav_path]
# Build command - use python -m piper to avoid shebang issues
cmd = [self.python_executable, "-m", "piper", "--model", model_file, "--output-file", output_wav_path]
if speaker:
speaker_id = self._get_speaker_id(speaker, model)
@ -279,7 +281,7 @@ class PiperEngine(TTSEngineBase):
logger.warning(f"Failed to cleanup temp file {temp_file}: {e}")
def list_models(self):
models_dir = "/models/piper/"
models_dir = os.path.join(settings.MODELS_DIR, "piper")
if not os.path.isdir(models_dir):
return []
return [name for name in os.listdir(models_dir)
@ -296,13 +298,13 @@ class PiperEngine(TTSEngineBase):
def healthcheck(self):
status = "ok"
if not self.piper_executable:
status = "missing_piper_executable"
if not self.python_executable:
status = "missing_python_executable"
return {"status": status, "engine": "piper"}
async def selftest(self):
if not self.piper_executable:
return {"selftest": False, "error": "Piper executable not found.", "engine": "piper"}
if not self.python_executable:
return {"selftest": False, "error": "Python executable not found.", "engine": "piper"}
try:
models = self.list_models()
if not models:

View File

@ -0,0 +1,20 @@
version: '3.8'
services:
app-kokoro-test:
image: ${REGISTRY}/${USERNAME}/${IMAGE_NAME}:dev-kokoro-v5
container_name: audio-engine-hub_kokoro_test
restart: unless-stopped
volumes:
# Mount local app directory for hot-reloading in dev
- ./app:/home/appuser/app
# Mount models directory to a top-level directory in the container
- ./app/models:/models
# Mount asset directory to persist generated audio files
- ./asset:/home/appuser/app/asset
# Use Dockerfile's CMD (gunicorn) instead of uvicorn for production
# command: ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
ports:
- "8001:8000"
env_file:
- .env.kokoro-test

View File

@ -0,0 +1,19 @@
version: '3.8'
services:
app-xtts-test:
image: audioenginehub-xtts-test:latest
build: .
container_name: audio-engine-hub_xtts_test
restart: unless-stopped
volumes:
# Mount local app directory for hot-reloading in dev
- ./app:/home/appuser/app
# Mount models directory to a top-level directory in the container
- ./app/models:/models
# Mount asset directory to persist generated audio files
- ./asset:/home/appuser/app/asset
ports:
- "8002:8000"
env_file:
- .env.xtts-test

View File

@ -12,7 +12,8 @@ services:
- ./app/models:/models
# Mount asset directory to persist generated audio files
- ./asset:/home/appuser/app/asset
command: ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000", "--reload"]
# Use Dockerfile's CMD (gunicorn) instead of uvicorn for production
# command: ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]
ports:
- "${APP_PORT:-8000}:8000"
env_file:

View File

@ -0,0 +1,84 @@
# Technische Beschreibung: Multi-Stage Docker Build für GPU-fähige PyTorch-Umgebungen
## Zielsetzung
Der Ansatz ermöglicht ein Docker-Image, das GPU-beschleunigte Frameworks (PyTorch + CUDA + Torchaudio) enthält, dabei jedoch kompakt bleibt, ohne Runtime-Downloads auskommt und reproduzierbar gebaut werden kann.
---
## 1. Grundprinzip
### Stage 1 – Build-/Dependency Stage
- CUDA-fähiges Base-Image (z. B. `nvidia/cuda:12.1.0-runtime-ubuntu22.04`)
- Installation schwerer Abhängigkeiten:
- `torch` (CUDA-Build)
- `torchaudio`
- Enthält alle Build-Tools und Systemdependencies
- Ergebnis: vollständiges GPU-fähiges `site-packages`
### Stage 2 – Runtime Stage
- Leichtes Slim-Python-Image (z. B. `python:3.11-slim`)
- Nur das fertige `site-packages` aus Stage 1 wird kopiert
- Keine Build-Kette, keine Compiler, keine großen Pakete
---
## 2. Vorteile gegenüber anderen Ansätzen
### Gegenüber Single-Stage Builds
| Problem | Multi-Stage Lösung |
|--------|---------------------|
| Finales Image wird 5–10 GB groß | Nur Runtime-Layer → deutlich kleiner |
| Komplexer Build im finalen Image | Build isoliert in Stage 1 |
| Sicherheitsrisiken | Minimale Angriffsfläche im Runtime-Image |
### Gegenüber Runtime-Installation (EntryPoint)
| Runtime-Install | Multi-Stage |
|------------------|-------------|
| Lange Startup-Zeit | Installation beim Build |
| Internetzugang nötig | Runtime benötigt kein Netzwerk |
| Nicht deterministisch | Reproduzierbare Builds |
---
## 3. Technischer Ablauf im Detail
### Stage 1: Builder
- CUDA-fähiges Image
- Installation aller Dependencies inkl. Torch + CUDA
- Resultierendes `site-packages` wird vorbereitet
### Stage 2: Runtime
- Schlankes Python-Image
- Kopieren der vorbereiteten Pakete aus Stage 1
- Hinzufügen der Anwendung
- Setzen des Entrypoints
**Host-Anforderungen:**
- NVIDIA Treiber
- NVIDIA Container Toolkit
- Start mit `--gpus all`
---
## 4. CI/CD Eignung
- Deterministische Builds
- Weniger Storage
- Schnelleres Deployment
- Keine externen Abhängigkeiten bei Runtime
- Ideal für Kubernetes, GitLab CI, GitHub Actions, ArgoCD
---
## 5. Risiken & Mitigation
| Risiko | Mitigation |
|--------|------------|
| CUDA-Version mismatch | Versionen pinnen & Kompatibilitätsmatrix nutzen |
| GPU nicht verfügbar | CPU-Fallback implementieren |
| Torch-Wheel entfernt | internes Wheel-Repository verwenden |
---
## 6. TL;DR
Multi-Stage Build =
**Torch/CUDA in eigener Build-Stage installieren → fertige Pakete in ein leichtes Runtime-Image kopieren → kleines, GPU-fähiges, sauberes Deployment.**

View File

@ -0,0 +1,27 @@
#!/bin/bash
# Patch misaki/espeak.py to work with newer phonemizer versions
# The newer phonemizer API doesn't have set_data_path() method
MISAKI_FILE="/opt/venv/lib/python3.11/site-packages/misaki/espeak.py"
if [ -f "$MISAKI_FILE" ]; then
echo "Patching misaki/espeak.py for phonemizer compatibility..."
# Comment out the problematic line
sed -i 's/^EspeakWrapper\.set_data_path(espeakng_loader\.get_data_path())/# EspeakWrapper.set_data_path(espeakng_loader.get_data_path()) # Patched: not needed with current phonemizer/' "$MISAKI_FILE"
echo "Patch applied successfully!"
else
echo "Warning: misaki/espeak.py not found at $MISAKI_FILE"
fi
# Download SpaCy model for Kokoro text processing
echo "Downloading SpaCy en_core_web_sm model for Kokoro..."
python3.11 -m spacy download en_core_web_sm
echo "SpaCy model downloaded successfully!"
# Create symlink for espeak-ng-data to fix hardcoded path issue
echo "Creating espeak-ng-data symlink..."
mkdir -p /home/runner/work/espeakng-loader/espeakng-loader/espeak-ng/_dynamic/share/
ln -sf /usr/lib/x86_64-linux-gnu/espeak-ng-data /home/runner/work/espeakng-loader/espeakng-loader/espeak-ng/_dynamic/share/espeak-ng-data
echo "Symlink created successfully!"

79
scripts/tts_client.py Executable file
View File

@ -0,0 +1,79 @@
#!/usr/bin/env python3
"""
A simple CLI client to test the AudioEngineHub TTS server.
This script requires the 'requests' library.
Install it with: pip install requests
"""
import argparse
import requests
import os
from urllib.parse import urljoin
def main():
parser = argparse.ArgumentParser(description="CLI client for AudioEngineHub TTS server.")
parser.add_argument("--server-url", required=True, help="Base URL of the TTS server (e.g., http://localhost:8000).")
parser.add_argument("--text", required=True, help="Text to synthesize.")
parser.add_argument("--engine", required=True, help="TTS engine to use (e.g., piper).")
parser.add_argument("--model", help="Model to use (optional).")
parser.add_argument("--speaker", help="Speaker to use (optional).")
parser.add_argument("--output", required=True, help="Path to save the output audio file (e.g., output.ogg).")
args = parser.parse_args()
# --- 1. Make the TTS request ---
tts_url = urljoin(args.server_url, "/tts")
payload = {
"text": args.text,
"engine": args.engine,
"format": "ogg" # Or determine from output file extension
}
if args.model:
payload["model"] = args.model
if args.speaker:
payload["speaker"] = args.speaker
print(f"Requesting synthesis from {tts_url} with payload: {payload}")
# Add a short delay to mitigate potential connection race conditions
import time
time.sleep(1)
try:
response = requests.post(tts_url, json=payload)
response.raise_for_status() # Raise an exception for bad status codes (4xx or 5xx)
except requests.exceptions.RequestException as e:
print(f"ERROR: Could not connect to the server: {e}")
return
try:
data = response.json()
except requests.exceptions.JSONDecodeError:
print(f"ERROR: Failed to decode JSON response from server. Response text: {response.text}")
return
if "audio_url" not in data:
print(f"ERROR: Server response did not contain 'audio_url'. Response: {data}")
return
# --- 2. Download the audio file ---
audio_url = urljoin(args.server_url, data["audio_url"])
print(f"Downloading audio from {audio_url}...")
try:
audio_response = requests.get(audio_url, stream=True)
audio_response.raise_for_status()
except requests.exceptions.RequestException as e:
print(f"ERROR: Could not download the audio file: {e}")
return
# --- 3. Save the audio file ---
try:
with open(args.output, "wb") as f:
for chunk in audio_response.iter_content(chunk_size=8192):
f.write(chunk)
print(f"Successfully saved audio to '{args.output}'")
except IOError as e:
print(f"ERROR: Could not write to output file '{args.output}': {e}")
if __name__ == "__main__":
main()

View File

@ -136,3 +136,28 @@ To facilitate pushing and pulling Docker images from a remote registry (e.g., `g
* The `down` and `help` targets were updated accordingly.
This allows for flexible deployment, supporting both local development with on-demand building and production-like environments pulling from a registry.
### Debugging "Always the Same Text" Bug
The user reported that the TTS server was not synthesizing the provided text but was always returning audio for a fixed, incorrect text.
1. **Initial Reproduction Attempt (and "422 Unprocessable Entity" error):**
* Attempted to reproduce the bug by sending different texts via `scripts/tts_client.py`.
* Encountered a `422 Unprocessable Entity` error, which led to debugging model loading.
* **Resolution 1 (Model Loading):** Fixed by correcting the volume mount source in `docker-compose.yml` (`./app/models` to `/models`) and updating paths in `app/engines/piper.py` to absolute `/models/piper/`.
* **Resolution 2 (Startup Race Condition):** Added a startup event handler in `app/main.py` to wait for the models directory to be available, preventing a race condition.
* **Resolution 3 (Client Connection):** Mitigated `ConnectionResetError` from client by adding `time.sleep(1)` before `requests.post` call in `scripts/tts_client.py`, indicating a subtle client-server connection timing issue.
2. **Debugging the "Always the Same Text" Issue:**
* **Initial Test:** Generated audio for "This is a test with the vctk model." and "This is a completely different sentence to verify the bug."
* **Result:** `cmp` showed the files were different, contradicting the user's report that the text was always the same.
* **User Clarification:** User confirmed that *listening* to the files revealed the same speech output, indicating a deeper issue beyond simple file difference.
* **Investigation of `piper.py`:**
* The original method for passing text to the `piper` executable via `stdin` (`--stdin`) seemed correct.
* Manual testing `echo "text" | piper ...` inside the container proved `piper` executable works correctly with `stdin`.
* **Hypothesis:** The `asyncio.subprocess.communicate(input=...)` call was not reliably passing text to `piper`.
* **Attempted Fix 1 (Explicit stdin write):** Modified `piper.py` to manually write to `process.stdin`, `drain`, and `close`. This led to `ConnectionResetError` (server crash).
* **Attempted Fix 2 (Revert and `--input-file` strategy):** Reverted `piper.py` back to `process.communicate()` (after fixing `cmd` to use `--stdin`). Then, changed strategy to use a temporary file for input (`--input-file`) instead of `stdin`. This also led to `ConnectionResetError`.
* **Persistent Crash:** The server consistently crashed after each attempt to synthesize actual audio. Debugging was hampered by `uvicorn`'s reloader hiding tracebacks, even after disabling it. It became apparent that the crash was very low-level, possibly within the `piper` executable's interaction with the file system or system resources.
3. **Current Status:** The server still crashes when the `synthesize` method is called to generate real audio. The exact cause of this crash is still unknown, as no Python traceback is being produced in the server logs. Further debugging is required to stabilize the `piper` engine's subprocess execution.