audio-engine-hub/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

AudioEngineHub is a local-first, modular, multi-engine Text-to-Speech (TTS) server built with FastAPI. It provides a unified API to interact with various TTS engines (Piper, Kokoro, StyleTTS, ChatTTS, F5-TTS) with caching, audio format conversion, and Docker deployment support.

## Architecture

### Core Components

**FastAPI Application Factory (`app/main.py`)**
- Uses `create_app()` factory pattern to dynamically load engines based on configuration
- Maintains `app.ENGINE_REGISTRY` dict mapping engine names to instantiated engine objects
- Engines are loaded from `settings.ACTIVE_ENGINES` at startup
- All available engines defined in `ALL_ENGINES` dict (lines 35-41)
- Startup event checks for models directory availability with retry logic (lines 192-213)

**Engine Architecture (`app/engines/`)**
- All engines inherit from `TTSEngineBase` abstract class
- Required methods: `synthesize()`, `list_models()`, `list_voices()`, `healthcheck()`, `selftest()`
- All `synthesize()` methods are async and return file paths to generated audio
- Engines are stateless and instantiated once per app lifecycle

**Configuration (`app/config.py`)**
- Uses pydantic-settings for type-safe configuration
- `ACTIVE_ENGINES` is a Set[str] loaded from `.env` file as JSON array
- Default model path: `/models/piper/{model_name}/{model_name}.onnx`
- Audio cache directory: `/home/appuser/app/asset/audio`

**Utility Modules**
- `app/utils/text.py`: Text chunking for long inputs
- `app/utils/audio.py`: Audio concatenation via ffmpeg
- `app/utils/cache.py`: Cache key generation based on request parameters

### Volume Mounts

The `docker-compose.yml` mounts three critical paths:
1. `./app:/home/appuser/app` - Hot-reload for development
2. `./app/models:/models` - TTS model files (not included in image)
3. `./asset:/home/appuser/app/asset` - Persistent audio cache

**CRITICAL**: The model mount is `./app/models:/models` (maps to `/models` inside container). The engine code uses absolute path `/models/piper/` to locate models. Do NOT use `./models` as the source - models must be in the `app/models/` directory on the host.

### Model Structure

Piper models follow this directory structure:
```
app/models/piper/{model_name}/
  ├── {model_name}.onnx           # Model file
  └── {model_name}.onnx.json      # Config with speaker_id_map
```

The `speaker_id_map` in the JSON config maps speaker names to numeric IDs. See `app/engines/piper.py:30-47` for speaker resolution logic.

## Development Commands

### Local Development
```bash
# Build and run from local source (auto-finds free port)
make dev-up

# Run with pre-built registry image
make up

# Stop containers
make down

# View logs
make logs

# Access container shell
make shell
```

### Testing
```bash
# Run pytest with coverage (requires .venv activated or auto-activates)
make test

# Health check on running container
make health-check
```

### Container Registry Workflow
```bash
# Login (one-time setup)
docker login git.wlkns.org

# Build, tag, and push to registry
make push

# Pull latest from registry
make pull
```

## Key Implementation Details

### TTS Request Flow
1. Request arrives at `/tts` endpoint (line 69)
2. Engine lookup and healthcheck (lines 72-78)
3. Model and speaker validation (lines 81-87)
4. Cache key generation and lookup (lines 90-108)
5. If not cached: synthesize (optionally with chunking) (lines 111-122)
6. Cache result and return URL or base64 (lines 125-143)

### Audio Format Handling
**IMPORTANT**: Piper engine currently returns WAV only. The ffmpeg conversion code is commented out at lines 99-118 in `app/engines/piper.py`. This was done to debug an ffmpeg issue. When re-enabling format conversion:
- Uncomment lines 103-118
- Remove line 101 (direct WAV return)
- Ensure ffmpeg is available in container

### Critical: Piper Engine Input Method (`app/engines/piper.py`)
The Piper engine uses `--input-file` with a temporary file instead of stdin. Previous attempts to use `--stdin` with `asyncio.subprocess.communicate(input=...)` caused issues. Current implementation (lines 71-96):
1. Creates temporary text file with input text
2. Passes file path via `--input-file` flag to piper executable
3. Cleans up temp file in finally block

**Known Issue**: There were reports of the Piper engine occasionally synthesizing incorrect text or causing server crashes. If investigating synthesis issues:
- Check the temporary file creation/cleanup in `piper.py:71-96`
- Verify the text is being written correctly to the temp file
- Test manually inside container: `echo "text" | piper --model /models/piper/{model}/{model}.onnx --output_file test.wav`

### Async Patterns
- All synthesis operations are async
- Blocking I/O (file operations, ffmpeg) wrapped in `asyncio.to_thread()`
- Concurrent chunk synthesis via `asyncio.gather()` (line 115)

### Engine Registration
To add a new engine:
1. Create class in `app/engines/` inheriting from `TTSEngineBase`
2. Add to `ALL_ENGINES` dict in `app/main.py`
3. Add engine name to `ACTIVE_ENGINES` in `.env`

## Testing Strategy

Tests use pytest with async support and fixtures in `tests/conftest.py`:
- `app_client`: TestClient for the FastAPI app
- `piper_engine`: Direct access to PiperEngine instance from registry

Run tests with `make test` which sets `PYTHONPATH` and activates venv automatically.

## Configuration Notes

**`.env` Format**:
```bash
ACTIVE_ENGINES='["piper", "styletts"]'  # JSON array as string
HOST=0.0.0.0
PORT=8000
```

**Registry Configuration** (in Makefile):
- `REGISTRY=git.wlkns.org`
- `USERNAME=stephan`
- `IMAGE_NAME=audio-engine-hub`
- `TAG=latest`

## Common Patterns

### Adding a New Endpoint
Follow the pattern in `app/main.py`:
- Add route to the app instance inside `create_app()`
- Use `app.ENGINE_REGISTRY` to access active engines
- Handle engine not found with 404
- Validate models/speakers before processing

### Working with Engines
```python
# Access from registry
engine = app.ENGINE_REGISTRY.get("piper")

# Always check health before use
health = engine.healthcheck()
if health.get("status") != "ok":
    # Handle unhealthy engine

# Get available resources
models = engine.list_models()
voices = engine.list_voices(model_name)

# Synthesize
audio_path = await engine.synthesize(text, speaker=voice, model=model, fmt="wav")
```

## Docker Multi-Stage Build

The Dockerfile uses a two-stage build:
1. **Builder stage**: Creates venv and installs Python dependencies
2. **Runner stage**: Slim image with ffmpeg, copies venv and app code, runs as unprivileged `appuser`

Production uses gunicorn with uvicorn workers (2 workers default).

## Critical Issues & Lessons Learned

### App Instance Must Be at Module Level
The FastAPI `app` instance MUST be created at module level in `app/main.py` (line 227: `app = create_app()`). Do not only create it inside `if __name__ == "__main__"` block, or uvicorn/gunicorn will fail with "Attribute 'app' not found" error.

### Startup Race Condition with Volume Mounts
The startup event handler (`app/main.py:192-213`) waits for `/models/piper` directory to exist and be non-empty. This prevents race conditions where the app starts before Docker has finished mounting volumes. Max retries: 10, retry delay: 2 seconds.

### Model Path Must Be Absolute
Engine implementations use absolute path `/models/piper/` not relative paths. This ensures consistency regardless of working directory and matches the Docker volume mount structure.

### Engine Status

**Currently Working:**
- `piper`: Functional, uses real Piper TTS executable with ONNX models
- `kokoro`: Fully functional, uses Kokoro-82M TTS library with 54 voices across 8 languages

**Dummy Implementations (for testing only):**
- `styletts`: Returns hardcoded model/voice lists, generates dummy audio
- `chattts`: Dummy implementation
- `f5-tts`: Implemented but may be inactive by default

### Kokoro Engine (`app/engines/kokoro.py`)

**Overview:**
- Uses `kokoro` Python library (KPipeline) for high-performance TTS synthesis
- 82M parameter model delivering ~90× real-time performance on RTX 3090 Ti
- 54 voices across 8 languages: EN-US, EN-GB, FR, ES, JA, ZH, IT, PT, HI, KO
- Outputs 24kHz audio natively (WAV), converts to OGG/MP3 via ffmpeg
- Model size: ~200MB per language, auto-downloaded from Hugging Face on first use
- Voice metadata: `app/engines/kokoro_voices.py` contains all 54 voices with metadata

**Architecture:**
```python
class KokoroEngine(TTSEngineBase):
    def __init__(self):
        # GPU/CPU detection via settings.KOKORO_DEVICE
        # Pipeline instances cached per language code

    async def synthesize(text, speaker, model, fmt):
        # Uses KPipeline for synthesis (24kHz native output)
        # Format conversion via ffmpeg (_run_ffmpeg_blocking)
        # Returns temp file path to generated audio

    def list_models(self):
        # Returns 10 language models:
        # kokoro-en-us, kokoro-en-gb, kokoro-fr, kokoro-es,
        # kokoro-ja, kokoro-zh, kokoro-it, kokoro-pt, kokoro-hi, kokoro-ko

    def list_voices(self, model=None):
        # Returns all 54 voices or filtered by language
        # Uses get_voices_for_model() from kokoro_voices.py
```

**Language Code Mapping:**
The engine maps model names to Kokoro's internal language codes:
- `kokoro-en-us` → `'a'` (American English)
- `kokoro-en-gb` → `'b'` (British English)
- `kokoro-fr` → `'fr'` (French)
- `kokoro-es` → `'es'` (Spanish)
- `kokoro-ja` → `'ja'` (Japanese)
- `kokoro-zh` → `'zh'` (Chinese)
- `kokoro-it` → `'it'` (Italian)
- `kokoro-pt` → `'pt'` (Portuguese)
- `kokoro-hi` → `'hi'` (Hindi)
- `kokoro-ko` → `'ko'` (Korean)

**Voice Organization (`app/engines/kokoro_voices.py`):**
- All 54 voices documented with metadata (gender, language, description)
- Naming convention: `{language}{gender}_{name}` (e.g., `af_bella`, `am_adam`)
- Popular voices: `af_bella`, `af_sarah`, `af_sky`, `am_adam`, `am_michael`
- Helper functions: `get_voices_for_model()`, `get_voice_info()`

**Model Download & Caching:**
- Models auto-download from Hugging Face on first synthesis
- Cached in `~/.cache/huggingface/` (inside container)
- First synthesis may take 30-60s due to download + compilation
- Subsequent syntheses are fast (~90× real-time on GPU)

**Configuration (`app/config.py`):**
- `KOKORO_DEVICE`: "cuda" or "cpu" (default: "cuda")
- `KOKORO_TIMEOUT_SECONDS`: Synthesis timeout (default: 30)

**Error Handling:**
- Applies all bug fixes from Piper engine (timeouts, temp file cleanup, logging)
- Graceful GPU fallback if CUDA unavailable
- Voice validation before synthesis
- Comprehensive error logging with context

### Debugging Tips

**Port Conflicts**: The Makefile automatically finds free ports starting from 8000. Run with `sudo make dev-up` for most reliable port detection.

**Container Crashes**: If the container crashes during synthesis:
1. Check logs: `make logs`
2. Disable uvicorn reload in docker-compose.yml command to see full tracebacks
3. Test piper executable directly inside container: `make shell`

**Model Loading Issues**: If `list_models()` returns empty:
1. Verify volume mount in docker-compose.yml points to `./app/models:/models`
2. Check models exist on host: `ls app/models/piper/`
3. Check inside container: `docker exec -it audio-engine-hub_app ls -la /models/piper/`

**Connection Reset Errors in Client**: Add small delay (`time.sleep(1)`) before making requests if experiencing `ConnectionResetError`. This indicates server needs time to fully initialize.