audio-engine-hub/KOKORO_IMPLEMENTATION_STATUS.md

# Kokoro TTS Implementation Status

**Date Started**: 2025-12-05
**Last Updated**: 2025-12-05 00:48 CET
**Status**: 🟡 In Progress (Phase 2 & 3 partially complete)

---

## Quick Summary

We are implementing Kokoro-82M TTS engine as a new engine in AudioEngineHub. Kokoro is a lightweight 82M parameter model that delivers quality comparable to models 5-15× its size, with ~90× real-time performance on consumer GPUs.

**Progress**: ~40% complete
- ✅ Phase 1: Research & Planning (100%)
- ✅ Phase 2: Engine Implementation (100%)
- 🟡 Phase 3: Docker Integration (50%)
- ⏳ Phase 4: Configuration (0%)
- ⏳ Phase 5: Testing (0%)
- ⏳ Phase 6: Documentation (0%)

---

## What We've Completed

### ✅ Phase 1: Research & Planning
- [x] Researched Kokoro TTS capabilities and architecture
- [x] Identified Python library (`kokoro>=0.9.2`)
- [x] Documented all 54 voices across 8 languages
- [x] Created comprehensive implementation plan in `KOKORO_IMPLEMENTATION_PLAN.md`

**Key Findings:**
- Kokoro uses 82M parameters (extremely lightweight)
- Performance: ~90× real-time on RTX 3090 Ti, ~210× on RTX 4090
- Output: 24kHz high-fidelity audio
- License: Apache 2.0 (open-source, commercial use allowed)
- 54 voices across 8 languages (EN-US, EN-GB, JA, ZH, ES, FR, HI, IT, PT)

### ✅ Phase 2: Engine Implementation (Complete)
- [x] **Created `app/engines/kokoro.py`** - Full engine implementation with:
  - Complete `KokoroEngine` class implementing `TTSEngineBase`
  - `synthesize()` method with all bug fixes from Piper (timeouts, cleanup, logging)
  - GPU/CPU support with automatic fallback
  - Language code mapping (10 models: kokoro-en-us, kokoro-en-gb, etc.)
  - Format conversion (WAV/OGG/MP3) via ffmpeg
  - Comprehensive error handling and logging
  - Model pipeline caching for performance

- [x] **Created `app/engines/kokoro_voices.py`** - Voice metadata with:
  - Complete list of all 54 Kokoro voices
  - Voice metadata (gender, language, description)
  - Language-based voice filtering
  - Helper functions: `get_voices_for_model()`, `get_voice_info()`

**File Locations:**
- `/home/stephan/Projekte/KI/AudioEngineHub/app/engines/kokoro.py` (316 lines)
- `/home/stephan/Projekte/KI/AudioEngineHub/app/engines/kokoro_voices.py` (222 lines)

### 🟡 Phase 3: Docker Integration (50% Complete)
- [x] **Updated `Dockerfile`** - Added espeak-ng system dependency
  - Line 17: Added `espeak-ng` to apt-get install

- [ ] **Update `requirements.txt`** - Need to add Kokoro Python dependencies
  - **NEXT STEP**: Add these lines to requirements.txt:
    ```
    # Kokoro TTS Engine
    kokoro>=0.9.2
    soundfile
    phonemizer
    scipy
    munch
    ```

---

## What's Left To Do

### ⏳ Phase 4: Configuration (Not Started)

#### 1. Update `app/config.py`
Add Kokoro configuration settings:
```python
class Settings(BaseSettings):
    # ... existing settings ...

    # Kokoro Engine Configuration
    KOKORO_DEVICE: str = "cuda"  # or "cpu"
    KOKORO_TIMEOUT_SECONDS: int = 30
```

**File**: `/home/stephan/Projekte/KI/AudioEngineHub/app/config.py`
**Location**: Add after line 25 (after FFMPEG_TIMEOUT_SECONDS)

#### 2. Update `app/main.py`
Register Kokoro engine in the engine registry:
```python
from app.engines.kokoro import KokoroEngine  # Add this import

ALL_ENGINES = {
    "piper": PiperEngine,
    "styletts": StyleTTSEngine,
    "chattts": ChatTTSEngine,
    "f5-tts": F5TTSEngine,
    "kokoro": KokoroEngine,  # Add this line
}
```

**File**: `/home/stephan/Projekte/KI/AudioEngineHub/app/main.py`
**Location**: Line 28 (import), Line 38 (registry)

#### 3. Update `.env.example`
Add Kokoro configuration documentation:
```bash
# Kokoro Engine Configuration
KOKORO_DEVICE=cuda  # cuda or cpu
KOKORO_TIMEOUT_SECONDS=30
```

**File**: `/home/stephan/Projekte/KI/AudioEngineHub/.env.example`
**Location**: Add after FFMPEG_TIMEOUT_SECONDS

#### 4. Optional: Update `.env`
To enable Kokoro by default:
```bash
ACTIVE_ENGINES='["piper", "kokoro"]'
```

---

### ⏳ Phase 5: Testing (Not Started)

#### 1. Test Import and Basic Functionality
```bash
# Test engine loads without errors
python3 -c "from app.engines.kokoro import KokoroEngine; print(KokoroEngine().healthcheck())"
```

Expected output:
```json
{
  "status": "not_available",  # OK if kokoro not installed yet
  "engine": "kokoro",
  "library_available": false,
  "error": "Kokoro library not installed..."
}
```

#### 2. Install Kokoro Dependencies (Outside Docker First)
```bash
# Install espeak-ng
sudo apt-get install espeak-ng

# Install Python packages
pip install kokoro>=0.9.2 soundfile phonemizer scipy munch
```

#### 3. Test Synthesis Locally
```bash
# Run engine selftest
cd /home/stephan/Projekte/KI/AudioEngineHub
python3 app/engines/kokoro.py
```

Expected output:
```
Healthcheck: {'status': 'ok', 'engine': 'kokoro', ...}
Models: ['kokoro-en-us', 'kokoro-en-gb', ...]
Voices: ['af_alloy', 'af_aoede', 'af_bella', ...] ...
Selftest: {'selftest': True, 'models': [...], 'voices_count': 54}
```

#### 4. Test via API
```bash
# Start development server
make dev-up

# Test API endpoint
curl -X POST http://localhost:8000/tts \
  -H "Content-Type: application/json" \
  -d '{"text":"Hello from Kokoro!","engine":"kokoro","model":"kokoro-en-us","speaker":"af_bella","format":"ogg"}'
```

#### 5. Test with Voice Wizard
```bash
# Launch voice wizard and test preview with Kokoro voices
cd /home/stephan/Games/University/tts
./scripts/voice-assignment-wizard.sh
```

---

### ⏳ Phase 6: Documentation (Not Started)

#### 1. Update `README.md`
Add Kokoro to supported engines list:
```markdown
## Supported TTS Engines

- **Piper** - Fast, lightweight, ONNX-based TTS with 100+ voices
- **Kokoro** - 82M parameter high-quality TTS with 54 voices across 8 languages 🆕
- **StyleTTS** - Expressive style-based TTS (coming soon)
- **ChatTTS** - Conversational TTS (coming soon)
- **F5-TTS** - Advanced flow-based TTS (coming soon)
```

#### 2. Update `CLAUDE.md`
Add Kokoro engine section:
```markdown
### Kokoro Engine (`app/engines/kokoro.py`)
- **Library**: `kokoro` Python package (KPipeline)
- **Models**: 10 language models (en-us, en-gb, ja, zh, es, fr, hi, it, pt, ko)
- **Voices**: 54 voices with gender and language metadata
- **Output**: 24kHz audio natively (WAV), converts to OGG/MP3 via ffmpeg
- **Performance**: ~90× real-time on RTX 3090 Ti
- **Model Size**: 82M parameters (~200MB download per language)
- **Voice Metadata**: `app/engines/kokoro_voices.py` contains all 54 voices with metadata
```

#### 3. Create Usage Guide (Optional)
Create `docs/kokoro-usage.md` with:
- Voice selection guide
- Language support matrix
- Performance tips (GPU vs CPU)
- Troubleshooting common issues

---

## Implementation Checklist

### Phase 1: Research ✅
- [x] Research Kokoro capabilities
- [x] Document API and dependencies
- [x] Create implementation plan

### Phase 2: Engine Implementation ✅
- [x] Create `app/engines/kokoro.py`
- [x] Implement `synthesize()` method
- [x] Implement `list_models()` method
- [x] Implement `list_voices()` method
- [x] Implement `healthcheck()` method
- [x] Implement `selftest()` method
- [x] Create `app/engines/kokoro_voices.py`

### Phase 3: Docker Integration 🟡
- [x] Update Dockerfile (added espeak-ng)
- [ ] **→ NEXT: Update requirements.txt (add Kokoro packages)**
- [ ] Test Docker build
- [ ] Verify GPU access in container (if available)

### Phase 4: Configuration ⏳
- [ ] Update `app/config.py` with Kokoro settings
- [ ] Update `.env.example` with Kokoro variables
- [ ] Add KokoroEngine import to `app/main.py`
- [ ] Add Kokoro to ALL_ENGINES in `app/main.py`
- [ ] Optional: Update `.env` to enable Kokoro

### Phase 5: Testing ⏳
- [ ] Test engine imports without errors
- [ ] Install Kokoro dependencies locally
- [ ] Run selftest (`python3 app/engines/kokoro.py`)
- [ ] Test synthesis via API
- [ ] Test with voice wizard
- [ ] Verify caching works correctly
- [ ] Check for temp file leaks
- [ ] Performance benchmarks

### Phase 6: Documentation ⏳
- [ ] Update README.md
- [ ] Update CLAUDE.md
- [ ] Optional: Create Kokoro usage guide
- [ ] Document voice selection
- [ ] Add troubleshooting section

### Phase 7: Deployment ⏳
- [ ] Rebuild Docker image
- [ ] Test in development environment
- [ ] Commit changes to git
- [ ] Push to remote
- [ ] Tag release (optional)

---

## Files Created/Modified

### ✅ Created Files
1. **`KOKORO_IMPLEMENTATION_PLAN.md`** - Complete implementation plan (280 lines)
2. **`app/engines/kokoro.py`** - Kokoro engine implementation (316 lines)
3. **`app/engines/kokoro_voices.py`** - Voice metadata (222 lines)
4. **`KOKORO_IMPLEMENTATION_STATUS.md`** - This file

### ✅ Modified Files
1. **`Dockerfile`** - Added espeak-ng dependency (line 17)

### ⏳ Files To Modify
1. **`requirements.txt`** - Add Kokoro Python packages
2. **`app/config.py`** - Add Kokoro configuration
3. **`app/main.py`** - Register Kokoro engine
4. **`.env.example`** - Document Kokoro config
5. **`README.md`** - Add Kokoro to engines list
6. **`CLAUDE.md`** - Add Kokoro engine details

---

## Next Session: Action Plan

### Step 1: Complete Docker Integration (5 minutes)
```bash
cd /home/stephan/Projekte/KI/AudioEngineHub

# Add to requirements.txt (after line 11):
cat >> requirements.txt << 'EOF'

# Kokoro TTS Engine
kokoro>=0.9.2
soundfile
phonemizer
scipy
munch
EOF
```

### Step 2: Complete Configuration (10 minutes)
1. Edit `app/config.py` - add KOKORO_DEVICE and KOKORO_TIMEOUT_SECONDS
2. Edit `app/main.py` - import KokoroEngine and add to ALL_ENGINES
3. Edit `.env.example` - document new config options

### Step 3: Test Locally (15 minutes)
1. Install dependencies: `sudo apt-get install espeak-ng && pip install kokoro>=0.9.2 soundfile phonemizer scipy munch`
2. Run selftest: `python3 app/engines/kokoro.py`
3. Start server: `make dev-up` (will rebuild Docker image)
4. Test API synthesis with Kokoro
5. Test voice wizard preview

### Step 4: Documentation (10 minutes)
1. Update README.md with Kokoro
2. Update CLAUDE.md with Kokoro engine details
3. Commit all changes to git

### Total Time Remaining: ~40 minutes

---

## Key Technical Details

### Language Code Mapping
```python
KOKORO_LANG_CODES = {
    "kokoro-en-us": "a",   # American English
    "kokoro-en-gb": "b",   # British English
    "kokoro-fr": "fr",     # French
    "kokoro-es": "es",     # Spanish
    "kokoro-ja": "ja",     # Japanese
    "kokoro-zh": "zh",     # Chinese
    "kokoro-it": "it",     # Italian
    "kokoro-pt": "pt",     # Portuguese
    "kokoro-hi": "hi",     # Hindi
    "kokoro-ko": "ko",     # Korean
}
```

### Voice Organization
- **54 total voices** across 8 languages
- **Naming convention**: `{language}{gender}_{name}`
  - `af_` = American Female
  - `am_` = American Male
  - `bf_` = British Female
  - `bm_` = British Male
  - `jf_/jm_` = Japanese F/M
  - `zf_/zm_` = Chinese F/M
  - etc.
- **Most popular**: af_bella, af_sarah, af_sky, am_adam, am_michael

### API Usage Example
```json
{
  "text": "Hello from Kokoro TTS!",
  "engine": "kokoro",
  "model": "kokoro-en-us",
  "speaker": "af_bella",
  "format": "ogg"
}
```

---

## Known Issues & Considerations

### 1. Model Download on First Use
- **Issue**: First synthesis will be slow (~30-60s) due to model download
- **Size**: ~200MB per language model
- **Location**: Models cached in `~/.cache/huggingface/`
- **Solution**: Expected behavior, document in README

### 2. GPU vs CPU Performance
- **GPU**: ~90× real-time (RTX 3090 Ti)
- **CPU**: ~5-10× real-time (estimated)
- **Fallback**: Code supports both, configurable via KOKORO_DEVICE env var

### 3. Voice-Language Compatibility
- **Issue**: Not all voices work with all languages
- **Solution**: `kokoro_voices.py` filters voices by language
- **API**: `/speakers?engine=kokoro&model=kokoro-en-us` returns only compatible voices

### 4. Dependencies
- **espeak-ng**: Required system package for phonemization
- **soundfile**: Required for WAV file I/O
- **phonemizer**: Required for text-to-phoneme conversion
- **scipy, munch**: Required by Kokoro library

---

## Testing Commands Reference

### Local Testing (Outside Docker)
```bash
# Install system dependency
sudo apt-get install espeak-ng

# Install Python packages
pip install kokoro>=0.9.2 soundfile phonemizer scipy munch

# Test engine
cd /home/stephan/Projekte/KI/AudioEngineHub
python3 app/engines/kokoro.py

# Test voice metadata
python3 -c "from app.engines.kokoro_voices import ALL_VOICES, get_voices_for_model; print(f'Total voices: {len(ALL_VOICES)}'); print(f'EN-US voices: {get_voices_for_model(\"kokoro-en-us\")}')"
```

### Docker Testing
```bash
# Rebuild and start
make down
make dev-up

# Check logs
make logs

# Health check
make health-check

# Test synthesis
curl -X POST http://localhost:8000/tts \
  -H "Content-Type: application/json" \
  -d '{"text":"Testing Kokoro TTS engine","engine":"kokoro","model":"kokoro-en-us","speaker":"af_bella","format":"ogg"}' | jq
```

### Voice Wizard Testing
```bash
cd /home/stephan/Games/University/tts
./scripts/voice-assignment-wizard.sh

# Should show Kokoro engine in dropdown
# Should list 54 voices (or subset by language)
# Preview should work with all Kokoro voices
```

---

## Success Criteria

Before marking this implementation as complete, verify:

- [ ] Kokoro engine passes healthcheck
- [ ] All 10 language models are listed in `/models` endpoint
- [ ] 54 voices are accessible via `/speakers` endpoint
- [ ] Voice filtering by model/language works correctly
- [ ] Synthesis produces valid audio files
- [ ] Synthesis speed: >50× real-time on GPU (if available)
- [ ] Format conversion works (WAV/OGG/MP3)
- [ ] Cache versioning works correctly
- [ ] No temp file leaks after synthesis
- [ ] No hung processes under load
- [ ] Voice wizard integration works
- [ ] Documentation is complete and accurate

---

## Resources & References

### Official Documentation
- [Kokoro-82M Hugging Face](https://huggingface.co/hexgrad/Kokoro-82M)
- [VOICES.md - Complete Voice List](https://huggingface.co/hexgrad/Kokoro-82M/blob/main/VOICES.md)
- [Kokoro GitHub Repository](https://github.com/hexgrad/kokoro)
- [Kokoro Official Website](https://kokorotts.net/)

### Technical Articles
- [Analytics Vidhya - Kokoro-82M Review](https://www.analyticsvidhya.com/blog/2025/01/kokoro-82m/)

### Installation Guides
- [Kokoro-82M Installation Instructions](https://huggingface.co/hexgrad/Kokoro-82M#installation)
- [Kokoro Live Demo](https://huggingface.co/spaces/hexgrad/Kokoro-TTS)

---

## Timeline & Estimates

| Phase | Status | Time Spent | Remaining | Total |
|-------|--------|------------|-----------|-------|
| Phase 1: Research | ✅ Complete | 2h | 0h | 2h |
| Phase 2: Implementation | ✅ Complete | 2h | 0h | 2h |
| Phase 3: Docker | 🟡 50% | 0.5h | 0.5h | 1h |
| Phase 4: Configuration | ⏳ Pending | 0h | 0.5h | 0.5h |
| Phase 5: Testing | ⏳ Pending | 0h | 1h | 1h |
| Phase 6: Documentation | ⏳ Pending | 0h | 0.5h | 0.5h |
| **Total** | **40% Complete** | **4.5h** | **2.5h** | **7h** |

**Original Estimate**: 8-12 hours
**Current Progress**: ~5 hours spent, ~2-3 hours remaining
**On Track**: Yes, ahead of schedule

---

## Questions for User (Next Session)

1. **GPU Availability**: Do you have a CUDA-capable GPU? This affects:
   - Performance expectations (90× vs 5× real-time)
   - Default KOKORO_DEVICE setting (cuda vs cpu)

2. **Language Priority**: Which languages do you need most?
   - Only EN-US model will be tested initially
   - Other languages can be tested on demand

3. **Docker vs Local**: Prefer testing locally first or directly in Docker?
   - Local testing is faster for iteration
   - Docker testing validates full deployment

4. **Voice Wizard**: Should we integrate Kokoro metadata into the voice wizard UI?
   - Could show language, gender, description for each voice
   - Requires changes to audioengine_client.py

---

**End of Status Report**

Last updated: 2025-12-05 00:48 CET
Next session: Continue with Step 1 (Complete Docker Integration)