Fix XTTS loader compatibility and add default voice

This commit is contained in:
2025-12-09 11:37:55 +01:00
parent 8cd91b6f22
commit ca3e66f17a
14 changed files with 483 additions and 128 deletions

31
AGENTS.md Normal file
View File

@ -0,0 +1,31 @@
# Repository Guidelines
## Project Structure & Modules
- Gateway (`gateway/`): FastAPI entrypoint in `main.py`, routes under `api/` (`health.py`, `openai_speech.py`, `voices.py`), shared helpers in `core/`, voice registry in `voices/` and runtime port in `port.txt`.
- Worker (`worker/`): Queue consumer in `core/queue_worker.py`, XTTS2 synthesis in `engine/xtts2_loader.py`, audio export in `engine/audio_export.py`, voice assets in `voices/`.
- Tooling: `Makefile` drives Docker workflow, `docker-compose.yml` wires gateway/worker/redis, helper scripts in `scripts/` (`find_port.py`, `selftest.py`).
## Build, Test, and Run
- `make build` – build gateway + worker images.
- `make up` – start stack with auto port selection (writes `gateway/port.txt`).
- `make status` / `make logs` – check containers and follow logs.
- `make selftest` – end-to-end smoke test against the running stack (health + sample synthesis).
- Local debug (no Docker): install deps with `pip install -r gateway/requirements.gateway.txt` and `pip install -r worker/requirements.worker.txt`, then `python gateway/main.py` and `python worker/main.py`; start Redis via `docker run -p 6379:6379 redis:7`.
## Coding Style & Naming
- Python, prefer PEP8 with 4-space indents and snake_case names for modules, functions, and vars; keep route names aligned with OpenAI-compatible paths (`/v1/audio/speech`, `/v1/voices/register`).
- Keep modules small and focused (API logic in `gateway/api`, queue/Redis helpers in `core`).
- Favor explicit config via env vars (`REDIS_HOST`, `GATEWAY_PORT`); avoid hardcoded ports besides the 8000–8100 scan range.
## Testing Guidelines
- Primary check is the smoke test: run `make selftest` after changes that touch API, queue, or audio paths.
- For new logic, add lightweight unit tests (e.g., under `gateway/tests/` or `worker/tests/`) named `test_<feature>.py`; prefer pytest-style asserts.
- When adding audio or queue code, include sanity checks (e.g., validate `mime` and byte length) to avoid silent failures.
## Commit & Pull Request Practices
- Commits: short, imperative subjects (e.g., `add queue timeout guard`, `tune xtts export`). Group related changes; avoid mixing refactors with feature work.
- Pull Requests: describe intent, list test commands executed (e.g., `make selftest`), mention affected endpoints or worker behaviors, and link issues when available. Provide screenshots or audio sample paths only if UX or output format changes.
## Security & Operations Notes
- Do not commit voice assets beyond small samples; keep secrets out of the repo and prefer env vars or Docker secrets.
- Gateway listens on the selected local port only; expose externally via reverse proxy/HTTPS in production. Keep worker services internal and behind the queue.

133
GEMINI.md Normal file
View File

@ -0,0 +1,133 @@
# XTTS2 OpenAI-Compatible TTS Server
## Project Overview
This project is a high-performance, modular, and scalable Text-to-Speech (TTS) platform. It provides an API fully compatible with the **OpenAI Speech API**, powered by the **XTTS2** model for high-quality synthesis and zero-shot voice cloning.
### Architecture
The system follows a distributed architecture:
* **Gateway (`gateway/`):** A FastAPI service that handles HTTP requests, validates input, and manages the voice registry. It pushes synthesis jobs to a Redis queue. It features dynamic port selection (8000-8100).
* **Redis:** Acts as the message broker (Queue) and cache between the Gateway and Workers.
* **Worker (`worker/`):** A background service that pulls jobs from Redis, performs the actual TTS inference using XTTS2 (with GPU acceleration if available), and returns the audio data. These can be scaled horizontally.
### Key Technologies
* **Language:** Python 3.10+
* **Framework:** FastAPI (Gateway)
* **ML Model:** Coqui XTTS v2
* **Infrastructure:** Docker, Docker Compose, Redis
* **Tooling:** Makefile for orchestration
## Building and Running
The project relies heavily on `make` for orchestration.
### Docker (Recommended)
1. **Build Images:**
```bash
make build
```
2. **Start Services:**
```bash
make up
```
* This runs a port scanner to find a free port between 8000-8100.
* The chosen port is saved to `gateway/port.txt`.
3. **Check Status:**
```bash
make status
```
4. **View Logs:**
```bash
make logs
```
5. **Stop Services:**
```bash
make down
```
### Scaling Workers
To handle higher load, you can spawn multiple worker containers:
```bash
make worker-scale N=3
```
### Verification
Run the self-test suite to verify Redis connectivity, worker processing, and audio synthesis:
```bash
make selftest
```
## Development Conventions
### Project Structure
* `gateway/`: Code for the API server.
* `main.py`: Entry point.
* `api/`: Endpoint definitions (`openai_speech.py`, `voices.py`).
* `core/`: Configuration and utilities.
* `worker/`: Code for the inference engine.
* `engine/`: XTTS2 model loading and audio export logic.
* `core/`: Queue processing and GPU detection.
* `scripts/`: Utility scripts (e.g., `find_port.py`, `selftest.py`).
### Local Development (Non-Docker)
1. Create a virtual environment:
```bash
python3 -m venv .venv
source .venv/bin/activate
```
2. Install dependencies:
```bash
pip install -r gateway/requirements.gateway.txt
pip install -r worker/requirements.worker.txt
```
3. Run Redis locally (e.g., `docker run -p 6379:6379 redis:7`).
4. Start Gateway: `python gateway/main.py`
5. Start Worker: `python worker/main.py`
### API Usage
The API mirrors OpenAI's structure.
**Generate Audio:**
```http
POST /v1/audio/speech
Content-Type: application/json
{
"model": "xtts-v2",
"input": "Hello world",
"voice": "auto",
"format": "wav"
}
```
**Register Voice:**
```http
POST /v1/voices/register
Content-Type: application/json
{
"name": "my-voice",
"samples": ["https://example.com/sample.wav"]
}
```
### Logging & Debugging
* **Gateway Logs:** `gateway/logs/gateway.log`
* **Port Info:** `gateway/port.txt` contains the active port.
* **GPU:** Workers will automatically detect and use CUDA if available. Check `nvidia-smi` to monitor usage.

View File

@ -10,120 +10,100 @@ DOCKER := docker compose
.DEFAULT_GOAL := help
help:
@echo ""
@echo "🚀 XTTS2 TTS Server – Makefile (Pro Mode)"
@echo "-------------------------------------------"
@echo " make build → Images bauen (Gateway + Worker)"
@echo " make up → Services starten (mit Portscan)"
@echo " make down → Services stoppen"
@echo " make restart → Neustart"
@echo " make logs → Logs aller Services anzeigen"
@echo " make status → Docker Status"
@echo " make worker-scale N=3 → Worker skalieren"
@echo " make prune → Docker aufräumen"
@echo " make selftest → System-Selbsttest"
@echo " make port → Zeigt aktuellen Gateway Port"
@echo "-------------------------------------------"
@echo ""
@echo "🚀 XTTS2 TTS Server – Makefile (Pro Mode)"
@echo "-------------------------------------------"
@echo " make build → Images bauen (Gateway + Worker)"
@echo " make up → Services starten (mit Portscan)"
@echo " make down → Services stoppen"
@echo " make restart → Neustart"
@echo " make logs → Logs aller Services anzeigen"
@echo " make status → Docker Status"
@echo " make worker-scale N=3 → Worker skalieren"
@echo " make prune → Docker aufräumen"
@echo " make selftest → System-Selbsttest"
@echo " make port → Zeigt aktuellen Gateway Port"
@echo "-------------------------------------------"
# ---------------------------------------------------------
# Build
# ---------------------------------------------------------
build:
$(DOCKER) build
$(DOCKER) build
# ---------------------------------------------------------
# Deploy
# ---------------------------------------------------------
up:
@echo "🔍 Suche freien Port zwischen 8000–8100..."
@PORT=`$(PYTHON) $(PORT_SCRIPT)`;
if [ "$$PORT" = "ERR_NO_FREE_PORT" ]; then
echo "❌ Kein freier Port gefunden!"; exit 1;
fi;
echo "🎧 Freier Port gefunden: $$PORT";
echo "$$PORT" > $(PORT_FILE);
echo "📄 Port gespeichert in $(PORT_FILE)";
export GATEWAY_PORT=$$PORT;
$(DOCKER) up -d --build;
echo "🚀 Gateway läuft auf [http://localhost:$$PORT](http://localhost:$$PORT)"
@echo "🔍 Suche freien Port zwischen 8000–8100..."
@PORT=`$(PYTHON) $(PORT_SCRIPT)`; \
if [ "$$PORT" = "ERR_NO_FREE_PORT" ]; then \
echo "❌ Kein freier Port gefunden!"; exit 1; \
fi; \
echo "🎧 Freier Port gefunden: $$PORT"; \
echo "$$PORT" > $(PORT_FILE); \
echo "📄 Port gespeichert in $(PORT_FILE)"; \
export GATEWAY_PORT=$$PORT; \
$(DOCKER) up -d --build; \
echo "🚀 Gateway läuft auf http://localhost:$$PORT"
# ---------------------------------------------------------
# Stop
# ---------------------------------------------------------
down:
$(DOCKER) down
$(DOCKER) down
# ---------------------------------------------------------
# Restart
# ---------------------------------------------------------
restart: down up
# ---------------------------------------------------------
# Logs
# ---------------------------------------------------------
logs:
$(DOCKER) logs -f
$(DOCKER) logs -f
# ---------------------------------------------------------
# Status
# ---------------------------------------------------------
status:
$(DOCKER) ps
$(DOCKER) ps
# ---------------------------------------------------------
# Worker Scaling
# ---------------------------------------------------------
worker-scale:
@if [ -z "$(N)" ]; then echo "Bitte N angeben: make worker-scale N=3"; exit 1; fi
$(DOCKER) up -d --scale worker=$(N)
@if [ -z "$(N)" ]; then echo "Bitte N angeben: make worker-scale N=3"; exit 1; fi
$(DOCKER) up -d --scale worker=$(N)
# ---------------------------------------------------------
# Cleanup
# ---------------------------------------------------------
prune:
$(DOCKER) down
docker system prune -f
$(DOCKER) down
docker system prune -f
# ---------------------------------------------------------
# Selftest
# ---------------------------------------------------------
selftest:
@echo "🧪 Starte Selbsttest..."
$(PYTHON) scripts/selftest.py
@echo "🧪 Starte Selbsttest..."
$(PYTHON) scripts/selftest.py
# ---------------------------------------------------------
# Show Port
# ---------------------------------------------------------
port:
@echo "📡 Aktueller Port:"
@cat $(PORT_FILE)
@echo "📡 Aktueller Port:"
@cat $(PORT_FILE)

View File

@ -1,54 +1,55 @@
# docker-compose.yml – XTTS2 TTS Server
# Multi-Service Orchestrierung (Gateway, Worker, Redis)
version: "3.9"
services:
redis:
image: redis:7
container_name: redis
restart: always
ports:
- "6379:6379"
redis:
image: redis:7
container_name: redis
restart: always
ports:
- "6379:6379"
gateway:
build:
context: ./gateway
dockerfile: Dockerfile
container_name: tts-gateway
restart: always
environment:
- REDIS_HOST=redis
- GATEWAY_PORT=${GATEWAY_PORT}
volumes:
- ./gateway/voices:/app/voices
- ./gateway/logs:/app/logs
- ./gateway/port.txt:/app/port.txt
ports:
- "${GATEWAY_PORT}:8000"
depends_on:
- redis
gateway:
build:
context: ./gateway
dockerfile: Dockerfile
container_name: tts-gateway
restart: always
environment:
- REDIS_HOST=redis
- GATEWAY_PORT=${GATEWAY_PORT}
volumes:
- ./gateway/voices:/app/voices
- ./gateway/logs:/app/logs
- ./gateway/port.txt:/app/port.txt
ports:
- "${GATEWAY_PORT:-8000}:${GATEWAY_PORT:-8000}"
depends_on:
- redis
worker:
build:
context: ./worker
dockerfile: Dockerfile
container_name: tts-worker
restart: always
environment:
- REDIS_HOST=redis
- GPU_MODE=AUTO
deploy:
resources:
reservations:
devices:
- capabilities: [gpu]
volumes:
- ./worker/voices:/app/voices
- tts-models:/root/.local/share/tts
depends_on:
- redis
worker:
build:
context: ./worker
dockerfile: Dockerfile
container_name: tts-worker
restart: always
environment:
- REDIS_HOST=redis
- GPU_MODE=AUTO
deploy:
resources:
reservations:
devices:
- capabilities: [gpu]
volumes:
- ./worker/voices:/app/voices
depends_on:
- redis
tts-models:
networks:
default:
name: wlkns-net
default:
name: wlkns-net

View File

@ -8,6 +8,7 @@ class SpeechRequest(BaseModel):
model: str
input: str
voice: str = "auto"
language: str | None = None
format: str = "wav"
voice_sample_url: str | None = None
voice_sample_base64: str | None = None

View File

@ -12,7 +12,7 @@ def push_job(data: dict) -> str:
r.lpush(QUEUE, json.dumps(data))
return job_id
def await_result(job_id: str, timeout=30):
def await_result(job_id: str, timeout=120):
key = f"{RESULT}:{job_id}"
start=time.time()
while time.time()-start < timeout:

1
gateway/port.txt Normal file
View File

@ -0,0 +1 @@
8003

View File

@ -1,7 +1,8 @@
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 AS base
ENV PYTHONUNBUFFERED=1
WORKDIR /app
COPY requirements.worker.txt .
RUN apt-get update && apt-get install -y python3-pip ffmpeg
RUN apt-get update && apt-get install -y python3-pip ffmpeg espeak-ng
RUN pip3 install -r requirements.worker.txt
COPY . .
CMD ["python3","main.py"]

View File

@ -1,19 +1,39 @@
from pydub import AudioSegment
import io
import numpy as np
def convert_audio(audio_data, fmt: str, sample_rate: int = 24000):
"""
Converts raw audio data (numpy array or list of floats) to the target format.
Assumes mono audio.
"""
# Ensure numpy array
if not isinstance(audio_data, np.ndarray):
audio_data = np.array(audio_data)
# Check if float and normalize/convert to int16
if audio_data.dtype.kind == 'f':
# Clip to Avoid wrap-around
audio_data = np.clip(audio_data, -1.0, 1.0)
# Convert to 16-bit PCM
audio_data = (audio_data * 32767).astype(np.int16)
def convert_audio(raw_bytes: bytes, fmt: str):
# raw mono 32-bit float fake waveform
seg = AudioSegment(
raw_bytes,
frame_rate=22050,
sample_width=4,
audio_data.tobytes(),
frame_rate=sample_rate,
sample_width=2, # 16-bit
channels=1
)
buf=io.BytesIO()
buf = io.BytesIO()
seg.export(buf, format=fmt)
mime={
"wav":"audio/wav",
"mp3":"audio/mpeg",
"ogg":"audio/ogg"
}.get(fmt,"audio/wav")
mime = {
"wav": "audio/wav",
"mp3": "audio/mpeg",
"ogg": "audio/ogg",
"flac": "audio/flac",
"aac": "audio/aac"
}.get(fmt, "audio/wav")
return buf.getvalue(), mime

View File

@ -1,10 +1,163 @@
# Dummy XTTS2 logic placeholder
# Replace with real TTS model loading
import os
# Auto-agree to Coqui TOS (Must be before imports)
os.environ["COQUI_TOS_AGREED"] = "1"
import json
import base64
import tempfile
import requests
import torch
import numpy as np
# Singleton for lazy loading
_model = None
def _allow_xtts_config_pickle():
"""Allow loading XTTS configs with torch >=2.6 safe loading."""
add_safe = getattr(torch.serialization, "add_safe_globals", None)
if not add_safe:
return
allowed = []
try:
from TTS.tts.configs.xtts_config import XttsConfig
from TTS.tts.models.xtts import XttsAudioConfig
allowed += [XttsConfig, XttsAudioConfig]
except Exception as e:
print(f"⚠️ Could not register safe globals for XTTS config: {e}")
try:
import TTS.config.shared_configs as shared_configs
allowed += [v for v in shared_configs.__dict__.values() if isinstance(v, type)]
except Exception as e:
print(f"⚠️ Could not register shared config globals: {e}")
try:
import TTS.tts.models.xtts as xtts_models
allowed += [v for v in xtts_models.__dict__.values() if isinstance(v, type)]
except Exception as e:
print(f"⚠️ Could not register XTTS model globals: {e}")
if allowed:
add_safe(allowed)
def get_model():
global _model
if _model is None:
print("⏳ Loading XTTS Model (Lazy Load)....")
# Lazy Import to prevent startup hang
from TTS.api import TTS
_allow_xtts_config_pickle()
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"🔧 XTTS Running on: {device}")
# Load Model (download if needed)
# Using default XTTS v2 model
_model = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to(device)
print("✅ XTTS Model loaded successfully.")
return _model
def synthesize(job: dict):
# return artificial sine wave placeholder
import numpy as np
sr=22050
t=np.linspace(0,0.3,int(sr*0.3))
tone=(0.1*np.sin(2*np.pi*440*t)).astype('float32')
return tone.tobytes()
from langdetect import detect
model = get_model()
...
text = job.get("input")
if not text:
raise ValueError("No input text provided")
# Language handling
language = job.get("language")
if not language:
try:
# Simple detection
detected = detect(text)
# XTTS expects 2-letter codes usually.
# We assume detected is valid or mapped if needed.
# Supported: en, es, fr, de, it, pt, pl, tr, ru, nl, cs, ar, zh-cn, ja, hu, ko
language = detected
print(f"🌍 Auto-detected language: {language}")
except:
language = "en"
print("⚠️ Language detection failed, using 'en'")
# Speaker Handling
speaker_wav = None
temp_files = []
try:
# Priority 1: Direct URL
if job.get("voice_sample_url"):
try:
print(f"⬇️ Downloading voice sample from {job['voice_sample_url']}")
r = requests.get(job["voice_sample_url"], timeout=10)
r.raise_for_status()
t = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
t.write(r.content)
t.close()
speaker_wav = t.name
temp_files.append(t.name)
except Exception as e:
print(f"❌ Failed to download voice sample: {e}")
# Priority 2: Base64
if not speaker_wav and job.get("voice_sample_base64"):
try:
b64 = job["voice_sample_base64"]
decoded = base64.b64decode(b64)
t = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
t.write(decoded)
t.close()
speaker_wav = t.name
temp_files.append(t.name)
except Exception as e:
print(f"❌ Failed to decode base64 voice: {e}")
# Priority 3: Registry / Local File
if not speaker_wav:
voice_id = job.get("voice", "auto")
if voice_id and voice_id != "auto":
# Look in worker/voices/
base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
voice_path = os.path.join(base_dir, "voices", f"{voice_id}.wav")
# Check for other extensions if wav missing
if not os.path.exists(voice_path):
for ext in [".mp3", ".ogg", ".m4a"]:
p = os.path.join(base_dir, "voices", f"{voice_id}{ext}")
if os.path.exists(p):
voice_path = p
break
if os.path.exists(voice_path):
speaker_wav = voice_path
print(f"🗣️ Using registered voice: {voice_id}")
else:
print(f"⚠️ Voice '{voice_id}' not found in registry.")
# Priority 4: Default/Auto Voice
if not speaker_wav:
# Fallback to a default file if it exists
base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
default_path = os.path.join(base_dir, "voices", "default.wav")
if os.path.exists(default_path):
speaker_wav = default_path
print("⚠️ Using default.wav")
else:
# If completely nothing, we can't synthesize with XTTS
# Unless we use speaker_idxs (only for multi-speaker models w/o cloning?)
# XTTS v2 IS zero-shot, needs reference.
raise ValueError("No speaker reference found (url, base64, registry, or default.wav)")
# Run Inference
print(f"🎤 Synthesizing: '{text[:30]}...' Lang: {language}")
# XTTS API returns List[float]
wav = model.tts(text=text, speaker_wav=speaker_wav, language=language)
return wav
finally:
# Cleanup
for f in temp_files:
try:
os.remove(f)
except:
pass

View File

@ -1,9 +1,14 @@
print("DEBUG: Starting worker...", flush=True)
import time, json, os
print("DEBUG: Imported stdlib", flush=True)
from core.queue_worker import fetch_job, store_result
print("DEBUG: Imported queue_worker", flush=True)
from engine.xtts2_loader import synthesize
print("DEBUG: Imported xtts2_loader", flush=True)
from engine.audio_export import convert_audio
print("DEBUG: Imported audio_export", flush=True)
print("Worker gestartet. Warte auf Jobs…")
print("Worker gestartet. Warte auf Jobs…", flush=True)
while True:
job = fetch_job()
@ -11,6 +16,13 @@ while True:
time.sleep(0.1)
continue
audio = synthesize(job)
out, mime = convert_audio(audio, job.get("format","wav"))
store_result(job["job_id"], out, mime)
try:
start = time.time()
print(f"🔄 Processing Job {job['job_id']}...", flush=True)
audio = synthesize(job)
out, mime = convert_audio(audio, job.get("format","wav"))
store_result(job["job_id"], out, mime)
print(f"✅ Job {job['job_id']} done in {time.time()-start:.2f}s")
except Exception as e:
print(f"❌ Error processing job {job.get('job_id')}: {e}")
# Optional: Store error state if protocol supports it

View File

@ -3,3 +3,8 @@ redis
torch
numpy
requests
transformers==4.42.4
TTS==0.22.0
scipy
langdetect
torchcodec

17
worker/voices/README.md Normal file
View File

@ -0,0 +1,17 @@
# Voice Registry
Place `.wav` files here to register them as permanent voices.
## Usage
If you place a file named `narrator.wav` in this directory:
1. Restart the worker (or mount this volume dynamically).
2. Send a request with `"voice": "narrator"`.
The system will use this file as the speaker reference for XTTS cloning.
## Formats
Supported formats: `.wav`, `.mp3`, `.ogg`, `.m4a`.
Recommended: Mono, 22050Hz or 24000Hz WAV (16-bit).

BIN
worker/voices/default.wav Normal file

Binary file not shown.