Fix XTTS loader compatibility and add default voice
This commit is contained in:
31
AGENTS.md
Normal file
31
AGENTS.md
Normal file
@ -0,0 +1,31 @@
|
||||
# Repository Guidelines
|
||||
|
||||
## Project Structure & Modules
|
||||
- Gateway (`gateway/`): FastAPI entrypoint in `main.py`, routes under `api/` (`health.py`, `openai_speech.py`, `voices.py`), shared helpers in `core/`, voice registry in `voices/` and runtime port in `port.txt`.
|
||||
- Worker (`worker/`): Queue consumer in `core/queue_worker.py`, XTTS2 synthesis in `engine/xtts2_loader.py`, audio export in `engine/audio_export.py`, voice assets in `voices/`.
|
||||
- Tooling: `Makefile` drives Docker workflow, `docker-compose.yml` wires gateway/worker/redis, helper scripts in `scripts/` (`find_port.py`, `selftest.py`).
|
||||
|
||||
## Build, Test, and Run
|
||||
- `make build` – build gateway + worker images.
|
||||
- `make up` – start stack with auto port selection (writes `gateway/port.txt`).
|
||||
- `make status` / `make logs` – check containers and follow logs.
|
||||
- `make selftest` – end-to-end smoke test against the running stack (health + sample synthesis).
|
||||
- Local debug (no Docker): install deps with `pip install -r gateway/requirements.gateway.txt` and `pip install -r worker/requirements.worker.txt`, then `python gateway/main.py` and `python worker/main.py`; start Redis via `docker run -p 6379:6379 redis:7`.
|
||||
|
||||
## Coding Style & Naming
|
||||
- Python, prefer PEP8 with 4-space indents and snake_case names for modules, functions, and vars; keep route names aligned with OpenAI-compatible paths (`/v1/audio/speech`, `/v1/voices/register`).
|
||||
- Keep modules small and focused (API logic in `gateway/api`, queue/Redis helpers in `core`).
|
||||
- Favor explicit config via env vars (`REDIS_HOST`, `GATEWAY_PORT`); avoid hardcoded ports besides the 8000–8100 scan range.
|
||||
|
||||
## Testing Guidelines
|
||||
- Primary check is the smoke test: run `make selftest` after changes that touch API, queue, or audio paths.
|
||||
- For new logic, add lightweight unit tests (e.g., under `gateway/tests/` or `worker/tests/`) named `test_<feature>.py`; prefer pytest-style asserts.
|
||||
- When adding audio or queue code, include sanity checks (e.g., validate `mime` and byte length) to avoid silent failures.
|
||||
|
||||
## Commit & Pull Request Practices
|
||||
- Commits: short, imperative subjects (e.g., `add queue timeout guard`, `tune xtts export`). Group related changes; avoid mixing refactors with feature work.
|
||||
- Pull Requests: describe intent, list test commands executed (e.g., `make selftest`), mention affected endpoints or worker behaviors, and link issues when available. Provide screenshots or audio sample paths only if UX or output format changes.
|
||||
|
||||
## Security & Operations Notes
|
||||
- Do not commit voice assets beyond small samples; keep secrets out of the repo and prefer env vars or Docker secrets.
|
||||
- Gateway listens on the selected local port only; expose externally via reverse proxy/HTTPS in production. Keep worker services internal and behind the queue.
|
||||
133
GEMINI.md
Normal file
133
GEMINI.md
Normal file
@ -0,0 +1,133 @@
|
||||
# XTTS2 OpenAI-Compatible TTS Server
|
||||
|
||||
## Project Overview
|
||||
|
||||
This project is a high-performance, modular, and scalable Text-to-Speech (TTS) platform. It provides an API fully compatible with the **OpenAI Speech API**, powered by the **XTTS2** model for high-quality synthesis and zero-shot voice cloning.
|
||||
|
||||
### Architecture
|
||||
|
||||
The system follows a distributed architecture:
|
||||
|
||||
* **Gateway (`gateway/`):** A FastAPI service that handles HTTP requests, validates input, and manages the voice registry. It pushes synthesis jobs to a Redis queue. It features dynamic port selection (8000-8100).
|
||||
* **Redis:** Acts as the message broker (Queue) and cache between the Gateway and Workers.
|
||||
* **Worker (`worker/`):** A background service that pulls jobs from Redis, performs the actual TTS inference using XTTS2 (with GPU acceleration if available), and returns the audio data. These can be scaled horizontally.
|
||||
|
||||
### Key Technologies
|
||||
|
||||
* **Language:** Python 3.10+
|
||||
* **Framework:** FastAPI (Gateway)
|
||||
* **ML Model:** Coqui XTTS v2
|
||||
* **Infrastructure:** Docker, Docker Compose, Redis
|
||||
* **Tooling:** Makefile for orchestration
|
||||
|
||||
## Building and Running
|
||||
|
||||
The project relies heavily on `make` for orchestration.
|
||||
|
||||
### Docker (Recommended)
|
||||
|
||||
1. **Build Images:**
|
||||
```bash
|
||||
make build
|
||||
```
|
||||
|
||||
2. **Start Services:**
|
||||
```bash
|
||||
make up
|
||||
```
|
||||
* This runs a port scanner to find a free port between 8000-8100.
|
||||
* The chosen port is saved to `gateway/port.txt`.
|
||||
|
||||
3. **Check Status:**
|
||||
```bash
|
||||
make status
|
||||
```
|
||||
|
||||
4. **View Logs:**
|
||||
```bash
|
||||
make logs
|
||||
```
|
||||
|
||||
5. **Stop Services:**
|
||||
```bash
|
||||
make down
|
||||
```
|
||||
|
||||
### Scaling Workers
|
||||
|
||||
To handle higher load, you can spawn multiple worker containers:
|
||||
|
||||
```bash
|
||||
make worker-scale N=3
|
||||
```
|
||||
|
||||
### Verification
|
||||
|
||||
Run the self-test suite to verify Redis connectivity, worker processing, and audio synthesis:
|
||||
|
||||
```bash
|
||||
make selftest
|
||||
```
|
||||
|
||||
## Development Conventions
|
||||
|
||||
### Project Structure
|
||||
|
||||
* `gateway/`: Code for the API server.
|
||||
* `main.py`: Entry point.
|
||||
* `api/`: Endpoint definitions (`openai_speech.py`, `voices.py`).
|
||||
* `core/`: Configuration and utilities.
|
||||
* `worker/`: Code for the inference engine.
|
||||
* `engine/`: XTTS2 model loading and audio export logic.
|
||||
* `core/`: Queue processing and GPU detection.
|
||||
* `scripts/`: Utility scripts (e.g., `find_port.py`, `selftest.py`).
|
||||
|
||||
### Local Development (Non-Docker)
|
||||
|
||||
1. Create a virtual environment:
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
```
|
||||
2. Install dependencies:
|
||||
```bash
|
||||
pip install -r gateway/requirements.gateway.txt
|
||||
pip install -r worker/requirements.worker.txt
|
||||
```
|
||||
3. Run Redis locally (e.g., `docker run -p 6379:6379 redis:7`).
|
||||
4. Start Gateway: `python gateway/main.py`
|
||||
5. Start Worker: `python worker/main.py`
|
||||
|
||||
### API Usage
|
||||
|
||||
The API mirrors OpenAI's structure.
|
||||
|
||||
**Generate Audio:**
|
||||
```http
|
||||
POST /v1/audio/speech
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"model": "xtts-v2",
|
||||
"input": "Hello world",
|
||||
"voice": "auto",
|
||||
"format": "wav"
|
||||
}
|
||||
```
|
||||
|
||||
**Register Voice:**
|
||||
```http
|
||||
POST /v1/voices/register
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"name": "my-voice",
|
||||
"samples": ["https://example.com/sample.wav"]
|
||||
}
|
||||
```
|
||||
|
||||
### Logging & Debugging
|
||||
|
||||
* **Gateway Logs:** `gateway/logs/gateway.log`
|
||||
* **Port Info:** `gateway/port.txt` contains the active port.
|
||||
* **GPU:** Workers will automatically detect and use CUDA if available. Check `nvidia-smi` to monitor usage.
|
||||
94
Makefile
94
Makefile
@ -10,120 +10,100 @@ DOCKER := docker compose
|
||||
.DEFAULT_GOAL := help
|
||||
|
||||
help:
|
||||
@echo ""
|
||||
@echo "🚀 XTTS2 TTS Server – Makefile (Pro Mode)"
|
||||
@echo "-------------------------------------------"
|
||||
@echo " make build → Images bauen (Gateway + Worker)"
|
||||
@echo " make up → Services starten (mit Portscan)"
|
||||
@echo " make down → Services stoppen"
|
||||
@echo " make restart → Neustart"
|
||||
@echo " make logs → Logs aller Services anzeigen"
|
||||
@echo " make status → Docker Status"
|
||||
@echo " make worker-scale N=3 → Worker skalieren"
|
||||
@echo " make prune → Docker aufräumen"
|
||||
@echo " make selftest → System-Selbsttest"
|
||||
@echo " make port → Zeigt aktuellen Gateway Port"
|
||||
@echo "-------------------------------------------"
|
||||
@echo ""
|
||||
@echo "🚀 XTTS2 TTS Server – Makefile (Pro Mode)"
|
||||
@echo "-------------------------------------------"
|
||||
@echo " make build → Images bauen (Gateway + Worker)"
|
||||
@echo " make up → Services starten (mit Portscan)"
|
||||
@echo " make down → Services stoppen"
|
||||
@echo " make restart → Neustart"
|
||||
@echo " make logs → Logs aller Services anzeigen"
|
||||
@echo " make status → Docker Status"
|
||||
@echo " make worker-scale N=3 → Worker skalieren"
|
||||
@echo " make prune → Docker aufräumen"
|
||||
@echo " make selftest → System-Selbsttest"
|
||||
@echo " make port → Zeigt aktuellen Gateway Port"
|
||||
@echo "-------------------------------------------"
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
# Build
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
build:
|
||||
$(DOCKER) build
|
||||
$(DOCKER) build
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
# Deploy
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
up:
|
||||
@echo "🔍 Suche freien Port zwischen 8000–8100..."
|
||||
@PORT=`$(PYTHON) $(PORT_SCRIPT)`;
|
||||
if [ "$$PORT" = "ERR_NO_FREE_PORT" ]; then
|
||||
echo "❌ Kein freier Port gefunden!"; exit 1;
|
||||
fi;
|
||||
echo "🎧 Freier Port gefunden: $$PORT";
|
||||
echo "$$PORT" > $(PORT_FILE);
|
||||
echo "📄 Port gespeichert in $(PORT_FILE)";
|
||||
export GATEWAY_PORT=$$PORT;
|
||||
$(DOCKER) up -d --build;
|
||||
echo "🚀 Gateway läuft auf [http://localhost:$$PORT](http://localhost:$$PORT)"
|
||||
@echo "🔍 Suche freien Port zwischen 8000–8100..."
|
||||
@PORT=`$(PYTHON) $(PORT_SCRIPT)`; \
|
||||
if [ "$$PORT" = "ERR_NO_FREE_PORT" ]; then \
|
||||
echo "❌ Kein freier Port gefunden!"; exit 1; \
|
||||
fi; \
|
||||
echo "🎧 Freier Port gefunden: $$PORT"; \
|
||||
echo "$$PORT" > $(PORT_FILE); \
|
||||
echo "📄 Port gespeichert in $(PORT_FILE)"; \
|
||||
export GATEWAY_PORT=$$PORT; \
|
||||
$(DOCKER) up -d --build; \
|
||||
echo "🚀 Gateway läuft auf http://localhost:$$PORT"
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
# Stop
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
down:
|
||||
$(DOCKER) down
|
||||
$(DOCKER) down
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
# Restart
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
restart: down up
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
# Logs
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
logs:
|
||||
$(DOCKER) logs -f
|
||||
$(DOCKER) logs -f
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
# Status
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
status:
|
||||
$(DOCKER) ps
|
||||
$(DOCKER) ps
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
# Worker Scaling
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
worker-scale:
|
||||
@if [ -z "$(N)" ]; then echo "Bitte N angeben: make worker-scale N=3"; exit 1; fi
|
||||
$(DOCKER) up -d --scale worker=$(N)
|
||||
@if [ -z "$(N)" ]; then echo "Bitte N angeben: make worker-scale N=3"; exit 1; fi
|
||||
$(DOCKER) up -d --scale worker=$(N)
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
# Cleanup
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
prune:
|
||||
$(DOCKER) down
|
||||
docker system prune -f
|
||||
$(DOCKER) down
|
||||
docker system prune -f
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
# Selftest
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
selftest:
|
||||
@echo "🧪 Starte Selbsttest..."
|
||||
$(PYTHON) scripts/selftest.py
|
||||
@echo "🧪 Starte Selbsttest..."
|
||||
$(PYTHON) scripts/selftest.py
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
# Show Port
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
||||
port:
|
||||
@echo "📡 Aktueller Port:"
|
||||
@cat $(PORT_FILE)
|
||||
@echo "📡 Aktueller Port:"
|
||||
@cat $(PORT_FILE)
|
||||
@ -1,54 +1,55 @@
|
||||
# docker-compose.yml – XTTS2 TTS Server
|
||||
|
||||
# Multi-Service Orchestrierung (Gateway, Worker, Redis)
|
||||
|
||||
version: "3.9"
|
||||
|
||||
services:
|
||||
redis:
|
||||
image: redis:7
|
||||
container_name: redis
|
||||
restart: always
|
||||
ports:
|
||||
- "6379:6379"
|
||||
redis:
|
||||
image: redis:7
|
||||
container_name: redis
|
||||
restart: always
|
||||
ports:
|
||||
- "6379:6379"
|
||||
|
||||
gateway:
|
||||
build:
|
||||
context: ./gateway
|
||||
dockerfile: Dockerfile
|
||||
container_name: tts-gateway
|
||||
restart: always
|
||||
environment:
|
||||
- REDIS_HOST=redis
|
||||
- GATEWAY_PORT=${GATEWAY_PORT}
|
||||
volumes:
|
||||
- ./gateway/voices:/app/voices
|
||||
- ./gateway/logs:/app/logs
|
||||
- ./gateway/port.txt:/app/port.txt
|
||||
ports:
|
||||
- "${GATEWAY_PORT}:8000"
|
||||
depends_on:
|
||||
- redis
|
||||
gateway:
|
||||
build:
|
||||
context: ./gateway
|
||||
dockerfile: Dockerfile
|
||||
container_name: tts-gateway
|
||||
restart: always
|
||||
environment:
|
||||
- REDIS_HOST=redis
|
||||
- GATEWAY_PORT=${GATEWAY_PORT}
|
||||
volumes:
|
||||
- ./gateway/voices:/app/voices
|
||||
- ./gateway/logs:/app/logs
|
||||
- ./gateway/port.txt:/app/port.txt
|
||||
ports:
|
||||
- "${GATEWAY_PORT:-8000}:${GATEWAY_PORT:-8000}"
|
||||
depends_on:
|
||||
- redis
|
||||
|
||||
worker:
|
||||
build:
|
||||
context: ./worker
|
||||
dockerfile: Dockerfile
|
||||
container_name: tts-worker
|
||||
restart: always
|
||||
environment:
|
||||
- REDIS_HOST=redis
|
||||
- GPU_MODE=AUTO
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- capabilities: [gpu]
|
||||
volumes:
|
||||
- ./worker/voices:/app/voices
|
||||
- tts-models:/root/.local/share/tts
|
||||
depends_on:
|
||||
- redis
|
||||
|
||||
worker:
|
||||
build:
|
||||
context: ./worker
|
||||
dockerfile: Dockerfile
|
||||
container_name: tts-worker
|
||||
restart: always
|
||||
environment:
|
||||
- REDIS_HOST=redis
|
||||
- GPU_MODE=AUTO
|
||||
deploy:
|
||||
resources:
|
||||
reservations:
|
||||
devices:
|
||||
- capabilities: [gpu]
|
||||
volumes:
|
||||
- ./worker/voices:/app/voices
|
||||
depends_on:
|
||||
- redis
|
||||
tts-models:
|
||||
|
||||
networks:
|
||||
default:
|
||||
name: wlkns-net
|
||||
default:
|
||||
name: wlkns-net
|
||||
@ -8,6 +8,7 @@ class SpeechRequest(BaseModel):
|
||||
model: str
|
||||
input: str
|
||||
voice: str = "auto"
|
||||
language: str | None = None
|
||||
format: str = "wav"
|
||||
voice_sample_url: str | None = None
|
||||
voice_sample_base64: str | None = None
|
||||
|
||||
@ -12,7 +12,7 @@ def push_job(data: dict) -> str:
|
||||
r.lpush(QUEUE, json.dumps(data))
|
||||
return job_id
|
||||
|
||||
def await_result(job_id: str, timeout=30):
|
||||
def await_result(job_id: str, timeout=120):
|
||||
key = f"{RESULT}:{job_id}"
|
||||
start=time.time()
|
||||
while time.time()-start < timeout:
|
||||
|
||||
1
gateway/port.txt
Normal file
1
gateway/port.txt
Normal file
@ -0,0 +1 @@
|
||||
8003
|
||||
@ -1,7 +1,8 @@
|
||||
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 AS base
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
WORKDIR /app
|
||||
COPY requirements.worker.txt .
|
||||
RUN apt-get update && apt-get install -y python3-pip ffmpeg
|
||||
RUN apt-get update && apt-get install -y python3-pip ffmpeg espeak-ng
|
||||
RUN pip3 install -r requirements.worker.txt
|
||||
COPY . .
|
||||
CMD ["python3","main.py"]
|
||||
|
||||
@ -1,19 +1,39 @@
|
||||
from pydub import AudioSegment
|
||||
import io
|
||||
import numpy as np
|
||||
|
||||
def convert_audio(raw_bytes: bytes, fmt: str):
|
||||
# raw mono 32-bit float fake waveform
|
||||
def convert_audio(audio_data, fmt: str, sample_rate: int = 24000):
|
||||
"""
|
||||
Converts raw audio data (numpy array or list of floats) to the target format.
|
||||
Assumes mono audio.
|
||||
"""
|
||||
# Ensure numpy array
|
||||
if not isinstance(audio_data, np.ndarray):
|
||||
audio_data = np.array(audio_data)
|
||||
|
||||
# Check if float and normalize/convert to int16
|
||||
if audio_data.dtype.kind == 'f':
|
||||
# Clip to Avoid wrap-around
|
||||
audio_data = np.clip(audio_data, -1.0, 1.0)
|
||||
# Convert to 16-bit PCM
|
||||
audio_data = (audio_data * 32767).astype(np.int16)
|
||||
|
||||
seg = AudioSegment(
|
||||
raw_bytes,
|
||||
frame_rate=22050,
|
||||
sample_width=4,
|
||||
audio_data.tobytes(),
|
||||
frame_rate=sample_rate,
|
||||
sample_width=2, # 16-bit
|
||||
channels=1
|
||||
)
|
||||
buf=io.BytesIO()
|
||||
|
||||
buf = io.BytesIO()
|
||||
seg.export(buf, format=fmt)
|
||||
mime={
|
||||
"wav":"audio/wav",
|
||||
"mp3":"audio/mpeg",
|
||||
"ogg":"audio/ogg"
|
||||
}.get(fmt,"audio/wav")
|
||||
return buf.getvalue(), mime
|
||||
|
||||
mime = {
|
||||
"wav": "audio/wav",
|
||||
"mp3": "audio/mpeg",
|
||||
"ogg": "audio/ogg",
|
||||
"flac": "audio/flac",
|
||||
"aac": "audio/aac"
|
||||
}.get(fmt, "audio/wav")
|
||||
|
||||
return buf.getvalue(), mime
|
||||
@ -1,10 +1,163 @@
|
||||
# Dummy XTTS2 logic placeholder
|
||||
# Replace with real TTS model loading
|
||||
import os
|
||||
# Auto-agree to Coqui TOS (Must be before imports)
|
||||
os.environ["COQUI_TOS_AGREED"] = "1"
|
||||
|
||||
import json
|
||||
import base64
|
||||
import tempfile
|
||||
import requests
|
||||
import torch
|
||||
import numpy as np
|
||||
|
||||
# Singleton for lazy loading
|
||||
_model = None
|
||||
|
||||
def _allow_xtts_config_pickle():
|
||||
"""Allow loading XTTS configs with torch >=2.6 safe loading."""
|
||||
add_safe = getattr(torch.serialization, "add_safe_globals", None)
|
||||
if not add_safe:
|
||||
return
|
||||
allowed = []
|
||||
try:
|
||||
from TTS.tts.configs.xtts_config import XttsConfig
|
||||
from TTS.tts.models.xtts import XttsAudioConfig
|
||||
allowed += [XttsConfig, XttsAudioConfig]
|
||||
except Exception as e:
|
||||
print(f"⚠️ Could not register safe globals for XTTS config: {e}")
|
||||
try:
|
||||
import TTS.config.shared_configs as shared_configs
|
||||
allowed += [v for v in shared_configs.__dict__.values() if isinstance(v, type)]
|
||||
except Exception as e:
|
||||
print(f"⚠️ Could not register shared config globals: {e}")
|
||||
try:
|
||||
import TTS.tts.models.xtts as xtts_models
|
||||
allowed += [v for v in xtts_models.__dict__.values() if isinstance(v, type)]
|
||||
except Exception as e:
|
||||
print(f"⚠️ Could not register XTTS model globals: {e}")
|
||||
if allowed:
|
||||
add_safe(allowed)
|
||||
|
||||
def get_model():
|
||||
global _model
|
||||
if _model is None:
|
||||
print("⏳ Loading XTTS Model (Lazy Load)....")
|
||||
# Lazy Import to prevent startup hang
|
||||
from TTS.api import TTS
|
||||
_allow_xtts_config_pickle()
|
||||
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
print(f"🔧 XTTS Running on: {device}")
|
||||
|
||||
# Load Model (download if needed)
|
||||
# Using default XTTS v2 model
|
||||
_model = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to(device)
|
||||
print("✅ XTTS Model loaded successfully.")
|
||||
return _model
|
||||
|
||||
def synthesize(job: dict):
|
||||
# return artificial sine wave placeholder
|
||||
import numpy as np
|
||||
sr=22050
|
||||
t=np.linspace(0,0.3,int(sr*0.3))
|
||||
tone=(0.1*np.sin(2*np.pi*440*t)).astype('float32')
|
||||
return tone.tobytes()
|
||||
from langdetect import detect
|
||||
model = get_model()
|
||||
...
|
||||
|
||||
text = job.get("input")
|
||||
if not text:
|
||||
raise ValueError("No input text provided")
|
||||
|
||||
# Language handling
|
||||
language = job.get("language")
|
||||
if not language:
|
||||
try:
|
||||
# Simple detection
|
||||
detected = detect(text)
|
||||
# XTTS expects 2-letter codes usually.
|
||||
# We assume detected is valid or mapped if needed.
|
||||
# Supported: en, es, fr, de, it, pt, pl, tr, ru, nl, cs, ar, zh-cn, ja, hu, ko
|
||||
language = detected
|
||||
print(f"🌍 Auto-detected language: {language}")
|
||||
except:
|
||||
language = "en"
|
||||
print("⚠️ Language detection failed, using 'en'")
|
||||
|
||||
# Speaker Handling
|
||||
speaker_wav = None
|
||||
temp_files = []
|
||||
|
||||
try:
|
||||
# Priority 1: Direct URL
|
||||
if job.get("voice_sample_url"):
|
||||
try:
|
||||
print(f"⬇️ Downloading voice sample from {job['voice_sample_url']}")
|
||||
r = requests.get(job["voice_sample_url"], timeout=10)
|
||||
r.raise_for_status()
|
||||
t = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
|
||||
t.write(r.content)
|
||||
t.close()
|
||||
speaker_wav = t.name
|
||||
temp_files.append(t.name)
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to download voice sample: {e}")
|
||||
|
||||
# Priority 2: Base64
|
||||
if not speaker_wav and job.get("voice_sample_base64"):
|
||||
try:
|
||||
b64 = job["voice_sample_base64"]
|
||||
decoded = base64.b64decode(b64)
|
||||
t = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
|
||||
t.write(decoded)
|
||||
t.close()
|
||||
speaker_wav = t.name
|
||||
temp_files.append(t.name)
|
||||
except Exception as e:
|
||||
print(f"❌ Failed to decode base64 voice: {e}")
|
||||
|
||||
# Priority 3: Registry / Local File
|
||||
if not speaker_wav:
|
||||
voice_id = job.get("voice", "auto")
|
||||
if voice_id and voice_id != "auto":
|
||||
# Look in worker/voices/
|
||||
base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
voice_path = os.path.join(base_dir, "voices", f"{voice_id}.wav")
|
||||
|
||||
# Check for other extensions if wav missing
|
||||
if not os.path.exists(voice_path):
|
||||
for ext in [".mp3", ".ogg", ".m4a"]:
|
||||
p = os.path.join(base_dir, "voices", f"{voice_id}{ext}")
|
||||
if os.path.exists(p):
|
||||
voice_path = p
|
||||
break
|
||||
|
||||
if os.path.exists(voice_path):
|
||||
speaker_wav = voice_path
|
||||
print(f"🗣️ Using registered voice: {voice_id}")
|
||||
else:
|
||||
print(f"⚠️ Voice '{voice_id}' not found in registry.")
|
||||
|
||||
# Priority 4: Default/Auto Voice
|
||||
if not speaker_wav:
|
||||
# Fallback to a default file if it exists
|
||||
base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
default_path = os.path.join(base_dir, "voices", "default.wav")
|
||||
if os.path.exists(default_path):
|
||||
speaker_wav = default_path
|
||||
print("⚠️ Using default.wav")
|
||||
else:
|
||||
# If completely nothing, we can't synthesize with XTTS
|
||||
# Unless we use speaker_idxs (only for multi-speaker models w/o cloning?)
|
||||
# XTTS v2 IS zero-shot, needs reference.
|
||||
raise ValueError("No speaker reference found (url, base64, registry, or default.wav)")
|
||||
|
||||
# Run Inference
|
||||
print(f"🎤 Synthesizing: '{text[:30]}...' Lang: {language}")
|
||||
|
||||
# XTTS API returns List[float]
|
||||
wav = model.tts(text=text, speaker_wav=speaker_wav, language=language)
|
||||
|
||||
return wav
|
||||
|
||||
finally:
|
||||
# Cleanup
|
||||
for f in temp_files:
|
||||
try:
|
||||
os.remove(f)
|
||||
except:
|
||||
pass
|
||||
|
||||
@ -1,9 +1,14 @@
|
||||
print("DEBUG: Starting worker...", flush=True)
|
||||
import time, json, os
|
||||
print("DEBUG: Imported stdlib", flush=True)
|
||||
from core.queue_worker import fetch_job, store_result
|
||||
print("DEBUG: Imported queue_worker", flush=True)
|
||||
from engine.xtts2_loader import synthesize
|
||||
print("DEBUG: Imported xtts2_loader", flush=True)
|
||||
from engine.audio_export import convert_audio
|
||||
print("DEBUG: Imported audio_export", flush=True)
|
||||
|
||||
print("Worker gestartet. Warte auf Jobs…")
|
||||
print("Worker gestartet. Warte auf Jobs…", flush=True)
|
||||
|
||||
while True:
|
||||
job = fetch_job()
|
||||
@ -11,6 +16,13 @@ while True:
|
||||
time.sleep(0.1)
|
||||
continue
|
||||
|
||||
audio = synthesize(job)
|
||||
out, mime = convert_audio(audio, job.get("format","wav"))
|
||||
store_result(job["job_id"], out, mime)
|
||||
try:
|
||||
start = time.time()
|
||||
print(f"🔄 Processing Job {job['job_id']}...", flush=True)
|
||||
audio = synthesize(job)
|
||||
out, mime = convert_audio(audio, job.get("format","wav"))
|
||||
store_result(job["job_id"], out, mime)
|
||||
print(f"✅ Job {job['job_id']} done in {time.time()-start:.2f}s")
|
||||
except Exception as e:
|
||||
print(f"❌ Error processing job {job.get('job_id')}: {e}")
|
||||
# Optional: Store error state if protocol supports it
|
||||
|
||||
@ -3,3 +3,8 @@ redis
|
||||
torch
|
||||
numpy
|
||||
requests
|
||||
transformers==4.42.4
|
||||
TTS==0.22.0
|
||||
scipy
|
||||
langdetect
|
||||
torchcodec
|
||||
|
||||
17
worker/voices/README.md
Normal file
17
worker/voices/README.md
Normal file
@ -0,0 +1,17 @@
|
||||
# Voice Registry
|
||||
|
||||
Place `.wav` files here to register them as permanent voices.
|
||||
|
||||
## Usage
|
||||
|
||||
If you place a file named `narrator.wav` in this directory:
|
||||
|
||||
1. Restart the worker (or mount this volume dynamically).
|
||||
2. Send a request with `"voice": "narrator"`.
|
||||
|
||||
The system will use this file as the speaker reference for XTTS cloning.
|
||||
|
||||
## Formats
|
||||
|
||||
Supported formats: `.wav`, `.mp3`, `.ogg`, `.m4a`.
|
||||
Recommended: Mono, 22050Hz or 24000Hz WAV (16-bit).
|
||||
BIN
worker/voices/default.wav
Normal file
BIN
worker/voices/default.wav
Normal file
Binary file not shown.
Reference in New Issue
Block a user