Fix XTTS loader compatibility and add default voice
This commit is contained in:
31
AGENTS.md
Normal file
31
AGENTS.md
Normal file
@ -0,0 +1,31 @@
|
|||||||
|
# Repository Guidelines
|
||||||
|
|
||||||
|
## Project Structure & Modules
|
||||||
|
- Gateway (`gateway/`): FastAPI entrypoint in `main.py`, routes under `api/` (`health.py`, `openai_speech.py`, `voices.py`), shared helpers in `core/`, voice registry in `voices/` and runtime port in `port.txt`.
|
||||||
|
- Worker (`worker/`): Queue consumer in `core/queue_worker.py`, XTTS2 synthesis in `engine/xtts2_loader.py`, audio export in `engine/audio_export.py`, voice assets in `voices/`.
|
||||||
|
- Tooling: `Makefile` drives Docker workflow, `docker-compose.yml` wires gateway/worker/redis, helper scripts in `scripts/` (`find_port.py`, `selftest.py`).
|
||||||
|
|
||||||
|
## Build, Test, and Run
|
||||||
|
- `make build` – build gateway + worker images.
|
||||||
|
- `make up` – start stack with auto port selection (writes `gateway/port.txt`).
|
||||||
|
- `make status` / `make logs` – check containers and follow logs.
|
||||||
|
- `make selftest` – end-to-end smoke test against the running stack (health + sample synthesis).
|
||||||
|
- Local debug (no Docker): install deps with `pip install -r gateway/requirements.gateway.txt` and `pip install -r worker/requirements.worker.txt`, then `python gateway/main.py` and `python worker/main.py`; start Redis via `docker run -p 6379:6379 redis:7`.
|
||||||
|
|
||||||
|
## Coding Style & Naming
|
||||||
|
- Python, prefer PEP8 with 4-space indents and snake_case names for modules, functions, and vars; keep route names aligned with OpenAI-compatible paths (`/v1/audio/speech`, `/v1/voices/register`).
|
||||||
|
- Keep modules small and focused (API logic in `gateway/api`, queue/Redis helpers in `core`).
|
||||||
|
- Favor explicit config via env vars (`REDIS_HOST`, `GATEWAY_PORT`); avoid hardcoded ports besides the 8000–8100 scan range.
|
||||||
|
|
||||||
|
## Testing Guidelines
|
||||||
|
- Primary check is the smoke test: run `make selftest` after changes that touch API, queue, or audio paths.
|
||||||
|
- For new logic, add lightweight unit tests (e.g., under `gateway/tests/` or `worker/tests/`) named `test_<feature>.py`; prefer pytest-style asserts.
|
||||||
|
- When adding audio or queue code, include sanity checks (e.g., validate `mime` and byte length) to avoid silent failures.
|
||||||
|
|
||||||
|
## Commit & Pull Request Practices
|
||||||
|
- Commits: short, imperative subjects (e.g., `add queue timeout guard`, `tune xtts export`). Group related changes; avoid mixing refactors with feature work.
|
||||||
|
- Pull Requests: describe intent, list test commands executed (e.g., `make selftest`), mention affected endpoints or worker behaviors, and link issues when available. Provide screenshots or audio sample paths only if UX or output format changes.
|
||||||
|
|
||||||
|
## Security & Operations Notes
|
||||||
|
- Do not commit voice assets beyond small samples; keep secrets out of the repo and prefer env vars or Docker secrets.
|
||||||
|
- Gateway listens on the selected local port only; expose externally via reverse proxy/HTTPS in production. Keep worker services internal and behind the queue.
|
||||||
133
GEMINI.md
Normal file
133
GEMINI.md
Normal file
@ -0,0 +1,133 @@
|
|||||||
|
# XTTS2 OpenAI-Compatible TTS Server
|
||||||
|
|
||||||
|
## Project Overview
|
||||||
|
|
||||||
|
This project is a high-performance, modular, and scalable Text-to-Speech (TTS) platform. It provides an API fully compatible with the **OpenAI Speech API**, powered by the **XTTS2** model for high-quality synthesis and zero-shot voice cloning.
|
||||||
|
|
||||||
|
### Architecture
|
||||||
|
|
||||||
|
The system follows a distributed architecture:
|
||||||
|
|
||||||
|
* **Gateway (`gateway/`):** A FastAPI service that handles HTTP requests, validates input, and manages the voice registry. It pushes synthesis jobs to a Redis queue. It features dynamic port selection (8000-8100).
|
||||||
|
* **Redis:** Acts as the message broker (Queue) and cache between the Gateway and Workers.
|
||||||
|
* **Worker (`worker/`):** A background service that pulls jobs from Redis, performs the actual TTS inference using XTTS2 (with GPU acceleration if available), and returns the audio data. These can be scaled horizontally.
|
||||||
|
|
||||||
|
### Key Technologies
|
||||||
|
|
||||||
|
* **Language:** Python 3.10+
|
||||||
|
* **Framework:** FastAPI (Gateway)
|
||||||
|
* **ML Model:** Coqui XTTS v2
|
||||||
|
* **Infrastructure:** Docker, Docker Compose, Redis
|
||||||
|
* **Tooling:** Makefile for orchestration
|
||||||
|
|
||||||
|
## Building and Running
|
||||||
|
|
||||||
|
The project relies heavily on `make` for orchestration.
|
||||||
|
|
||||||
|
### Docker (Recommended)
|
||||||
|
|
||||||
|
1. **Build Images:**
|
||||||
|
```bash
|
||||||
|
make build
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Start Services:**
|
||||||
|
```bash
|
||||||
|
make up
|
||||||
|
```
|
||||||
|
* This runs a port scanner to find a free port between 8000-8100.
|
||||||
|
* The chosen port is saved to `gateway/port.txt`.
|
||||||
|
|
||||||
|
3. **Check Status:**
|
||||||
|
```bash
|
||||||
|
make status
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **View Logs:**
|
||||||
|
```bash
|
||||||
|
make logs
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **Stop Services:**
|
||||||
|
```bash
|
||||||
|
make down
|
||||||
|
```
|
||||||
|
|
||||||
|
### Scaling Workers
|
||||||
|
|
||||||
|
To handle higher load, you can spawn multiple worker containers:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make worker-scale N=3
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verification
|
||||||
|
|
||||||
|
Run the self-test suite to verify Redis connectivity, worker processing, and audio synthesis:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make selftest
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development Conventions
|
||||||
|
|
||||||
|
### Project Structure
|
||||||
|
|
||||||
|
* `gateway/`: Code for the API server.
|
||||||
|
* `main.py`: Entry point.
|
||||||
|
* `api/`: Endpoint definitions (`openai_speech.py`, `voices.py`).
|
||||||
|
* `core/`: Configuration and utilities.
|
||||||
|
* `worker/`: Code for the inference engine.
|
||||||
|
* `engine/`: XTTS2 model loading and audio export logic.
|
||||||
|
* `core/`: Queue processing and GPU detection.
|
||||||
|
* `scripts/`: Utility scripts (e.g., `find_port.py`, `selftest.py`).
|
||||||
|
|
||||||
|
### Local Development (Non-Docker)
|
||||||
|
|
||||||
|
1. Create a virtual environment:
|
||||||
|
```bash
|
||||||
|
python3 -m venv .venv
|
||||||
|
source .venv/bin/activate
|
||||||
|
```
|
||||||
|
2. Install dependencies:
|
||||||
|
```bash
|
||||||
|
pip install -r gateway/requirements.gateway.txt
|
||||||
|
pip install -r worker/requirements.worker.txt
|
||||||
|
```
|
||||||
|
3. Run Redis locally (e.g., `docker run -p 6379:6379 redis:7`).
|
||||||
|
4. Start Gateway: `python gateway/main.py`
|
||||||
|
5. Start Worker: `python worker/main.py`
|
||||||
|
|
||||||
|
### API Usage
|
||||||
|
|
||||||
|
The API mirrors OpenAI's structure.
|
||||||
|
|
||||||
|
**Generate Audio:**
|
||||||
|
```http
|
||||||
|
POST /v1/audio/speech
|
||||||
|
Content-Type: application/json
|
||||||
|
|
||||||
|
{
|
||||||
|
"model": "xtts-v2",
|
||||||
|
"input": "Hello world",
|
||||||
|
"voice": "auto",
|
||||||
|
"format": "wav"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Register Voice:**
|
||||||
|
```http
|
||||||
|
POST /v1/voices/register
|
||||||
|
Content-Type: application/json
|
||||||
|
|
||||||
|
{
|
||||||
|
"name": "my-voice",
|
||||||
|
"samples": ["https://example.com/sample.wav"]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Logging & Debugging
|
||||||
|
|
||||||
|
* **Gateway Logs:** `gateway/logs/gateway.log`
|
||||||
|
* **Port Info:** `gateway/port.txt` contains the active port.
|
||||||
|
* **GPU:** Workers will automatically detect and use CUDA if available. Check `nvidia-smi` to monitor usage.
|
||||||
94
Makefile
94
Makefile
@ -10,120 +10,100 @@ DOCKER := docker compose
|
|||||||
.DEFAULT_GOAL := help
|
.DEFAULT_GOAL := help
|
||||||
|
|
||||||
help:
|
help:
|
||||||
@echo ""
|
@echo ""
|
||||||
@echo "🚀 XTTS2 TTS Server – Makefile (Pro Mode)"
|
@echo "🚀 XTTS2 TTS Server – Makefile (Pro Mode)"
|
||||||
@echo "-------------------------------------------"
|
@echo "-------------------------------------------"
|
||||||
@echo " make build → Images bauen (Gateway + Worker)"
|
@echo " make build → Images bauen (Gateway + Worker)"
|
||||||
@echo " make up → Services starten (mit Portscan)"
|
@echo " make up → Services starten (mit Portscan)"
|
||||||
@echo " make down → Services stoppen"
|
@echo " make down → Services stoppen"
|
||||||
@echo " make restart → Neustart"
|
@echo " make restart → Neustart"
|
||||||
@echo " make logs → Logs aller Services anzeigen"
|
@echo " make logs → Logs aller Services anzeigen"
|
||||||
@echo " make status → Docker Status"
|
@echo " make status → Docker Status"
|
||||||
@echo " make worker-scale N=3 → Worker skalieren"
|
@echo " make worker-scale N=3 → Worker skalieren"
|
||||||
@echo " make prune → Docker aufräumen"
|
@echo " make prune → Docker aufräumen"
|
||||||
@echo " make selftest → System-Selbsttest"
|
@echo " make selftest → System-Selbsttest"
|
||||||
@echo " make port → Zeigt aktuellen Gateway Port"
|
@echo " make port → Zeigt aktuellen Gateway Port"
|
||||||
@echo "-------------------------------------------"
|
@echo "-------------------------------------------"
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
# Build
|
# Build
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
build:
|
build:
|
||||||
$(DOCKER) build
|
$(DOCKER) build
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
# Deploy
|
# Deploy
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
up:
|
up:
|
||||||
@echo "🔍 Suche freien Port zwischen 8000–8100..."
|
@echo "🔍 Suche freien Port zwischen 8000–8100..."
|
||||||
@PORT=`$(PYTHON) $(PORT_SCRIPT)`;
|
@PORT=`$(PYTHON) $(PORT_SCRIPT)`; \
|
||||||
if [ "$$PORT" = "ERR_NO_FREE_PORT" ]; then
|
if [ "$$PORT" = "ERR_NO_FREE_PORT" ]; then \
|
||||||
echo "❌ Kein freier Port gefunden!"; exit 1;
|
echo "❌ Kein freier Port gefunden!"; exit 1; \
|
||||||
fi;
|
fi; \
|
||||||
echo "🎧 Freier Port gefunden: $$PORT";
|
echo "🎧 Freier Port gefunden: $$PORT"; \
|
||||||
echo "$$PORT" > $(PORT_FILE);
|
echo "$$PORT" > $(PORT_FILE); \
|
||||||
echo "📄 Port gespeichert in $(PORT_FILE)";
|
echo "📄 Port gespeichert in $(PORT_FILE)"; \
|
||||||
export GATEWAY_PORT=$$PORT;
|
export GATEWAY_PORT=$$PORT; \
|
||||||
$(DOCKER) up -d --build;
|
$(DOCKER) up -d --build; \
|
||||||
echo "🚀 Gateway läuft auf [http://localhost:$$PORT](http://localhost:$$PORT)"
|
echo "🚀 Gateway läuft auf http://localhost:$$PORT"
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
# Stop
|
# Stop
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
down:
|
down:
|
||||||
$(DOCKER) down
|
$(DOCKER) down
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
# Restart
|
# Restart
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
restart: down up
|
restart: down up
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
# Logs
|
# Logs
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
logs:
|
logs:
|
||||||
$(DOCKER) logs -f
|
$(DOCKER) logs -f
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
# Status
|
# Status
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
status:
|
status:
|
||||||
$(DOCKER) ps
|
$(DOCKER) ps
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
# Worker Scaling
|
# Worker Scaling
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
worker-scale:
|
worker-scale:
|
||||||
@if [ -z "$(N)" ]; then echo "Bitte N angeben: make worker-scale N=3"; exit 1; fi
|
@if [ -z "$(N)" ]; then echo "Bitte N angeben: make worker-scale N=3"; exit 1; fi
|
||||||
$(DOCKER) up -d --scale worker=$(N)
|
$(DOCKER) up -d --scale worker=$(N)
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
# Cleanup
|
# Cleanup
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
prune:
|
prune:
|
||||||
$(DOCKER) down
|
$(DOCKER) down
|
||||||
docker system prune -f
|
docker system prune -f
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
# Selftest
|
# Selftest
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
selftest:
|
selftest:
|
||||||
@echo "🧪 Starte Selbsttest..."
|
@echo "🧪 Starte Selbsttest..."
|
||||||
$(PYTHON) scripts/selftest.py
|
$(PYTHON) scripts/selftest.py
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
# Show Port
|
# Show Port
|
||||||
|
|
||||||
# ---------------------------------------------------------
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
port:
|
port:
|
||||||
@echo "📡 Aktueller Port:"
|
@echo "📡 Aktueller Port:"
|
||||||
@cat $(PORT_FILE)
|
@cat $(PORT_FILE)
|
||||||
@ -1,54 +1,55 @@
|
|||||||
# docker-compose.yml – XTTS2 TTS Server
|
# docker-compose.yml – XTTS2 TTS Server
|
||||||
|
|
||||||
# Multi-Service Orchestrierung (Gateway, Worker, Redis)
|
|
||||||
|
|
||||||
version: "3.9"
|
version: "3.9"
|
||||||
|
|
||||||
services:
|
services:
|
||||||
redis:
|
redis:
|
||||||
image: redis:7
|
image: redis:7
|
||||||
container_name: redis
|
container_name: redis
|
||||||
restart: always
|
restart: always
|
||||||
ports:
|
ports:
|
||||||
- "6379:6379"
|
- "6379:6379"
|
||||||
|
|
||||||
gateway:
|
gateway:
|
||||||
build:
|
build:
|
||||||
context: ./gateway
|
context: ./gateway
|
||||||
dockerfile: Dockerfile
|
dockerfile: Dockerfile
|
||||||
container_name: tts-gateway
|
container_name: tts-gateway
|
||||||
restart: always
|
restart: always
|
||||||
environment:
|
environment:
|
||||||
- REDIS_HOST=redis
|
- REDIS_HOST=redis
|
||||||
- GATEWAY_PORT=${GATEWAY_PORT}
|
- GATEWAY_PORT=${GATEWAY_PORT}
|
||||||
volumes:
|
volumes:
|
||||||
- ./gateway/voices:/app/voices
|
- ./gateway/voices:/app/voices
|
||||||
- ./gateway/logs:/app/logs
|
- ./gateway/logs:/app/logs
|
||||||
- ./gateway/port.txt:/app/port.txt
|
- ./gateway/port.txt:/app/port.txt
|
||||||
ports:
|
ports:
|
||||||
- "${GATEWAY_PORT}:8000"
|
- "${GATEWAY_PORT:-8000}:${GATEWAY_PORT:-8000}"
|
||||||
depends_on:
|
depends_on:
|
||||||
- redis
|
- redis
|
||||||
|
|
||||||
|
worker:
|
||||||
|
build:
|
||||||
|
context: ./worker
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
container_name: tts-worker
|
||||||
|
restart: always
|
||||||
|
environment:
|
||||||
|
- REDIS_HOST=redis
|
||||||
|
- GPU_MODE=AUTO
|
||||||
|
deploy:
|
||||||
|
resources:
|
||||||
|
reservations:
|
||||||
|
devices:
|
||||||
|
- capabilities: [gpu]
|
||||||
|
volumes:
|
||||||
|
- ./worker/voices:/app/voices
|
||||||
|
- tts-models:/root/.local/share/tts
|
||||||
|
depends_on:
|
||||||
|
- redis
|
||||||
|
|
||||||
worker:
|
|
||||||
build:
|
|
||||||
context: ./worker
|
|
||||||
dockerfile: Dockerfile
|
|
||||||
container_name: tts-worker
|
|
||||||
restart: always
|
|
||||||
environment:
|
|
||||||
- REDIS_HOST=redis
|
|
||||||
- GPU_MODE=AUTO
|
|
||||||
deploy:
|
|
||||||
resources:
|
|
||||||
reservations:
|
|
||||||
devices:
|
|
||||||
- capabilities: [gpu]
|
|
||||||
volumes:
|
volumes:
|
||||||
- ./worker/voices:/app/voices
|
tts-models:
|
||||||
depends_on:
|
|
||||||
- redis
|
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
default:
|
default:
|
||||||
name: wlkns-net
|
name: wlkns-net
|
||||||
@ -8,6 +8,7 @@ class SpeechRequest(BaseModel):
|
|||||||
model: str
|
model: str
|
||||||
input: str
|
input: str
|
||||||
voice: str = "auto"
|
voice: str = "auto"
|
||||||
|
language: str | None = None
|
||||||
format: str = "wav"
|
format: str = "wav"
|
||||||
voice_sample_url: str | None = None
|
voice_sample_url: str | None = None
|
||||||
voice_sample_base64: str | None = None
|
voice_sample_base64: str | None = None
|
||||||
|
|||||||
@ -12,7 +12,7 @@ def push_job(data: dict) -> str:
|
|||||||
r.lpush(QUEUE, json.dumps(data))
|
r.lpush(QUEUE, json.dumps(data))
|
||||||
return job_id
|
return job_id
|
||||||
|
|
||||||
def await_result(job_id: str, timeout=30):
|
def await_result(job_id: str, timeout=120):
|
||||||
key = f"{RESULT}:{job_id}"
|
key = f"{RESULT}:{job_id}"
|
||||||
start=time.time()
|
start=time.time()
|
||||||
while time.time()-start < timeout:
|
while time.time()-start < timeout:
|
||||||
|
|||||||
1
gateway/port.txt
Normal file
1
gateway/port.txt
Normal file
@ -0,0 +1 @@
|
|||||||
|
8003
|
||||||
@ -1,7 +1,8 @@
|
|||||||
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 AS base
|
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 AS base
|
||||||
|
ENV PYTHONUNBUFFERED=1
|
||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
COPY requirements.worker.txt .
|
COPY requirements.worker.txt .
|
||||||
RUN apt-get update && apt-get install -y python3-pip ffmpeg
|
RUN apt-get update && apt-get install -y python3-pip ffmpeg espeak-ng
|
||||||
RUN pip3 install -r requirements.worker.txt
|
RUN pip3 install -r requirements.worker.txt
|
||||||
COPY . .
|
COPY . .
|
||||||
CMD ["python3","main.py"]
|
CMD ["python3","main.py"]
|
||||||
|
|||||||
@ -1,19 +1,39 @@
|
|||||||
from pydub import AudioSegment
|
from pydub import AudioSegment
|
||||||
import io
|
import io
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
def convert_audio(audio_data, fmt: str, sample_rate: int = 24000):
|
||||||
|
"""
|
||||||
|
Converts raw audio data (numpy array or list of floats) to the target format.
|
||||||
|
Assumes mono audio.
|
||||||
|
"""
|
||||||
|
# Ensure numpy array
|
||||||
|
if not isinstance(audio_data, np.ndarray):
|
||||||
|
audio_data = np.array(audio_data)
|
||||||
|
|
||||||
|
# Check if float and normalize/convert to int16
|
||||||
|
if audio_data.dtype.kind == 'f':
|
||||||
|
# Clip to Avoid wrap-around
|
||||||
|
audio_data = np.clip(audio_data, -1.0, 1.0)
|
||||||
|
# Convert to 16-bit PCM
|
||||||
|
audio_data = (audio_data * 32767).astype(np.int16)
|
||||||
|
|
||||||
def convert_audio(raw_bytes: bytes, fmt: str):
|
|
||||||
# raw mono 32-bit float fake waveform
|
|
||||||
seg = AudioSegment(
|
seg = AudioSegment(
|
||||||
raw_bytes,
|
audio_data.tobytes(),
|
||||||
frame_rate=22050,
|
frame_rate=sample_rate,
|
||||||
sample_width=4,
|
sample_width=2, # 16-bit
|
||||||
channels=1
|
channels=1
|
||||||
)
|
)
|
||||||
buf=io.BytesIO()
|
|
||||||
|
buf = io.BytesIO()
|
||||||
seg.export(buf, format=fmt)
|
seg.export(buf, format=fmt)
|
||||||
mime={
|
|
||||||
"wav":"audio/wav",
|
mime = {
|
||||||
"mp3":"audio/mpeg",
|
"wav": "audio/wav",
|
||||||
"ogg":"audio/ogg"
|
"mp3": "audio/mpeg",
|
||||||
}.get(fmt,"audio/wav")
|
"ogg": "audio/ogg",
|
||||||
|
"flac": "audio/flac",
|
||||||
|
"aac": "audio/aac"
|
||||||
|
}.get(fmt, "audio/wav")
|
||||||
|
|
||||||
return buf.getvalue(), mime
|
return buf.getvalue(), mime
|
||||||
@ -1,10 +1,163 @@
|
|||||||
# Dummy XTTS2 logic placeholder
|
import os
|
||||||
# Replace with real TTS model loading
|
# Auto-agree to Coqui TOS (Must be before imports)
|
||||||
|
os.environ["COQUI_TOS_AGREED"] = "1"
|
||||||
|
|
||||||
|
import json
|
||||||
|
import base64
|
||||||
|
import tempfile
|
||||||
|
import requests
|
||||||
|
import torch
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
# Singleton for lazy loading
|
||||||
|
_model = None
|
||||||
|
|
||||||
|
def _allow_xtts_config_pickle():
|
||||||
|
"""Allow loading XTTS configs with torch >=2.6 safe loading."""
|
||||||
|
add_safe = getattr(torch.serialization, "add_safe_globals", None)
|
||||||
|
if not add_safe:
|
||||||
|
return
|
||||||
|
allowed = []
|
||||||
|
try:
|
||||||
|
from TTS.tts.configs.xtts_config import XttsConfig
|
||||||
|
from TTS.tts.models.xtts import XttsAudioConfig
|
||||||
|
allowed += [XttsConfig, XttsAudioConfig]
|
||||||
|
except Exception as e:
|
||||||
|
print(f"⚠️ Could not register safe globals for XTTS config: {e}")
|
||||||
|
try:
|
||||||
|
import TTS.config.shared_configs as shared_configs
|
||||||
|
allowed += [v for v in shared_configs.__dict__.values() if isinstance(v, type)]
|
||||||
|
except Exception as e:
|
||||||
|
print(f"⚠️ Could not register shared config globals: {e}")
|
||||||
|
try:
|
||||||
|
import TTS.tts.models.xtts as xtts_models
|
||||||
|
allowed += [v for v in xtts_models.__dict__.values() if isinstance(v, type)]
|
||||||
|
except Exception as e:
|
||||||
|
print(f"⚠️ Could not register XTTS model globals: {e}")
|
||||||
|
if allowed:
|
||||||
|
add_safe(allowed)
|
||||||
|
|
||||||
|
def get_model():
|
||||||
|
global _model
|
||||||
|
if _model is None:
|
||||||
|
print("⏳ Loading XTTS Model (Lazy Load)....")
|
||||||
|
# Lazy Import to prevent startup hang
|
||||||
|
from TTS.api import TTS
|
||||||
|
_allow_xtts_config_pickle()
|
||||||
|
|
||||||
|
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||||
|
print(f"🔧 XTTS Running on: {device}")
|
||||||
|
|
||||||
|
# Load Model (download if needed)
|
||||||
|
# Using default XTTS v2 model
|
||||||
|
_model = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to(device)
|
||||||
|
print("✅ XTTS Model loaded successfully.")
|
||||||
|
return _model
|
||||||
|
|
||||||
def synthesize(job: dict):
|
def synthesize(job: dict):
|
||||||
# return artificial sine wave placeholder
|
from langdetect import detect
|
||||||
import numpy as np
|
model = get_model()
|
||||||
sr=22050
|
...
|
||||||
t=np.linspace(0,0.3,int(sr*0.3))
|
|
||||||
tone=(0.1*np.sin(2*np.pi*440*t)).astype('float32')
|
text = job.get("input")
|
||||||
return tone.tobytes()
|
if not text:
|
||||||
|
raise ValueError("No input text provided")
|
||||||
|
|
||||||
|
# Language handling
|
||||||
|
language = job.get("language")
|
||||||
|
if not language:
|
||||||
|
try:
|
||||||
|
# Simple detection
|
||||||
|
detected = detect(text)
|
||||||
|
# XTTS expects 2-letter codes usually.
|
||||||
|
# We assume detected is valid or mapped if needed.
|
||||||
|
# Supported: en, es, fr, de, it, pt, pl, tr, ru, nl, cs, ar, zh-cn, ja, hu, ko
|
||||||
|
language = detected
|
||||||
|
print(f"🌍 Auto-detected language: {language}")
|
||||||
|
except:
|
||||||
|
language = "en"
|
||||||
|
print("⚠️ Language detection failed, using 'en'")
|
||||||
|
|
||||||
|
# Speaker Handling
|
||||||
|
speaker_wav = None
|
||||||
|
temp_files = []
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Priority 1: Direct URL
|
||||||
|
if job.get("voice_sample_url"):
|
||||||
|
try:
|
||||||
|
print(f"⬇️ Downloading voice sample from {job['voice_sample_url']}")
|
||||||
|
r = requests.get(job["voice_sample_url"], timeout=10)
|
||||||
|
r.raise_for_status()
|
||||||
|
t = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
|
||||||
|
t.write(r.content)
|
||||||
|
t.close()
|
||||||
|
speaker_wav = t.name
|
||||||
|
temp_files.append(t.name)
|
||||||
|
except Exception as e:
|
||||||
|
print(f"❌ Failed to download voice sample: {e}")
|
||||||
|
|
||||||
|
# Priority 2: Base64
|
||||||
|
if not speaker_wav and job.get("voice_sample_base64"):
|
||||||
|
try:
|
||||||
|
b64 = job["voice_sample_base64"]
|
||||||
|
decoded = base64.b64decode(b64)
|
||||||
|
t = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
|
||||||
|
t.write(decoded)
|
||||||
|
t.close()
|
||||||
|
speaker_wav = t.name
|
||||||
|
temp_files.append(t.name)
|
||||||
|
except Exception as e:
|
||||||
|
print(f"❌ Failed to decode base64 voice: {e}")
|
||||||
|
|
||||||
|
# Priority 3: Registry / Local File
|
||||||
|
if not speaker_wav:
|
||||||
|
voice_id = job.get("voice", "auto")
|
||||||
|
if voice_id and voice_id != "auto":
|
||||||
|
# Look in worker/voices/
|
||||||
|
base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
voice_path = os.path.join(base_dir, "voices", f"{voice_id}.wav")
|
||||||
|
|
||||||
|
# Check for other extensions if wav missing
|
||||||
|
if not os.path.exists(voice_path):
|
||||||
|
for ext in [".mp3", ".ogg", ".m4a"]:
|
||||||
|
p = os.path.join(base_dir, "voices", f"{voice_id}{ext}")
|
||||||
|
if os.path.exists(p):
|
||||||
|
voice_path = p
|
||||||
|
break
|
||||||
|
|
||||||
|
if os.path.exists(voice_path):
|
||||||
|
speaker_wav = voice_path
|
||||||
|
print(f"🗣️ Using registered voice: {voice_id}")
|
||||||
|
else:
|
||||||
|
print(f"⚠️ Voice '{voice_id}' not found in registry.")
|
||||||
|
|
||||||
|
# Priority 4: Default/Auto Voice
|
||||||
|
if not speaker_wav:
|
||||||
|
# Fallback to a default file if it exists
|
||||||
|
base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
default_path = os.path.join(base_dir, "voices", "default.wav")
|
||||||
|
if os.path.exists(default_path):
|
||||||
|
speaker_wav = default_path
|
||||||
|
print("⚠️ Using default.wav")
|
||||||
|
else:
|
||||||
|
# If completely nothing, we can't synthesize with XTTS
|
||||||
|
# Unless we use speaker_idxs (only for multi-speaker models w/o cloning?)
|
||||||
|
# XTTS v2 IS zero-shot, needs reference.
|
||||||
|
raise ValueError("No speaker reference found (url, base64, registry, or default.wav)")
|
||||||
|
|
||||||
|
# Run Inference
|
||||||
|
print(f"🎤 Synthesizing: '{text[:30]}...' Lang: {language}")
|
||||||
|
|
||||||
|
# XTTS API returns List[float]
|
||||||
|
wav = model.tts(text=text, speaker_wav=speaker_wav, language=language)
|
||||||
|
|
||||||
|
return wav
|
||||||
|
|
||||||
|
finally:
|
||||||
|
# Cleanup
|
||||||
|
for f in temp_files:
|
||||||
|
try:
|
||||||
|
os.remove(f)
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
|||||||
@ -1,9 +1,14 @@
|
|||||||
|
print("DEBUG: Starting worker...", flush=True)
|
||||||
import time, json, os
|
import time, json, os
|
||||||
|
print("DEBUG: Imported stdlib", flush=True)
|
||||||
from core.queue_worker import fetch_job, store_result
|
from core.queue_worker import fetch_job, store_result
|
||||||
|
print("DEBUG: Imported queue_worker", flush=True)
|
||||||
from engine.xtts2_loader import synthesize
|
from engine.xtts2_loader import synthesize
|
||||||
|
print("DEBUG: Imported xtts2_loader", flush=True)
|
||||||
from engine.audio_export import convert_audio
|
from engine.audio_export import convert_audio
|
||||||
|
print("DEBUG: Imported audio_export", flush=True)
|
||||||
|
|
||||||
print("Worker gestartet. Warte auf Jobs…")
|
print("Worker gestartet. Warte auf Jobs…", flush=True)
|
||||||
|
|
||||||
while True:
|
while True:
|
||||||
job = fetch_job()
|
job = fetch_job()
|
||||||
@ -11,6 +16,13 @@ while True:
|
|||||||
time.sleep(0.1)
|
time.sleep(0.1)
|
||||||
continue
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
start = time.time()
|
||||||
|
print(f"🔄 Processing Job {job['job_id']}...", flush=True)
|
||||||
audio = synthesize(job)
|
audio = synthesize(job)
|
||||||
out, mime = convert_audio(audio, job.get("format","wav"))
|
out, mime = convert_audio(audio, job.get("format","wav"))
|
||||||
store_result(job["job_id"], out, mime)
|
store_result(job["job_id"], out, mime)
|
||||||
|
print(f"✅ Job {job['job_id']} done in {time.time()-start:.2f}s")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"❌ Error processing job {job.get('job_id')}: {e}")
|
||||||
|
# Optional: Store error state if protocol supports it
|
||||||
|
|||||||
@ -3,3 +3,8 @@ redis
|
|||||||
torch
|
torch
|
||||||
numpy
|
numpy
|
||||||
requests
|
requests
|
||||||
|
transformers==4.42.4
|
||||||
|
TTS==0.22.0
|
||||||
|
scipy
|
||||||
|
langdetect
|
||||||
|
torchcodec
|
||||||
|
|||||||
17
worker/voices/README.md
Normal file
17
worker/voices/README.md
Normal file
@ -0,0 +1,17 @@
|
|||||||
|
# Voice Registry
|
||||||
|
|
||||||
|
Place `.wav` files here to register them as permanent voices.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
If you place a file named `narrator.wav` in this directory:
|
||||||
|
|
||||||
|
1. Restart the worker (or mount this volume dynamically).
|
||||||
|
2. Send a request with `"voice": "narrator"`.
|
||||||
|
|
||||||
|
The system will use this file as the speaker reference for XTTS cloning.
|
||||||
|
|
||||||
|
## Formats
|
||||||
|
|
||||||
|
Supported formats: `.wav`, `.mp3`, `.ogg`, `.m4a`.
|
||||||
|
Recommended: Mono, 22050Hz or 24000Hz WAV (16-bit).
|
||||||
BIN
worker/voices/default.wav
Normal file
BIN
worker/voices/default.wav
Normal file
Binary file not shown.
Reference in New Issue
Block a user