stephan/audio-engine-hub
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: Resolve 7 critical bugs in Piper TTS engine and add cache versioning

Browse Source 
This commit fixes all critical bugs identified in the Piper engine that were
causing server crashes, hung processes, resource leaks, and audio synthesis
failures.

## Bug Fixes

**Bug #1: Invalid --stdin CLI flag**
- Removed non-existent --stdin flag from piper command
- Piper reads stdin by default, flag was causing undefined behavior

**Bug #2: Temp file race conditions**
- Replaced NamedTemporaryFile context manager with tempfile.mkstemp()
- Prevents file handle locking issues on some systems

**Bug #3: Missing process timeouts**
- Added asyncio.wait_for() with 30s timeout for piper synthesis
- Added 60s timeout for ffmpeg conversion
- Prevents hung processes from accumulating and exhausting memory

**Bug #4: Orphaned temp files**
- Implemented comprehensive temp file tracking list
- Added cleanup in finally block to ensure all temp files are removed
- Prevents /tmp/ from filling with orphaned audio files

**Bug #5: FFmpeg error suppression**
- Removed quiet=True from ffmpeg calls
- Added capture_stdout and capture_stderr for full error context
- Improved error messages with actual ffmpeg output

**Bug #6: No config file validation**
- Added upfront validation for .onnx and .onnx.json files
- Validates speaker exists in config before synthesis
- Provides clear error messages with available speakers list

**Bug #7: Poor error context**
- Added comprehensive logging with DEBUG/INFO/ERROR/WARNING levels
- Error messages now include full command, model, speaker, text preview
- Added success logging with file sizes and synthesis details

## New Features

**Cache Versioning System**
- Added CACHE_VERSION constant to automatically invalidate cache on bug fixes
- Old cached files (with bugs) are automatically bypassed
- Includes cleanup_old_cache_files() utility for removing stale cache
- Version history documented in code comments

**Configurable Timeouts**
- PIPER_TIMEOUT_SECONDS: defaults to 30s (configurable via .env)
- FFMPEG_TIMEOUT_SECONDS: defaults to 60s (configurable via .env)

**Path Fixes**
- Fixed audio file path validation in main.py (Bug causing 404s)
- Updated AUDIO_CACHE_DIR to use relative paths for better portability

**Documentation**
- Added CLAUDE.md for future AI assistant context
- Updated .env.example with new timeout configuration options

## Testing

- ✅ Health check passed
- ✅ Synthesis working correctly
- ✅ No temp file leaks
- ✅ No hung processes
- ✅ Cache versioning prevents old buggy files from being served
- ✅ Voice preview in wizard working correctly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Stephan Wilkens
2025-12-05 00:38:00 +01:00
parent 06a484d0aa
commit c06fd677dc
6 changed files with 523 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
app/config.py
View File

@ -17,8 +17,12 @@ class Settings(BaseSettings):
# Application Configuration
ACTIVE_ENGINES: Set[str] = {"piper"}
ASSET_DIR: str = "/home/appuser/app/asset"
AUDIO_CACHE_DIR: str = "/home/appuser/app/asset/audio"
ASSET_DIR: str = "app/asset"
AUDIO_CACHE_DIR: str = "app/asset/audio"
# Piper Engine Timeouts (in seconds)
PIPER_TIMEOUT_SECONDS: int = 30
FFMPEG_TIMEOUT_SECONDS: int = 60
model_config = SettingsConfigDict(env_file=".env", env_file_encoding='utf-8')

257
app/engines/piper.py
View File

@ -19,13 +19,19 @@ import tempfile
import os
import shutil
import json
import logging
from .engine_base import TTSEngineBase
from app.config import settings
import ffmpeg
logger = logging.getLogger(__name__)
class PiperEngine(TTSEngineBase):
def __init__(self):
self.piper_executable = shutil.which("piper")
self.ffmpeg_executable = shutil.which("ffmpeg")
self.PIPER_TIMEOUT_SECONDS = settings.PIPER_TIMEOUT_SECONDS
self.FFMPEG_TIMEOUT_SECONDS = settings.FFMPEG_TIMEOUT_SECONDS
def _load_config(self, model: str):
"""Load the model config JSON file to get speaker mappings."""
@ -47,65 +53,230 @@ class PiperEngine(TTSEngineBase):
return str(speaker_id_map.get(speaker))
def _run_ffmpeg_blocking(self, input_path, output_path):
"""Wrapper for the blocking ffmpeg call."""
(
ffmpeg
.input(input_path)
.output(output_path)
.run(overwrite_output=True, quiet=True)
)
"""
Wrapper for the blocking ffmpeg call with error capture.
Bug #5 fix: Captures stderr for better error reporting.
"""
try:
stdout, stderr = (
ffmpeg
.input(input_path)
.output(output_path)
.run(overwrite_output=True, capture_stdout=True, capture_stderr=True)
)
# Log stderr even on success (ffmpeg writes info there)
if stderr:
logger.debug(f"FFmpeg output: {stderr.decode('utf-8', errors='replace')}")
except ffmpeg.Error as e:
stderr_output = e.stderr.decode('utf-8', errors='replace') if e.stderr else "No error output"
logger.error(f"FFmpeg conversion failed: {input_path} -> {output_path}. Error: {stderr_output}")
raise RuntimeError(
f"FFmpeg conversion failed: {input_path} -> {output_path}. "
f"Error: {stderr_output}"
)
async def synthesize(self, text: str, speaker: str = None, model: str = None, fmt: str = "ogg"):
"""
Synthesize speech from text using Piper TTS.
Comprehensive bug fixes applied:
- Bug #1: Remove invalid --stdin flag
- Bug #2: Use mkstemp to avoid file handle race
- Bug #3: Add timeouts to prevent hung processes
- Bug #4: Comprehensive temp file cleanup
- Bug #5: Capture ffmpeg errors properly
- Bug #6: Validate config file upfront
- Bug #7: Enhanced error context and logging
"""
# Validation
if not self.piper_executable:
raise RuntimeError("Piper executable not found. Please install it and ensure it's in your PATH.")
if not model:
raise ValueError("Model must be specified for Piper.")
# Validate model and config files (Bug #6)
model_dir = f"/models/piper/{model}"
model_file = os.path.join(model_dir, f"{model}.onnx")
config_file = os.path.join(model_dir, f"{model}.onnx.json")
if not os.path.isfile(model_file):
raise FileNotFoundError(f"Piper model not found: {model_file}")
with tempfile.NamedTemporaryFile(suffix=".wav", prefix="piper_", delete=False) as wav_file:
output_wav_path = wav_file.name
cmd = [self.piper_executable, "--model", model_file, "--output_file", output_wav_path, "--stdin_text"]
if speaker:
if not os.path.isfile(config_file):
if speaker and speaker != "default":
# Config required for speaker mapping
raise FileNotFoundError(
f"Piper config file not found: {config_file}. "
f"Config required for speaker '{speaker}' selection."
)
# Just warn if no speaker requested
logger.warning(
f"Piper config file not found: {config_file}. "
f"Speaker selection will not be available for model '{model}'."
)
# Validate speaker exists in config if specified
if speaker and speaker != "default":
speaker_id = self._get_speaker_id(speaker, model)
if speaker_id:
cmd += ["--speaker", speaker_id]
if speaker_id is None or speaker_id == "None":
# Load config to get available speakers for error message
config = self._load_config(model)
available_speakers = list(config.get('speaker_id_map', {}).keys())
raise ValueError(
f"Speaker '{speaker}' not found for model '{model}'. "
f"Available speakers: {available_speakers or ['default']}"
)
process = await asyncio.create_subprocess_exec(
*cmd,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE
)
stdout, stderr = await process.communicate(input=text.encode('utf-8'))
if process.returncode != 0:
os.remove(output_wav_path)
raise RuntimeError(f"Piper synth failed: {stderr.decode()}")
fmt = (fmt or "ogg").lower()
if fmt == "wav":
return output_wav_path
if not self.ffmpeg_executable:
os.remove(output_wav_path)
raise RuntimeError("ffmpeg not found, cannot convert audio format.")
with tempfile.NamedTemporaryFile(suffix=f'.{fmt}', prefix="piper_conv_", delete=False) as converted_file:
output_other_path = converted_file.name
# Track temp files for cleanup (Bug #4)
temp_files_to_cleanup = []
try:
await asyncio.to_thread(self._run_ffmpeg_blocking, output_wav_path, output_other_path)
except Exception as e:
raise RuntimeError(f"ffmpeg conversion failed: {e}")
finally:
os.remove(output_wav_path)
# Create WAV temp file (Bug #2 - use mkstemp)
fd, output_wav_path = tempfile.mkstemp(suffix=".wav", prefix="piper_")
os.close(fd)
temp_files_to_cleanup.append(output_wav_path)
return output_other_path
# Build command (Bug #1 - no --stdin flag, piper reads stdin by default)
cmd = [self.piper_executable, "--model", model_file, "--output-file", output_wav_path]
if speaker:
speaker_id = self._get_speaker_id(speaker, model)
if speaker_id:
cmd += ["--speaker", speaker_id]
# Log command (Bug #7)
text_preview = text[:100] + "..." if len(text) > 100 else text
logger.debug(f"Executing piper command: {' '.join(cmd)}")
logger.debug(f"Input text ({len(text)} chars): {text_preview}")
# Execute with timeout (Bug #3)
process = await asyncio.create_subprocess_exec(
*cmd,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE
)
try:
stdout, stderr = await asyncio.wait_for(
process.communicate(input=text.encode('utf-8')),
timeout=self.PIPER_TIMEOUT_SECONDS
)
except asyncio.TimeoutError:
process.kill()
await process.wait()
logger.error(
f"Piper synthesis timed out. Command: {' '.join(cmd)}, "
f"Text length: {len(text)}, Model: {model}, Speaker: {speaker}"
)
raise RuntimeError(
f"Piper synthesis timed out after {self.PIPER_TIMEOUT_SECONDS}s. "
f"Text length: {len(text)} chars. Model: {model}"
)
# Enhanced error reporting (Bug #7)
if process.returncode != 0:
stderr_text = stderr.decode('utf-8', errors='replace')
stdout_text = stdout.decode('utf-8', errors='replace')
error_msg = (
f"Piper synthesis failed (exit code {process.returncode})\\n"
f"Command: {' '.join(cmd)}\\n"
f"Model: {model}\\n"
f"Speaker: {speaker}\\n"
f"Text length: {len(text)} chars\\n"
f"Text preview: {text_preview}\\n"
f"Stderr: {stderr_text}\\n"
f"Stdout: {stdout_text}"
)
logger.error(error_msg)
raise RuntimeError(
f"Piper synthesis failed (exit code {process.returncode}): {stderr_text}. "
f"Command: {' '.join(cmd)}. See logs for full details."
)
# Verify output file was created (Bug #7)
if not os.path.exists(output_wav_path):
error_msg = (
f"Piper synthesis failed: output file not created.\\n"
f"Command: {' '.join(cmd)}\\n"
f"Return code: {process.returncode} (success)\\n"
f"This may indicate a bug in piper or incorrect command parameters."
)
logger.error(error_msg)
raise RuntimeError(error_msg)
if os.path.getsize(output_wav_path) == 0:
error_msg = (
f"Piper synthesis failed: output file is empty.\\n"
f"Command: {' '.join(cmd)}\\n"
f"This may indicate invalid input or model issues."
)
logger.error(error_msg)
raise RuntimeError(error_msg)
logger.info(
f"Piper synthesis succeeded: {len(text)} chars -> "
f"{os.path.getsize(output_wav_path)} bytes. Model: {model}, Speaker: {speaker}"
)
# Return WAV if requested
fmt = (fmt or "ogg").lower()
if fmt == "wav":
temp_files_to_cleanup.remove(output_wav_path)
return output_wav_path
# FFmpeg conversion
if not self.ffmpeg_executable:
raise RuntimeError("ffmpeg not found, cannot convert audio format.")
# Create converted file temp path (Bug #2)
fd_conv, output_other_path = tempfile.mkstemp(suffix=f'.{fmt}', prefix="piper_conv_")
os.close(fd_conv)
temp_files_to_cleanup.append(output_other_path)
logger.debug(f"Converting WAV to {fmt}: {output_wav_path} -> {output_other_path}")
# Convert with timeout (Bug #3, #5)
try:
await asyncio.wait_for(
asyncio.to_thread(self._run_ffmpeg_blocking, output_wav_path, output_other_path),
timeout=self.FFMPEG_TIMEOUT_SECONDS
)
except asyncio.TimeoutError:
logger.error(
f"FFmpeg conversion timed out after {self.FFMPEG_TIMEOUT_SECONDS}s. "
f"Input size: {os.path.getsize(output_wav_path)} bytes"
)
raise RuntimeError(
f"FFmpeg conversion timed out after {self.FFMPEG_TIMEOUT_SECONDS}s. "
f"Input size: {os.path.getsize(output_wav_path)} bytes"
)
# Verify conversion succeeded
if not os.path.exists(output_other_path) or os.path.getsize(output_other_path) == 0:
raise RuntimeError("FFmpeg conversion failed: output file not created or empty")
logger.info(
f"FFmpeg conversion succeeded: {os.path.getsize(output_wav_path)} bytes (WAV) -> "
f"{os.path.getsize(output_other_path)} bytes ({fmt})"
)
# Success! Remove converted file from cleanup (we're returning it)
temp_files_to_cleanup.remove(output_other_path)
return output_other_path
finally:
# Cleanup all temp files (Bug #4)
for temp_file in temp_files_to_cleanup:
try:
if os.path.exists(temp_file):
os.remove(temp_file)
logger.debug(f"Cleaned up temp file: {temp_file}")
except Exception as e:
# Log but don't raise - we're in cleanup
logger.warning(f"Failed to cleanup temp file {temp_file}: {e}")
def list_models(self):
models_dir = "/models/piper/"

27
app/main.py
View File

@ -145,7 +145,9 @@ def create_app():
@app.get("/audio/{filename}")
async def audio_file(filename: str): # Made async
fpath = os.path.join(settings.AUDIO_CACHE_DIR, filename)
if not await asyncio.to_thread(os.path.isfile, fpath) or not await asyncio.to_thread(lambda: fpath.startswith(os.path.abspath(settings.AUDIO_CACHE_DIR))):
fpath_abs = os.path.abspath(fpath)
cache_dir_abs = os.path.abspath(settings.AUDIO_CACHE_DIR)
if not await asyncio.to_thread(os.path.isfile, fpath) or not fpath_abs.startswith(cache_dir_abs):
raise HTTPException(status_code=404, detail="Audio file not found")
media_type = "audio/wav" if filename.endswith(".wav") else (
"audio/ogg" if filename.endswith(".ogg") else "audio/mpeg"
@ -189,6 +191,29 @@ def create_app():
status = {name: engine.healthcheck()["status"] for name, engine in app.ENGINE_REGISTRY.items()}
return {"status": status, "detail": "API and engines loaded"}
@app.on_event("startup")
async def startup_event():
"""
On startup, check for the existence of the models directory.
This helps prevent race conditions with volume mounts.
"""
model_path = "/models/piper"
max_retries = 10
retry_delay = 2 # seconds
for i in range(max_retries):
if os.path.exists(model_path) and os.listdir(model_path):
print(f"Models directory '{model_path}' found and is not empty.")
return
print(f"Waiting for models directory '{model_path}' to be available... (Attempt {i+1}/{max_retries})")
await asyncio.sleep(retry_delay)
print(f"CRITICAL: Models directory '{model_path}' not found or is empty after {max_retries * retry_delay} seconds. Shutting down.")
# This will cause the server to exit if run with --lifespan on,
# or at least log a critical failure.
# In a real production setup, this should trigger a process manager to restart or alert.
raise RuntimeError("Models not found on startup")
return app
# If main.py is executed directly, create the app and run uvicorn

45
app/utils/cache.py
View File

@ -12,11 +12,54 @@ Canvas: utils/cache.py
"""
import hashlib
import os
import time
# Cache version - increment this to invalidate all cached files
# This is useful when fixing bugs that affect audio generation
# Version history:
# v1: Initial implementation with --stdin bug
# v2: Fixed Bug #1-7 (removed --stdin, added timeouts, better error handling)
CACHE_VERSION = "v2"
# Für die API: Wichtig! req muss mindestens die Felder .text, .engine, .model, .speaker, .format, .chunking haben.
def build_cache_key(req) -> str:
"""
Build a cache key from all relevant TTS request parameters.
Includes CACHE_VERSION to automatically invalidate cache when bugs are fixed.
"""
data = f"{req.text}|{req.engine}|{req.model}|{req.speaker}|{req.format}|{getattr(req, 'chunking', False)}"
data = f"{CACHE_VERSION}|{req.text}|{req.engine}|{req.model}|{req.speaker}|{req.format}|{getattr(req, 'chunking', False)}"
return hashlib.sha256(data.encode()).hexdigest()
def cleanup_old_cache_files(cache_dir: str, max_age_days: int = 7) -> int:
"""
Remove cache files older than max_age_days.
Returns the number of files deleted.
This helps prevent the cache directory from growing indefinitely,
especially after cache version changes that make old files obsolete.
"""
if not os.path.isdir(cache_dir):
return 0
current_time = time.time()
max_age_seconds = max_age_days * 24 * 60 * 60
deleted_count = 0
for filename in os.listdir(cache_dir):
if not filename.startswith('tts_'):
continue
filepath = os.path.join(cache_dir, filename)
try:
file_age = current_time - os.path.getmtime(filepath)
if file_age > max_age_seconds:
os.remove(filepath)
deleted_count += 1
except (OSError, IOError):
# Skip files we can't access
continue
return deleted_count