audio-engine-hub/session_resumee.md

Session Resumee - AudioEngineHub Project Refactoring

Date: Donnerstag, 4. Dezember 2025

Objective: Refactor the AudioEngineHub project to follow best practices for containerization, error handling, and project structure, based on a provided guide.md document.

---

Initial Project State & Overview

The project was a modular FastAPI-based TTS server with a TTSEngineBase interface. Key initial findings:

• piper was functional.
• styletts and chattts were dummy implementations.
• f5_tts was implemented but inactive.
• Configuration was scattered and hardcoded.
• Error handling was basic.
• Tests (pytest and unittest) existed but were limited and inconsistent.
• No containerization strategy was in place, leading to potential dependency hell.

Refactoring Phase 1: Robustness & Configuration

1. Centralized Configuration:

   • Replaced hardcoded values with pydantic-settings for .env file management.
   • config.py created (later moved to app/config.py).
   • IMAGE_NAME in Makefile was also user-configurable.
   • requirements.txt was updated with pydantic-settings.
   • docker-compose.yml was updated to use .env.

2. Configurable Engines Feature:

   • Implemented dynamic ENGINE_REGISTRY loading based on ACTIVE_ENGINES setting in .env.
   • Allows easy activation/deactivation of TTS engines.

3. Robust Error Handling:

   • Implemented comprehensive input validation in the /tts endpoint (checking engine, model, speaker existence).
   • Added dependency checks (e.g., ffmpeg, piper executables) to engines, reporting HTTP 503 for unavailable engines.
   • Secured tempfile.mktemp usage by replacing it with tempfile.NamedTemporaryFile.
   • Wrapped synthesis logic in try-except blocks to catch and propagate engine-specific errors as HTTP 500.

4. Asynchronous Operations:

   • Changed TTSEngineBase.synthesize and selftest to async.
   • Refactored all concrete engine implementations (piper, f5_tts, styletts, chattts) to use async def methods.
   • Updated app/main.py's /tts endpoint to be async and use await for engine calls and asyncio.gather for concurrent chunk synthesis.
   • Wrapped blocking I/O (file ops, ffmpeg) and CPU-bound tasks in asyncio.to_thread.
   • Updated tests/test_f5_tts.py to correctly await async calls.

Refactoring Phase 2: Containerization & Workflow (Based on guide.md)

1. Integrated Makefile:

   • Created a Makefile with targets for build, run, stop, logs, shell, up, down, test, tag, push, clean, help.
   • Included robust shell functions for port checking (check_traefik_ports_free, check_app_port_free).
   • Set SHELL := /bin/bash in Makefile to ensure correct shell interpretation.
   • Ensured PYTHONPATH=$(PWD) is set for make test.

2. Adopted Multi-Stage Dockerfile:

   • Implemented a multi-stage Dockerfile (builder/runner stages).
   • builder stage creates a Python virtual environment and installs requirements.txt (including gunicorn).
   • runner stage uses python:3.11-slim, installs runtime system dependencies (ffmpeg), creates an unprivileged appuser, and sets the production CMD to gunicorn with uvicorn workers.

3. Refactored Project Structure (app/ package):

   • Created an app/ directory.
   • Moved main.py, config.py, engines/, models/, utils/ into app/.
   • Created app/__init__.py.
   • Updated all Python import paths (from app.config import settings, from app.engines.piper import PiperEngine, etc.).
   • Updated internal references in engine files (e.g., model_dir, voices_dir).

4. Refined docker-compose.yml with Traefik:

   • Integrated traefik service for dynamic reverse proxying during local development.
   • Modified app service with Traefik labels and connected both services to a web network.
   • Adjusted Docker volumes mounts to match the new app/ structure (e.g., ./models:/home/appuser/app/models).
   • Updated app service command for Uvicorn hot-reloading in dev.

5. Centralized Testing Workflow:

   • Removed test_run.sh.
   • Integrated make test for running pytest --cov=. app/ tests/.

6. Dedicated Documentation:

   • Created docs/ directory.
   • Moved guide.md to docs/guide.md.

Verification & Troubleshooting

• Tests: All unit/integration tests (make test) are passing.
• Local Run (Virtual Env): Initial local runs (python app/main.py) failed due to ModuleNotFoundError (fixed by python -m app.main) and PermissionError (fixed by needing to mock or redirect settings.AUDIO_CACHE_DIR for local direct execution, but not strictly needed for successful app execution through uvicorn). The current approach is to verify in Docker.
• Docker Compose:
  • Initial make up failures were due to an outdated docker-compose client (1.29.2) and later, Makefile syntax issues (fixed by setting SHELL := /bin/bash and fixing macros).
  • docker-compose was eventually updated to the docker compose CLI plugin (v5.0.0).
  • make up command finally succeeded in bringing up containers.
  • The curl http://localhost/health command returned 404 page not found. This indicates a potential routing issue with Traefik or the application not being responsive on the expected path within the container. (This is the last unresolved issue).

---

Final Debugging & Resolution (Post-Refactoring)

After the major refactoring, the service was unable to start and was returning 404 errors. The final debugging session addressed these issues.

1. FastAPI App Startup Fix:

   • Problem: The Uvicorn server was failing with Attribute "app" not found in module "app.main".
   • Solution: The app instance was only created inside the if __name__ == "__main__" block. Moved app = create_app() to the module's global scope in app/main.py so Uvicorn could find it.

2. Traefik & Docker Compose Issues:

   • Problem: The initial docker-compose.yml included a Traefik service for reverse proxying, which was causing multiple issues (invalid container names due to missing .env variables, Docker client API version errors). The user also clarified that they use an existing reverse proxy and did not want a new Traefik container deployed.
   • Solution: Removed the traefik service entirely from docker-compose.yml. The app service's port was exposed directly to the host.

3. Model Loading Failure:

   • Problem: The piper engine was not loading any models, returning an empty list and causing /tts requests to fail with a 422 Unprocessable Entity error.
   • Diagnosis: The root cause was an incorrect volume mount. The docker-compose.yml was attempting to mount a non-existent, empty ./models directory from the host. The actual models were located in ./app/models.
   • Solution:
     1. Corrected the docker-compose.yml volume mount to point to the correct source directory: - ./app/models:/models.
     2. Updated the app/engines/piper.py code to use the absolute path /models/piper/ to look for models inside the container, making the configuration more robust and independent of the working directory.

4. Developer Experience (DX) Improvements:

   • Automatic Port Finding: The Makefile was enhanced. The make up and make run commands now automatically find a free port starting from 8000, preventing port conflicts.
   • Health Check Command: A make health-check target was added. It runs a sanity check against the deployed container's /health endpoint to verify that the service is up and all engines are reporting an "ok" status.
   • Documentation:
     • The README.md was completely rewritten to provide a clear and up-to-date guide for getting started, usage, and available make commands.
     • A note was added to docs/guide.md to clarify that it describes an older, more advanced setup and to point readers to the new README.md.

Final Status: The service is now fully deployable via make up and passes make health-check. All identified startup issues and bugs have been resolved.

Container Registry Integration

To facilitate pushing and pulling Docker images from a remote registry (e.g., git.wlkns.org), the Makefile and docker-compose.yml were updated:

1. Makefile Configuration:

   • Added REGISTRY := git.wlkns.org and USERNAME := stephan variables.
   • Modified the tag target to tag images in the format $(REGISTRY)/$(USERNAME)/$(IMAGE_NAME):$(TAG).
   • Modified the push target to depend on build and tag, then push the image to the configured registry.

2. docker-compose.yml Integration:

   • Changed the app service definition to use image: ${REGISTRY}/${USERNAME}/${IMAGE_NAME}:${TAG} instead of build: ., making it pull from the registry by default.

3. Makefile Workflow Enhancements:

   • Added a pull target (make pull) to explicitly download the image from the registry.
   • The up target (make up) was modified to start the service using the image specified in docker-compose.yml (which now points to the registry).
   • A new dev-up target (make dev-up) was introduced for local development, which explicitly builds the image from source (docker compose up --build -d) before starting the service.
   • The down and help targets were updated accordingly.

This allows for flexible deployment, supporting both local development with on-demand building and production-like environments pulling from a registry.

Debugging "Always the Same Text" Bug

The user reported that the TTS server was not synthesizing the provided text but was always returning audio for a fixed, incorrect text.

1. Initial Reproduction Attempt (and "422 Unprocessable Entity" error):

   • Attempted to reproduce the bug by sending different texts via scripts/tts_client.py.
   • Encountered a 422 Unprocessable Entity error, which led to debugging model loading.
   • Resolution 1 (Model Loading): Fixed by correcting the volume mount source in docker-compose.yml (./app/models to /models) and updating paths in app/engines/piper.py to absolute /models/piper/.
   • Resolution 2 (Startup Race Condition): Added a startup event handler in app/main.py to wait for the models directory to be available, preventing a race condition.
   • Resolution 3 (Client Connection): Mitigated ConnectionResetError from client by adding time.sleep(1) before requests.post call in scripts/tts_client.py, indicating a subtle client-server connection timing issue.

2. Debugging the "Always the Same Text" Issue:

   • Initial Test: Generated audio for "This is a test with the vctk model." and "This is a completely different sentence to verify the bug."
   • Result: cmp showed the files were different, contradicting the user's report that the text was always the same.
   • User Clarification: User confirmed that listening to the files revealed the same speech output, indicating a deeper issue beyond simple file difference.
   • Investigation of piper.py:
     • The original method for passing text to the piper executable via stdin (--stdin) seemed correct.
     • Manual testing echo "text" | piper ... inside the container proved piper executable works correctly with stdin.
     • Hypothesis: The asyncio.subprocess.communicate(input=...) call was not reliably passing text to piper.
   • Attempted Fix 1 (Explicit stdin write): Modified piper.py to manually write to process.stdin, drain, and close. This led to ConnectionResetError (server crash).
   • Attempted Fix 2 (Revert and --input-file strategy): Reverted piper.py back to process.communicate() (after fixing cmd to use --stdin). Then, changed strategy to use a temporary file for input (--input-file) instead of stdin. This also led to ConnectionResetError.
   • Persistent Crash: The server consistently crashed after each attempt to synthesize actual audio. Debugging was hampered by uvicorn's reloader hiding tracebacks, even after disabling it. It became apparent that the crash was very low-level, possibly within the piper executable's interaction with the file system or system resources.

3. Current Status: The server still crashes when the synthesize method is called to generate real audio. The exact cause of this crash is still unknown, as no Python traceback is being produced in the server logs. Further debugging is required to stabilize the piper engine's subprocess execution.