Development Backend Deployment Setup

Overview

This document details the complete setup and configuration for deploying the FastAPI backend to the Virtuozzo Application Platform development server (dev-backend.mightybox.site).

Infrastructure Details

Platform: Virtuozzo Application Platform (Jelastic PaaS) Server: Apache 2.4.66 with mod_wsgi 4.9.4 Python: 3.14.2 Environment: /var/www/webroot/ Virtual Environment: /var/www/webroot/virtenv/ Application Root: /var/www/webroot/ROOT/

Deployment Architecture

GitHub Actions Workflow

File: .github/workflows/deploy-backend.yml Trigger: Push to development branch affecting backend/** or manual workflow_dispatch

Deployment Steps

1. Package Creation
2. Checkout code
3. Create tar.gz package with: app/, requirements.txt, pyproject.toml, alembic.ini, wsgi.py

4. Excludes: __pycache__, *.pyc, *.egg-info

5. File Transfer

6. Uses SFTP (bypasses shell to avoid output interference)

7. Uploads package to /tmp/backend-deploy.tar.gz

8. Deployment Execution

9. Backup existing deployment
10. Extract new files to /var/www/webroot/ROOT/
11. Create/update virtual environment
12. Install dependencies in venv
13. Configure Apache environment variables
14. Run database migrations
15. Restart Apache

Critical Issues Resolved

1. SSH Key Formatting

Problem: echo command strips newlines from SSH private keys, causing "error in libcrypto".

Solution:

printf "%s\n" "${{ secrets.DEV_SSH_PRIVATE_KEY }}" > ~/.ssh/id_rsa

Why: printf preserves exact formatting including required newlines for OpenSSH keys.

2. Remote Shell Output Interference

Problem: Apache user's shell produced output during SSH sessions, causing: - scp: "Received message too long 1416128883" (hex: 0x54686973 = "This") - rsync: "protocol version mismatch -- is your shell clean?"

Initial Issue: Apache user had /sbin/nologin shell.

Solution:

sudo usermod -s /bin/bash apache

Better Approach: Used SFTP which bypasses shell entirely.

3. Virtual Environment Requirement

Problem: Virtuozzo platform expects applications to use virtual environments at /var/www/webroot/virtenv/.

Apache Config Reference:

WSGIDaemonProcess apache python-path="/var/www/webroot/virtenv/lib/python/:/var/www/webroot/"

Solution:

# Create virtual environment
python3 -m venv /var/www/webroot/virtenv

# Install packages in venv
/var/www/webroot/virtenv/bin/pip install -r requirements.txt

# Run migrations in venv
/var/www/webroot/virtenv/bin/alembic upgrade head

4. WSGI_SCRIPT Environment Variable

Problem: Apache config uses ${WSGI_SCRIPT} variable but it wasn't defined, causing "Config variable not defined" warnings.

Apache Config:

WSGIScriptAlias / ${WSGI_SCRIPT}

Solution:

echo "WSGI_SCRIPT=/var/www/webroot/ROOT/wsgi.py" | sudo tee /usr/local/etc/httpd_vars

Note: Systemd EnvironmentFile requires KEY=VALUE format, not export KEY=VALUE.

5. PyO3 Subinterpreter Incompatibility (Critical)

Problem: FastAPI uses Pydantic, which depends on pydantic-core (Rust extension via PyO3). PyO3 doesn't support Python subinterpreters used by mod_wsgi by default.

Error:

ImportError: PyO3 modules do not yet support subinterpreters
https://github.com/PyO3/pyo3/issues/576

Solution: Force mod_wsgi to use the main Python interpreter:

WSGIApplicationGroup %{GLOBAL}

Implementation in workflow:

if ! sudo grep -q "WSGIApplicationGroup %{GLOBAL}" /etc/httpd/conf.d/wsgi.conf; then
  sudo sed -i 's/WSGIProcessGroup apache/WSGIProcessGroup apache\nWSGIApplicationGroup %{GLOBAL}/' /etc/httpd/conf.d/wsgi.conf
fi

WSGI Configuration

wsgi.py Entry Point

File: backend/wsgi.py

import sys
import os

# Add the application directory to the Python path
sys.path.insert(0, os.path.dirname(__file__))

# Import the FastAPI ASGI application
from app.main import app

# FastAPI is ASGI, but mod_wsgi expects WSGI
# Use a2wsgi to convert ASGI app to WSGI
from a2wsgi import ASGIMiddleware

application = ASGIMiddleware(app)

Key Points: - FastAPI is an ASGI application - mod_wsgi expects WSGI applications - a2wsgi package provides ASGI-to-WSGI adapter - The WSGI server expects a callable named application

Dependencies

Added to pyproject.toml:

"a2wsgi==1.10.9",  # ASGI-to-WSGI adapter for mod_wsgi on Virtuozzo platform

Apache Configuration

mod_wsgi Settings

File: /etc/httpd/conf.d/wsgi.conf

WSGIDaemonProcess apache user=apache group=apache processes=2 threads=10 \
    python-path="/var/www/webroot/virtenv/lib/python/:/var/www/webroot/" \
    home="/var/www/webroot/"

WSGIScriptAlias / ${WSGI_SCRIPT}
WSGISocketPrefix "/tmp/wsgi"
WSGIPassAuthorization On
WSGIProcessGroup apache
WSGIApplicationGroup %{GLOBAL}  # Critical for PyO3/Pydantic

Environment Variables

File: /usr/local/etc/httpd_vars

WSGI_SCRIPT=/var/www/webroot/ROOT/wsgi.py

Loaded by: /etc/systemd/system/httpd.service

EnvironmentFile=/etc/sysconfig/httpd
EnvironmentFile=-/usr/local/etc/httpd_vars

GitHub Secrets Configuration

Required secrets in repository settings:

Secret Name	Description	Example Value
DEV_SSH_PRIVATE_KEY	SSH private key for apache user	Full OpenSSH private key with headers
DEV_SSH_HOST	Server hostname/IP	15.204.15.210
DEV_SERVER_USER	SSH username	apache

SSH Key Format: The secret should contain a complete OpenSSH private key including the BEGIN and END headers.

Important: Must include newline after the END header.

Testing the Deployment

Manual Deployment Trigger

gh workflow run deploy-backend.yml --ref development

Verify Deployment

# Test API root
curl https://dev-backend.mightybox.site/

# Expected: {"detail":"Not Found"}

# Test Swagger UI
curl https://dev-backend.mightybox.site/docs

# Expected: HTML page with Swagger UI

# Test OpenAPI schema
curl https://dev-backend.mightybox.site/openapi.json

# Expected: JSON with full API schema

Check Logs

# Apache error log
ssh apache@server 'sudo tail -f /var/log/httpd/error_log'

# SSL-specific errors
ssh apache@server 'sudo tail -f /var/log/httpd/ssl-error.log'

Troubleshooting

500 Internal Server Error with No Logs

Symptom: Apache returns 500 errors but no Python tracebacks in logs.

Check: 1. Verify WSGI_SCRIPT environment variable is set 2. Check if PyO3 subinterpreter error exists in SSL logs 3. Ensure WSGIApplicationGroup %{GLOBAL} is in wsgi.conf

ImportError: Cannot import FastAPI modules

Symptom: Python import errors for FastAPI, Pydantic, or other packages.

Check: 1. Virtual environment exists at /var/www/webroot/virtenv/ 2. Packages installed in venv: /var/www/webroot/virtenv/bin/pip list 3. Python path in Apache config points to venv

SSH Connection Failures

Symptom: "Permission denied" or "error in libcrypto"

Check: 1. SSH key format in GitHub secret (proper headers/footers with newlines) 2. Key permissions on server (should be 600) 3. Apache user shell (should be /bin/bash)

SFTP "Received message too long"

Symptom: File transfer fails with message length error.

Check: 1. Remote shell producing output during non-interactive sessions 2. Check apache user's .bashrc for echo statements 3. Wrap output in interactive check: if [ -t 0 ]; then echo "message"; fi

Deployment Workflow Summary

graph TD
    A[Push to development] --> B[GitHub Actions Triggered]
    B --> C[Create deployment package]
    C --> D[Setup SSH with printf]
    D --> E[SFTP upload to /tmp/]
    E --> F[SSH to server]
    F --> G[Backup existing deployment]
    G --> H[Extract new files]
    H --> I[Create/update venv]
    I --> J[Install dependencies in venv]
    J --> K[Set WSGI_SCRIPT variable]
    K --> L[Configure PyO3 compatibility]
    L --> M[Run migrations]
    M --> N[Restart Apache]
    N --> O[Cleanup]

References

• Virtuozzo Python Documentation
• mod_wsgi Documentation
• PyO3 Subinterpreter Issue
• a2wsgi Package

Maintenance

Updating Dependencies

Dependencies are installed from requirements.txt which uses editable install (-e .), pulling from pyproject.toml.

To update dependencies: 1. Modify backend/pyproject.toml 2. Commit and push to development branch 3. Deployment workflow automatically installs updated packages

Database Migrations

Migrations run automatically during deployment if alembic directory exists:

if [ -d "alembic" ]; then
  sudo /var/www/webroot/virtenv/bin/alembic upgrade head
fi

Manual Rollback

Backups are created before each deployment:

/var/www/webroot/backups/ROOT.backup.YYYYMMDD_HHMMSS/

To rollback:

cd /var/www/webroot
sudo rm -rf ROOT
sudo cp -r backups/ROOT.backup.YYYYMMDD_HHMMSS ROOT
sudo systemctl restart httpd

Security Considerations

1. SSH Keys: Private keys stored as GitHub encrypted secrets
2. File Permissions: Deployment sets proper ownership (apache:apache)
3. Virtual Environment: Isolates application dependencies
4. HTTPS: SSL/TLS enforced via Apache configuration
5. Authentication: WSGIPassAuthorization On passes auth headers to application

Performance

Apache Configuration: - Processes: 2 - Threads per process: 10 - Max concurrent requests: 20

Optimization Notes: - Virtual environment cached between deployments - Only changed files deployed (tar.gz package) - Database migrations run incrementally - Apache graceful restart minimizes downtime

---

Last Updated: 2026-01-19 Deployment URL: https://dev-backend.mightybox.site Status: ✅ Production-ready