Troubleshooting Guide¶

Common authentication issues and solutions.

Quick Diagnosis¶

Symptom	Likely Cause	Quick Fix
401 on every request	Missing/invalid access token	User must re-login
403 on endpoint	Missing permission	Check user's role/permissions
423 on login	Concurrent login pending	Approve from existing session
429 on login	Rate limit exceeded	Wait 5 minutes
CSRF errors	Missing CSRF token	Check cookie/header sync
Redis connection errors	Redis unavailable	Check `REDIS_URL`
Token refresh fails	Session expired	User must re-login

Authentication Issues¶

"Not authenticated" (401)¶

Symptoms: - All requests return 401 - /api/v1/auth/me returns 401

Causes: 1. Access token cookie missing 2. Access token expired 3. JWT_SECRET_KEY mismatch between services 4. Token blacklisted

Diagnosis:

# Check if cookie is present
import requests
response = requests.get("http://localhost:8000/api/v1/auth/me")
print(response.cookies.get("mbpanel_access"))  # Should not be None

# Decode token to check expiry
from app.core.jwt import decode_token
try:
    payload = decode_token(token)
    print(f"Expires at: {payload.exp}")
except TokenValidationError as e:
    print(f"Invalid token: {e}")

Solutions: 1. User must re-login 2. Check JWT_SECRET_KEY is consistent 3. Check Redis for blacklist: redis-cli KEYS "blacklist:*"

"Missing permission" (403)¶

Symptoms: - 403 on specific endpoint - Error message: "Missing permission: X"

Causes: 1. User's role lacks required permission 2. User on wrong team 3. Permission slug mismatch

Diagnosis:

# Check user's permissions via API
curl http://localhost:8000/api/v1/auth/me \
  --cookie "mbpanel_access=<token>"

# Check database
psql -d mbpanel -c "
SELECT u.email, r.name, array_agg(p.slug)
FROM users u
JOIN team_members tm ON u.id = tm.user_id
JOIN roles r ON tm.role_id = r.id
JOIN role_permissions rp ON r.id = rp.role_id
JOIN permissions p ON rp.permission_slug = p.slug
WHERE u.id = <user_id>
GROUP BY u.email, r.name;
"

Solutions: 1. Grant permission to user's role 2. Add user to correct team 3. Check permission slug matches exactly

"CSRF token mismatch" (403)¶

Symptoms: - 403 on /auth/refresh, /auth/logout - Error: "CSRF token mismatch"

Causes: 1. CSRF cookie not sent 2. CSRF header not sent 3. Cookie/header mismatch 4. Expired CSRF token

Diagnosis:

// Browser console
console.log(document.cookie);  // Check for mbpanel_csrf
console.log(getCookie('mbpanel_csrf'));  // Should match header

// Check header in Network tab
// Look for X-CSRF-Token header

Solutions: 1. Ensure cookies are sent: credentials: 'include' 2. Set CSRF header: headers: {'X-CSRF-Token': csrfCookie} 3. Check cookie domain/path settings 4. Re-login to get fresh CSRF token

Symptoms: - 423 on /auth/session-exchange - Response: {"status": "pending", "token": "..."}

Causes: 1. User already has active session 2. New login from different device 3. Existing session hasn't approved/denied

Diagnosis:

# Check active sessions
psql -d mbpanel -c "
SELECT * FROM active_sessions
WHERE user_id = <user_id> AND revoked_at IS NULL;
"

# Check Redis for concurrent login state
redis-cli KEYS "concurrent_login:*"
redis-cli GET "concurrent_login:state:<token>"

Solutions: 1. User approves from existing session (via email or UI) 2. User denies from existing session 3. Logout from existing session first

Token Issues¶

Access Token Expired¶

Symptoms: - 401 errors - Token validation fails with "expired"

Diagnosis:

from app.core.jwt import decode_token, TokenValidationError
try:
    payload = decode_token(access_token)
    print(f"Token expires at: {payload.exp}")
except TokenValidationError as e:
    print(f"Token error: {e}")

Solution: - Client should call /auth/refresh automatically - If refresh also fails, user must re-login

Refresh Token Not Working¶

Symptoms: - /auth/refresh returns 401 or 403 - Session not extending

Causes: 1. Refresh token expired 2. Refresh token blacklisted 3. Session idle timeout exceeded 4. Redis session state missing

Diagnosis:

# Check Redis for refresh session
redis-cli GET "refresh:<jti>"

# Check if blacklisted
redis-cli EXISTS "blacklist:<jti>"

# Check ActiveSession
psql -d mbpanel -c "
SELECT * FROM active_sessions
WHERE refresh_token_jti = '<jti>';
"

Solutions: 1. User must re-login (token expired) 2. Check Redis connection 3. Check REFRESH_IDLE_TIMEOUT_MINUTES setting

Redis Issues¶

Redis Connection Refused¶

Symptoms: - 503 errors on login - "Redis connection refused" in logs

Diagnosis:

# Check Redis is running
docker-compose ps redis

# Test Redis connection
docker-compose exec redis redis-cli PING

# Check REDIS_URL
echo $REDIS_URL

Solutions: 1. Start Redis: docker-compose up -d redis 2. Check REDIS_URL (use service name: redis) 3. Check network connectivity

Redis Key Missing¶

Symptoms: - "Pre-auth token expired" immediately - Refresh fails with session not found

Diagnosis:

# Check for key
redis-cli GET "preauth:<token>"
redis-cli GET "refresh:<jti>"

# Check TTL
redis-cli TTL "preauth:<token>"

Solutions: 1. Token may have expired (TTL too short) 2. Redis may have been flushed (FLUSHALL) 3. Check key prefix matches constants

Database Issues¶

User Not Found¶

Symptoms: - 404 on user operations - "User not found" errors

Diagnosis:

psql -d mbpanel -c "
SELECT * FROM users WHERE email = '<email>';
"

Solutions: 1. Check email is correct 2. Check user is not soft-deleted 3. Check correct database

Team Membership Not Found¶

Symptoms: - 403 on team-scoped operations - "Not a member of team" error

Diagnosis:

psql -d mbpanel -c "
SELECT * FROM team_members
WHERE user_id = <user_id> AND team_id = <team_id>;
"

Solutions: 1. Add user to team via invite 2. Check user is on correct team

Email Issues¶

OTP Not Received¶

Symptoms: - User doesn't receive OTP email - "Email send failed" in logs

Diagnosis:

# Check Postmark credentials
echo $POSTMARK_SERVER_TOKEN
echo $POSTMARK_DEVICE_APPROVAL_TEMPLATE_ID

# Check logs for email errors
grep "email_send_failed" logs/

Solutions: 1. Check Postmark API key is valid 2. Check template ID is correct 3. Check spam folder 4. Check email address is valid

Verification Link Not Working¶

Symptoms: - Clicking email link shows error - "Token expired or invalid"

Causes: 1. Token TTL expired 2. Token already used 3. Wrong base URL in config

Diagnosis:

# Check token in DB
psql -d mbpanel -c "
SELECT * FROM email_verification_tokens
WHERE token = '<token>' AND used_at IS NULL;
"

# Check TTL
echo $EMAIL_VERIFICATION_TOKEN_TTL_HOURS

Solutions: 1. Request new verification email 2. Check BASE_URL setting 3. Check email link is not truncated

Virtuozzo Issues¶

Session Key Refresh Fails¶

Symptoms: - "Virtuozzo session refresh failed" - 503 on session exchange

Causes: 1. Owner-only check failed (user is not Owner) 2. Virtuozzo API unavailable 3. Invalid VZ credentials

Diagnosis:

# Check user is Owner
psql -d mbpanel -c "
SELECT * FROM teams WHERE id = <team_id> AND owner_user_id = <user_id>;
"

# Check VZ credentials exist
psql -d mbpanel -c "
SELECT vz_email, session_key_expires_at
FROM teams WHERE id = <team_id>;
"

# Check VZ API
curl -X POST "$VZ_VIRTUOZZO_SIGNIN_URL" \
  -d "email=$VZ_EMAIL&password=$VZ_PASSWORD"

Solutions: 1. Only Owner can refresh VZ session 2. Check VZ credentials are correct 3. Check VZ API is accessible 4. Update VZ credentials if invalid

Performance Issues¶

Symptoms: - Login takes > 2 seconds - Timeout errors

Diagnosis:

# Check database query performance
psql -d mbpanel -c "EXPLAIN ANALYZE SELECT * FROM users WHERE email = '...';"

# Check Redis latency
redis-cli --latency

# Check network latency
ping redis
ping postgres

Solutions: 1. Add database indexes 2. Check network connectivity 3. Use Redis caching 4. Check for N+1 queries

High Memory Usage¶

Symptoms: - Redis memory growing - OOM errors

Diagnosis:

# Check Redis memory
redis-cli INFO memory

# Check key count
redis-cli DBSIZE

# Check large keys
redis-cli --bigkeys

Solutions: 1. Set appropriate TTLs 2. Clean up expired keys 3. Monitor key patterns

Debugging Tips¶

Enable Debug Logging¶

# backend/app/core/config.py
LOG_LEVEL = "DEBUG"

# Or via environment
LOG_LEVEL=DEBUG uvicorn app.main:app

Check Middleware Order¶

# Verify middleware is registered
# backend/app/core/app_factory.py
print(app.middleware_middleware)  # Should show all middleware

Monitor Redis in Real-Time¶

# Monitor all auth-related keys
redis-cli --scan --pattern "preauth:*" | head -10
redis-cli --scan --pattern "refresh:*" | head -10
redis-cli --scan --pattern "blacklist:*" | head -10

# Monitor pub/sub
redis-cli MONITOR

Trace Token Lifecycle¶

# 1. Login - get pre-auth token
# 2. Check Redis
redis-cli GET "preauth:<token>"

# 3. Session exchange - get JTI
# 4. Check Redis
redis-cli GET "refresh:<jti>"

# 5. Logout - check blacklist
redis-cli EXISTS "blacklist:<jti>"

Quick Start - Setup and debugging
API Reference - Endpoint details
Error Codes - Complete error reference

Troubleshooting Guide¶

Quick Diagnosis¶

Authentication Issues¶

"Not authenticated" (401)¶

"Missing permission" (403)¶

"CSRF token mismatch" (403)¶

"Concurrent login approval pending" (423)¶

Token Issues¶

Access Token Expired¶

Refresh Token Not Working¶

Redis Issues¶

Redis Connection Refused¶

Redis Key Missing¶

Database Issues¶

User Not Found¶

Team Membership Not Found¶

Email Issues¶

OTP Not Received¶

Verification Link Not Working¶

Virtuozzo Issues¶

Session Key Refresh Fails¶

Performance Issues¶

Slow Login¶

High Memory Usage¶

Debugging Tips¶

Enable Debug Logging¶

Check Middleware Order¶

Monitor Redis in Real-Time¶

Trace Token Lifecycle¶

Related Documentation¶