Skip to content

CI Cost Optimization Strategy

Problem Statement

GitHub Actions was running 4 parallel jobs with complex test matrices, mutation testing, and extensive integration tests. This resulted in: - 12-15 minutes of runner time per push - High monthly GitHub Actions costs - Unnecessary complexity for simple changes

Solution: Shift-Left Testing

Philosophy: Run comprehensive tests locally in pre-commit hooks. Only run critical security tests in CI.

What Runs Where

GitHub Actions (CI) - ~2-3 minutes

Only critical security tests: - tests/unit/core/test_jwt_tokens.py - JWT token security - tests/unit/core/test_security_hashing.py - Password hashing - tests/unit/core/test_external_http.py - External API security - tests/unit/core/test_security_events.py - Security event logging - tests/unit/core/test_rate_limit.py - Rate limiting

Coverage requirement: 95% on core security modules

Pre-Commit Hook (Local) - ~2-4 minutes

Comprehensive test suite: 1. Fast Checks (<10s) - Ruff linting with auto-fix - Import sorting

  1. Core Security Tests (<30s)

  2. Same tests as CI (ensures local/CI parity)

  3. Full Unit Tests (<60s)

  4. All unit tests across all modules

  5. Coverage reporting

  6. Integration Tests (<90s)

  7. API integration tests
  8. External service mocks

  9. E2E workflows

  10. Mutation Testing (optional, <120s)

  11. Security-critical modules only
  12. Can be skipped with SKIP_MUTATION=1

Cost Savings

Before

  • Jobs: 4 parallel (Lint, Unit Tests, Integration Tests, Coverage Gate, Mutation)
  • Runtime: 12-15 minutes
  • Monthly cost (100 pushes): ~20-25 hours of runner time

After

  • Jobs: 1 (Core Security Tests only)
  • Runtime: 2-3 minutes
  • Monthly cost (100 pushes): ~3-5 hours of runner time

Savings: ~80% reduction in GitHub Actions costs

Usage

For Developers

# Normal commit - runs comprehensive tests automatically
git commit -m "feat: add new feature"

# Emergency commit - skip tests (use sparingly!)
git commit --no-verify -m "hotfix: critical production issue"

# Run tests manually
./tools/hooks/pre-commit-comprehensive.sh

# Skip mutation testing for speed
SKIP_MUTATION=1 git commit -m "feat: quick iteration"

For CI/CD

The simplified CI workflow runs automatically on push to dev-* branches. It: 1. Runs only core security tests 2. Fails fast on security issues 3. Provides clear feedback in <3 minutes

Migration Guide

Old Workflow (Deprecated)

# .github/workflows/ci-backend.yml (483 lines)
- Lint job
- Test matrix (unit + integration)
- Coverage gate
- Mutation testing

New Workflow (Simplified)

# .github/workflows/ci-backend-simplified.yml (130 lines)
- Single job: Core security tests only

Monitoring

Success Metrics

  • ✅ CI runtime: <3 minutes (target: 2-3 min)
  • ✅ Pre-commit runtime: <4 minutes (target: 2-4 min)
  • ✅ Core security coverage: >95%
  • ✅ Developer satisfaction: Fast feedback loop

Failure Handling

If CI fails: - Security test failure = BLOCK MERGE - Must be fixed immediately

If pre-commit fails: - Developer can skip with --no-verify for emergencies - Must run manually ASAP: ./tools/hooks/pre-commit-comprehensive.sh

Best Practices

  1. Always run pre-commit tests before pushing
  2. Catches issues early
  3. Prevents CI failures

  4. Saves time for everyone

  5. Don't skip tests unless emergency

  6. Use --no-verify only for hotfixes

  7. Run tests manually immediately after

  8. Keep tests fast

  9. Mark slow tests with @pytest.mark.slow
  10. Mark load tests with @pytest.mark.load

  11. These are excluded from pre-commit

  12. Monitor CI costs

  13. Review GitHub Actions usage monthly
  14. Optimize if runtime creeps up

Rollback Plan

If the simplified CI causes issues:

# Revert to old workflow
git revert <commit-hash>

# Or temporarily use old workflow
# Edit .github/workflows/auto-merge-to-development.yml
uses: ./.github/workflows/ci-backend.yml  # Old workflow

References