004 · CI/CD Pipeline Architecture

Last updated: 2025-11-28

1. Overview

Our CI/CD stack is built entirely on GitHub Actions and follows a trunk-based strategy. All code merges into the development branch (the trunk). Each push to development triggers a suite of workflows that validate, secure, and package both the backend (FastAPI) and frontend (Next.js) services. Deployment artifacts are produced per service so infrastructure can consume them independently.

2. Workflows at a Glance

Workflow	File	Trigger	Purpose
Backend CI	.github/workflows/ci-backend.yml	push/pull_request on development touching backend/**	Python unit tests, Ruff linting, mypy, security checks. Spins up Postgres + Redis services.
Frontend CI	.github/workflows/ci-frontend.yml	push/pull_request on development touching frontend/**	Yarn install cache prime, TypeScript type-check, ESLint, unit tests placeholder, Next.js build artifacts.
Security Scanning	.github/workflows/security.yml	push, pull_request, nightly schedule	TruffleHog secrets scan, dependency audits (pip-audit / yarn audit), CodeQL, container scans, license review, summary.
Deploy Monorepo (Trunk)	.github/workflows/deploy-monorepo.yml	push to development	Detects whether backend and/or frontend changed, builds each service independently, packages artifacts ready for deployment.
Legacy helpers	.github/workflows/backend-ci.yml, .github/workflows/integration-ci.yml	push/pull_request on main	Kept for backward compatibility; they run a slimmed-down backend test suite.

3. Flow Details

3.1 Trigger & Validation

1. Developer merges to development.
2. ci-backend.yml and/or ci-frontend.yml fire, depending on path filters.
3. Both workflows pin runtimes (actions/setup-python@v5.0.0, actions/setup-node@v4.0.2).
4. Dependencies are cached via actions/cache@0057852… (pip, yarn, Next.js cache).
5. Backend coverage is scoped to app.main and app.database.database so the minimum threshold is meaningful while we grow the suite.
6. security.yml runs in parallel to re-scan secrets, dependencies, and container images.

3.2 Packaging (Deploy Monorepo)

The trunk deployment workflow enforces service separation while sharing the repo:

1. detect-changes uses dorny/paths-filter to emit booleans (frontend, backend).
2. build-backend (conditional):
3. Upgrades pip, installs backend/requirements/prod.txt.
4. Produces backend-${SHA}.tar.gz (source minus caches) and optionally builds a Docker image (placeholder).
5. build-frontend (conditional):
6. Runs yarn install --frozen-lockfile in frontend/.
7. Executes yarn build (Next.js 14 app router).
8. Tars .next/, public/, package.json, yarn.lock, next.config.js into frontend-${SHA}.tar.gz.
9. Artifacts are uploaded via actions/upload-artifact@v4.3.1 for downstream consumption.

3.3 Promotion Plan

GitHub Actions Environments (DEV/STAGING/PROD) are planned but not yet wired. The intent is:

• DEV deploy job runs automatically after packaging.
• STAGING deploy job requires a manual approval (“promote” button) and consumes the same artifacts.
• PROD deploy job requires N approvals + optional time delay and references the STAGING-approved artifact.

Implementation tasks for promotions are tracked in docs/development/tasks/ci_cd_backlog.md.

4. Artifact Summary

Artifact	Produced By	Contents	Destination
backend-${SHA}.tar.gz	Deploy Monorepo / build-backend	FastAPI app source, migrations, scripts (sans caches)	DEV/STAGING/PROD backend servers or registry
frontend-${SHA}.tar.gz	Deploy Monorepo / build-frontend	Next.js .next/ build, public/, configs	Static hosting servers / CDN
Security audit JSON	security.yml	backend/pip-audit-results.json, frontend/npm-audit-results.json	Workflow artifacts, compliance storage

5. Extending the Pipeline

1. Hook up server deployments: add SCP/rsync/Docker push steps in deploy-monorepo.yml after artifacts are created, or create environment-specific workflows (deploy-dev.yml, etc.).
2. Enable promotions: configure GitHub Environments with approvals and reference them in the deploy jobs.
3. Observability hooks: integrate Slack/Teams notifications on deploy success/failure via actions/github-script or webhooks.

6. Documentation Ownership Schema (SPEC-001)

All documentation-producing services must own a Backstage entity so TechDocs can map the correct namespace/system/component. The canonical schema for this monorepo is:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: mbpanel
  description: FastAPI + Next.js system powering the MB Panel experience.
  tags:
    - fastapi
    - nextjs
    - docs-platform
spec:
  type: service
  lifecycle: production
  owner: team-devex
  system: mbpanel
  domain: developer-experience

Future repos or packages derived from this mono repo must keep the same domain/system lineage and set their owner to the active on-call group. This schema is referenced by the docs orchestrator CLI and the TechDocs pipelines to guarantee 100% traceability required by SPEC-001.

This document should be updated whenever workflow triggers change, new environments are introduced, new documentation artifacts are added, or the ownership schema evolves.

7. Traceability Matrix (Commit → Artifact → Portal)

Source Path / Domain	CLI Subcommand(s)	Produced Artifact(s)	CI Status Check / Workflow	Published Surface
backend/app/** (FastAPI)	docs-cli extract fastapi	openapi.json, docs/api/**	docs-extract, docs-verify	TechDocs API section + Backstage API catalog
frontend/** (Next.js + shared TS libs)	docs-cli extract typedoc	docs/typedoc/**	docs-extract, docs-verify	TechDocs TypeDoc bundle
asyncapi/** (events)	docs-cli extract asyncapi	docs/events/**, bundled AsyncAPI HTML	docs-extract, docs-verify, adr-lint (if breaking)	TechDocs Events section + Backstage API entity
infra/** (Terraform)	docs-cli extract terraform	docs/infra/** (terraform-docs)	docs-extract, docs-verify	TechDocs Infra section
adr/**	docs-cli verify adr	ADR index + metadata manifest	adr-lint	TechDocs ADR catalog
catalog-info.yaml + metadata	docs-cli catalog:validate	Catalog validation reports	catalog-validate	Backstage Software Catalog
Monorepo root commit (any)	docs-cli render, docs-cli publish	/site TechDocs bundle, provenance records	docs-render, docs-publish, portal-refresh	S3/GCS TechDocs storage + Backstage portal

Every commit must touch at least one row in this matrix; docs-verify enforces the mapping by checking that generated artifacts match the touched paths. The CLI emits provenance linking the commit SHA, artifact checksum, and TechDocs object URI so AI agents can prove the lineage demanded by SPEC-001.