Skip to content

SigNoz Smoke Verify

Preconditions

Start the local stack (MBPanel + open-source SigNoz):

docker compose -f docker-compose.yml -f docker-compose.signoz.yml up -d

SigNoz UI: http://localhost:8080

Smoke checklist (curl)

Set the backend base URL (adjust if you changed BACKEND_PORT):

BASE=http://localhost:8000

If you don't have jq installed, you can either: - omit | jq . entirely, or - use: python3 -m json.tool

1) Confirm HTTP is alive (creates a request span)

curl -sS "$BASE/api/v1/health/ping"
# or:
# curl -sS "$BASE/api/v1/health/ping" | python3 -m json.tool

2) Trigger DB + Redis + RabbitMQ (and broker consume) in one call

This endpoint is disabled in production (ENVIRONMENT=production returns 404).

curl -sS -X POST "$BASE/api/v1/health/observability-smoke"
# or:
# curl -sS -X POST "$BASE/api/v1/health/observability-smoke" | python3 -m json.tool

What this single request exercises: - Redis: INCR + EXPIRE on a non-PII key (mbpanel:observability:smoke) - PostgreSQL: SELECT 1 via SQLAlchemy async session - RabbitMQ: publish an event with routing key team.0.job.0.status - SSE broker: consumes that event and runs its internal fan-out path (even if no SSE client is connected)

Where to look in SigNoz UI

A) Traces

  1. Open SigNoz: http://localhost:8080
  2. Go to Traces → Explorer
  3. Filter Service = mbpanel-api
  4. Set time range to “Last 15 min”

You should see traces whose root span name looks like: - GET /api/v1/health/ping - POST /api/v1/health/observability-smoke

Within the observability-smoke trace, look for child spans like: - redis.incr / redis.expire - db.query - rabbitmq.publish - sse.broker.handle_message (from the broker consumer loop)

Import: FastAPI Errors dashboard (JSON)

This repo includes an importable SigNoz dashboard JSON derived from SigNoz’s official HTTP API Monitoring template.

  • File: docs/development/signoz/dashboards/fastapi-errors-mbpanel-api.json
  • What it shows:
  • HTTP 5xx rate by route
  • HTTP 4xx rate by route
  • Top failing endpoints (5xx)
  • Slowest failing requests (5xx)
  • 5xx latency P90 by route

To import: - SigNoz UI → Dashboards → New Dashboard → Import JSON → upload the file above.

B) Metrics

MBPanel’s metrics export pipeline is enabled, but MBPanel currently doesn’t define custom OTEL metrics instruments. So it’s normal to see no MBPanel-specific application metrics yet until we add meters/counters/histograms.

If you want metrics next, tell me what you want first (HTTP latency histogram, RabbitMQ consumed/dropped counters, Redis latency, etc.) and I’ll add them.

C) Logs

SigNoz’s Logs Explorer supports native OpenTelemetry attributes (service name, environment, trace/span IDs, etc.) per their log ingestion guide.

  1. In the SigNoz UI, open Logs → Explorer.
  2. Filter Service = mbpanel-api and Environment = development.
  3. Trigger the smoke endpoint again (POST /api/v1/health/observability-smoke) and watch new request_completed / request_failed events arrive with matching trace_id / span_id. You can click the trace icon in the log row to jump directly into the correlated trace.
  4. Optional: add severity_text filters (INFO, ERROR) or search for structured keys like team_id, user_id, or correlation_id.

If no data shows up, confirm:

  • OTEL_LOGS_ENABLED=true on the backend container (see docker-compose.signoz.yml), and
  • the SigNoz collector’s logs pipeline is healthy (docker logs signoz-otel-collector should show Exporting log records messages).