Observability

When a message doesn't arrive, you need to answer one question fast: where in the two flows did it stop? Good observability makes that a single log query. The rules are the usual ones — structured logs, a correlation ID on every request, sensible levels — with one hard constraint specific to this integration: never log the API key, the signing secret, or message payloads.

Structured logging

Log JSON, not free text. In TypeScript, pino is a good default: JSON in production, pretty-printed in development, with a base that stamps every line with the pid and environment.

// src/config/logger.ts
import pino from 'pino';
import { env } from './env';

export const logger = pino({
  level: env.LOG_LEVEL,
  transport:
    env.NODE_ENV === 'development'
      ? { target: 'pino-pretty', options: { colorize: true } }
      : undefined, // JSON to stdout in production
  base: { pid: process.pid, env: env.NODE_ENV },
});

# src/app/utils/logging.py
import logging

def configure_logging(level: str = "INFO") -> None:
    logging.basicConfig(
        level=getattr(logging, level.upper(), logging.INFO),
        format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    )
    # In production, attach a JSON formatter (e.g. python-json-logger) so
    # structured `extra={...}` fields land as queryable keys.

Write logs to stdout as JSON and let your platform (the container runtime, a log shipper, or your host) collect them. Don't manage log files inside the app — a stateless service shouldn't own local state, including logs. See Scaling & deployment.

Correlation: a request ID on every request

Attach a unique ID to each inbound request, echo it back in an X-Request-Id header, and include it in every log line for that request. That single field lets you reconstruct a whole request from scattered log lines — and lets a client quote it in a support ticket.

// src/middleware/request-logger.ts
export function requestLogger(req, res, next) {
  const start = Date.now();
  const requestId = randomUUID();
  res.setHeader('X-Request-Id', requestId);

  logger.trace(
    { requestId, method: req.method, path: req.path, ip: req.ip },
    'Incoming request',
  );

  res.on('finish', () => {
    const logData = {
      requestId,
      method: req.method,
      path: req.path,
      statusCode: res.statusCode,
      duration: `${Date.now() - start}ms`,
    };
    // Level chosen from the outcome — see below.
    if (res.statusCode >= 500) logger.error(logData, 'Request failed');
    else if (res.statusCode >= 400) logger.warn(logData, 'Request failed');
    else logger.debug(logData, 'Request completed');
  });

  next();
}

# src/app/middleware.py
class RequestIdMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        request_id = request.headers.get("X-Request-ID", str(uuid.uuid4()))
        request.state.request_id = request_id
        response = await call_next(request)
        response.headers["X-Request-ID"] = request_id
        return response

class LoggingMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        start = time.time()
        request_id = getattr(request.state, "request_id", "unknown")
        response = await call_next(request)
        logger.info("Request completed", extra={
            "request_id": request_id,
            "method": request.method,
            "path": request.url.path,
            "status_code": response.status_code,
            "duration_ms": round((time.time() - start) * 1000, 2),
        })
        return response

Honor an inbound X-Request-Id if the caller already set one (as shown in the Python snippet above), so the ID flows across service boundaries. Generate a fresh one only when none is present.

What to log at each level

Pick the level from the outcome, not the code path. The request logger above is the template: 2xx → debug, 4xx → warn, 5xx → error.

Level	Use for	Example
`trace` / `debug`	Normal request lifecycle, health checks.	"Incoming request", "Request completed"
`info`	Business milestones worth keeping.	"Message sent" (with `message_id`), "Webhook endpoint registered"
`warn`	Handled problems / client errors.	`4xx` responses, an unverified webhook rejected `401`
`error`	Unhandled failures needing attention.	`5xx` responses, SDK call threw
`fatal`	Process-ending conditions.	Uncaught exception, unhandled rejection (see `server.ts`)

Logging webhook processing (without secrets)

The receiver is where observability and security collide. Log enough to trace a delivery — and nothing that leaks.

// Safe: identifiers and outcomes only.
logger.info(
  {
    requestId,
    webhookId: req.headers['x-webhook-id'], // config UUID, not per-event
    eventType: req.body.event,              // e.g. "message.delivered"
    messageId: payload.message_id,
    status: payload.message_status,
  },
  'Webhook processed',
);

// NEVER: logger.info(req.headers)  → leaks X-Webhook-Signature + secret material
// NEVER: logger.debug(req.body)    → leaks recipient numbers + message content

logger.info("Webhook processed", extra={
    "request_id": request_id,
    "webhook_id": request.headers.get("x-webhook-id"),
    "event_type": payload["event"],
    "message_id": payload["payload"]["message_id"],
    "status": payload["payload"]["message_status"],
})
# Never log request.headers or the raw body.

X-Webhook-ID is the webhook configuration UUID, not a per-event ID — it's the same across every delivery from one endpoint. It's useful context, but derive your idempotency/dedupe key from the payload (message_id + message_status), not from this header. See Status tracking.

Metrics and tracing

Logs tell you what happened to one request; metrics tell you the shape of the system. Emit at least:

Counters: messages sent, webhooks received, webhooks rejected (bad signature / stale timestamp), send failures.
Histograms: SDK call latency, webhook processing duration, request duration (you already compute it above).
Gauges: if you offload webhook work to a queue, its depth.

This section is optional — skip it until you're actually debugging a cross-service latency problem the logs above can't answer.

For distributed tracing, instrument with OpenTelemetry, a vendor-neutral standard for recording timed operations (spans) and stitching them into one end-to-end trace per request, even across services. Both flows here are natural spans: wrap the messages.send SDK call in a span on the outbound path, and start a span in the receiver keyed by the correlation ID on the inbound path. Carry the same request/correlation ID as baggage — OpenTelemetry's term for a small piece of context attached to a trace and automatically propagated to every span within it — so a log line and a span can be correlated back to each other.

Keep instrumentation in middleware and the service layer, not in controllers. The layered architecture means one span around the service method covers every route that calls it.

Observability

Structured logging

Correlation: a request ID on every request

What to log at each level

Logging webhook processing (without secrets)

Metrics and tracing

Next steps

Scaling & deployment

Security

On this page