0007 — Grafana LGTM monitoring with provisioned alerts

Status: accepted (2026-07-23)

Context

The platform needs metrics, logs, and traces without a SaaS bill, on the same VPS. The backend already exports Prometheus metrics (fiberprometheus at /metrics) and OTLP-HTTP traces (otelfiber) — it just needs somewhere to send them. Karat's LGTM stack is the working template, but it ships only dashboards: alerting there is whatever was clicked together in the UI, which is invisible to code review and lost on reprovision.

Decision

Run the Karat-shaped stack in monitoring/: Grafana, Prometheus, Loki, Tempo, Alloy, node-exporter, and cAdvisor as mon_* containers on their own aim_mon network (Grafana reachable only through Caddy at monitor.aimstoreorg.com), memory-capped for the small box, datasources and dashboards file-provisioned from git. Improvement over Karat: alert rules are provisioned from git too (monitoring/grafana/provisioning/), so API-down, error-rate, and resource alerts exist from the first deploy and survive a rebuild instead of depending on UI clicks.

Consequences

• Monitoring config — including alerting — is code-reviewed and redeployable (.github/workflows/deploy-monitoring.yml); edits made in the Grafana UI are not persisted to git.
• Tempo actually receives traces here (the backend ships OTLP), unlike Karat where the receiver sat idle.
• Alert-rule changes go through a PR, which is slower than clicking but is the point.
• Retention stays bounded by VPS disk; GRAFANA_ADMIN_PASSWORD is required from /opt/aim-store/monitoring/.env.