Skip to content

Production roadmap audit and first hardening pass

Audited on 17 November 2026 from 0564c4e2 and read-only production metadata. Three parallel workstreams covered reliability/recovery, backend/financial processing, and customer-facing requests. Implementation is on codex/production-roadmap-hardening; application and database deployments are separate operations.

Current architecture

Area Existing implementation Direction
API Go/Fiber, pgx and go-redis (aim-backend/cmd/api/main.go) Extend the current API and its validation/error handling
Storefront React/Vite with an Express payment server Keep gateway callbacks and wallet-credit contracts
Portals/mobile Next.js portals and Expo/React Native Keep existing TypeScript and native release workflows
Edge/deployment Caddy TLS, Docker Compose, private networks and pinned addresses Retain the guarded service-by-service deployment model
Authoritative data PostgreSQL 16 Keep financial state and durable job/outbox records transactional
Cache/rate limits Redis 7 with authentication, AOF and no-eviction policy Repair the limiter; evaluate any Valkey migration separately
Jobs Go payment maintenance and durable support outbox workers Extend existing workers when a concrete workflow needs it
Images/documents Content-addressed public files and a separate private document store Preserve access boundaries in any object-storage migration
Monitoring Sentry, PostHog, Prometheus, Grafana, Loki/Alloy and Kuma Monitoring changes are already merged in PR #266

The generic Node/Prisma/BullMQ/Nginx blueprint needs adapting to this existing implementation. A framework, ORM or queue replacement is not required for the first reliability improvements.

Read-only live observations

  • Host capacity: 2 CPUs, 7.8 GiB RAM and 39.4 GiB free disk at audit time. This is inventory, not a concurrent-user capacity claim.
  • Public authoritative DNS uses solar.dns-parking.com and lunar.dns-parking.com; the API A record points directly at the existing VPS. Cloudflare DNS/CDN/WAF activation has not been established by this audit.
  • Latest local scheduled backup observed: full-20261117-040637, about 12 hours old. It had no Kuma recovery artifact.
  • The server has neither deploy/offsite-backup.env nor rclone installed. The documented off-site path is unconfigured. Local backup success does not protect against loss of the VPS disk.
  • API/website containers already have runtime log rotation. PostgreSQL/Redis do not. Production Compose lacked a shared policy, so future recreates could lose bounds that were set manually.

No private configuration values, database rows or customer data were collected.

Implemented source changes

Workstream Change Verification
Rate limits Atomic Redis increment/expiry; repair persistent legacy counters; bounded Redis socket/fallback budget with the original shared client unchanged Targeted race tests; real-Redis expiry, repair and concurrent-client cases in CI
Public catalogue Opaque infrastructure errors, genuine 404 handling, one Sentry event per failed request Closed-pool HTTP tests plus isolated PostgreSQL visibility fixtures
Catalogue clients Bounded initial catalogue requests, including response-body stalls and cancellation Focused Node request tests; existing API contracts retained
Kuma recovery Consistent SQLite online snapshot, matching private credentials, checksums, integrity checks and disposable recovery verification Offline WAL/recovery and failure-path tests; no production snapshot run
Off-site transfer Atomic completed-set receipt, exact inventory and hashes, downloaded-byte verification, read-only dry-run and fail-closed retention 32 isolated receipt/transfer tests; no production upload
Database recovery Fresh isolated restore instructions with stop-on-error and transactional restore; synthetic CI comparison includes both public and accounting tables Existing restore-drill workflow extended; both schema inventories must be nonempty
Logs Shared 10m × 3 json-file rotation policy for all nine production services, with env overrides Compose candidate resolved successfully against the server's existing configuration; no containers recreated

These are source changes. Runtime logging options require scoped container recreation; stateful services need a planned maintenance operation. The backup workflow will use the committed helper on its normal run after release. Read the Kuma recovery guide before recovery.

Local verification

  • 16 synthetic SQLite backup/recovery tests passed, including committed WAL state, configuration/password pairing, non-overwrite and secret-safe errors.
  • 97 targeted catalogue/Shop by Price/product-summary tests passed, including 20 new deadline/cancellation cases. Targeted helper TypeScript and storefront ESLint checks passed.
  • Two isolated limiter race tests passed locally; three real-Redis cases require the explicit disposable Redis fixture supplied by CI.
  • The public catalogue HTTP/visibility tests and full Go build are delegated to CI because local disk space is constrained. Production PostgreSQL/Redis are not used as test fixtures. Pinned Kuma library compatibility was checked using transient in-memory data.
  • Production Compose candidate parsing verified all nine log policies without recreating containers. This is configuration validation, not a runtime rollout.

Existing financial safeguards to retain

  • Checkout commit locks intent, wallet and stock inside one transaction (internal/repository/postgres/commit_from_intent.go). Integer-paise split constraints and idempotency indexes already exist in migration 062.
  • Journal balance and append-only controls already exist in migration 046; metal ledger immutability exists in migration 075.
  • Gateway webhook verification precedes processing (internal/handler/razorpay_webhook_handler.go).
  • Payment maintenance is already started by cmd/api/main.go and runs recovery, verification, reconciliation and refund work. The older comment in reconciliation_worker.go describing it as unscheduled is stale.
  • Real bank payout execution remains deliberately disabled until an approved provider is integrated. A recorded SELL obligation is not a completed bank payout. This pass does not change financial policy or move money.

Remaining work and dependencies

  1. Off-site backups: select an existing private R2/S3 bucket or another approved destination, configure credentials privately and verify restoration from that destination. Add explicit freshness/off-site-success alerts. Bucket details are pending; no cloud account or billing setting was changed. Cloudflare's connector was subsequently authenticated for admin@aimstoreorg.com; its only visible account returns R2 error 10042 (not enabled). On 17 November 2026 the owner chose to keep R2 inactive; activation checkout was exited without enabling a subscription. Live off-site backups remain unconfigured. The R2 recovery guide covers private bucket access, authenticated crypt encryption and independent key custody. Local agent skills/MCP setup does not activate R2 or backups.
  2. Restore objectives: agree an RPO/RTO, rehearse recovery of a secured real backup, and then evaluate continuous WAL/PITR. Synthetic row-count equality does not prove live ledger correctness or a real-backup recovery time.
  3. FCM/customer messaging: implement token registration/revocation, consent and delivery workers with native release verification. Firebase currently supplies authentication. Monitoring's Resend SMTP setup does not activate the separate support-email worker; customer delivery needs explicit setup and verification.
  4. Notification UX: reconcile origin/feat/support-notifications-app before implementing another notification centre; that branch already has support/order navigation and popup coordination work.
  5. Catalogue scale: design compatible server pagination/search and measure query/index behaviour. Clients currently retrieve the complete catalogue. PostgreSQL search is the first implementation to evaluate before adding a dedicated search service.
  6. Storage/edge: configure R2/S3 and Cloudflare only with account/domain access and an incremental migration plan. Public media and private customer documents need distinct access policies.
  7. Security/capacity: add isolated vulnerability scanning and a bounded, opt-in staging load harness. Measure CPU, memory, connection pools and query latency before choosing production resource caps or claiming support for a particular concurrency level. No load test was run against production.

References