Production roadmap audit and first hardening pass¶
Audited on 17 November 2026 from 0564c4e2 and read-only production metadata.
Three parallel workstreams covered reliability/recovery, backend/financial
processing, and customer-facing requests. Implementation is on
codex/production-roadmap-hardening; application and database deployments are
separate operations.
Current architecture¶
| Area | Existing implementation | Direction |
|---|---|---|
| API | Go/Fiber, pgx and go-redis (aim-backend/cmd/api/main.go) |
Extend the current API and its validation/error handling |
| Storefront | React/Vite with an Express payment server | Keep gateway callbacks and wallet-credit contracts |
| Portals/mobile | Next.js portals and Expo/React Native | Keep existing TypeScript and native release workflows |
| Edge/deployment | Caddy TLS, Docker Compose, private networks and pinned addresses | Retain the guarded service-by-service deployment model |
| Authoritative data | PostgreSQL 16 | Keep financial state and durable job/outbox records transactional |
| Cache/rate limits | Redis 7 with authentication, AOF and no-eviction policy | Repair the limiter; evaluate any Valkey migration separately |
| Jobs | Go payment maintenance and durable support outbox workers | Extend existing workers when a concrete workflow needs it |
| Images/documents | Content-addressed public files and a separate private document store | Preserve access boundaries in any object-storage migration |
| Monitoring | Sentry, PostHog, Prometheus, Grafana, Loki/Alloy and Kuma | Monitoring changes are already merged in PR #266 |
The generic Node/Prisma/BullMQ/Nginx blueprint needs adapting to this existing implementation. A framework, ORM or queue replacement is not required for the first reliability improvements.
Read-only live observations¶
- Host capacity: 2 CPUs, 7.8 GiB RAM and 39.4 GiB free disk at audit time. This is inventory, not a concurrent-user capacity claim.
- Public authoritative DNS uses
solar.dns-parking.comandlunar.dns-parking.com; the API A record points directly at the existing VPS. Cloudflare DNS/CDN/WAF activation has not been established by this audit. - Latest local scheduled backup observed:
full-20261117-040637, about 12 hours old. It had no Kuma recovery artifact. - The server has neither
deploy/offsite-backup.envnorrcloneinstalled. The documented off-site path is unconfigured. Local backup success does not protect against loss of the VPS disk. - API/website containers already have runtime log rotation. PostgreSQL/Redis do not. Production Compose lacked a shared policy, so future recreates could lose bounds that were set manually.
No private configuration values, database rows or customer data were collected.
Implemented source changes¶
| Workstream | Change | Verification |
|---|---|---|
| Rate limits | Atomic Redis increment/expiry; repair persistent legacy counters; bounded Redis socket/fallback budget with the original shared client unchanged | Targeted race tests; real-Redis expiry, repair and concurrent-client cases in CI |
| Public catalogue | Opaque infrastructure errors, genuine 404 handling, one Sentry event per failed request | Closed-pool HTTP tests plus isolated PostgreSQL visibility fixtures |
| Catalogue clients | Bounded initial catalogue requests, including response-body stalls and cancellation | Focused Node request tests; existing API contracts retained |
| Kuma recovery | Consistent SQLite online snapshot, matching private credentials, checksums, integrity checks and disposable recovery verification | Offline WAL/recovery and failure-path tests; no production snapshot run |
| Off-site transfer | Atomic completed-set receipt, exact inventory and hashes, downloaded-byte verification, read-only dry-run and fail-closed retention | 32 isolated receipt/transfer tests; no production upload |
| Database recovery | Fresh isolated restore instructions with stop-on-error and transactional restore; synthetic CI comparison includes both public and accounting tables |
Existing restore-drill workflow extended; both schema inventories must be nonempty |
| Logs | Shared 10m × 3 json-file rotation policy for all nine production services, with env overrides |
Compose candidate resolved successfully against the server's existing configuration; no containers recreated |
These are source changes. Runtime logging options require scoped container recreation; stateful services need a planned maintenance operation. The backup workflow will use the committed helper on its normal run after release. Read the Kuma recovery guide before recovery.
Local verification¶
- 16 synthetic SQLite backup/recovery tests passed, including committed WAL state, configuration/password pairing, non-overwrite and secret-safe errors.
- 97 targeted catalogue/Shop by Price/product-summary tests passed, including 20 new deadline/cancellation cases. Targeted helper TypeScript and storefront ESLint checks passed.
- Two isolated limiter race tests passed locally; three real-Redis cases require the explicit disposable Redis fixture supplied by CI.
- The public catalogue HTTP/visibility tests and full Go build are delegated to CI because local disk space is constrained. Production PostgreSQL/Redis are not used as test fixtures. Pinned Kuma library compatibility was checked using transient in-memory data.
- Production Compose candidate parsing verified all nine log policies without recreating containers. This is configuration validation, not a runtime rollout.
Existing financial safeguards to retain¶
- Checkout commit locks intent, wallet and stock inside one transaction
(
internal/repository/postgres/commit_from_intent.go). Integer-paise split constraints and idempotency indexes already exist in migration 062. - Journal balance and append-only controls already exist in migration 046; metal ledger immutability exists in migration 075.
- Gateway webhook verification precedes processing
(
internal/handler/razorpay_webhook_handler.go). - Payment maintenance is already started by
cmd/api/main.goand runs recovery, verification, reconciliation and refund work. The older comment inreconciliation_worker.godescribing it as unscheduled is stale. - Real bank payout execution remains deliberately disabled until an approved provider is integrated. A recorded SELL obligation is not a completed bank payout. This pass does not change financial policy or move money.
Remaining work and dependencies¶
- Off-site backups: select an existing private R2/S3 bucket or another
approved destination, configure credentials privately and verify restoration
from that destination. Add explicit freshness/off-site-success alerts. Bucket
details are pending; no cloud account or billing setting was changed.
Cloudflare's connector was subsequently authenticated for
admin@aimstoreorg.com; its only visible account returns R2 error 10042 (not enabled). On 17 November 2026 the owner chose to keep R2 inactive; activation checkout was exited without enabling a subscription. Live off-site backups remain unconfigured. The R2 recovery guide covers private bucket access, authenticated crypt encryption and independent key custody. Local agent skills/MCP setup does not activate R2 or backups. - Restore objectives: agree an RPO/RTO, rehearse recovery of a secured real backup, and then evaluate continuous WAL/PITR. Synthetic row-count equality does not prove live ledger correctness or a real-backup recovery time.
- FCM/customer messaging: implement token registration/revocation, consent and delivery workers with native release verification. Firebase currently supplies authentication. Monitoring's Resend SMTP setup does not activate the separate support-email worker; customer delivery needs explicit setup and verification.
- Notification UX: reconcile
origin/feat/support-notifications-appbefore implementing another notification centre; that branch already has support/order navigation and popup coordination work. - Catalogue scale: design compatible server pagination/search and measure query/index behaviour. Clients currently retrieve the complete catalogue. PostgreSQL search is the first implementation to evaluate before adding a dedicated search service.
- Storage/edge: configure R2/S3 and Cloudflare only with account/domain access and an incremental migration plan. Public media and private customer documents need distinct access policies.
- Security/capacity: add isolated vulnerability scanning and a bounded, opt-in staging load harness. Measure CPU, memory, connection pools and query latency before choosing production resource caps or claiming support for a particular concurrency level. No load test was run against production.