Skip to content

Private R2 backup setup and recovery

Use a private bucket dedicated to backups. Production sets contain database records, private documents, application keys and monitoring credentials. Keep public media in a different bucket. R2 activation, bucket creation and a verified restore are separate checkpoints; installing the Cloudflare connector does not complete any of them.

Account and destination

Inspect the intended Cloudflare account before configuring the VPS. Confirm the bucket's jurisdiction and its S3 endpoint from the dashboard. Disable the managed r2.dev endpoint and all public custom domains on the backup bucket; do not change the application's DNS or public-media access as part of backup setup. Also review any Worker bindings that can serve objects from the bucket.

An API response 10042: Please enable R2 through the Cloudflare Dashboard means R2 activation is still required for that account. Activation accepts a recurring, usage-billed subscription even when the amount due now is zero. Confirm that step with the account owner. Standard storage includes a free usage allowance; it is not a hard spending cap. Check current R2 pricing and monitor the account's billed usage separately from the backup success check.

Create an account-owned R2 S3 token with Object Read & Write, restricted to the backup bucket. Enter its Access Key ID and Secret Access Key privately on the VPS. Do not paste them into chat, GitHub secrets for this workflow, a shell command line, screenshots or source control. The existing workflow reads the server's rclone configuration and needs no cloud credential in GitHub.

Encrypted remote configuration

Install a supported rclone release on the VPS from the official distribution or the OS package manager. Using rclone config in a private terminal:

  1. Create an S3 remote named aim-r2, provider Cloudflare, region auto, the dashboard's S3 endpoint and the bucket-restricted credentials. Use no_check_bucket = true so it does not attempt to create the existing bucket.
  2. Create a crypt remote named aim-backup-crypt, wrapping aim-r2:<private-bucket>/aim-store-production-v1. Enable standard file-name encryption and directory-name encryption. Generate strong independent encryption password and salt through the interactive setup.
  3. Protect rclone's configuration with mode 600 and its parent directory with 700. The stored obscured passwords are recoverable credentials, not protection against access to the configuration file.
  4. Save an independent recovery copy of the remote configuration and encryption passwords in an access-controlled password manager or encrypted offline storage outside the VPS. A backup of the key stored inside the encrypted backup cannot recover the key after VPS loss.

Rclone crypt authenticates encrypted content. Use a new prefix for this format; do not point it at existing plaintext or .enc backups. Retain the separate legacy keys and recovery instructions for those older formats. The script's optional AES-CBC setting remains for compatibility and does not supply content authentication; leave it unset when using the crypt remote.

Create /opt/aim-store/deploy/offsite-backup.env privately, mode 600:

OFFSITE_RCLONE_REMOTE=aim-backup-crypt:
OFFSITE_RETENTION_DAYS=30

Completion and transfer

Release the matching workflow, backup-set.py and offsite-backup.sh together. The workflow fingerprints its committed helpers, takes the shared server lock, captures and checks the artifacts, and publishes backup-set-manifest.json last. This receipt records SHA-256 and byte size for the exact file inventory. Existing unreceipted sets are preserved, but cannot be uploaded by the hardened script. Produce a new successful backup instead of automatically sealing old or failed sets. The receipt detects corruption; it is not a signed provenance record or a transactional snapshot across database and live uploaded files.

The workflow retains 14 completed local sets. Failed, legacy and milestone directories do not count towards that limit and need separate disk-space review. The off-site script fails if the newest validly named set is incomplete; it does not conceal a failed backup by silently falling back to an older one.

For manual runs on the VPS, use the same lock as deployment:

flock -w 300 /tmp/aim-store-deploy.lock \
  bash /opt/aim-store/deploy/offsite-backup.sh --dry-run

Dry-run validates local hashes and reads the remote directory listing; it does not prove write permission. A real run uploads, downloads the remote bytes for comparison, revalidates the local set, then applies remote retention. Verification reads the entire set and adds storage operations and VPS I/O. A failed listing, upload or verification stops retention. The current uploaded set is preserved even if its timestamp is older than the configured retention window.

Recovery rehearsal

On an isolated recovery host, recreate the S3 and crypt remotes from the independent recovery material. Use an empty directory bearing the original backup-set name, not the production backup directory:

umask 077
RECOVERY_ROOT=$(mktemp -d)
chmod 700 "$RECOVERY_ROOT"
SET=full-<UTC-timestamp>
mkdir -m 700 "$RECOVERY_ROOT/$SET"
rclone copy "aim-backup-crypt:$SET" "$RECOVERY_ROOT/$SET"
python3 /path/to/reviewed/deploy/backup-set.py verify \
  --backup-dir "$RECOVERY_ROOT/$SET"

Crypt decrypts through the remote automatically; no .enc loop is needed. Keep recovered artifacts private. Validate archive/gzip integrity and restore PostgreSQL into a fresh isolated database using the deployment recovery procedure. Follow the Kuma recovery guide for its database and matching credentials. Do not start recovered notification or payment workers against production providers during a rehearsal. Record completion time, artifact checks and recovery results without recording their contents.

Off-site protection is complete only after a real upload and this independent download/restore rehearsal succeed. Add backup freshness and off-site-success alerts separately; a green uptime dashboard does not prove recoverability.

References