mcpField
R
r2-glacier-migration

r2-glacier-migration

github.com/pollinationsCloud & DevOps5.3k71 installsUpdated today
Repository

About r2-glacier-migration

A skill for transferring objects from Cloudflare R2 to AWS S3 Glacier Deep Archive. It documents streaming transfer commands, status checks, and checkpoint-based resumption for large bucket migrations.

Key features

  1. 01Stream R2 objects to Glacier Deep Archive
  2. 02Configure batch size and concurrency
  3. 03Inspect transfer status and logs
  4. 04Resume interrupted transfers from checkpoints

Use cases

  1. 01Archive R2 bucket contents in AWS S3
  2. 02Check progress of a long-running migration
  3. 03Continue a transfer after interruption

Install

Copy the command and run it in a terminal.

$npx skills add https://github.com/pollinations/pollinations --skill r2-glacier-migration

After installing, type/r2-glacier-migrationin Claude Code to confirm it loaded.

Bundled filesscripts/package.jsonscripts/r2-to-glacier-streaming.js

Details

From the project README

R2 → AWS Glacier Deep Archive Migration

Migrate ~42.6 TB from Cloudflare R2 to AWS S3 Glacier Deep Archive.

Cost Savings

  • R2: ~$638/month → Glacier: ~$42/month
  • Savings: ~$596/month

Buckets

BucketSizeObjectsStatus
pollinations-text16.96 TB1.86B🔄 In progress
pollinations-images25.64 TB221M⏳ Pending

Quick Commands

Check Status

# Is transfer running?
ssh ninon "screen -ls"

# View recent logs
ssh ninon "tail -50 ~/r2-glacier/transfer-text.log"

# Check checkpoint
ssh ninon "cat /tmp/r2-glacier-pollinations-text-checkpoint.json | jq ."

Attach to Screen (Interactive)

ssh -t ninon "screen -r r2-text"
# Detach: Ctrl+A, D

Resume if Crashed

ssh ninon "screen -dmS r2-text bash -c '~/r2-glacier/run-text-transfer.sh; exec bash'"

File Locations

LocationPathPurpose
Skillscripts/r2-to-glacier-streaming.jsMain transfer script
Skillscripts/package.jsonDependencies
ninon~/r2-glacier/Deployed script + deps
ninon~/r2-glacier/run-text-transfer.shWrapper with credentials
ninon~/r2-glacier/transfer-text.logTransfer log
ninon/tmp/r2-glacier-*-checkpoint.jsonResume checkpoint

Script Usage

# Test (small batch)
node r2-to-glacier-streaming.js -b pollinations-text --batch-size 100 --max-batches 1

# Full transfer
node r2-to-glacier-streaming.js -b pollinations-text --batch-size 50000 --concurrency 30

# Resume after interruption
node r2-to-glacier-streaming.js -b pollinations-text --resume

# Dry run (count only)
node r2-to-glacier-streaming.js -b pollinations-text --dry-run

AWS Setup

  • Bucket: s3://pollinations-archive
  • Region: us-east-1
  • Storage Class: DEEP_ARCHIVE

Verify Archives

aws s3 ls s3://pollinations-archive/ --recursive --human-readable | head -20

Important Notes

  • ⚠️ 180-day minimum on Glacier Deep Archive - don't delete R2 until verified!
  • Some objects fail with "Header overflow" (very large responses) - acceptable loss
  • Text bucket (1.86B objects) takes several days
  • R2 egress is free, AWS ingress is free

Start Images Bucket (After Text Complete)

  1. Update ~/r2-glacier/run-text-transfer.sh on ninon:

    # Change bucket name
node r2-to-glacier-streaming.js -b pollinations-images --batch-size 50000 --concurrency 30

  2. Start new screen:

    ssh ninon "screen -dmS r2-images bash -c '~/r2-glacier/run-images-transfer.sh; exec bash'"

Tracking

  • Issue: #5860
  • PR: #5861

Comments

Markdown supported