Sync files on one S3 bucket to another S3 bucket with support for large files, session tokens, and single file operations.

• Directory and Single File Sync: Sync entire directories or individual files between S3 buckets
• Large File Support: Automatic chunked copying for files over 500MB using AWS SDK multipart upload
• Session Token Support: Compatible with temporary AWS credentials including session tokens
• Cross-Region Support: Sync between different AWS regions and S3-compatible endpoints
• Configurable Staging: Customizable local staging directory for operations

• AWS S3 CLI
• NodeJS 18+

Options can be provided either as command line options or environment variables.

% npx @eyevinn/s3-sync \
  --source-access-key=<source-access-key> \
  --source-secret-key=<source-secret-key> \
  --source-region=<source-region> \
  --dest-endpoint=<dest-endpoint> \
  --dest-access-key=<dest-access-key> \
  --dest-secret-key=<dest-secret-key> \
  s3://source/files/ s3://dest/files/

% npx @eyevinn/s3-sync \
  --single-file \
  --source-access-key=<source-access-key> \
  --source-secret-key=<source-secret-key> \
  s3://source/large-file.mp4 s3://dest/large-file.mp4

% npx @eyevinn/s3-sync \
  --source-access-key=<source-access-key> \
  --source-secret-key=<source-secret-key> \
  --source-session-token=<source-session-token> \
  --dest-access-key=<dest-access-key> \
  --dest-secret-key=<dest-secret-key> \
  --dest-session-token=<dest-session-token> \
  s3://source/files/ s3://dest/files/

Store credentials in a file called .env:

SOURCE_ACCESS_KEY=<source-access-key>
SOURCE_SECRET_KEY=<source-secret-key>
SOURCE_SESSION_TOKEN=<source-session-token>  # Optional
SOURCE_REGION=<source-region>
SOURCE_ENDPOINT=<source-endpoint>             # Optional
DEST_ACCESS_KEY=<dest-access-key>
DEST_SECRET_KEY=<dest-secret-key>
DEST_SESSION_TOKEN=<dest-session-token>       # Optional
DEST_REGION=<dest-region>
DEST_ENDPOINT=<dest-endpoint>                 # Optional
STAGING_DIR=/tmp/data                         # Optional, default: /tmp/data
AWS_CLI_PATH=/usr/local/bin/aws               # Optional, default: aws

% set -a ; source .env ; set +a
% npx @eyevinn/s3-sync s3://source/files s3://dest/files/

Files larger than 500MB are automatically processed using chunked copying with the AWS SDK for improved performance and reliability. This feature:

• Uses multipart upload for better error recovery
• Processes files in 100MB chunks
• Provides progress logging for large transfers
• Automatically falls back to traditional copy for smaller files

Also available as an open web service in Eyevinn Open Source Cloud. This enables you to synchronize two S3 buckets without having to go through your local computer. Read this documentation for usage instructions.

% npm install
% set -a ; source .env ; set +a
% npm start -- s3://source/files s3://dest/files/

See CONTRIBUTING

This project is licensed under the MIT License, see LICENSE.

Join our community on Slack where you can post any questions regarding any of our open source projects. Eyevinn's consulting business can also offer you:

• Further development of this component
• Customization and integration of this component into your platform
• Support and maintenance agreement

Contact [email protected] if you are interested.

Eyevinn Technology is an independent consultant firm specialized in video and streaming. Independent in a way that we are not commercially tied to any platform or technology vendor. As our way to innovate and push the industry forward we develop proof-of-concepts and tools. The things we learn and the code we write we share with the industry in blogs and by open sourcing the code we have written.

Want to know more about Eyevinn and how it is to work here. Contact us at [email protected]!