Simple snowball sync

Sync folders of data onto a snowball

Features:

Start and stop the process at any stage
Compute SRI sha256 hash of all files and stored inside of the snowball in a manifiest.json
Compress small files into a tar.gz so snowball can batch extract when importing into s3
Validate uploaded files against the computed hash

Manifest

Create a list of files to be synced

sss manifest /media/$USER/external-drive

Start the syncing processes

sss sync manifest.json --endpoint 10.254.32.104 --target s3://linz-snowball-a/drive1

Syncing can be stopped and started at any time

Options:

--filter Files under this limit will be compressed into a .tar.gz and uploaded using the snowball batching (Mb) --filter 1 only directly upload files > 1MB
--concurrency Number of files to upload at a single time
--verbose More verbose logging