push-text

This script adds two simple command line features:

• push-text -- send the current folder up to your encrypted s3 bucket, recursively.
• pull-text -- bring all the files from your s3 bucket down to local.

If you like taking notes in plain text or markdown and want the security of an encrypted s3 location as a backup, this might be a good solution for you.

Demo

Here's an example of using these commands with the _example folder provided in this repo.

Getting Started

1. Install command line tool: npm i -g push-text
2. Create an s3 Bucket and save your credentials (make sure to enable encryption)
3. Navigate to the folder on your local machine that contains all your text notes
4. Run push-text init to set up the configuration
5. Add your aws credentials to the .aws.json file

Using the Tool

1. Run push-text to persist your text files to your encrypted s3 bucket
2. Run pull-text to update text files in this folder from s3. Note: that this will overwrite changes in existing local files from the remote.

Features

• Recursive. Preserves folder structure and persists your whole organized tree of text files.
• Content Hashing. Save bandwidth and time on identical files when saving and fetching. (Compares local md5 against s3's ETag)
• Small Minded. Ignores large files, just in case you have big log text files or other junk in the folder you want to back up.

Example .aws.json file:

AWS credentials are handled in a configuration file called .aws.json. You can run push-text init to autogenerate one.

This file lives in the folder you want to syncronize and looks like this:

{
    "accessKeyId": "MY-KEY",
    "secretAccessKey": "MY-SECRET",
    "region": "MY-REGION",
    "Bucket": "MY-BUCKET"
}

Yes, the capitalization is correct on Bucket; this is to match the config API provided by AWS s3.

Explanation, Limitations

The "sync" algorithm is basically a "force save" followed by a "force get":

1. save all local files to the remote.

• If the content hash on remote is identical, skip
• If the remote exists, overwrite
• if the remote is missing, write

2. get all remote files to local.

• If the content hash on remote is identical, skip
• If the local exists, overwrite
• If the local is missing, write

The result is that if you always run them in a pair you'll have a pretty reasonable syncing behavior, but it can fail pretty spectacularly if...

• an out of date client runs sync (overwrites with old data)
• You run get before save (get nothing, then write it all back)

So, this is better suited to a single-author / many-viewer model.

Other notes:

Be careful about running pull-text. If you have more recent changes locally you'll overwrite them. If you have brand new files those will be ok, but it has to make smart chocies about merge and I haven't solved that problem yet.

Until I build a real 2-way-sync algorithm, this thing tends to replicate files you might have wanted gone. If you create a file, run push-text, then later delete it; it's going to stay in the remote. pull-text will bring it back. I've opted to hang on to old stuff because text is small and I prefer that default, but you can always log into s3 and clear things out.

Interesting / Inspiring / Similar

• Plain Text Project
• imdone.io and https://github.com/imdone/imdone-core
• QOwnNotes

Developing

To develop locally, first navigate to this project folder. Then run:

npm link

This makes your local code override the global module's behavior.

You can then make changes and test using push-text and pull-text as usual.

TODO:

Unidirectional Backup + View

• [ ] provide push-text init or similar, which would autocreate a starter .aws.json file in the current directory so you know the right key names to start with.
• [ ] Enable a way to run push-text automatically (as a watcher) on configured folders all the time
• [ ] Build a mobile + web viewer for an s3 bucket full of text files.

web viewer spec:

mvp
o Specify aws config in UI, and it's stored so I don't have to type it every time
o Navigate files and folders
o view `txt` and `md`

nice to have
o save scroll position for each file
o local annotations (check / highlight / cross out a line of text)

Bidirectional Backup / Sync

• [ ] Add a push-text status (or info or dry-run) command, so you can see what would happen if you were to run the command. Can extend with syncing behavior later.
• [ ] Come up with a text syncing solution that will work for multiple authors and is unlikely to lose data (helpful: most edits are additive, mass removals are suspect.)
• [ ] When it says "uploading these n files", be clear about which ones are brand new, and which are overwriting the remote

Neat

• [ ] If the current system has git, use it as a backup solution (eg: so the user can recover from mistakes in a sync or undo changes)
• [ ] Save the last k states of the tree of text files to support "oops" undo. (especially useful if you pull-text on an old state and overwrite your latest changes.) S3 versioning might be leveragable, except I want to undo a delete of local-only state.
• [ ] calculate % differences (eg: 5% different. 7% added in remote, 2% added by local, etc)
• [ ] re-write imperitive chained callback code with async functions (but wait for node 8 to be popular enough that this makes sense)

Isolated Features / Extensibility (easy pull request ideas)

• [ ] Add a --help command to show how to use the different features
• [ ] Add a post-install guide: "navigate to the folder that holds your text files, and type such-and-so..."
• [ ] Add support for small files with no extension, also support .mdown
• [ ] Add a way for the terminal to launch or link to the bucket creation process
• [ ] Add a step-by-step bucket install setup
• [ ] Add pull-text --force if you want to overwrite all your local files on purpose
• [ ] Should I look at mimetype instead of filename to determine file type? (eg: word docs with no extension)
• [x] Support init command to make it easy and obvious to configure this tool
• [x] Configurable whitelist for file extensions
• [x] Configurable max file size
• [ ] Make Bucket command-line configurable (requires refactor to change init sequence)
• [ ] If count of files in array (during upload or download) is larger than k, show count instead of individual files
• [ ] Add a verbose mode that shows "In Sync" output, otherwise don't show that.
• [ ] Don't upload empty folders (probably they had things other than text files in them, so not useful)
• [ ] Support all cfg properties on the command line as well, so you could, for example, specify a different bucket like push-text --Bucket foo
• [ ] Change all callback APIs to promises
• [ ] Make console output beautiful. Start with spacing and column alignment. Limit hashes to 8 chars.
• [ ] Add colors to console where appropriate (if the dependency tree effect isn't bad)

Confidence

• [ ] Unit test purely functional code
• [ ] Integration tests with fs. (tree of text files, doesn't include non-text, excludes > size k)
• [ ] Integration tests with s3? (use throwaway account info, depend on network, get back to repeatable state afterwards)