backer
wip distributed backup / file mirroring tool
Idea
Mirror your files between all your computers and servers, so when a harddrive breaks or a computer gets stolen you don't lose anything.
Like bittorrent sync, but open source and in node.js, so it runs on SmartOS too.
Like dropbox, but stores your data only on your machines. No third party - except possibly internet providers - involved.
Associated projects
Possible features
- revisions
- webgui
- public sharing
- collaborative live editing? maybe just using the file system limits too much
Data structure
The filesystem is represented as a merkle tree, so backer can efficiently figure out what files changed.
Replication
On connection sync using merkle trees, then maybe use scuttlebutt and only send file diffs.
Problem: efficient tree diffing
Encryption
Since backer is only used for transferring data and never stores it on any server/computer but the user's, only transfor needs to be encrypted.
Encrypting your own harddrive is out of scope and there's plenty of tools that do this, one popular being truecrypt.
- Maybe public/private key cryptography?
JS api
For maximum composability the transport part of the api will just be a duplex stream that is to be piped to another backer's duplex stream. This way it works over tcp, websockets, in memory, or over any another network or streaming interface.
var a = ;var b = ;a;
Over tcp it would look like this:
// computer Avar backer = ; var back = ;net; // computer Bvar backer = ;var reconnect = ; var back = ;;
There will be events emitted on the backer instance, which can then for example be fed to a web frontend.
Resources
- scuttlebutt
- btsync
- dht
- delta encoding
- merkle tree
- nodedrop (failed but tried the same)
- Build your own private, encrypted, open-source Dropbox-esque sync folder
- unison research papers (11)
Possible dependencies
- dominictarr/merkle
- roderwang/node-webkit
- shama/nodewebkit (npm installer)
Collaborators
There's a lot still to be figured out, so if you either are a mad scientist that juggles chainsaws while writing distributed systems, or want to be come one, this is the right place for you!
Plus, there's going to be need for designs, blog posts, a public facing website, etc.!
License
MIT