Never Patronize Mothers

    blob-store-replication-stream

    1.3.3 • Public • Published

    blob-store-replication-stream

    BuildStatus

    Replicate two abstract-blob-store compatible stores together (with caveats).

    It would be very useful if any two blob stores could be replicated to each other over a simple duplex stream.

    That's just what this module does, but with one large caveat: the abstract-blob-store interface doesn't provide a method for getting a list of the names of all blobs in the store. Without this, a full sync between blob stores isn't possible.

    The caveat: any blob store you wish to use for replication must have a ._list(cb) function monkey patched onto it, which should return an array of keys (strings) in the callback cb. This will generally mean delving into the innards of each blob store you wish to support and figuring out how to add this functionality.

    Usage

    var createReplicationStream = require('blob-store-replication-stream')
     
    var StoreFs = require('fs-blob-store')
    StoreFs.prototype._list = function (cb) {
      require('fs').readdir(this.path, cb)
    }
     
    var StoreMem = require('abstract-blob-store')
    StoreMem.prototype._list = function (cb) {
      process.nextTick(cb, null, Object.keys(this.data))
    }
     
    var fs = StoreFs('./fs')
    var mem = StoreMem()
     
    var ws = fs.createWriteStream('foo', function (err, metadata) {
      console.log('fs key', metadata.key)
      var ws = mem.createWriteStream('bar', function (err, metadata) {
        console.log('mem key', metadata.key)
        doReplicate()
      })
      ws.end('yo mem world')
    })
    ws.end('hello fs world')
     
    function doReplicate () {
      var r1 = createReplicationStream(fs)
      var r2 = createReplicationStream(mem)
     
      r1.pipe(r2).pipe(r1)
     
      r1.on('end', onDone)
      r2.on('end', onDone)
     
      var pending = 2
      function onDone () {
        if (!--pending) {
          fs.createReadStream('bar').pipe(process.stdout)
          mem.createReadStream('foo').pipe(process.stdout)
        }
      }
    }

    outputs

    fs key foo
    mem key bar
    yo mem world
    hello fs world
    

    API

    var createReplicationStream = require('blob-store-replication-stream')

    var stream = createReplicationStream(store[, opts])

    Creates the duplex stream stream from the abstract-blob-store instance store.

    Pipe this into another replication stream, and have that other replication stream also pipe into this, to create a full duplex channel of communication.

    This function will throw an Error if no store._list() function is present.

    Valid opts include:

    • mode:
      • 'sync': (default) entries will sent and fetched. the two stores will contain the same blobs post-replication
      • 'pull': entries will only be fetched, not sent
      • 'push': entries will only be sent, not fetched
    • filter: a function that takes a filename and returns true if it should be synced to the other side, or false if it shouldn't

    In addition, stream will emit 'progress' events with parameters filesSent and filesTotal for tracking sync progress.

    Install

    With npm installed, run

    $ npm install blob-store-replication-stream
    

    License

    ISC

    Keywords

    none

    Install

    npm i blob-store-replication-stream

    DownloadsWeekly Downloads

    23

    Version

    1.3.3

    License

    ISC

    Unpacked Size

    29.1 kB

    Total Files

    6

    Last publish

    Collaborators

    • andrew_chou
    • substack
    • karissa
    • digidem-admin
    • gmaclennan
    • noffle