pending-streams

    1.0.1 • Public • Published

    Pending streams

    Provides writable and readable streams for pending files.

    • Writable will write to a temporary file and generate the final file only in the very end
    • Readable will be able to grab data written by Writable on the go, reading what has already been written previously, waiting for new data, finishing as soon as writing is over

    Install

    npm install pending-streams --save

    Usage

    var pending = require("pending-streams");
     
    var writable = new pending.Writable(filename, options);
     
    var readable = new pending.Readable(filename, options);

    Sample

    Look at the sample folder for a fully working example:

    cd sample
    node reader.js # will wait for writer to start working 
     
    # from another terminal 
    node writer.js # will start writing, ends after 15 seconds 
    # first "reader.js" should show chunks on the go 
     
    # from another terminal, before 15 seconds 
    node reader.js # look how it showed the previously written data… 
    # and then waits for next chunks 
     
    # from another terminal, after 15 seconds 
    node reader.js # works just like cat then 

    API

    Readable(filename, [options])

    • filename (string, mandatory) is the path to final file
    • options (object, optional):
      • wait (boolean, defaults = false) if true the stream will not fail if no file is found, it will just wait for a writer to start working
      • suffix (string, default = ".part") is the suffix added to filename to generate temporary file
      • highWaterMark is supposedly correctly implemented
      • other options are passed as-is to stream.Readable, you should be careful (I suppose objectMode for example may break everything)

    Writable(filename, [options])

    • filename (string, mandatory) is the path to final file
    • options (object, optional):
      • suffix (string, default = ".part") is the suffix added to filename to generate temporary file
      • other options are passed as-is to stream.Writable, you should be careful

    How it works

    It's quite simple and naive:

    • Writable works with a temporary file until it's finished, then renames it
    • Readable reads from the final file if found (then there is nothing else to do, no pending chunks), else it reads from the temporary file, and as soon as it has finished reading the temporary file it restarts from the number of bytes already consumed

    When should you use it?

    This module was written to implement a direct upload/download tool without keeping streams in memory.

    Writing from a process, reading from another, you may not want to wait for the first process to end, and may not want to implement some specific communication between the processes. Pending streams could do the job.

    TODO

    • Optimizations (I guess there can be a lot, like using fs.read instead of fs streams in PendingReadable)
    • Tests (don't cry, at least there is a sample folder)
    • Work with other things than filesystem
    • Readable: detect when file is deleted or restarted

    Contributions are very welcome: create an issue, or even better a pull request.

    Install

    npm i pending-streams

    DownloadsWeekly Downloads

    7

    Version

    1.0.1

    License

    ISC

    Last publish

    Collaborators

    • naholyr