Naked Panda Meditations

    fast-zlib
    TypeScript icon, indicating that this package has built-in type declarations

    2.0.1 • Public • Published

    fast-zlib

    A simple zlib wrapper for Node.js that enables synchronous shared context compression.

    Shared Context / Context Takeover / Sliding Window

    When working with data streams, zlib makes use of a "sliding window", in which a dictionary of common patterns is built and kept on both ends of the stream. When the compressor encounters a common pattern, it replaces it with a reference to the decompressor's dictionary and thus doesnt need to send that piece of data at all. This referenced pattern will simply be reconstructed by the decompressor using data from its dictionary. This is also known as context sharing or context takeover.

    Node's native zlib module does not offer a public API to perform this task synchronously and instead offers an asynchronous API using transform streams. Because zlib itself is synchronous, artificially defering it can cause issues such as high latency and memory fragmentation, especially when processing a high volume of small chunks of data like websocket messages. (see ws#1369 and node#8871)

    Node does however include all the necessary tools and functionality in its private and undocumented APIs, for instance its _processChunk() method, which this package makes use of to provide an easy and fast way to synchronously process chunks of data without losing the zlib context.

    Usage

    Fast-zlib exports a modified synchronous version of all zlib classes:

    const zlib = require("fast-zlib");
    
    let inflate = new zlib.Inflate();
    let deflate = new zlib.Deflate();
    let inflateRaw = new zlib.InflateRaw();
    let deflateRaw = new zlib.DeflateRaw();;
    let gzip = new zlib.Gzip();
    let unzip = new zlib.Unzip();
    let gunzip = new zlib.Gunzip();
    let brotli = new zlib.BrotliCompress();
    let debrotli = new zlib.BrotliDecompress();

    ES6 imports and typescript users can also use the following:

    import zlib from "fast-zlib";
    let deflate = new zlib.Deflate();
    import { Deflate } from "fast-zlib";
    let deflate = new Deflate();

    Each fast-zlib class is a wrapper around the original zlib class and looks like this:

    constructor(options?)

    Create a new instance of a fast-zlib class.

    • options - An optional object of zlib or brotli options as per node's zlib documentation.

    .process(data, flag?)

    Process a chunk of data.

    • data - A Buffer of data to be compressed or decompressed. Non-Buffers will be internally converted to Buffer.
    • flag - An optional flush flag to override the default flag.
    • => Buffer - A Buffer of processed data.

    .close()

    Close the zlib handler and shut down the instance.

    • => void

    .instance

    Access the underlying zlib instance for advanced usage.

    Examples

    Usage is very simple, compress a chunk of data and decompress it elsewhere. The instance keeps track of its compression state and sliding window contexts.

    let zlib = require("fast-zlib");
    
    let deflate = new zlib.Deflate(); // create a deflator
    let inflate = new zlib.Inflate(); // create an inflator
    
    let data = "123456789";
    
    let chunk1 = deflate.process(data);
    // Buffer(17) [120, 156, 50, 52, 50, 54, 49, 53, 51, 183, 176, 4, 0, 0, 0, 255, 255]
    // the first chunk of data is fully processed
    
    let chunk2 = deflate.process(data);
    // Buffer(9) [50, 132, 49, 0, 0, 0, 0, 255, 255]
    // reusable patterns from previous compressions are referenced
    
    let chunk3 = deflate.process(data);
    // Buffer(8) [130, 51, 0, 0, 0, 0, 255, 255]
    // and the context continues to adapt to all subsequent chunks
    
    inflate.process(chunk1).toString(); // "123456789"
    inflate.process(chunk2).toString(); // "123456789"
    inflate.process(chunk3).toString(); // "123456789"

    Decompression must be done in exactly the same order as compression because chunks sequentially complement each other. Attempting to decode a chunk out of order may throw an error and reset the decompressor so it has to either restart from the beginning or you will have to destroy both and create a new pair.

    let chunk1 = deflate.process(data);
    let chunk2 = deflate.process(data);
    
    inflate.process(chunk2); // error
    inflate.process(chunk1); // works
    inflate.process(chunk2); // works

    Each zlib class can be passed an options object as per zlib's documentation.

    let deflateRaw = new zlib.DeflateRaw({
        chunkSize: 128 * 1024,
        level: 8
    });
    let inflateRaw = new zlib.InflateRaw({
        chunkSize: 64 * 1024
    });

    This library uses Z_SYNC_FLUSH as the default flush flag in order to return data immediately. For more control over the compression process, flush flags can be set as an option and also passed directly to the process function.

    let deflate = new zlib.Deflate({
        flush: zlib.constants.Z_NO_FLUSH // set default flag to Z_NO_FLUSH
    });
    let inflate = new zlib.Inflate();
    
    deflate.process("123"); // add data
    deflate.process("456");
    deflate.process("789");
    
    // process all data added so far by passing Z_SYNC_FLUSH
    let data = deflate.process("hij", zlib.constants.Z_SYNC_FLUSH);
    
    inflate.process(data).toString(); // 123456789hij

    Other flush flags are also available and can be used to achieve fine control over the process. Not all classes support the same flags and there might be differences in behavior between them. For example brotli uses BROTLI_OPERATION_PROCESS and BROTLI_OPERATION_FLUSH instead of Z_NO_FLUSH and Z_SYNC_FLUSH. Check zlib's documentation for more details about how each class works.

    // default flag is zlib.BROTLI_OPERATION_FLUSH
    let compress = new zlib.BrotliCompress();
    let decompress = new zlib.BrotliDecompress();
    
    let compressed = compress.process("abc");
    decompress.process(compressed).toString(); // abc
    
    compress.process("123", zlib.constants.BROTLI_OPERATION_PROCESS);
    compress.process("456", zlib.constants.BROTLI_OPERATION_PROCESS);
    let data = compress.process("789", zlib.constants.BROTLI_OPERATION_FLUSH);
    
    decompress.process(data).toString(); // 123456789

    Benchmark

    Tested on Node.js v15.4.0 running on an i5 7300HQ 2.5ghz with default zlib options

    Deflate performance on randomized json messages of various sizes

    Library ~ 0.03kb ~ 0.5kb ~ 11kb
    zlib (stream) 38140 op/s 18496 op/s 1621 op/s
    pako (stream) 38975 op/s 13377 op/s 835 op/s
    minizlib 74851 op/s 26104 op/s 1892 op/s
    zlib-sync* - - -
    fast-zlib 113056 op/s 29612 op/s 1948 op/s

    Inflate performance on the same messages

    Library ~ 0.03kb ~ 0.5kb ~ 11kb
    zlib (stream) 61274 op/s 34272 op/s 5262 op/s
    pako (stream)* - - -
    minizlib 198000 op/s 96283 op/s 13944 op/s
    zlib-sync 407455 op/s 127178 op/s 14875 op/s
    fast-zlib 641532 op/s 145444 op/s 15194 op/s

    *zlib-sync does not yet support compression.
    *pako v2 removed support for decompressing with Z_SYNC_FLUSH.

    More benchmarks can be found at zlib-benchmark

    Install

    npm i fast-zlib

    DownloadsWeekly Downloads

    520

    Version

    2.0.1

    License

    MIT

    Unpacked Size

    15.3 kB

    Total Files

    8

    Last publish

    Collaborators

    • timotejroiko