Nattily Polished Marbles

    orle

    3.0.0 • Public • Published

    npm version CircleCI

    orle

    orle (pronounced "Oh, really?") is simple run length encoder for Javascript typed arrays.

    Usage

    Using orle is simple. There are two exposed methods: encode and decode.

    Encoding

    If you pass a typed array (such as Uint16Array or Float32Array) to encode, that data type is used. encode returns a promise with an encoded buffer. If you pass a non-typed array, it will try to find the most compact data type possible to encode the data. Note: if decimals are found, it will use Float64Array to preserve precision.

    const orle = require('orle');
    const buffer = await orle.encode([1,1,1,1,1,1,5,6,1,1,1,1,1,1,1,1,1]);
    

    or

    const orle = require('orle');
    const buffer = await orle.encode(new Int8Array([1,1,1,1,1,1,5,6,1,1,1,1,1,1,1,1,1]));
    

    If it's advantageous to gzip part of the payload, it will do so automatically. You can opt out of this by passing options to encode orle.encode(arr, { gzip: false }).

    Notes

    This package primarily optimizes typed arrays, which don't support null or undefined as elements. null or undefined get coerced to 0 when encoding.

    Decoding

    Pass a buffer to decode and you will get back a promise of a typed array.

    const orle = require('orle');
    const arr = await orle.decode(buffer);
    

    Compression

    Obviously your results may vary. If you have a large array with entirely non-repeating numbers, this will add about 9 bytes to the total payload. If you have a large array of entirely repeating numbers, the resulting payload will be about 10 bytes.

    Format

    The binary format is pretty simple:

    Data Version

    Bytes: 1 Sample Value: 7

    Data Type/Lookup Table Flag

    Bytes: 1 Sample Value: The first 7 bits are an unsigned int representing different data formats. Possible formats are:

    • 0: Int32
    • 1: Int16
    • 2: Int8
    • 3: Uint32
    • 4: Uint16
    • 5: Uint8
    • 6: Float32
    • 7: Float64
    • 8: String

    If the last bit is set, that indicates that a lookup table is present If the second last bit is set, that indicates that the Payload is gzipped

    Run Value Size

    Bytes: 1 Sample Value: The size of each "run" value. Unsigned 8 bit int:

    • 0: Int32
    • 1: Int16
    • 2: Int8

    Runs

    Bytes: size-of-each-run-value * (number-of-distinct-runs+1) Sample Value: Store each set of run values. Positive values indicates that the value is repeated that number of times. Negative values indicates that there is a run of distinct values. 0 indicates there are no more runs defined

    Lookup Table Length

    Bytes: 0 or 1 Sample Value: If the lookup table bit was set, this indicates how many items are in the LUT (max of 256)

    Lookup Table

    Bytes: size-of-each-LUT-value * number-of-items-in-LUT Sample Value: The data is serialized flat and is obviously variable length. There are a maximum of 256 values in the lookup table. If a Lookup table is used, the payload items are serialized as Uint8s indicating the index into this array that they map to

    GZip Payload

    Bytes: A 4 byte UInt32 followed by a gzip payload Note: The Payload is gzipped if the second bit of the data type lookup is set. If the payload is gzipped, the rest of the storage format remains the same, except it is gzipped/gunzipped first.

    Payload

    Bytes: size-of-each-value * number-of-values-stored Sample Value: The actual payload. Important notes: The order here is very important and has to map to the runs previously defined. If a run is "positive" then the item should only appear once here.
    Note on strings: Strings are stored as a Uint32LE number representing followed by that number of bytes of a JSON representation of the string array.
    Note on Lookup Tables: If a lookup table is being used, the size of each value will be 1 and it will be a Uint8 value representing the value to use from the lookup table.

    Keywords

    none

    Install

    npm i orle

    DownloadsWeekly Downloads

    7

    Version

    3.0.0

    License

    ISC

    Unpacked Size

    29.5 kB

    Total Files

    9

    Last publish

    Collaborators

    • jbreckman