November: Procrastination Month

    buffer-plus

    1.3.0 • Public • Published

    Buffer Plus

    Enhanced high performance Node.js style Buffer with auto re-allocation, custom. types and static schematic support.

    It's really fast that you should try.

    Introduction

    Node.js native Buffer provides a easy way to manipulate binary data, which is useful for communicating between different languagesm, or tranfering data over network.

    But it's annoying to calculate offset and buffer size when the binary data structure becomes complex. And it also annoying when we want to insert some bytes or needs to forward/rewind position.

    This library aims to solve these problems with light weight overhead. Let programmer to define a flexiable custom. data types, or even define static schematic to present complex. data structure.

    Key Features

    • Similiar API styles to native node.js Buffer API.
    • Keeps track of read and write positions automatically.
    • Automatic buffer resizing to make sure read/write operations safely.
    • Support many extra methods to deal with variant integer, 64-bit integers, and string.
    • Support customized data type definition.
    • Support schematic definition with static data/field ordering.
    • High performance, the static schematic decode/encode speed are the same as well programmed native Buffer API.

    Installation

    Via NPM

        npm install buffer-plus

    Via Yarn

        yarn add buffer-plus

    The Order of Binary Fields is Matter

    When we want to exchange data which encoded or serialized in binary format between different programming languages or devices, there are several common ways like JSON, messagepack, protocol buffers...etc.

    These encode/decode methods are great, but also with significant byte size and parsing time overhead...etc. And some methods like JSON has a problem: The order of object properties in the structure is not guaranteed due to the nature of these methods need to handle dynamic data structure.

    It means the decode side needs to know the whole data size first, store whole data into buffer, then parse data. But in the case we defined the fixed data structure and want to one-phase data handling, the order of fields is matter.

    We can read field bytes by bytes in one-phase only the order of fields are fixed, and the length of fields are well defined in data structure.

    When we need to exchange data over high latency network, the data size is the key part of performance bottleneck than the decode/encode speed. And the most size-efficiency way to exchange data is with fixed, static data schematic.

    BufferPlus API Reference

    Refer the delicate BufferPlus API Reference

    Examples

    Simple Schematic Example

    const BufferPlus = require('buffer-plus');
    const assert = require('assert');
     
    // account data
    const account = {
        name: 'user1',
        age: 18,
        languages: ['en-US', 'en-UK'],
        serial: 0x123456781234567,
    };
     
    // account schema definition
    const AccountSchema = {
        type: 'object',
        properties: {
            name: {type: 'string'},
            age: {type: 'uint8'},
            languages: {
                type: 'array',
                items: {type: 'string'},
            },
            serial: {type: 'uint64le'},
        },
        order: ['name', 'age', 'languages', 'serial'],
    };
     
    // create a BufferPlus instance
    const bp = BufferPlus.create();
     
    // create Account schema
    BufferPlus.createSchema('Account', AccountSchema);
     
    // write account data with Account schema
    bp.writeSchema('Account', account);
     
    // move to buffer begining
    bp.moveTo(0);
     
    // read account from buffer
    const decodedAccount = bp.readSchema('Account');
     
    // compare decoded and original account
    assert.deepStrictEqual(account, decodedAccount);
     

    Simple Customized Type Example

     
    const BufferPlus = require('buffer-plus');
    const crypto = require('crypto');
    const assert = require('assert');
     
    function getMd5Hash(value) {
        return crypto.createHash('md5').update(value).digest('hex');
    }
    const bp = BufferPlus.alloc(1024);
     
    BufferPlus.addCustomType('HashString',
            // writeHashString method
            (buffer) => {
                const hashLen = buffer.readUInt32LE();
                const valueLen = buffer.readUInt32LE();
                const hash = buffer.readString(hashLen);
                const value = buffer.readString(valueLen);
                return {value: value, hash: hash};
            },
            // readHashString method
            (buffer, value) => {
                const hash = getMd5Hash(value);
                buffer.writeUInt32LE(hash.length);
                buffer.writeUInt32LE(value.length);
                buffer.writeString(hash);
                buffer.writeString(value);
            },
            // byteLengthHashString method
            (buffer, value) => {
                const hash = getMd5Hash(value);
                return 8 + BufferPlus.byteLength(value) + BufferPlus.byteLength(hash);
            }
    );
     
    const str = 'test hash string';
    bp.writeHashString(str);
    bp.moveTo(0);
     
    const hashStr = bp.readHashString();
    assert.strictEqual(str, hashStr.value);
    assert.strictEqual(getMd5Hash(hashStr.value), hashStr.hash);

    Install

    npm i buffer-plus

    DownloadsWeekly Downloads

    39

    Version

    1.3.0

    License

    MIT

    Unpacked Size

    265 kB

    Total Files

    24

    Last publish

    Collaborators

    • arlo