@3m1/binary-object

Binary Object

Manage binary data with strictly typed JavaScript Object-oriented programming.

Summary

• Install
• Usage
  • First: polyfill if needed
  • Second: decorators or not
  • API
  • Examples
• Memory owner
• Use cases
  • WebAssembly
  • Disable Garbage Collector (GC)
  • Workers API
  • Saving/restoring states
  • Accessing binary data files
  • Accessing binary APIs
  • Develop backend DB APIs
• See also

Install

With npm:

npm install binary-object

With yarn:

yarn add binary-object

Usage

First: polyfill if needed

This library uses TextEncoder and TextDecoder to transform text to and from binary data. These are JavaScript native functions, but Node lacks them. You need to polyfill them first:

if(!('TextEncoder' in global)) {
  import('util').then(nodeUtil => {;
    global.TextEncoder = nodeUtil.TextEncoder;
    global.TextDecoder = nodeUtil.TextDecoder;
  });
}

For this to work on Node >= 14, you need to install util first:

npm install --save util

Second: decorators or not

This library encourages the use of class member decorators, available in Typescript, but at a stage 2 proposal. To add it into your Babel configuration, you will need something like:

    "plugins": [
      "@babel/plugin-transform-runtime",
      [ "@babel/plugin-proposal-decorators", { "legacy": true } ],
      [ "@babel/plugin-proposal-class-properties", { "loose": true } ]
    ]

If you don't want to use decorators, you will need to use it like (not tested):

class MyBinaryClass extends Binary { /* Functions/logic/statics */ }
Object.defineProperty(MyBinaryClass.prototype, "someMember", binary(Types.Float32));

API

See autogenerated jsdoc API documentation at this repo's GitHub Page.

Examples

import {Binary, binary, Types} from 'binary-object';

// Declare object shape
class BinaryTest extends Binary {
  @binary(Types.Uint32)
  id = 0;
  @binary(Types.Float64)
  testFloat = 0;

  // Array of 10 low precision floating point numbers
  @binary(Types.Array(Types.Float32, 10))
  testFloatArray;

  showId = () => console.log(`My id is ${this.id}`);
}

// Allocate memory
const binTest = new ArrayBuffer( BinaryTest.binarySize );

// Instantiate memory parser
const proxyObject = new BinaryTest(binTest);

// Use it
proxyObject.id = 12345;
proxyObject.testFloat = 7.8;

expect(proxyObject.id).toBe(12345);
expect(proxyObject.testFloat).toBe(7.8);

Accessing and modifying arrays and array elements also work:

const testArray = proxyObject.testFloatArray;

// Iteratibility
expect([...testArray]).toEqual(new Array(10).fill(0));

// Elements access and modification
expect(testArray[0]).toBe(0);
testArray[0] = 1;
expect(testArray[0]).toBe(1);
expect(testArray[1]).toBe(0);
testArray[1] = 2;
expect(testArray[0]).toBe(1);
expect(testArray[1]).toBe(2);

// Modify full array
const newArr = [1,0,0,0,0,0,0,0,0,1];
proxyObject.testFloatArray = newArr;
expect([...testArray]).toEqual(newArr);

You can define padded arrays for better performance and, maybe, enforced by API:

  const type = Types.Uint32;
  const length = 5;
  class BinaryPadTest extends Binary {
    @binary(Types.Uint8)
    someMemberAtTheBegining;

    // Here, `true` reffers to padding
    @binary(Types.Array(type, length, true))
    testArray;

    @binary(Types.Uint8)
    someMemberAtTheEnd;
  }

  const binTest2 = new ArrayBuffer(BinaryPadTest.binarySize);
  const testObj = new BinaryPadTest(binTest2);

  // The first byte of someMemberAtTheBegining forces to consume
  // a full testArray element before it (for padding)
  const expectedSize = ((length + 1) * type.bytes) + 1;
  expect(BinaryPadTest.binarySize).toBe(expectedSize);
  expect(testObj.testArray).toBeInstanceOf(type.extractor); // Uint32Array

Object composition is also allowed:

class BinaryArrayOfNestedTest extends Binary {
  // Array of 3 BinaryTest objects
  @binary( Types.Array( Types.Struct(BinaryTest), 3))
  testNested;

  get id() { return this.testNested[0].id }
  set id(value) {
    this.testNested[0].id = value;
    return true
  }

  @binary(Types.Uint32)
  someNumber = 0;

  showId = this.testNested[0].showId;
}

const binTest2 = new ArrayBuffer( BinaryArrayOfNestedTest.binarySize );
const proxyNested = new BinaryArrayOfNestedTest( binTest2 );

Object composition from different memory sources is also allowed:

class Position extends Binary {
  @binary(Types.Uint32)
  x;
  @binary(Types.Uint32)
  y;

  static testCollision(pos1, pos2) {
    ...
  }

  collision = (pos2) => this.constructor.testCollision(this, pos2);
}

class Player extends Binary {
  @binary( Types.Struct(Position) )
  position;
  @binary(Types.Float64)
  life;

  // Non managed binary data (like pointers)
  bullets = [];
}

class Bullet extends Binary {
  @binary( Types.Struct(Position) )
  position;
  @binary(Types.Float64)
  direction;
}

class Game {
  constructor() {
    // malloc for players
    this.binPlayers = new ArrayBuffer( Player.binarySize * 2 );
    this.player1 = new Player( this.binPlayers );
    this.player2 = new Player( this.binPlayers, Player.binarySize );

    // malloc for bullets
    this.maxBullets = 100;
    this.binBullets = new ArrayBuffer( Bullet.binarySize * this.maxBullets );

    // Optimize bullets by using a unique DataView for all of them
    this.dvBullets = new DataView(this.binBullets);

    // Half of the bullets for each player
    Bullets.arrayFactory(
      this.dvBullets,
      maxBullets / 2,
      0,
      player1.bullets);
    Bullets.arrayFactory(
      this.dvBullets,
      maxBullets / 2,
      maxBullets / 2,
      player2.bullets);
  }

  // You can move Bullets using a parallel worker or a WASM code block

  // Sometimes, check if a bullet touched a victim
  testTouched(attacker, victim) {
    const {position: {collision}} = victim;
    const touched = attacker.bullets.some( ({position}) => collision(position) );
  }
}

Transform your binary data into a JSON string:

console.log( JSON.stringify(proxyNested) );

Assign data from JS objects:

proxyNested.testNested = {testFloatArray: [0,0,0,0,0,0,0,0,0,1]};
expect( [...proxyNested.testNested.testFloatArray] ).toEqual([0,0,0,0,0,0,0,0,0,1]);

Allocate and parse a big memory chunk as array of objects:

const iterations = 5e5;
const binTest3 = new ArrayBuffer( BinaryTest.binarySize * iterations );
const dv3 = new DataView(binTest3);
const bObjList = new Array(iterations);
for(let i = 0; i < iterations; i++) {
  bObjList.push(new BinaryTest(dv3, BinaryTest.binarySize * i));
}
bObjList.forEach( (obj, index) => obj.id = `i${index}`);
const ids = bObjList.map( ({id}) => id);

Memory owner

This library does not aims to own the memory, allowing to use it in different ways. The memory can be allocated from a parallel worker, from the WebAssembly code or from another library or API which can be consumed using ArrayBuffer or SharedBuffer. It's recommended to wrap this library with the one managing the memory.

Note: Opting in into managing your own memory (as in C) requires you to understand what you are doing at a low level. Things like dynamic array sizes or text manipulation can be a pain if you don't understand the basics (not to say about endianness or memory padding). There are several standards and known ways to manage the memroy. This library aims to make it much easier to parse it (read and write), but you'll still need to know how it works. Specifically, this libs does not aims to own the memory pieces: you'll need to create/allocate/move/copy/reallocate/free them around instantiating the JS binary objects.

Use cases

The environments where direct memory management would be desirable are minimal in JS. Still, there are some edge cases where it could be benefficial or mandatory to do so.

WebAssembly

WebAssembly allows coding in some low level language (like C, C++ or Rust) and compiling to some binary code which can be executed by a JS interpreter (with WebAssembly API, of course).

The code compiled into WebAssembly can use dedicated libraries to communicate with the JS code. It allows accessing the DOM (thus, the window object), but also sharing pieces of memory using WebAssembly.Memory. Using that, you could write complex but very fast code to do heavy calculations in your compiled code, and trigger some kind of JS re-render using the currently calculated values at some time or frame-rate.

Disable Garbage Collector (GC)

As app states is mostly managed by the ArrayBuffer (except some JS internal caches), you can minimize GC works, which will run your game (app) much smoothly without unexpected background GC tasks.

Workers API

Similar to the WebAssembly use case, some one could create a SharedBuffer and use it from different Workers, allowing your app to use more than one CPU. For example in a game: you could have the main worker which updates the DOM (or a canvas or a OpenGL) using the data from the buffer, while having several workers calculating 3D collisions.

Saving/restoring states

As all the data is in a single memory piece, you can easily save it to somewhere and load it later to restore the state. This could be benefficial for games, as well as AIs, 3D renderers, scientific calculations, password crackers, etc.

Accessing binary data files

Dynamic binary reading, processing and writing (to server or directly to user), like images, audio, video, medical data (DICOM), etc.

Accessing binary APIs

You could take full advantage of browsers USB and/or Bluetooth APIs, but also you could easily communicate against binary API servers or IoT. Though, as those are usually read-only or stream based, you would preffer using DataStreams.js instead.

Develop backend DB APIs

Some libraries allows maintaining shared pieces of memory between the backend app and the database. This lib could help developing Node database middlewares.

See also

There are several JS projects aiming to handle binary data:

• buffer-backed-object: creates objects that are backed by an ArrayBuffer
• @bnaya/objectbuffer (source code): JavaScript Object like api, backed by an arraybuffer
• DataStream.js: library for reading data from ArrayBuffers
• Restructure
• Restructure (next)
• buffercodec
• Buffer Plus
• Binary Protocol
• Binary-parser
• @yotamshacham/schema
• Structron
• cppmsg
• bin-protocol
• byte-data
• binobject
• Binarcular
• Uttori Data Tools
• @avro/types
• Structurae
• @thi.ng/unionstruct
• Typed Array Buffer Schema
• @sighmir/bstruct
• binary-parser-encoder
• binary-encoder
• c-struct
• binary-transfer
• Superbuffer
• @binary-files/structjs
• js Binary Schema Parser
• jDataView/jBinary