Binary Data Representation (serialization format)
Oh no! Not another BSON?!?!
The fact of the proliferation of serialized data formats, and of binary JSON-like formats in particular, is a fiat demonstration that there isn't actually a solid consensus for a single unified and universal format. As such, I (Danfuzz) think it's a reasonable and prudent idea to continue the exploration of the design of these things.
Here are the properties being aimed for here:
single-representation (non-equal binary forms implies non-isomorphic / non-equal object forms)
without gratuitous quoting (e.g., look what happens when you put a JSON-encoded string inside another JSON-encoded string; avoid that!)
supports a buffer (binary blob) type
possible. For example, this format differentiates
undefined. Why? Because you can tell the difference between them
in the purely-local case, and you shouldn't necessarily have to
change your code if you decide to interpose a serialization (e.g., a
machine boundary). I have no love of
undefined myself and wish I
could do away with it, but I'm not about to try to do so wishfully.
supports "holes" where non-pure-data elements may be filled in. More specifically, supports clients that want to be able to set up function/callback-like things for over-the-wire communication.
supports non-tree graphs of objects. This includes both circular structures and arbitrary acyclic graphs. Non-cycles might be tempting to convert to trees on serialization, but that would have poor interplay with the idea of "holes" (above). That is, once you have "holes," object identity starts to matter a lot more.
partially streamable. If you want to incrementally stream a (notionally) single structure, that can be done by defining a layer above this one.
random-access. If you need an index into a structure, then (as with the previous item) it can be built as an additional layer.
A few references:
One of the major reasons for the development of this module was to have the ability to sensibly manage and manipulate non-data "holes" in what is otherwise a graph of data objects, particularly so that two sides of a communication link can succeed in communicating about at least some "non-data-y things" despite not being able to actually share an address space.
In this example, imagine that the two sides (serializing and parsing) do a bit more than the trivial transformations here. Hopefully, this is enough to give a flavor of things.
(You can find this example in the
var bidar = require"../";return "muffins";return "are";return "tasty";return "[" + hole + "]";return "<" + hole + ">";var message = muffins are "very" tasty ;// The "sending" side.var serialized = bidarserializemessage holeFilter;// The "receiving" side.var parsed = bidarparseserialized holeReverser;console.logparsed;
Here is a transcript of running it:
$ node example/transform.js[ '<[muffins]>', '<[are]>', 'very', '<[tasty]>' ]
The format will be more fully described once it has baked a bit and seems stable. In the mean time, refer to the source. (It's got comments.)
This library provides just a few top-level functions.
Serialize an object graph, rooted at the given object, into binary form. Returns a buffer of the result. The hole filter, if provided is used as a transformation on "holes" encoutered in the object graph.
A "hole" is any object that is not representable as pure data. This includes functions and (non-array, non-buffer) objects whose prototype is not the default object prototype.
The hole filter is called with a single argument -- the original "hole" object encountered in the graph -- and is expected to return a serializable pure data object in response. That object is marked as a "hole replacement" in the serialized output.
Parse a serialized form into an object graph, returning the root of the
graph as originally serialized. The
holeFiller, if specified, is a
function which is consulted to transform "hole replacements" (see the
serialize() for more details) back into directly-usable
The entire contents of the given buffer are expeceted to be consulted in the process of parsing. If there is extra data at the end of the buffer that is not needed, then this function throws an exception.
This will also throw an exception if there is insufficient data to complete the operation.
And finally, this will throw an exception if given a serialized form
that has holes, if
holeFiller was not specified.
This is like
parse(), except that only a portion of the input buffer
is expected to be used. The
startIndex (defaults to
0) is the
index into the buffer at which to begin parsing. The optional
is the same as with
The return value of this call is a map of two values:
root is bound
to the root of the parsed object graph.
bytesConsumed is bound to the
number of bytes that were read from the buffer during the parse operation.
This is a sub-module that contains convenient utilities for type determination and related assertions. The author does not particularly recommend the use of this module. It may drastically change over time, or may simply disappear.
npm install bidar
Or grab the source and
node-waf configure build
If there are extra properties on Buffer objects, they get silently
ignored. E.g.: On serialization,
foo will get dropped with this
x = new Buffer(10); x.foo = "bar";
Regex objects are currently considered holes. They should perhaps be handled specially (as Buffer objects currently are).
Date objects are currently considered holes. They should perhaps be handled specially (as Buffer objects currently are).
Questions, comments, bug reports, and pull requests are all welcome. Submit them at the project on GitHub.
Bug reports that include steps-to-reproduce (including code) are the best. Even better, make them in the form of pull requests that update the test suite. Thanks!
Copyright 2012 The Obvious Corporation.
Licensed under the Apache License, Version 2.0.
See the top-level file