Nanoprogrammed Penultimate Musicianship

    pbf
    DefinitelyTyped icon, indicating that this package has TypeScript declarations provided by the separate @types/pbf package

    3.2.1 • Public • Published

    pbf

    build status Coverage Status

    A low-level, fast, ultra-lightweight (3KB gzipped) JavaScript library for decoding and encoding protocol buffers, a compact binary format for structured data serialization. Works both in Node and the browser. Supports lazy decoding and detailed customization of the reading/writing code.

    Performance

    This library is extremely fast — much faster than native JSON.parse/JSON.stringify and the protocol-buffers module. Here's a result from running a real-world benchmark on Node v6.5 (decoding and encoding a sample of 439 vector tiles, 22.6 MB total):

    • pbf decode: 387ms, or 57 MB/s
    • pbf encode: 396ms, or 56 MB/s
    • protocol-buffers decode: 837ms, or 26 MB/s
    • protocol-buffers encode: 4197ms, or 5 MB/s
    • JSON.parse: 1540ms, or 125 MB/s (parsing an equivalent 77.5 MB JSON file)
    • JSON.stringify: 607ms, or 49 MB/s

    Examples

    Using Compiled Code

    Install pbf and compile a JavaScript module from a .proto file:

    $ npm install -g pbf
    $ pbf example.proto > example.js

    Then read and write objects using the module like this:

    var Pbf = require('pbf');
    var Example = require('./example.js').Example;
     
    // read
    var pbf = new Pbf(buffer);
    var obj = Example.read(pbf);
     
    // write
    var pbf = new Pbf();
    Example.write(obj, pbf);
    var buffer = pbf.finish();

    Alternatively, you can compile a module directly in the code:

    var compile = require('pbf/compile');
    var schema = require('protocol-buffers-schema');
     
    var proto = schema.parse(fs.readFileSync('example.proto'));
    var Test = compile(proto).Test;

    If you use webpack as your module bundler, you can use pbf-loader to load .proto files directly. It returns a compiled module ready to be used.

    Given you already configured your webpack.config.js, the code above would look like:

    var Pbf = require('pbf');
    var proto = require('./example.proto');
     
    var Test = proto.Test;

    Custom Reading

    var data = new Pbf(buffer).readFields(readData, {});
     
    function readData(tag, data, pbf) {
        if (tag === 1) data.name = pbf.readString();
        else if (tag === 2) data.version = pbf.readVarint();
        else if (tag === 3) data.layer = pbf.readMessage(readLayer, {});
    }
    function readLayer(tag, layer, pbf) {
        if (tag === 1) layer.name = pbf.readString();
        else if (tag === 3) layer.size = pbf.readVarint();
    }

    Custom Writing

    var pbf = new Pbf();
    writeData(data, pbf);
    var buffer = pbf.finish();
     
    function writeData(data, pbf) {
        pbf.writeStringField(1, data.name);
        pbf.writeVarintField(2, data.version);
        pbf.writeMessage(3, writeLayer, data.layer);
    }
    function writeLayer(layer, pbf) {
        pbf.writeStringField(1, layer.name);
        pbf.writeVarintField(2, layer.size);
    }

    Install

    Node and Browserify:

    npm install pbf

    Making a browser build:

    npm install
    npm run build-dev # dist/pbf-dev.js (development build) 
    npm run build-min # dist/pbf.js (minified production build) 

    CDN link: https://unpkg.com/pbf@3.0.5/dist/pbf.js

    API

    Create a Pbf object, optionally given a Buffer or Uint8Array as input data:

    // parse a pbf file from disk in Node
    var pbf = new Pbf(fs.readFileSync('data.pbf'));
     
    // parse a pbf file in a browser after an ajax request with responseType="arraybuffer"
    var pbf = new Pbf(new Uint8Array(xhr.response));

    Pbf object properties:

    pbf.length; // length of the underlying buffer
    pbf.pos; // current offset for reading or writing

    Reading

    Read a sequence of fields:

    pbf.readFields(function (tag) {
        if (tag === 1) pbf.readVarint();
        else if (tag === 2) pbf.readString();
        else ...
    });

    It optionally accepts an object that will be passed to the reading function for easier construction of decoded data, and also passes the Pbf object as a third argument:

    var result = pbf.readFields(callback, {})
     
    function callback(tag, result, pbf) {
        if (tag === 1) result.id = pbf.readVarint();
    }

    To read an embedded message, use pbf.readMessage(fn[, obj]) (in the same way as read).

    Read values:

    var value = pbf.readVarint();
    var str = pbf.readString();
    var numbers = pbf.readPackedVarint();

    For lazy or partial decoding, simply save the position instead of reading a value, then later set it back to the saved value and read:

    var fooPos = -1;
    pbf.readFields(function (tag) {
        if (tag === 1) fooPos = pbf.pos;
    });
    ...
    pbf.pos = fooPos;
    pbf.readMessage(readFoo);

    Scalar reading methods:

    • readVarint(isSigned) (pass true if you expect negative varints)
    • readSVarint()
    • readFixed32()
    • readFixed64()
    • readSFixed32()
    • readSFixed64()
    • readBoolean()
    • readFloat()
    • readDouble()
    • readString()
    • readBytes()
    • skip(value)

    Packed reading methods:

    • readPackedVarint(arr, isSigned) (appends read items to arr)
    • readPackedSVarint(arr)
    • readPackedFixed32(arr)
    • readPackedFixed64(arr)
    • readPackedSFixed32(arr)
    • readPackedSFixed64(arr)
    • readPackedBoolean(arr)
    • readPackedFloat(arr)
    • readPackedDouble(arr)

    Writing

    Write values:

    pbf.writeVarint(123);
    pbf.writeString("Hello world");

    Write an embedded message:

    pbf.writeMessage(1, writeObj, obj);
     
    function writeObj(obj, pbf) {
        pbf.writeStringField(obj.name);
        pbf.writeVarintField(obj.version);
    }

    Field writing methods:

    • writeVarintField(tag, val)
    • writeSVarintField(tag, val)
    • writeFixed32Field(tag, val)
    • writeFixed64Field(tag, val)
    • writeSFixed32Field(tag, val)
    • writeSFixed64Field(tag, val)
    • writeBooleanField(tag, val)
    • writeFloatField(tag, val)
    • writeDoubleField(tag, val)
    • writeStringField(tag, val)
    • writeBytesField(tag, buffer)

    Packed field writing methods:

    • writePackedVarint(tag, val)
    • writePackedSVarint(tag, val)
    • writePackedSFixed32(tag, val)
    • writePackedSFixed64(tag, val)
    • writePackedBoolean(tag, val)
    • writePackedFloat(tag, val)
    • writePackedDouble(tag, val)

    Scalar writing methods:

    • writeVarint(val)
    • writeSVarint(val)
    • writeSFixed32(val)
    • writeSFixed64(val)
    • writeBoolean(val)
    • writeFloat(val)
    • writeDouble(val)
    • writeString(val)
    • writeBytes(buffer)

    Message writing methods:

    • writeMessage(tag, fn[, obj])
    • writeRawMessage(fn[, obj])

    Misc methods:

    • realloc(minBytes) - pad the underlying buffer size to accommodate the given number of bytes; note that the size increases exponentially, so it won't necessarily equal the size of data written
    • finish() - make the current buffer ready for reading and return the data as a buffer slice
    • destroy() - dispose the buffer

    For an example of a real-world usage of the library, see vector-tile-js.

    Proto Schema to JavaScript

    If installed globally, pbf provides a binary that compiles proto files into JavaScript modules. Usage:

    $ pbf <proto_path> [--no-write] [--no-read] [--browser]

    The --no-write and --no-read switches remove corresponding code in the output. The --browser switch makes the module work in browsers instead of Node.

    The resulting module exports each message by name with the following methods:

    • read(pbf) - decodes an object from the given Pbf instance
    • write(obj, pbf) - encodes an object into the given Pbf instance (usually empty)

    The resulting code is clean and simple, so feel free to customize it.

    Changelog

    3.2.1 (Oct 11, 2019)

    • Significantly improved performance when decoding large strings in the browser.

    3.2.0 (Mar 11, 2019)

    • Improved decoding to be able to parse repeated fields even if they were specified as packed, and vise versa.
    • Improved packed encoding to skip empty arrays (previously, it would write a tag).
    • Fixed an off-by-one data corruption bug when writing a message larger than 0x10000000 bytes.

    3.1.0 (Sep 27, 2017)

    • Added support for Protocol Buffer 3 maps to proto compiler.

    3.0.5 (Nov 30, 2016)

    • Fixed an error appearing in some versions of IE11 and old Android browsers.

    3.0.4 (Nov 14, 2016)

    • Fixed compiling repeated packed enum fields.

    3.0.3 (Nov 14, 2016)

    • Fixed a regression that broke compiling repeated enum fields with defaults.

    3.0.2 (Sep 30, 2016)

    • Fixed a regression that broke decoding of packed fields with a tag that didn't fit into one byte.

    3.0.1 (Sep 20, 2016)

    • Fixed a regression that broke encoding of long strings.

    3.0.0 (Aug 30, 2016)

    This release include tons of compatibility/robustness fixes, and a more reliable Node implementation. Decoding performance is expected to get up to ~15% slower than v2.0 in Node (browsers are unaffected), but encoding got faster by ~15% in return.

    Encoder/decoder
    • Breaking: changed Node implementation to use Uint8Array instead of Buffer internally (and produce corresponding result on finish()), making it fully match the browser implementation for consistency and simplicity.
    • Fixed writeVarint to write 0 when given NaN or other non-number to avoid producing a broken Protobuf message.
    • Changed readPacked* methods signature to accept an optional arr argument to append the results to (to support messages with repeated fields that mix packed/non-packed encoding).
    • Added an optional isSigned argument to readVarint that enables proper reading of negative varints.
    • Deprecated readVarint64() (it still works, but it's recommended to be changed to readVarint(true)).
    • Faster string encoding.
    Proto compiler
    • Breaking: Full support for defaults field values (both implicit and explicit); they're now included in the decoded JSON objects.
    • Fixed reading of repeated fields with mixed packed/non-packed encoding for compatibility.
    • Fixed proto3 compiler to use packed by default for repeated scalar fields.
    • Fixed reading of negative varint types.
    • Fixed packed fields to decode into [] if they're not present.
    • Fixed nested message references handling.
    • Fixed packed=false being interpreted as packed.
    • Added a comment to generated code with pbf version number.

    2.0.1 (May 28, 2016)

    • Fixed a regression with writeVarint that affected certain numbers.

    2.0.0 (May 28, 2016)

    • Significantly improved the proto compiler, which now produces a much safer reading/writing code.
    • Added the ability to compile a read/write module from a protobuf schema directly in the code.
    • Proto compiler: fixed name resolutions and collisions in schemas with nested messages.
    • Proto compiler: fixed broken top-level enums.

    1.3.7 (May 28, 2016)

    • Fixed a regression with writeVarint that affected certain numbers.

    1.3.6 (May 27, 2016)

    • Improved read and write performance (both ~15% faster).
    • Improved generated code for default values.

    1.3.5 (Oct 5, 2015)

    • Added support for syntax keyword proto files (by updating resolve-protobuf-schema dependency).

    1.3.4 (Jul 31, 2015)

    • Added writeRawMessage method for writing a message without a tag, useful for creating pbfs with multiple top-level messages.

    1.3.2 (Mar 5, 2015)

    • Added readVarint64 method for proper decoding of negative int64-encoded values.

    1.3.1 (Feb 20, 2015)

    • Fixed pbf proto compile tool generating broken writing code.

    1.3.0 (Feb 5, 2015)

    • Added pbf binary that compiles .proto files into Pbf-based JavaScript modules.

    1.2.0 (Jan 5, 2015)

    Breaking API changes
    • Changed writeMessage signature to (tag, fn, obj) (see example in the docs) for a huge encoding performance improvement.
    • Replaced readPacked and writePacked methods that accept type as a string with readPackedVarint, etc. for each type (better performance and simpler API).
    Improvements
    • 5x faster encoding in Node (vector tile benchmark).
    • 40x faster encoding and 3x faster decoding in the browser (vector tile benchmark).

    1.1.4 (Jan 2, 2015)

    • Significantly improved readPacked and writePacked performance (the tile reading benchmark is now 70% faster).

    1.1.3 (Dec 26, 2014)

    Brings tons of improvements and fixes over the previous version (0.0.2). Basically makes the library complete.

    Improvements
    • Improved performance of both reading and writing.
    • Made the browser build 3 times smaller.
    • Added convenience readFields and readMessage methods for a much easier reading API.
    • Added reading methods: readFloat, readBoolean, readSFixed32, readSFixed64.
    • Added writing methods: writeUInt64, writeSFixed32, writeSFixed64.
    • Improved readDouble and readString to use native Buffer methods under Node.
    • Improved readString and writeString to use HTML5 TextEncoder and TextDecoder where available.
    • Made Pbf buffer argument optional.
    • Added extensive docs and examples in the readme.
    • Added an extensive test suite that brings test coverage up to 100%.
    Breaking API changes
    • Renamed readBuffer/writeBuffer to readBytes/writeBytes.
    • Renamed readUInt32/writeUInt32 to readFixed32/writeFixed32, etc.
    • Renamed writeTaggedVarint to writeVarintField, etc.
    • Changed writePacked signature from (type, tag, items) to (tag, type, items).
    Bugfixes
    • Fixed readVarint to handle varints bigger than 6 bytes.
    • Fixed readSVarint to handle number bigger than 2^30.
    • Fixed writeVarint failing on some integers.
    • Fixed writeVarint not throwing an error on numbers that are too big.
    • Fixed readUInt64 always failing.
    • Fixed writing to an empty buffer always failing.

    Install

    npm i pbf

    DownloadsWeekly Downloads

    730,579

    Version

    3.2.1

    License

    BSD-3-Clause

    Unpacked Size

    125 kB

    Total Files

    8

    Last publish

    Collaborators

    • aarthykc
    • ajashton
    • ajithranka
    • aliceykuo
    • alinapaz
    • allierowan
    • alulsh
    • amishas157
    • amyleew
    • anandthakker
    • ansis
    • aparlato
    • apendleton
    • arunasank
    • benjamintd
    • bhousel
    • bkowshik
    • boblannon
    • brendanmcfarland
    • bsrinivasa
    • bsudekum
    • camillacaros
    • captainbarbosa
    • chaupow
    • chetangowda
    • chrisloer
    • colleenmcginnis
    • danieljh
    • danpat
    • danswick
    • dasulit
    • davidtheclark
    • dnomadb
    • emilymcafee
    • emilymdubois
    • enf
    • fa7c0n
    • freenerd
    • geohacker
    • ghoshkaj
    • gretacb
    • ian29
    • ianshward
    • ingalls
    • isiyu
    • jacquestardie
    • jfirebaugh
    • jothirnadh
    • jrpruit1
    • k-mahoney
    • kaibot3000
    • kaidalgleish
    • karenzshea
    • katydecorah
    • kepta
    • kkaefer
    • l-r
    • laurier
    • lbud
    • lily-chai
    • lndsay
    • lukasmartinelli
    • lxbarth
    • lyzidiamond
    • maning
    • manoharuss
    • mapbox-admin
    • mapbox-machine-user
    • mapbox-npm
    • mapbox-npm-01
    • mapbox-npm-02
    • mapbox-npm-03
    • mapbox-npm-04
    • mapbox-npm-05
    • mapbox-npm-06
    • mapbox-npm-07
    • mapbox-npm-08
    • mapbox-npm-09
    • mapbox-npm-advanced-actions
    • mapbox-npm-ci
    • mapsam
    • mattficke
    • mcwhittemore
    • miccolis
    • miles-dev
    • mollymerp
    • morganherlocker
    • mourner
    • msirenko
    • nammala
    • natslaughter
    • nickcordella
    • nickidlugash
    • nikhilprabhakar
    • oini
    • oliikit
    • oxidase
    • pdgoodman
    • perrygeo
    • planemad
    • pratikyadav
    • rasagy
    • rclark
    • rumc
    • ryan-baumann
    • saikia.abhishek
    • samanbb
    • samely
    • sbma44
    • sgillies
    • springmeyer
    • srividyacb
    • tcql
    • themarex
    • tony-cj
    • tristen
    • upendrayadav
    • vincents
    • virginiayung
    • willwhite
    • yhahn
    • zmully