what-the-pack
Ultra-fast MessagePack for NodeJS & Browsers.
implementation notes
- this implementation uses pre-allocated buffers and buffer.copy() for encoding, instead of regular arrays
- uses a buffer polyfill if used in browser environments
- has dictionary support, to further reduce payload size
backward compatibility notes with other libraries
- used extensions
fixext 1, type 0, data 0
=undefined
fixext 1, type 0, data 1
=NaN
fixext 1, type 0, data 2
=+Infinity
fixext 1, type 0, data 3
=-Infinity
Buffers
,ArrayBuffers
andTypedArrays
Buffers
: encoded as Buffers, decoded as BuffersArrayBuffers
: encoded as Buffers, decoded as Buffers
const decoded = ;const your_arraybuffer = decodedbuffer;TypedArrays
: encoded as Buffers, decoded as Buffers
const decoded = ;const your_typedarray = decodedbuffer;
usage
yarn add what-the-pack
const MessagePack = ;const encode decode = MessagePack; // 4MB const data = name: 'Lunox' age: 20; const encoded = ;const decoded = ; console;
result
encoded: <Buffer 82 a4 6e 61 6d 65 a5 4c 75 6e 6f 78 a3 61 67 65 14> decoded: name: 'Lunox' age: 20
pre-allocating a larger buffer
const MessagePack = ;const encode decode = MessagePack; // 1GBconst data = // large data goes here;
2^7 = 128 B
2^8 = 256 B
2^9 = 512 B
2^10 = 1.02 kB
2^11 = 2.05 kB
2^12 = 4.1 kB
2^13 = 8.19 kB
2^14 = 16.4 kB
2^15 = 32.8 kB
2^16 = 65.5 kB
2^17 = 131 kB
2^18 = 262 kB
2^19 = 524 kB
2^20 = 1.05 MB
2^21 = 2.1 MB
2^22 = 4.19 MB
2^23 = 8.39 MB
2^24 = 16.8 MB
2^25 = 33.6 MB
2^26 = 67.1 MB
2^27 = 134 MB
2^28 = 268 MB
2^29 = 537 MB
2^30 = 1.07 GB
using dictionaries (added in 1.1.3)
- this feature isn't in MessagePack spec but added as a convenience feature in 1.1.3
- dictionaries allow us to decrease our buffer output size by recognizing strings used as object keys and replacing them with shorter-byte integer values during the encoding process
- these shorter-byte placeholder values are then restored to their respective strings during the decoding process
- the trade-off in using dictionaries is an insignificantly slower encoding and decoding time in exchange of a significantly smaller buffer output, which results into a lower network bandwidth and storage consumption in the long run
- the best part: the byte placeholders starts from -32 then increments upwards, values -32 to 127 are encoded in single byte, which means your first (32 + 128) = 160 keys will be encoded as a single byte instead of encoding the whole string
const MessagePack = ;const encode decode register = MessagePack; // 4MBlet encoded decoded data;data = name: 'Lunox' age: 20 ; encoded = ;decoded = ;console;/** - encoded: <Buffer 82 a4 6e 61 6d 65 a5 4c 75 6e 6f 78 a3 61 67 65 14> (17) - decoded: { name: 'Lunox', age: 20 } **/ ;encoded = ;decoded = ;console;/** - encoded: <Buffer 82 e0 a5 4c 75 6e 6f 78 e1 14> (10) - decoded: { name: 'Lunox', age: 20 } **/
minified build for browsers
<!-- latest umd build --><script src="https://unpkg.com/what-the-pack/dist/MessagePack.min.js"></script> <!-- exposed as 'MessagePack' --><script> const encode decode = MessagePack; // 4MB const data = name: 'Lunox' age: 20 ; const encoded = ; const decoded = ; console;</script>
using with browser websockets
server
const WebSocket = ;const MessagePack = ;const encode decode = MessagePack; // 4MB const wss = /- options go here */;wss;
client
- On browsers,
Buffer
object is exposed asMessagePack.Buffer
- On browsers, call
MessagePack.Buffer.from(x)
on received ArrayBuffers
// Create WebSocket connection.const socket = 'ws://localhost:8080';const encode decode Buffer = MessagePack; // 4MB // Connection openedsocket; // Listen for messagessocket;
benchmarks
yarn run benchmark
$ yarn run benchmarkJSON stringify tiny x 1,477,866 ops/sec ±0.58% JSON stringify small x 232,645 ops/sec ±0.25% JSON stringify medium x 117,357 ops/sec ±0.31% JSON stringify large x 24.01 ops/sec ±0.37% JSON parse tiny x 1,301,925 ops/sec ±3.18% JSON parse small x 264,410 ops/sec ±0.57% JSON parse medium x 133,865 ops/sec ±0.52% JSON parse large x 31.52 ops/sec ±0.34% what-the-pack encode tiny x 1,175,981 ops/sec ±0.39% what-the-pack encode small x 365,533 ops/sec ±0.85% what-the-pack encode medium x 173,746 ops/sec ±0.41% what-the-pack encode large x 218 ops/sec ±0.85% what-the-pack decode tiny x 1,130,260 ops/sec ±0.30% what-the-pack decode small x 254,931 ops/sec ±0.79% what-the-pack decode medium x 146,809 ops/sec ±0.79% what-the-pack decode large x 211 ops/sec ±0.37% notepack.encode tiny x 1,291,361 ops/sec ±0.22% notepack encode small x 325,882 ops/sec ±1.20% notepack encode medium x 133,398 ops/sec ±0.20% notepack encode large x 231 ops/sec ±1.65% notepack decode tiny x 1,097,597 ops/sec ±0.67% notepack decode small x 231,895 ops/sec ±0.69% notepack decode medium x 137,385 ops/sec ±2.45% notepack decode large x 210 ops/sec ±0.85%
tests
yarn run test
$ yarn run test PASS ./test.js √ fixstr √ str 8 √ str 16 √ str 32 √ zero √ positive fixint √ negative fixint √ uint 8 √ uint 16 √ uint 32 √ uint 64 √ int 8 √ int 16 √ int 32 √ int 64 √ float 32 √ float 64 √ true, false, undefined, NaN, +Infinity, -Infinity √ flat & nested empty arrays √ flat arrays √ nested arrays √ highly nested arrays √ buffers, bin8 √ buffers, bin16 √ buffers, bin32 √ arraybuffers as buffer √ typedarrays as buffer √ tiny object √ small object √ medium object √ large object console.log index.js:49 MessagePack: Setting buffer limit to 1.07 GB Test Suites: 1 passed, 1 totalTests: 31 passed, 31 totalSnapshots: 0 totalTime: 5.477sRan all test suites.Done in 6.59s.
changelog
- 1.x
- basic support
- dictionary support
- 2.0.0
- rewrite to use raw functions instead of classes
- update dev-deps
- jest test-cov @
86.06%
- statements
389/452
- branches
137/169
- functions
11/12
- lines
374/428
- statements
- 2.0.x
- fix tempBufferLength check
- rebuild for browser
- fix leak on buffer decode
references
buffer re-alloc idea
- https://github.com/darrachequesne/notepack/issues/12#issuecomment-320872590 (Manuel Astudillo)
notepack.io
- https://github.com/darrachequesne/notepack (MIT, Damien Arrachequesne)
notepack
- https://github.com/hypergeometric/notepack (MIT, Ben Shepheard)
buffer
:- https://github.com/feross/buffer (MIT, Feross Aboukhadijeh)
pretty-bytes
- https://github.com/sindresorhus/pretty-bytes (MIT, Sindre Sorhus)
MIT | @davalapar