buffers provide functions for manipulating NodeJS
Install using either
# yarn$ yarn add '@react-frontend-developer/buffers'# npm$ npm install '@react-frontend-developer/buffers'
Semantics of the NodeJS buffers is not instantly obvious. A NodeJS
Uint8Array, actually, they largely share the same API, but their semantics is different here and there.
Buffer, similar to
Uint8Array is a view casted on the underlying
ArrayBuffer, but in NodeJS, a small
Buffer (small meaning
< 4KB bytes) will be casted on a bigger underlying
ArrayBuffer to prevent memory segmentation. This can be easily observed. Let's create a
Buffer with just a couple of elements:
const buffer = Buffer // Buffer <01, 02, 03>bufferlength // 3bufferbufferbyteLength // 8192bufferbyteOffset // 3432 - will be different for youBufferpoolSize // 8192
We see the
buffer property to access the underlying
ArrayBuffer. Buffer will use a shared
ArrayBuffer as long as the requested size is less than or equal
const largeBuffer = BufferlargeBufferbufferbyteLength // 4096 (and not 8192)
Please notice, that there is a difference between calling:
Buffer.alloc(20, 'a'). The former will use shared memory pool, while the latter will allocate dedicated
20 bytes. Also, when you use one
Buffer.from methods the semantics depends on the provided input. For example:
const largeBufferAlloc = BufferlargeBufferAllocbufferbyteLength // 20const largeBufferAllocUnsafe = BufferlargeBufferAllocUnsafebufferbyteLength // 8192const bufferFromSmallString = BufferbufferFromSmallStringbufferbyteLength // 8192const bufferFromRegularArray = BufferbufferFromSmallStringbufferbyteLength // 8192const uint8Array = length: 5const bufferFromUint8Array = BufferbufferFromUint8ArraybufferbyteLength // 8192const bufferFromArrayBuffer = BufferbufferFromArrayBufferbufferbyteLength // 5
The last case is the most interesting. Providing an array buffer as the input to
Buffer.from will create a
Buffer that uses the provided
ArrayBuffer as its underlying buffer (thus, no copy):
console // trueconsole // false
Yes, it is easy to get lost if you do not use
TypedArrays on a daily basis.
This is why we created this package. We got some options fixed for you and we made API a bit more descriptive so that it is easier to predict the result.
BTW: we have captured the above examples in RunKit: https://runkit.com/marcinczenko/5b44a0b41ff01e0012d13bbf so that you can play with it.
So our convenience package provides two abstractions:
Buffers provides a set of convenience methods to convert between
TypedArrays using the copy and move semantics.
TypedArrays provide convenience methods for creating
TypedArrays from Strings - something we miss in the existing API.
Remember to import
TypedArrays in your code:
// or using requireconst Buffers TypedArrays =
We also prepared for you a RunKit where you can try it immediately: https://runkit.com/marcinczenko/5b437ad7b034eb0012c56c85
You have a Buffer and want to convert it to TypedArray
Buffer is the older brother of
Uint8Array - thus, we provides two methods to conveniently move between the two:
Buffers.copyToUint8Array is super save. It takes a NodeJS
Buffer as the input and returns
Uint8Array with its own underlying
ArrayBuffer so that you do not have to worry if the Buffer is modified after conversion. You also do not have to care about buffer sizes, message pools, etc:
const buffer = Bufferconst uint8Array = Buffers
In some case, you may not want to copy. Imagine you have a large (like, really large)
Buffer and you want to use it as
Uint8Array from now on, but you do not want to copy the bytes to a new
ArrayBuffer. This is where
const largeBuffer = Bufferconst uint8Array = Buffers
You will get an
Uint8Array view on the same underlying
ArrayBuffer - thus no copying.
There is one small trap though. You have to make sure that
Buffer provided as the input to
moveToUint8Array does not use a memory pool (i.e. buffer.length === buffer.buffer.byteLength) or you will get a copy instead of a move. So, pragmatically, if your buffer is
4096+ bytes in size, you do not have to worry as this is always the case. Otherwise, it depends how your buffer was created. If it uses a pre-allocated pool, you will get a copy, otherwise, we will move. In other words, a copy may only occur for small buffers - not a big deal thus.
Let's make it clear - using
moveToUint8Array is usually a pre-mature optimization, especially when you move outside of Buffer's native environment - NodeJS. So for most of the cases, sticking with copy semantics is recommended.
So, we have a convenience method that converts
Uint8Array. What about other types like
In order to convert from
Buffer to an arbitrary
TypedArray type, you have to perform two steps:
ArrayBuffer- either by copying or moving
- Create the desired view using the resulting
ArrayBufferas the input
const buffer = Bufferconst arrayBuffer = Buffers// now create the view of your likingconst uint16Array = arrayBufferconst uint32Array = arrayBufferconst uint8Array = arrayBuffer
Of course, also here, we provide
Buffers.moveToArrayBuffer for move semantics.
TypedArray to a string
Buffer has great support for string conversions:
Buffer // Buffer <31, 32, 33, 34>Buffer // Buffer <31, 00, 32, 00, 33, 00, 34, 00>Buffer // Buffer <12, 34>Buffer // Buffer <12, 34>
See more on RunKit: https://runkit.com/marcinczenko/5b3f3974226fb000128824a6.
TypedArray however, we do not have any equivalent API. Therefore, our
TypedArrays provides some convenience methods for converting from and to strings.
TypedArrays allows you to convert any
ArrayBuffer to a string and vice versa in any encoding supported by
Bufferunderneath to perform the conversion.
And so we have
TypedArrays.string2ab to convert a string to
TypedArrays.ab2string to convert
ArrayBuffer to a string.
We found that conversions to an from
Uint8Array are the most frequent. For this reason in
TypedArrays we included two convenience methods: