Naughty Programmer's Madness

    TypeScript icon, indicating that this package has built-in type declarations

    2.0.1 • Public • Published


    buffers provide functions for manipulating NodeJS Buffer and JavaScript ArrayBuffer.


    Install using either yarn or npm:

    # yarn
    $ yarn add '@react-frontend-developer/buffers'
    # npm
    $ npm install '@react-frontend-developer/buffers'

    Using Buffer

    Semantics of the NodeJS buffers is not instantly obvious. A NodeJS Buffer is similar to a JavaScript Uint8Array, actually, they largely share the same API, but their semantics is different here and there.

    A Buffer, similar to Uint8Array is a view casted on the underlying ArrayBuffer. In JavaScript, the (byte) size of the view is the same as the (byte) size of the underlying ArrayBuffer, but in NodeJS, a small Buffer (small meaning < 4KB bytes) will be casted on a bigger underlying ArrayBuffer to prevent memory segmentation. This can be easily observed. Let's create a Buffer with just a couple of elements:

    const buffer = Buffer.from([1,2,3]) // Buffer <01, 02, 03>
    buffer.length                       // 3
    buffer.buffer.byteLength            // 8192
    buffer.byteOffset                   // 3432 - will be different for you
    Buffer.poolSize                     // 8192

    We see the buffer property to access the underlying ArrayBuffer. Buffer will use a shared ArrayBuffer as long as the requested size is less than or equal 4096 bytes:

    const largeBuffer = Buffer.allocUnsafe(4096).fill('a')
    largeBuffer.buffer.byteLength       // 4096 (and not 8192)

    Please notice, that there is a difference between calling: Buffer.allocUnsafe(20).fill('a') and Buffer.alloc(20, 'a'). The former will use shared memory pool, while the latter will allocate dedicated buffer of 20 bytes. Also, when you use one Buffer.from methods the semantics depends on the provided input. For example:

    const largeBufferAlloc = Buffer.alloc(20, 'a')
    largeBufferAlloc.buffer.byteLength  // 20
    const largeBufferAllocUnsafe = Buffer.allocUnsafe(20).fill('a')
    largeBufferAllocUnsafe.buffer.byteLength  // 8192
    const bufferFromSmallString = Buffer.from('Short...')
    bufferFromSmallString.buffer.byteLength   // 8192
    const bufferFromRegularArray = Buffer.from([1,2,3])
    bufferFromSmallString.buffer.byteLength   // 8192
    const uint8Array = new Uint8Array({length: 5})
    const bufferFromUint8Array = Buffer.from(uint8Array)
    bufferFromUint8Array.buffer.byteLength    // 8192
    const bufferFromArrayBuffer = Buffer.from(uint8Array.buffer)
    bufferFromArrayBuffer.buffer.byteLength   // 5

    The last case is the most interesting. Providing an array buffer as the input to Buffer.from will create a Buffer that uses the provided ArrayBuffer as its underlying buffer (thus, no copy):

    console.log(uint8Array.buffer === bufferFromArrayBuffer.buffer) // true
    console.log(uint8Array.buffer === bufferFromUint8Array.buffer)  // false

    Yes, it is easy to get lost if you do not use Buffers and TypedArrays on a daily basis.

    This is why we created this package. We got some options fixed for you and we made API a bit more descriptive so that it is easier to predict the result.

    BTW: we have captured the above examples in RunKit: so that you can play with it.

    Using buffers

    So our convenience package provides two abstractions: Buffers and TypedArrays. Buffers provides a set of convenience methods to convert between Buffers and TypedArrays using the copy and move semantics. TypedArrays provide convenience methods for creating TypedArrays from Strings - something we miss in the existing API.


    Remember to import Buffers and TypedArrays in your code:

    import { Buffers, TypedArrays } from '@react-frontend-developer/buffers'
    // or using require
    const { Buffers, TypedArrays } = require('@react-frontend-developer/buffers')

    We also prepared for you a RunKit where you can try it immediately:

    You have a Buffer and want to convert it to TypedArray

    So Buffer is the older brother of Uint8Array - thus, we provide two methods to conveniently move between the two: Buffers.copyToUint8Array and Buffers.moveToUint8Array. Buffers.copyToUint8Array is super save. It takes a NodeJS Buffer as the input and returns Uint8Array with its own underlying ArrayBuffer so that you do not have to worry if the Buffer is modified after conversion. You also do not have to care about buffer sizes, message pools, etc:

    const buffer = Buffer.alloc(20, 'a')
    const uint8Array = Buffers.copyToUint8Array(buffer)

    In some case, you may not want to copy. Imagine you have a large (like, really large) Buffer and you want to use it as Uint8Array from now on, but you do not want to copy the bytes to a new ArrayBuffer. This is where Buffers.moveToUint8Array comes:

    const largeBuffer = Buffer.allocUnsafe(4096).fill('a')
    const uint8Array = Buffers.moveToUint8Array(buffer)

    You will get an Uint8Array view on the same underlying ArrayBuffer - thus no copying.

    There is one small trap though. You have to make sure that Buffer provided as the input to moveToUint8Array does not use a memory pool (i.e. buffer.length === buffer.buffer.byteLength) or you will get a copy instead of a move. So, pragmatically, if your buffer is 4096+ bytes in size, you do not have to worry as this is always the case. Otherwise, it depends how your buffer was created. If it uses a pre-allocated pool, you will get a copy, otherwise, we will move. In other words, a copy may only occur for small buffers - not a big deal thus.

    Let's make it clear - using moveToUint8Array is usually a pre-mature optimization, especially when you move outside of Buffer's native environment - NodeJS. So for most of the cases, sticking with copy semantics is recommended.

    Other TypedArray types

    So, we have a convenience method that converts Buffer to Uint8Array. What about other types like Uint16Array or Uint32Array?

    In order to convert from Buffer to an arbitrary TypedArray type, you have to perform two steps:

    1. Convert Buffer to ArrayBuffer - either by copying or moving
    2. Create the desired view using the resulting ArrayBuffer as the input

    For instance:

    const buffer = Buffer.from('Some string...', 'utf16le')
    const arrayBuffer = Buffers.copyToArrayBuffer(buffer)
    // now create the view of your liking
    const uint16Array = new Uint16Array(arrayBuffer)
    const uint32Array = new Uint32Array(arrayBuffer)
    const uint8Array = new Uint8Array(arrayBuffer)

    Of course, also here, we provide Buffers.moveToArrayBuffer for move semantics.

    Converting TypedArray to a string

    Buffer has great support for string conversions:

    Buffer.from('1234', 'utf8')       // Buffer <31, 32, 33, 34>
    Buffer.from('1234', 'utf16le')    // Buffer <31, 00, 32, 00, 33, 00, 34, 00>
    Buffer.from('1234', 'hex')        // Buffer <12, 34>
    Buffer.from('\u00124', 'binary')  // Buffer <12, 34>

    See more on RunKit:

    For TypedArray however, we do not have any equivalent API. Therefore, our TypedArrays provides some convenience methods for converting from and to strings.

    TypedArrays allows you to convert any ArrayBuffer to a string and vice versa in any encoding supported by Buffer.from and Buffer.toString().

    TypedArrays is using Buffer underneath to perform the conversion.

    And so we have TypedArrays.string2ab to convert a string to ArrayBuffer and TypedArrays.ab2string to convert ArrayBuffer to a string.

    We found that conversions to an from Uint8Array are the most frequent. For this reason in TypedArrays we included two convenience methods: string2Uint8Array and uint8Array2string.


    npm i @react-frontend-developer/buffers

    DownloadsWeekly Downloads






    Unpacked Size

    56.5 kB

    Total Files


    Last publish


    • everydayproductive
    • markspanbroek
    • jeroenknoops