set-sparse

    2.0.0 • Public • Published

    set-sparse

    A simple function that sets the FSCTL_SET_SPARSE or FSCTL_SET_ZERO_DATA windows flag via DeviceIoControl.

    Why

    If you're creating large files with large sections filled with zeros (like in torrents, or hypercores) sometimes you don't want to take up the full size of the file even though you've only written 100kb. This sets the flag in Windows to "compress" the zeros. Mac and Linux do this by default.

    this is a file with a lot of zeros, notice the Size vs Size on disk

    Usage

    const sparse = require('set-sparse')
    const create_if_nonexistant = true
    
    const success = sparse.setSparse('./file-to-set', create_if_nonexistant)
    // you can now write to sections of your file
    const { open } = require('fs/promises')
    const fd = await open('./file-to-set', 'r+')
    await fd.write(Buffer.from([1]), 0, 1, 10000000) // 10mb file
    fd.close() // file is *still small!*
    
    // And if we start with a dense file
    const { open, stat } = require('fs/promises')
    const fd = await open('./other-file', 'w+')
    const TEN_MB = Buffer.alloc(10000000, 'repeating-bytes')
    await fd.write(TEN_MB, 0, TEN_MB.length, 0) // 10mb file
    fd.close() // file is TEN_MB even set as sparse
    sparse.setSparse('./other-file') // still 10mb
    // punch a hole leaving 1 non-zero byte on either side
    sparse.holePunch('./other-file', 1, TEN_MB.length)
    await stat('./other-file') // reclaim all those zeros, saving space!

    On Mac and Linux this library is a no-op and will always return false.

    Caveats

    • There is no Windows API to turn off sparse mode.
    • Setting the sparse flag does not tell Windows to go clear up space. You should set the flag before writing data to the file.
    • Do not open the file in node with await open(file, 'w+') this will disable sparse mode. Use a+ or r+.

    API

    setSparse(filepath[, create_if_nonexistant])

    • filepath String: a relative or absolute path.
    • create_if_nonexistant Boolean = false: if setSparse should create the file if it doesn't exist

    This function is synchronous but not dependent on the file's existing size and is generally very fast.

    Returns true if operation succeeded.

    holePunch(filepath, start_byte, beyond_final_zero)

    • filepath String: a relative or absolute path.
    • start_byte Number: the starting (inclusive) byte to set to zero
    • beyond_final_zero Number: the ending (exclusive) byte to set to zero

    This function is synchronous and runtime (whilst good) is dependent on the size of the new hole.

    Returns true if operation succeeded.

    Development

    npm install --global windows-build-tools
    npm install
    # hack...
    npm run build
    npm run prepack
    npm run test

    License

    MIT

    Install

    npm i set-sparse

    DownloadsWeekly Downloads

    7

    Version

    2.0.0

    License

    MIT

    Unpacked Size

    101 kB

    Total Files

    4

    Last publish

    Collaborators

    • xori