better-fd-slicer
Safe fs.ReadStream
and fs.WriteStream
using the same fd.
Let's say that you want to perform a parallel upload of a file to a remote
server. To do this, we want to create multiple read streams. The first thing
you might think of is to use the {start: 0, end: 0}
API of
fs.createReadStream
. This gives you two choices:
- Use the same file descriptor for all
fs.ReadStream
objects. - Open the file multiple times, resulting in a separate file descriptor for each read stream.
Neither of these are acceptable options. The first one is a severe bug,
because the API docs for fs.write
state:
Note that it is unsafe to use
fs.write
multiple times on the same file without waiting for the callback. For this scenario,fs.createWriteStream
is strongly recommended.
fs.createWriteStream
will solve the problem if you only create one of them
for the file descriptor, but it will exhibit this unsafety if you create
multiple write streams per file descriptor.
The second option suffers from a race condition. For each additional time the file is opened after the first, it is possible that the file is modified. So in our parallel uploading example, we might upload a corrupt file that never existed on the client's computer.
This module solves this problem by providing createReadStream
and
createWriteStream
that operate on a shared file descriptor and provides
the convenient stream API while still allowing slicing and dicing.
This module also gives you some additional power that the builtin
fs.createWriteStream
do not give you. These features are:
- Emitting a 'progress' event on write.
- Ability to set a maximum size and emit an error if this size is exceeded.
- Ability to create an
FdSlicer
instance from aBuffer
. This enables you to provide API for handling files as well as buffers using the same API.
Usage
import {createFromFd} from 'better-fd-slicer';
import fs from 'node:fs';
fs.open("file.txt", 'r', function (err, fd) {
if (err) throw err;
const slicer = createFromFd(fd);
const firstPart = slicer.createReadStream({start: 0, end: 100});
const secondPart = slicer.createReadStream({start: 100});
const firstOut = fs.createWriteStream("first.txt");
const secondOut = fs.createWriteStream("second.txt");
firstPart.pipe(firstOut);
secondPart.pipe(secondOut);
});
You can also create from a buffer:
import {createFromBuffer} from 'better-fd-slicer';
import fs from 'node:fs';
const slicer = createFromBuffer(someBuffer);
const firstPart = slicer.createReadStream({start: 0, end: 100});
const secondPart = slicer.createReadStream({start: 100});
const firstOut = fs.createWriteStream("first.txt");
const secondOut = fs.createWriteStream("second.txt");
firstPart.pipe(firstOut);
secondPart.pipe(secondOut);
API Documentation
fdSlicer.createFromFd(fd, [options])
import {createFromFd} from 'better-fd-slicer';
fs.open("file.txt", 'r', function (err, fd) {
if (err) throw err;
const slicer = createFromFd(fd);
// ...
});
Make sure fd
is a properly initialized file descriptor. If you want to
use createReadStream
make sure you open it for reading and if you want
to use createWriteStream
make sure you open it for writing.
options
is an optional object which can contain:
-
autoClose
- if set totrue
, the file descriptor will be automatically closed once the last stream that references it is closed. Defaults tofalse
.ref()
andunref()
can be used to increase or decrease the reference count, respectively.
fdSlicer.createFromBuffer(buffer, [options])
import {createFromBuffer} from 'better-fd-slicer';
const slicer = createFromBuffer(someBuffer);
// ...
options
is an optional object which can contain:
-
maxChunkSize
- ANumber
of bytes. seecreateReadStream()
. If falsey, defaults to unlimited.
Properties
fd
The file descriptor passed in. undefined
if created from a buffer.
Methods
createReadStream(options)
Available options
:
-
start
- Number. The offset into the file to start reading from. Defaults to 0. -
end
- Number. Exclusive upper bound offset into the file to stop reading from. -
highWaterMark
- Number. The maximum number of bytes to store in the internal buffer before ceasing to read from the underlying resource. Defaults to 16 KB. -
encoding
- String. If specified, then buffers will be decoded to strings using the specified encoding. Defaults tonull
.
The ReadableStream that this returns has these additional methods:
-
destroy(err)
- stop streaming.err
is optional and is the error that will be emitted in order to cause the streaming to stop. Defaults tonew Error("stream destroyed")
.
If maxChunkSize
was specified (see createFromBuffer()
), the read stream
will provide chunks of at most that size. Normally, the read stream provides
the entire range requested in a single chunk, but this can cause performance
problems in some circumstances.
See thejoshwolfe/yauzl#87.
createWriteStream(options)
Available options
:
-
start
- Number. The offset into the file to start writing to. Defaults to 0. -
end
- Number. Exclusive upper bound offset into the file. If this offset is reached, the write stream will emit an 'error' event and stop functioning. In this situation,err.code === 'ETOOBIG'
. Defaults toInfinity
. -
highWaterMark
- Number. Buffer level whenwrite()
starts returning false. Defaults to 16KB. -
decodeStrings
- Boolean. Whether or not to decode strings into Buffers before passing them to_write()
. Defaults totrue
.
The WritableStream that this returns has these additional methods:
-
destroy()
- stop streaming
And these additional properties:
-
bytesWritten
- number of bytes written to the stream
And these additional events:
- 'progress' - emitted when
bytesWritten
changes.
read(buffer, offset, length, position, callback)
Equivalent to fs.read
, but with concurrency protection.
callback
must be defined.
write(buffer, offset, length, position, callback)
Equivalent to fs.write
, but with concurrency protection.
callback
must be defined.
ref()
Increase the autoClose
reference count by 1.
unref()
Decrease the autoClose
reference count by 1.
Events
'error'
Emitted if fs.close
returns an error when auto closing.
'close'
Emitted when FdSlicer closes the file descriptor due to autoClose
. Never
emitted if created from a buffer.
Credits
This library is heavily based on an excellent fd-slicer library, and uses some of it (modernized and improved) code.