A cdb implementation in node.js, supporting both read and write capabilities, using 64bit pointers and es6 features.
Notice that the pointers were increased to 64 bits to allow larger database. The hash-size also supports 64 bits - cdb's default hash-function gives results of 32 bits, but other hash functions could be used instead. Key-Length and Data-Length remain 4 bytes (32 bits) - this allows only 4GB for each key and each value, but saves space if the database contains lots of short key-value pairs (which is the typical use-case).
npm install constant-db64
Changes from original v2.0.0
- Replacing error-first-callbacks with promises using async-await
- Writable is not an EventEmitter
- Using 64 bits for pointers and hash-values
- Writable and Readable are classes and therefore begin with a capital letter
- Converting keys to buffers instead of hashing utf8 strings directly (with
- New default hash function that uses all 64bits
- Two Raw-Data-Readers: files, buffers. Users can implement their own data reader (i.e. for online storage buckets)
- Optional cache-wrapper for raw-data readers.
Changes from v1.0.0
- Dropped promise support
- Completely rewritten!
get()calls should be much faster.
const Writable = Writable;const writer = './cdbfile';await writer;writer;await writer;console;
const readable = readable;const reader = './cdbfile';await reader;const data = await reader;console; // results in 'hello world!'await reader;console;
To create a new readable instance:
const constantDb = ;const reader = filename;
You can choose a different hash function by calling:
Your hash function must return a BigInt.
For faster results that (using cache):
const cacheReader = constandDbrawDataReaders;const reader = cacheReader
new RawDataReaderCacheWrapper(filename, options) can be called with the following options:
Opens the file (calls the raw-reader's
open() function), and immediately caches the header table for the cdb (4098 bytes).
Attempts to find the specified key, the data
Buffer for that key (if found) or undefined (if not found). If an offset is specified, the cdb will return data for the nth record matching that key.
Returns an async iterator (which also implements
AsyncIterable), for finding multiple values for the same key. This should be slightly faster than calling
get() with an offset.
Closes the file (calls the raw-reader's
close() function). No more records can be read after closing.
Custom Raw-Data Reader
You can also implement your own "Raw-Data Reader" (for example, to read data from an online storage bucket). Such object should have the following fields:
- required: async
read(start, length)function that returns a
- optional: async
open()function that will be called when the
- optional: async
close()function that will be called when the
To use your raw-data reader, pass it as the first param to:
new Readable(myRawDataReader) (instead of
filename), or pass it as the first param to
new RawDataReaderCacheWrapper(myRawDataReader) (instead of
To create a new Writable instance:
Unlike raw-data readers, the library does not support custom "raw-data writers" (i.e. for writing to an online storage bucket instead of a local file) because creating a constant-db file is more complicated than reading from a constant-db file. In a typical use-case of a constant database you would want to create the database file locally, upload it to your online-storage, and use it with your custom raw-data reader.
Opens the file for writing. This will overwrite any file that currently exists, or create a new one if necessary.
Writes a record to the cdb.
Finalizes the cdb and closes the file. Calling
close() is necessary to write out the header and subtables required for the cdb!
npm run benchmark