symdb

2.3.13 • Public • Published

symdb

A JSON database that uses symbolic links for indexing

reasoning

There are a lot of JSON databases available on npm. Of the ones I investigated, most store the objects for a collection in a single json file. Upon loading a collection, the whole json file is loaded in to memory. While this is probably the fastest method for accessing and updating objects in a collection, it could be problematic for large collections. It also does not really lend itself to replication in an easy way.

goals

  • Use the filesystem
    • each object should be stored in their own .json file
    • directories and symbolic links should be used for indexing

example

const SymDb = require('symdb');

const db = new SymDb({ root : './db' });

const Product = db.Model('product', {
    product_id : Number
    , name : String
    , description : String
    , type : String
});

async function go() {
    let obj = await Product.add({
        product_id : 1
        , name : 'Test'
        , type : 'test-product'
    });

    //you'll notice that the object now has a ._id attribute that is a uuid
    console.log(obj); 

    let results = await Product.get({ type : 'test-product' });

    //results is an array of objects whose type value is 'test-product'
    console.log(results);
}

go();

api

symdb = new SymDb(opts)

  • opts.root - string - the path to the root directory in which database files should be stored

Model = symdb.Model(name, schema)

  • name - string - the name of the model/collection
  • schema - object - an object which contains key:Type pairs
    • the Types are generally, String, Number, or some other function that will format the value to how you want it indexed.
    • NOTE: this is not thoroughly tested and needs love

Model.get(lookup[, context][, callback]) => Promise

let results = Model.get({
    weight : SymDb.gt(42)
});

// also these
SymDb.gt(10)
SymDb.gte(10)
SymDb.lt(9)
SymDb.lte(9)
SymDb.startsWith('bart')
SymDb.contains('bart')
SymDb.between(1, 10)
SymDb.contains(['a','b', 'c'])
SymDb.compare(function (z) { return z === 1234 })

Model.getSync(lookup[, context]) => Array

Model.add(obj[, context][, callback]) => Promise

Model.addSync(obj[, context][, callback]) => Object

Model.update(obj[, context][, callback]) => Promise

Model.updateSync(obj[, context][, callback]) => Object

Model.del(obj[, context][, callback]) => Promise

Model.delSync(obj[, context][, callback]) => Object

Model Events

Example:

Model.on('update:before', (event, cb) => {
    //cb must be called when done;

    event.data.password = null;

    return cb();
});

Callback with an error to prevent the operation from continuing

Model.on('add:before', (event, cb) => {
    if (!event.user.canAdd) {
        return cb(new Error('user does not have add permissions'));
    }

    return cb();
});

try {
    let obj = await Model.add({ href : 'https://www.google.com' }, { user : { canAdd : false } });
}
catch (e) {
    //should have thrown 'user does not have add permissions'
}

Model.on('get:before', (event, cb) => {})

Model.on('get:after', (event, cb) => {})

Model.on('get-sync:before', (event, cb) => {})

Model.on('get-sync:after', (event, cb) => {})

Model.on('add:before', (event, cb) => {})

Model.on('add:after', (event, cb) => {})

Model.on('add-sync:before', (event, cb) => {})

Model.on('add-sync:after', (event, cb) => {})

Model.on('update:before', (event, cb) => {})

Model.on('update:after', (event, cb) => {})

Model.on('update-sync:before', (event, cb) => {})

Model.on('update-sync:after', (event, cb) => {})

Model.on('delete:before', (event, cb) => {})

Model.on('delete:after', (event, cb) => {})

Model.on('delete-sync:before', (event, cb) => {})

Model.on('delete-sync:after', (event, cb) => {})

Model.on('save:before', (event, cb) => {})

Model.on('save:after', (event, cb) => {})

Model.on('save-sync:before', (event, cb) => {})

Model.on('save-sync:after', (event, cb) => {})

todo

  • [ ] docs
  • [ ] wildcard lookups
  • [ ] case-insensitive lookups
  • [ ] range lookups
  • [x] lookups on non-indexed attributes
  • [x] deep attribute indexing
  • [ ] fulltext search
  • [ ] fix cleanup of empty index directories
  • [x] rewrite .update() handling to not call delete() then save()
  • [x] paging
  • [x] sorting
  • [ ] https://github.com/davedoesdev/getdents
  • [ ] automatic blob storage (Buffers, ReadStreams, SymDbFile)
    • [x] Buffers
    • [ ] Readable Streams
    • [ ] SymDbFile (a wrapper around a long string to be stored in a file outside of the json object)
    • [ ] need to handle deleting blobs on update:before
    • [ ] toggle blobs on/off per db/model
  • [ ] change on-disk format to have a wrapping json object that contains metadata
    • [ ] does the object have blobs?
    • [ ] if so, which keys?
    • [ ] keep symbolic links references in the metadata?
  • [x] synchronous versions of all model operations

license

MIT

Package Sidebar

Install

npm i symdb

Weekly Downloads

1

Version

2.3.13

License

MIT

Unpacked Size

102 kB

Total Files

21

Last publish

Collaborators

  • wankdanker