EruDB

About

EruDB (e-roo-deebee) is a persistent, distributed, and fault-tolerant key-value data store written in ES6, powered by NodeJS, and backed by LevelDB. It features durable replication, service discovery, real-time events and superior at-rest and at-transport encryption features.

EruDB is designed and maintained by Seandon 'Eru' Mooy.

This is a truly open project! Pull requests will be merged and we will happily give out contributor access to those who contribute!

Why

Firstly, because there is no simple cross platform datastore that can be easily installed as an NPM dependency. Secondly, solutions like RethinkDB and Redis are excellent, but have large dependency chains, do not work on all platforms, are not npm-tracked dependencies, and are easily misconfigured.

Usage:

• Have a valid build chain for node-gyp
• Install: npm install --save erudb
• CLI Usage: erudb -X GET localhost:9040/database/key

Within your app:

import EruDB from 'erudb'

Client example:

const db = new EruDB.Client({
  host: '127.0.0.1',
  port: 9040
}).select('database')
 
db.get('key').then((err, value) => {
  console.log(value)
})

Server example:

const db = new EruDB.Service({
  port: 9040
  peers: [ 'addr1', 'addr2...' ]
}).ensure({
  // Ensure a database exists
  databases: [ 'database' ],
  // Ensure a record exists
  records: {
    database: [
      { name: 'Sample Record' }
    ]
  }
})

Note that all Services are also Clients!

const db = new EruDB.Service({
  port: 9040
}).select('database').get('key').then((err, value) => {
  console.log(value)
})

EruDB also understands service discovery - Service can simply accept a list of peers - but you may also use load balancing (or DNS round-robbin) and an expected number of peers and EruDB will sort that out for you as well!

const db = new EruDB.Service({
  port: 9040,
  expectedPeerCount: 3,
  peers: {
    loadBalancer: 'load_balancer:port'
  }
}).select('database').get('key').then((err, value) => {
  console.log(value)
})

Alternatively, a Redis cache can also be used for service discovery (works great with elasticache!):

const db = new EruDB.Service({
  port: 9040,
  peers: {
    redis: {
      host: 'some_redis',
      port: 9000,
      db: 0,
      key: 'erudb_peers'
    }
  }
}).select('database').get('key').then((err, value) => {
  console.log(value)
})

Design

Service

Concepts & Terms

Safe, Strong, Weak

A Safe operation waits for quorum and accepted replication of data. Safe operations are the slowest & most reliable of all operations in EruDB.

A Strong operation waits for the master shard to store and accept the data. It does not wait for quorum replication, but returns once the master shard has OKed the data. Strong mode is the default mode for reads and writes in EruDB.

A Weak operation does not wait for data safety. It returns immediately upon handing the data to the EruDB Service. This mode is only appropriate for cache data and should be considered volatile.

Disk

A disk is a single independent storage unit. A disk can be a physical storage medium, a logical volume, or a cloud storage service like AWS S3. Multiple instances of EruDB can write to the same disk, but each instance will be aware of the actual replication that is represented. For example, EruDB will not attempt to replicate data to another instance of EruDB which would write to the same disk as the original instance. A warning will be issued if EruDB cannot reach it's desired replication factor relative to the number of disks in the cluster. This default is 3.

Host

A host is a single independent compute unit. In the case of Docker, a host is not the container, but the physical host which the container resides on. A warning will be issued if EruDB cannot reach it's desired replication factor relative to the number of hosts in the cluster. This default is 3.

Group

A group is an association of hosts which share some single point of failure. By default, we assume that the 3rd octet of the instances IP address represents a group (10.0.0.0 and 10.0.1.0 are separate "groups"). This can be overridden by configuration or service discovery plugins. A warning will be issued if EruDB cannot reach it's desired replication factor relative to the number of groups in the cluster. This default is 3.

Region

A region an association of hosts which share a geographical location. Regions, by default, use public IP detection to determine geographically distinct hosts, and can be overridden by configuration or discovery plugins. A warning will be issued if EruDB cannot reach it's desired replication factor relative to the number of regions in the cluster. This default is 3.

Index

An Index is not a natively supported feature of EruDB. EruDB is a key-value store. However, EruDB will feature ElasticSearch and Apache Spark integration for it's query language, allowing full text indexing of your data store.

Limited by design

"Limited by design" is an important philosophy of EruDB. Here are some things you cannot do with EruDB:

1. You cannot use plaintext communication, ever. EruDB instances are secure.
2. You cannot avoid authorization, ever. EruDB instances are auditable.
3. You cannot insert invalid data, ever. EruDB instances are sane.
4. You cannot disable warnings for clusters which are not highly available by design. EruDB clusters are resiliant.

Configuration

Some settings can be changed in realtime via the management interface, but this is highly discouraged. Instead, treat EruDB like an immutable web service and launch new instances with new configuration and destroy old instances when ready.

Administration

Discovery

Security

Fault Tolerance

Logging

Client

Strong and Weak writes

Strong and Weak reads

Events

Query syntax

Indexing tips