newtondb

Newton

⚠️ This package is under active development: Compatibility and APIs may change.

A simple, easy to use and extendible JSON database.

Table of contents

• Introduction
  • Key features
• Installation
• Basic usage
• Basic principles
  • Adapters
  • Collections
  • Indexing
  • Chaining
  • Committing mutations
• Adapters
  • MemoryAdapter
  • FileAdapter
• Database
  • new Database(options)
  • .read()
  • .write()
  • .$
  • .data
  • .observe()
  • .unobserve()
• Collections
  • new Collection(options)
  • .get()
  • .find()
  • .data
  • .count
  • .exists
  • .select()
  • .insert()
  • .set()
  • .replace()
  • .or
    • Upserting data
  • .delete()
  • .orderBy()
  • .limit()
  • .offset()
  • .commit()
  • .assert()
  • .observe()
  • .unobserve()
• Querying
  • By primary key
  • By function
  • Basic conditions
  • Advanced conditions
    • every and some
    • not
    • Operators
      • equal
      • notEqual
      • lessThan
      • lessThanInclusive
      • greaterThan
      • greaterThanInclusive
      • in
      • notIn
      • contains
      • doesNotContain
      • startsWith
      • endsWith
      • matchesRegex
      • doesNotMatchRegex
    • Preprocessors
      • toUpper
      • toLower
      • toString
      • toNumber
      • toLength
      • substring
• Guides and concepts
  • Type inference
• License

Introduction

JSON is central to Javascript and Typescript development. It's commonly used when you need to transfer data between one medium and another, such as when you're consuming and sending data to and from APIs and when persisting and hydrating application data to a remote store (be it the file system, an S3 bucket, session storage, local storage, etc.)

Most of the time, Javascript's Object and Array prototype methods are sufficient when interfacing with your data, however there are times when you may need to interface with a JSON data source as you might a more traditional database. You might find yourself needing to:

• Performantly query large data sets (of which arrays are notoriously poor for).
• Safely execute serializable queries (e.g. user defined queries).
• Safely execute data transformations.
• Set up observers to listen to changes in your data.
• Automatically hydrate from and persist changes to a remote store (e.g. file system, local storage, s3 bucket, etc.)

That's where Newton can help out.

Key features

Although Newton doesn't aim to replace a traditional database, it does borrow on common features to let you interact with your data more effectively. It does this by providing:

• A serializable query language to query your data.
• Adapters to read and write to commonly used stores (filesystem, s3, local/session storage, etc.)
• Indices - primary, secondary and sort indexes to improve the efficiency of reads.
• Serializable data transformations.
• Query caching, eager/lazy loading.
• Transactions.
• Observers (hooks).

and more. Above everything else, Newton's mission is to allow you to interface with your data while optimizing for performance and extendability.

Back to top

Installation

Using npm:

$ npm install newtondb

Or with yarn:

$ yarn add newtondb

Back to top

Basic usage

Using a single collection:

import { Database } from "newtondb";

const scientists = [
  { name: "Isaac Newton", born: "1643-01-04T12:00:00.000Z" },
  { name: "Albert Einstein", born: "1879-03-14T12:00:00.000Z" },
];
const db = new Database(scientists);

db.$.get({ name: "Isaac Newton" }).data;

// => { name: "Isaac Newton", born: "1643-01-04T12:00:00.000Z" }

Using multiple collections:

import { Database } from "newtondb";

const db = {
  scientists: [
    { name: "Isaac Newton", born: "1643-01-04T12:00:00.000Z" },
    { name: "Albert Einstein", born: "1879-03-14T12:00:00.000Z" }
  ],
  universities: [
    { name: "University of Zurich", location: "Zurich, Switzerland" }
  ]
];
const db = new Database(db);

db.$.universities.get({ location: "Zurich, Switzerland" }).data;

// => { name: "University of Zurich", location: "Zurich, Switzerland" }

Back to top

Basic principles

Adapters

An Adapter is what Newton uses to read and write to a data source. In simple terms, an Adapter is merely an instance of class with both a read and write method to be able to read from and write changes to your data source.

When instantiating Newton, you can either pass through an explicit instance of an Adapter, or you can pass through your data directly and Newton will attempt to infer and instantiate an adapter on your behalf using the following rules:

• If an array of objects, or an object whose properties are all arrays is passed through, Newton will instantiate a new MemoryAdapter instance.
• If a file path is passed through, Newton will instantiate a new FileAdapter instance.

You can extend Newton by creating your own Adapters and passing instances of those adapters through when you instantiate Newton.

Back to top

Collections

When thinking of data sources expressed in JSON, you will often have arrays/lists of data objects of a given type:

[
  { "name": "Isaac Newton", "born": "1643-01-04T12:00:00.000Z" },
  { "name": "Albert Einstein", "born": "1879-03-14T12:00:00.000Z" }
]

We define this as a Collection, where a Collection can have a type (in the above example we define a Collection of type Scientist).

One might also have a JSON data structure that defines various arrays of data of different types:

{
  "scientists": [
    { "name": "Isaac Newton", "born": "1643-01-04T12:00:00.000Z" },
    { "name": "Albert Einstein", "born": "1879-03-14T12:00:00.000Z" }
  ],
  "universities": [
    { "name": "University of Zurich", "location": "Zurich, Switzerland" }
  ]
}

We define this as a Database which contains two collections:

1. scientists of type Scientist
2. universities or type University

Newton will take as input either a single Collection or a Database with one or more collections.

Back to top

Indexing

Newton operates on arrays/lists of data. However, there are performance implications when operating on arrays that starts to become more troublesome as the size of your dataset grows. Namely, when given a query or a predicate, internally you have to iterate over the entire list to determine which objects match your predicate.

Newton solves this by internally maintaining both a linked list and a set of hash maps to efficiently query your data.

For example, most data sources will often have a primary key composed of one or more attributes that uniquely identifies the item:

[
  { "code": "isa", "name": "Isaac Newton", "university": "berlin" },
  { "code": "alb", "name": "Albert Einstein", "university": "cambridge" }
]

When instantiating newton, if you set the primaryKey configuration option to ["code"], a hash map would be created internally with code as the key, so that when you were to query it, newton could return the record from a single map lookup rather than iterating over the entire list:

$.get("isa").data;

// => { "code": "isa", "name": "Isaac Newton", "university": "berlin" }

You can configure one or more secondary indexes to maintain hashmaps for attributes that are commonly queried. For example, if you had a data source of 20,000 scientists, and you often queried against universities, you may want to create a secondary index for the university attribute. When you then executed the following query:

$.find({ university: "berlin", isAlive: true });

Rather than iterating over all 20,000 records, newton would instead iterate over the records in the hashmap with university as the hash (in which there might only be 100 records). You can set up multiple secondary indexes to increase performance even more as your dataset grows.

Back to top

Chaining

Newton functions using a concept of operation chaining, where the data output from one operation feeds in as input to the subsequent operation. For example, when updating a record, Newton's update function doesn't take as input a query of records to update against. Rather, if you wanted to update a set of records that matched a particular query, you would first find those records and then call set:

// update all records where "university" = 'berlin' to "university" = 'University of Berlin'
$.find({ university: "berlin" }).set({ university: "University of Berlin" });

This allows you to set up complex chains and transformations on your data.

Back to top

Committing mutations

Mutations are only persisted to the original data source when .commit is called on your chain.

$.scientists
  .find({ university: "berlin" })
  .set({ university: "University of Berlin" }).data;

// => [ { "code": "isa", "name": "Isaac Newton", "university": "University of Berlin" } ]

In the above example, the university attribute for all scientists studying at the "berlin" university is set to "University of Berlin", and you can access that data through the .data property. However, if you were to then query for scientists attending the "University of Berlin" you would receive an empty result:

$.scientists.find({ university: "University of Berlin" }).data;

// => []

In order to persist mutations within your chain to the original data source, you must call .commit:

$.scientists
  .find({ university: "berlin" })
  .set({ university: "University of Berlin" })
  .commit(); // commits the mutations defined in the chain

You can then query against the updated items:

$.scientists.find({ university: "University of Berlin" }).count;

// => 1

Back to top

Adapters

MemoryAdapter

Reads an object directly from memory.

Usage:

import { Database } from "newtondb";
import { MemoryAdapter } from "newtondb/adapters/memory-adapter";

const adapter = new MemoryAdapter({
  scientists: [
    { code: "isa", name: "Isaac Newton", university: "berlin" },
    { code: "alb", name: "Albert Einstein", university: "cambridge" },
  ],
  universities: [
    { id: "berlin", name: "University of Berlin" },
    { id: "cambridge", name: "University of Cambridge" },
  ],
});

const db = new Database(adapter);
await db.read();

db.$.scientists.find({ code: "isa" });

// => { code: "isa", name: "Isaac Newton", university: "berlin" }

Back to top

FileAdapter

Reads a JSON file from the local filesystem.

Usage:

import { Database } from "newtondb";
import { FileAdapter } from "newtondb/adapters/file-adapter";

const adapter = new FileAdapter("./db.json");
const db = new Database(adapter);
await db.read();

db.$.scientists.find({ code: "isa" });

// => { code: "isa", name: "Isaac Newton", university: "berlin" }

Back to top

Database

new Database(adapter, options)

Instantiates a new collection newton instance. The first argument takes either an Adapter instance, or a data object (in which case newton will instantiate a MemoryAdapter on your behalf).

Instantiating with an adapter:

import { Database } from "newtondb";
import { FileAdapter } from "newtondb/adapters/file-adapter";

const adapter = new FileAdapter("./db.json");
const db = new Database(adapter);
await db.load();

Instantiating with a data object:

import { Database } from "newton";

const db = new Database({
  scientists: [
    // ...
  ]
  universities: [
    // ...
  ]
});

Newton can be instantiated with either a single collection, or multiple collections. A single collection is defined by an array of objects of the same type, whereas multiple collections is defined as an object whose properties each contain an array of the same type. See: using multiple collections.

Options

The following options can be passed through to Newton:

Option	Type	Required	Default value	Description
writeOnCommit	boolean	false	true	If true, Newton will call the write() method on your adapter after each commit, persisting data mutations to your data source. Note: this option is ignored when using the MemoryAdapter.
collection	object	false	{}	Can be used to configure each collection. See below.

DatabaseCollectionOptions

When Newton is instantiated with a single collection, the DatabaseCollectionOptions object is a single instance of the CollectionOptions object. For example:

Setting collection options for a single collection:

const scientists = [
  { code: "isa", name: "Isaac Newton", university: "berlin" },
  { code: "alb", name: "Albert Einstein", university: "cambridge" },
];

const db = new Database(scientists, {
  collection: {
    primaryKey: "code",
  },
});

When instantiating Newton with multiple collections, the collection option takes the shape of an object whose properties are the same as your database shape, where each value is an instance of CollectionOptions object. For example:

Setting collection options when using multiple collections:

const scientists = [
  { code: "isa", name: "Isaac Newton", university: "berlin" },
  { code: "alb", name: "Albert Einstein", university: "cambridge" },
];

const universities = [
  { name: "University of Zurich", location: "Zurich, Switzerland" },
];

const db = new Database(scientists, {
  collection: {
    scientists: {
      primaryKey: "code",
    },
  },
});

You can configure as many collections as you like. When omitted from your options, each collection uses the default settings ({}):

const db = new Database(scientists, {
  collection: {
    scientists: {
      primaryKey: "code",
    },
    universities: {
      primaryKey: ["name", "location"],
    },
  },
});

Back to top

.read()

When reading your data from any source other than memory, you must call .read() before you can interact with your database. read() is an asynchronous function that returns a Promise when complete:

import { Database } from "newtondb";
import { FileAdapter } from "newtondb/adapters/file-adapter";

const db = new Database(new FileAdapter("./db.json"));
await db.read();

// can now interact with your db

In addition to loading your data, read() triggers some basic bootstrapping of your collections. If you try to interact with your database prior to calling read, a NotReadyError exception will be thrown:

import { Database } from "newtondb";
import { FileAdapter } from "newtondb/adapters/file-adapter";

const db = new Database(new FileAdapter("./db.json"));
db.$.find({ name: "isaac newton" }); // will throw a NotReadyError exception

Back to top

.write()

Will write the current state of your database to its source by triggering the write() method in the Adapter you instantiated the database with. Returns a Promise which will resolve to true when the write operation was successful and false when it was unsuccessful.

const db = new Database(new FileAdapter("./db.json"));
await db.read();

db.find({ name: "isaac newton" }).set({ alive: false }).commit();
await db.write();

When newton is instantiated with writeOnCommit set to true (the default option), commits will automatically be written:

const db = new Database(new FileAdapter("./db.json"), { writeOnCommit: true });
await db.read();

db.find({ name: "isaac newton" }).set({ alive: false }).commit();

// .write() is not necessary as the changes would have already been written

Back to top

.$

When newton is instantiated with a single collection, $ will return that collection instance:

Instantiating with a single collection:

const db = new Database(scientists);
db.$.find({ name: "isaac newton" });

When instantiated with multiple collections, $ will return an object whose values are collection instances:

const db = new Database({ scientists, universities });

db.$.scientists.find({ name: "isaac newton" });
db.$.universities.find({ name: "university of berlin" });

Back to top

.data

Returns the data of the entire database.

const scientists = [
  { name: "Isaac Newton", born: "1643-01-04T12:00:00.000Z" },
  { name: "Albert Einstein", born: "1879-03-14T12:00:00.000Z" },
];

const db = new Database(scientists);
db.$.find({ name: "Isaac Newton" })
  .set({ name: "Isaac Newton (deceased)" })
  .commit();

db.data;

Which returns:

[
  { "name": "Isaac Newton (deceased)", "born": "1643-01-04T12:00:00.000Z" },
  { "name": "Albert Einstein", "born": "1879-03-14T12:00:00.000Z" }
]

Back to top

.observe()

Sets up an observer which is triggered whenever CRUD operations occur on the database.

⚠️ Note: when configuring an observer at the database level, it is triggered each time any collection is updated. You can also configure observers on individual collections.

When instantiating a database with a single collection, the observe() method expects a function with a single argument of type MutationEvent:

MutationEvent:

type MutationEvent<T> = InsertEvent<T> | DeleteEvent<T> | UpdateEvent<T>;

MutationEvent is either an instance of InsertEvent, DeleteEvent or UpdateEvent:

type InsertEvent<T> = { event: "insert"; data: T };
type DeleteEvent<T> = { event: "delete"; data: T };
type UpdateEvent<T> = { event: "updated"; data: { old: T; new: T } };

When using typescript, you can narrow in on the data using the event name:

const db = new Database(scientists);
db.observe(({ event, data }) => {
  if (event === "insert") {
    // data will be of type `T` (`Scientist` in our case)
  }
});

When instantiating a database with multiple collections, observe() expects a function with two arguments, the first being the collection name and the second a MutationEvent argument:

const db = new Database({ scientists, universities });

db.observe((collection, event) => {
  console.log(`collection ${collection} triggered an event`);
});

Return

observe() returns a numeric ID of the observer. You can pass this ID to unobserve() to cancel the observer.

Back to top

.unobserve()

Takes as input a numeric ID (the output from observe()) and cancels an observer.

const db = new Database({ scientists, universities });

const observer = db.observe((collection, event) => {
  console.log(`collection ${collection} triggered an event`);

  db.unobserve(observer); // cancel after the first event
});

Throws an ObserverError exception when the observer is not found.

Back to top

Collections

new Collection(options)

Instantiates a new collection instance. The following options are supported:

Option	Type	Required	Default value	Description
primaryKey	string | string[]	false	undefined	A single property (or an array of properties for a composite key) that is used to uniquely identify a record. (id is commonly used). Not required, but will dramatically speed up read operations when querying by primary key.
copy	boolean	false	false	When mutations are committed using commit, the original data object will be updated. This can sometimes lead to unintended side effects (when using the MemoryAdapter). Set copy to true to create a deep copy of the collection data on instantiation.

Back to top

.get()

Returns a single record. Most commonly used when querying your collection by a unique identifier:

$.get({ code: "isa" }).data;

// => { "code": "isa", "name": "Isaac Newton", "university": "berlin" }

When your collection has been instantiated with a primary key, and your primary key is a single property whose value is a scalar (e.g. a string or a number), you can call .get with that scalar value and Newton will infer the fact that you're querying against your primary key:

$.get("isa").data;

// => { "code": "isa", "name": "Isaac Newton", "university": "berlin" }

You can query using a primary key, a basic condition, an advanced condition or a function.

Back to top

.find()

Returns multiple records:

$.find({ university: "cambridge" }).data;

// => [ { "code": "alb", "name": "Albert Einstein", "university": "cambridge" } ]

Will return an empty array when no results are found.

You can query using a primary key, a basic condition, an advanced condition or a function.

Back to top

.data

The data property returns an array of data as it currently exists within your chain. For example, referencing .data on the root collection will return an array of all data in your collection:

$.data;

// => [ { "name": "Isaac Newton", "born": "1643-01-04T12:00:00.000Z" }, ... ]

When you start chaining operations, .data will return an array of data as it currently exists within your chain:

$.find({ name: "Isaac Newton" }).data;

// => [ { "name": "Isaac Newton", "born": "1643-01-04T12:00:00.000Z" } ]

Back to top

.count

The count property returns the amount of records currently within your chain. When executed from the base collection, it will return the total amount of records in your collection:

$.count;

// => 100

When you start chaining operations, .count will return the amount of records that currently exist within your chain:

$.find({ name: "Isaac Newton" }).count;

// => 1

Back to top

.exists

The exists property is a shorthand for .count > 0 and simply returns true or false if there is a non-zero amount of items currently within your chain:

$.get("isa").exists;

// => true

Or when it doesn't exist:

$.get("not isaac newton").exists;

// => false

Back to top

.select()

By default, when a query returns records, the result includes all of those records' attributes. To only return a subset of an object's properties, call .select with an array of properties to return:

$.get({ name: "Isaac Newton" }).select(["university"]).data;

// => { university: "Cambridge" }

Given the result of one operation is fed into another, the order of select doesn't matter. The above will produce the same output as:

$.select(["university"]).get({ name: "Isaac Newton" }).data;

// => { university: "Cambridge" }

Back to top

.insert()

Inserts one or more records into the database.

Inserting a single record:

$.insert({
  name: "Nicolaus Copernicus",
  born: "1473-02-19T12:00:00.000Z",
}).commit();

You can insert multiple records by passing through an array of objects to insert:

$.insert([
  { name: "Nicolaus Copernicus", born: "1473-02-19T12:00:00.000Z" },
  { name: "Edwin Hubble", born: "1989-11-10T12:00:00.000Z" },
]).commit();

Back to top

.set()

Updates a set of attributes on one or more records.

// update isaac newton's college to "n/a" and set isAlive to false
$.find({ name: "Isaac Newton" })
  .set({ college: "n/a", isAlive: false })
  .commit();

set can also take as input a function whose first argument is the current value of the record, and which must return a subset of the record to update:

// uppercase all universities using .set
$.set(({ university }) => ({
  university: university.toUpperCase(),
})).commit();

⚠️ this differs from replace() in that it will only update/set the attributes passed through, whereas replace() will replace the entire document.

Back to top

.replace()

Replaces an entire document with a new document:

const newNewton = {
  name: "Isaac Newton",
  isAlive: false,
  diedOn: "1727-03-31T12:00:00.000Z",
};

$.get("Isaac Newton").replace(newNewton).commit();

replace can also take as input a function whose first argument is the current value of the record, and which must return a complete new record:

// uppercase all universities using .replace
$.replace((record) => ({
  ...record,
  university: university.toUpperCase(),
})).commit();

Back to top

.or

When a .get() or .find() operation doesn't return any data, the .or property can be used to conditionally execute methods on chain:

// will throw an Error if Isaac Newton can not be found
const isaac = $.get("Isaac Newton").or.throw();

Upsert

Importantly, or doesn't have to be used immediately after the find/get operation - this allows you to perform conditional operations such as updating an existing record or inserting a new record (upserting):

const isaac = $.get("Isaac Newton")
  .set({ university: "Trinity College" })
  .or.insert({
    id: 100,
    name: "Isaac Newton",
    university: "Trinity College",
  })
  .commit();

Back to top

.delete()

Deletes one or more records from the collection.

delete() doesn't take any arguments. Rather, it deletes the records that currently exist within the chain at the time that it's called. For example:

// delete all records from a collection
$.delete().commit();

// delete all scientists from cambridge university
$.find({ university: "cambridge" }).delete().commit();

// delete a single record
$.get("isaac newton").delete().commit();

Back to top

.orderBy()

orderBy can be used to sort records by one or more properties. It takes as input a single object whose properties are a key of your collection's properties, and whose value is either asc (for ascending) or desc (for descending).

For example, using the below dataset:

const students = [
  { name: "roger galilei", university: "mit" },
  { name: "kip tesla", university: "harvard" },
  { name: "rosalind faraday", university: "harvard" },
  { name: "thomas franklin", university: "mit" },
  { name: "albert currie", university: "harvard" },
];

To sort by university in descending order and name in ascending order:

$.orderBy({ university: "desc", name: "asc" }).data;

This will produce the following:

[
  { "name": "roger galilei", "university": "mit" },
  { "name": "thomas franklin", "university": "mit" },
  { "name": "albert currie", "university": "harvard" },
  { "name": "kip tesla", "university": "harvard" },
  { "name": "rosalind faraday", "university": "harvard" }
]

Given the order by which you sort is important, orderBy() will adhere to the order of the properties in the object passed through.

For example, in the above example, { university: "desc", name: "asc" } was passed through. orderBy would first sort by university in descending order, and then by name in ascending order.

If you were to instead pass through { name: "asc", university: "desc" }, orderBy would first sort by name in ascending order and then by university in descending order. This would produce a different result:

[
  { "name": "albert currie", "university": "harvard" },
  { "name": "kip tesla", "university": "harvard" },
  { "name": "roger galilei", "university": "mit" },
  { "name": "rosalind faraday", "university": "harvard" },
  { "name": "thomas franklin", "university": "mit" }
]

Back to top

.limit()

You can use limit to only return the first n amount of records within your chain:

$.find({ university: "cambridge" }).limit(5).data;

Will return the first 5 records with university set to "cambridge".

You can use limit with offset to implement an offset based pagination on your data.

Back to top

.offset()

offset will skip the first n records from your query. For example, to skip the first 5 records:

$.find({ university: "cambridge" }).offset(5).data;

offset can be used with limit to implement an offset based pagination:

const pageSize = 10;
const currentPage = 3;

$.find()
  .limit(pageSize)
  .offset((currentPage - 1) * pageSize).data;

Back to top

.commit()

The following operations can mutate (change) your data:

• set()
• replace()
• delete()
• insert()

Mutations will only be persisted/committed to your collection when .commit() is called. This is useful as it allows you to:

1. Perform temporary transformations on your data, and
2. Create complex chains

What's more, by requiring a call to commit Newton confirms your intent to mutate the original data source, reducing the risk for unintended side effects throughout your application.

Back to top

.assert()

Runs an assertion on your chain, and continues the chain execution if the assertion passes and raises an AssertionError when it fails.

Takes as input a function whose single argument is the chain instance and which returns a boolean:

import { AssertionError } from "newtondb";

try {
  $.get({ name: "isaac newton" })
    .assert(({ exists }) => exists)
    .set({ university: "unknown" })
    .commit();
} catch (e: unknown) {
  if (e instanceof AssertionError) {
    // record does not exist
  }
}

You can optionally pass through a string as the first argument and a function as the second to describe your assertion:

import { AssertionError } from "newtondb";

try {
  $.get({ name: "isaac newton" })
    .assert(
      "the record the user is attempting to update exists",
      ({ exists }) => exists
    )
    .set({ university: "unknown" })
    .commit();
} catch (e: unknown) {
  if (e instanceof AssertionError) {
    // record does not exist
  }
}

Back to top

.observe()

When mutations to the data source are committed, one or more of the following events will be raised:

• insert: raised when a record is inserted into the collection
• delete: raised when a record is deleted from the collection
• updated: raised when a record is updated

You can pass callbacks to the observe method that will be triggered when these events occur.

On insert:

const onInsert = $.observe("insert", (record) => {
  //
});

On delete:

const onDelete = $.observe("delete", (record) => {
  //
});

On update:

const onUpdate = $.observe("updated", (record, historical) => {
  // historical.old = item before update
  // historical.new = item after update
});

You can also pass through a wildcard observer which will be triggered on every event:

const wildcardObserver = $.observe((event, data) => {
  // event: "insert" | "delete" | "updated"
  // data: event data
});

Calls to .observe() will return an numeric id of the observer. This id should be passed to unobserve() to cancel the observer.

Back to top

.unobserve()

Cancels an observer set with the .observe() method. Takes as input a numeric ID (which should correspond to the output of the original .observe call).

Back to top

Querying

Newton allows you to query your data using through the following mechanisms:

• By primary key
• By function
• By a simple condition
• By an advanced condition

The examples in this section will use the following dataset:

[
  { "id": 1, "name": "isaac newton", "born": 1643, "alive": false },
  { "id": 2, "name": "albert einstein", "born": 1879, "alive": false },
  { "id": 3, "name": "galileo galilei", "born": 1564, "alive": false },
  { "id": 4, "name": "marie curie", "born": 1867, "alive": false },
  { "id": 5, "name": "roger penrose", "born": 1931, "alive": true },
  { "id": 6, "name": "rosalind franklin", "born": 1920, "alive": true }
]

Back to top

By primary key

When you instantiate Newton you can optionally define a primary key:

const db = new Database(scientists, { primaryKey: "id" });

⚠️ Performance warning: while primaryKey is optional, it is highly recommended you set this when you instantiate Newton in order to optimize read performance.

If the value of your primary key is a scalar value (string or number), you can query your collection by the value directly:

$.get(2).data;

// =>  { id: 3, name: 'galileo galilei', born: 1564, alive: false }

If you are using a composite primary key, you'll have to pass through an object:

const $ = new Collection(scientists, { primaryKey: ["name", "born"] });

$.get({ name: "albert einstein", born: 1879 }).data;

// => { "id": 2, "name": "albert einstein", "born": 1879, "alive": false }

Back to top

By function

A function predicate can be passed to get() and find(), which takes as input a single argument with the record, and should return true if the record passes the predicate and false if not.

For example, to return scientists who are currently alive:

$.find((record) => record.alive).data;

This will return the following:

[
  { "id": 5, "name": "roger penrose", "born": 1931, "alive": true },
  { "id": 6, "name": "rosalind franklin", "born": 1920, "alive": true }
]

⚠️ Optimization warning: Newton will have to iterate over each item in your collection to test whether or not the predicate is truthy. Where possible, you should try and use a basic or advanced condition with secondary indexes to optimize read operations.

Back to top

By basic condition

You can pass a simple key-value query to perform an exact match on items in your collection:

$.find({ alive: true }).data;

Which returns:

[
  { "id": 5, "name": "roger penrose", "born": 1931, "alive": true },
  { "id": 6, "name": "rosalind franklin", "born": 1920, "alive": true }
]

You can pass multiple properties through:

$.find({ alive: true, born: 1920 }).data;

// => [ { "id": 6, "name": "rosalind franklin", "born": 1920, "alive": true } ]

Back to top

By advanced condition

An advanced condition is an object with contains a property, an operator and a value:

$.find({
  property: "born",
  operator: "greaterThan",
  value: 1900,
}).select(["name", "born"]).data;

// => [ {"name":"roger penrose","born":1931}, {"name":"rosalind franklin","born":1920} ]

every and some

You can create complex conditions by using a combination of some and every. Both properties accept an array of conditions. some will evaluate as true if any condition within the array evaluates as true, whereas every will evaluate to true only when all conditions within the array evaluate as true.

every

You can use every to return all records that meet all of the conditions:

$.find({
  every: [
    { property: "born", operator: "greaterThan", value: 1800 },
    { property: "name", operator: "startsWith", value: "r" },
  ],
}).select(["name", "born"]).data;

This query will return all scientists who were born after the year 1800 and whose name starts with the letter r:

[
  { "name": "roger penrose", "born": 1931 },
  { "name": "rosalind franklin", "born": 1920 }
]

some

You can use some to return all records that meet any of the conditions:

$.find({
  some: [
    { property: "born", operator: "greaterThan", value: 1800 },
    { property: "name", operator: "startsWith", value: "a" },
  ],
}).select(["name", "born"]).data;

This query will return all scientists who were born after the year 1800 or whose name starts with the letter a:

[
  { "name": "albert einstein", "born": 1879 },
  { "name": "marie curie", "born": 1867 },
  { "name": "roger penrose", "born": 1931 },
  { "name": "rosalind franklin", "born": 1920 }
]

not

not can be used to return the reverse of the condition. Similar to standard Javascript, !false would return true and !true would return false.

The following returns all scientists who are alive:

$.find({
  not: {
    { property: "alive", operator: "equal", value: false }
  },
}).data;

Returned value:

[
  { "id": 5, "name": "roger penrose", "born": 1931, "alive": true },
  { "id": 6, "name": "rosalind franklin", "born": 1920, "alive": true }
]

Nesting conditions

You can nest conditions to create complex rules:

$.find({
  some: [
    {
      every: [
        { property: "born", operator: "greaterThan", value: 1800 },
        { property: "alive", operator: "equal", value: true },
      ],
    },
    {
      some: [
        { property: "name", operator: "startsWith", value: "albert" },
        { property: "name", operator: "endsWith", value: "newton" },
      ],
    },
  ],
}).select(["name", "born"]).data;

This query will return all scientists where:

1. They were born after the year 1800 and are alive, or
2. Whose first name starts with "albert" or ends with "newton":

[
  { "name": "isaac newton", "born": 1643 },
  { "name": "albert einstein", "born": 1879 },
  { "name": "roger penrose", "born": 1931 },
  { "name": "rosalind franklin", "born": 1920 }
]

Back to top

Operators

Conditions require one of the following operators:

equal

Performs a strict equality (===) match:

$.find({ property: "born", operator: "equal", value: 1643 }).data;

Returns the following:

[{ "id": 1, "name": "isaac newton", "born": 1643, "alive": false }]

Back to top

notEqual

Performs a strict inequality (!==) match:

$.find({ property: "alive", operator: "notEqual", value: true }).data;

Returns the following:

[
  { "id": 1, "name": "isaac newton", "born": 1643, "alive": false },
  { "id": 2, "name": "albert einstein", "born": 1879, "alive": false },
  { "id": 3, "name": "galileo galilei", "born": 1564, "alive": false },
  { "id": 4, "name": "marie curie", "born": 1867, "alive": false }
]

Back to top

startsWith

Checks if a string starts with a given value.

$.find({ property: "name", operator: "startsWith", value: "ro" }).data;

Returns the following:

[
  { "id": 5, "name": "roger penrose", "born": 1931, "alive": true },
  { "id": 6, "name": "rosalind franklin", "born": 1920, "alive": true }
]

Back to top

endsWith

Checks if a string ends with a given value.

$.find({ property: "name", operator: "endsWith", value: "n" }).data;

Returns the following:

[
  { "id": 1, "name": "isaac newton", "born": 1643, "alive": false },
  { "id": 2, "name": "albert einstein", "born": 1879, "alive": false },
  { "id": 6, "name": "rosalind franklin", "born": 1920, "alive": true }
]

Back to top

greaterThan

Checks if a numeric value is greater than a given value:

$.find({ property: "born", operator: "greaterThan", value: 1879 }).data;

Returns the following:

[
  { "id": 5, "name": "roger penrose", "born": 1931, "alive": true },
  { "id": 6, "name": "rosalind franklin", "born": 1920, "alive": true }
]

Back to top

greaterThanInclusive

Checks if a numeric value is greater than or equal to a given value:

$.find({ property: "born", operator: "greaterThanInclusive", value: 1879 })
  .data;

Returns the following:

[
  { "id": 2, "name": "albert einstein", "born": 1879, "alive": false },
  { "id": 5, "name": "roger penrose", "born": 1931, "alive": true },
  { "id": 6, "name": "rosalind franklin", "born": 1920, "alive": true }
]

Back to top

lessThan

Checks if a numeric value is less than than a given value:

$.find({
  property: "born",
  operator: "lessThan",
  value: 1867,
}).data;

Returns the following:

[
  { "id": 1, "name": "isaac newton", "born": 1643, "alive": false },
  { "id": 3, "name": "galileo galilei", "born": 1564, "alive": false }
]

Back to top

lessThanInclusive

Checks if a numeric value is less than than or equal to a given value:

$.find({
  property: "born",
  operator: "lessThanInclusive",
  value: 1867,
}).data;

Returns the following:

[
  { "id": 1, "name": "isaac newton", "born": 1643, "alive": false },
  { "id": 3, "name": "galileo galilei", "born": 1564, "alive": false },
  { "id": 4, "name": "marie curie", "born": 1867, "alive": false }
]

Back to top

in

Checks if a value exists within an array of allowed values:

$.find({ property: "born", operator: "in", value: [1867, 1920] }).data;

Returns the following:

[
  { "id": 4, "name": "marie curie", "born": 1867, "alive": false },
  { "id": 6, "name": "rosalind franklin", "born": 1920, "alive": true }
]

Back to top

notIn

Checks if a value does not exist within an array of values:

$.find({ property: "born", operator: "notIn", value: [1867, 1920] }).data;

Returns the following:

[
  { "id": 1, "name": "isaac newton", "born": 1643, "alive": false },
  { "id": 2, "name": "albert einstein", "born": 1879, "alive": false },
  { "id": 3, "name": "galileo galilei", "born": 1564, "alive": false },
  { "id": 5, "name": "roger penrose", "born": 1931, "alive": true }
]

Back to top

contains

We'll use the following dataset for this example (as well as the examples in doesNotContain):

[
  {
    "name": "lise meitner",
    "awards": ["leibniz medal", "liebenn prize", "ellen richards prize"]
  },
  {
    "name": "vera rubin",
    "awards": [
      "gruber international cosmology prize",
      "richtmyer memorial award"
    ]
  },
  {
    "name": "chien-shiung wu",
    "awards": ["john price wetherill medal"]
  }
]

Checks if an array or string contains a value:

$.find({ property: "name", operator: "contains", value: "-" }).data;

Returns the following:

[{ "name": "chien-shiung wu", "awards": ["john price wetherill medal"] }]

contains can also be used to check if an array contains a given value:

$.find({
  property: "awards",
  operator: "contains",
  value: "richtmyer memorial award",
}).data;

Returns the following:

[
  {
    "name": "vera rubin",
    "awards": [
      "gruber international cosmology prize",
      "richtmyer memorial award"
    ]
  }
]

Back to top

doesNotContain

Checks if an array or string does not contain a value:

$.find({ property: "name", operator: "doesNotContain", value: "r" }).data;

Returns the following:

[{ "name": "chien-shiung wu", "awards": ["john price wetherill medal"] }]

doesNotContain can also be used to check if an array does not contain a given value:

$.find({
  property: "awards",
  operator: "doesNotContain",
  value: "richtmyer memorial award",
}).data;

Returns the following:

[
  {
    "name": "lise meitner",
    "awards": ["leibniz medal", "liebenn prize", "ellen richards prize"]
  },
  { "name": "chien-shiung wu", "awards": ["john price wetherill medal"] }
]

Back to top

matchesRegex

Checks if a string matches a regular expression:

$.find({
  property: "name",
  operator: "matchesRegex",
  value: "^ro(g|s)",
}).data;

Returns the following:

[
  { "id": 5, "name": "roger penrose", "born": 1931, "alive": true },
  { "id": 6, "name": "rosalind franklin", "born": 1920, "alive": true }
]

Back to top

doesNotMatchRegex

Checks if a string does not match a regular expression:

$.find({
  property: "name",
  operator: "doesNotMatchRegex",
  value: "^ro(g|s)",
}).data;

Returns the following:

[
  { "id": 1, "name": "isaac newton", "born": 1643, "alive": false },
  { "id": 2, "name": "albert einstein", "born": 1879, "alive": false },
  { "id": 3, "name": "galileo galilei", "born": 1564, "alive": false },
  { "id": 4, "name": "marie curie", "born": 1867, "alive": false }
]

Back to top

Preprocessors

Preprocessor functions can optionally be applied to the values you're evaluating against in your condition. They can be used to check for:

• case insensitivity
• empty/non-empty checks
• type coercion

To apply a preprocessor to a property, instead of passing a property through as a string, pass an object through with a name and preProcess property:

$.find({
  property: { name: "name", preProcess: ["toUpper"] },
  operator: "contains",
  value: "ISAAC",
}).data;

// => [ { id: 1, name: 'isaac newton', born: 1643, alive: false } ]

preProcess is an array which can contain one or more preprocessors. When a preprocessor doesn't require any arguments (toUpper, toLower, toString, toNumber, toLength) you can pass the preprocessor through as a string (as shown in the above example). For functions that require one or more arguments (substring, concat), pass through an object where fn is the name of the preprocessor and args is an array of arguments:

$.find({
  property: {
    name: "name",
    preProcess: ["toUpper", { fn: "substring", args: [0, 3] }],
  },
  operator: "equal",
  value: "ISA",
}).data;

// => [ { id: 1, name: 'isaac newton', born: 1643, alive: false } ]

Back to top

toUpper

Converts the property to all uppercase before evaluating the condition.

$.find({
  property: { name: "name", preProcess: ["toUpper"] },
  operator: "equal",
  value: "ISAAC",
}).data;

// => [ { id: 1, name: 'isaac newton', born: 1643, alive: false } ]

Back to top

toLower

Converts the property to all lowercase before evaluating the condition.

$.find({
  property: { name: "name", preProcess: ["toLower"] },
  operator: "equal",
  value: "isaac",
}).data;

// => [ { id: 1, name: 'isaac newton', born: 1643, alive: false } ]

Back to top

toString

Converts the property to a string before evaluating the condition.

The below example won't return any data since born is of type number on the original object and we are doing a comparison of type string (remembering that the equal operators perform a strict equality (===) check):

$.find({ property: "born", operator: "equal", value: "1867" }).data;

// => []

If you want to compare a number against a string value, you can coerce the original value to a string using the toString preprocessor:

$.find({
  property: { name: "born", preProcess: ["toString"] },
  operator: "equal",
  value: "1867",
}).data;

// => [ { id: 4, name: 'marie curie', born: 1867, alive: false } ]

Back to top

toNumber

Converts the property to a number before evaluating the condition.

Using the following dataset:

[
  { "element": "hydrogen", "atomicNumber": "1" },
  { "element": "helium", "atomicNumber": "2" },
  { "element": "lithium", "atomicNumber": "3" },
  { "element": "beryllium", "atomicNumber": "4" },
  { "element": "boron", "atomicNumber": "5" }
]

Executing the following query will return an empty result, as we are trying to perform an equal operation (===) on data of type string with a number:

$.find({ property: "atomicNumber", operator: "equal", value: 2 }).data;

// => []

If we want to perform an equality match on different data types, we can first coerce the value to a number:

If you want to compare a number against a string value, you can coerce the original value to a string using the toString preprocessor:

$.find({
  property: { name: "atomicNumber", preProcess: ["toNumber"] },
  operator: "equal",
  value: 2,
}).data;

// => [ { element: 'helium', atomicNumber: '2' } ]

Back to top

toLength

Returns the length of a string or the amount of items in an array. Can be used to check for non-empty values:

$.find({
  property: { name: "name", preProcess: ["toLength"] },
  operator: "greaterThan",
  value: 0,
}).data;

Returns the following:

[
  { "id": 1, "name": "isaac newton", "born": 1643, "alive": false },
  { "id": 2, "name": "albert einstein", "born": 1879, "alive": false },
  { "id": 3, "name": "galileo galilei", "born": 1564, "alive": false },
  { "id": 4, "name": "marie curie", "born": 1867, "alive": false },
  { "id": 5, "name": "roger penrose", "born": 1931, "alive": true },
  { "id": 6, "name": "rosalind franklin", "born": 1920, "alive": true }
]

Can also be used on arrays:

const schedule = new Database([
  { department: "it", subjects: ["data structures and algorithms"] },
  { department: "physics", subjects: ["newtonian mechanics"] },
  { department: "maths", subjects: [] },
]);

schedule.$.find({
  property: { name: "subjects", preProcess: ["toLength"] },
  operator: "greaterThan",
  value: 0,
}).data;

Returns records with a non-empty subjects property:

[
  { "department": "it", "subjects": ["data structures and algorithms"] },
  { "department": "physics", "subjects": ["newtonian mechanics"] }
]

Back to top

substring

Returns the part of the string between the start and end indexes, or to the end of the string:

$.find({
  property: {
    name: "name",
    preProcess: [{ fn: "substring", args: [1, 4] }],
  },
  operator: "equal",
  value: "oge",
}).data;

// => [ { id: 5, name: 'roger penrose', born: 1931, alive: true } ]

Back to top

Guides and concepts

Type inference

When using the MemoryAdapter, newton will automatically infer the shape of your data based on the value passed in:

Using a single collection:

const db = new Database([
  { name: "Isaac Newton", born: "1643-01-04T12:00:00.000Z" },
  { name: "Albert Einstein", born: "1879-03-14T12:00:00.000Z" },
]);

Using multiple collections:

const db = new Database({
  scientists: [
    { name: "Isaac Newton", born: "1643-01-04T12:00:00.000Z" },
    { name: "Albert Einstein", born: "1879-03-14T12:00:00.000Z" },
  ],
  universities: [
    { name: "University of Zurich", location: "Zurich, Switzerland" },
  ],
});

When the shape of the data can't be inferred automatically, you can pass through the shape of the data when instantiating your database:

Instantiating with the shape of your database:

const adapter = new FileAdapter("./db.json");
const db = new Database<{
  scientists: Scientist[];
  universities: University[];
}>(adapter);

Back to top

License

MIT