@worldbrain/storex

Storex is a minimal storage layer as a foundation for easing common problems around storing and moving data around. Allowing you to describe your data layout as a graph and providing different plugins, it helps you interact with (No)SQL databases, data migration, offline first applications architecture, creating and consuming REST/GraphQL APIs, permission management, finding optimization opportunaties and more. The aim is to provide a minimalistic common ground/language for working with your data, providing packages for solving the most common problems around data, while giving you easy access to the underlying machinery to do the things that are specific to your application. Everything together that means that every problem you encounter while rapidly iterating towards a serious product, from choosing a suitable DB to suddenly realizing you need to migrate your data model, or even switch DBs, will get a ton easier because you don't have to solve them for the 109th time yourself.

This project started as the storage layer for Memex, a tool to organize your web-research for yourself and collaboratively, in collaboration with YouAPT. Download it here, and check out our vision here.

Status: Proof of concept used in production to interact with IndexedDB while having the freedom to shift to the cloud and decentralize storage in the near future. Needs a lot more development, but implemented functionality is working. That being said, the API is subject to change between minor versions until the 1.0 release. Please consider contributing through easy to pick up tasks to get you started!

Installation

Storex is a collection of Node.js modules (written in TypeScript) available through NPM, meant to be used both client- and server-side. To start, you need the core and a backend:

$ npm install @worldbrain/storex --save

$ # For a client-side DB
$ npm install @worldbrain/storex-backend-dexie --save # IndexedDB through Dexie library

$ # For a server-side SQL DB
$ npm install @worldbrain/storex-backend-sequelize --save # MySQL, PostgreSQL, SQLite, MSSQL through Sequelize

$ # For a Firestore mBaaS DB
$ npm install @worldbrain/storex-backend-firestore --save

Basic usage

First, configure a StorageBackend and set up the StorageManager, which will be the main point of access to define, query and manipulate your data. For more in-depth information on how to do all of this, please refer to the docs.

import StorageManager from 'storex'
import { DexieStorageBackend } from 'storex-backend-dexie'

const storageBackend = new DexieStorageBackend({dbName: 'my-awesome-product'})
const storageManager = new StorageManager({ backend: storageBackend })
storageManager.registry.registerCollections({
    user: {
        version: new Date(2018, 11, 11),
        fields: {
            identifier: { type: 'string' },
            isActive: { type: 'boolean' },
        },
        indices: [
            { field: 'identifier' },
        ]
    },
    todoList: {
        version: new Date(2018, 7, 11),
        fields: {
            title: { type: 'string' },
        },
        relationships: [
            {childOf: 'user'} # creates one-to-many relationship
        ],
        indices: []
    },
    todoListEntry: {
        version: new Date(2018, 7, 11),
        fields: {
            content: {type: 'text'},
            done: {type: 'boolean'}
        },
        relationships: [
            {childOf: 'todoList', reverseAlias: 'entries'}
        ]
    }
})
await storageManager.finishInitialization()

const user = await storageManager.collection('user').createObject({
    identifier: 'email:[e-mail]',
    isActive: true,
    todoLists: [{
        title: 'Procrastinate this as much as possible',
        entries: [
            {content: 'Write intro article', done: true},
            {content: 'Write docs', done: false},
            {content: 'Publish article', done: false},
        ]
    }]
})
# user now contains things generated by underlying backend, like ids and random keys if you have such fields
console.log(user.id)

await storageManager.collection('todoList').findObjects({user: user.id}) # You can also use MongoDB-like queries

Further documentation

You can find the docs here. Also, we'll be writing more and more automated tests which also serve as documentation.

Storex ecosystem

The power of Storex comes from having modular packages that can be recombined in different contexts based on the core 'language' or patterns Storex provides to talk about data. Officially supported packages will be included in the @worldbrain npm namespace. This helps us to endorse patterns that emerge throughout the ecosystem in a controlled, collectively governed way. These are the currently supported packages:

• Storage modules: An pattern of organizing your storage logic modules so other parts of your application and tooling can have more meta-information about the application. This includes standard ways of defining your data structure and schema changes, describing how you access your data, describing higher-level methods, and describing access rules. This allows for really cool stuff like automatically detecting operations on fields that are not indexed, automatically generating client-server APIs for moving your data around, generating rules for other systems (like Firebase) to manage user rights, and a lot more.
• Schema migrations: The functionality you need, provided in a modular and adaptable way, to describe and execute schema migration as your application evolves. Written with a diverse context in mind, like generating SQL scripts for DBAs, running migration in Serverless environments, converting exported data on import, etc.
• Client-server communication usign GraphQL: This allows you to move your storage layer, expressed in storage modules, server-side transparently by using GraphQL as the transport layer with your back-end. You consume this API with the client so the business logic consuming your storage logic can stay exactly the same, making GraphQL an implementation detail, not something running throughout your code.
• Data tools: A collection of data tools allowing you to apply schema migration on live data (coming in through the network or read from files), generate large sets of fake, complex data for testing purposes, and tools for dumping and loading test fixtures.
• Schema visualization using Graphviz: Still in its infancy, this creates a GraphViz DOT files showing you the relationships between your collections.

Also, current officially supported back-ends are:

• Dexie: Manages interaction with IndexedDB for you, so your application can run fully client-side. Use for your daily development workflow, for fully client-side applications, or offline-first applications.
• Sequelize: Interact with any SQL database supported by Sequelize, such as MySQL or PostgreSQL. Meant to be used server-side.
• Firestore: Store your data in Firestore, so you don't have to build a whole back-end yourself. Can be used directly from the client-side, or inside Cloud Functions. Includes a Security Rule generator that removes a lot of the pain of writing secure and maintainable security rules.

Status and future development

In addtion to the functionality described above, these are some features to highlight that are implemented and used in production:

• One DB abstraction layer for client-side, server-side and mBaaS code: Use the Dexie backend for IndexedDB client-side applications, the Sequelize backend for server-side SQL-based databases, or the Firestore backend for mBaaS-based applications. This allows you to write storage-related business logic portable between front- and back-end, while easily switching to non-SQL storage back-ends later if you so desire, so you can flexible adjust your architecture as your application evolves.
• Defining data in a DB-agnostic way as a graph of collections: By registering your data collections with the StorageManager, you can have an easily introspectable representation of your data model
• Automatic creation of relationships in DB-agnostic way: One-to-one, one-to-many and many-to-many relationships declared in DB-agnostic ways are automatically being taken care of by the underlying StorageBackend on creation.
• MongoDB-style querying: Basic CRUD operations take MongoDB-style queries, which will then be translated by the underlying StorageBackend.
• Client-side full-text search using Dexie backend: By passing a stemmer into the Dexie storage backend, you can full-text search text fields using the fastest client-side full-text search engine yet!
• Run automated storage-related tests in memory: Using the Dexie back-end, you can pass in a fake IndexedDB implementation to run your storage in-memory for faster automated and manual testing.
• Version management of data models: For each collection, you can pass in an array of different date-versioned collection versions, and you'll be able to iterate over your data model versions through time.

The following items are on the roadmap in no particular order:

• Synching functionality for offline-first and p2p applications (in development)
• Relationship fetching & filtering: This would allow passing in an extra option to find(One)Object(s) signalling the back-end to also fetch relationship, which would translate to JOINs in SQL databases and use other configurable methods in other kinds of databases. Also, you could filter by relationships, like collection('user').findObjects({'email.active': true}).
• Field types for handling user uploads: Allowing you to reference user uploads in your data-model, while choosing your own back-end to host them.
• A caching layer: Allows you to cache certain explicitly-configured queries in stores like Memcache and Redis
• Composite back-end writing to multiple back-ends at once: When you're switching databases or cloud providers, there may be period where your application needs to the exact same data to multiple database systems at once.
• Assisting migrations from one database to another: Creating standard procedures allowing copying data from one database to another with any paradigm translations that might be needed.
• Server-side full-text search server integration": Allow for example to store your data in MondoDB, but your full-text index in ElasticSearch.
• Query analytics: Report query performance and production usage patterns to your anaylics backend to give you insight into possible optimization opportunities (such as what kind of indices to create.)

Also, Storex was built with decentralization in mind. The first available backends are Dexie allowing you to user data client side, Sequelize for storing data server-side in relational databases (MySQL, PostgreSQL, AWS Aurora), and Firestore for quickly buiding cloud-based MVPs that can be relatively easily be migrated to other solutions. In the future, we see it possible to create backends for decentralized systems like DAT to ease the transition and integration between centralized and decentralized back-ends as easy as possible.

Contributing

There are different ways you can contribute:

• Report bugs: The most simple way. Scream at us :) If you can find the time though, we'd really appreciate it if you could attach a PR with a failing unit test.
• Tackle outstanding bugs: Great way to dip your toes in the water. Some of the reported bugs may already have failing unit tests attached to them!
• Propose new features: Open an issue to describe a new feature you'd like to see. Please take the time to explain your reasoning behind the request, how it would benefit the Storex ecosystem and things like use-cases if relevant.
• Implement new features: We try our best to make extensive descriptions of how features should work, and a clear decision process if the features are yet to be designed. Features to be implemented can be found here with help how to pick up those features.

Building for development

Storex is written in Typescript. As such, there's a lot of type information serving as documentation for the reader of the code. To start writing code, do the following:

1. Check out the Storex workspace repo: git clone --recursive [e-mail]:WorldBrain/storex-workspace.git.
2. Run yarn bootstrap inside the checked out repo.
3. cd to any package you'd like to work on, like cd packages/@worldbrain/storex-backend-dexie
4. Create your own feature branch if necessay: git checkout -b feature/my-awesome-feature
5. Run yarn test:watch

More info on https://github.com/WorldBrain/storex-workspace

Getting in touch

For guidance on how to contribute, providing insight into your use cases, partnerships or any other collaboration opportunities, don't hesitate to get in touch here.