InjestDB
A peer-to-peer database for dat:// applications. How it works
Example
Setup a database for social profiles, which can publish status updates and like other users' posts.
const Injest = var db = 'social-profiles'db
Next we add source archives to be ingested (added ot the dataset). The source archives are persisted in IndexedDB, so this doesn't have to be done every run.
await db
Now we can begin querying the database for records.
// get the first profile record where name === 'bob'var bobProfile = await dbprofiles // get all profile records which match this queryvar bobProfiles = await dbprofiles // get the 30 latest broadcasts from all source archivesvar recentBroadcasts = await dbbroadcasts // most recent first // get the 30 latest broadcasts by a specific archive (bob)// - this uses a compound index to filter by origin, and then sort by createdAtvar bobsRecentBroadcasts = await dbbroadcasts // most recent first // get the # of likes for a broadcastvar numLikes = await dblikes // _url is an autogenerated attribute which represents the URL of the record
We can also use Injest to create, modify, and delete records (and their matching files).
// update bob's nameawait dbprofiles // publish a new broadcast for bobvar broadcastUrl = await dbbroadcasts // modify the broadcastawait dbbroadcasts // like the broadcastawait dblikes // delete the broadcastawait dbbroadcasts // delete all likes on the broadcast (that we control)await dblikes
TODOs
Injest is still in development.
- Indexer
- Core query engine
- Persisted tables and table reindex on schema change
- Mutation methods (add/update/delete)
- Events
- Multikey indexes
- Validation: filename must match primaryKey on non-singular tables
- Support for .or() queries
- Complete documentation
API quick reference
var db = nameInjestDB Promise<Void>InjestDB Promise<Void>db Promise<Void>db Promise<Void>db Promise<Void>db Promise<Void>db Promise<Void>db Promise<Void>dbdb Promise<url>db 'open' db 'open-failed' errordb 'versionchange' db 'indexes-updated' archive archiveVersion dbtable InjestTableInjestTable# Promise<url>InjestTable# Promise<Number>InjestTable#deleteurl Promise<url>InjestTable# Promise<Void>InjestTable# InjestRecordsetInjestTable# Promise<Object>InjestTable# Promise<Object>InjestTable# Promise<Object>InjestTable# Promise<Object>InjestTable# BooleanInjestTable# InjestRecordsetInjestTable# Promise<Object>InjestTable# StringInjestTable# InjestRecordsetInjestTable# InjestRecordsetInjestTable# Promise<url>InjestTable# InjestRecordsetInjestTable# ObjectInjestTable# Promise<Array>InjestTable# InjestRecordsetInjestTable# Promise<Number>InjestTable# Promise<Number>InjestTable# Promise<Number>InjestTable# Promise<Number>InjestTable# Promise<Void | url>InjestTable# Promise<Void | url>InjestTable# InjestWhereClauseInjestTable 'index-updated' archive archiveVersion InjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordsetInjestWhereClause# InjestRecordset InjestRecordset# InjestRecordsetInjestRecordset# Promise<Number>InjestRecordset#delete Promise<Number>InjestRecordset# InjestRecordsetInjestRecordset# Promise<Void>InjestRecordset# Promise<Void>InjestRecordset# Promise<Void>InjestRecordset# InjestRecordsetInjestRecordset# Promise<Object>InjestRecordset# Promise<Array<String>>InjestRecordset# Promise<Object>InjestRecordset# InjestRecordsetInjestRecordset# InjestRecordsetInjestRecordset# InjestWhereClauseInjestRecordset# Promise<Number>InjestRecordset# Promise<Array<String>>InjestRecordset# InjestRecordsetInjestRecordset# Promise<Array<Object>>InjestRecordset# Promise<Array<String>>InjestRecordset# InjestRecordsetInjestRecordset# Promise<Number>InjestRecordset# InjestWhereClause
API
db.schema(definition)
version: Number // the version # of the schema, should increment by 1 on each change tableName: // is there only one record-file per archive? // - if true, will look for the file at `/${tableName}.json` // - if false, will look for files at `/${tableName}/*.json` singular: Boolean // attribute to build filenames for newly-created records // ie `/${tableName}/${record[primaryKey]}.json` // only required if !singular primaryKey: String // specify which fields are indexed for querying // each is a keypath, see https://www.w3.org/TR/IndexedDB/#dfn-key-path // can specify compound indexes with a + separator in the keypath // eg one index - index: 'firstName' // eg two indexes - index: ['firstName', 'lastName'] // eg add a compound index - index: ['firstName', 'lastName', 'firstName+lastName'] // eg index an array's values - index: ['firstName', '*favoriteFruits'] index: String|Array<String> // validator & sanitizer // takes the ingested file (must be valid json) // returns the record to be stored // returns falsy or throws to not store the record validator: FunctionObject Object
How it works
InjestDB abstracts over the DatArchive API to provide a simple database-like interface. It is based heavily on Dexie.js and built using IndexedDB.
Injest works by scanning a set of source archives for files that match a path pattern. Those files are indexed ("ingested") so that they can be queried easily. Injest also provides a simple interface for adding, editing, and removing records on the archives that the local user owns.
Injest sits on top of Dat archives. It duplicates the data it's handling into IndexedDB, and that duplicated data acts as a throwaway cache -- it can be reconstructed at any time from the Dat archives.
Injest treats individual files in the Dat archive as individual records in a table. As a result, there's a direct mapping for each table to a folder of .json files. For instance, if you had a 'tweets' table, it would map to the /tweets/*.json
files. Injest's mutators, such as put or add or update, simply write those json files. Injest's readers & query-ers, such as get() or where(), read from the IndexedDB cache.
Injest watches its source archives for changes to the json files. When they change, it reads them and updates IndexedDB, thus the query results stay up-to-date. The flow is, roughly: put() -> archive/tweets/12345.json -> indexer -> indexeddb -> get()
.