ssb-review-level

A view implemented on top of leveldb, for use with ssb-revisions.

Provides indexes which are persistent and can be streamed in order.

This is more or less a drop-in replacement for flumeview-level, for scuttlebutt applications that require mutable documents.

See also: ssb-review-reduce

Differences to flumeview-level

• In case a message is a revision of a prior message (e.g. it has revisionRoot and revisionBranch properties), your map function is called twice: once for the old value, once for the new value. (your map function typically does not care whether it is called for the old or new value. However, if it does, this information is provided in the third argument: true for new, false for old).

• all entries returned by map(new_value) are written to leveldb (same as with flumeview-level)

• all entries retunred by map(old_value) that are not also included in what is retunred by map(new_value) are deleted from leveldb.

• the stream retunred by read({live:true}) may contain {type: 'del', key: [..]} items, if an object/document no longer is part of the query result.

• the key-value pair argument to map() has a thrid property: meta. It's an object containing two booleans: forked and incomplete that indicated problems with the history of the object.

• if revisions are present, ssb-revisions makes sure that map is called in the correct, causal order.

Example

var ReviewLevel = require('ssb-review-level')
 
ssb.revisions.use('my_view', ReviewLevel(1, function map (kv) {
  return [ [kv.value.content.foo, kv.value.content.revisionRoot || kv.key] ] // array of array-keys
}))
 
ssb.publish({foo: 'bar'}, function (err, msg_A) {
  ssb.publish({foo: 'baz'}, function (err, msg_B) {
    ssb.publish({
      revisionRoot: msg_B.key,
      revisionBranch: msg_B.key
      foo: 'bar'
    }, function (err, msg_C) {
      // query ranges via pull-streams
      pull(
        revisions.my_view.read({gt: ['bar', null], lt: ['bar', undefined], live: true}),
        ... => {
          key: ['bar', msg_A.key],
          value: { 
            key: msg_A.key,
            value: { content: { foo: 'bar' } }
          } 
        }
        ... => {
          key: ['bar', msg_B.key],
          value: { 
            key: msg_C.key,
            value: {
              content: {
                foo: 'bar',
                revisionRoot: msg_B.key,
                revisionBranch: msg_B.key
              }
            }
          }
        }
      )
    })
  })
})

In the example above, msg C is a revision of msg B that causes the original message's key (B) to be included in the query result (because the revised message B now has the foo value that matches the query)

More examples can be found here and here

The following was adapted from flumeview-level's README.

API

ReviewLevel(version, map) => function

version

The version of the view. Incrementing this number will cause the view to be re-built

map

A function with signature (value, seq, is_new), where value is the item from the log coming past, and seq is the location of that value in the flume log. is_new is true if the function is called with a new (or the original) value, and false if it is called with the old value. In most cases you can ignore all arguments but the first.

This function must return an Array that's either empty or contains unique index key(s). These index keys can then be queired to retrieve the stored value (see get and read below).

Examples of index key(s) you might return:

• [] - i.e. don't add any indexes for this value
• ['@mix'] - make an index entry for this value under string @mix
• ['@mix', '@mixmix'] - make an index entries for this value under both @mix AND @mixmix
• [['@mix', 1524805117433]] - make an index entry for this value under the key ['@mix', 1524805117433] (anything can be a key in leveldb)

This last case is useful when you might want multiple entries under a particular key like @mix - if just use @mix then the index will get overwritten by future values coming in with the same key. Extending the key to include some unique aspect (like a timestamp or the seq of the value) means you can have multiple indexes in your view which have a similar key.

In a scenario with mutable documents however, you most likely want an index key like this:

• ['@mix', kv.value.content.revisionRoot || kv.key]

This ensures that a) differnt documents don't overwrite each other's index entry, and b) later revisions of the same document do overwrite previous index entries. ssb-revisions makes sure that map is called in the correct (causal) order (newest last)

function

ssb-review-level returns a function which follows the ssb-review pattern, enabling it to be installed into an instance of ssb-revisions.

get(key, cb)

The keys for the values in map above would be '@mix', '@mixmix', or ['@mix', 1524805117433]

read(opts) => pull-stream

opts is similar to a level db query (see level docs).

e.g.

{
  live: true,     // this is an addition to the classic query options of level
  gte: '@mi',     // gte = greater than or equal to
  lt: null,       // lt = less than
  reverse: true,
  keys: true,
  values: true,
  seqs: false,
}

If you've created indexes that are Arrays (quite likely), you need to understand how Arrays and other value are ordered by leveldb. This is because using leveldb is all about ordering keys so that you can do queries efficiently. Because of the way a log-structured-merge-tree works (what level is) it can read adjacent records quickly (with a single seek) but jumping around is not as fast. Read about the pattern of ordering of keys/ indexes flumeview-level uses here (actually uses charwise under the hood, but follows the bytewise spec).

Example of more advanced query:

{
  gte: ['@mix', null],
  lte: ['@mix' undefined],
}

Assume this is an index where the keys are of the form [@mentions, revisionRoot], then this query will get all documents where @mix is mentioned in the latest revision. (note undefined is the highest, null the lowest value in bytewise comparator)

If you wanted to get all mentions which started with @m you could use:

{
  gte: ['@m', null],
  lt: ['@m~', undefined],
}

Here null is the lowest value in the comparator, and the ~ is just a slightly unreliable hack to catch values below @m~ as ~ is quite a high character (e.g. above Z) for lexicographic ordering (there are higher characters but english people are less likely to type them, check ltgt to generate reliable limiting values).

Here's some lexographically ordered strings to help you catch the vibe: '@nevernever', '@m', '@manowar', '@ma~', '@mo', '@m~'

License

MIT