Incrementally reduce an event-log into a snapshot view of those events
The idea is that you store all your raw data as a massive log of append only events, i.e. you never update or delete anything.
Then you use snapshot-reduce to reduce the event log into a coherent snapshot of a subset of current state.
Snapshot reduce allows you to accumulate a subset of a event log into a snapshot representation of that data. The actual mapReduce is based on mongoDB and is incremental.
var snapshotReduce =var passback =// Get a mongoDB instance however you wantvar db = someMongoDbvar inputCollection = dbvar outputCollection = dbvar command =
For this to work you must adhere to some constraints of the event-log input. For example if your input is:
"eventType" : "add""timestamp" : 1360877429848"value" :"title" : "some room""type" : "colingo-room""id" : "1"
This is a message that says "a room was added". You may have another message like
"eventType" : "add""timestamp" : 1360878425642"value" :"name" : "Jake""message" : "test""type" : "colingo-room~messages""parentId" : "1""id" : "3dedc0cc-917b-48ed-a3c9-6166011fb458"
Which says a message was added to a room. The room it was added
to was the room with id
All inputs should have inline
type is used to query a subset of the event-log to
map reduce. The
timestamp field is used to support
incremental map reduce, i.e. it will only reduce values that
have a timestamp higher then the last one from last time.
The two events above would be reduced to
title: "some room"id: "1"type: "colingo-room"messages:name: "Jake"message: "test"id: "3dedc0cc-917b-48ed-a3c9-6166011fb458"type: "colingo-room~messages"parentId: "1"__lastTimestamp__: 1360878425642
__lastTimestamp__ field is used to do an incremental map
reduce and basically allows snapshot-reduce to reduce the
entire history of the event log again.
You can also do the reverse, turn a snapshot into the minimal amount of raw events needed to construct that snapshot
var unpackSnapshot =var assert =var into =var list =assert
npm install snapshot-reduce