syncronize couchdb data into leveldb
The following example illustrates the simplest use case. It will synchronize couchdb data into a leveldb located at
/tmp/level-npm and store the data as (key, value) = (data.id, JSON.stringify(data.doc)), where data is JSON chunks received from the couch.
var levelCouchSync = require'level-couch-sync'var levelup = require'levelup'var sublevel = require'level-sublevel'var db = sublevellevelup'/tmp/level-npm'levelCouchSync'' db 'registry-sync'
If you provide a map/iterator function you can decide for yourself what kind of data your want to persist. An easy
way to accomplish this, is to create more sublevels and shove data into them. This example shows how you can store
basic package metadata in a sublevel named
var levelCouchSync = require'level-couch-sync'var levelup = require'levelup'var sublevel = require'level-sublevel'var db = sublevellevelup'/tmp/level-npm-advanced'var packageDb = dbsublevel'package'levelCouchSync'' db 'registry-sync'var doc = datadocemitdataid JSONstringifyname : docnameauthor : docauthorrepository : docrepositorypackageDb
emit() call adds a (key, value, sublevel) triplet to a batch operation that is executed once the iterator
returns, which means you can call
emit() many times during each time the iterator is invoked.
levelCouchSync() returns an EventEmitter, which you can attach listeners to. The following example logs package
versions and progress to stdout. See the events section for more details.
// ..var sync = levelCouchSyncurl db 'registry-sync'syncon'data'console.logdataid datadocversions && Objectkeysdatadocversionssyncon'progress'console.logMathfloorratio*10000/100 + '%'
Run the samples in the
example/ folder and try it out! It should work on all systems where
levelup can be
compiled. If you want to take a closer look at what the data looks like you can use
lev, which is an awesome cli tool for viewing any leveldb. All you need is a path
The API is very simple and only contain one function.
This function returns an
EventEmitter instance and has three mandatory arguments and one optional.
sourceUrlis a string pointing out the url to the couch we are getting the updates from
dbmust be a
level-sublevelinstance and is used to store the data if there is no
metaDbmust be a
level-sublevelinstance or a string. If it's a string, a sublevel will be created with that name.
metaDbhandles metadata of the ongoing transfer and keeps track of the
update_seq, which means that if the process crashes, it will automatically continue where it left off
map(data, emit)is an iterator function called for each JSON data received from the couch. The first argument
datais the JSON received from the couch.
emit(key, value, sublevel)is a function you call each time you want to persist some data. It takes the following three arguments:
keyis a string and is the key used to store the value
valueis an object that you are free to build as you please
level-sublevelinstance used to store the
level-couch-sync emits various events as the leveldb is syncronized with the couch:
sync.emit('data', data)emitted for each data object received from the couch
sync.emit('progress', ratio)emitted each time data has been written to levelup. The
ratiois defined as how much data that has been written from the current update sequence. When there is something to be read from the couch then
0 < ratio < 1.0and when
ratio > 1.0it means we are syncing live!
sync.emit('fail', err)emitted when there is an error fetching the
sourceUrlfrom couchdb before the request will be tried again using fibonacci backoff
sync.emit('max', maxSeq)emitted when a request has been made to the source url.
maxSeqis the value of the
update_seqproperty in the JSON response
you can create a progress bar like used in npmd, just provide a name for the couchdb, and a function that returns a tagline describing the document that was updated.
name is the url of the couchdb instance, and the tagline will be