scurry
A leveldb-backed consistent hash ring, for your caching needs. I can see the day when you might want to put data in this & feel reasonably sort of confident you might get it back out again. If this scares you, it's supposed to.
One part each
Rod Vagg's levelup leveldb bindings for node + sublevel to create buckets.
Dominic Tarr's crtd, which uses his scuttlebutt implementation to keep a document in sync.
light-cycle, a lightweight consistent hash ring structure that can be mixed into most anything.
restify to provide a simple RESTful api to data in the buckets.
Shake with ice
Run a server:
node index.js --id=node-one -m -p 3333 -g 4114 -d ./db | ./node_modules/.bin/bunyan -o short
Run a client or five:
node index.js --id=node-two -p 3334 -g 4114 -s 10.0.0.5 -d ./db2 | ./node_modules/.bin/bunyan -o shortnode index.js --id=node-three -p 3335 -g 4114 -s 10.0.0.5 -d ./db3 | ./node_modules/.bin/bunyan -o shortnode index.js --id=node-four -p 3336 -g 4114 -s 10.0.0.5 -d ./db4 | ./node_modules/.bin/bunyan -o short
Replace 10.0.0.5
with the IP address of your server.
Strain into a chilled glass
Then stuff some data in:
http PUT 10.0.0.5:3334/vodkas/1 name="Sobieski" rating=5http PUT 10.0.0.5:3335/vodkas/2 name="Tito's Handmade" rating=5http PUT 10.0.0.5:3335/vodkas/3 name="Bimber" rating=4
Get it back out: http GET 10.0.0.5:3336/vodkas/2 | json
(Human-friendly shell commands courtesy of httpie).
Garnish with a twist of orange peel
API endpoints exposed:
GET /:bucket
: stream sorted keys for a bucket (works!)POST /:bucket
: add an item to the cache; id is generated for you & returnedPUT /:bucket/:id
: add/update an item in the cache; 204 responseGET /:bucket/:id
: get an item from the cacheHEAD /:bucket/:id
: headers for an itemDEL /:bucket/:id
: remove an item
TTLs
Send this header to specify a time-to-live for your cached data: X-Scurry-TTL: [seconds]
Conditional requests
Scurry sends an ETag header and a last-modified timestamp.
Storage format
As of version 0.0.4, the data stored in the LevelDB nodes is json structured as follows:
version: 1 // storage version ts: Date // timestamp of last set() payload: value // base64-encoded string if buffer, JSON string if not etag: crc // md5 hex digest of payload base64: valueIsB64String // true if the payload had to be base64 encoded 'content-type': metadata'content-type' // content-type if passed in;
TODO
Upcoming releases:
- Release 0.0.4 will probably make streaming keys work, maybe. Done!
- Release 0.0.5 will finalize the storage format. Probably done!
- Release 0.0.6 will contemplate eviction. Not yet!
General goals:
- Implement key streaming from multiple nodes. See notes in endpoints.handleGetBucket().
- The RESTful server needs error handling.
- Reconnect on errors.
- Error handling.
- Better logging. Configurable, for one thing.
- Back ends should be pluggable; the API is very small.
- Stretch goal: replication?
License
MIT.