simpledb

An Amazon AWS SimpleDB library for Node.js that is user-friendly and fault-tolerant

simpledb

If you're using this library, feel free to contact me on twitter if you have any questions! :) @rjrodger

NOTE: this project follows the Open-Open policy - if you submit a pull request or an issue, you get commit rights, so feel free to merge yourself after asking for feedback from the other contribs.

IMPORTANT: YOUR CODE CONTRIBUTIONS (if any) ARE MADE UNDER THE MIT LICENSE. By submitting a pull request or issue you indicate agreement with this condition.

Please open an issue to indicate a release should be published to NPM, and we can discuss.

Current Version: 0.2.0

Tested on: node 0.12.4

A user-friendly fault-tolerant library for Amazon AWS SimpleDB access. The core SimpleDB actions are mapped to functions:

var simpledb = require('simpledb')
var sdb      = new simpledb.SimpleDB({keyid:'YOUR_AWS_KEY_ID',secret:'YOUR_AWS_SECRET_KEY'})

sdb.createDomain( 'yourdomain', function( error ) {

  sdb.putItem('yourdomain', 'item1', {attr1:'one', attr2:'two'}, function( error ) {

    sdb.getItem('yourdomain', 'item1', function( error, result ) {
      console.log( 'attr1 = '+result.attr1 )
      console.log( 'attr2 = '+result.attr2 )
    })
  })
})

Any given SimpleDB request has a non-trivial chance of failing. This library implements the exponential back-off retry algorithm as recommended in the SimpleDB developer guide.

This library depends on the excellent aws-lib module: aws-lib

Key Features:

simple API
fully configurable
detailed logging
all request attributes can be overridden
fully tested

Core Functions:

createDomain ("CreateDomain")
domainMetadata ("DomainMetadata")
listDomains ("ListDomains")
deleteDomain ("DeleteDomain")
putItem ("PutAttributes")
batchPutItem ("BatchPutAttributes")
batchDeleteItem ("BatchDeleteAttributes")
getItem ("GetAttributes")
deleteItem ("DeleteAttributes")
select ("Select")
request (any action)

This is still an early version so there's probably some wierdness - use at your risk. Secure connections are not supported on node 0.3.x.

Installation

npm install simpledb

And in your code:

var simpledb = require('simpledb')

Or clone the git repository: git clone git://github.com/rjrodger/simpledb.git

The simpledb module depends on the aws-lib module. npm will install this automatically.

Usage

This module uses the node.js-style callback convention. All functions take a callback function as their last argument. This callback function should accept three arguments:

callback( error, result, meta )

Where error is an object ({Code:'...',Message:'...'}) describing any errors that occured. If the function was successful then error is null.

So, you check if error is null to see if you can continue working:

sdb.listDomains( function( error, result, meta ) {
  if( error ) {
    console.log('listDomains failed: '+error.Message )
  }
  else {
    // do stuff with result, an array of domain names
  }
})

The result parameter contains the results of a successful action and what the result parameter is depends on the action. It could be a string, an array or an object.

The meta parameter contains a description of the request, including the underlying details from Amazon. Take a look with:

console.log( JSON.stringify(meta) )

Conventions

Where possible, the SimpleDB naming style is preserved: CamelCaseBaby. Names of functions and their parameters also mostly match SimpleDB.

The simpledb.SimpleDB wrapper options (maxtry, secure, etc) are not directly related to Amazon, and so have their own names.

It is sometimes necessary to embed meta-directives into the Amazon query or result objects. These non-Amazon attributes always begin with the $ character, but are in CamelCase. For example: $AsArrays.

This wrapper is based on the REST API. I looked at the SOAP API but... yeah. No X.509 for you. Yet.

API

For the API examples, assume the following lines of code at the top of your source code file:

var simpledb = require('simpledb')

var sdb = new simpledb.SimpleDB(
  {keyid:'YOUR_AWS_KEY_ID',secret:'YOUR_AWS_SECRET_KEY'},
  simpledb.debuglogger
)

This gives you the standard wrapper, with a basic debugger that prints to STDOUT.

You should really also read the Amazon SimpleDB documentation so that you understand how SimpleDB works: Amazon SimpleDB Developer Guide

As a get-out-of-jail, you can provide request attribute overrides. You supply these in an optional override argument just before the callback argument. You can use an override on any of the SimpleDB action wrapper functions.

sdb.getItem('domain','itemname', {ConsistentRead:'false'} ,function(err,res,meta){ ... })

In the above code, {ConsistentRead:"false"} is the optional override argument.

simpledb.SimpleDB: `simpledb.SimpleDB( options, logger )`

options: (required) set of options; keyid and secret are required
logger: (optional) callback for log events

var sdb = new simpledb.SimpleDB( options, logger )

Create a new SimpleDB wrapper. The options argument sets general options for the requests. The logger argument receives logging events so that you can debug and/or record SimpleDB interactions.

options: required

keyid: (required), your Amazon AWS Key ID
secret: (required), your Amazon Secret Key

For further options, see the section on options below

logger: optional

See the section on logging below

createDomain: `sdb.createDomain(domain,override,callback)`

domain: (required) the name of the domain
override: (optional) SimpleDB attributes to override function defaults
callback: (required) callback function accepting parameters callback(error, result, metadata)

Create a domain. A domain is like a SQL table, sort of.

sdb.createDomain('<domain>',function(err,res,meta){
  if( !err ) {
    console.log('Mul-ti-pass!')
  }
}

Where <domain> is the name of your domain.

domainMetadata: `sdb.domainMetadata(domain,override,callback)`

domain: (required) the name of the domain
override: (optional) SimpleDB attributes to override function defaults
callback: (required) callback function accepting parameters callback(error, result, metadata)

Get some statistics about your domain, such as a count of items and how much storage it is using (you pay Amazon for this!).

sdb.domainMetadata('<domain>',function(err,res,meta){
   console.log('Mmm, floor pie! '+JSON.stringify(res) )
}

Where <domain> is the name of your domain.

listDomains: `sdb.listDomains(override,callback)`

override: (optional) SimpleDB attributes to override function defaults
callback: (required) callback function accepting parameters callback(error, result, metadata)

Returns a list of your domain names as an array of strings. Restricted to the specified SimpleDB host (default=sdb.amazonaws.com). See the simpledb.SimpleDB options to change this.

sdb.listDomains(function(err,res,meta){
   console.log('You hear that? That's market bacon hitting the pan: '+JSON.stringify(res) )
}

deleteDomain: `sdb.deleteDomain(domain,override,callback)`

domain: (required) the name of the domain
override: (optional) SimpleDB attributes to override function defaults
callback: (required) callback function accepting parameters callback(error, result, metadata)

Delete a domain. Cannot be undone!

sdb.deleteDomain('<domain>',function(err,res,meta){
  if( !err ) {
    console.log('God made the world, but we made the field.')
  }
}

Where <domain> is the name of your domain.

putItem: `sdb.putItem(domain,itemname,attrs,override,callback)`

domain: (required) the name of the domain
itemname: (required) the unique name of the item in the domain
attrs: (required) the item attributes to store
override: (optional) SimpleDB attributes to override function defaults
callback: (required) callback function accepting parameters callback(error, result, metadata)

Store an item in SimpleDB.

sdb.putItem('<domain>','<itemname>',
  {
    <attr>:'<value>',
    ...
  },
  function(err,res,meta){
    console.log("Memories, you're talking about memories: "+JSON.stringify(res))
  })

Where <itemname> is the unique name of your item, and <attr>:"<value>" are the attribute-value pairs for your item. The value must be either a string or an array of strings.

If you want to use conditional puts, you'll need to add some override values:

sdb.putItem('<domain>','<itemname>',
  {
    <attr1>:'<value>',
    <attr2>:['<value1>','<value2>',...]
    ...
  },
  {
    'Expected.1.Name':'VersionNumber',
    'Expected.1.Value':'1'
  },
  function(err,res,meta){
    console.log("Nobody expects the spanish inquistion! "+JSON.stringify(res))
  })

batchPutItem: `sdb.batchPutItem( domain, items, override, callback )`

domain: (required) the name of the domain
items: (required) the list of items to store
override: (optional) SimpleDB attributes to override function defaults
callback: (required) callback function accepting parameters callback(error, result, metadata)

Store multiple items in the same request. This is more efficient. The items argument is an array of item objects. Each item object must have a $ItemName meta-attribute that specifies the name of the item.

sdb.batchPutItem('<domain>',
  [
    { $ItemName:'<itemname1>', <attr>:'<value>', ...},
    { $ItemName:'<itemname2>', <attr>:'<value>', ...}
  ],function(err,res,meta){
    console.log("And what was your ownership share diluted down to?"+JSON.stringify(res))
  })

batchDeleteItem: `sdb.batchDeleteItem( domain, items, override, callback )`

domain: (required) the name of the domain
items: (required) the list of items to delete
override: (optional) SimpleDB attributes to override function defaults
callback: (required) callback function accepting parameters callback(error, result, metadata)

Delete multiple items in one request. This is more efficient. The items argument is an array of item objects. Each item object must have a $ItemName meta-attribute that specifies the name of the item.

sdb.batchDeleteItem('<domain>',
  [
    { $ItemName:'<itemname1>', <attr>:'<value>', ...},
    { $ItemName:'<itemname2>', <attr>:'<value>', ...}
  ],function(err,res,meta){
    console.log("Done"+JSON.stringify(res))
  })

getItem: `sdb.getItem( domain, itemname, override, callback )`

domain: (required) the name of the domain
itemname: (required) the unique name of the item in the domain
override: (optional) SimpleDB attributes to override function defaults
callback: (required) callback function accepting parameters callback(error, result, metadata)

Get an item from SimpleDB using the item's unique name. The values of the item's attributes are returned as strings. You can provide an $AsArrays meta-directive in the override argument. When true, all attribute values are returned as arrays. As SimpleDb is schemaless, it is not possible to know in advance if an attribute is multi-valued. In the default case, {$AsArrays:false}, multiple values are returned as string, with the value list comma-separated. SimpleDB returns multiple values in alphabetical order.

sdb.getItem('<domain>','<itemname>',function( error , result, meta ){
  console.log("Those are good burgers, Walter: "+JSON.stringify(res))
})

sdb.getItem('<domain>','<itemname>',{$AsArrays:true},function( error, result, meta ){
  console.log("I've been watching television so much the shows are starting to run together: "+JSON.stringify(res))
})

By default, simpledb uses consistent reads. For improved performance, if this is suitable for your application, you can set the consistent option to false when creating simpledb.SimpleDB. Or you can set it on a case-by-case basis, using an override: {ConsistentRead:"false"}

deleteItem: `sdb.deleteItem( domain, itemname, attrs, override, callback )`

domain: (required) the name of the domain
itemname: (required) the unique name of the item in the domain
attrs: (optional) the attributes to delete
override: (optional) SimpleDB attributes to override function defaults
callback: (required) callback function accepting parameters callback(error, result, metadata)

Delete an item from SimpleDB. The attrs argument is optional, and can be:

an array of attribute names: all matching attributes will be deleted
an object whose properties are attribute names: attributes of the item will be deleted if they have the same value as the object properties. Values can be either a single string, or an array of string values, in which case all matching attributes are deleted.

If no attributes are specified, the item is completely removed. If present, only the specified attributes are removed. If all the attributes of an item are removed, then it will also be completely deleted.

sdb.deleteItem('<domain>','<itemname>',function( error, result, meta ){
  console.log("Well, Ted, like I said the last time: it won't happen again: "+JSON.stringify(res))
})

sdb.deleteItem('<domain>','<itemname>',[ '<attr>', ... ], function( error, result, meta ){
  console.log("I felt like destroying something beautiful. "+JSON.stringify(res))
})

sdb.deleteItem('<domain>','<itemname>',
  { '<attr1>': '<value1>', 'attr2': ['<value2>, ... ], ... },
  function( error, result, meta ){
    console.log("I don't know what to write about. "+JSON.stringify(res))
  }
)

select: `sdb.select( query, override, callback )`

query: (required) SimpleDB select expression
override: (optional) SimpleDB attributes to override function defaults
callback: (required) callback function accepting parameters callback(error, result, metadata)

Perform a SELECT-style query on a SimpleDB domain. The syntax is almost-but-not-quite SQL. You should read the Amazon documentation: Using Select

The results are returned as an array of items. Each item contains an $ItemName meta-attribute providing you with the name of the item.

If you need to handle NextToken you'll need to do this manually with the override argument. You can get the NextToken from the meta parameter to your callback.

sdb.select("select * from <domain> where <attribute> = '<value>'",function( error, result, meta ){
  console.log("I'll get you, my pretty, and your little dog too! "+JSON.stringify(result)+" "+JSON.stringify(meta))
})

request: `sdb.request( action, attrs, callback )`

action: (required) SimpleDB action
attrs: (required) SimpleDB request attributes
callback: (required) callback function accepting parameters callback(error, result, metadata)

Make a direct request to SimpleDB. You're on your own! Again, read Amazon SimpleDB Developer Guide Unlike the other functions above, the request function is not a SimpleDB action wrapper. Use it when the wrapper functions have painted themselves into a corner.

sdb.request("<action>",
  {
    <attribute>:"<value>",
    ...
  },
  function( error, result, meta ){
    console.log("Gotta keep 'em separated: "+JSON.stringify(res))
  })

Where <action> is the SimpleDB action, such as GetItem, and <attribute>:"<value>" are the SimpleDB REST request attribute pairs.

client: `sdb.client`

The aws-lib client object. Use this to send raw requests. Go hardcore.

handle: `sdb.handle( start, action, query, tryIndex, last, response, stop, callback, )`

start: Date object, start time of request
action: string, name of SimpleDB action
query: full SimpleDB query
tryIndex: number of tries attempted, up to maxtry
last: true if this is the last request that will be made
response: result from SimpleDB
stop: stop(true|false), function to stop retries in case of errors
callback: action-specific callback, as provided by functions like getItem, putItem, etc.

Replace this with your own implementation to change the handling of SimpleDB responses. Most useful is to modify the response in some way and then call the original function. Also good for testing.

This example counts the number of requests made:

var resultcount = 0

var orighandle = sdb.handle
sdb.handle = function(start,action,query,tryIndex,last,response,stop,callback){
  res.$ResultCount = resultcount++
  orighandle(start,action,query,tryIndex,last,response,stop,callback)
}

Options

The additional options that can be given to simpledb.SimpleDB are:

secure: (optional, default=false), if true, use HTTPS
consistent: (optional, default=true), if true, ask for consistent reads
test: (optional, default=false), if true, don't actually send anything to SimpleDB
host: (optional, default=sdb.amazon.com), SimpleDB host
path: (optional, default=/), SimpleDB path
version: optional), default=2009-04-15, SimpleDB API version
maxtry: (optional, default=4), maximum number of retries when SimpleDB fails
delaymin: (optional, default=0), minimum delay in milliseconds
delayscale: (optional, default=100), delay multiplier, in milliseconds
randomdelay: (optional, default=true), apply a random delay multiplier between 0 and 1
expbase: (optional, default=4), exponent base, for the formula that calculates delay time when SimpleDB fails
nolimit: (optional, default=false), if true, it will return results over the max limit of 2500 with subsequent api requests

Logging

You can provide a logger callback when you are creating the simpledb.SimpleDB object to get notifications of request processing events. A simple logger that prints to STDOUT is provided by simpledb.debuglogger:

var sdb = new simpledb.SimpleDB( {...}, simpledb.debuglogger )

The logger callback accepts the following arguments: logger( type, date, ... )

type: string, one of create, request, handle, error, status
date: a Date object
...: remaining arguments depend on type

For type=create, fired when the simpledb.SimpleDB object is created, the arguments are:

opts: options object
awsopts: aws-lib options

For type=request, fired just before a request is made to SimpleDB, the arguments are:

start: Date object, start time of request
action: string, name of SimpleDB action
query: full SimpleDB query

For type=handle, fired after each response from SimpleDB, the arguments are:

start: Date object, start time of request
action: string, name of SimpleDB action
query: full SimpleDB query
tryIndex: number of tries attempted, up to maxtry
last: true if this is the last request that will be made
response: result from SimpleDB

For type=error, fired after any response with an error, the arguments are:

start: Date object, start time of request
action: string, name of SimpleDB action
query: full SimpleDB query
tryIndex: number of tries attempted, up to maxtry
last: true if this is the last request that will be made
retry: true if a retry will be attempted
err: the error that occurred, an object like {Code:'...',Message:'...'}, where Code is the Amazon error code
res: the result
meta: the request meta data