Cloudant Envoy (beta)
Beta software
Note: this is beta; it's not battle tested or supported in any way. If you find bugs (of which there will be plenty), do let us know – or better, consider a pull request. You know what beta means, right?
Introduction
Cloudant Envoy is a microservice that acts as a replication target for your PouchDB web app or Cloudant Sync-based native app. Envoy allows your client side code can adopt a "one database per user" design pattern, with a copy of a user's data stored on the mobile device and synced to the cloud when online, while invisibly storing all the users' data in one large database. This prevents the proliferation of database that occurs as users are added and facilitates simpler backup and server-side reporting.
Envoy implements a subset of the CouchDB API and can be used as a replication target for PouchDB or Cloudant Sync, or used with custom replicators such as pouchdb-envoy.
Envoy includes a demo web app (hosted at the /demo/
path) which demonstrates how a basic offline-first, progressive web app can be deployed that scales easily as users are added.
Why Cloudant Envoy?
Database-per-user is a common pattern with CouchDB when there is a requirement for each application user to have their own set of documents which can be synced (e.g. to a mobile device or browser). On the surface, this is a good solution - Cloudant handles a large number of databases within a single installation very well. However, there are some problems:
- Querying data across many databases (e.g. for analytics) can be difficult. Cloudant does not have native support for a cross-database query and aggregating the data into a single database using e.g. replication can be resource intensive or require manual scheduling.
- There is a high management cost of many (thousands of) databases. Consider backup, replication in cross-region scenarios, design document management. These all require coordination / resource management outside of Cloudant.
Envoy aims to work around these problems by emulating database-per-user using a single backend database and a proxy.
When should I consider Envoy instead of a db-per-user pattern?
- you want to perform queries across all per-user databases.
- you want to replicate all user data to a remote target (e.g. another Cloudant account or IBM dashDB).
- you want to take advantage of additional HTTP features (e.g. compression) that are not natively supported by Cloudant.
When should I consider Envoy instead of a single database with filtered replication?
Cloudant Envoy is essentially performing a filtered replication under the hood (using the new Mango/Erlang native filtering supported in CouchDB 2.0). However, you may want to consider it if:
- you do not want to give all users read/write access to other users data.
- you want to take advantage of additional HTTP features (e.g. compression) that are not natively supported by Cloudant.
TODO / known limitations
- multipart requests
- document ownership is non-transferable (essentially the same as db per user)
- _design documents cannot be created through the proxy (they will get saved as normal documents)
Installation
Via npm
Cloudant Envoy is published to npm. It can be installed and run if you have Node.js and npm installed:
npm install -g cloudant-envoyexport COUCH_HOST='https://key:passwd@account.cloudant.com'envoy
Deploy to Bluemix
Deploy Cloudant Envoy to Bluemix by clicking the Deploy to Bluemix button below.
Don't have a Bluemix account? If you haven't already, you'll be prompted to sign up for a Bluemix account when you click the button. Sign up, verify your email address, then return here and click the the Deploy to Bluemix button again. Your new credentials let you deploy to the platform and also to code online with Bluemix and Git. If you have questions about working in Bluemix, find answers in the Bluemix Docs.
Manual installation
To install the code yourself, clone the repo and run npm install
. The Envoy server needs admin credentials for the backing Cloudant database, and it expects a COUCH_HOST
environment variable to be set:
export COUCH_HOST='https://key:passwd@account.cloudant.com'
After those variables are set, you can start the Envoy server with npm start
. Note that the port is the port that Envoy will listen to, not the port of the Cloudant server.
Configuration
Environment variables
- PORT - the port number Envoy will listen on. When running in Bluemix, Envoy detects the Cloud Foundry port assigned to this app automatically. When running locally, you'll need to provide your own e.g.
export PORT=8001
- COUCH_HOST - The URL of the Cloudant service to connected to. Not required in Bluemix, as the attached Cloudant service is detected automatically.
COUCH_HOST
is required when running locally e.g.export COUCH_HOST='https://key:passwd@account.cloudant.com'
- ENVOY_DATABASE_NAME - the name of the Cloudant database to use. Defaults to
envoy
- LOG_FORMAT - the type of logging to output. One of
combined
,common
,dev
,short
,tiny
,off
. Defaults tooff
. (see https://www.npmjs.com/package/morgan) - DEBUG - see debugging section
- ENVOY_AUTH - which authentication plugin to use. One of
default
,couchdb_user
- ENVOY_ACCESS - which access control plugin to use. One of
default
,id
,meta
- PRODUCTION - when set to 'true', disables the
POST /_adduser
endpoint - ENVOY_STATIC - when set to a full path, this directory is served out as static content from the Envoy express app.
- ENVOY_USERS_DATABASE_NAME - the name of the database to store users when using the
default
auth plugin. Defaults toenvoyusers
.
Using Envoy in your own app
You can install Envoy and run it from your own Node.js application by install the module:
npm install --save cloudant-envoy
And then "require" it into your app:
var envoy = ;
where it will pick up its configuration from environment variables. You may also pass in object to Envoy on startup with your runtime options e.g.:
var opts = couchHost: 'http://username:password@mycluster.cloudant.com' databaseName: 'myenvoy' port: 9000 logFormat: 'dev' production: true ; var envoy = opts; envoyevents;
Sessions
By default Envoy uses the Express Session session handler. This is an in-memory store and is only designed for test deployments. If you want to use any of the compatible session stores you may pass in a sessionHandler
option at startup e.g.:
var session = ; var RedisStore = session; var path = ; var options = url: 'redis://127.0.0.1:6379/0' ; var sessionHandler = ; var opts = sessionHandler :sessionHandler port: 9000 ; var envoy = opts;
Static files
You can get Envoy to server out static files for you. Simply pass in a 'static' option or use the ENVOY_STATIC environment variable:
var opts = port: 9000 static: path ; var envoy = opts;
Custom routes
You can supply your own custom routes to Envoy too:
var express = router = express; // my custom routerouter; var opts = databaseName: dbName port: appEnvport logFormat: 'dev' production: true static: path router: router; var envoy = opts;
Custom middleware
You can supply your own custom middleware to Envoy
var { console };var opts = databaseName: dbName port: appEnvport logFormat: 'dev' production: true static: path router: router middleware: myLogger;var envoy = opts;
The middleware
property expects an array of Express middleware functions.
Debugging
Debugging messages are controlled by the DEBUG
environment variable. To see detailed debugging outlining the API calls being made between Envoy and Cloudant then set the DEBUG
environment variable to cloudant,nano
e.g
export DEBUG=cloudant,nanonode app.js
or
DEBUG=cloudant,nano node app.js
Envoy API end points
Envoy supports a subset of the CouchDB API, sufficient to support replication. Additionally, a new Envoy-specific endpoint _add_user
is there to facilitate testing; see below.
Replication | CRUD | Utility |
---|---|---|
GET /db/_all_docs | GET /db/id | POST /_adduser |
POST /db/_all_docs | POST /db | GET /_auth |
POST /db/_bulk_docs | POST /db/id | GET /_logout |
POST /db/_bulk_get | PUT /db/id | OPTIONS /* |
GET /db/_bulk_get | DELETE /db/id | |
GET /db/_changes | GET / | Search |
POST /db/_revs_diff | GET /db | POST /db/_find |
Envoy-specic APIs
GET /_auth
Allows a remote client to either check whether it is logged in or to establish a login session (credentials are passed using an HTTP Basic Authorization header):
# not logged in - Envoy returns 403 response > curl https://myenvoy.mybluemix.net/_auth# log in - Envoy returns 200 response and saves cookie > curl https://myenvoy.mybluemix.net/_auth --user glynn:password{"loggedin":true,"username":"glynn"}
POST /_auth
Allows a remote client to either check whether it is logged in or to establish a login session (credentials are passed via a JSON-formatted HTTP POST):
# not logged in - Envoy returns 403 response > curl https://myenvoy.mybluemix.net/_auth# log in - Envoy returns 200 response and saves cookie > curl -H "Content-Type: application/json" -X POST -d '{"name":"glynn","password":"password"}' https://myenvoy.mybluemix.net/_auth{"loggedin":true,"username":"glynn"}
GET /_logout
Allows a remote client to logout of a session.
> curl https://myenvoy.mybluemix.net/_logout
POST /_adduser
Allows the creation of new Envoy users. Supply a username
and password
parameter in a form-encoded post e.g.
> curl -X POST -d 'username=rita&password=password' https://myenvoy.mybluemix.net/_adduser
This API is only for evaluation purposes and should be disabled in production, by setting the PRODUCTION
environment variable.
Plugins
At startup, Envoy can load in plugin modules to modify Envoy's behaviour. The plugin source code can be found in the lib/plugins
directory, with two plugin types supported:
auth
– controls how authentication occurs in Envoy.access
– controls how the ownership of a document is stored.
Default auth plugin
The default
auth plugin uses the envoyusers
database (by default) to store a database of users who are allowed to authenticate. Add a user to your envoyusers
database with a document like this:
"_id": "glynn" "_rev": "1-58bbb25716001c681febaccf6d48a9b8" "type": "user" "name": "glynn" "roles": "username": "glynn" "salt": "monkey" "password": "9ef16a44310564ecd1b6894c46d93c58281b07af"
Where the password
field is the sha1
of the "salt" field concatenated with the user's password e.g.
> '9ef16a44310564ecd1b6894c46d93c58281b07af'
Additional plugins can be selected using the ENVOY_AUTH
environment variable. e.g. a value of "couchdb_user" will use the CouchDB "_users" database.
Default access plugin
The default
access plugin stores the ownership of a document in the _id
field of the document. If the owner of the document is glynn
, then a document whose _id
is test
will actually get stored with an _id
of
3d07659e67fa5a86a06945ec4cdb2754c9fc67bf-test
where
3d07659e67fa5a86a06945ec4cdb2754c9fc67bf
is the sha1
hash of the username.
The replication client doesn't see this additional meta data – it is transparently added and stripped out on its way through Envoy – but it allows Envoy to efficiently store many users data in the same database and retrieve the data each user owns efficiently.
Additional plugins can be selected using the ENVOY_ACCESS
environment variable.
Frequently Asked Questions
Help!
If you have an issue with Envoy, then please check to see if someone else has raised it already in our issues page. Please raise an issue for any problems you encounter and we'll see if we can help.