Petit Service 🍬
Set of helpers designed for a Microservice architecture.
Available helpers:
- Service Loader
- AMQ publisher/consumer
- Redis cache
- MongoDb client
- Express common middlewares
- Google Cloud Monitoring (Trace, Debug, Errors)
- Logger
- Database helpers
- Database tasks
Service Loader
Start different services (sequentially) with graceful exit handler.
Supported features:
- Check if TCP hosts are reachable
- Start Redis cache manager
- Start MongoDB client
- Start AMQ publisher/consumer
- Start postgres database connection using knex (check if it's alive every 30 sec)
- Start HTTP server (express or http)
- Gracefully exit all services on exceptions
Full Example:
const serviceLoader = require('petitservice/lib/serviceLoader');
const logger = require('petitservice/lib/logger');
const config = require('./config');
return serviceLoader()
.ping([
config.apiUrl,
config.redisUrl,
config.amqpUrl,
config.mongoUrl,
])
.cache({ host: 'localhost', port: 6379, auth_pass: 'xxx' })
.mongo(config.mongoUrl)
.amq({
amqUrl: 'amqp://guest:guest@localhost:5672',
consumerQueue: 'my-consumer-queue',
assertQueues: ['my-publisher-queue'],
consumer: (data, ack) => {
logger.info(data);
ack(); // acknowledge the message
},
})
.then(() => {
logger.info('Do something during starting...');
})
.db({
pgUrl: config.pgUrl,
pgDatabase: config.pgDatabase,
})
.express(() => require('./express_app.js'), config.httpPort)
.onExit(() => {
logger.info('Do something during exit...');
})
.done(() => {
logger.info('Everything is started!');
});
Available functions chains:
-
ping(urlList, [options]): Check if hostnames are alive
-
urlList
(array of url to ping) format:protocol://[user:pass@]hostname:port
-
options.failureMax
(optional integer, how many attempts should we try before we exit the process, default: 5) -
options.frequency
(optional integer, how many milliseconds should wait before checking again hostnames, default: 30000)
-
-
db([options]): Initiate database, checks if database is alive and destroy knex on exit
-
options.pgUrl
(optional string, postgres url, default: set in./lib/db/config.js
) -
options.pgDatabase
(optional string, database to query, default: postgres) -
options.failureMax
(optional integer, how many attempts should we try before we exit the process, default: 5) -
options.frequency
(optional integer, how many milliseconds should wait before checking again the database, default: 30000)
-
-
cache(redisOpts): Start cache for
petitservice/lib/cache
-
redisOpts
(required object, { host, port, auth_pass })
-
-
mongo(mongoUrl): Start mongoDb for
petitservice/lib/mongo
-
mongoUrl
(required string, mongo url)
-
-
amq(options): Connect to RabbitMQ using amqp.node, and close it when the program exit.
options
is required.-
options.amqUrl
(required string, amq url) -
options.assertQueues
(optional array, list of queue to create if they haven't been created before. if consumerQueue is set, it will be asserted as well automatically. -
options.consumerCb(data, ack)
(required function): handles a message to consume. It gets as parameter respectively the receiveddata
(already JSON parsed) andack
function to run when we want to acknowledge the message. -
options.consumerQueue
(required string): consumer queue name -
options.consumerPrefetch
(optional integer): how many message we consume simultaneously - default: 1 -
options.onError(err, msg)
(optional function): error handler when there's an exception on the consumer. Gets as parametererr
as error object andmsg
as raw message. Defaults a warning message.
-
-
express(expressApp, port): Start express HTTP server, and close it when exit
-
expressApp
(required function that returns express app, https://github.com/expressjs/express) - We advice you to use the require inside this function. -
port
(integer, HTTP port. default:80
)
-
-
then(cb): Run a function during starting
-
cb
(function that performs action, can return a promise as well)
-
- done([callback]): Add this at the end of the chain to start the service. it can take a callback function as parameter that executes when everything is loaded.
-
onExit(cb): Action to perform when closing Gracefully
-
cb
(function that performs action, can return a promise as well)
-
AMQ publisher/consumer
Connect to RabbitMQ using amqp.node, Asserts queues, and consume messages from a queue.
Example using serviceLoader
const serviceLoader = require('petitservice/lib/serviceLoader');
const amq = require('petitservice/lib/amq');
serviceLoader()
.amq({
amqUrl: 'amqp://guest:guest@localhost:5672',
consumerQueue: 'my-consumer-queue',
assertQueues: ['my-publisher-queue'],
consumer: (data, ack) => {
logger.info(data);
ack(); // acknowledge the message
},
})
.done(() => {
amq.publish({ myKey: 'myValue' }, 'my-publisher-queue');
});
Example without serviceLoader
const amq = require('petitservice/lib/amq');
// Using serviceLoader
amq.start({
amqUrl: 'amqp://guest:guest@localhost:5672',
consumerQueue: 'my-consumer-queue',
assertQueues: ['my-publisher-queue'],
consumer: (data, ack) => {
logger.info(data);
ack(); // acknowledge the message
},
})
.then(() => {
// publish a message
amq.publish({ myKey: 'myValue' }, 'my-publisher-queue');
});
Available methods:
- start(options): Connect to RabbitMQ and assert publisher/consumer. Documentation on the options is available on serviceLoader
-
publish(payload, queueName, [options]): Publish a payload to a queue.
- payload: (object) data to publish
- queueName: (string) where we'd like to publish the data
-
options: (optional object) Publication options (for sendToQueue), default
{ persistent: false, expiration: 60000 }
- close(): Close amq connection
- getChannel(): returns active channel
- getConnection(): returns active connection
Redis Cache
Cache manager using Redis
Full example
const cache = require('petitservice/lib/cache');
const logger = require('petitservice/lib/logger');
cache.start(config.redisUrl);
const userId = 12;
const getUser = (userId) => models.getUser(userId);
// Get / Set / Delete
const cacheKey = 'bob';
cache.getValue(cacheKey)
.then((cachedValue) => {
if (!cachedValue) {
logger.info(`Setting value for ${cacheKey}`);
return cache.getValue(cacheKey, 'alice', 60);
}
logger.info(`I remember ${cachedValue}`);
})
.then((cachedValue) => {
logger.info(`Bye ${cacheKey}`);
return cache.delValue(cacheKey);
});
// Wrap a function
cache.wrap(userId, () => models.getUser(userId), 10)
.then((cacheUser) => {
logger.info(`Bonjour ${cacheUser.firstName}`);
});
// Delayed Execution
const id = 'abc';
const prm = () => {
return models.insertKeystroke(id, Math.random());
}
cache.delayedExec(id, prm, 10); // <= prm will be discarded after 10 sec
cache.delayedExec(id, prm, 10); // <= prm will be resolved after 10 sec
Available methods:
- start(redisUrl): Instantiate the cache so that we can call get/set data.
- getValue(key): Get value by its key
- setValue(key, value, ttl): Set a value using a key (ttl - Time to live - is in seconds)
- delValue(key): Delete a value by its key
-
wrap(key, fallbackPromise, ttl, isJSON): Wrap a promise in cache.
- key: (string) identifier
- fallbackPromise: (function that returns a promise) How we get the data to read/write
- ttl: (integer) Time to live in seconds
- isJSON: (boolean - default:false) encode objects to a JSON before saving into the cache
-
delayedExec(identifier, prm, delayTime): Delay an promise execution of a promise across different microservices. The promise is resolved only if not another delayedExecution has been trigged during the same timeframe (delayTime).
- identifier: (string) how we identify this execution
- prm: (function that returns a promise) the promise that we want to execute
- delayTime: (integer - in seconds) timeframe when there wasn't any delayed execution with the same identifier
MongoDb client
MongoDB helpers
Full example
const mongo = require('petitservice/lib/mongo');
const logger = require('petitservice/lib/logger');
return mongo.start(config.mongoUrl)
.then(() => {
const db = mongo.db(config.mongoDbname);
const collection = db.collection('documents');
// Insert some documents
return collection.insertMany([
{a : 1}, {a : 2}, {a : 3}
])
.then(() => collection.find({}).toArray())
.then((docs) => logger.info(docs));
})
.then(() => mongo.close());
Available methods:
- start(mongoUrl): Instantiate the mongoDb client
- db(key): Get database instance
- close(): Close mongodb client
Express common middlewares
Set common middlewares for an express app
Full example
const express = require('express');
const expressMiddleWare = require('petitservice/lib/expressMiddleWare');
const app = express();
expressMiddleWare.addStandard(app);
expressMiddleWare.addCompression(app);
expressMiddleWare.addLogs(app);
app.get('/', (req, res) => {
res.send('Bonjour!');
});
expressMiddleWare.addErrorHandlers(app);
Available methods:
-
addStandard(app): Add the following middlewares:
- Disable 'x-powered-by' header
- Define 'trsut proxy' to accept forwared ip
- Body parser (json and form data)
- Set a health check endpoints '/~health' that returns 'ok'
- Remove trailing slashes on urls
- addCompression(app): Adds GZIP compression middleware
- addLogs(app): logs http request using the logger module (See section about logger)
- addErrorHandlers(app, isHTML):
- Report errors to google cloud engine if it's defined
- Endpoint to handle not found pages (if isHTML is set to true it will render the view
404
) - Endpoint to handle internal errors (if isHTML is set to true it will render the view
500
)
Google Cloud Monitoring (Trace, Debug, Errors)
Monitoring using Google Stackdriver: Debug, Trace, Errors.
Full Example:
const gcloud = require('petitservice/lib/gcloud');
// Environment variable:
// - ENABLE_GCLOUD_TRACE: "1"
// - ENABLE_GCLOUD_ERROR: "1"
// - ENABLE_GCLOUD_DEBUG: "1"
// - GCLOUD_PROJECT: "my-project"
// - GCLOUD_STACKDRIVER_CREDENTIALS: "xxxx"
gcloud.init(process.cwd(), {
trace: {
ignoreUrls: [/^\/asserts/, /\/~*health/],
}
});
Available methods:
-
init(projectRootDirectory, [options]): Initiate gcloud
- options: (optional object) more details below
- projectRootDirectory: (required, string) Project root directory (where package.json is located)
- reportError(): Report an error to gcloud-errors, error must be an Error object
- expressMiddleWare(): gcloud-errors express middleware
- startSpan(): gcloud-trace startSpan (see trace documentation)
- endSpan(): gcloud-trace endSpan (see trace documentation)
- runInSpan(): gcloud-trace runInSpan (see trace documentation)
- runInRootSpan(): gcloud-trace runInRootSpan (see trace documentation)
Available options:
- credentials: object, gcloud credentials (default: base64decode(GCLOUD_STACKDRIVER_CREDENTIALS))
- trace: object, options to override default configuration: https://github.com/GoogleCloudPlatform/cloud-trace-nodejs/
- debug: object, options to override default configuration: https://github.com/GoogleCloudPlatform/cloud-debug-nodejs/
- error: object, options to override default configuration: https://github.com/GoogleCloudPlatform/cloud-errors-nodejs/
Environment variables:
- GCLOUD_STACKDRIVER_CREDENTIALS: required string, base64 of the gcloud json key
- GCLOUD_PROJECT: required string: gcloud project name
- ENABLE_GCLOUD_TRACE: option binary: Enable gcloud trace
- ENABLE_GCLOUD_ERROR: option binary: Enable gcloud error reporting
- ENABLE_GCLOUD_DEBUG: option binary: Enable gcloud debug
- GCLOUD_DEBUG_LOGLEVEL: Log level for gcloud/debug (default: 1)
- GCLOUD_TRACE_LOGLEVEL: Log level for gcloud/trace (default: 1)
- GCLOUD_ERRORS_LOGLEVEL: Log level for gcloud/error (default: 1)
Logger
Log data on the console (using winston), and report errors to gcloud/error if enabled
The default LogLevels depends on the NOD_ENV:
-
debug
fordevelopment
env -
info
forproduction
env -
error
fortest
env
Full Example:
// You may also set the log level using the environment variable: LOG_LEVEL: 'debug'
const logger = require('petitservice/lib/logger');
logger.debug('bonjour');
logger.info('un café et un croissant chaud');
logger.error(new Error('Something broke'));
// You can use middlewares for express
// Request logs
app.use(logger.requestLogger);
// Error logs
if (logger.gcloudErrorsMiddleWare) {
app.use(logger.gcloudErrorsMiddleWare);
}
app.use(logger.errorLogger);
Exported methods:
- requestLogger: Express middleware to log requests
- errorLogger: Express middleware to log errors
- gcloudErrorsMiddleWare: Express middleware to report express errors to gcloud
- error
- outputError (like error, but without reporting that to gcloud)
- warn
- info
- log
- verbose
- debug
- silly
Database helpers
Connect to a Postgres database using knex.
Check section Service Loader for details to initiate the database
Full Example:
const serviceLoader = require('petitservice/lib/serviceLoader');
const logger = require('petitservice/lib/logger');
const db = require('petitservice/lib/db');
serviceLoader()
.db({
pgUrl: 'postgres://root:@localhost',
pgDatabase: 'postgres',
})
.done(() => {
const db = db.getKnexObject();
db.raw('SELECT 1;')
.then((data) => {
logger.info(data);
});
});
Exported methods:
- init([pgUrl], [pgDatabase]): Initiate knex object in memory
-
pgUrl: (option string, postgres url, default: set in
./lib/db/config.js
) -
pgDatabase: (option string, postgres database, default: set in
./lib/db/config.js
) - getKnexObject(): returns knex object
Used environment variables
- POSTGRES_PORT_5432_TCP_ADDR: Postgres hostname
- POSTGRES_PORT_5432_TCP_PORT: Postgres port
- POSTGRES_ENV_POSTGRES_USER: Postgres username
- POSTGRES_ENV_POSTGRES_PASSWORD: Postgres password
- PG_DATABASE: Postgres database
Database tasks
require('petitservice/lib/db/tasks')
:
Available tasks on -
run(action, database): Run a task to the database
- action: can be
createdb
,dropdb
,migrate
,seed
,init
(createdb + migrate + seed),refresh
(dropdb + init) - database: database name
- action: can be
- createdb(database): Create database
- dropdb(database): Drop database
- migrate(database): Migrate database
- seed(database): Seed database
Full example:
node ./dbTasks.js createdb development
// dbTasks.js
const tasks = require('petitservice/lib/db/tasks');
const pgDatabases = {
production: 'myDb',
test: 'myDb_test',
development: 'myDb_dev',
};
const env = process.argv[process.argv.length - 1];
const action = process.argv[process.argv.length - 2];
tasks.run(action, pgDatabases[env]);
Generate GCLOUD_STACKDRIVER_CREDENTIALS
Enable API
https://console.cloud.google.com/flows/enableapi?apiid=clouddebugger.googleapis.com
https://console.cloud.google.com/flows/enableapi?apiid=cloudtrace.googleapis.com
https://console.cloud.google.com/flows/enableapi?apiid=logging.googleapis.com
https://console.cloud.google.com/flows/enableapi?apiid=clouderrorreporting.googleapis.com
If not running on google cloud
# CREATE GCLOUD_STACKDRIVER_CREDENTIALS
PROJECT_NAME="***"
gcloud iam service-accounts create stackdriver --display-name=stackdriver
gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member serviceAccount:stackdriver@${PROJECT_NAME}.iam.gserviceaccount.com --role roles/cloudtrace.admin
gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member serviceAccount:stackdriver@${PROJECT_NAME}.iam.gserviceaccount.com --role roles/clouddebugger.agent
gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member serviceAccount:stackdriver@${PROJECT_NAME}.iam.gserviceaccount.com --role roles/logging.admin
gcloud projects add-iam-policy-binding ${PROJECT_NAME} --member serviceAccount:stackdriver@${PROJECT_NAME}.iam.gserviceaccount.com --role roles/errorreporting.admin
If not running on google cloud
Contribute
You are welcomed to fork the project and make pull requests. Or just file an issue or suggestion 😊