@astronomer/integration-worker

integration-worker

Worker to read analytics events from a stream and broadcast to a 3rd party API.

Installation

npm install --save @astronomer/integration-worker

Usage

This package exports 2 objects:

• connect: A method to connect to a mongo database.
• IntegrationWorker: A Kinesis consumer that will read events from a stream and broadcast them to a 3rd party API.

connect(options, callback)

connect({
  mongoUrl: process.env.MONGO_URL,
  mongooseOptions: { // these are passed directly to mongoose
      mongos: true,
  },
}, () => {
    // do something after connection succeeds
});

IntegrationWorker(options)

const worker = new IntegrationWorker({
    encryptionKey: 'supersecret',
    integrationCode: 'google-analytics',
    Integration: GoogleAnalytics,
    redisUrl: 'localhost',
    redisPort: 6379, // optional: default 6379
    logger: myLogger // an instance of bunyan
});

Example

Connect to mongo and instantiate an IntegrationWorker. Then pass the created IntegrationWorker to the aws-kcl.

const kcl = require('aws-kcl');
const { connect, IntegrationWorker } = require('@astronomer/integration-worker');
const GoogleAnalytics = require('@astronomerio/integration-google-analytics');

connect({
  mongoUrl: process.env.MONGO_URL,
    mongooseOptions: {
    mongos: true,
  },
}, () => {
  const worker = new IntegrationWorker({
    encryptionKey: process.env.ENCRYPTION_KEY,
    integrationCode: 'google-analytics',
    Integration: GoogleAnalytics,
    redisUrl: 'localhost',
  });
  kcl(worker).run();
});