kued
Extensions for Kue (Daemonization, Checkpointing, etc.)
Workers
Kued simplifies worker creation by organizing Job handling around classes:
const Worker = Worker; // This is where you can register handlers. { this; this; // Retrieve up to 50 messages at a time (default is 1). this; this; } { ; } { ; } { ; }
The queue is available to Worker
s, so the outcome of one job can be another:
const Worker = Worker; // This is where you can register handlers. { this; } { thisqueue; }
Time-based Intervals.
Sometimes you just need tasks executed at a regular interval. For this usecase, we have a special CronWorker
:
const CronWorker = CronWorker; { const cron = "cronTime": "10 * * * * *" "runOnInit": true ; supercron config logger _queue; } { thisqueue; }
Checkpoints.
With distributed workers, sometimes it's important to not process a record for some reason, like if it's older than a last "modified" point. For this use case, Kued provides a "checkpointing" feature that applies a predicate test before processing a record.
Checkpoints wrap a typical subscription (i.e. use of Worker.process()
):
const Worker = Worker;const moment = ; // This is where you can register handlers. { this ; } { const quote = jobdata; thisqueue; }
Bridges.
Another common requirement of a worker is to receive messages from sources other than Kue. A bridge is a mechanism to take messages off of a provider (like IronMQ) and forward it to a Worker
.
const BridgeFactory = BridgeFactory;const bridgeFactory = config logger; // Specific options for the Bridgeconst opts = {}; // This is not implemented yet.const bridge = bridgeFactory;
TaskManager
TaskManager is simply a tiny wrapper around Kue used to build and submit Jobs. It's more of a convenience mechanism for managing the Kue connection outside of using a worker.
const moment = ;const TaskManager = TaskManager;const taskManager = config logger; const stockQuote = symbol: 'ABCD' value: 352 datetime: ; // Simplest form of enqueuing a tasktaskManager; // Omit the Callback and get the Kue job (make further adjustments as necessary,// but don't forget to call `save()` to submit the job.taskManager;
Daemonizing
Worker config file: workers.json
And to start that workers, just use the workers.js
entrypoint:
# Your project
npm install
./node_modules/.bin/kued workers --config=workers.json
Evergreen Integration
Evergreen is side project of ours for making Node configuration easier and more powerful. We created a simple wrapper to allow a Workgroup
to be spawned after importing an Evergreen compatible configuration file.
# Get the range of options by specifying the --help flag
everkued --help
# Example of spawning a Workgroup:
everkued --config config.json -workgroup ImageProcessor \
--workers-path foo.bar.workers --modules trbl-evergreen-mongo
Keep in mind, this is used to spawn only one workgroup.
Task Enqueuing from Cli
# Your project
npm install
./node_modules/.bin/kued task --config=tasks.json taskname --params