@momsfriendlydevco/agents

Deferred async job scheduler and runner for local or remote batch queues.

Features:

• Run background jobs periodically, as required or with use-cache-or-rerun methods
• Supports multiple runners
• Can be used in JavaScript as a module or from the command line as a runner
• Supports both CJS and ESM modules by default

var Agents = require('@momsfriendlydevco/agents');

var agents = new Agents();

// Calculate the first 100 prime numbers within PM2 / AWS or some other runner
agents.get('primes', {limit: 100}) 
	.then(result => { ... }) // Do something with the result


// Set up a job to calculate the first million prime numbers
// This doesn't return the result but the session we can use to ask about its status
agents.get('primes', {limit: 1e6}, {want: 'session'}) 
	.then(session => {
		// Set up a function to page the session and ask if its done yet
		var checkSession = ()=> agents.getSession(session).then(session => {
			if (session.status == 'complete') {
				console.log('Yey there are', session.result.length, 'primes');
			} else {
				setTimeout(checkSession, 1000); // Check again in 1s
			}
		});
		checkSession(); // Kick off the initial check
	})

ESM / ES6 compatibility

This module is CJS and ESM compatible by default via the Inclusion transpiler.

However if a newer require system is required, the require option can be overloaded to use a third party module to pre-compile ESM modules to CJS:

var agents = new Agents({
	require: path => myCustomRequire(path)
		.then(mod => mod.default || mod), // Other cleanup methods when done
});

Adding agents to package.json

Add the agent binary command to your package.json:

{
  "name": "abi-ims",
  "version": "0.0.0",
  "description": "Inventory Management Sub-system for ABI",
  "type": "module",
  "main": "server.js",
  "scripts": {
    "agents": "agents -l",
    "agents:mintProductsDelta": "agents myAgentToRun --command-line-options"
  },
  "agents": {
    "paths": [
      "./agents/*.js"
    ],
    "runner": {
      "modules": [
        "pm2",
        "inline"
      ]
    }
  }
}

The above maps npm run agents to listing the available agents within a project, run run agents:mintProductsDelta runs a specific agent with the specified command line. The agents branch contains the various defaults the agent binary will use when executing such as the default paths to search and the runners to use.

Module API

new Agents([settings])

Create an agents instance and set its initial objects.

The following settings are supported (in dotted form):

Setting	Type	Default	Description
autoInit	boolean	false	Whether constructing the object will also call agents.init() automatically. Use agents.on('ready') to trap when the agent has loaded
autoInstall	boolean	true	Whether any timed jobs should be installed via Cron
allowImmediate	boolean	true	Allow any agents detected as {immediate: true} to execute
checkProcess	number	1000	Interval between PM2 process status checks
logThrottle	number	250	Delay between log output from agent.logThrottle() or agent.progress()
paths	array	['examples/**/*.agent.js']	Paths to search for agent files
keyRewrite	function	key => key	How to mangle the allocated cacheKey when assigning agents
cache	object	See below	The cache options. All settings except cache.calculate are specific to @momsfriendlydevco/cache
cache.calculate	function	See notes	Function used to select the caching module to use
runner	object	See below	The runner options
runner.modules	array	['inline']	Supported runner modules
runner.calculate	function	session => 'inline'	How to select the applicable runner
runner.pm2	object	See below	Various configuration options for PM2
runner.pm2.procName	function	cacheKey => cacheKey	The name of the PM2 process, calculated from the cacheKey
runner.pm2.execFile	string	${__dirname}/agents.js	The code Node agents.js file used to execute the inline process
runner.pm2.execFileInterpreter	string	"node"	The interpreter to set for the agents.js file
runner.pm2.execFileInterpreterArgs	array	See code	Arguments passed to the interpreter of agents.js
runner.pm2.cwd	string	${__dirname}	The working directory of the agents.js file
runner.pm2.env	function	See code	The environment to pass to the agents.js file
runner.pm2.logFileScan	boolean	true	Examine the main pm2 log file to determine additional detail if PM2 claims a process exited correctly
runner.pm2.logFilePath	boolean	$HOME/.pm2/pm2.log	Log file to examine for process exit information
runner.pm2.logFileTailSize	number	2048	How many bytes backwards from the end of the main PM2 log file to read
agentDefaults	object	See code	Options set as defaults when reading in each agent file

Notes:

• The cache used is @momsfriendlydevco/cache the all of the cache settings excepting calculate are passed to it to initialize the caches
• The default cache.calculate code simply selects the first module specified in cache.modules

agents.init([settings])

Initialize the agent manager, including starting all runners and caches. Returns a promise.

agents.settings

The storage object for the agents settings, use agents.set() to populate this conveniently. See the main constructor for a list of settings.

agents.set(key, [val])

Set a single key (dotted or array notation is supported) or merge an entire object into agents.settings Returns the agents chainable object.

agents.refresh()

Function which refreshes available agents, this is automatically called by agents.init(). Returns a promise.

agents.has(id)

Asks if the supplied agent ID is valid. Returns a boolean.

agents.destroy()

Asks the agents object to clean up all caches, runners and any other async objects. Returns a promise.

agents.get(id, [agentSettings], [settings])

Either returns a cached agent value or, if none are found, calls agents.run() automatically to obtain one. Returns a promise.

agents.getSize(id, [agentSettings], [settings])

Returns the size of a cached response or undefined if none is found. Returns a promise.

agents.run(id, [agentSettings], [settings])

This is the main function which runs an agent and waits for its response. Returns a promise.

agents.invalidate(id, [agentSettings], [settings])

Invalidates an agent response - that is, this function will remove the cached value. This causes subsequent calls to agents.get() to recalculate. Returns a promise.

agents.list()

Returns a list of all available agents, their cache status and other meta information. Returns a promise.

Agent API

The following is a minimal example of an agent file:

module.exports = {
	id: 'myAgent',
	hasReturn: true,
	immediate: false,
	methods: ['pm2', 'inline'],
	triggers: ['build'],
	expires: '1h',
	worker: function() {
		var agent = this;
		// ... Agent work here ... //
	},
};

See the examples directory for more comprehensive examples.

Agents have the following settings:

Setting	Type	Default	Description
id	string		The unique ID identifying the agent
worker	function		Worker function to call as (callback, settings)
clearOnBuild	boolean	false	Whether the agent cache should be invalidated on any system build
expires	boolean / string	false	Additional cache expiry options to hold the agent response
hasReturn	boolean	true	Whether the agent is expected to return a value, an warning occurs if the agent does not
immediate	boolean	false	Indicates that the agent should always be executed when the agent module loads the agent for the first time
methods	array<string>	['inline', 'pm2']	Method types this agent is compatible with
path	string	Calculated	Calculate as the source agent path when refreshing
show	boolean	true	Whether to show the agent in the agent list
triggers	triggers	[]	Additional triggers that make the agent run. Examples could include things like "boot", "build" etc.

Notes:

• All defaults are configurable in agents.settings.agentDefaults

When executed all agent functions are given a context which has access to the following convenience API methods:

agent.log(...msg)

Output generic information. This is the same functionality as console.log() but can be captured by the upstream agent invoker.

agent.logThrottled(...msg)

Functionally the same as agent.log() but this function is designed to deal with blast data by throttling the actual output.

agent.log.flush()

Manually release any throttled log content.

agent.warn(...msg)

Output a warning. This is the same functionality as console.warn() but can be captured by the upstream agent invoker.

agent.log.colors

Convenience object which provides a Chalk instance.

agent.progress([title], currentProgress, [maxProgress])

Output a progress indicator using agent.logThrottled(). Upstream agent invokers have access to this output and could present a progress bar of some kind.

Tips

Recommended emitter bindings

var prefix = colors.blue('[agents]');

agents
	.on('init', ()=> console.log(prefix, 'Created agents interface'))
	.on('ready', ()=> console.log(prefix, 'Agents interface ready'))
	.on('destroy', ()=> console.log(prefix, 'Destroying agents interface'))
	.on('destroyed', ()=> console.log(prefix, 'Destroyed agents interface'))
	.on('refreshWarn', (path, msg) => console.log(prefix, colors.yellow('WARNING'), colors.cyan(path), msg))
	.on('refresh', ids => console.log(prefix, 'Loaded agents:', ids.map(i => colors.cyan(i)).join(', ')))
	.on('tick', id => console.log(prefix, 'Refreshing agent', colors.cyan(id), 'from cron timing', colors.cyan(agents.agents[id].timing)))
	.on('scheduled', id => console.log(prefix, 'Installed agent', colors.cyan(id), 'with timing', colors.cyan(agents.agents[id].timing)))
	.on('runImmediate', id => console.log(prefix, 'Agent', colors.cyan(id), 'marked for immediate run!'))
	.on('log', (session, ...args) => console.log.apply(this, args))
	.on('warn', (session, ...args) => console.log.apply(this, ['WARN'].concat(args)))