flume-spool

The repository is to support flume spooling directory source for node.

Note: only support Node > 4.0.0 and Apache Flume > 1.3.0

Installation

$ npm install flume-spool

Usage

Opening a log stream:

const FlumeSpool = require('flume-spool')
const log = new FlumeSpool('/path/to/spool/directory', [options])
 
log.write('some logs \n')

See the examples/ folder for more examples.

Options

Available options:

Name	Type	Default	Description
interval	number	60000(60s)	The time interval for transfering files from temporary directory to spool directory.
tempDir	string	./flume_temp	The directory for temporary logs.
autoDelete	boolean	false	Remove completed files using node.
autoDeleteInterval	number	600000(10min)	---
autoDeleteSuffixReg	regexp	/.COMPLETED$/	---
streamEncoding	string	utf8	The config for the log stream.
streamMode	number	0644	---
streamFlags	string	a	---

Methods

• write(chunk<String|Buffer>): write data to log stream
• close(): close log stream

Flume Config

Example for the agent using spooling directory source:

agent.sources.s1.type = spooldir
agent.sources.s1.channels = ch1
agent.sources.s1.spoolDir = /path/to/spool/directory
agent.sources.s1.fileHeader = true
# auto delete completed files
agent.sources.s1.deletePolicy = immediate

See FlumeUserGuide for all available options.

Events

The instance will emit some events in running:

Event	Description
open	emit when being initialized
transfer	emit when the temporary file is transfered
close	emit when being closed
error	emit when an error occurs

Tests

$ npm test

License

Apache-2.0