regexp-stream-tokenizer
This is a simple regular expression based tokenizer for streams.
IMPORTANT: If you return null
from your function, the stream will end there.
IMPORTANT: Only supports object mode streams.
var tokenizer = ; var words = ; // Sink receives tokens: 'The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog'words;words // Separators are excluded by default, but can be includedvar wordsAndSeparators = ; // Sink receives tokens: 'The', ' ', 'quick', ' ', 'brown', ' ', 'fox', ' ', 'jumps', ' ', 'over', ...words;words
API
options regexp
Create a stream.Transform
instance with objectMode: true
that will tokenize the input stream using the regexp.
var Tx =
Create a reusable stream.Transform
TYPE that can be called via new Tx
or Tx()
to create an instance.
Arguments
options
excludeZBS
(boolean): defaultstrue
.token
(boolean|string|function): defaultstrue
.separator
(boolean|string|function): defaultsfalse
.leaveBehind
(string|Array): optionally provides pseudo-lookbehind support.- all other through2 options.
regexp
(RegExp): The regular expression using which the stream will be tokenized.