tokenator: a Node.js tokenization library
tokenator is a simple to use library that rework a string splitting it to meaningful tokens.
Table of Contents
This library transforms a series of characters to tokens identified by input strings.
e.g. "The itsy bitsy spider crawled up the water spout. | Down came the rain, and washed the spider out. | Out came the sun, and dried up all the rain, | and the itsy bitsy spider went up the spout again" Splitting strings: spider, rain Result tokens: > "The itsy bitsy " > "spider" > " crawled up the water spout. | Down came the " > "rain" > ", and washed the " > "spider" > " out. | Out came the sun, and dried up all the " > "rain" > ", | and the itsy bitsy " > "spider" > " went up the spout again"
npm install tokenator
The signatures of the available methods are:
function tokenator(source: string, ...separators: string): string function tokenatorStream(...separators: string): (separatorsEncoding: string = "utf-8") => stream.Transform
Code examples could be consulted within the test folder:
These functions work over multiple separators, like in the example above. The tokenatorStream() function creates a Transform stream. NodeJS Stream usage specification is available on the NodeJS official documentation.