Tokenizer
The task is simple: take a String as input, and return an Array of Strings, one per word. Words may repeat.
The logic is not simple.
This tokenizer implements the Unicode 8.0 Text Segmentation Algorithm. That makes it valid for English and European languages; but it's terrible for Chinese, Japanese, and other languages that do not have any characters between words.
Usage
First, add it to your project: npm install --save overview-js-tokenizer
Next, use it. It turns a String input into an Array of String tokens:
var tokenizer = require('overview-js-tokenizer');
var inputString = "The cat's meowed 1,000 times! Really!";
console.log(tokenizer.tokenize(inputString));
// output: [ "The", "cat's", "meowed", "1,000", "times", "Really" ]Constraints
The input must be a valid Unicode. In particular, a string like \uDC00\uD800
is invalid (it's a low surrogate followed by a high surrogate); that will cause
undefined behavior. (This constraint is true of most programs that deal with
Strings.)
Developing
Download, npm install, and run npm run prepublish to generate data files.
Run mocha -w in the background as you implement features. Write tests in
test and code in lib.
Memory concerns
If you're filtering for only a few words out of a huge input String, be aware that in v8, a reference to even a one-character substring of the huge String will keep the huge String in memory. See https://code.google.com/p/v8/issues/detail?id=2869 for more depth and a workaround.
TODO
Pull requests are welcome! In particular, this library could use:
- More unit tests: we really don't test much here
- Options: especially those suggested at http://www.unicode.org/reports/tr29
- Optimization: we have zillions of function calls and allocations
LICENSE
AGPL-3.0. This project is (c) Overview Services Inc. Please contact us should you desire a more permissive license.