A tokenizer for Google-like search text


Tokenize a search keyword text in the same manner as Google does.

  • Split space-delimitered keyword string into an array of keywords
  • Treat dobule-quoted keywords as phrase
var tokenizer = require('search-text-tokenizer');
console.log(tokenizer('red bull'));  // ['red', 'bull'] 
console.log(tokenizer('"red bull" "gives you wings"'));  // ['"red bull"', '"gives you wings"'] 
$ npm install search-text-tokenizer

To run the test suite first invoke the following command in the project directory.

$ npm install

then run the tests:

$ npm test

MIT © Tatsuya Oiwa