search-text-tokenizer

A tokenizer for Google-like search queries

search-text-tokenizer

A search query tokeniser inspired by Google.

  • Split a space-delimitered query string into an array of terms
  • Treat quoted terms as phrases
  • Support tagged terms (tag:term)
var tokenizer = require( 'search-text-tokenizer' );
 
console.log( tokenizer( 'red bull' ) );
// [ { term: 'red' }, { term: 'bull' } ] 
 
var result = console.log( tokenizer( '"red bull" "gives you wings"' ) );
// [ { term: 'red bull', phrase: true }, { term: 'gives you wings', phrase: true } ] 
 
result = console.log( tokenizer( 'author:tolkien' ) );
// [ { term: 'tolkien', tag: 'author' } ] 
$ npm install search-text-tokenizer

To run the test suite enter these commands in the project directory.

$ npm install
$ npm test

MIT © Tatsuya Oiwa, Dannii Willis