node package manager
Orgs are free. Discover, share, and reuse code in your team. Create a free org »

tokenize

tokenize NPM version Downloads Build Status Coveralls Status

JavaScript tokenizer

Example:

var tokenize = require('tokenize');
tokenize('dad cat', [{type: "variable", regex: /[a-z]+/},{type:"whitespace", regex: /\s/}], function(err,results){
})