node package manager

tokenize

tokenize NPM version Downloads Build Status Coveralls Status

JavaScript tokenizer

Example:

var tokenize = require('tokenize');
tokenize('dad cat', [{type: "variable", regex: /[a-z]+/},{type:"whitespace", regex: /\s/}], function(err,results){
})