A JS implementation of a tokenizer to split up source code based on tokens
npm install code-tokenizer --save
How to use
Tokenizer = require 'code-tokenizer' tokenizer = new Tokenizer() tokens = tokenize 'print "Josh"' # ['print'] tokens = tokenize 'path/to/file.html', 'file' # ['<div>', 'id', 'class', '</div>']
This is based on the Github Linguist project. This node.js implementation strips any data strings or comments from the data and returns an array of language symbols.
You can use this to do sanity checking of source code, parsing, detection, etc. This is a general purpose library and I'm always looking to hear how you're using it. Happy Coding!
All tests are in the test/tokenizer.coffee file. It tests a variety of cases from different language types. To run the test for yourself, just do