Tokenizer can be installed from NPM by running the lines below in your working directory containing package.json file for use in NodeJS, browserify or webpack.
npm install libcore-tokenizer --save
The following lines defines tokens and regular expressions and uses the tokenizer instance to tokenize the string.
var Tokenizer =tokenizer =subject = "12345abc67890";// declare tokenstokenizer;// tokenize (indexFrom, stringSubject)tokenizer;// result:// ["number", "12345", 5]tokenizer;// result: ["sequence", "abc", 8]tokenizer;// result: ["number", "67890", 13]tokenizer;// result: ["$", "", 14] -- end token, next call on next index would result in nulltokenizer;// result: null
You can also export state matrix of the state machine to JSON string by running the lines below.
var Tokenizer =tokenizer = ;// declare tokenstokenizer;console;// result: JSON string containing state matrix object
You can import state matrix from JSON string or from Native Object.
var Tokenizer =exportedJSON =tokenizer = ;tokenizer;// torkenizer is now ready for tokenizing string
Supported Regex Operators for Regular Expression patterns
- Kleen Star
- Character Class
, and Negated Character Class
[^]operators. Character Ranges like
0-9a-zis also supported.
()operator (Capture is not yet implemented)