tokenizer
Split text into tokens.
Install
$ npm install --save hx-tokenizer
Usage
const tokenizer = ;const tokens = tokenizer;// tokens == ["Lorem", "ipsum", ",", "something", "."]const text = tokenizer;// text == "Lorem ipsum, something."
API
tokenizer.tokenize(text)
tokenizer.join(tokens)
License
MIT © Hexacta