textml – Text Analysis with JavaScript
Require the module
// load module
const textml = require('textml')
API
tokenize()
Tokenize a text input (a type string
input)
const text = 'Hello, world! How are you today?'
/* defaults */
const options = {
delimiter: ' ',
pattern: '[.,:?!/]',
preprocess: true,
removePunctuation: true,
removeDigits: true,
lowercase: true
}
const tokens = textml.tokenize(text, options)
// ['hello', 'world', 'how', 'are', 'you', 'today']