textml

textml – Text Analysis with JavaScript

Require the module

// load module
const textml = require('textml')

API

tokenize()

Tokenize a text input (a type string input)

const text = 'Hello, world! How are you today?'

/* defaults */
const options = {
  delimiter: ' ',
  pattern: '[.,:?!/]',
  preprocess: true,
  removePunctuation: true,
  removeDigits: true,
  lowercase: true
}

const tokens = textml.tokenize(text, options)
// ['hello', 'world', 'how', 'are', 'you', 'today']