Simple Text Tokenizer

Tokenize text to paragraphs, sentences, subsentences, and words.

Installation

Use npm:

npm install simple-text-tokenizer

How to

Import functions

 
import * as tokenizer from 'text-tokenizer'

Tokenize text to paragraphs

getParagraphTokens('this is the text of paragraph1\n\n\n this is the text of paragraph1\n');

Tokenize paragraph to sentences:

getSentenceTokens('this is the text of sentence1. And this is sentence2!');

Tokenize sentence to subsentences

getSubSentenceTokens('this is the text of subsentence1, this is sentence2; and this is the 3rd one!');

Tokenize sentence to words

getSubSentenceTokens('this is the text of subsentence1, this is sentence2; and this is the 3rd one!');