npm

95 packages found

M
Q
P

Description

A regex that tokenizes JavaScript.

Keywords

Publisher

published 5.0.05 months ago
M
Q
P

Description

Lexical analyzer built in Javascript

Keywords

Publisher

published 1.0.4a year ago
M
Q
P

Description

A simple iterative lexer written in TypeScript

Keywords

Publisher

published 0.7.46 months ago
M
Q
P

Description

Tokenizer for tokenizing sentences, for BERT or other NLP preprocessing.

Keywords

Publisher

published 1.0.26 months ago
M
Q
P

Description

String ngram splitter.

Keywords

Publisher

published 0.2.13 months ago
M
Q
P

Description

NLP Functions for amplifying negations, managing elisions, creating ngrams, stems, phonetic codes to tokens and more.

Keywords

Publisher

published 2.0.57 months ago
M
Q
P

Description

Multilingual tokenizer that automatically tags each token with its type

Keywords

Publisher

published 5.2.17 months ago
M
Q
P

Description

transform stream to tokenize html

Keywords

Publisher

published 2.0.03 years ago
M
Q
P

Description

lexing json

Keywords

Publisher

published 1.1.13 years ago
M
Q
P

Description

Simple utility to get unique / repeated words from strings or phrases

Keywords

Publisher

published 2.0.010 months ago
M
Q
P

Description

personal tokenize util

Keywords

Publisher

published 1.0.19 months ago
M
Q
P

Description

Lemmize and tokenize string which contains Chinese and English words

Keywords

Publisher

published 1.0.07 months ago
M
Q
P

Description

Tokenize Excel formulas

Keywords

Publisher

published 2.3.110 months ago
M
Q
P

Description

Transform stream that tokenizes CSS

Keywords

Publisher

published 1.0.15 years ago
M
Q
P

Description

Uses esprima to extract line and block comments from a string of JavaScript. Also optionally parses code context (the next line of code after a comment).

Keywords

Publisher

published 1.1.0a year ago
M
Q
P

Description

Simple synchronous string tokenizer using Regex

Keywords

Publisher

published 2.0.02 years ago
M
Q
P

Description

tokenize c/c++ source code

Keywords

Publisher

published 1.0.03 years ago
M
Q
P
M
Q
P

Description

Tokenizes utterances that contain a mix of English and Chinese words.

Keywords

Publisher

published 1.1.0a year ago
M
Q
P

Description

A streaming JSON tokenizer

Keywords

Publisher

published 1.1.02 years ago