Search results

11 packages found

Multilingual tokenizer that automatically tags each token with its type

published version 5.3.0, 3 years ago18 dependents licensed under $MIT
56,845

Port from Apache Lucene

published version 5.3.3, 9 years ago0 dependents licensed under $ISC
121

A text tokenizing library that handles strings, by tokenizing them into arrays, depending on intended format, like sentences, sub sentences, paragraphs, words...

published version 1.2.0, 7 months ago0 dependents licensed under $MIT
75

NFA Tokenizer

published version 0.1.8, 7 years ago1 dependents licensed under $MIT
69

A super simple basic parser / tokenizer for easier processing of various configuration files

published version 1.1.0, 6 months ago0 dependents licensed under $MIT
46

RWKV / gpt-NeoX / Pythia, 0-dep tokenizer library, for nodejs

published version 1.0.5, 2 years ago1 dependents licensed under $MIT
44

Mathematical expression solver / Reverse Polish Notation calculator for NodeJS

published version 1.0.4, 4 years ago0 dependents licensed under $MIT
37

Lexer generator from RegExp spec

published version 0.1.10, 4 years ago0 dependents licensed under $MIT
39

Lexical tokenizer with grammar classes ready to go.

published version 1.0.10, 3 years ago0 dependents licensed under $ISC
32

module for nksoft tokens

published version 1.1.2, 4 years ago0 dependents licensed under $ISC
31

A simple lexer for javascript

published version 1.0.2, 4 years ago1 dependents licensed under $GPL-3.0-or-later
21