Search results

codepoint-iterator

Fast uint8array to utf-8 codepoint iterator for streams and array buffers by @okikio & @jonathantneal

• utf8
• uint8array
• streams
• iterable-streams
• codepoints
• tokenizer
• iterable-callback
• codepoint-array

okikio

published 1.1.1 • 7 months agopublished 1.1.1 7 months ago

@novel-segment/table-core-abstract

• NLP
• PanGuSegment
• PoS tagging
• analyzer
• async
• chinese
• chinese segmentation
• data
• dict
• dictionary
• file
• hanzi
• jieba
• load
• View more

bluelovers

published 1.0.23 • 5 months agopublished 1.0.23 5 months ago

leac

Lexer / tokenizer

• lexer
• tokenizer
• lex
• token

killy.mxi

published 0.6.0 • 2 years agopublished 0.6.0 2 years ago

@santerijps/tokenizer

A simple generic string tokenizer

• tokenizer
• token
• string

santerijps

published 1.0.3 • 5 months agopublished 1.0.3 5 months ago

@sglkc/kuromoji

Forked version of kuromoji with better compatibility for browsers

• japanese
• morphological analyzer
• nlp
• pos
• pos tagger
• tokenizer

sglkc

published 1.1.0 • a year agopublished 1.1.0 a year ago

react-bootstrap-typeahead-with-restart

React typeahead with Bootstrap styling

• auto complete
• auto suggest
• auto-complete
• auto-suggest
• autocomplete
• autosuggest
• bootstrap
• bootstrap tokenizer
• bootstrap typeahead
• bootstrap-tokenizer
• bootstrap-typeahead
• react
• react autocomplete
• react autosuggest
• View more

nthansen

published 1.0.0 • a year agopublished 1.0.0 a year ago

@aiktb/kuromoji

Forked version of kuromoji with better compatibility for browsers

• japanese
• morphological analyzer
• nlp
• pos
• pos tagger
• tokenizer
• browser

~aiktb

published 1.0.2 • 6 months agopublished 1.0.2 6 months ago

@lordluceus/mistral-tokenizer

TS tokenizer for Mistral-based LLMs

• mistral
• llm
• tokenizer

lordluceus

published 1.2.2 • 5 months agopublished 1.2.2 5 months ago

words-n-numbers

Tokenizing strings of text. Extracting arrays of words and optionally number, emojis, tags, usernames and email addresses from strings. For Node.js and the browser. When you need more than just [a-z] regular expressions.

• nlp
• regex
• words
• tokenizer
• tokenization
• offline-first
• numbers
• emojis
• extract

eklem

published 9.1.2 • 10 months agopublished 9.1.2 10 months ago

@language-kit/lexer

Transforms a string into a list of tokens

• lexer
• tokenizer
• language

henryque

published 1.1.6 • 6 months agopublished 1.1.6 6 months ago

@novel-segment/table-synonym-pangu

• NLP
• PanGuSegment
• PoS tagging
• analyzer
• async
• chinese
• chinese segmentation
• data
• dict
• dictionary
• file
• hanzi
• jieba
• load
• View more

bluelovers

published 1.0.22 • 5 months agopublished 1.0.22 5 months ago

@novel-segment/table-line

原版 node-segment 的格式

• NLP
• PanGuSegment
• PoS tagging
• analyzer
• async
• chinese
• chinese segmentation
• data
• dict
• dictionary
• file
• hanzi
• jieba
• load
• View more

bluelovers

published 1.0.22 • 5 months agopublished 1.0.22 5 months ago

@novel-segment/table-blacklist

• NLP
• PanGuSegment
• PoS tagging
• analyzer
• async
• chinese
• chinese segmentation
• data
• dict
• dictionary
• file
• hanzi
• jieba
• load
• View more

bluelovers

published 1.0.19 • 5 months agopublished 1.0.19 5 months ago

vscode-textmate-languageservice

Textmate token-based language service for Visual Studio Code.

• vscode
• vscode-extension
• textmate
• grammar
• language-features
• language-service
• language
• lsp
• parse
• syntax
• tokenization
• tokenizer

zm-sdet

published 3.0.1 • 7 months agopublished 3.0.1 7 months ago

@novel-segment/table-stopword

• NLP
• PanGuSegment
• PoS tagging
• analyzer
• async
• chinese
• chinese segmentation
• data
• dict
• dictionary
• file
• hanzi
• jieba
• load
• View more

bluelovers

published 1.0.19 • 5 months agopublished 1.0.19 5 months ago

@novel-segment/table-synonym

• NLP
• PanGuSegment
• PoS tagging
• analyzer
• async
• chinese
• chinese segmentation
• data
• dict
• dictionary
• file
• hanzi
• jieba
• load
• View more

bluelovers

published 1.0.19 • 5 months agopublished 1.0.19 5 months ago

@lenml/tokenizer-llama2

llama2 tokenizer for NodeJS/Browser

• llama
• llama2
• llama3
• chatgpt
• mistral
• tokenizer

luke_zhang

published 1.1.2 • 11 days agopublished 1.1.2 11 days ago

omni-tokenizer

Node based tokenizers for open source models hosted on HuggingFace.

• tokenizer
• mistral
• mixtral
• gemma

jakecyr

published 2.0.1 • 6 months agopublished 2.0.1 6 months ago

@lenml/tokenizer-gpt4

gpt4 tokenizer for NodeJS/Browser

• llama
• llama2
• llama3
• chatgpt
• mistral
• tokenizer
• gpt4

luke_zhang

published 1.1.2 • 11 days agopublished 1.1.2 11 days ago

@lenml/tokenizer-command_r_plus

command_r_plus tokenizer for NodeJS/Browser

• llama
• llama2
• llama3
• chatgpt
• mistral
• tokenizer
• command_r_plus

luke_zhang

published 1.1.2 • 11 days agopublished 1.1.2 11 days ago