Search results
460 packages found
Input field with tagging/token/chip capabilities written in raw JavaScript
Turns a string into tokens.
Tokenize JavaScript
Rich text and markdown tokenization made easy.
A text tokenizing library that handles strings, by tokenizing them into arrays, depending on intended format, like sentences, sub sentences, paragraphs, words...
- NLP tokenizer
- Tokenizer
- Sentence tokenizer
- text tokenizer
- node.js tokenizer
- flexible text tokenizer
- general text tokenizer
- paragraph tokenizer
- sub sentence tokenizer
- stable tokenizer
- easy text tokenizer
- new text tokenizer
A CLI command for converting JSX files into standard JS.
This project is about to implement a simple javascript parser and visualize the ast
Enhances comprehension for MindScript-powered models by tokenizing text.
a lightweight no-dependency fork of transformers.js (only tokenizers)
- llama
- llama2
- llama3
- chatgpt
- mistral
- tokenizer
- transformers
- transformers.js
- gpt4
- gpt4o
- gpt3.5
- vicuna
- chatglm
- baichuan
Token generator/validator without external dependencies
- random
- token
- generator
- javascript
- keygen
- string
- tokenizer
- number token
- number generator
- random string
- unique token
- unique string token
- validator
- token validator
- View more
A super fast html-parser stream that outputs tag, text and closing nodes.
Takes in a text/file/stream and generates random sentences that sound like they could have been in the text
Create a string representation of an object or primative
React-based typeahead and typeahead-tokenizer
Toolkit to make streamable scanners and parsers.
Tokenize a string.
Split text into sentence strings or word arrays with Sentence Boundary Detection (SBD). Based on http://github.com/Tessmore/sbd by Fabiën Tesselaar.
TypeScript definition for strtok3 token