Search results

433 packages found

Tiny JavaScript tokenizer.

published 9.0.0 2 months ago
M
Q
P
M
Q
P

A tool set for CSS: fast detailed parser (CSS → AST), walker (AST traversal), generator (AST → CSS) and lexer (validation and matching) based on specs and browser implementations

published 2.3.1 a year ago
M
Q
P

small commonmark compliant markdown parser with positional info and concrete tokens

published 4.0.0 9 months ago
M
Q
P

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

published 2.1.2 6 months ago
M
Q
P

Tokenize CSS

published 2.2.4 15 days ago
M
Q
P

Chevrotain is a high performance fault tolerant javascript parsing DSL for building recursive decent parsers

published 11.0.3 7 months ago
M
Q
P
M
Q
P

stream-json is the micro-library of Node.js stream components for creating custom JSON processing pipelines with a minimal memory footprint. It can parse JSON files far exceeding available memory streaming individual primitives using a SAX-inspired API. I

published 1.8.0 10 months ago
M
Q
P

JS tokenizer for LLaMA-based LLMs

published 1.2.1 4 days ago
M
Q
P

Isomorphic utilities for GPT-3 tokenization and prompt building.

published 1.2.0 a year ago
M
Q
P

Simple algorithm to tokenize Chinese texts into words using CC-CEDICT.

published 1.0.1 8 months ago
M
Q
P

General natural language (tokenizing, stemming (English, Russian, Spanish), part-of-speech tagging, sentiment analysis, classification, inflection, phonetics, tfidf, WordNet, jaro-winkler, Levenshtein distance, Dice's Coefficient) facilities for node.

published 6.12.0 16 days ago
M
Q
P

Parse PHP code from JS and returns its AST

published 3.1.5 10 months ago
M
Q
P

This project is about to implement a simple javascript parser and visualize the ast

published 1.0.9 2 years ago
M
Q
P

A promise based streaming tokenizer

published 7.0.0 2 years ago
M
Q
P

Dot notation tokenizer

published 1.3.0 2 years ago
M
Q
P

A lua parser

published 1.0.9 4 months ago
M
Q
P

JavaScript implementation of Japanese morphological analyzer

published 0.1.4 9 months ago
M
Q
P

A CLI tool to concatenate all text files in your CWD with headers for GPT prompt engineering.

published 1.0.2 2 months ago
M
Q
P