Search results

8 packages found

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

published version 2.9.0, 2 months ago115 dependents licensed under $MIT
724,947

Javascript BPE Encoder Decoder for GPT-2 / GPT-3. The "gpt-3-encoder" module provides functions for encoding and decoding text using the Byte Pair Encoding (BPE) algorithm. It can be used to process text data for input into machine learning models, or to

published version 0.1.0, 2 years ago2 dependents licensed under $MIT
2,787

Javascript BPE Encoder Decoder for GPT-2 / GPT-3. The "gpt-3-encoder" module provides functions for encoding and decoding text using the Byte Pair Encoding (BPE) algorithm. It can be used to process text data for input into machine learning models, or to

published version 1.4.0-rc5, 2 years ago0 dependents licensed under $MIT
1,587

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 / Claude Instant / Claude 2

published version 0.3.2, 2 years ago1 dependents licensed under $MIT
48

Build your own vocabulary from application-specific corpus using Byte pair encoding (BPE) algorithm.

published version 2.2.0, a year ago0 dependents licensed under $BSD-2-Clause
44

Lightweight trimmed down encoder/decoder/tokenizer/token counter for gpt3 that is compatible with both node and browser environments

published version 0.0.1, 2 years ago1 dependents licensed under $MIT
34

A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package

published version 1.0.2, 2 years ago1 dependents licensed under $Apache-2.0
26

Fast tokenizer.

published version 0.1.0, 6 months ago0 dependents licensed under $MIT
9