Search results

gpt-tokenizer

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

• BPE
• encoder
• decoder
• tokenizer
• GPT
• GPT-2
• GPT-3
• GPT-3.5
• GPT-4
• GPT-4o
• NLP
• Natural Language Processing
• Text Generation
• OpenAI
• View more

niieani

published 2.3.0 • 2 days agopublished 2.3.0 2 days ago

bpe-tokenizer

Build your own vocabulary from application-specific corpus using Byte pair encoding (BPE) algorithm.

• BPE
• byte pair encoding
• algorithm
• sqlite
• browser
• cross-platform
• isomorphic
• natural language processing
• NLP
• GPT
• tokenizer
• typescript

beenotung

published 2.2.0 • 6 months agopublished 2.2.0 6 months ago

@jill64/cf-tiktoken

⏳ js-tiktoken on Cloudflare Pages

• bpe
• cloudflare
• gpt
• llm
• pages
• tiktoken

jill64

published 1.0.5 • a month agopublished 1.0.5 a month ago

gpt-turbo-plugin-stats

GPT Turbo plugin that calculates your conversation size and cost

• openai
• chatgpt
• chat
• gpt
• gpt3
• gpt-3
• gpt3.5
• gpt-3.5
• gpt4
• gpt-4
• completion
• chatcompletion
• conversation
• conversation ai
• View more

maxijonson

published 5.0.1 • a year agopublished 5.0.1 a year ago

@syonfox/gpt-3-encoder

Javascript BPE Encoder Decoder for GPT-2 / GPT-3. The "gpt-3-encoder" module provides functions for encoding and decoding text using the Byte Pair Encoding (BPE) algorithm. It can be used to process text data for input into machine learning models, or to

• JavaScript
• BPE
• Encoder
• Decoder
• GPT-2
• GPT-3
• Natural Language Processing (NLP)
• Text Generation
• Machine Learning

syonfox

published 1.4.0-rc5 • 2 years agopublished 1.4.0-rc5 2 years ago

gptoken

Javascript BPE Encoder Decoder for GPT-2 / GPT-3. The "gpt-3-encoder" module provides functions for encoding and decoding text using the Byte Pair Encoding (BPE) algorithm. It can be used to process text data for input into machine learning models, or to

• JavaScript
• BPE
• Encoder
• Decoder
• GPT-2
• GPT-3
• Natural Language Processing (NLP)
• Text Generation
• Machine Learning

syonfox

published 0.1.0 • 2 years agopublished 0.1.0 2 years ago

llm-tokenizer

A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package

• OpenAI
• GPT-3
• GPT-4
• GPT-3.5
• ChatGPT
• LLMs
• GPT
• tokenizer
• BPE
• encoder
• decoder
• Machine Learning

thrawnk

published 1.0.2 • a year agopublished 1.0.2 a year ago

@meistrari/gpt-tokenizer

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 / Claude Instant / Claude 2

• BPE
• encoder
• decoder
• tokenizer
• GPT
• GPT-2
• GPT-3
• GPT-3.5
• GPT-4
• NLP
• Natural Language Processing
• Text Generation
• OpenAI
• Machine Learning

henrykunh

published 0.3.2 • a year agopublished 0.3.2 a year ago

n8n-nodes-gpt-tokenizer

n8n node for working with BPE Tokens with OpenAI's GPT models in mind.

• n8n-community-node-package
• n8n
• nodemation
• n8n-node
• gpt
• bpe
• tokenizer

geckse

published 0.1.1 • a year agopublished 0.1.1 a year ago

gpt-tok

Lightweight trimmed down encoder/decoder/tokenizer/token counter for gpt3 that is compatible with both node and browser environments

• BPE
• encoder
• decoder
• tokenizer
• GPT
• GPT-2
• GPT-3
• GPT-3.5
• GPT-4
• NLP
• Natural Language Processing
• Text Generation
• OpenAI
• Machine Learning

iskandarreza

published 0.0.1 • a year agopublished 0.0.1 a year ago