❤
npm
Sign UpSign In

Search results

40 packages found

Sort by: Default
  • Default
  • Most downloaded this week
  • Most downloaded this month
  • Most dependents
  • Recently published

@lenml/llama2-tokenizer

Our library `@lenml/llama2-tokenizer` has been deprecated. We are excited to introduce our new library `@lenml/tokenizers` as its replacement, offering a broader set of features and an enhanced experience.

luke_zhang
published version 1.1.4, a year ago1 dependents licensed under $MIT
6,056

@lenml/llama2-tokenizer-vocab-llama2

luke_zhang
published version 1.1.1, a year ago1 dependents licensed under $MIT
6,131

@lenml/tokenizers

a lightweight no-dependency fork of transformers.js (only tokenizers)

luke_zhang
published version 3.4.0, a month ago32 dependents licensed under $Apache-2.0
6,430

@lenml/tokenizer-llama3

llama3 tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago1 dependents licensed under $Apache-2.0
5,117

@axflow/models

Zero-dependency, modular SDK for building robust natural language applications

benjreinhart
published version 0.0.25, a year ago0 dependents licensed under $MIT
3,706

@lenml/tokenizer-gemini

gemini tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
886

@lenml/tokenizer-gpt4

gpt4 tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
574

@lenml/tokenizer-claude

claude tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
551

@lenml/tokenizer-gpt4o

gpt4o tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
469

@lenml/tokenizer-gemma2

gemma2 tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
425

@lenml/tokenizer-text_embedding_ada002

text_embedding_ada002 tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
227

@lenml/tokenizer-llama2

llama2 tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
218

@lenml/tokenizer-gemma

gemma tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago1 dependents licensed under $Apache-2.0
230

@lenml/tokenizer-text_davinci002

text_davinci002 tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
196

@lenml/tokenizer-baichuan2

baichuan2 tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
198

@lenml/tokenizer-qwen2_5

qwen2_5 tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
201

@lenml/tokenizer-internlm2

internlm2 tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
184

@lenml/tokenizer-chatglm3

chatglm3 tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
187

@lenml/tokenizer-gpt35turbo

gpt35turbo tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
172

@lenml/tokenizer-gpt35turbo16k

gpt35turbo16k tokenizer for NodeJS/Browser

luke_zhang
published version 3.4.2, a month ago0 dependents licensed under $Apache-2.0
176