This npm package is designed to count the number of OpenAI tokens in a given text or messages array. It supports various OpenAI text and chat models, and it has been verified for 100% accuracy.
You can install the package using npm:
npm install openai-gpt-token-counter
For CommonJS:
const openaiTokenCounter = require('openai-gpt-token-counter');
For ES6 Imports:
import openaiTokenCounter from 'openai-gpt-token-counter';
To count the number of tokens in a text for a specific OpenAI text model (e.g. text-davinci-003), use the text
method:
const text = "This is a test sentence.";
const model = "text-davinci-003"; // Replace with your desired OpenAI model
const tokenCount = openaiTokenCounter.text(text, model);
console.log(`Token count: ${tokenCount}`);
To count the number of tokens in chat messages for a specific OpenAI chat model, use the chat
method:
const messages = [
{ role: "user", content: "Say this is a test!" },
// Add more messages if needed
];
const model = "gpt-4"; // Replace with your desired OpenAI chat model
const tokenCount = openaiTokenCounter.chat(messages, model);
console.log(`Token count: ${tokenCount}`);
For chat models, provide an array of messages, where each message is an object with the following structure:
const messages = [
{ role: "system", content: "System prompt to guide the AI" },
{ role: "user", content: "Message content from the user" },
{ role: "assistant", content: "AI response to the user's message" },
// Add more messages as needed
];
The role
property can be one of "user"
, "system"
, or "assistant"
. The content
property holds the actual text of the message.
This package supports all OpenAI chat/text models, but the official ones we tested on are:
- GPT3 (text-davinci-003, text-curie-001, text-babbage-001, text-ada-001)
- GPT3.5 Turbo:
"gpt-3.5-turbo"
- GPT3.5 16K:
"gpt-3.5-turbo-16k"
- GPT4:
"gpt-4"
- GPT4 32K:
"gpt-4-32k"
Use the base model system to calculate the token count for fine-tuned models. For example, if you have a fine-tuned model based on gpt-4
, you can use the gpt-4
model to calculate the token count. Please report on the Github repository if you find any issues with fine-tuned models.
This module has been tested and verified for 100% accuracy against the OpenAI API's token count. Don't take my word for it, run this example test code to see the accuracy:
import openaiTokenCounter from 'openai-gpt-token-counter';
import { Configuration, OpenAIApi } from "openai";
const configuration = new Configuration({
apiKey: process.env.OPENAI_API_KEY,
});
const openai = new OpenAIApi(configuration);
(async () => {
const model = "gpt-3.5-turbo";
const texts = [
"Hello world",
"This is a slightly longer sentence with more words.",
"And this is an even longer sentence that has an excessive number of words..."
];
for (let text of texts) {
console.log(`Testing text: "${text}"`);
const messages = [{ role: "user", content: text }];
const tokenCount = openaiTokenCounter.chat(messages, model);
console.log(`openai-gpt-token-counter Token count: ${tokenCount}`);
const chatCompletion = await openai.createChatCompletion({
model: model,
messages: messages,
});
console.log(`OpenAI API Token count: ${chatCompletion.data.usage.prompt_tokens}`);
console.log("\n");
}
})();
Please note that this package does not support embeddings. It is specifically designed for counting the number of tokens in text or chat messages for OpenAI models. Though this is on our roadmap, we do not have an ETA for when this feature will be added.
If you encounter any issues or have suggestions for improvements, please feel free to open an issue on the GitHub repository. Contributions through pull requests are also welcome!