PrompTest

The Prompt testing library for LLM that allows comparing patterns of prompts.

Installation

$ npm i --save-dev promptest

Usage

// prompt.test.js
const { Configuration, OpenAIApi } = require("openai")
const { variable, promptest } = require('promptest')

const configuration = new Configuration({
  apiKey: process.env.OPENAI_API_KEY,
})
const openai = new OpenAIApi(configuration)

variable("name", ["John", "Jane"])
variable("age", ["20", "30"])

promptest(
  "Your name is {{name}}, and you are {{age}} years old.",
  "Hello!",
  async (subject, input) => {
    const chatCompletion = await openai.createChatCompletion({
      model: "gpt-3.5-turbo",
      messages: [
        { role: "system", content: subject },
        { role: "user", content: input },
      ],
    })
    return chatCompletion.data.choices[0].message.content
  }
).then(console.log).catch(console.error)

npx node prompt.test.js

Variable

variable("name", ["John", "Jane"])

variable is called with variable name and patterns.

Subject

The subject prompt template is defined with {{name}}.

promptest(
  "Your name is {{name}}, and you are {{age}} years old.",
  // ...
)

Input

Input is ordinary user prompt for testing.

promptest(
  // ..
  "Hello!"
  // ..
)

Callback

The callback is test function. The arguments are subject prompt and input prompt.

promptest(
  // ...
  async (subject: string, input: string) => {
    const chatCompletion = await openai.createChatCompletion({
      model: "gpt-3.5-turbo",
      messages: [
        { role: "system", content: subject },
        { role: "user", content: input },
      ],
    })
    return chatCompletion.data.choices[0].message.content
  }
)

Result

The result is JSON.

{
  "input": "Hello!",
  "results": [
    {
      "subject": "Your name is John, and you are 20 years old",
      "output": "Hello! How can I assist you today?"
    },
    {
      "subject": "Your name is Jane, and you are 20 years old",
      "output": "Hello! How can I assist you today?"
    },
    {
      "subject": "Your name is John, and you are 30 years old",
      "output": "Hello! How can I help you today?"
    },
    {
      "subject": "Your name is Jane, and you are 30 years old",
      "output": "Hello! How can I assist you today?"
    },
  ]
}

Development

After checking out the repo, run npm i to install dependencies. Then, run npm run test to run the tests.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/moekidev/promptest. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the code of conduct.

License

The gem is available as open source under the terms of the MIT License.

Code of Conduct

Everyone interacting in the Promptest project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the code of conduct.