browser-llm-engine

A browser-friendly library for running large language models (LLMs) directly in the browser using Wllama. This library provides a simple interface to load .gguf or .bin models (e.g., from Hugging Face) and generate text completions, including streaming token support.

---

Features

• Plug-and-Play: Easy to integrate into your web projects.
• Local or Remote Models: Load a URL from Hugging Face or pass local File objects.
• Token-by-Token Streaming: Handle partial results in real-time via onNewToken callback.
• Templates: Leverages Jinja to format chat-based prompts.
• Lightweight: Bundles a minimal set of dependencies.

---

Table of Contents

1. Installation
2. Usage
   • Quick Start
   • Streaming
   • Loading Local Files
3. Preset Models
4. API
   • createLlmEngine
   • loadModel
   • formatChat
   • createCompletion
   • exit
5. Local Development
6. License

---

Installation

npm install browser-llm-engine

Or with Yarn:

yarn add browser-llm-engine

---

Usage

Quick Start

import { createLlmEngine, CHAT_ROLE, PRESET_MODELS } from 'browser-llm-engine';

(async () => {
  // 1) Create an engine instance
  const llm = createLlmEngine({
    // Optional: provide custom WASM paths or config
    wasmPaths: {}
  });

  // 2) Load a preset model from the library
  const modelUrl = PRESET_MODELS["SmolLM2 (360M)"].url;
  await llm.loadModel(modelUrl, {
    progressCallback: (progress) => console.log(`Loading: ${progress}%`),
  });

  // 3) Generate a completion
  const result = await llm.createCompletion("Hello from the browser!");
  console.log("Full model response:", result);

  // 4) Clean up
  await llm.exit();
})();

That’s it! You have a working LLM in the browser.

---

Streaming

To get partial tokens as they are generated, supply an onNewToken callback:

const llm = createLlmEngine();
await llm.loadModel(PRESET_MODELS["SmolLM2 (360M)"].url);

let outputSoFar = "";
await llm.createCompletion("What's the weather today?", {
  nPredict: 128,
  sampling: { temp: 0.7, penalty_repeat: 1.1 },
  onNewToken: (token) => {
    outputSoFar += token;
    console.log("Streamed token:", token);
  }
});

console.log("Final streamed output:", outputSoFar);

---

Loading Local Files

If you want to load the model from your local machine:

<input type="file" id="modelFile" multiple />
<script type="module">
  import { createLlmEngine } from 'browser-llm-engine';

  const fileInput = document.getElementById("modelFile");
  const llm = createLlmEngine();

  fileInput.addEventListener("change", async () => {
    try {
      // fileInput.files is a FileList
      await llm.loadModel(fileInput.files);
      console.log("Model loaded locally!");
    } catch (error) {
      console.error("Failed to load local model:", error);
    }
  });
</script>

---

Preset Models

The library includes a models.json with references to a few hosted models. You can get them via:

import { PRESET_MODELS } from 'browser-llm-engine';

console.log("Available models:", PRESET_MODELS);

Feel free to add or remove entries if you fork this library.

---

API

createLlmEngine(config?)

Creates a new engine instance.

• Parameters:
  • config (Object) – Optional configuration, e.g. { wasmPaths: { ... } }.

loadModel(source, options?)

Loads the model from either a remote URL or local File objects.

• Parameters:
  • source (String | File[] | FileList) – The source of the model.
  • options (Object) – Additional load options:
    • progressCallback (function): (progress) => {} for tracking loading progress
    • useCache (Boolean): Cache the model for faster reloads
    • allowOffline (Boolean): If false, tries to fetch from network

formatChat(messages, useProvidedTemplate?)

Takes an array of messages (each with role and content) and formats them into a single prompt with Jinja.

createCompletion(prompt, options?)

Creates the text completion for a given prompt.

• Parameters:
  • prompt (String) – The text to generate from.
  • options (Object) – Fine-tuning generation:
    • nPredict (Number) – Maximum tokens to predict (default 512)
    • sampling (Object) – e.g. { temp: 0.7, penalty_repeat: 1.1 }
    • onNewToken (function) – A callback for streaming tokens

exit()

Cleans up resources used by Wllama.

• Example:

  await llm.exit();

---

Local Development

If you want to develop locally:

1. Clone the repo:

   git clone https://github.com/you/browser-llm-engine.git
   cd browser-llm-engine

2. Install dependencies:

   npm install

3. Build the library:

   npm run build

   This will create dist/ with both ESM and CJS bundles.
4. (Optional) Start a dev server (if you add a script in package.json):

   npm run dev

5. Open index.html (or any dev test page) in your browser to play around with the library.

---

License

This project is released under the MIT License. Feel free to fork, adapt, and contribute!

---

Happy coding and enjoy using your LLM in the browser!