@convai/experience-embed

A library for embedding Pixel Streaming functionality in web applications, supporting both React and vanilla JavaScript/TypeScript implementations.

Installation

npm install @convai/experience-embed

Step-by-Step Usage Guide

React Implementation

Step 1: Import the Component

First, import the PixelStreamComponent in your React application:

import { PixelStreamComponent } from "@convai/experience-embed";

Step 2: Set Up the Component Reference

Create a ref to access the component's methods:

import { useRef } from "react";
import { PixelStreamComponentHandles } from "@convai/experience-embed";

// Inside your component:
const pixelStreamRef = useRef<PixelStreamComponentHandles>(null);

Step 3: Available Methods

The component provides several methods through the ref:

enableCamera(): Enables the camera for the experience
disableCamera(): Disables the camera
enableCharacterAudio(): Enables character audio
disableCharacterAudio(): Disables character audio
initializeExperience(): Initializes the experience

Example usage:

const handleEnableCamera = async () => {
  await pixelStreamRef.current?.enableCamera();
};

Step 4: Create the Component

Add the PixelStreamComponent to your JSX with the required props:

<PixelStreamComponent
  ref={pixelStreamRef}
  expId="your-experiment-id"
  InitialScreen={<div>Loading your experience...</div>}
/>

Step 5: Customize the Initial Screen (Optional)

You can create a custom loading screen:

const CustomLoadingScreen = () => (
  <div
    style={{
      width: "100%",
      height: "100%",
      display: "flex",
      alignItems: "center",
      justifyContent: "center",
      background: "#000",
      color: "#fff",
    }}
  >
    <h2>Loading your experience...</h2>
  </div>
);

// Use it in your component:
<PixelStreamComponent
  ref={pixelStreamRef}
  expId="your-experiment-id"
  InitialScreen={<CustomLoadingScreen />}
/>;

Core (Vanilla TypeScript) Implementation

Step 1: Import the Client

Import the PixelStreamClient in your TypeScript file:

import { PixelStreamClient } from "@convai/experience-embed";

Step 2: Create Container Element

Create a container element in your HTML:

<div id="pixel-stream-container" style="width: 100%; height: 600px;"></div>

Step 3: Initialize the Client

Create a new instance of PixelStreamClient:

const container = document.getElementById("pixel-stream-container");
if (container) {
  const pixelStream = new PixelStreamClient({
    container: container,
    expId: "your-experiment-id",
    InitialScreen: document.createElement("div"), // Optional custom loading screen
  });
}

Step 4: Available Methods

The client provides the following methods:

// Enable camera
pixelStream.enableCamera().then((success) => {
  console.log("Camera enabled:", success);
});

// Disable camera
pixelStream.disableCamera().then((success) => {
  console.log("Camera disabled:", success);
});

// Enable character audio
pixelStream.enableCharacterAudio();

// Disable character audio
pixelStream.disableCharacterAudio();

// Initialize experience
pixelStream.initializeExperience();

CDN Implementation

You can use the @convai/experience-embed package directly in your HTML without any build tools by loading it from a CDN.

Step 1: Include the UMD Script

Include the library via CDN in your HTML:

<script src="https://unpkg.com/@convai/experience-embed/dist/core/convai-embed.umd.js"></script>

Step 2: Create Container Element

Create a container element in your HTML:

<div id="pixel-stream-container" style="width: 100%; height: 600px;"></div>

Step 3: Initialize the Client

Create a new instance of PixelStreamClient:

<script>
  const container = document.getElementById("pixel-stream-container");

  const customInitialScreen = document.createElement("div");
  customInitialScreen.style.cssText = `
    width: 100%;
    height: 100%;
    position: absolute;
    top: 0;
    left: 0;
    z-index: 10;
    display: flex;
    align-items: center;
    justify-content: center;
    background: rgba(0, 0, 0, 0.8);
    color: white;
    font-family: Arial, sans-serif;
  `;
  customInitialScreen.innerHTML = `
    <div style="text-align: center;">
      <h2>Welcome to the Experience</h2>
      <p>Click anywhere to start</p>
    </div>
  `;

  const pixelStream = new PixelStreamCore.PixelStreamClient({
    expId: "your-experiment-id",
    container: container,
    InitialScreen: customInitialScreen,
  });

  // Make it available globally for debugging if needed
  window.pixelStream = pixelStream;
</script>

Step 4: Available Methods

Once initialized, the following methods are available on the pixelStream object:

initializeExperience() Starts the pixel streaming session:

pixelStream.initializeExperience();

enableCamera() Requests access to the user’s camera and sends the video stream to the experience:

pixelStream.enableCamera().then((success) => {
  console.log("Camera enabled:", success);
});

disableCamera() Stops the camera stream:

pixelStream.disableCamera().then((success) => {
  console.log("Camera disabled:", success);
});

enableCharacterAudio() Enables audio from the character (from Unreal/Convai):

pixelStream.enableCharacterAudio();

disableCharacterAudio() Enables audio from the character (from Unreal/Convai):

pixelStream.disableCharacterAudio();

Step 5: Add Interactive Buttons (Optional)

You can add buttons in your HTML to trigger these methods:

<div class="controls">
  <button onclick="pixelStream.initializeExperience()">Start Experience</button>
  <button onclick="pixelStream.enableCamera()">Enable Camera</button>
  <button onclick="pixelStream.disableCamera()">Disable Camera</button>
  <button onclick="pixelStream.enableCharacterAudio()">Enable Audio</button>
  <button onclick="pixelStream.disableCharacterAudio()">Disable Audio</button>
</div>

API Reference

PixelStreamComponent Props

Prop	Type	Required	Description
expId	string	Yes	Your experiment ID
InitialScreen	React.ReactNode	No	Custom loading screen component
serviceUrls	object	No	Custom service URLs configuration
serviceUrls.sessionFetch	string	No	Custom URL for fetching the session
serviceUrls.pixelStreamBase	string	No	Custom base URL for the pixel stream

PixelStreamClient Options

Option	Type	Required	Description
container	HTMLElement	Yes	The container element for the stream
expId	string	Yes	Your experiment ID
InitialScreen	HTMLElement	No	Custom loading screen element
serviceUrls	object	No	Custom service URLs configuration
serviceUrls.sessionFetch	string	No	Custom URL for fetching the session
serviceUrls.pixelStreamBase	string	No	Custom base URL for the pixel stream