@cloudflare/realtimekit-virtual-background

RealtimeKit Video Background Transformer

A package that adds video background & blur to participant video feeds in your RealtimeKit Meetings.
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

• About the Project
  • Built With
• Getting Started
  • Prerequisites
  • Installation
• Usage
• Contributing
• License

About The Project

This package provides the foundation to add video background and blur to a participant's video feed in your RealtimeKit Meeting with your custom UI implementation.

If you are using the rtk-meeting component directly and prefer abstraction over this package, refer to UI Kit Addons.

Built With

• RealtimeKit
• Typescript

Getting Started

Installation

npm install @cloudflare/realtimekit-virtual-background

Usage

Disable the default per frame rendering of video middleware to improve speed and quality by letting this middleware control it on its own.

await meeting.self.setVideoMiddlewareGlobalConfig({
  disablePerFrameCanvasRendering: true
});

A videoBackgroundTransformer object can be created using the RealtimeKitVideoBackgroundTransformer.init({meeting: meeting}) method.

const videoBackgroundTransformer = await RealtimeKitVideoBackgroundTransformer.init({
  meeting,
});

Types of middlewares exposed by videoBackgroundTransformer:

1. createStaticBackgroundVideoMiddleware expects an imageUrl as a parameter and then creates the image the background for the current user.

meeting.self.addVideoMiddleware(
    await videoBackgroundTransformer.createStaticBackgroundVideoMiddleware(imageUrl)
);

2. createBackgroundBlurVideoMiddleware expects blurStrength (0-100) as a parameter (50% by default) and blurs the background of the user by the given blurStrength.

meeting.self.addVideoMiddleware(
  await videoBackgroundTransformer.createBackgroundBlurVideoMiddleware(50)
);

Note: Some browsers or their old versions might not have support for WebGL or the browser APIs that this package uses. We would recommend checking the support beforehand using:

if(RealtimeKitVideoBackgroundTransformer.isSupported()){
  const videoBackgroundTransformer = await RealtimeKitVideoBackgroundTransformer.init({
    meeting: meeting,
  });
  meeting.self.addVideoMiddleware(
    await videoBackgroundTransformer.createStaticBackgroundVideoMiddleware(`REPLACE_THIS_WITH_IMAGE_URL`)
  );
}

Note: Image URLs must allow CORS to avoid tainting the canvas. You can find such images on https://unsplash.com/ & https://imgur.com.

If in case you want to tweak the segmentation for better, sharper results, Please pass the desired segmentation config while initialising RealtimeKitVideoBackgroundTransformer.

const videoBackgroundTransformer = await RealtimeKitVideoBackgroundTransformer.init({
  meeting,
  segmentationConfig: {
    model: 'mlkit', // 'meet' | 'mlkit'
    backend: 'wasmSimd',
    inputResolution: '256x256', // '256x144' for meet
    pipeline: 'webgl2', // 'webgl2' | 'canvas2dCpu' // canvas2dCpu gives sharper blur, webgl2 is faster.
    targetFps: 35,
  }
});