trtc-js-sdk-infinity

Feature Description

This article focuses on how to publish more than 50 streams to a single room. It is suitable for metaverse scenarios.

Prerequisites

• You need to read these tutorials How to run TRTC demo and Basic Audio/Video Call.

• The business side needs to maintain its own user list and the remote users' publishing state (whether remote users publishing video, audio, screen sharing). TRTC Web SDK does not throw stream-added, stream-updated, stream-removed, mute-video, mute-audio, unmute- video, unmute-audio and other events related to the user's publishing state.

• Environment supports

  Chrome	Edge	Firefox	Safari	iOS Safari	Android Chrome
  69+	80+	59+	11+	11+	69+

Implementation Process

1. Install trtc sdk and import

• Install

  npm i trtc-js-sdk-infinity

• Import trtc to your project.

  import TRTC from 'trtc-js-sdk-infinity';

2. Join room

• API docs: What is userSig?, TRTC.createClient, Client.join.

  const client = TRTC.createClient({
    mode: 'infinity', // use infinity mode
    sdkAppId: 140000000, // your sdkAppId
    userId: 'B', // your userId
    userSig: 'fake_sig', // your userSig
  });
  
  await client.join({ roomId: 6969 });

3. Publish stream

• API docs: TRTC.createStream, LocalStream.initialize, Client.publish, LocalStream.play.

  // publish microphone and camera stream
  const localStream = TRTC.createStream({
    audio: true, // capture microphone
    video: true // capture camera
  });
  await localStream.initialize();
  await client.publish(localStream);
  await localStream.play('local_stream_element_id');

  // publish screen sharing stream
  const screenStream = TRTC.createStream({
    screen: true // capture screen
  });
  await screenStream.initialize();
  await client.publish(screenStream);

  // publish screenStream with screenAudio
  const screenStream = TRTC.createStream({
    screen: true, // capture screen
    screeAudio: true // capture screenAudio
  });
  await screenStream.initialize();
  await client.publish(screenStream);

4. Open/close camera & microphone

• API docs: LocalStream.addTrack, LocalStream.removeTrack, LocalStream.getAudioTrack, LocalStream.getVideoTrack.

  // close camera
  const videoTrack = localStream.getVideoTrack();
  await localStream.removeTrack(videoTrack);
  videoTrack.stop();
  
  // open camera
  const tempStream = TRTC.createStream({ audio: false, video: true });
  await tempStream.initialize();
  await localStream.addTrack(tempStream.getVideoTrack());
  
  // close microphone
  const audioTrack = localStream.getAudioTrack();
  await localStream.removeTrack(audioTrack);
  audioTrack.stop();
  
  // open microphone
  const tempStream = TRTC.createStream({ audio: true, video: false });
  await tempStream.initialize();
  await localStream.addTrack(tempStream.getAudioTrack());

5. Subscribe and play remoteStream

• API docs: Client.subscribe, RemoteStream.play.

• The client.subscribe interface in this case differs slightly from the API docs.

  const { mainStream, auxiliaryStream } = await Client.subscribe(remoteUserId, subscriptionState);
  
  // mainStream & auxiliaryStream is instance of RemoteStream: https://web.sdk.qcloud.com/trtc/webrtc/doc/en/RemoteStream.html
  
  // mainStream contains remoteUser's audio(microphone or screenAudio) & video(camera)
  // if mainStream is undefined, means remoteUser is not publishing microphone & screenAudio & camera.
  
  // auxiliaryStream contains remoteUser's screenSharing.
  // if auxiliaryStream is undefined, means remoteUser is not publishing screenSharing.

  Parameter	Type	Description
  remoteUserId	String	Remote userId of published streams.
  subscriptionState	Object
  subscriptionState.audio	Boolean	Whether to subscribe audio(usually means microphone or screenAudio) stream of remote user.
  subscriptionState.video	Boolean	Whether to subscribe video(usually means camera) stream of remote user.
  subscriptionState.auxiliary	Boolean	Whether to subscribe auxiliary(usually means screen sharing) stream of remote user.

  const remoteStreamMap = new Map(); // Map<userId_streamType, remoteStream>
  
  // In a metaverse scenario, when another user publish a stream and walks up to the current user, the current user can call the client.subscribe interface to subscribe to the audio/video stream.
  const remoteUserId = 'A';
  const subscriptionState =   { 
      audio: true, // Whether to subscribe to the audio (microphone or screenAudio)
      video: true, // Whether to subscribe to the camera
      auxiliary: true // Whether to subscribe to the screen sharing
  }
  const { mainStream, auxiliaryStream } = await client.subscribe(remoteUserId, subscriptionState);
  
  remoteStreamMap.set(`${mainStream.getUserId()}_${mainStream.getType()}`);
  remoteStreamMap.set(`${auxiliaryStream.getUserId()}_${auxiliaryStream.getType()}`);
  
  // mainStream contains microphone and camera data.
  await mainStream.play('main_stream_element_id'); // main_stream_element_id is the element id of a div tag in the DOM.
  // auxiliaryStream contains screen sharing data.
  await auxiliaryStream.play('aux_stream_element_id');

• The subscriptionState needs to be the same as the remote user's publishing state, otherwise client.subscribe will fails.

  // eg: A is publishing microphone + camera + screen sharing, then B trying to subscribe A
  try {
    const { mainStream } = await await client.subscribe('A', { 
      audio: true,
      video: true,
      auxiliary: false // A is publishing screen sharing，but auxiliary is set to false(should be set to true), client.subscribe will be failed.
    });
  } catch(error) {
    // subscribe state is inconsistent with remote publishing state
    if (error.getCode() === 16461) {
      console.error(error);
      // error.data is the publishing state of A
      console.warn(error.data);
      // your can retry subscription by passing error.data to client.subscribe()
      client.subscribe('A', error.data);
    }
  }

• Subscriber should resubscribe when publisher's publishingState changed.

  // 1. A publishingState is { audio: true, video: true, screen: false, screenAudio: false }
  // 2. B subscribe A
  let mainStream = undefined;
  const result = await client.subscribe('A', { audio: true, video: true, auxiliary: false });
  if (result.mainStream !== mainStream) {
    mainStream = result.mainStream;
    if (mainStream) {
      mainStream.play('div_id_in_dom_object')
    }
  }
  
  
  // 3. A close camera, and send publishingState to B.
  // 4. B need to call subscribe again when received A's publishingState.
  const result = await client.subscribe('A', { audio: true, video: false, auxiliary: false })
  if (result.mainStream !== mainStream) {
    mainStream = result.mainStream;
    if (mainStream) {
      mainStream.play('div_id_in_dom_object')
    }
  }

• How to set subscriptionState.audio?

  // remotePublishingState is other user's publishing state, broadcast from your server.
  const subscriptionState = {
    audio: remotePublishingState.audio || remotePublishedState.screenAudio; // microphone and screenAudio use the same audio track
    video: remotePublishingState.video
    auxiliary: remotePublishingState.screen
  }
  
  const { mainStream, auxiliaryStream } = await client.subscribe(remoteUserId, subscriptionState);
  
  // mainStream contains screenAudio

6. Unsubscribe remote user

   // When other users unpublished stream, leaved room, or walked away on metaverse scene. You need to call client.unsubscribe to unsubscribe the audio/video stream.
   await client.unsubscribe('A');
   
   // clear streams and stop playback
   const mainStream = remoteStreamMap.get('A_main');
   const auxiliaryStream = remoteStreamMap.get('A_auxiliary');
   if (mainStream) {
     mainStream.stop(); // stop playback
   }
   if (auxiliaryStream) {
     auxiliaryStream.stop(); // stop playback
   }
   remoteStreamMap.delete('A_main');
   remoteStreamMap.delete('A_auxiliary');

7. Unpublish stream

   await client.unpublish(localStream);
   await client.unpublish(screenStream);

8. Leave room

   await client.leave();

How to maintain a user list on the business side

Option one

1. When a user publish stream, open/close the camera, open/close the microphone, or unpublish stream, you need send the publishing state of this user to your server.

2. Then, your server need to sends the user's publishing state to the other users in the room, ensuring that the other users in the room are notified.

3. The other users can determine whether they need to subscribe to this user's stream based on their position in the metaverse.

   const localStream = TRTC.createStream({ userId: 'B', audio: true, video: true });
   await localStream.initialize();
   await client.publish(localStream);
   
   const publishingState = { userId: 'B', audio: true, video: true, screen: false, screenAudio: false }
   
   // send B publishingState to your server and broadcast to other users.

   const screenStream = TRTC.createStream({
     screen: true,
     screenAudio: true
   });
   await screenStream.initialize();
   await client.publish(screenStream);
   
   publishingState.screenAudio = screenStream.hasAudio();
   
   // send B publishingState to your server and broadcast to other users.

   // close camera
   const videoTrack = localStream.getVideoTrack();
   await localStream.removeTrack(videoTrack);
   videoTrack.stop();
   
   publishingState.video = false;
   
   // send B publishingState to your server and broadcast to other users.

   await client.unpublish(localStream);
   
   publishingState.audio = false;
   publishingState.video = false;
   
   // send B publishingState to your server and broadcast to other users.

Option two

The business side manages the user list through TRTC's server-side callback events. For details, refer to: https://www.tencentcloud.com/document/product/647/39558.

Best Practices for Handling Reconnections

1. Detect current network quality

Listen to client.on('network-quality') event before join room.

client.on('network-quality', event => {
  console.log(`network-quality, uplinkNetworkQuality:${event.uplinkNetworkQuality}, downlinkNetworkQuality: ${event.downlinkNetworkQuality}`)
});

await client.join({ roomId });
...

2. About reconnection

The SDK has two types of connections:

• 1. Websocket connection for signaling.
• 2. Stream connection for audio/video transmission.

When a network problem causes a disconnection, TRTC SDK will try to automatically restore the connection. You can refer to the following code example to achieve a better user experience.

// refer to: https://web.sdk.qcloud.com/trtc/webrtc/doc/en/module-ClientEvent.html#.CONNECTION_STATE_CHANGED
client.on('connection-state-changed', event => {
  const prevState = event.prevState;
  const curState = event.state;

  // DISCONNECTED -> CONNECTING: websocket is connecting.
  // CONNECTING -> CONNECTED: websocket is connected.
  // CONNECTED -> DISCONNECTED: websocket is disconnected.
  // DISCONNECTED -> RECONNECTING: websocket is reconnecting.
  // RECONNECTING -> CONNECTED: websocket is reconnected.
  // RECONNECTING -> DISCONNECTED: websocket reconnect failed.
})

client.on('stream-connection-state-changed', event => {
  const prevState = event.prevState;
  const curState = event.state;

  // DISCONNECTED -> CONNECTING: stream connection is connecting.
  // CONNECTING -> CONNECTED: stream connection is connected.
  // CONNECTED -> DISCONNECTED: stream connection is disconnected.
  // DISCONNECTED -> RECONNECTING: stream connection is reconnecting.
  // RECONNECTING -> CONNECTED: stream connection is reconnected.
  // RECONNECTING -> DISCONNECTED: stream connection reconnect failed.
})

ChangeLog

Beta 42

Bug Fixed

• Fix the issue that setting the resolution of screen sharing does't works.

Beta 41

Improvement

• Optimize the capture logic of screen sharing to improve screen sharing smoothness.

Beta 40

Feature

• Support "dynamic calls" for localStream.setScreenProfile.

Beta 39

Improvement

• Optimize recapture logic in case of abnormal camera capture.

Beta 38

Breaking Change

• Change the value error code SUBSCRIBE_STATE_NOT_MATCH to 16462(16461 before).

Improvement

• Add API_CALL_ABORTED(error code is 16461) error for subscribe & unsubscribe API.

  // case 1:  
  // If you execute the following code without await, the second clientB.subscribe will fail and throw error(16461).
  // The final subscription state will be clientB subscribe clientA with { audio: true, video: false, auxiliary: false }.
  
  clientB.subscribe('clientA', { audio: true, video: false, auxiliary: false });
  clientB.subscribe('clientA', { audio: true, video: true, auxiliary: false }); // will throw API_CALL_ABORTED error. error.getCode() === 16461
  clientB.subscribe('clientA', { audio: true, video: false, auxiliary: false });

  // If you execute the following code without await, the second and third clientB.subscribe will fail and throw error(16461).
  // The final subscription state will be clientB unsubscribe clientA
  clientB.subscribe('clientA', { audio: true, video: false, auxiliary: false });
  clientB.subscribe('clientA', { audio: true, video: true, auxiliary: false }); // will throw API_CALL_ABORTED error. error.getCode() === 16461
  clientB.subscribe('clientA', { audio: false, video: true, auxiliary: false }); // will throw API_CALL_ABORTED error. error.getCode() === 16461
  clientB.unsubscribe('clientA')

  1. You can ignore API_CALL_ABORTED error, TRTC SDK will ensure that the final subscription state meets the expectations of the business-side call.
  2. This has the advantage of updating the subscription state faster when the client.subscribe & client.unsubscribe APIs are called at high frequencies.
  3. Subscribe & unsubscribe for different userId's do not affect each other

Beta 37

Improvement

• Optimize reconnection logic.

Beta 36

Bug fixed

• Remote video may be black after subscribe video(for single client).

Beta 35

Bug fixed

• Publish screenAudio may fails(for single client).
• Cannot receive screensharing after remote user reconnected(for single client).