Skip to content

Multistream Comprehensive Guide

Jump to bottom
Rajesh Kumar edited this page Apr 18, 2024 · 4 revisions

[Work in progress...]

Developers have a variety of options to customize how they choose to display remote videos with multistream media.

The initial step in utilizing Multistream is to include enableMultistream within the joinOptions parameter for both the meeting.join() and meeting.joinWithMedia() methods.

e.g.

const joinOptions = {
  enableMultistream: true,
  ...
};

await meeting.join(joinOptions);

// or

await meeting.joinWithMedia({joinOptions, mediaOptions});

Once the media is added, we must invoke the meeting.publishStream() function.

Presently, the entire document is segmented into three primary categories:

  1. Layouts: Determining the preferred visualization of remote streams.
  2. Event Listeners: Monitoring all remote streams and additional events.
  3. Methods and RemoteMediaManager: Facilitating layout updates and other operations.

Concepts and Terminology

TODO

Layout

When referring to "layout," we are discussing the arrangement of remote streams on the user interface. Multistream provides the freedom to design the layout according to your preferences.

The arguments for addMedia() remain unchanged. For further details, please refer to this link.

A new property, remoteMediaManagerConfig, has been introduced to the AddMediaOptions for multistream functionality.

If no value is provided for remoteMediaManagerConfig, the default settings are as follows. It's worth noting that to implement a custom layout, you can utilize the same data format as demonstrated in the predefined layouts.

Below is the default configuration value for the remote media manager.

const remoteMediaManagerConfig = {
  audio: {
    numOfActiveSpeakerStreams: 3,
    numOfScreenShareStreams: 1,
  },
  video: {
    preferLiveVideo: true,
    initialLayoutId: 'AllEqual',

    layouts: {
      AllEqual: AllEqualLayout,
      OnePlusFive: OnePlusFiveLayout,
      Single: SingleLayout,
      Stage: Stage2x2With6ThumbnailsLayout,
      ScreenShareView: RemoteScreenShareWithSmallThumbnailsLayout,
    }
  }
}

The predefined layouts for the default configuration are outlined below.

Type Description
AllEqual (max 9) All equal
OnePlusFive One big pane + five small
Single Single Video pane
Stage Stage with thumbnails
ScreenShareView Screen share with thumbnails

Let's delve into the visualization of all the predefined layouts.

  1. AllEqual


  1. OnePlusFive



  1. Single



  1. Stage



  1. ScreenShareView



Below are code samples demonstrating the predefined layouts.

// An "all equal" grid, with size up to 3 x 3 = 9:
const AllEqualLayout: VideoLayout = {
  activeSpeakerVideoPaneGroups: [
    {
      id: 'main',
      numPanes: 9,
      size: 'best',
      priority: 255,
    },
  ],
};

// A layout with 1 big pane for the highest priority active speaker and 5 small panes for other active speakers:
const OnePlusFiveLayout: VideoLayout = {
  activeSpeakerVideoPaneGroups: [
    {
      id: 'mainBigOne',
      numPanes: 1,
      size: 'large',
      priority: 255,
    },
    {
      id: 'secondarySetOfSmallPanes',
      numPanes: 5,
      size: 'very small',
      priority: 254,
    },
  ],
};

// A layout with just a single remote active speaker video pane:
const SingleLayout: VideoLayout = {
  activeSpeakerVideoPaneGroups: [
    {
      id: 'main',
      numPanes: 1,
      size: 'best',
      priority: 255,
    },
  ],
};

// A staged layout with 4 pre-selected meeting participants in the main 2x2 grid and 6 small panes for other active speakers at the top:
const Stage2x2With6ThumbnailsLayout: VideoLayout = {
  activeSpeakerVideoPaneGroups: [
    {
      id: 'thumbnails',
      numPanes: 6,
      size: 'thumbnail',
      priority: 255,
    },
  ],
  memberVideoPanes: [
    {id: 'stage-1', size: 'medium', csi: undefined},
    {id: 'stage-2', size: 'medium', csi: undefined},
    {id: 'stage-3', size: 'medium', csi: undefined},
    {id: 'stage-4', size: 'medium', csi: undefined},
  ],
};

// A strip of 8 small video panes (thumbnails) displayed at the top of a remote screenshare:
const RemoteScreenShareWithSmallThumbnailsLayout: VideoLayout = {
  screenShareVideo: {size: 'best'},
  activeSpeakerVideoPaneGroups: [
    {
      id: 'thumbnails',
      numPanes: 8,
      size: 'thumbnail',
      priority: 255,
    },
  ],
};

Explanation of the RemoteMediaManager Object

{
  audio: {
    numOfActiveSpeakerStreams: number; // number of audio streams we want to receive
    numOfScreenShareStreams: number; // 1 should be enough, because in webex only 1 person at a time can be presenting screen share
  };
  video: {
    preferLiveVideo: boolean; // applies to all pane groups with active speaker policy
    initialLayoutId: LayoutId;

    layouts: {[key: LayoutId]: VideoLayout}; // a map of all available layouts, a layout can be set via setLayout() method
  };
}

Properties of audio object

Name Description Type
numOfActiveSpeakerStreams The maximum number of speakers you can hear simultaneously. number
numOfScreenShareStreams The maximum number of screenshare streams (one should suffice, as only one person can present at a time in Webex). number

Properties of video object

Name Description Type
preferLiveVideo If set to true, you only receive the video stream of participants who have activated their video. boolean
initialLayoutId One of the keys defined within the layouts object. string
layouts Key-Value pairs where the value determines the layout (Additional details provided in the following table). object

Properties of VideoLayout object

Name Description Type
screenShareVideo Defines the screenshare stream. { size: PaneSize }
activeSpeakerVideoPaneGroups Defines the active speaker panes Array<ActiveSpeakerPaneObject>
memberVideoPanes Defines the member video panes Array<MemberPaneObject>

PaneSize can take the following values:

PaneSize Description
thumbnail The smallest possible resolution, 90p or less
very small 180p or less
small 360p or less
medium 720p or less
large 1080p or less
best Highest possible resolution

Properties of ActiveSpeakerPaneObject

Name Description Type
id Arbitrary value string
numPanes Number of streams that you will receive number
size Size of each stream PaneSize (string)
priority Priority of the Streams number (0-255)

Properties of MemberPaneObject

Name Description Type
id Arbitrary value (e.g. stage-1) string
size Size of the stream PaneSize (string)
csi It serves as a unique identifier for a stream (When a client generates a stream, it assigns a CSI value to it). number

Note:

  1. Member video panes are exclusively accessible within the stage layout, allowing for the pinning of specific participants' positions in the layout. The stage layout is instantiated when memberVideoPanes is included in the configuration of layouts. In this scenario, developers are responsible for managing and displaying participants as they join or leave, contrasting with activeSpeakerVideoPaneGroups, where a consistent list of streams featuring active speakers/participants is always received. Additionally, the active speaker video pane can be pinned. Therefore, participants can be pinned either using memberVideoPanes or by invoking pinActiveSpeakerVideoPane(remoteMedia, csi) for activeSpeakerVideoPaneGroups.

  2. Each member/participant can have up to four CSI values for each stream type (audio, video, screenshareVideo, screenshareAudio). Furthermore, for a single stream type like video, multiple CSIs may exist depending on the number of devices a participant has joined with.

  3. In most scenarios, you'll primarily utilize the activeSpeakerVideoPaneGroups.

Event Listeners

Events can be listened to using the meeting.on() method.

Below is a sample code demonstrating how to listen for events.

meeting.on('[event name]', (data) => {
  console.log(data);
});

e.g.

meeting.on('media:remoteVideo:layoutChanged', ({
  layoutId, activeSpeakerVideoPanes, memberVideoPanes, screenShareVideo
}) => {
  console.log('layoutId: ', layoutId);
  console.log('activeSpeakerVideoPanes:', activeSpeakerVideoPanes);
  console.log('memberVideoPanes:', memberVideoPanes);
  console.log('screenShareVideo:', screenShareVideo);
});

Note: The event media:remoteVideo:layoutChanged will be triggered with the initialLayoutId the first time, even if there are no changes in the layout.

Media Events

Event name Data Received
media:remoteAudio:created List of audio streams. These streams remain unchanged regardless of any layout changes.
media:remoteScreenShareAudio:created Object representing the screen share audio media group
media:remoteVideo:layoutChanged { layoutId, activeSpeakerVideoPanes, memberVideoPanes, screenShareVideo }
media:remoteVideoSourceCountChanged Number of remote video sources - { numTotalSource, numLiveSources }
media:remoteAudioSourceCountChanged Number of remote audio sources - { numTotalSource, numLiveSources }
media:activeSpeakerChanged List of member IDs for active speakers - { memberIds }

media:remoteScreenShareAudio:created

To obtain the list of remote media elements from the group, you must invoke getRemoteMedia().

const remoteMedia = screenShareAudioMediaGroup.getRemoteMedia()[0];

const remoteMediaStream = remoteMedia.stream;

Asynchronous: No

Parameters

Name Description Type Mandatory
filter A filtering string to retrieve a particular type of remote media. all (Default), pinned, unpinned No

Returns: Array<RemoteMedia>

media:remoteVideo:layoutChanged

Data Received Description
layoutId AllEqual, OnePlusFive, Single, Stage or, ScreenShareView
activeSpeakerVideoPanes [groupId, groupRemoteMedia]
memberVideoPanes [paneId, remoteMedia]
screenShareVideo RemoteMedia - { id, stream, sourceState (no source, live) , memberId }

Note: The layoutId list displayed here corresponds to those defined in the configuration. These IDs may vary depending on your configuration settings.

media:remoteVideoSourceCountChanged

Data Received Description
numTotalSource Total number of all video sources
numLiveSources Total number of live video sources

media:remoteAudioSourceCountChanged

Data Received Description
numTotalSource Total number of all audio sources
numLiveSources Total number of live audio sources

Meeting Events

Event name Description
meeting:stoppedSharingLocal TODO
meeting:startedSharingRemote TODO
meeting:stoppedSharingRemote TODO

Remote Media Events

The remoteMedia received may contain streams from various participants at different times. It dynamically changes as other participants begin speaking. Therefore, to accurately display the corresponding name/data alongside the stream, the memberId is also provided within the remoteMedia object. This memberId can be utilized to retrieve the name.

Events can be listened to using the remoteMedia.on() method.

e.g.

remoteMedia.on('[Event name]', (data) => {
  console.log(data);
});
Event name Description
sourceUpdate This event is triggered when the source of the remoteMedia changes, such as when another participant begins speaking.
stopped This event is triggered when the participant has deactivated their camera.

With sourceUpdate, you can inspect the memberId and sourceState to obtain the most recent information about the remoteMedia.

Information received during the sourceUpdate event.

Data Received Description
'no source' There is no video.
'invalid source' Source is invalid.
'live' The video is available.
'avatar' Camera is muted or didn't have any video. Avatar can be shown.
'bandwidth disabled' There is not enough bandwidth to show video. Avatar can be shown.
'policy violation' Restriction due to policy where they can not receive the video.

Methods

There exist multiple methods for managing various scenarios and updating the layout.

Get an array of a member's CSIs

const csiList = meeting.members.getCsisForMember(memberId, mediaType='video', mediaContent='main');

Asynchronous: No

Parameters

Name Description Type Mandatory
memberId Member's ID string Yes
mediaType The media type audio, video (Default) No
mediaContent For what type of media content main (Default), slides No

Returns: Array<number>

Find member by CSI

const member = meeting.members.findMemberByCsi(csi);

Asynchronous: No

Parameters:

Name Description Type Mandatory
csi CSI number linked to the stream number Yes

Returns: Member

Remote Media Manager

In multistream meetings, the current API meeting.addMedia() will solely establish the media connection without publishing any tracks. However, it will configure streams to receive remote media based on the provided remoteMediaManagerConfig as one of the options.

The Meeting.addMedia() method accepts an optional configuration object to set up the RemoteMediaManager.

meeting.addMedia({
  ...,
  remoteMediaManagerConfig?: RemoteMediaManagerConfiguration;
})

RemoteMediaManager configuration object

{
  audio: {
    numOfActiveSpeakerStreams: number; // number of audio streams we want to receive
    numOfScreenShareStreams: number; // 1 should be enough, because in webex only 1 person at a time can be presenting screen share
  };
  video: {
    preferLiveVideo: boolean; // applies to all pane groups with active speaker policy
    initialLayoutId: LayoutId;

    layouts: {[key: LayoutId]: VideoLayout}; // a map of all available layouts, a layout can be set via setLayout() method
  };
}

Note: This object's details are outlined in the Explanation of the RemoteMediaManager Object section.

Set remote video CSI

meeting.remoteMediaManager.setRemoteVideoCsi(remoteMedia, csi);

Asynchronous: No

Parameters:

Name Description Type Mandatory
remoteMedia RemoteMedia object RemoteMedia Yes
csi CSI number assiciated with the stream number Yes

Returns: void

Add member video pane

const remoteMedia = await meeting.remoteMediaManager.addMemberVideoPane(
  { id: PaneId; size: PaneSize; csi?: CSI; }
);

Asynchronous: Yes

Parameters: An object with following parameters

Name Description Type Mandatory
id Pane ID string Yes
size Size of the video pane thumbnail, very small, small, medium, large, best Yes
csi CSI number assiciated with the stream number No

Returns: Promise<RemoteMedia>

Get currently selected layout ID

const layoutId = meeting.remoteMediaManager?.getLayoutId();

Asynchronous: No

Parameters: No

Returns: string

Change the Layout

Calling this method will trigger the VideoLayoutChanged event. TODO: Verify the event name

await meeting.remoteMediaManager.setLayout(layoutId);

Asynchronous: Yes

Parameters:

Name Description Type Mandatory
layoutId ID of the defined layout string Yes

Returns: Promise<void>

Set the prefer live video

meeting.remoteMediaManager.setPreferLiveVideo(preferLiveVideo);

Asynchronous: No

Parameters:

Name Description Type Mandatory
preferLiveVideo Enabling this option will exclusively retrieve streams with active video. boolean Yes

Returns - void

Set CSIs for multiple RemoteMedia instances belonging to RemoteMediaGroup

meeting.remoteMediaManager.setActiveSpeakerCsis(remoteMediaCsis);

Asynchronous: No

Parameters: An array of object with the following properties. i.e. Array<{ remoteMedia, csi }>

Name Description Type Mandatory
remoteMedia The remote media object RemoteMedia Yes
csi CSI number assiciated with the stream number No

Returns: void

Note: For each entry in the remoteMediaCsis array:

  • if csi is specified, the RemoteMedia instance is pinned to that CSI
  • if csi is undefined, the RemoteMedia instance gets unpinned

Set a new CSI on a given remote media object

meeting.remoteMediaManager.setRemoteVideoCsi(remoteMedia, csi);

Asynchronous: No

Parameters:

Name Description Type Mandatory
remoteMedia The remote media object to be altered RemoteMedia Yes
csi A new CSI value. It can be null if you wish to stop receiving media. number, null Yes

Returns - void

Add a new member video pane to the currently selected layout

const remoteMedia = await meeting.remoteMediaManager.addMemberVideoPane(newPane);

Asynchronous: Yes

Parameters: An object with the following properties

Name Description Type Mandatory
id Pane ID string Yes
size Pane size thumbnail, very small, small, medium, large, best Yes
csi A new CSI value number No

Returns - Promise<RemoteMedia>

Note: Changes to the layout are lost after a layout change.

Remove a member video pane from the currently selected layout

await meeting.remoteMediaManager.removeMemberVideoPane(paneId);

Asynchronous: Yes

Parameters: An object with the following properties

Name Description Type Mandatory
paneId Pane ID string Yes

Returns - Promise<void>

Note: Changes to the layout are lost after a layout change.

Pin an active speaker remote media object to the given CSI value

meeting.remoteMediaManager.pinActiveSpeakerVideoPane(remoteMedia, csi);

Pins an active speaker remote media object to the given CSI value. From that moment onwards the remote media will only play audio/video from that specific CSI until unpinActiveSpeakerVideoPane() is called or current layout is changed.

Asynchronous: No

Parameters: An object with the following properties

Name Description Type Mandatory
remoteMedia Remote media object reference RemoteMedia Yes
csi CSI value to pin to, if undefined, then current CSI value is used number No

Returns - void

Unpin a remote media object from the fixed CSI value it was pinned to

meeting.remoteMediaManager.unpinActiveSpeakerVideoPane(remoteMedia);

Asynchronous: No

Parameters:

Name Description Type Mandatory
remoteMedia Remote media object reference RemoteMedia Yes

Returns - void

Check if a given remote media object belongs to an active speaker group and has been pinned

const isPinned = meeting.remoteMediaManager.isPinned(remoteMedia);

Asynchronous: No

Parameters:

Name Description Type Mandatory
remoteMedia Remote media object reference RemoteMedia Yes

Returns - boolean

Note: Throws an error if the remote media object doesn't belong to any active speaker remote media group.

Set Size Hint

If the client is displaying remote video on screens of various sizes and occasionally requires higher or lower resolution video for proper display or bandwidth conservation, they can specify a size hint. Upon receiving this hint, the remote will begin transmitting the video in the requested resolution.

setSizeHint() can be invoked on the remoteMedia object. Each video stream corresponds to a remoteMedia object.

remoteMedia.setSizeHint(width, height);

Asynchronous: No

Parameters:

Name Description Type Mandatory
width Width of the video element number Yes
height Height of the video element number Yes

Returns - void

Kitchen Sink App

To explore Multistream functionality, feel free to explore our Meeting Samples App.

Under Manage Meeting section, check the box with label Use a multistream connection before clicking on Join Meeting or Join with Media.

Caution

The Documentation is no longer be current.

  • Click here to view updated content for Calling SDK from our Developer Portal.

  • Click here to view updated content for Meetings SDK from our Developer Portal.

Webex Web SDK

Calling

Meeting

Clone this wiki locally