Skip to content

Implementing Video Receiver Pipelines

Jump to bottom
Kolesnik, Gennadiy edited this page Apr 6, 2026 · 2 revisions

For general information about receiver pipelines refer to the Receiver Pipelines section section.

Video Processing Pipeline Architecture

This document describes a video processing pipeline designed to handle network-streamed video data efficiently by applying various transformations to optimize visual quality and reduce network and computational load. The ssdk::video::MonoscopicVideoInput and ssdk::video::VideoReceiverPipeline components, combined with the processing slots, provide a flexible infrastructure to handle format changes, enhance quality, and manage resolution.

Platform Scope The concepts described in this document (multi-stage pipelines, presenters, and AMF components chained via slots) fully apply to Windows and Linux. On Android, video presentation is handled differently: the decoder renders directly into a view surface and there is no traditional multi-stage post-processing pipeline. Platform-specific differences are called out where relevant.

Core Components

Network Transport and MonoscopicVideoInput

Video data is obtained from the ssdk::transport_common::ClientTransport interface and relayed to the ssdk::video::VideoDispatcher object through the transport_common::ClientTransport::VideoReceiverCallback. The ssdk::video::VideoDispatcher object is responsible for routing various video streams to their respective video inputs, represented by the ssdk::video::MonoscopicVideoInput class. Upon receiving the video buffers, ssdk::video::MonoscopicVideoInput decodes compressed video frames and forwards the decoded frames to the ssdk::video::VideoReceiverPipeline-based objects.

On Windows and Linux, the decoded frames are then passed through a chain of AMF components and finally presented using a dedicated presenter component. On Android, the decoder targets a view surface directly, and only frame metadata is propagated through the pipeline abstraction.

VideoReceiverPipeline Base Class

ssdk::video::VideoReceiverPipeline provides the common, platform-agnostic base for receive-side video processing. It owns the basic slot infrastructure and exposes methods such as:

  • SubmitInput() – entry point for decoded frames or pass-through surfaces.
  • OnInputChanged() – reinitializes internal state when input format changes (codec, resolution, frame rate, color space, etc.).
  • Shutdown() / lifecycle helpers – to tear down pipeline resources.

Platform-specific implementations extend this base as follows:

  • Windows and Linux: ssdk::video::VideoReceiverPipelinePC derives from VideoReceiverPipeline and implements a full multi-stage pipeline, including optional denoiser, high-quality upscaler, and color-space / format conversion, with a video presenter used as the final sink.
  • Android: ssdk::video::VideoReceiverAndroid (located in sdk/video/android) adapts the same base concepts but does not build a traditional multi-stage GPU pipeline. Instead, it configures the decoder to output directly to a platform-specific view surface (e.g., a Surface/SurfaceView/TextureView), and uses the pipeline infrastructure only to propagate timing and metadata.

Slots and AMF Components (Windows and Linux)

On Windows and Linux, once decoded or passed through, video data is submitted to the processing pipeline using the VideoReceiverPipeline::SubmitInput() method. The pipeline consists of a series of slots, each containing an amf::AMFComponent object.

The slots can be implemented as either synchronous or asynchronous, based on the component’s processing characteristics:

  1. Synchronous Slots (ssdk::util::SynchronousSlot): Executes SubmitInput and QueryOutput in a single thread. This approach is compatible with components that produce a matching output frame for every input frame.
  2. Asynchronous Slots (ssdk::util::AsynchronousSlot): Executes SubmitInput and QueryOutput in separate threads, ideal for high-complexity components with varied input-output ratios.

Pipeline Components on Windows and Linux

In VideoReceiverPipelinePC, the following key components are typically chained together, each serving a distinct purpose in video enhancement, resolution handling, and color space conversion:

1. Video Denoiser/Compression Artifact Remover

This component, implemented using the AMFVQEnhancer in the AMF runtime, improves visual clarity by removing noise and compression artifacts from the video. It enables rendering and streaming at lower resolutions and bitrates, reducing the load on the network and encoding/decoding resources while enhancing quality.

  • Performance Benefits: By reducing required bitrate and resolution, this component improves streaming efficiency and reduces bandwidth.
  • Hardware Requirements: Available only on Windows systems with AMD GPUs/APUs.
  • Performance Considerations: This process is computationally intensive, especially on lower-end GPUs, and can be a bottleneck at high resolutions or frame rates.

2. High-Quality Upscaler

The AMFHQScaler component provides advanced upscaling for scenarios where the stream resolution is lower than the display resolution. This upscaling is applied only when necessary.

  • Performance Benefits: Allows video streaming at a reduced resolution, with high-quality upscaling on the client side, optimizing both network and GPU resource usage.
  • Hardware Requirements: Available only on Windows systems with AMD GPUs/APUs.
  • Performance Considerations: Like the denoiser, this component is computationally heavy, potentially causing bottlenecks on lower-end GPUs with high resolutions or frame rates.

3. Video Converter

The Video Converter performs essential color space conversion and can also perform resolution scaling (either upscaling or downscaling) when required. This component operates as follows:

  • Color Space Conversion: Converts the YUV format output by the decoder to the RGB format required by the presenter.
  • Scaling: Uses bilinear or bicubic scalers but defers scaling to the presenter itself when the high-performance upscaler is not being used. This approach maintains performance by only applying essential conversions within the pipeline.

Format Change Handling

When a format change occurs in the video stream (e.g., codec, resolution, frame rate, color space), MonoscopicVideoInput invokes VideoReceiverPipeline::OnInputChanged() to reinitialize pipeline components for compatibility with the new format. The base class takes care of common state management, while VideoReceiverPipelinePC and VideoReceiverAndroid perform any platform-specific reconfiguration (for example, re-creating the presenter chain on PC or updating the decoder / surface configuration on Android).

Data Flow and Processing in the Pipeline

Windows and Linux (VideoReceiverPipelinePC)

On PC platforms, the video pipeline, which sits between MonoscopicVideoInput and the video presenter, processes video as follows:

  1. Input Handling: MonoscopicVideoInput submits decoded frames to VideoReceiverPipeline::SubmitInput().
  2. Sequential Processing: VideoReceiverPipelinePC pushes each frame through the configured slots (denoiser, upscaler, converter, and any additional components) depending on the stream’s resolution, display requirements, and available hardware.
  3. Synchronous and Asynchronous Execution: Each component is evaluated to determine whether synchronous or asynchronous execution is optimal.
  4. Final Output: The presenter receives frames for display, with resolution and color matching as required.

Recommendations for Optimal Slot Configuration

To optimize performance on Windows and Linux, consider the following:

  • Denoiser and Upscaler on Capable Hardware: Ensure these components are executed asynchronously if they require heavy processing, especially at high resolutions.
  • Experiment with Slot Type: For components with stable input-output behavior, synchronous slots may perform best; otherwise, asynchronous slots provide flexibility and responsiveness for components with dynamic output requirements.

This structure allows the video pipeline to adapt to different hardware and network conditions, providing a high-quality streaming experience on AMD GPUs while efficiently managing computational and network resources on PC, while Android uses a simplified, surface-based presentation model backed by the same base VideoReceiverPipeline infrastructure for timing and control.

Android (VideoReceiverAndroid)

On Android, there is no separate video presenter component in the Streaming SDK pipeline. Instead:

  • The video decoder is configured to output directly to a platform-specific view surface owned by the Android UI layer.
  • There is effectively no multi-stage GPU post-processing pipeline equivalent to the Windows/Linux implementation.
  • The decoder component still implements amf::AMFComponent::QueryOutput(), but on Android it only returns an amf::AMFPropertyStorage object containing per-frame metadata (such as presentation timestamps and other properties).
  • VideoReceiverAndroid uses this metadata primarily for audio/video synchronization, passing timing information to the ssdk::util::AVSynchronizer and related components, while the actual pixel presentation is handled by the Android view hierarchy.

This design keeps the synchronization model consistent across platforms while letting Android rely on its native presentation stack.

Clone this wiki locally