feat: Add V3 background segmentation video frame processor#3223
Conversation
72f5adb to
fd06b99
Compare
|
Note: Multiclass-model based integration tests in |
| static async create( | ||
| logger: Logger, | ||
| config: BackgroundSegmentationVideoFrameProcessorConfig, | ||
| modelType: ModelType = ModelType.SELFIE_GENERAL, |
There was a problem hiding this comment.
I thought we were defaulting to multi-class?
There was a problem hiding this comment.
The default matches the CDN processor's default. We decided to default to SELFIE_GENERAL because the multiclass model (18MB) auto-downgrades to general on CPU-only and low-performance devices as part of QA bugs fix for some device-specific issues.
cc: @dinmin-amzn
There was a problem hiding this comment.
i think it's reasonable, but how drastric is the auto downgrade? as a followup how can we get more default users on multiclass?
There was a problem hiding this comment.
The auto-downgrade is seamless — the CDN processor detects device capability at init and silently loads general (8MB) instead of multiclass (18MB). The user sees no error or interruption; they just get the lighter model. Visual quality difference is subtle for most scenarios (both models handle blur/replacement well), with multiclass providing better segmentation accuracy per our benchmark data. Once we have confidence for devices-specific issues, we can flip the default easily back to Multiclass model as it is a one-liner change.
fd06b99 to
82707c4
Compare
| describe('videoProcessorMetricsDidReceive', () => { | ||
| it('stores video processor metrics', () => { | ||
| statsCollector = new StatsCollector(audioVideoController, logger, interval); | ||
| const metrics = { | ||
| frameRate: 15, | ||
| processors: [{ averageProcessLatency: 10.5, processorName: 'TestProcessor' }], | ||
| }; | ||
| statsCollector.videoProcessorMetricsDidReceive(metrics); | ||
| }); | ||
| }); | ||
|
|
||
| describe('backgroundSegmentationMetricsDidReceive', () => { | ||
| it('stores background segmentation metrics', () => { | ||
| statsCollector = new StatsCollector(audioVideoController, logger, interval); | ||
| statsCollector.backgroundSegmentationMetricsDidReceive({ | ||
| metricName: 'test', | ||
| timestamp: Date.now(), | ||
| }); | ||
| }); | ||
|
|
||
| it('drops oldest when max exceeded', () => { | ||
| statsCollector = new StatsCollector(audioVideoController, logger, interval); | ||
| for (let i = 0; i < 55; i++) { | ||
| statsCollector.backgroundSegmentationMetricsDidReceive({ | ||
| metricName: `m-${i}`, | ||
| timestamp: Date.now(), | ||
| }); | ||
| } | ||
| }); |
There was a problem hiding this comment.
How do these tests verify the correct result?
| it('passes metrics collector to internal metrics', async () => { | ||
| sandbox.stub(BackgroundSegmentationCompatibilityChecker, 'checkCompatibility').returns({ | ||
| isCompatible: false, | ||
| missingFeatures: ['webgl2'], | ||
| }); | ||
|
|
||
| const processor = await BackgroundSegmentationVideoFrameProcessor.create(logger, { | ||
| type: ProcessorEffect.BLUR, | ||
| }); | ||
|
|
||
| const collector = { backgroundSegmentationMetricsDidReceive: sandbox.stub() }; | ||
| processor.setMetricsCollector(collector); | ||
| }); |
There was a problem hiding this comment.
How is validation done in this test?
There was a problem hiding this comment.
My bad, I need to add some assertions here.
Implement a new CDN-based background segmentation processor that supports blur (low/medium/high), image replacement, and color replacement effects with two model types (selfie_general and selfie_multiclass). Core: - BackgroundSegmentationVideoFrameProcessor with CDN asset loading - BackgroundSegmentationAssetLoader with request deduplication - BackgroundSegmentationCompatibilityChecker (WebGL2, WASM, Workers) - BackgroundSegmentationMetrics observer for StatsCollector integration - Dynamic filter switching via setConfig/setModelType without recreation Observability: - Video processor metrics (frame rate, per-processor latency) via StatsCollector - Event publishing for filter lifecycle (backgroundFilterStarted, backgroundFilterConfigSelected, backgroundFilterFailed) - Extended VideoFXEventAttributes with V3-specific fields Demo: - Replace legacy MediaPipe/TensorFlow-based filters with V3 segmentation - Add blur strength options (low/medium/high) for both model types - Fix iOS crash on rapid filter switching (concurrency guard) - Fix filter re-application after offline/online transition - Remove @tensorflow-models/body-segmentation dependency Testing: - Add unit tests for BackgroundSegmentation (processor, asset loader, compatibility checker, metrics) - Update VideoProcessingTest to use V3 background segmentation filters (replaces V2 VideoFx tests) - Add VideoTestSteps utility methods for blur, image replacement, and color replacement with model parameter - Add fake video stream (output.y4m) for consistent pixel-based filter verification - Add blue pixel percentage check and pixel diff verification for filter validation
f5eb316 to
104c1c6
Compare
3 participants
Issue #:
Implement a new CDN-based background segmentation processor that supports blur (low/medium/high), image replacement, and color replacement effects with two model types (selfie_general and selfie_multiclass).
Description of changes:
Core:
Observability:
Demo:
Integration tests:
Testing:
Can these tested using a demo application? Please provide reproducible step-by-step instructions.
Yes, these changes can be tested using demo application. Here are the steps:
- BackgroundSegmentation - (Blue) — replaces background with solid blue using general model variant
- BackgroundSegmentation - (Blur Low) / (Blur Medium) / (Blur High) - replaces background with blur effect using general model variant
- BackgroundSegmentation - (Replacement) — replaces background with encoded image using general model variant
- BackgroundSegmentation - (Multiclass-Blue) — replaces background with solid blue using multiclass model variant
- BackgroundSegmentation - (Multiclass-Blur Low/Medium/High) — replaces background with blur effect using multiclass model variant
- BackgroundSegmentation - (Multiclass-Replacement) — replaces background with encoded image using multiclass model variant
Checklist:
Have you successfully run
npm run build:releaselocally? YesDo you add, modify, or delete public API definitions? If yes, has that been reviewed and approved?
Do you change the wire protocol, e.g. the request method? If yes, has that been reviewed and approved?
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.