Skip to content

Commit 14c1d7e

Browse files
runningcodeclaudemarkushigetsentry-bot
authored
feat(replay): Add ReplayFrameObserver for snapshot testing (#5386)
* feat(replay): Add beforeStoreFrame callback (JAVA-504) Add an experimental callback that fires right before a replay frame is stored to disk. The callback receives the masked bitmap (via Hint), timestamp, and current screen name. This enables snapshot testing of replay masking without needing to decode stored video segments. Includes a Kotlin extension for ergonomic usage: options.sessionReplay.beforeStoreFrame { bitmap, ts, screen -> ... } Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(replay): Add replay snapshot UI test with Sauce Labs collection (JAVA-504) Add ReplaySnapshotTest that uses the beforeStoreFrame callback to capture masked replay frames during a Compose UI test. Frames are written to the Downloads/sauce_labs_custom_screenshots/ directory, which is the standard path Sauce Labs collects screenshots from. CI changes: - Add *.png to Sauce Labs artifact match patterns - Upload collected replay snapshots via sentry-cli build snapshots Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(replay): Use Java API in snapshot test to avoid extension dep (JAVA-504) The Kotlin extension `beforeStoreFrame` comes from `sentry-android-replay` which may not resolve in the UI test module. Use the Java callback API directly instead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(replay): Skip snapshot test on GH emulators and add changelog (JAVA-504) GH Actions emulators don't support screenshot capture for replay, so the ReplaySnapshotTest needs the same assumeThat guard used by ReplayTest. Also adds a changelog entry for the beforeStoreFrame callback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Apply suggestion from @markushi Co-authored-by: Markus Hintersteiner <markus.hintersteiner@sentry.io> * refactor(replay): Replace beforeStoreFrame with ReplaySnapshotObserver (JAVA-504) Move the frame observer API from the core sentry module to sentry-android-replay so it can use Bitmap directly instead of the Hint indirection. The new ReplaySnapshotObserver fun interface lives in the replay module and is set on ReplayIntegration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(replay): Mark ReplaySnapshotObserver as experimental and use Set in test (JAVA-504) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(replay): Add @ApiStatus.Experimental to ReplaySnapshotObserver (JAVA-504) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(replay): Make snapshotObserver public for cross-module access (JAVA-504) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(replay): Exclude ReplaySnapshotTest when integrations disabled (JAVA-504) Move ReplaySnapshotTest to a conditional androidTestReplay source set so it's only compiled when APPLY_SENTRY_INTEGRATIONS is true. The test imports replay classes that aren't on the classpath otherwise. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(replay): Copy bitmap before passing to ReplaySnapshotObserver (JAVA-504) Consumers of the observer API receive a copy of the bitmap instead of the replay system's shared instance. This eliminates race conditions and crashes when consumers store or use the bitmap asynchronously. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(replay): Move ReplaySnapshotObserver to SentryReplayOptions with Hint API (JAVA-504) Move ReplaySnapshotObserver from the replay module to SentryReplayOptions in the core module and change the callback signature to use Hint instead of Bitmap. The bitmap is now accessible via TypeCheckHint.REPLAY_FRAME_BITMAP. This allows configuring the observer during Sentry.init{} alongside other replay options, removing the need to cast replayController to ReplayIntegration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(replay): Remove unnecessary jetbrains-annotations dependency (JAVA-504) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(replay): Rename ReplaySnapshotObserver to ReplayFrameObserver (JAVA-504) Rename the interface to ReplayFrameObserver and the callback method to onMaskedFrameCaptured to clarify that frames have masking applied. Also update the changelog with a usage snippet. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Format code * fix(replay): Call onMaskedFrameCaptured in File-based onScreenshotRecorded (JAVA-504) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(changelog): Move replay entry to Unreleased section (JAVA-504) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Markus Hintersteiner <markus.hintersteiner@sentry.io> Co-authored-by: Sentry Github Bot <bot+github-bot@sentry.io>
1 parent 58e0436 commit 14c1d7e

10 files changed

Lines changed: 319 additions & 1 deletion

File tree

‎.github/workflows/integration-tests-ui.yml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,28 @@ jobs:
7373
if: env.SAUCE_USERNAME != null
7474

7575

76+
- name: Install Sentry CLI
77+
if: ${{ !cancelled() && env.SAUCE_USERNAME != null }}
78+
run: curl -sL https://sentry.io/get-cli/ | bash
79+
80+
- name: Upload Replay Snapshots to Sentry
81+
if: ${{ !cancelled() && env.SAUCE_USERNAME != null }}
82+
run: |
83+
shopt -s globstar nullglob
84+
pngs=(artifacts/**/*.png)
85+
if [ ${#pngs[@]} -gt 0 ]; then
86+
mkdir -p replay-snapshots
87+
cp "${pngs[@]}" replay-snapshots/
88+
sentry-cli build snapshots ./replay-snapshots \
89+
--app-id sentry-android-replay
90+
else
91+
echo "No replay snapshot files found, skipping upload"
92+
fi
93+
env:
94+
SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
95+
SENTRY_ORG: sentry-sdks
96+
SENTRY_PROJECT: sentry-android
97+
7698
- name: Upload test results to Codecov
7799
if: ${{ !cancelled() }}
78100
uses: codecov/test-results-action@0fa95f0e1eeaafde2c782583b36b28ad0d8c77d3

‎.sauce/sentry-uitest-android-ui.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,4 +32,5 @@ artifacts:
3232
when: always
3333
match:
3434
- junit.xml
35+
- "*.png"
3536
directory: ./artifacts/

‎CHANGELOG.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,28 @@
11
# Changelog
22

3+
## Unreleased
4+
5+
### Features
6+
7+
- Session Replay: Add `ReplayFrameObserver` for observing captured replay frames ([#5386](https://github.com/getsentry/sentry-java/pull/5386))
8+
9+
```kotlin
10+
SentryAndroid.init(context) { options ->
11+
options.sessionReplay.frameObserver =
12+
SentryReplayOptions.ReplayFrameObserver { hint, frameTimestamp, screenName ->
13+
val bitmap = hint.getAs(TypeCheckHint.REPLAY_FRAME_BITMAP, Bitmap::class.java)
14+
if (bitmap != null) {
15+
try {
16+
// Process the masked replay frame
17+
myAnalyzer.processFrame(bitmap, frameTimestamp, screenName)
18+
} finally {
19+
bitmap.recycle()
20+
}
21+
}
22+
}
23+
}
24+
```
25+
326
## 8.42.0
427

528
### Features

‎sentry-android-integration-tests/sentry-uitest-android/build.gradle.kts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,10 @@ android {
8383

8484
val applySentryIntegrations = System.getenv("APPLY_SENTRY_INTEGRATIONS")?.toBoolean() ?: true
8585

86+
if (applySentryIntegrations) {
87+
android.sourceSets["androidTest"].java.srcDirs("src/androidTestReplay/java")
88+
}
89+
8690
dependencies {
8791
implementation(
8892
kotlin(Config.kotlinStdLib, org.jetbrains.kotlin.config.KotlinCompilerVersion.VERSION)

‎sentry-android-integration-tests/sentry-uitest-android/src/androidTestReplay/java/io/sentry/uitest/android/ReplaySnapshotTest.kt‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
package io.sentry.uitest.android
2+
3+
import android.graphics.Bitmap
4+
import android.os.Environment
5+
import androidx.lifecycle.Lifecycle
6+
import androidx.test.core.app.launchActivity
7+
import io.sentry.SentryReplayOptions
8+
import io.sentry.TypeCheckHint
9+
import java.io.File
10+
import java.util.concurrent.CopyOnWriteArraySet
11+
import java.util.concurrent.CountDownLatch
12+
import java.util.concurrent.TimeUnit
13+
import kotlin.test.Test
14+
import kotlin.test.assertTrue
15+
import org.hamcrest.CoreMatchers.`is`
16+
import org.junit.Assume.assumeThat
17+
import org.junit.Before
18+
19+
class ReplaySnapshotTest : BaseUiTest() {
20+
21+
@Before
22+
fun setup() {
23+
// GH Actions emulators don't support capturing screenshots for replay
24+
@Suppress("KotlinConstantConditions")
25+
assumeThat(BuildConfig.ENVIRONMENT != "github", `is`(true))
26+
}
27+
28+
@Test
29+
fun captureComposeReplayFrameSnapshots() {
30+
val snapshotsDir =
31+
File(
32+
Environment.getExternalStoragePublicDirectory(Environment.DIRECTORY_DOWNLOADS),
33+
"sauce_labs_custom_screenshots",
34+
)
35+
.apply {
36+
deleteRecursively()
37+
mkdirs()
38+
}
39+
val frameReceived = CountDownLatch(1)
40+
val capturedScreens = CopyOnWriteArraySet<String>()
41+
42+
val activityScenario = launchActivity<ComposeActivity>()
43+
activityScenario.moveToState(Lifecycle.State.RESUMED)
44+
45+
initSentry {
46+
it.sessionReplay.sessionSampleRate = 1.0
47+
it.sessionReplay.frameObserver =
48+
SentryReplayOptions.ReplayFrameObserver { hint, frameTimestamp, screenName ->
49+
val bitmap =
50+
hint.getAs(TypeCheckHint.REPLAY_FRAME_BITMAP, Bitmap::class.java)
51+
?: return@ReplayFrameObserver
52+
val name = screenName ?: "unknown"
53+
if (capturedScreens.add(name)) {
54+
val file = File(snapshotsDir, "${name}_$frameTimestamp.png")
55+
file.outputStream().use { out -> bitmap.compress(Bitmap.CompressFormat.PNG, 100, out) }
56+
}
57+
bitmap.recycle()
58+
frameReceived.countDown()
59+
}
60+
}
61+
62+
assertTrue(frameReceived.await(10, TimeUnit.SECONDS), "Expected at least one replay frame")
63+
assertTrue(capturedScreens.isNotEmpty(), "Expected at least one screen captured")
64+
65+
val files = snapshotsDir.listFiles()?.filter { it.extension == "png" } ?: emptyList()
66+
assertTrue(files.isNotEmpty(), "Expected snapshot PNG files on disk")
67+
assertTrue(files.all { it.length() > 0 }, "Snapshot files should not be empty")
68+
69+
activityScenario.moveToState(Lifecycle.State.DESTROYED)
70+
}
71+
}

‎sentry-android-replay/src/main/java/io/sentry/android/replay/ReplayIntegration.kt‎

Lines changed: 37 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,13 @@ package io.sentry.android.replay
22

33
import android.content.Context
44
import android.graphics.Bitmap
5+
import android.graphics.BitmapFactory
56
import android.os.Build
67
import android.view.MotionEvent
78
import io.sentry.Breadcrumb
89
import io.sentry.DataCategory.All
910
import io.sentry.DataCategory.Replay
11+
import io.sentry.Hint
1012
import io.sentry.IConnectionStatusProvider.ConnectionStatus
1113
import io.sentry.IConnectionStatusProvider.ConnectionStatus.DISCONNECTED
1214
import io.sentry.IConnectionStatusProvider.IConnectionStatusObserver
@@ -17,8 +19,10 @@ import io.sentry.ReplayBreadcrumbConverter
1719
import io.sentry.ReplayController
1820
import io.sentry.SentryIntegrationPackageStorage
1921
import io.sentry.SentryLevel.DEBUG
22+
import io.sentry.SentryLevel.ERROR
2023
import io.sentry.SentryLevel.INFO
2124
import io.sentry.SentryOptions
25+
import io.sentry.TypeCheckHint
2226
import io.sentry.android.replay.ReplayState.CLOSED
2327
import io.sentry.android.replay.ReplayState.PAUSED
2428
import io.sentry.android.replay.ReplayState.RESUMED
@@ -308,13 +312,45 @@ public class ReplayIntegration(
308312
var screen: String? = null
309313
scopes?.configureScope { screen = it.screen?.substringAfterLast('.') }
310314
captureStrategy?.onScreenshotRecorded(bitmap) { frameTimeStamp ->
315+
val observer = options.sessionReplay.frameObserver
316+
if (observer != null) {
317+
val copy = bitmap.copy(bitmap.config!!, false)
318+
if (copy != null) {
319+
try {
320+
val hint = Hint()
321+
hint.set(TypeCheckHint.REPLAY_FRAME_BITMAP, copy)
322+
observer.onMaskedFrameCaptured(hint, frameTimeStamp, screen)
323+
} catch (e: Throwable) {
324+
options.logger.log(ERROR, "Error in ReplayFrameObserver", e)
325+
copy.recycle()
326+
}
327+
}
328+
}
311329
addFrame(bitmap, frameTimeStamp, screen)
312330
}
313331
checkCanRecord()
314332
}
315333

316334
override fun onScreenshotRecorded(screenshot: File, frameTimestamp: Long) {
317-
captureStrategy?.onScreenshotRecorded { _ -> addFrame(screenshot, frameTimestamp) }
335+
var screen: String? = null
336+
scopes?.configureScope { screen = it.screen?.substringAfterLast('.') }
337+
captureStrategy?.onScreenshotRecorded { _ ->
338+
val observer = options.sessionReplay.frameObserver
339+
if (observer != null) {
340+
val bitmap = BitmapFactory.decodeFile(screenshot.absolutePath)
341+
if (bitmap != null) {
342+
try {
343+
val hint = Hint()
344+
hint.set(TypeCheckHint.REPLAY_FRAME_BITMAP, bitmap)
345+
observer.onMaskedFrameCaptured(hint, frameTimestamp, screen)
346+
} catch (e: Throwable) {
347+
options.logger.log(ERROR, "Error in ReplayFrameObserver", e)
348+
bitmap.recycle()
349+
}
350+
}
351+
}
352+
addFrame(screenshot, frameTimestamp, screen)
353+
}
318354
checkCanRecord()
319355
}
320356

‎sentry-android-replay/src/test/java/io/sentry/android/replay/ReplayIntegrationTest.kt‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ import io.sentry.SentryEvent
1818
import io.sentry.SentryIntegrationPackageStorage
1919
import io.sentry.SentryOptions
2020
import io.sentry.SentryReplayEvent.ReplayType
21+
import io.sentry.SentryReplayOptions
22+
import io.sentry.TypeCheckHint
2123
import io.sentry.android.replay.ReplayCache.Companion.ONGOING_SEGMENT
2224
import io.sentry.android.replay.ReplayCache.Companion.SEGMENT_KEY_BIT_RATE
2325
import io.sentry.android.replay.ReplayCache.Companion.SEGMENT_KEY_FRAME_RATE
@@ -63,6 +65,7 @@ import org.mockito.kotlin.anyOrNull
6365
import org.mockito.kotlin.argThat
6466
import org.mockito.kotlin.check
6567
import org.mockito.kotlin.doAnswer
68+
import org.mockito.kotlin.doReturn
6669
import org.mockito.kotlin.eq
6770
import org.mockito.kotlin.mock
6871
import org.mockito.kotlin.never
@@ -969,6 +972,106 @@ class ReplayIntegrationTest {
969972
assertFalse(replay.isDebugMaskingOverlayEnabled)
970973
}
971974

975+
@Test
976+
fun `snapshot observer is invoked with bitmap and metadata`() {
977+
var callbackInvoked = false
978+
var receivedTimestamp = 0L
979+
var receivedScreen: String? = null
980+
var receivedBitmap: Bitmap? = null
981+
982+
val captureStrategy =
983+
mock<CaptureStrategy> {
984+
doAnswer {
985+
((it.arguments[1] as ReplayCache.(frameTimestamp: Long) -> Unit)).invoke(
986+
fixture.replayCache,
987+
1720693523997,
988+
)
989+
}
990+
.whenever(mock)
991+
.onScreenshotRecorded(anyOrNull<Bitmap>(), any())
992+
}
993+
val replay = fixture.getSut(context, replayCaptureStrategyProvider = { captureStrategy })
994+
995+
fixture.scopes.configureScope { it.screen = "MainActivity" }
996+
replay.register(fixture.scopes, fixture.options)
997+
replay.start()
998+
999+
fixture.options.sessionReplay.frameObserver =
1000+
SentryReplayOptions.ReplayFrameObserver { hint, frameTimestamp, screenName ->
1001+
callbackInvoked = true
1002+
receivedTimestamp = frameTimestamp
1003+
receivedScreen = screenName
1004+
receivedBitmap = hint.getAs(TypeCheckHint.REPLAY_FRAME_BITMAP, Bitmap::class.java)
1005+
}
1006+
1007+
val copyBitmap = mock<Bitmap>()
1008+
val sourceBitmap =
1009+
mock<Bitmap> {
1010+
on { config } doReturn ARGB_8888
1011+
on { copy(any(), any()) } doReturn copyBitmap
1012+
}
1013+
replay.onScreenshotRecorded(sourceBitmap)
1014+
1015+
assertTrue(callbackInvoked)
1016+
assertEquals(1720693523997, receivedTimestamp)
1017+
assertEquals("MainActivity", receivedScreen)
1018+
assertEquals(copyBitmap, receivedBitmap)
1019+
}
1020+
1021+
@Test
1022+
fun `snapshot observer exception does not prevent frame storage`() {
1023+
val captureStrategy =
1024+
mock<CaptureStrategy> {
1025+
doAnswer {
1026+
((it.arguments[1] as ReplayCache.(frameTimestamp: Long) -> Unit)).invoke(
1027+
fixture.replayCache,
1028+
1720693523997,
1029+
)
1030+
}
1031+
.whenever(mock)
1032+
.onScreenshotRecorded(anyOrNull<Bitmap>(), any())
1033+
}
1034+
val replay = fixture.getSut(context, replayCaptureStrategyProvider = { captureStrategy })
1035+
1036+
replay.register(fixture.scopes, fixture.options)
1037+
replay.start()
1038+
1039+
fixture.options.sessionReplay.frameObserver =
1040+
SentryReplayOptions.ReplayFrameObserver { _, _, _ -> throw RuntimeException("test") }
1041+
1042+
val sourceBitmap =
1043+
mock<Bitmap> {
1044+
on { config } doReturn ARGB_8888
1045+
on { copy(any(), any()) } doReturn mock<Bitmap>()
1046+
}
1047+
replay.onScreenshotRecorded(sourceBitmap)
1048+
1049+
verify(fixture.replayCache).addFrame(any<Bitmap>(), any(), anyOrNull())
1050+
}
1051+
1052+
@Test
1053+
fun `snapshot observer is not invoked when null`() {
1054+
val captureStrategy =
1055+
mock<CaptureStrategy> {
1056+
doAnswer {
1057+
((it.arguments[1] as ReplayCache.(frameTimestamp: Long) -> Unit)).invoke(
1058+
fixture.replayCache,
1059+
1720693523997,
1060+
)
1061+
}
1062+
.whenever(mock)
1063+
.onScreenshotRecorded(anyOrNull<Bitmap>(), any())
1064+
}
1065+
val replay = fixture.getSut(context, replayCaptureStrategyProvider = { captureStrategy })
1066+
1067+
replay.register(fixture.scopes, fixture.options)
1068+
replay.start()
1069+
1070+
replay.onScreenshotRecorded(mock<Bitmap>())
1071+
1072+
verify(fixture.replayCache).addFrame(any<Bitmap>(), any(), anyOrNull())
1073+
}
1074+
9721075
private fun getSessionCaptureStrategy(options: SentryOptions): SessionCaptureStrategy =
9731076
SessionCaptureStrategy(
9741077
options,

‎sentry/api/sentry.api‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4063,6 +4063,7 @@ public final class io/sentry/SentryReplayOptions : io/sentry/SentryMaskingOption
40634063
public fun addUnmaskViewClass (Ljava/lang/String;)V
40644064
public fun getBeforeErrorSampling ()Lio/sentry/SentryReplayOptions$BeforeErrorSamplingCallback;
40654065
public fun getErrorReplayDuration ()J
4066+
public fun getFrameObserver ()Lio/sentry/SentryReplayOptions$ReplayFrameObserver;
40664067
public fun getFrameRate ()I
40674068
public fun getNetworkDetailAllowUrls ()Ljava/util/List;
40684069
public fun getNetworkDetailDenyUrls ()Ljava/util/List;
@@ -4085,6 +4086,7 @@ public final class io/sentry/SentryReplayOptions : io/sentry/SentryMaskingOption
40854086
public fun setBeforeErrorSampling (Lio/sentry/SentryReplayOptions$BeforeErrorSamplingCallback;)V
40864087
public fun setCaptureSurfaceViews (Z)V
40874088
public fun setDebug (Z)V
4089+
public fun setFrameObserver (Lio/sentry/SentryReplayOptions$ReplayFrameObserver;)V
40884090
public fun setMaskAllImages (Z)V
40894091
public fun setMaskAllText (Z)V
40904092
public fun setNetworkCaptureBodies (Z)V
@@ -4105,6 +4107,10 @@ public abstract interface class io/sentry/SentryReplayOptions$BeforeErrorSamplin
41054107
public abstract fun execute (Lio/sentry/SentryEvent;Lio/sentry/Hint;)Z
41064108
}
41074109

4110+
public abstract interface class io/sentry/SentryReplayOptions$ReplayFrameObserver {
4111+
public abstract fun onMaskedFrameCaptured (Lio/sentry/Hint;JLjava/lang/String;)V
4112+
}
4113+
41084114
public final class io/sentry/SentryReplayOptions$SentryReplayQuality : java/lang/Enum {
41094115
public static final field HIGH Lio/sentry/SentryReplayOptions$SentryReplayQuality;
41104116
public static final field LOW Lio/sentry/SentryReplayOptions$SentryReplayQuality;
@@ -4651,6 +4657,7 @@ public final class io/sentry/TypeCheckHint {
46514657
public static final field OKHTTP_RESPONSE Ljava/lang/String;
46524658
public static final field OPEN_FEIGN_REQUEST Ljava/lang/String;
46534659
public static final field OPEN_FEIGN_RESPONSE Ljava/lang/String;
4660+
public static final field REPLAY_FRAME_BITMAP Ljava/lang/String;
46544661
public static final field SENTRY_DART_SDK_NAME Ljava/lang/String;
46554662
public static final field SENTRY_DOTNET_SDK_NAME Ljava/lang/String;
46564663
public static final field SENTRY_EVENT_DROP_REASON Ljava/lang/String;

0 commit comments

Comments
 (0)