Add DINOv2 dense feature visualization sample (ViT-S/14 on CompiledModel GPU)#223
Open
john-rocky wants to merge 2 commits into
Open
Add DINOv2 dense feature visualization sample (ViT-S/14 on CompiledModel GPU)#223john-rocky wants to merge 2 commits into
john-rocky wants to merge 2 commits into
Conversation
…del GPU) Self-supervised DINOv2 ViT-S/14 (Meta, Apache-2.0) running fully on the LiteRT CompiledModel GPU; a top-3 PCA of the dense patch tokens -> RGB visualizes what the backbone sees. New feature_visualization/dinov2_kotlin_gpu sample. Proven ViT recipes: fused-qkv -> 4D attention (C12), SafeLayerNorm (1/64 pre-square for DINOv2 massive activations), LayerScale baked into projections, tanh-GELU (sigmoid approx drifts to feature corr 0.968 over 12 blocks; tanh -> 0.99999). pos_embed baked at a fixed 448 grid at creation -> no GATHER_ND. Pixel 8a: 864/864 nodes, 1 partition, ~8 ms; device fp16 features vs fp32 corr 0.996. conversion/validate_dinov2.py = CompiledModel-API parity. Host PCA (power iteration) validated vs SVD. Model: litert-community/DINOv2-ViT-S14-LiteRT.
Rework the UI layer to the canonical Compose + MVVM sample shape while leaving the inference path byte-identical. - MainActivity: thin ComponentActivity Compose host (viewModels factory, collectAsStateWithLifecycle, PickVisualMedia gallery launcher). - MainViewModel: owns Dinov2Features, loads the fp16 model from assets in a try/catch (surfacing the throwable as an inline error), and confines every model call to a single-thread dispatcher. The old Activity's sideBySide() render math is moved here verbatim; only the GPU inference is timed, as before. - UiState / ImageUtils / view (Dinov2Screen, Theme, Color) and res/values (strings, themes, colors) added; no hardcoded UI strings. - Build files adopt the version catalog and Compose (AGP 8.9.1 / Kotlin 2.2.21 / compileSdk 36); LiteRT stays pinned at 2.1.5. The existing assets provisioning via download_model.gradle is preserved. Dinov2Features.kt (the CompiledModel + host PCA helper) is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
New
feature_visualization/dinov2_kotlin_gpusample: the self-supervised DINOv2 ViT-S/14 backbone (Meta, Apache-2.0) running fully on the LiteRTCompiledModelGPU, with a dense-feature visualization — a top-3 PCA of the patch tokens mapped to RGB. Semantically similar patches (object parts vs background) share a color, so the object pops out with no labels or segmentation.com.google.ai.edge.examples.dinov2): pick a photo → image and its DINOv2 feature-PCA side by side. The PCA (power iteration on the 384×384 covariance) is host-side.conversion/:build_dinov2.py(litert-torch) +validate_dinov2.py(CompiledModel Python API parity) + README.download_model.gradle, not committed).GPU re-authoring (proven ViT recipes)
[1,heads,N,d](≤4D), manualsoftmax(qkᵀ/√d)·v.ls1/ls2) baked into the following projections.GATHER_ND).Verification (Pixel 8a, LiteRT 2.1.5, fp16)
validate_dinov2.pyPASS).:app:assembleDebugbuilds.Companion LiteRT-Models entry: john-rocky/LiteRT-Models#58.