Add HSEmotion facial emotion recognition sample (EfficientNet-B0 on CompiledModel GPU)#222
Open
john-rocky wants to merge 2 commits into
Open
Conversation
…ompiledModel GPU) HSEmotion (av-savchenko / EmotiEffLib, Apache-2.0): an EfficientNet-B0 fine-tuned on AffectNet, recognizing 8 emotions fully on the LiteRT CompiledModel GPU. New facial_emotion_recognition/hsemotion_kotlin_gpu sample. Two conversion hurdles: (1) the released weights are an old-timm pickle whose forward is broken under current timm -> lift the state dict into a fresh timm tf_efficientnet_b0. (2) the SqueezeExcite global mean over the 112x112 stem map overflows the fp16 accumulator on the delegate -> all-NaN output; replaced by a hierarchical avg_pool mean (fp16-safe). Pixel 8a: 342/342 nodes, 1 partition, ~2 ms; device fp16 top-1 == desktop fp32 (logits corr 0.99997). conversion/validate_hsemotion.py = CompiledModel-API parity. App uses android.media.FaceDetector to crop the face. Model: litert-community/HSEmotion-B0-LiteRT.
Rework the HSEmotion sample UI from a single Activity-with-Views layout into the canonical Compose + MVVM shape used by the sibling samples. This is a UI-layer-only change; the inference helper is untouched. - MainActivity is now a thin ComponentActivity that hosts the Compose UI and wires the gallery picker. - MainViewModel owns EmotionClassifier, loads it from assets, and confines model creation plus every classify call to a single-parallelism dispatcher (the helper reuses native I/O buffers). The face-classification timing math moves over verbatim from the old Activity. - UiState is an immutable snapshot carrying the input face bitmap, the ranked emotion predictions, the inference time, and any error text. - EmotionScreen renders the status header, image picker, the input face, and a Column of emotion-score rows. All user-visible strings live in strings.xml, including the score format string. - Model loading stays in assets via EmotionClassifier (unchanged); load failures now surface as an inline errorMessage. The build-time download_model.gradle fetch is retained. - Adopt the version-catalog build (AGP 8.9.1 / Kotlin 2.2.21 / compileSdk 36) matching the sibling samples. LiteRT stays pinned at 2.1.5. EmotionClassifier.kt (the inference helper) is byte-identical.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
New
facial_emotion_recognition/hsemotion_kotlin_gpusample: HSEmotion (EmotiEffLib, Apache-2.0) — an EfficientNet-B0 fine-tuned on AffectNet, recognizing the 8 emotions (anger, contempt, disgust, fear, happiness, neutral, sadness, surprise) fully on the LiteRTCompiledModelGPU.com.google.ai.edge.examples.hsemotion): pick a face photo → the app crops the face with the built-inandroid.media.FaceDetector→ emotion distribution. ~2 ms/inference.conversion/:build_hsemotion.py(litert-torch) +validate_hsemotion.py(CompiledModel Python API parity) + README.download_model.gradle, not committed).Two conversion hurdles
conv_s2d). The state dict is lifted into a fresh timmtf_efficientnet_b0(num_classes=8,classifier.0.*→classifier.*), which has a working forward — 358/360 tensors match by name+shape.x.mean((2,3))over the 112×112 stem map is a single fp16 reduction whose partial sum overflows 65504 → the GPU delegate emits an all-NaN output (it reduces in fp16 even for an fp32 graph; desktop fp16 CPU is exact). Replaced by a hierarchical mean — repeatedavg_pool2dover equal-size tiling windows (≤ 49 elements each) — mathematically identical, fp16-safe.Verification (Pixel 8a, LiteRT 2.1.5, fp16)
validate_hsemotion.pyPASS).:app:assembleDebugbuilds.Companion LiteRT-Models entry: john-rocky/LiteRT-Models#57.