Merge pull request #3470 from pareenaverma/review_branch

Tech review of AI Plank Tutor LP

Author: pareenaverma

content/learning-paths/mobile-graphics-and-gaming/ai-plank-tutor/1-introduce-plank-tutor.md

@@ -45,7 +45,7 @@ This keeps the LLM prompt small, reduces latency, and makes the behavior easier
 
 ## Clone the starter project
 
-Clone the Learning Path code examples repository:
+Clone the `PlankTutor` Learning Path code example repository. 
 
 ```console
 git clone https://gitlab.arm.com/learning-code-examples/code-examples.git
@@ -70,7 +70,7 @@ The starter project contains the app structure, layout, image asset, MediaPipe p
 
 If Android Studio prompts you to trust the project, accept the prompt.
 
-The starter app is intentionally incomplete, but it should sync successfully before you add code.
+The starter app is intentionally incomplete, but it should sync successfully before you add code. If the `ai-plank-tutor/android` directory is missing, check that you cloned the `PlankTutor` branch.
 
 ## Inspect the provided files
 
@@ -79,12 +79,16 @@ Start by looking at the files that are already provided for you.
 Open `app/build.gradle` and confirm that the Android, CameraX, lifecycle, and MediaPipe dependencies are already present.
 Arm's AI Chat dependency is not included yet. You will add it later, when you implement local LLM inference.
 
+Also note the `packaging { jniLibs { useLegacyPackaging = true } }` setting. The AI Chat library you add later uses native libraries, and this packaging setting lets the app load those libraries correctly.
+
 Open `app/src/main/AndroidManifest.xml` and confirm that the app requests camera access:
 
 ```xml
 <uses-permission android:name="android.permission.CAMERA" />
 ```
 
+The app package is `com.arm.demo.AIPlankTutor`. You will use that package name later when copying the LLM model into the app-specific external files directory with `adb`.
+
 Open `app/src/main/res/layout/activity_main.xml` and review the main UI. The layout already contains:
 
 - An `ImageView` for the instructor plank image.
@@ -94,6 +98,6 @@ Open `app/src/main/res/layout/activity_main.xml` and review the main UI. The lay
 
 Open `app/src/main/res/drawable/plank.jpg` and review the instructor reference image.
 
-Code is under the long path `app/src/main/java/com/arm/demo/AIPlankTutor`. Under that, open `data/PlankPoseData.kt` and note the hard-coded plank reference data. This file contains the instructor's reference landmarks and angle weights used by the scoring step. This was generated from the reference plank image in an offline step so it doesn't need any runtime compute.
+Code is under the long path `app/src/main/java/com/arm/demo/AIPlankTutor`. Copy this path exactly, including the capitalized `AIPlankTutor` package directory. Under that, open `data/PlankPoseData.kt` and note the hard-coded plank reference data. This file contains the instructor's reference landmarks and angle weights used by the scoring step. This was generated from the reference plank image in an offline step so it doesn't need any runtime compute.
 
-Android code starts from the `MainActivity.kt` file, and we will look at that in the next step.
+Android code starts from the `MainActivity.kt` file, and you will look at that in the next step.

content/learning-paths/mobile-graphics-and-gaming/ai-plank-tutor/2-camera-pose-landmarks.md

@@ -84,6 +84,8 @@ The analyzer uses `STRATEGY_KEEP_ONLY_LATEST` because pose detection should work
 
 The output image format is `RGBA_8888`, which makes the frame data easy to copy into a `Bitmap` before passing it to MediaPipe.
 
+The analyzer runs on a background executor with one worker. That keeps camera frame conversion off the UI thread and serializes calls into the landmarker helper, which is useful because live-stream inference is driven by timestamped frames.
+
 ## Send frames to the landmarker
 
 In `bindCameraUseCases()` we just set `ImageAnalysis` to call `detectPose()` for every analyzed frame. Now, replace the TODO in `detectPose()` with this code:
@@ -164,11 +166,13 @@ The app uses `RunningMode.LIVE_STREAM` because frames arrive continuously from t
 
 The app also requests one pose with `setNumPoses(1)`. This keeps the example focused on a single learner.
 
+The `Delegate.GPU` option asks MediaPipe to use the device GPU for the pose model. If initialization fails on a particular device, the error listener and Logcat message are the first places to check.
+
 ## Convert camera frames to MPImage
 
 When we call from `MainActivity`, it is with a CameraX `ImageProxy`, but the MediaPipe analyzer expects an `MPImage`.
 
-Replace the TODO in `detectLiveStream()` with this code to convert between the two and start `poseLandmaker`'s analysis:
+Replace the TODO in `detectLiveStream()` with this code to convert between the two and start the Pose Landmarker analysis:
 
 ```kotlin
 fun detectLiveStream(
@@ -216,6 +220,8 @@ The call to `imageProxy.use { ... }` closes the frame after its pixels are copie
 
 The matrix rotates the image using the camera frame metadata. For the front camera, it also mirrors the image so the detected pose matches the preview the learner sees.
 
+`SystemClock.uptimeMillis()` provides a monotonic timestamp for the frame. MediaPipe requires timestamps for video and live-stream inputs, and `detectAsync()` returns immediately; the result arrives later through the result listener configured above.
+
 ## Return the first detected pose
 
 Finally, replace the TODO in `returnLiveStreamResult()` with this code:
@@ -235,6 +241,8 @@ MediaPipe returns a list of detected poses. This app asks for one pose, so it fo
 
 Build and run the app on your Android device.
 
-When prompted, allow camera access. You should see the front camera preview in the right side of the app.
+When prompted, allow camera access. Expect to see the front camera preview in the right side of the app.
+
+The score will not update yet. At this point, the app is collecting pose landmarks and passing them to `MainViewModel`; the scoring logic is added in the next section.
 
-The score will not update yet. At this point, the app is collecting pose landmarks and passing them to `MainViewModel`; the scoring logic is added in the next section.
+If the preview opens but no landmarks are produced later, make sure your full body is visible in the frame and check Logcat for `PoseLandmarkerHelper`.

content/learning-paths/mobile-graphics-and-gaming/ai-plank-tutor/3-plank-score.md

@@ -68,6 +68,8 @@ private fun midpoint(pointA: NormalizedLandmark, pointB: NormalizedLandmark) =
 
 MediaPipe landmarks are normalized coordinates. The `x` and `y` values are relative to the input image, and `z` gives relative depth.
 
+While this demo compares normalized landmarks from a fixed instructor reference, the score is still fairly view-dependent. It works best when the learner is side-on to the camera, fully visible, and at a similar orientation to the reference image.
+
 ## Calculate a 3D angle
 
 For each angle in `extractAnglesFrom()` it uses `calculateAngleFor()` - replace its TODO with this code:
@@ -135,7 +137,7 @@ This score is a simple pose-matching signal for the demo app. It is not a clinic
 
 ## Emit score data from the ViewModel
 
-Open `ui\viewmodels\MainViewModel.kt`.
+Open `ui/viewmodels/MainViewModel.kt`.
 
 Replace the TODO inside the `userPoseResults` mapping block with this code:
 
@@ -164,7 +166,7 @@ The UI wants a rounded string score. The underlying `UserPoseResult` keeps the f
 
 ## Display the score
 
-Open `ui\MainActivity.kt`.
+Open `ui/MainActivity.kt`.
 
 The starter project keeps `collectAppState()` as a safe no-op so the app can run before scoring and speech are implemented. Replace that function with this version:
 
@@ -194,4 +196,6 @@ Build and run the app on your Android device.
 
 When you move in front of the camera, the score should update as MediaPipe detects your landmarks. The score will vary with camera angle and distance from the device, but mainly with how closely your body position matches the reference plank pose.
 
+If the score stays blank, verify that `onResults()` is receiving landmarks and that your body remains inside the camera frame. If the score jumps sharply, try a steadier side-on camera angle.
+
 At this point, the app can detect and score the plank pose. The next section converts the largest angle differences into a short text prompt for the local LLM.

content/learning-paths/mobile-graphics-and-gaming/ai-plank-tutor/4-data-to-prompt.md

@@ -26,6 +26,8 @@ I'm doing Plank pose. My joints have the following significant angle differences
 
 This keeps the prompt small and gives the model only the facts it needs to produce one coaching cue.
 
+This is the main design pattern in the app: the vision model turns camera frames into structured pose data, deterministic Kotlin code turns that data into a small set of facts, and the LLM takes these facts and returns appropriate advice.
+
 ## Format the largest angle differences
 
 Open `ui/landmarker/PoseScoreHelper.kt`.
@@ -61,7 +63,7 @@ This function pairs each reference angle with the corresponding learner angle, c
 The returned map uses the joint name as the key and a rounded string angle difference as the value. The string value is useful because the next function can insert it directly into the prompt.
 
 {{% notice Note %}}
-The sign of the difference is important. A positive value means the learner needs to straighten that joint compared with the reference. A negative value means the learner needs to bend it more.
+The sign of the difference is important. `difference = reference - learner`, so a positive value means the learner's angle is smaller than the reference and needs to straighten. A negative value means the learner's angle is larger than the reference and needs to bend more.
 {{% /notice %}}
 
 ## Generate the LLM prompt
@@ -98,7 +100,7 @@ The fallback prompt handles the case where the learner is already close to the r
 
 ## Emit prompts from the ViewModel
 
-Open `ui\viewmodels\MainViewModel.kt`.
+Open `ui/viewmodels/MainViewModel.kt`.
 
 Replace the TODO in `userPosePrompt` with this code:
 
@@ -120,6 +122,8 @@ The prompt is paired with the score because fuller app versions might use both v
 
 The `_isSpeaking` filter prevents the app from generating more feedback while the previous correction is still being spoken. Speech is added later, but keeping the filter here makes the data flow ready for that final step.
 
+The `flowOn(Dispatchers.Default)` call keeps prompt formatting off the main thread. The work is small, but keeping camera, scoring, prompt generation, LLM inference, and speech on clear execution paths makes the pipeline easier to tune.
+
 ## Check the prompt in Logcat
 
 The local LLM is added in the next section. For now, add a temporary collector so you can see the prompt in Logcat.
@@ -148,12 +152,14 @@ This Logcat collector is only for checking this section. In the next section, yo
 
 Build and run the app on your Android device.
 
-Open Logcat and filter for `MainActivity`. As you move in front of the camera, you should see short prompts that describe the largest differences from the reference plank pose.
+Open Logcat and filter for `MainActivity`. As you move in front of the camera, expect short prompts that describe the largest differences from the reference plank pose.
+
+This Logcat check is the validation step for the prompt stage. 
 
 The app now has the complete input side of the tutor pipeline:
 
 ```text
 Camera frame -> pose landmarks -> angle score -> text prompt
 ```
 
-In the next section, you will add Arm's AI Chat library and send this prompt to a local LLM.
+In the next section, you will add Arm's AI Chat library and send this prompt to a local LLM.

content/learning-paths/mobile-graphics-and-gaming/ai-plank-tutor/5-llm-ai-chat.md

@@ -18,6 +18,7 @@ You will:
 - Load the model and set the tutor system prompt.
 - Send the pose prompt from the previous section to the local LLM.
 - Convert streamed LLM tokens into complete `Sentence` objects.
+- Release the native inference engine when the ViewModel is cleared.
 
 At the end of this section, the app will produce short text coaching corrections and show them as captions. Speech output is added in the next section.
 
@@ -35,6 +36,8 @@ Sync the project with Gradle.
 
 The AI Chat library provides the Android inference API used by this project and by the [Arm AI Chat app](https://play.google.com/store/apps/details?id=com.arm.aichat&hl=en_GB). It uses `llama.cpp` for GGUF inference, and `llama.cpp` integrates Arm [KleidiAI](https://developer.arm.com/ai/kleidi-libraries) kernels. Q4_0 GGUF models work particularly well with these kernels and can get strong acceleration on phones with [SME2](https://www.arm.com/technologies/sme2), SVE2, and Neon support.
 
+The starter project already has `mavenCentral()` in `settings.gradle` and native library packaging configured in `app/build.gradle`, so adding the dependency is the only Gradle change needed here.
+
 ## Add the model file to the device
 
 The app expects a Q4_0 GGUF model file named:
@@ -43,27 +46,46 @@ The app expects a Q4_0 GGUF model file named:
 Phi-4-mini-instruct-Q4_0.gguf
 ```
 
-Download [microsoft_Phi-4-mini-instruct-Q4_0.gguf](https://huggingface.co/bartowski/microsoft_Phi-4-mini-instruct-GGUF/blob/main/microsoft_Phi-4-mini-instruct-Q4_0.gguf) from Hugging Face. The model is 3.8B parameters, and around 2.2GB. Remove the "microsoft_" off the front of the name.
+Download [microsoft_Phi-4-mini-instruct-Q4_0.gguf](https://huggingface.co/bartowski/microsoft_Phi-4-mini-instruct-GGUF/blob/main/microsoft_Phi-4-mini-instruct-Q4_0.gguf) from Hugging Face. The model is 3.8B parameters, and about 2.3 GB. Rename the downloaded file by removing the leading `microsoft_` prefix.
+
+Keep at least 5 GB free on the device. The app imports the model from external app storage and then copies it into internal app storage the first time it loads, so the device temporarily needs room for both copies.
 
 The provided `LlmModelStore.kt` helper looks for the model in a few import locations. The most useful location while developing is the app-specific external files directory:
 
 ```text
 /sdcard/Android/data/com.arm.demo.AIPlankTutor/files/llm/Phi-4-mini-instruct-Q4_0.gguf
 ```
 
-With your device connected over USB, create the directory:
+With your device connected over USB, first confirm that `adb` can see the phone:
+
+```console
+adb devices
+```
+
+Expect a device listed as `device`:
+
+```output
+List of devices attached
+<device-id>    device
+```
+
+If the device is listed as `unauthorized`, unlock the phone and accept the USB debugging prompt.
+
+Next, create the app-specific external directory. This is the import location that `LlmModelStore` checks before copying the model into internal app storage:
 
 ```console
 adb shell mkdir -p /sdcard/Android/data/com.arm.demo.AIPlankTutor/files/llm
 ```
 
-Then push the model file:
+Then push the model file from the directory where you downloaded and renamed it:
 
 ```console
 adb push Phi-4-mini-instruct-Q4_0.gguf /sdcard/Android/data/com.arm.demo.AIPlankTutor/files/llm/
 ```
 
-When the app first loads the model, `LlmModelStore` copies it into the app's internal files directory. Later runs use the internal copy.
+The push can take a few minutes because the GGUF file is large. Do not disconnect the device while the transfer is running.
+
+When the app first loads the model, `LlmModelStore` checks that the file is readable and at least 2 GB, then copies it into the app's internal files directory. Later runs use the internal copy.
 
 {{% notice Note %}}
 If the model cannot be found, check Logcat for `LlmModelStore`. The error message lists every model path that was checked.
@@ -100,7 +122,7 @@ private val llmViewModel: LlmViewModel by viewModels {
 }
 ```
 
-`AiChat.getInferenceEngine()` creates the inference engine used by the ViewModel. `LlmModelStore` is the helper that finds or imports the GGUF model file. Now we need to make the `LlmViewModel` accept those parameters.
+`AiChat.getInferenceEngine()` creates a native-backed inference engine used by the ViewModel.`LlmModelStore` is the helper that finds or imports the GGUF model file. Now make the `LlmViewModel` accept those parameters.
 
 ## Add AI Chat to LlmViewModel
 
@@ -155,6 +177,8 @@ private suspend fun loadModelAndSystemPrompt() {
 
 You can read `TUTOR_SYSTEM_PROMPT` at the bottom of the file. The system prompt tells the model to act as a concise yoga teacher, use the joint-angle facts, and return one short correction.
 
+`loadModel()` parses the GGUF file and allocates the model in memory, so it can take noticeable time on the first run. The system prompt is set once after the model loads so each later user prompt can stay focused on the live pose differences.
+
 ## Send a prompt to AI Chat
 
 Replace the TODO in `sendUserPrompt()` with this code:
@@ -226,6 +250,19 @@ val sentences: Flow<Sentence> = _tokens
 
 The ViewModel now exposes a `Flow<Sentence>` instead of raw tokens. Page 6 will collect this Flow and send each sentence to Android text-to-speech.
 
+## Release the inference engine
+
+The AI Chat engine owns native resources. Add this cleanup function inside `LlmViewModel`:
+
+```kotlin
+override fun onCleared() {
+    inferenceEngine.destroy()
+    super.onCleared()
+}
+```
+
+Android calls `onCleared()` when the ViewModel is no longer used, which gives the app a clear place to release the native inference engine.
+
 ## Send pose prompts to the LLM
 
 Return to `MainActivity.kt`.
@@ -252,7 +289,7 @@ launch {
 }
 ```
 
-This prevents the app from generating a new prompt for every camera frame while the LLM is already responding.
+This prevents the app from generating a new prompt for every camera frame while the LLM is already responding. On-device LLM generation is much slower than camera frame delivery, so this backpressure step is what keeps the feedback loop understandable.
 
 Finally, add a temporary sentence collector so you can show the generated correction in the caption area and check it in Logcat:
 
@@ -270,10 +307,10 @@ The next section will replace this direct caption update with `SpeechManager`, s
 
 ## Run the app
 
-Build and run the app on your Android device. You should see the tutor's advice now at the bottom of the screen.
+Build and run the app on your Android device. Expect the tutor's advice at the bottom of the screen.
 
 To see what's happening behind the scenes, open Logcat and filter for `LlmViewModel` or `MainActivity`. The app should load the model, send pose prompts, and print short tutor corrections. 
 
-The first run can take longer because the model file may be copied into the app's internal storage and loaded into memory. Later runs should start faster.
+The first run can take longer because the model file may be copied into the app's internal storage and loaded into memory. Later runs should start faster. If model loading fails, also filter Logcat for `LlmModelStore`; it prints the exact import paths it checked.
 
 At this point, the app can generate local text feedback. The next section uses Android `TextToSpeech` to speak each completed sentence and show it as a caption.

content/learning-paths/mobile-graphics-and-gaming/ai-plank-tutor/6-android-tts.md

@@ -20,6 +20,8 @@ You will:
 
 At the end of this section, the app will speak each complete correction and show the same text as a caption while it is being spoken.
 
+Android `TextToSpeech` uses the TTS engine and voices installed on the device. The exact voice, language support, and quality can vary by phone, but the app code is the same.
+
 ## Initialize TextToSpeech
 
 Open `ui/SpeechManager.kt`.
@@ -86,6 +88,8 @@ fun launchSpeechGenerationAndPlaybackJob(speech: String) {
 
 Each phrase is queued with an utterance ID. `SpeechManager` keeps a small map from utterance ID to caption text so the app can show the caption only when that phrase starts speaking.
 
+The `speak()` call is asynchronous. A successful return value means Android accepted the phrase into the TTS queue, not that speech has already finished. The progress listener is what tells the app when playback starts and ends.
+
 ## Finish each utterance
 
 Replace the TODO in `finishUtterance()` with this code:
@@ -155,4 +159,6 @@ Build and run the app on your Android device.
 
 Move into a plank position in front of the camera. The score should update, the local LLM should generate a short correction, and Android TTS should speak it. The same correction should appear as a caption while it is being spoken.
 
+If captions appear but you do not hear speech, check the device media volume and confirm that a TTS voice is installed for the default language.
+
 At this point, the app has the full on-device pipeline: camera input, pose landmarks, joint-angle scoring, local LLM feedback, and spoken output.