Finish plank tutor learning path

Author: BmanClark

content/learning-paths/mobile-graphics-and-gaming/ai-plank-tutor/1-introduce-plank-tutor.md

@@ -1,5 +1,5 @@
 ---
-title: Introduction
+title: AI Plank Tutor
 weight: 2
 
 ### FIXED, DO NOT MODIFY
@@ -23,6 +23,8 @@ The finished app has two main visual areas:
 
 The app overlays a pose score and a short caption that matches the spoken coaching feedback.
 
+This Learning Path starts with a "shell" project with MediaPipe and camera integration mostly setup. If you wish to learn about that setup from an empty project, you could try another learning path, like [Build a Hands-Free Selfie Android Application with MediaPipe](https://learn.arm.com/learning-paths/mobile-graphics-and-gaming/build-android-selfie-app-using-mediapipe-multimodality/)
+
 ## App pipeline
 
 The app uses a small pipeline of on-device components:

content/learning-paths/mobile-graphics-and-gaming/ai-plank-tutor/2-camera-pose-landmarks.md

@@ -124,7 +124,7 @@ This is a callback from `PoseLandmarkerHelper` and sends the live landmarks onto
 
 ## Configure MediaPipe Pose Landmarker
 
-What happens between `detectPose()` and `onResults()`? Open `landmarker/PoseLandmarkerHelper.kt`.
+What happens between `detectPose()` and `onResults()`? Open `ui/landmarker/PoseLandmarkerHelper.kt`.
 
 The starter file already contains the MediaPipe imports, model path, confidence values, and the `LandmarkerListener` interface for the callbacks to `MainActivity`.
 

content/learning-paths/mobile-graphics-and-gaming/ai-plank-tutor/3-plank-score.md

@@ -0,0 +1,197 @@
+---
+title: Score the plank pose
+weight: 4
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Objective
+
+In this section, you will turn MediaPipe pose landmarks into a simple plank score.
+
+You will:
+
+- Calculate joint angles from pose landmarks.
+- Compare the learner's live angles with the provided instructor reference.
+- Convert the weighted angle differences into a score from 0 to 100.
+- Display the score in the app UI.
+
+The starter project already includes `PlankPoseData.kt`. This file contains the reference landmarks for the instructor plank image and a list of angle weights. The weights let the app make some joints more important than others when calculating the final score.
+
+## Review the scoring data
+
+Open `data/PlankPoseData.kt`.
+
+The file provides three pieces of data:
+
+- `POSE_NAME`, which is the name used later in the LLM prompt.
+- `referenceLandmarks`, which stores the instructor plank pose as MediaPipe landmarks.
+- `angleWeights`, which stores the relative importance of each measured angle.
+
+You do not need to edit this file. In a larger app, you would capture this data from an instructor image or video as a preprocessing step, and load for different poses as needed. In this Learning Path, it is hard-coded so the Android app can stay focused on live inference and feedback.
+
+## Extract joint angles
+
+Open `ui/landmarker/PoseScoreHelper.kt`.
+
+The file already contains landmark index constants and `KEY_JOINTS_COORDINATES`. Each entry in `KEY_JOINTS_COORDINATES` identifies three landmarks that form one angle. For example, an elbow angle uses shoulder, elbow, and wrist landmarks.
+
+Replace the TODO in `extractAnglesFrom()` with this code:
+
+```kotlin
+fun extractAnglesFrom(landmarks: List<NormalizedLandmark>): List<Double> {
+    val angles = KEY_JOINTS_COORDINATES.map { coords ->
+        calculateAngleFor(landmarks[coords[0]], landmarks[coords[1]], landmarks[coords[2]])
+    }.toMutableList()
+
+    val neck = midpoint(landmarks[IDX_L_SHOULDER], landmarks[IDX_R_SHOULDER])
+    val midHips = midpoint(landmarks[IDX_L_HIP], landmarks[IDX_R_HIP])
+    angles.add(calculateAngleFor(landmarks[IDX_NOSE], neck, midHips))
+
+    return angles
+}
+```
+
+The main list extracts wrist, elbow, shoulder, knee, ankle, and hip angles. The final angle estimates neck alignment by creating a midpoint between the shoulders and a midpoint between the hips, with the nose as the third landmark.
+
+Replace the TODO in `midpoint()` with this code for the neck joint helper function:
+
+```kotlin
+private fun midpoint(pointA: NormalizedLandmark, pointB: NormalizedLandmark) =
+    NormalizedLandmark.create(
+        (pointA.x() + pointB.x()) / 2,
+        (pointA.y() + pointB.y()) / 2,
+        (pointA.z() + pointB.z()) / 2
+    )
+```
+
+MediaPipe landmarks are normalized coordinates. The `x` and `y` values are relative to the input image, and `z` gives relative depth.
+
+## Calculate a 3D angle
+
+For each angle in `extractAnglesFrom()` it uses `calculateAngleFor()` - replace its TODO with this code:
+
+```kotlin
+private fun calculateAngleFor(
+    pointA: NormalizedLandmark,
+    pointB: NormalizedLandmark,
+    pointC: NormalizedLandmark
+): Double {
+    val bax = pointA.x() - pointB.x()
+    val bay = pointA.y() - pointB.y()
+    val baz = pointA.z() - pointB.z()
+
+    val bcx = pointC.x() - pointB.x()
+    val bcy = pointC.y() - pointB.y()
+    val bcz = pointC.z() - pointB.z()
+
+    val dotProduct = bax * bcx + bay * bcy + baz * bcz
+    val magnitudeBA = sqrt(bax.pow(2) + bay.pow(2) + baz.pow(2))
+    val magnitudeBC = sqrt(bcx.pow(2) + bcy.pow(2) + bcz.pow(2))
+    val cosine = (dotProduct / (magnitudeBA * magnitudeBC)).toDouble().coerceIn(-1.0, 1.0)
+
+    return Math.toDegrees(acos(cosine))
+}
+```
+
+This function calculates the angle at `pointB`. It treats `pointA - pointB` and `pointC - pointB` as two 3D vectors, then uses their dot product to calculate the angle between them.
+
+The `coerceIn(-1.0, 1.0)` call protects the `acos()` calculation from small floating-point rounding errors.
+
+## Calculate a weighted score
+
+Replace the TODO in `calculatePoseScore()` with this code:
+
+```kotlin
+fun calculatePoseScore(
+    correctAngles: List<Double>,
+    userAngles: List<Double>,
+    angleWeights: List<Float>,
+    maxDifference: Double = MAX_ANGLE_DIFFERENCE
+): Double {
+    require(correctAngles.isNotEmpty()) { "Correct angles cannot be empty." }
+    require(correctAngles.size == userAngles.size) { "Correct and user angle counts must match." }
+    require(correctAngles.size == angleWeights.size) { "Angle and weight counts must match." }
+
+    val weightedScoreSum = correctAngles.indices.sumOf { index ->
+        val difference = abs(correctAngles[index] - userAngles[index])
+        val normalizedScore = ((maxDifference - difference) / maxDifference)
+            .coerceIn(0.0, 1.0) * HIGHEST_SCORE
+        normalizedScore * angleWeights[index]
+    }
+
+    return weightedScoreSum / angleWeights.sum().toDouble()
+}
+```
+
+For each angle, the score starts at 100 and drops as the learner's angle differs from the reference angle. Differences at or above `MAX_ANGLE_DIFFERENCE` contribute zero for that angle.
+
+The final score is the weighted average across all measured angles.
+
+{{% notice Note %}}
+This score is a simple pose-matching signal for the demo app. It is not a clinical, safety, or fitness-quality assessment.
+{{% /notice %}}
+
+## Emit score data from the ViewModel
+
+Open `ui\viewmodels\MainViewModel.kt`.
+
+Replace the TODO inside the `userPoseResults` mapping block with this code:
+
+```kotlin
+val referenceAngles = extractAnglesFrom(PlankPoseData.referenceLandmarks)
+val userAngles = extractAnglesFrom(userLandmarks)
+val score = calculatePoseScore(
+    correctAngles = referenceAngles,
+    userAngles = userAngles,
+    angleWeights = PlankPoseData.angleWeights
+)
+
+UserPoseResult(referenceAngles, userAngles, score)
+```
+
+This code calculates angles for the instructor reference and the learner's live pose, then stores both angle lists with the score. The next page will reuse the angle lists to build the LLM prompt.
+
+Now replace the TODO in `userPoseScore` with this code:
+
+```kotlin
+val userPoseScore: Flow<String> =
+    sharedUserPoseResults.map { it.scoreVal.roundToInt().toString() }
+```
+
+The UI wants a rounded string score. The underlying `UserPoseResult` keeps the full `Double` value for later use.
+
+## Display the score
+
+Open `ui\MainActivity.kt`.
+
+The starter project keeps `collectAppState()` as a safe no-op so the app can run before scoring and speech are implemented. Replace that function with this version:
+
+```kotlin
+private suspend fun collectAppState() = coroutineScope {
+    launch {
+        mainViewModel.userPoseScore.collect {
+            score.text = getString(R.string.score) + it
+        }
+    }
+}
+```
+
+Add this import near the other coroutine imports:
+
+```kotlin
+import kotlinx.coroutines.coroutineScope
+```
+
+The existing `repeatOnLifecycle(Lifecycle.State.STARTED)` block already calls `collectAppState()`. Changing the function to `suspend` is valid because `repeatOnLifecycle` runs its block from a coroutine.
+
+Later sections will add more `launch { ... }` collectors inside this same `coroutineScope` block.
+
+## Run the app
+
+Build and run the app on your Android device.
+
+When you move in front of the camera, the score should update as MediaPipe detects your landmarks. The score will vary with camera angle and distance from the device, but mainly with how closely your body position matches the reference plank pose.
+
+At this point, the app can detect and score the plank pose. The next section converts the largest angle differences into a short text prompt for the local LLM.

content/learning-paths/mobile-graphics-and-gaming/ai-plank-tutor/4-data-to-prompt.md

@@ -0,0 +1,159 @@
+---
+title: Turn pose data into a prompt
+weight: 5
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Objective
+
+In this section, you will turn pose and score data into a compact text prompt for the local LLM.
+
+You will:
+
+- Compare the instructor and learner angle lists.
+- Sort the largest joint-angle differences.
+- Filter out small differences so the prompt stays short.
+- Format the remaining differences as plain text.
+- Expose the generated prompt from `MainViewModel`.
+
+The LLM will not receive images or raw landmarks. It will receive a short list of the largest pose differences, such as:
+
+```text
+I'm doing Plank pose. My joints have the following significant angle differences to the teacher's: Left hip flexion: -18, Right shoulder: 12
+```
+
+This keeps the prompt small and gives the model only the facts it needs to produce one coaching cue.
+
+## Format the largest angle differences
+
+Open `ui/landmarker/PoseScoreHelper.kt`.
+
+The file already contains `KEY_ANGLE_NAMES`, which maps each calculated angle to a human-readable joint name. The order of `KEY_ANGLE_NAMES` matches the order of the angles returned by `extractAnglesFrom()`.
+
+Replace the TODO in `angleDifference()` with this code:
+
+```kotlin
+fun angleDifference(
+    correctAngles: List<Double>,
+    userAngles: List<Double>,
+    filter: Double = -1.0,
+    maxEntries: Int = Int.MAX_VALUE
+): Map<String, String> {
+    require(correctAngles.isNotEmpty()) { "Correct angles cannot be empty." }
+    require(correctAngles.size == userAngles.size) { "Correct and user angle counts must match." }
+
+    return correctAngles.zip(userAngles)
+        .zip(KEY_ANGLE_NAMES) { (correct, user), name ->
+            val difference = correct - user
+            Triple(name, difference, "%.0f".format(difference))
+        }
+        .filter { (_, difference, _) -> filter <= 0 || abs(difference) >= filter }
+        .sortedByDescending { abs(it.second) }
+        .take(maxEntries)
+        .associate { (name, _, formatted) -> name to formatted }
+}
+```
+
+This function pairs each reference angle with the corresponding learner angle, calculates the difference, and sorts by the absolute size of that difference.
+
+The returned map uses the joint name as the key and a rounded string angle difference as the value. The string value is useful because the next function can insert it directly into the prompt.
+
+{{% notice Note %}}
+The sign of the difference is important. A positive value means the learner needs to straighten that joint compared with the reference. A negative value means the learner needs to bend it more.
+{{% /notice %}}
+
+## Generate the LLM prompt
+
+Replace the TODO in `generateLlmPromptFrom()` with this code:
+
+```kotlin
+fun generateLlmPromptFrom(
+    referenceAngles: List<Double>,
+    userAngles: List<Double>,
+    poseName: String
+): String {
+    val filteredAngles = angleDifference(
+        referenceAngles,
+        userAngles,
+        JOINT_DIFF_FILTER,
+        MAX_ENTRIES
+    )
+
+    if (filteredAngles.isEmpty()) {
+        return "I'm doing $poseName pose. My joint angles are close to the teacher's reference; give me one brief plank alignment cue."
+    }
+
+    val angleSummary = filteredAngles.entries.joinToString { (name, angleDiff) ->
+        "$name: $angleDiff"
+    }
+    return "I'm doing $poseName pose. My joints have the following significant angle differences to the teacher's: $angleSummary"
+}
+```
+
+This function calls `angleDifference()` with values that ensure it filters out small differences using `JOINT_DIFF_FILTER` and limits the prompt to `MAX_ENTRIES` joint differences.
+
+The fallback prompt handles the case where the learner is already close to the reference pose. The LLM still creates a useful instruction, but it is not forced to invent a large correction from tiny angle changes. In the full AI Yoga Tutor this fallback won't be hit as a different "praise" prompt is given when the score is high (greater than 70).
+
+## Emit prompts from the ViewModel
+
+Open `ui\viewmodels\MainViewModel.kt`.
+
+Replace the TODO in `userPosePrompt` with this code:
+
+```kotlin
+val userPosePrompt: Flow<Pair<String, Double>> =
+    sharedUserPoseResults
+        .filter { !_isSpeaking.value }
+        .map { userResult ->
+            generateLlmPromptFrom(
+                referenceAngles = userResult.referenceAngles,
+                userAngles = userResult.userAngles,
+                poseName = poseName
+            ) to userResult.scoreVal
+        }
+        .flowOn(Dispatchers.Default)
+```
+
+The prompt is paired with the score because fuller app versions might use both values when deciding what feedback is spoken. In this Learning Path, the LLM step uses the prompt text.
+
+The `_isSpeaking` filter prevents the app from generating more feedback while the previous correction is still being spoken. Speech is added later, but keeping the filter here makes the data flow ready for that final step.
+
+## Check the prompt in Logcat
+
+The local LLM is added in the next section. For now, add a temporary collector so you can see the prompt in Logcat.
+
+Open `MainActivity.kt` and update `collectAppState()` so it also collects `userPosePrompt`:
+
+```kotlin
+private suspend fun collectAppState() = coroutineScope {
+    launch {
+        mainViewModel.userPoseScore.collect {
+            score.text = getString(R.string.score) + it
+        }
+    }
+
+    launch {
+        mainViewModel.userPosePrompt.collect { prompt ->
+            Log.d(TAG, "Pose prompt: ${prompt.first}")
+        }
+    }
+}
+```
+
+This Logcat collector is only for checking this section. In the next section, you will replace it with code that sends the prompt to the local LLM.
+
+## Run the app
+
+Build and run the app on your Android device.
+
+Open Logcat and filter for `MainActivity`. As you move in front of the camera, you should see short prompts that describe the largest differences from the reference plank pose.
+
+The app now has the complete input side of the tutor pipeline:
+
+```text
+Camera frame -> pose landmarks -> angle score -> text prompt
+```
+
+In the next section, you will add Arm's AI Chat library and send this prompt to a local LLM.