|
| 1 | +--- |
| 2 | +title: Get pose landmarks from camera |
| 3 | +weight: 3 |
| 4 | + |
| 5 | +### FIXED, DO NOT MODIFY |
| 6 | +layout: learningpathall |
| 7 | +--- |
| 8 | + |
| 9 | +## Objective |
| 10 | + |
| 11 | +In this section, you will connect the Android camera to MediaPipe Pose Landmarker. |
| 12 | + |
| 13 | +You will: |
| 14 | + |
| 15 | +- Bind a CameraX preview to the app UI. |
| 16 | +- Add an `ImageAnalysis` use case for live camera frames. |
| 17 | +- Configure MediaPipe Pose Landmarker in live-stream mode. |
| 18 | +- Convert each CameraX `ImageProxy` into a MediaPipe `MPImage`. |
| 19 | +- Send the first detected pose landmark list to `MainViewModel`. |
| 20 | + |
| 21 | +At the end of this section, the app opens the front camera and passes live pose landmarks into the app. The score will still be incomplete until you add pose scoring in the next section. |
| 22 | + |
| 23 | +## Configure CameraX |
| 24 | + |
| 25 | +Open `ui/MainActivity.kt`. |
| 26 | + |
| 27 | +The starter project already requests camera permission and calls `setUpCamera()` from `onCreate()`. Replace the TODO in `setUpCamera()` with the following code: |
| 28 | + |
| 29 | +```kotlin |
| 30 | +private fun setUpCamera() { |
| 31 | + val cameraProviderFuture = ProcessCameraProvider.getInstance(this) |
| 32 | + cameraProviderFuture.addListener( |
| 33 | + { |
| 34 | + cameraProvider = cameraProviderFuture.get() |
| 35 | + bindCameraUseCases() |
| 36 | + }, Dispatchers.Main.asExecutor() |
| 37 | + ) |
| 38 | +} |
| 39 | +``` |
| 40 | + |
| 41 | +`ProcessCameraProvider` owns the camera use cases for the activity. When the provider is ready, the app stores it and calls `bindCameraUseCases()`. |
| 42 | + |
| 43 | +## Bind preview and image analysis |
| 44 | + |
| 45 | +The app needs two CameraX use cases: |
| 46 | + |
| 47 | +- `Preview`, which displays the camera feed in the `PreviewView`. |
| 48 | +- `ImageAnalysis`, which receives frames for pose detection. |
| 49 | + |
| 50 | +In `MainActivity.kt`, replace the TODO at the end of `bindCameraUseCases()` with this code: |
| 51 | + |
| 52 | +```kotlin |
| 53 | +val preview = Preview.Builder() |
| 54 | + .setResolutionSelector(resolutionSelector) |
| 55 | + .setTargetRotation(targetRotation) |
| 56 | + .build() |
| 57 | + |
| 58 | +val imageAnalyzer = ImageAnalysis.Builder() |
| 59 | + .setResolutionSelector(resolutionSelector) |
| 60 | + .setTargetRotation(targetRotation) |
| 61 | + .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) |
| 62 | + .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888) |
| 63 | + .build() |
| 64 | + .also { |
| 65 | + it.setAnalyzer( |
| 66 | + Dispatchers.Default.limitedParallelism(1).asExecutor() |
| 67 | + ) { image -> detectPose(image) } |
| 68 | + } |
| 69 | + |
| 70 | +cameraProvider.unbindAll() |
| 71 | + |
| 72 | +try { |
| 73 | + cameraProvider.bindToLifecycle( |
| 74 | + this, cameraSelector, preview, imageAnalyzer |
| 75 | + ) |
| 76 | + |
| 77 | + preview.surfaceProvider = cameraPreview.surfaceProvider |
| 78 | +} catch (exc: Exception) { |
| 79 | + Log.e(TAG, "Use case binding failed", exc) |
| 80 | +} |
| 81 | +``` |
| 82 | + |
| 83 | +The analyzer uses `STRATEGY_KEEP_ONLY_LATEST` because pose detection should work on the most recent frame. If the device is busy, old frames are dropped instead of queued. |
| 84 | + |
| 85 | +The output image format is `RGBA_8888`, which makes the frame data easy to copy into a `Bitmap` before passing it to MediaPipe. |
| 86 | + |
| 87 | +## Send frames to the landmarker |
| 88 | + |
| 89 | +In `bindCameraUseCases()` we just set `ImageAnalysis` to call `detectPose()` for every analyzed frame. Now, replace the TODO in `detectPose()` with this code: |
| 90 | + |
| 91 | +```kotlin |
| 92 | +private fun detectPose(imageProxy: ImageProxy) { |
| 93 | + if (!this::poseLandmarkerHelper.isInitialized || poseLandmarkerHelper.isClosed) { |
| 94 | + imageProxy.close() |
| 95 | + return |
| 96 | + } |
| 97 | + |
| 98 | + if (imageAnalysisEnabled) { |
| 99 | + poseLandmarkerHelper.detectLiveStream( |
| 100 | + imageProxy = imageProxy, |
| 101 | + isFrontCamera = true |
| 102 | + ) |
| 103 | + } else { |
| 104 | + imageProxy.close() |
| 105 | + } |
| 106 | +} |
| 107 | +``` |
| 108 | + |
| 109 | +This code checks that the MediaPipe helper is ready before using it. If the helper is not ready, it closes the `ImageProxy` immediately. |
| 110 | + |
| 111 | +{{% notice Note %}} |
| 112 | +Every `ImageProxy` from CameraX must be closed. In this app, `PoseLandmarkerHelper.detectLiveStream()` closes the image after copying its pixels. If the frame is skipped, `detectPose()` closes it directly. |
| 113 | +{{% /notice %}} |
| 114 | + |
| 115 | +Now replace the TODO in `onResults()` with this code: |
| 116 | + |
| 117 | +```kotlin |
| 118 | +override fun onResults(landmarks: List<NormalizedLandmark>?) { |
| 119 | + mainViewModel.handleUserPose(landmarks) |
| 120 | +} |
| 121 | +``` |
| 122 | + |
| 123 | +This is a callback from `PoseLandmarkerHelper` and sends the live landmarks onto the ViewModel. The next page will convert those landmarks into joint angles and a pose score. |
| 124 | + |
| 125 | +## Configure MediaPipe Pose Landmarker |
| 126 | + |
| 127 | +What happens between `detectPose()` and `onResults()`? Open `landmarker/PoseLandmarkerHelper.kt`. |
| 128 | + |
| 129 | +The starter file already contains the MediaPipe imports, model path, confidence values, and the `LandmarkerListener` interface for the callbacks to `MainActivity`. |
| 130 | + |
| 131 | +Replace the TODO in `setupPoseLandmarker()` with this code: |
| 132 | + |
| 133 | +```kotlin |
| 134 | +fun setupPoseLandmarker(context: Context) { |
| 135 | + try { |
| 136 | + val baseOptions = BaseOptions.builder() |
| 137 | + .setDelegate(Delegate.GPU) |
| 138 | + .setModelAssetPath(MODEL_PATH) |
| 139 | + .build() |
| 140 | + |
| 141 | + val options = PoseLandmarker.PoseLandmarkerOptions.builder() |
| 142 | + .setBaseOptions(baseOptions) |
| 143 | + .setNumPoses(1) |
| 144 | + .setMinPoseDetectionConfidence(MIN_POSE_DETECTION_CONFIDENCE) |
| 145 | + .setMinTrackingConfidence(MIN_POSE_TRACKING_CONFIDENCE) |
| 146 | + .setMinPosePresenceConfidence(MIN_POSE_PRESENCE_CONFIDENCE) |
| 147 | + .setRunningMode(RunningMode.LIVE_STREAM) |
| 148 | + .setResultListener(this::returnLiveStreamResult) |
| 149 | + .setErrorListener(this::returnLiveStreamError) |
| 150 | + .build() |
| 151 | + |
| 152 | + poseLandmarker = PoseLandmarker.createFromOptions(context, options) |
| 153 | + } catch (exception: IllegalStateException) { |
| 154 | + listener.onError("Pose Landmarker failed to initialize. See logs for details.") |
| 155 | + Log.e(TAG, "MediaPipe failed to load the pose landmarker", exception) |
| 156 | + } catch (exception: RuntimeException) { |
| 157 | + listener.onError("Pose Landmarker failed to initialize. See logs for details.") |
| 158 | + Log.e(TAG, "MediaPipe failed to create the pose landmarker", exception) |
| 159 | + } |
| 160 | +} |
| 161 | +``` |
| 162 | + |
| 163 | +The app uses `RunningMode.LIVE_STREAM` because frames arrive continuously from the camera. In this mode, MediaPipe returns results through callbacks instead of returning them directly from the detection call. |
| 164 | + |
| 165 | +The app also requests one pose with `setNumPoses(1)`. This keeps the example focused on a single learner. |
| 166 | + |
| 167 | +## Convert camera frames to MPImage |
| 168 | + |
| 169 | +When we call from `MainActivity`, it is with a CameraX `ImageProxy`, but the MediaPipe analyzer expects an `MPImage`. |
| 170 | + |
| 171 | +Replace the TODO in `detectLiveStream()` with this code to convert between the two and start `poseLandmaker`'s analysis: |
| 172 | + |
| 173 | +```kotlin |
| 174 | +fun detectLiveStream( |
| 175 | + imageProxy: ImageProxy, |
| 176 | + isFrontCamera: Boolean |
| 177 | +) { |
| 178 | + val frameTime = SystemClock.uptimeMillis() |
| 179 | + val imageWidth = imageProxy.width |
| 180 | + val imageHeight = imageProxy.height |
| 181 | + val rotationDegrees = imageProxy.imageInfo.rotationDegrees |
| 182 | + |
| 183 | + val bitmapBuffer = Bitmap.createBitmap( |
| 184 | + imageWidth, |
| 185 | + imageHeight, |
| 186 | + Bitmap.Config.ARGB_8888 |
| 187 | + ) |
| 188 | + |
| 189 | + imageProxy.use { image -> |
| 190 | + bitmapBuffer.copyPixelsFromBuffer(image.planes[0].buffer) |
| 191 | + } |
| 192 | + |
| 193 | + val matrix = Matrix().apply { |
| 194 | + postRotate(rotationDegrees.toFloat()) |
| 195 | + if (isFrontCamera) { |
| 196 | + postScale(-1f, 1f, imageWidth.toFloat(), imageHeight.toFloat()) |
| 197 | + } |
| 198 | + } |
| 199 | + |
| 200 | + val rotatedBitmap = Bitmap.createBitmap( |
| 201 | + bitmapBuffer, |
| 202 | + 0, |
| 203 | + 0, |
| 204 | + bitmapBuffer.width, |
| 205 | + bitmapBuffer.height, |
| 206 | + matrix, |
| 207 | + true |
| 208 | + ) |
| 209 | + |
| 210 | + val mpImage = BitmapImageBuilder(rotatedBitmap).build() |
| 211 | + poseLandmarker?.detectAsync(mpImage, frameTime) |
| 212 | +} |
| 213 | +``` |
| 214 | + |
| 215 | +The call to `imageProxy.use { ... }` closes the frame after its pixels are copied. |
| 216 | + |
| 217 | +The matrix rotates the image using the camera frame metadata. For the front camera, it also mirrors the image so the detected pose matches the preview the learner sees. |
| 218 | + |
| 219 | +## Return the first detected pose |
| 220 | + |
| 221 | +Finally, replace the TODO in `returnLiveStreamResult()` with this code: |
| 222 | + |
| 223 | +```kotlin |
| 224 | +private fun returnLiveStreamResult( |
| 225 | + result: PoseLandmarkerResult, |
| 226 | + @Suppress("UNUSED_PARAMETER") input: MPImage |
| 227 | +) { |
| 228 | + listener.onResults(result.landmarks().firstOrNull()) |
| 229 | +} |
| 230 | +``` |
| 231 | + |
| 232 | +MediaPipe returns a list of detected poses. This app asks for one pose, so it forwards only the first landmark list. |
| 233 | + |
| 234 | +## Run the app |
| 235 | + |
| 236 | +Build and run the app on your Android device. |
| 237 | + |
| 238 | +When prompted, allow camera access. You should see the front camera preview in the right side of the app. |
| 239 | + |
| 240 | +The score will not update yet. At this point, the app is collecting pose landmarks and passing them to `MainViewModel`; the scoring logic is added in the next section. |
0 commit comments