Mediapipe face mesh sop aligning with camera/render

[vertgo]

So I would love to use the mediapipe face mesh sop to overlay rendered face masks a la instagram with my face, but I cannot get the camera to align (I am using the face mesh TOP and face mesh SOP and then compositing them to see if I can get the mesh to align with the 2D points in the TOP). In the mediapipe documentation, they say the points should be considered the distance in centimeters, with a camera at the origin. I see you rotated the points in the geo, which makes sense, as the point recognition doesn't seem to be rotation invariant, but I don't understand the translation and scale of the points. Any ideas on how to align the render?

[vertgo]

I think one might need to solve the face geometry from the 3D to the 2D video points? There seems to be some kind of implication on the mediapipe page that the face pose transformation matrix is part of the face mesh module, but when asked about it, google's team just refers to someone else who solved it using cv2 solvepnp:
google-ai-edge/mediapipe#1457

[chungbwc]

I think there is no perfect solution for the Python binding at this moment. The face geometry details and the face effect modules are only available for the iOS and Android versions (https://google.github.io/mediapipe/solutions/face_mesh.html). It can be possible to use Script TOP and the 2D face mesh (x, y) to build a mask using the contour of the face mesh for such purpose. And I am aware of some readers playing around with this in IG. I have not tried out the Camera component in TouchDesigner and whether by putting in the accurate vertical FOV 63 degree as documented in MediaPipe may help or not. Thanks.