Skip to content

Add speak detection model#135

Merged
openminddev merged 7 commits into
mainfrom
add-speak-detection-model
Jul 9, 2026
Merged

Add speak detection model#135
openminddev merged 7 commits into
mainfrom
add-speak-detection-model

Conversation

@WenjinFu

@WenjinFu WenjinFu commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

This pull request introduces several major improvements to the face recognition and tracking system, focusing on performance, robustness, and new features. The most significant update is the addition of a threaded camera capture mechanism to avoid frame drops and increase FPS. It also adds a vision-only "who is speaking" scorer (VVAD) and a more accurate head pose estimation module, along with API changes to leverage these features.

Camera capture performance and reliability:

  • camera_reader.py: Refactored to use a background thread that continuously grabs the latest frame, ensuring read_frame() is non-blocking and always returns the freshest frame. This prevents frame drops and improves FPS by decoupling capture from processing. The threaded approach can be toggled via a parameter. [1] [2] [3] [4] [5] [6]

Vision-only speaking detection (VVAD):

  • face_tracker.py, http_api.py: Introduced a vvad attribute that allows the system to track and score which detected face is likely speaking, based only on visual cues. The face tracker now feeds mouth/chin crops to the VVAD model, and the HTTP API can bias enrollment toward the visually detected speaker. [1] [2] [3] [4] [5] [6] [7]

Improved head pose and frontality scoring:

  • head_pose.py: Added a new module for full head pose estimation (yaw, pitch, roll) using only five keypoints and optional chin landmark. This yields a more accurate "frontality" score, penalizing faces that look up or down, improving enrollment quality.

API and selection logic enhancements:

  • http_api.py: Added a /speaking endpoint and logic to bias the selection of the enrollment face toward the one visually detected as speaking, ensuring the correct person is enrolled during multi-person interactions. [1] [2] [3]

These changes collectively make the system more robust, responsive, and accurate in real-time face recognition and speaker identification scenarios.

@openminddev openminddev merged commit 7fda1bb into main Jul 9, 2026
7 checks passed
@openminddev openminddev deleted the add-speak-detection-model branch July 9, 2026 01:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants