Add speak detection model#135
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several major improvements to the face recognition and tracking system, focusing on performance, robustness, and new features. The most significant update is the addition of a threaded camera capture mechanism to avoid frame drops and increase FPS. It also adds a vision-only "who is speaking" scorer (VVAD) and a more accurate head pose estimation module, along with API changes to leverage these features.
Camera capture performance and reliability:
camera_reader.py: Refactored to use a background thread that continuously grabs the latest frame, ensuringread_frame()is non-blocking and always returns the freshest frame. This prevents frame drops and improves FPS by decoupling capture from processing. The threaded approach can be toggled via a parameter. [1] [2] [3] [4] [5] [6]Vision-only speaking detection (VVAD):
face_tracker.py,http_api.py: Introduced avvadattribute that allows the system to track and score which detected face is likely speaking, based only on visual cues. The face tracker now feeds mouth/chin crops to the VVAD model, and the HTTP API can bias enrollment toward the visually detected speaker. [1] [2] [3] [4] [5] [6] [7]Improved head pose and frontality scoring:
head_pose.py: Added a new module for full head pose estimation (yaw, pitch, roll) using only five keypoints and optional chin landmark. This yields a more accurate "frontality" score, penalizing faces that look up or down, improving enrollment quality.API and selection logic enhancements:
http_api.py: Added a/speakingendpoint and logic to bias the selection of the enrollment face toward the one visually detected as speaking, ensuring the correct person is enrolled during multi-person interactions. [1] [2] [3]These changes collectively make the system more robust, responsive, and accurate in real-time face recognition and speaker identification scenarios.