You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This classifier is build upon the excellent pose estimator [**OpenPose**](https://github.com/CMU-Perceptual-Computing-Lab/openpose) from **CMU Perceptual Computing Lab**. A GUI has been developped to ease dataset creation and real-world testing.
9
+
The OpenHand application uses the excellent full-body pose estimator [**OpenPose**](https://github.com/CMU-Perceptual-Computing-Lab/openpose) from **CMU Perceptual Computing Lab**. This application eases dataset creation and real-time pose classification.
8
10
9
11
-[Installation](#installation)
10
12
-[Under the hood](#under-the-hood)
@@ -17,7 +19,7 @@ This classifier is build upon the excellent pose estimator [**OpenPose**](https:
17
19
18
20
## Installation
19
21
20
-
Make sure that [`Poetry`](https://poetry.eustace.io/) is installed for Python 3.7 and above on your system.
22
+
Ensure that [`Poetry`](https://poetry.eustace.io/) is installed for Python 3.7 and above on your system.
21
23
22
24
1. Git clone the repository - `git clone https://github.com/ArthurFDLR/OpenHand-App`
23
25
@@ -31,27 +33,27 @@ Even if **OpenHand classifier** can run without [**OpenPose**](https://github.co
31
33
32
34
5. Once the installation is completed, change the variable `OPENPOSE_PATH` ( [`.\openhand_app\__init__.py`](https://github.com/ArthurFDLR/OpenHand-Classifier/blob/master/openhand_app/__init__.py)) to the location of the OpenPose installation folder on your system.
33
35
34
-
_Note:_ TensorFlow 2.4.0 is installed by default (can be changed through `Poetry`). GPU support thus recquieres CUDA 11.0 which might conflict with **OpenPose**recquierement. However, classification models available in the application are relatively light. Modern CPUs will handle these models inference process flawlessly.
36
+
_Note:_ TensorFlow 2.4.0 is installed by default (can be changed through `Poetry`). GPU support thus requires CUDA 11.0, which might conflict with **OpenPose**requirements. However, classification models available in the application are relatively light. Modern CPUs will handle these models' inference process flawlessly.
35
37
36
38
## Under the hood
37
39
38
40
### Features extraction
39
41
40
42
The 21 hand keypoints (2D) used as input for this classifier are produced by OpenPose. The hand output format is as follow:
More information can be found [here](https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/output.md#face-and-hands). Please note that even if only hand keypoints are used, [OpenPose recquiered the whole body to be analysed](https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/standalone_face_or_hand_keypoint_detector.md)in order to generate hand informations. Furtheremore keypoints coordinates are given in the frame of reference of the image feeded to OpenPose. Thus, the coordinates have to be normalized.
45
-
I addition to x, y coordinates, the accuracy of detection of each keypoints is provided. From now on, the sum of these values will be simply refered as accuracy.
46
+
More information is available [here](https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/output.md#face-and-hands). Please note that even though OpenHand focus on hand keypoints, [OpenPose requires the whole body to be analyzed](https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/standalone_face_or_hand_keypoint_detector.md) to generate hand data. Furthermore, keypoints coordinates are given in the frame of reference of the image fed to OpenPose. Thus, the coordinates have to be normalized.
47
+
I addition to x, y coordinates, the accuracy of detection of each keypoints is provided.
46
48
47
49
### Keypoints normalization
48
50
49
-
OpenPose outputs have to be formated and normalized prior to the artificial neural network (ANN) training. Coordinates are normalized relatively to finger length and the center of gravity of the hand.
51
+
OpenPose outputs have to be formatted and normalized before classification analysis. Coordinates are normalized relative to finger length and the center of gravity of the hand.
50
52
51
53
***Scaling:** First, the length of each fingers - defined as a set of lines of the same color, see above - is calculated. The euclidian distances of all segments of a finger are sumed *- e.g.* <imgsrc="https://render.githubusercontent.com/render/math?math=Thumb\_length = \sum_{i=0}^{3} d(\boldsymbol{k_i}, \boldsymbol{k_{i%2B1}})">.
52
54
Then, every coordinates composing the hand are divided by the greater finger length.
53
55
54
-
***Centering:** Keypoints are centered relatively to the center of mass of the hand which in this case, is simply defined as <imgsrc="https://render.githubusercontent.com/render/math?math=(\bar{\boldsymbol{k^x}}, \bar{\boldsymbol{k^y}})">.
56
+
***Centering:** Keypoints are centered relative to the center of mass of the hand which, in this case, is simply defined as <imgsrc="https://render.githubusercontent.com/render/math?math=(\bar{\boldsymbol{k^x}}, \bar{\boldsymbol{k^y}})">.
Now that coordinates are normalized, the input data is flatten to be fed to the ANNs as a list of 42 values between -1.0 and 1.0: <imgsrc="https://render.githubusercontent.com/render/math?math=(k^x_0, k^y_0, k^x_1, k^y_1 \dots k^x_{20}, k^y_{20})">
### Dataset creation - [*11090 samples for 27 categories*](https://github.com/ArthurFDLR/OpenHand-Classifier/tree/master/Datasets)
82
82
83
-
The dataset is composed of several classes. A class is composed of two text files, one for each hand. The dataset is structured as follow:
83
+
The dataset is composed of several classes consisting of two text files, one for each hand. The dataset is structured as follow:
84
84
85
85
```
86
86
.\AppHandClassifier\Datasets
@@ -100,7 +100,12 @@ The dataset is composed of several classes. A class is composed of two text file
100
100
.
101
101
```
102
102
103
-
The first line of a *data.txt* files contains the caracteristics of the dataset: class label, hand identifier (0 for left hand, 1 for right hand) and the minimum accuracy of detection. To add comments, begin a line with *##*. A sample is (at leat) composed of 3 lines: a header giving the detection accuracy, x coordinates, y coordinates.
103
+
The first line of a *data.txt* file contains the set's characteristics:
104
+
- Class label
105
+
- Hand identifier (0 for the left hand, 1 for the right hand)
106
+
- The minimum accuracy of detection
107
+
108
+
To add comments, begin a line with *##*. A sample is (at least) composed of 3 lines: a header giving the detection accuracy, x coordinates, y coordinates.
104
109
105
110
<details><summary>Click to show examples - First lines of 'Super' set for right hand</summary>
106
111
<p>
@@ -128,13 +133,37 @@ Note that a training set of 150 samples per hand and per pose seems enough to yi
128
133
129
134
### Pose classifier models
130
135
131
-
Classification models available in the application are stored in [`.\Models`](https://github.com/ArthurFDLR/OpenHand-App/tree/master/Models). Each model folder contain two HDF5 file containing model's architecture and weights values. While both model share the same architecture, they are respectively trained to analyse right or left hands. In addition a text file `class.txt` provides labels associated to the one-hot encoded output.
136
+
Classification models available in the application are stored in [`.\Models`](https://github.com/ArthurFDLR/OpenHand-App/tree/master/Models). Each model sub-folder contain two HDF5 files containing model's architecture and weights values. While both model share the same architecture, they are respectively trained to analyse right or left hands. In addition a text file `class.txt` provides labels associated to the one-hot encoded output.
137
+
138
+
```
139
+
.\AppHandClassifier\Models
140
+
│
141
+
└───model_1
142
+
| class.txt
143
+
│ model_1_left.h5
144
+
| model_1_right.h5
145
+
│
146
+
└───model_2
147
+
| class.txt
148
+
│ model_2_left.h5
149
+
| model_2_right.h5
150
+
.
151
+
.
152
+
```
132
153
133
-
See [**OpenHand-Models** repository](https://github.com/ArthurFDLR/OpenHand-Models) for more details and design your own model.
154
+
See [**OpenHand-Models** repository](https://github.com/ArthurFDLR/OpenHand-Models) for more details about model creation.
134
155
135
156
## User guide
136
157
137
-
🚧 Under construction 🚧
158
+
### Real-time pose classification
159
+
160
+
The video feed of the selected camera is feeded to OpenPose at all time. The analysis result is displayed on the left size of the application. Then, you simply have to select one of the available model in the drop-down at the bottom of the hand-analysis window. Hand keypoints extracted from the video feed by OpenPose are automatically normalized and feeded to the classifier.
161
+
162
+
### Create and manipulate dataset
163
+
164
+
First, you either have to load or create a new set of samples for a specific label and hand side. To do so, respectivelly choose *Open (Ctrl+O)* or *Create new (Ctrl+N)* in *Dataset* of the menu bar. You have to specify the hand side, the label and the accuracy treshold of the newly created samples set. The accuracy treshold define the minimum accuracy of hand keypoints detection from OpenPose of any sample in the set. This accuracy is displayed on top of hand keypoints graphs.
165
+
166
+
Now that a set is loaded in the application, you can record new samples from your video feed or inspect the set and delete inadequate samples. When your done, save the set through *Dataset -> Save (Ctrl+S)*.
0 commit comments