Skip to content

Commit 8f8967b

Browse files
Update C++ SDK docs (#106)
* Document changes to C++ SDK. * Add rotate with flip example. * Update static rest api doc * Add link from Python api page to C++ rotaion info * Minor updates to REST-API.html. * Update rotation and flip discussion. * Add semicolon. Co-authored-by: jrobble <jrobble@mitre.org>
1 parent ca008c9 commit 8f8967b

13 files changed

Lines changed: 5331 additions & 2974 deletions

File tree

docs/docs/CPP-Batch-Component-API.md

Lines changed: 115 additions & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -326,117 +326,84 @@ string SampleComponent::GetDetectionType() {
326326

327327
#### GetDetections(MPFImageJob …)
328328

329-
Used to detect objects in an image file. The MPFImageJob structure contains the data_uri specifying the location of the image file.
329+
Used to detect objects in an image file. The MPFImageJob structure contains
330+
the data_uri specifying the location of the image file.
330331

331-
Currently, the data_uri is always a local file path. For example, "/opt/mpf/share/remote-media/test-file.jpg". This is because all media is copied to the OpenMPF server before the job is executed.
332+
Currently, the data_uri is always a local file path. For example, "/opt/mpf/share/remote-media/test-file.jpg".
333+
This is because all media is copied to the OpenMPF server before the job is executed.
332334

333335
* Function Definition:
334336
```c++
335-
MPFDetectionError GetDetections(const MPFImageJob &job, vector<MPFImageLocation> &locations)
337+
std::vector<MPFImageLocation> GetDetections(const MPFImageJob &job);
336338
```
337339
338340
* Parameters:
339341
340342
| Parameter | Data Type | Description |
341343
|---|---|---|
342-
| job | `const MPFImageJob &` | Structure containing details about the work to be performed. See [`MPFImageJob`](#mpfimagejob) |
343-
| locations | `vector<MPFImageLocation> &` | The [`MPFImageLocation`](#mpfimagelocation) data for each detected object. |
344+
| job | `const MPFImageJob&` | Structure containing details about the work to be performed. See [`MPFImageJob`](#mpfimagejob) |
344345
345-
* Returns: `MPFDetectionError`
346+
* Returns: (`std::vector<MPFImageLocation>`) The [`MPFImageLocation`](#mpfimagelocation) data for each detected object.
346347
347-
* Example:
348-
349-
```c++
350-
MPFDetectionError SampleComponent::GetDetections(const MPFImageJob &job, vector<MPFImageLocation> &locations) {
351-
// Parse job
352-
// Generate image locations
353-
return MPF_DETECTION_SUCCESS;
354-
}
355-
```
356348
357349
#### GetDetections(MPFVideoJob …)
358350
359-
Used to detect objects in a video file. Prior to being sent to the component, videos are split into logical "segments" of video data and each segment (containing a range of frames) is assigned to a different job. Components are not guaranteed to receive requests in any order. For example, the first request processed by a component might receive a request for frames 300-399 of a Video A, while the next request may cover frames 900-999 of a Video B.
351+
Used to detect objects in a video file. Prior to being sent to the component, videos are split into logical "segments"
352+
of video data and each segment (containing a range of frames) is assigned to a different job. Components are not
353+
guaranteed to receive requests in any order. For example, the first request processed by a component might receive
354+
a request for frames 300-399 of a Video A, while the next request may cover frames 900-999 of a Video B.
360355
361356
* Function Definition:
362357
```c++
363-
MPFDetectionError getDetections(const MPFVideoJob &job, vector<MPFVideoTrack> tracks);
358+
std::vector<MPFVideoTrack> GetDetections(const MPFVideoJob &job);
364359
```
365360

366361
* Parameters:
367362

368363
| Parameter | Data Type | Description |
369364
|---|---|---|
370-
| job | `const MPFVideoJob &` | Structure containing details about the work to be performed. See [`MPFVideoJob`](#mpfvideojob) |
371-
| tracks | `vector<MPFVideoTrack> &` | The [`MPFVideoTrack`](#mpfvideotrack) data for each detected object. |
365+
| job | `const MPFVideoJob&` | Structure containing details about the work to be performed. See [`MPFVideoJob`](#mpfvideojob) |
372366

373-
* Returns: `MPFDetectionError`
367+
* Returns: (`std::vector<MPFVideoTrack>`) The [`MPFVideoTrack`](#mpfvideotrack) data for each detected object.
374368

375-
* Example:
376-
377-
```c++
378-
MPFDetectionError SampleComponent::GetDetections(const MPFAudioJob &job, vector<MPFAudioTrack> &tracks) {
379-
// Parse job
380-
// Generate tracks
381-
return MPF_DETECTION_SUCCESS;
382-
}
383-
```
384369

385370
#### GetDetections(MPFAudioJob …)
386371

387-
Used to detect objects in an audio file. Currently, audio files are not logically segmented, so a job will contain the entirety of the audio file.
372+
Used to detect objects in an audio file. Currently, audio files are not logically segmented, so a job will contain
373+
the entirety of the audio file.
388374

389375
* Function Definition:
390376
```c++
391-
MPFDetectionError GetDetections(const MPFAudioJob &job, vector<MPFAudioTrack> &tracks)
377+
std::vector<MPFAudioTrack> GetDetections(const MPFAudioJob &job);
392378
```
393379
394380
* Parameters:
395381
396382
| Parameter | Data Type | Description |
397383
|---|---|---|
398384
| job | `const MPFAudioJob &` | Structure containing details about the work to be performed. See [`MPFAudioJob`](#mpfaudiojob) |
399-
| tracks | `vector<MPFAudioTrack> &` | The [`MPFAudioTrack`](#mpfaudiotrack) data for each detected object |
400385
401-
* Returns: `MPFDetectionError`
402-
403-
* Example:
386+
* Returns: (`std::vector<MPFAudioTrack>`) The [`MPFAudioTrack`](#mpfaudiotrack) data for each detected object.
404387
405-
```c++
406-
MPFDetectionError GetDetections(const MPFAudioJob &job, vector<MPFAudioTrack> &tracks) {
407-
// Parse job
408-
// Generate tracks
409-
return MPF_DETECTION_SUCCESS;
410-
}
411-
```
412388
413389
#### GetDetections(MPFGenericJob …)
414390
415-
Used to detect objects in files that aren't video, image, or audio files. Such files are of the UNKNOWN type and handled generically. These files are not logically segmented, so a job will contain the entirety of the file.
391+
Used to detect objects in files that aren't video, image, or audio files. Such files are of the UNKNOWN type and
392+
handled generically. These files are not logically segmented, so a job will contain the entirety of the file.
416393
417394
* Function Definition:
418395
```c++
419-
MPFDetectionError GetDetections(const MPFGenericJob &job, vector<MPFGenericTrack> &tracks)
396+
std::vector<MPFGenericTrack> GetDetections(const MPFGenericJob &job);
420397
```
421398

422399
* Parameters:
423400

424401
| Parameter | Data Type | Description |
425402
|---|---|---|
426403
| job | `const MPFGenericJob &` | Structure containing details about the work to be performed. See [`MPFGenericJob`](#mpfgenericjob) |
427-
| tracks | `vector<MPFGenericTrack> &` | The [`MPFGenericTrack`](#mpfgenerictrack) data for each detected object |
428404

429-
* Returns: `MPFDetectionError`
405+
* Returns: (`std::vector<MPFGenericTrack>`) The [`MPFGenericTrack`](#mpfgenerictrack) data for each detected object.
430406

431-
* Example:
432-
433-
```c++
434-
MPFDetectionError GetDetections(const MPFGenericJob &job, vector<MPFGenericTrack> &tracks) {
435-
// Parse job
436-
// Generate tracks
437-
return MPF_DETECTION_SUCCESS;
438-
}
439-
```
440407

441408
### Detection Job Data Structures
442409

@@ -825,46 +792,90 @@ MPFImageLocation(
825792
| width | `int` | The width of the detected object. |
826793
| height | `int` | The height of the detected object. |
827794
| confidence | `float` | Represents the "quality" of the detection. The range depends on the detection algorithm. 0.0 is lowest quality. Higher values are higher quality. Using a standard range of [0.0 - 1.0] is advised. If the component is unable to supply a confidence value, it should return -1.0. |
828-
| detection_properties | `Properties &` | Optional additional information about the detected object. There is no restriction on the keys or the number of entries that can be added to the detection_properties map. For best practice, keys should be in all CAPS. See note about `ROTATION` below. |
795+
| detection_properties | `Properties &` | Optional additional information about the detected object. There is no restriction on the keys or the number of entries that can be added to the detection_properties map. For best practice, keys should be in all CAPS. See the [section](#rotation-flip-info) for `ROTATION` and `HORIZONTAL_FLIP` below, |
796+
797+
* Example:
798+
799+
A component that performs generic object classification can add an entry to `detection_properties` where the key is `CLASSIFICATION` and the value is the type of object detected.
800+
801+
<pre><code class="text" style="color:black">
802+
MPFImageLocation {
803+
x_left_upper = 0, y_left_upper = 0, width = 100, height = 50, confidence = 1.0,
804+
{ {"CLASSIFICATION", "backpack"} }
805+
}
806+
</code></pre>
807+
808+
<span id="rotation-flip-info"></span>
809+
##### Rotation and Horizontal Flip
829810

830811
When the `detection_properties` map contains a `ROTATION` key, it should be a floating point value in the interval
831812
`[0.0, 360.0)` indicating the orientation of the detection in degrees in the counter-clockwise direction.
832813
In order to view the detection in the upright orientation, it must be rotated the given number of degrees in the
833-
clockwise direction. When the `ROTATION` key is present, `x_left_upper` and `y_left_upper` indicate the top left of
834-
the correctly oriented detection. Similarly, `width` and `height` indicate the dimensions of the correctly oriented
835-
detection.
814+
clockwise direction.
836815

837-
* Example:
816+
The `detection_properties` map can also contain a `HORIZONTAL_FLIP` property that will either be `"true"` or `"false"`.
817+
The `detection_properties` map may have both `HORIZONTAL_FLIP` and `ROTATION` keys.
838818

839-
![Lenna 90 CCW with Markup](img/lenna_90_ccw_markup.png "Lenna 90 degrees CCW with Markup")
819+
The Workflow Manager performs the following algorithm to draw the bounding box when generating markup:
840820

821+
<ol>
822+
<li style="color:red">
823+
Draw the rectangle ignoring rotation and flip.
824+
</li>
841825

842-
In the above image with markup, the cyan dot in the bottom-left corner of the bounding box represents the top-left corner of the detection region when correctly oriented.
826+
<li style="color:blue">
827+
Rotate the rectangle counter-clockwise the given number of degrees around its top left corner.
828+
</li>
843829

830+
<li style="color:green">
831+
If the rectangle is flipped, flip horizontally around the top left corner.
832+
</li>
833+
</ol>
844834

845-
```c++
846-
MPFImageLocation detection;
847-
detection.x_left_upper = 156;
848-
detection.y_left_upper = 339;
849-
detection.width = 194;
850-
detection.height = 243;
851-
detection.confidence = 1.0;
852-
detection.detection_properties["ROTATION"] = "90.0";
853-
```
835+
<img src="../img/flip-rotate-steps-example.png" style="border: 1px solid black" alt="flip and rotate example">
854836

855-
* Example:
837+
In the image above you can see the three steps required to properly draw a bounding box.
838+
Step 1 is drawn in red. Step 2 is drawn in blue. Step 3 and the final result is drawn in green.
839+
The detection for the image above is:
856840

857-
A component that performs generic object classification can add an entry to `detection_properties` where the key is `CLASSIFICATION` and the value is the type of object detected.
841+
<pre><code class="text" style="color:black">
842+
MPFImageLocation {
843+
x_left_upper = 210, y_left_upper = 189, width = 177, height = 41, confidence = 1.0,
844+
{ {"ROTATION", "15"}, { "HORIZONTAL_FLIP", "true" } }
845+
}
846+
</code></pre>
858847

859-
```c++
860-
MPFImageLocation detection;
861-
detection.x_left_upper = 0;
862-
detection.y_left_upper = 0;
863-
detection.width = 100;
864-
detection.height = 50;
865-
detection.confidence = 1.0;
866-
detection.detection_properties["CLASSIFICATION"] = "backpack";
867-
```
848+
Note that the `x_left_upper`, `y_left_upper`, `width`, and `height` values describe the red rectangle. The addition
849+
of the `ROTATION` property results in the blue rectangle, and the addition of the `HORIZONTAL_FLIP` property results
850+
in the green rectangle.
851+
852+
One way to think about the process is "draw the unrotated and unflipped rectangle, stick a pin in the upper left corner,
853+
and then rotate and flip around the pin".
854+
855+
###### Rotation-Only Example
856+
857+
![Lenna 90 CCW with Markup](img/lenna_90_ccw_markup.png "Lenna 90 degrees CCW with Markup")
858+
859+
The Workflow Manager generated the above image by performing markup on the original image with the following
860+
detection:
861+
862+
<pre><code class="text" style="color:black">
863+
MPFImageLocation {
864+
x_left_upper = 156, y_left_upper = 339, width = 194, height = 243, confidence = 1.0,
865+
{ {"ROTATION", "90.0"} }
866+
}
867+
</code></pre>
868+
869+
The markup process followed steps 1 and 2 in the previous section, skipping step 3 because there is no
870+
`HORIZONTAL_FLIP`.
871+
872+
In order to properly extract the detection region from the original image, such as when generating an artifact, you
873+
would need to rotate the region in the above image 90 degrees clockwise around the cyan dot currently shown in the
874+
bottom-left corner so that the face is in the proper upright position.
875+
876+
When the rotation is properly corrected in this way, the cyan dot will appear in the top-left corner of the bounding
877+
box. That is why its position is described using the `x_left_upper`, and `y_left_upper` variables. They refer to the
878+
top-left corner of the correctly oriented region.
868879

869880
#### MPFVideoTrack
870881

@@ -950,15 +961,35 @@ MPFGenericTrack(
950961
| confidence | `float` | Represents the "quality" of the detection. The range depends on the detection algorithm. 0.0 is lowest quality. Higher values are higher quality. Using a standard range of [0.0 - 1.0] is advised. If the component is unable to supply a confidence value, it should return -1.0. |
951962
| detection_properties | `Properties &` | Optional additional information about the detection. There is no restriction on the keys or the number of entries that can be added to the detection_properties map. For best practice, keys should be in all CAPS. |
952963

964+
965+
### Exception Types
966+
967+
#### MPFDetectionException
968+
969+
Exception that should be thrown by the `GetDetections()` methods when an error occurs.
970+
The content of the `error_code` and `what()` members will appear in the JSON output object.
971+
972+
* Constructors:
973+
```c++
974+
MPFDetectionException(MPFDetectionError error_code, const std::string &what = "")
975+
MPFDetectionException(const std::string &what)
976+
```
977+
978+
| Member | Data Type | Description |
979+
|---|---|---|
980+
| error_code | `MPFDetectionError` | Specifies the error type. See [`MPFDetectionError`](#mpfdetectionerror).
981+
| what() | `const char*` | Textual description of the specific error. (Inherited from `std::exception`)
982+
983+
953984
### Enumeration Types
954985
955986
#### MPFDetectionError
956987
957-
Enum used to indicate the status of a `GetDetections` call. A component is not required to support all error types.
988+
Enum used to indicate the type of error that occurred in a `GetDetections()` method. It is used as a parameter to
989+
the `MPFDetectionException` constructor. A component is not required to support all error types.
958990
959991
| ENUM | Description |
960992
|---|---|
961-
| MPF_DETECTION_SUCCESS | The execution of any component function has completed normally with no errors. |
962993
| MPF_OTHER_DETECTION_ERROR_TYPE | The component function has failed for a reason that is not captured by any of the other error codes. |
963994
| MPF_DETECTION_NOT_INITIALIZED | The initialization of the component, or the initialization of any of its dependencies, has failed for any reason. |
964995
| MPF_UNRECOGNIZED_DATA_TYPE | The media data type received by a component is not one of the values contained in the MPFDetectionDataType enum. Note that this failure is normally caught by the Component Executable, before a job is passed to the component logic. |

docs/docs/Java-Batch-Component-API.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -903,7 +903,6 @@ Enum used to indicate the status of `getDetections` in a [`MPFComponentDetection
903903

904904
| ENUM | Description |
905905
|---|---|
906-
| MPF_DETECTION_SUCCESS | The execution of any component method has completed normally with no errors. |
907906
| MPF_OTHER_DETECTION_ERROR_TYPE | The component method has failed for a reason that is not captured by any of the other error codes. |
908907
| MPF_DETECTION_NOT_INITIALIZED | The initialization of the component, or the initialization of any of its dependencies, has failed for any reason. |
909908
| MPF_UNRECOGNIZED_DATA_TYPE | The media data type received by a component is not one of the values contained in the `MPFDataType` enum. Note that this failure is normally caught by the Component Executor before a job is passed to the component logic. |

docs/docs/Python-Batch-Component-API.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -512,6 +512,8 @@ def __init__(self, x_left_upper, y_left_upper, width, height, confidence=-1.0, d
512512
| confidence | `float` | Represents the "quality" of the detection. The range depends on the detection algorithm. 0.0 is lowest quality. Higher values are higher quality. Using a standard range of [0.0 - 1.0] is advised. If the component is unable to supply a confidence value, it should return -1.0. |
513513
| detection_properties | `dict[str, str]` | A dict with keys and values of type `str` containing optional additional information about the detected object. For best practice, keys should be in all CAPS. |
514514

515+
[See here for information about rotation and horizontal flipping.](CPP-Batch-Component-API.md#rotation-flip-info)
516+
515517
* Example:
516518

517519
A component that performs generic object classification can add an entry to `detection_properties` where the key is

0 commit comments

Comments
 (0)