Update release notes for R7.2.0. (#167)

Author: jrobble

docs/docs/Admin-Guide.md

@@ -1,4 +1,4 @@
-**NOTICE:** This software (or technical data) was produced for the U.S. Government under contract, and is subject to theCopyright 2023 The MITRE
+**NOTICE:** This software (or technical data) was produced for the U.S. Government under contract, and is subject to the
 Rights in Data-General Clause 52.227-14, Alt. IV (DEC 2007). Copyright 2023 The MITRE Corporation. All Rights Reserved.
 
 <div style="background-color:orange"><p style="color:white; padding:5px"><b>WARNING:</b> Please refer to the <a href="#user-configuration">User Configuration</a> section for changing the default user passwords.</p></div>

docs/docs/Component-Descriptor-Reference.md

@@ -123,10 +123,6 @@ Contains the following sub-fields:
   Required. Contains a brief description of the algorithm.
 
 
-* **detectionType:**
-  Required. Defines the type of entity associated with the algorithm. For example: `CLASS`, `FACE`, `MOTION`, `PERSON`, `SPEECH`, or `TEXT`.
-
-
 * **actionType:**
   Required. Defines the type of processing that the algorithm performs. Must be set to `DETECTION`.
 

docs/docs/Release-Notes.md

@@ -1,8 +1,156 @@
 **NOTICE:** This software (or technical data) was produced for the U.S. Government under contract, and is subject to the
 Rights in Data-General Clause 52.227-14, Alt. IV (DEC 2007). Copyright 2023 The MITRE Corporation. All Rights Reserved.
 
+# OpenMPF 7.2.x
+
+<h2>7.2.0: May 2023</h2>
+
+<h3>Documentation</h3>
+
+- Created a new [TiesDb Guide](TiesDb-Guide/index.html).
+- Updated the [Component Descriptor Reference](Component-Descriptor-Reference/index.html) with `outputChangedCounter`.
+- Updated the [REST API](REST-API/index.html) with a new `[POST] /rest/jobs/tiesdbrepost` endpoint.
+- Updated the REST API `[POST] /rest/jobs` response with `tiesDbCheckStatus` and `outputObjectUri`.
+
+<h3>TiesDb Re-Post</h3>
+
+- Added a new `[POST] /rest/jobs/tiesdbrepost` endpoint that accepts an array of job ids as an input and will attempt to
+  re-post the job assertions (records) to TiesDb for each one. 
+- Added a "TiesDb" column to the Job Status page. If there is a problem posting a record to the TiesDb server the column
+  will contain an "ERROR" button. Clicking on it will provide a description of the error and a button that can be used
+  to re-post the associated job records.
+
+<h3>TiesDb Checking</h3>
+
+- If the `TIES_DB_URL` job property or `ties.db.url` system property is set when submitting a job creation request, 
+  then the Workflow Manager will attempt to check TiesDb for existing job results before running the job again.
+- The Workflow Manager will attempt to use the most-recently-created job results, preferring jobs that completed without
+  errors or warnings, and preferring jobs that completed with warnings over completed with errors.
+- To prevent this check, set `SKIP_TIES_DB_CHECK=true`. That will force the job to run and attempt to post the new
+  job results to TiesDb.
+- When using TiesDb, we strongly recommend providing both the `MEDIA_HASH` and `MIME_TYPE` in the `media.metadata` map
+  in the job request. This will enable the Workflow Manager to skip media inspection. When using S3 object storage, this
+  means that the Workflow Manager will not need to download the media before checking TiesDb for existing job records.
+- The `[POST] /rest/jobs` response now contains a `tiesDbCheckStatus` and `outputObjectUri` field. `tiesDbCheckStatus`
+  will be set to one of the following values:
+    - `NOT_REQUESTED`
+    - `NO_TIES_DB_URL_IN_JOB`
+    - `MEDIA_HASHES_ABSENT`
+    - `MEDIA_MIME_TYPES_ABSENT`
+    - `NO_MATCH`
+    - `FOUND_MATCH`
+- When there is a `FOUND_MATCH`, the `outputObjectUri` will be set to the URI of the old TiesDb record if S3 copy is
+  not enabled.
+- By default, the `ties.db.s3.copy.enabled` system property is set to `true`. This means that the Workflow Manager will
+  attempt to copy all of the artifacts, markup, and derivative media associated with the job in TiesDb from the S3
+  locations associated with the old job to the new S3 location specified in the new job. A new JSON output object will
+  be generated. To disable this behavior set the system property, or `TIES_DB_S3_COPY_ENABLED`, to `false`. Then the
+  Workflow Manager will simply provide a link to the old JSON as the result of the new job.
+- If there is a problem copying between S3 locations, the "TiesDb" column to the Job Status page will show a
+  "COPY ERROR" button. Clicking on it will provide a description of the error.
+
+<h3>TiesDb Linked Media</h3>
+
+- Added support for the `LINKED_MEDIA_HASH` `media.mediaProperty` in the job creation request. When specified,
+  the value of the `LINKED_MEDIA_HASH` will be used instead of the actual media hash when creating a record in TiesDb,
+  and also when looking for existing records in TiesDb.
+- This feature can be used to submit a transcoded (or thumbnail) version of an image to process instead of the source
+  image. For example, the source image may be in a format not supported by OpenMPF. In this case, the value of the
+  `LINKED_MEDIA_HASH` can be set to the source image, but the rest of the job creation request would specify
+  the `media.path` and `media.mediaMetadata` for the transcoded version of that image.
+
+<h3>Output Changed Counter</h3>
+
+- Added the `output.changed.counter` system property to the Workflow Manager and `outputChangedCounter` field to each
+  component's `descriptor.json`. These values are used when calculating the hash for a job when its record is posted to
+  TiesDb, and also when checking TiesDb for existing records when a new job is submitted.
+- If the Workflow Manager is updated for any reason that should invalidate pre-existing job results, such as a
+  change to the fields in the JSON output object, or significant improvements to track merging, for example, then the
+  value of `output.changed.counter` should be incremented by one. This will ensure that records in TiesDb will not be
+  used so that all future jobs will need to be (re)run at least once until the counter is incremented again.
+- The same is true for each component. If a component is updated for any reason that should invalidate
+  pre-existing job results, such as changes to input or output properties, or substantial improvements to the algorithm,
+  then the value of `outputChangedCounter` should be incremented by one.
+
+<h3>Changes to JSON Output Object</h3>
+
+- New JSON output objects will include `tiesDbSourceJobId` and `tiesDbSourceMediaPath` when the Workflow Manager can use
+  previous job results stored in TiesDB. Note that the Workflow Manager will not generate a new JSON output object
+  unless `S3_RESULTS_BUCKET` is set to a valid value, S3 access and secret keys are provided, and
+  `TIES_DB_S3_COPY_ENABLED=true`.
+
+<h3>ffprobe for Media Inspection</h3>
+
+- The Workflow Manager media inspection behavior now uses `ffprobe` with `-print_format json` to return more precise
+  `FPS` values for the `media.mediaMetadata` in the JSON output object. For example, the previous version of the
+  Workflow Manager would return `29.97`, where the new version will return `29.97002997002997`. In multi-hour-long
+  vidoes this can prevent cases where the last few frames were being ignored.
+- The previous version of the Workflow Manager was using both `ffmpeg` and OpenCV to determine the number of frames in
+  a video. We removed the OpenCV frame counter in this version because the `ffprobe` approach is more accurate.
+  The `ffprobe` command replaces the old `ffmpeg` command. 
+
+<h3>Web User Interface</h3>
+
+- Updated the Job Status page to be more efficient. Searching a database of hundreds of thousands of jobs takes a long
+  time. By limiting the search to one page of results at a time the UI is more responsive.
+- Removed timeout and bootout. The user session will no longer automatically end due to time out, or due to the same
+  user logging in from a different host or browser. These behaviors were deemed too disruptive by end users.
+- Updated the Job Status page to include a "TiesDb" column that reports TiesDb status, such as when posting records
+  to TiesDb and when retrieving existing records.
+
+<h3>Features</h3>
+
+- [[#1438](https://github.com/openmpf/openmpf/issues/1438)] Create a REST endpoint that will attempt to re-post to TiesDb
+- [[#1613](https://github.com/openmpf/openmpf/issues/1613)] Check TiesDb before running a job
+- [[#1650](https://github.com/openmpf/openmpf/issues/1650)] Create TiesDb records for thumbnail jobs under the parent media
+
+<h3>Updates</h3>
+
+- [[#1342](https://github.com/openmpf/openmpf/issues/1342)] Use ffprobe to get FPS during media inspection
+- [[#1564](https://github.com/openmpf/openmpf/issues/1564)] Use ffprobe's JSON output instead of regexes during media inspection
+- [[#1601](https://github.com/openmpf/openmpf/issues/1601)] Update the Workflow Manager jobs table to be more efficient
+- [[#1611](https://github.com/openmpf/openmpf/issues/1611)] Remove Workflow Manager timeout and bootout behavior
+
 # OpenMPF 7.1.x
 
+<h2>7.1.12: March 2023</h2>
+
+<h3>Bug Fixes</h3>
+
+- [[#1667](https://github.com/openmpf/openmpf/issues/1667)] Handle Webp files with extra data at the end that cause components to crash
+
+<h2>7.1.10: March 2023</h2>
+
+<h3>Updates</h3>
+
+- [[#1662](https://github.com/openmpf/openmpf/issues/1662)] Monitor StorageBackend
+
+<h2>7.1.9: February 2023</h2>
+
+<h3>Bug Fixes</h3>
+
+- [[#1675](https://github.com/openmpf/openmpf/issues/1675)] Prevent upgrade of cudnn in yolo server dockerfile
+
+<h2>7.1.8: February 2023</h2>
+
+<h3>Bug Fixes</h3>
+
+- [[#1649](https://github.com/openmpf/openmpf/issues/1649)] Install specific version of libcudnn8 in Docker build
+
+<h2>7.1.7: February 2023</h2>
+
+<h3>Updates</h3>
+
+- [[#1674](https://github.com/openmpf/openmpf/issues/1674)] Update `SPEAKER_ID` logic, set `LONG_SPEAKER_ID=0`
+
+<h2>7.1.5: January 2023</h2>
+
+<h3>Features</h3>
+
+- [[#1542](https://github.com/openmpf/openmpf/issues/1542)] Update Azure Speech Detection component to select transcription language based on feed-forward track
+- [[#1543](https://github.com/openmpf/openmpf/issues/1543)] Update audio transcoder to accept subsegments
+- [[#1605](https://github.com/openmpf/openmpf/issues/1605)] Update Azure Translation to use detected language from upstream
+
 <h2>7.1.1: December 2022</h2>
 
 <h3>Bug Fixes</h3>

docs/site/Admin-Guide/index.html

@@ -232,7 +232,7 @@
           <div role="main">
             <div class="section">
 
-                <p><strong>NOTICE:</strong> This software (or technical data) was produced for the U.S. Government under contract, and is subject to theCopyright 2023 The MITRE
+                <p><strong>NOTICE:</strong> This software (or technical data) was produced for the U.S. Government under contract, and is subject to the
 Rights in Data-General Clause 52.227-14, Alt. IV (DEC 2007). Copyright 2023 The MITRE Corporation. All Rights Reserved.</p>
 <div style="background-color:orange"><p style="color:white; padding:5px"><b>WARNING:</b> Please refer to the <a href="#user-configuration">User Configuration</a> section for changing the default user passwords.</p></div>
 

docs/site/Component-Descriptor-Reference/index.html

@@ -315,10 +315,6 @@ <h3>algorithm</h3>
   Required. Contains a brief description of the algorithm.</p>
 </li>
 <li>
-<p><strong>detectionType:</strong>
-  Required. Defines the type of entity associated with the algorithm. For example: <code>CLASS</code>, <code>FACE</code>, <code>MOTION</code>, <code>PERSON</code>, <code>SPEECH</code>, or <code>TEXT</code>.</p>
-</li>
-<li>
 <p><strong>actionType:</strong>
   Required. Defines the type of processing that the algorithm performs. Must be set to <code>DETECTION</code>.</p>
 </li>