Merge branch 'master' into hf-merge/s3-prefix

Author: brosenberg42

docs/docs/Object-Storage-Guide.md

@@ -3,33 +3,33 @@ Rights in Data-General Clause 52.227-14, Alt. IV (DEC 2007). Copyright 2022 The
 
 # Object Storage Overview
 
-By default, OpenMPF will write markup files, JSON output objects, and extracted artifacts to directories in 
-`$MPF_HOME/share`. For multi-node deployments, `$MPF_HOME/share` points to a directory on a network share. 
+By default, OpenMPF will write markup files, JSON output objects, and extracted artifacts to directories in
+`$MPF_HOME/share`. For multi-node deployments, `$MPF_HOME/share` points to a directory on a network share.
 Most often, the share is managed by the Network File System (NFS) protocol, although using NFS is not a requirement.
 
-Alternatively, OpenMPF supports writing these files to an object storage server. That may be desirable in cloud 
+Alternatively, OpenMPF supports writing these files to an object storage server. That may be desirable in cloud
 deployments to better support integration between systems, and/or to consolidate file storage as a cost-saving measure.
 
-When a file cannot be uploaded to the server, the Workflow Manager will fall back to storing it in `$MPF_HOME/share`. 
-If and when a failure occurs, the JSON output object will contain a descriptive message in the `jobWarnings` field. 
+When a file cannot be uploaded to the server, the Workflow Manager will fall back to storing it in `$MPF_HOME/share`.
+If and when a failure occurs, the JSON output object will contain a descriptive message in the `jobWarnings` field.
 If the job completes without other issues, the final status will be `COMPLETE_WITH_WARNINGS`.
 
 # Common Object Storage Properties
 
-The following system properties are common to the various types of object storage solutions that 
+The following system properties are common to the various types of object storage solutions that
 OpenMPF supports:
 
 - `http.object.storage.upload.retry.count`
-    - The number of times OpenMPF will attempt to upload an object to the storage server after the 
+    - The number of times OpenMPF will attempt to upload an object to the storage server after the
       first failed attempt.
     - When using S3, the AWS SDK's default retry strategy is used.
-    - When using NGINX, exponential back off is used between retry attempts. There is a 500ms 
+    - When using NGINX, exponential back off is used between retry attempts. There is a 500ms
       delay before the first retry. The delay doubles for each subsequent retry.
 
 
 # S3 Object Storage
-OpenMPF supports downloading media and uploading results to an S3 compatible 
-server such as Ceph or Minio. The use of S3 is controlled through the 
+OpenMPF supports downloading media and uploading results to an S3 compatible
+server such as Ceph or Minio. The use of S3 is controlled through the
 following job properties and system properties:
 
 - `S3_ACCESS_KEY` job property or `s3.access.key` system property
@@ -39,23 +39,23 @@ following job properties and system properties:
 - `S3_SECRET_KEY` job property or `s3.secret.key` system property
     - The secret key that will be used when downloading and uploading to S3.
 - `S3_SESSION_TOKEN` job property or `s3.session.token` system property
-    - Only required when the S3 bucket is configured to require a session key. 
+    - Only required when the S3 bucket is configured to require a session key.
       This generally occurs when multi-factor authentication is required.
     - OpenMPF does not handle generating the session key.
 - `S3_USE_VIRTUAL_HOST` job property or `s3.use.virtual.host` system property
-    - When false or not provided, 
-      [path-style requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access) 
+    - When false or not provided,
+      [path-style requests](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access)
       will be used.
-    - When true, 
-      [virtual hosted-style](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access) 
+    - When true,
+      [virtual hosted-style](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access)
       access will be used.
     - When true, `S3_HOST` must also be provided.
-    - The CNAME configuration described 
-      [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#VirtualHostingCustomURLs) 
+    - The CNAME configuration described
+      [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#VirtualHostingCustomURLs)
       is not supported.
 - `S3_HOST` job property or `s3.host` system property
     - The host of the S3 server without the bucket name.
-    - If `S3_RESULTS_BUCKET=https://bucket.s3.amazonaws.com`, `S3_HOST` should be 
+    - If `S3_RESULTS_BUCKET=https://bucket.s3.amazonaws.com`, `S3_HOST` should be
       set to `s3.amazonaws.com`
     - Only used when `S3_USE_VIRTUAL_HOST=true`.
 - `S3_RESULTS_BUCKET` job property or `s3.results.bucket` system property
@@ -71,18 +71,20 @@ following job properties and system properties:
     - If you want to run a job where some media is in S3 and some is hosted elsewhere,
       you can set `S3_UPLOAD_ONLY` to `true` as a media specific property on the media that is hosted elsewhere.
 - `S3_REGION` job property or `s3.region` system property
-    - The [S3 region](https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints) 
-      to use when accessing S3. For example: `us-east-1` 
+    - The [S3 region](https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints)
+      to use when accessing S3. For example: `us-east-1`
     - Some S3 compatible servers like Minio ignore this.
+- `S3_UPLOAD_OBJECT_KEY_PREFIX` job property or `s3.upload.object.key.prefix` system property
+    - Specifies a prefix to prepend to object keys when uploading to S3.
 
 
 # Custom NGINX HTTP Object Storage
 
-OpenMPF supports a custom NGINX object storage server solution. If you're interested, please contact us. 
+OpenMPF supports a custom NGINX object storage server solution. If you're interested, please contact us.
 We can make the server-side code available upon request.
 
-For those who choose to run their own custom NGINX object storage server, please configure OpenMPF by setting 
-the `http.object.storage.nginx.service.uri` property to the URI of the NGINX server. 
+For those who choose to run their own custom NGINX object storage server, please configure OpenMPF by setting
+the `http.object.storage.nginx.service.uri` property to the URI of the NGINX server.
 The following system properties are unique to the custom NGINX object storage solution:
 
 - `http.object.storage.nginx.service.uri`
@@ -96,7 +98,7 @@ The following system properties are unique to the custom NGINX object storage so
     - The chunk size, in bytes, that is used to upload objects to the storage server.
     - In general, the default value is sufficient.
 
-The NGINX object storage server will determine the sha256 hash for the file once it's been uploaded. 
+The NGINX object storage server will determine the sha256 hash for the file once it's been uploaded.
 It then uses that hash to name the file and returns the file URI to OpenMPF.
 
 

docs/site/Object-Storage-Guide/index.html

@@ -226,30 +226,30 @@
                 <p><strong>NOTICE:</strong> This software (or technical data) was produced for the U.S. Government under contract, and is subject to the
 Rights in Data-General Clause 52.227-14, Alt. IV (DEC 2007). Copyright 2022 The MITRE Corporation. All Rights Reserved.</p>
 <h1 id="object-storage-overview">Object Storage Overview</h1>
-<p>By default, OpenMPF will write markup files, JSON output objects, and extracted artifacts to directories in 
-<code>$MPF_HOME/share</code>. For multi-node deployments, <code>$MPF_HOME/share</code> points to a directory on a network share. 
+<p>By default, OpenMPF will write markup files, JSON output objects, and extracted artifacts to directories in
+<code>$MPF_HOME/share</code>. For multi-node deployments, <code>$MPF_HOME/share</code> points to a directory on a network share.
 Most often, the share is managed by the Network File System (NFS) protocol, although using NFS is not a requirement.</p>
-<p>Alternatively, OpenMPF supports writing these files to an object storage server. That may be desirable in cloud 
+<p>Alternatively, OpenMPF supports writing these files to an object storage server. That may be desirable in cloud
 deployments to better support integration between systems, and/or to consolidate file storage as a cost-saving measure.</p>
-<p>When a file cannot be uploaded to the server, the Workflow Manager will fall back to storing it in <code>$MPF_HOME/share</code>. 
-If and when a failure occurs, the JSON output object will contain a descriptive message in the <code>jobWarnings</code> field. 
+<p>When a file cannot be uploaded to the server, the Workflow Manager will fall back to storing it in <code>$MPF_HOME/share</code>.
+If and when a failure occurs, the JSON output object will contain a descriptive message in the <code>jobWarnings</code> field.
 If the job completes without other issues, the final status will be <code>COMPLETE_WITH_WARNINGS</code>.</p>
 <h1 id="common-object-storage-properties">Common Object Storage Properties</h1>
-<p>The following system properties are common to the various types of object storage solutions that 
+<p>The following system properties are common to the various types of object storage solutions that
 OpenMPF supports:</p>
 <ul>
 <li><code>http.object.storage.upload.retry.count</code><ul>
-<li>The number of times OpenMPF will attempt to upload an object to the storage server after the 
+<li>The number of times OpenMPF will attempt to upload an object to the storage server after the
   first failed attempt.</li>
 <li>When using S3, the AWS SDK's default retry strategy is used.</li>
-<li>When using NGINX, exponential back off is used between retry attempts. There is a 500ms 
+<li>When using NGINX, exponential back off is used between retry attempts. There is a 500ms
   delay before the first retry. The delay doubles for each subsequent retry.</li>
 </ul>
 </li>
 </ul>
 <h1 id="s3-object-storage">S3 Object Storage</h1>
-<p>OpenMPF supports downloading media and uploading results to an S3 compatible 
-server such as Ceph or Minio. The use of S3 is controlled through the 
+<p>OpenMPF supports downloading media and uploading results to an S3 compatible
+server such as Ceph or Minio. The use of S3 is controlled through the
 following job properties and system properties:</p>
 <ul>
 <li><code>S3_ACCESS_KEY</code> job property or <code>s3.access.key</code> system property<ul>
@@ -263,27 +263,27 @@ <h1 id="s3-object-storage">S3 Object Storage</h1>
 </ul>
 </li>
 <li><code>S3_SESSION_TOKEN</code> job property or <code>s3.session.token</code> system property<ul>
-<li>Only required when the S3 bucket is configured to require a session key. 
+<li>Only required when the S3 bucket is configured to require a session key.
   This generally occurs when multi-factor authentication is required.</li>
 <li>OpenMPF does not handle generating the session key.</li>
 </ul>
 </li>
 <li><code>S3_USE_VIRTUAL_HOST</code> job property or <code>s3.use.virtual.host</code> system property<ul>
-<li>When false or not provided, 
-  <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access">path-style requests</a> 
+<li>When false or not provided,
+  <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access">path-style requests</a>
   will be used.</li>
-<li>When true, 
-  <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access">virtual hosted-style</a> 
+<li>When true,
+  <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access">virtual hosted-style</a>
   access will be used.</li>
 <li>When true, <code>S3_HOST</code> must also be provided.</li>
-<li>The CNAME configuration described 
-  <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#VirtualHostingCustomURLs">here</a> 
+<li>The CNAME configuration described
+  <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#VirtualHostingCustomURLs">here</a>
   is not supported.</li>
 </ul>
 </li>
 <li><code>S3_HOST</code> job property or <code>s3.host</code> system property<ul>
 <li>The host of the S3 server without the bucket name.</li>
-<li>If <code>S3_RESULTS_BUCKET=https://bucket.s3.amazonaws.com</code>, <code>S3_HOST</code> should be 
+<li>If <code>S3_RESULTS_BUCKET=https://bucket.s3.amazonaws.com</code>, <code>S3_HOST</code> should be
   set to <code>s3.amazonaws.com</code></li>
 <li>Only used when <code>S3_USE_VIRTUAL_HOST=true</code>.</li>
 </ul>
@@ -305,17 +305,21 @@ <h1 id="s3-object-storage">S3 Object Storage</h1>
 </ul>
 </li>
 <li><code>S3_REGION</code> job property or <code>s3.region</code> system property<ul>
-<li>The <a href="https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints">S3 region</a> 
-  to use when accessing S3. For example: <code>us-east-1</code> </li>
+<li>The <a href="https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints">S3 region</a>
+  to use when accessing S3. For example: <code>us-east-1</code></li>
 <li>Some S3 compatible servers like Minio ignore this.</li>
 </ul>
 </li>
+<li><code>S3_UPLOAD_OBJECT_KEY_PREFIX</code> job property or <code>s3.upload.object.key.prefix</code> system property<ul>
+<li>Specifies a prefix to prepend to object keys when uploading to S3.</li>
+</ul>
+</li>
 </ul>
 <h1 id="custom-nginx-http-object-storage">Custom NGINX HTTP Object Storage</h1>
-<p>OpenMPF supports a custom NGINX object storage server solution. If you're interested, please contact us. 
+<p>OpenMPF supports a custom NGINX object storage server solution. If you're interested, please contact us.
 We can make the server-side code available upon request.</p>
-<p>For those who choose to run their own custom NGINX object storage server, please configure OpenMPF by setting 
-the <code>http.object.storage.nginx.service.uri</code> property to the URI of the NGINX server. 
+<p>For those who choose to run their own custom NGINX object storage server, please configure OpenMPF by setting
+the <code>http.object.storage.nginx.service.uri</code> property to the URI of the NGINX server.
 The following system properties are unique to the custom NGINX object storage solution:</p>
 <ul>
 <li><code>http.object.storage.nginx.service.uri</code><ul>
@@ -335,7 +339,7 @@ <h1 id="custom-nginx-http-object-storage">Custom NGINX HTTP Object Storage</h1>
 </ul>
 </li>
 </ul>
-<p>The NGINX object storage server will determine the sha256 hash for the file once it's been uploaded. 
+<p>The NGINX object storage server will determine the sha256 hash for the file once it's been uploaded.
 It then uses that hash to name the file and returns the file URI to OpenMPF.</p>
 
             </div>

docs/site/index.html

@@ -376,5 +376,5 @@ <h1 id="overview">Overview</h1>
 
 <!--
 MkDocs version : 0.17.5
-Build Date UTC : 2022-10-27 16:39:52
+Build Date UTC : 2022-12-15 13:38:58
 -->