Skip to content

Commit 121e822

Browse files
authored
Clarify the difference between Docker and non-Docker behaviors (#130)
* Move old docs to "Additional Developer Resources". * Split "Media Segmentation Guide" into a separate page. * Update "Workflow Manager Architecture" doc.
1 parent 7032584 commit 121e822

72 files changed

Lines changed: 3003 additions & 2330 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
Welcome to the Open Media Processing Framework (OpenMPF) Web Site Project!
44

5-
## What is the OpenMPF?
5+
## What is OpenMPF?
66

77
OpenMPF provides a platform to perform content detection and extraction on bulk multimedia, enabling users to analyze, search, and share information through the extraction of objects, keywords, thumbnails, and other contextual data.
88

docs/docs/Admin-Guide.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
> **NOTICE:** This software (or technical data) was produced for the U.S. Government under contract, and is subject to the Rights in Data-General Clause 52.227-14, Alt. IV (DEC 2007).Copyright 2021 The MITRE Corporation. All Rights Reserved.> **IMPORTANT:** Please refer to the [User Configuration](#user-configuration) section for changing the default user passwords.> **NOTE:** This release contains basic support for processing video streams. Currently, the only way to make use of that functionality is through the REST API. Streaming jobs and services cannot be created or monitored through the web UI.# Web UIThe login procedure, as well as all of the pages accessible through the Workflow Manager sidebar, are the same for admin and non-admin users. Refer to the [User Guide](/User-Guide/index.html) for more information. The default account for an admin user has the username "admin" and password "mpfadm". We highly recommend changing the default username and password settings for any environment which is exposed on a network, especially production environments. The default settings are public knowledge, which could be a security risk. Please refer to the [User Configuration](#user-configuration) section below.This document will cover the additional functionality permitted to admin users through the Admin Console pages.## DashboardThe landing page for an admin user is the Job Status page:![Admin Landing Page](img/mpf-adm-landing.png "Admin Landing Page")The Job Status page displays a summary of the status for all jobs run by any user in the past. The current status and progress of any running job can be monitored from this view, which is updated automatically.## Node Configuration and StatusThis page provides a list of all of the services that are configured to run on the OpenMPF cluster, and enables an admin user to start, stop, or restart them on an individual basis. Only an admin user can perform these actions. If a non-admin user views this page, the "Action(s)" column is not displayed. This page also enables an admin user to edit the configuration for all nodes in the OpenMPF cluster. A non-admin user can only view the existing configuration.![Node and Process Status Page](img/mpf-adm-node.png "Node and Process Status Page")An admin user can add a node by using the "Add Node" button and selecting a node in the OpenMPF cluster from the drop-down list. You an also select to add all services at this time. A node and all if its configured services can be removed by clicking the trash can to the right of the node's hostname.An admin user can add services individually by selecting the node edit button at the bottom of the node. The number of service instances can be increased or decreased by using the drop-down. Click the "Submit" button to save the changes.Any node or service changes take effect immediately, no saving is required, except for adding services.When making changes, please be aware of the following: - It may take a minute for the configuration to take effect on the server. - If you remove an existing service from a node, any job that service is processing will be stopped, and you will need to resubmit that job. - If you create a new node, its configuration will not take effect until the OpenMPF software is properly installed and started on the associated host. - If you delete a node, you will need to manually turn off the hardware running that node (deleting a node does not shut down the machine).## Properties SettingsThis page allows an admin user to view and edit various OpenMPF properties:![Properties Settings Page](img/mpf-adm-property-settings.png "Properties Settings Page")An admin user can click inside of the "Value" field for any of the properties and type a new value. Doing so will change the color of the property to orange and display an orange icon to the right of the property name.Note that if the admin user types in the original value of the property, or clicks the "Reset" button, then it will return back to the normal coloration.WARNING: Changing the value of these properties can prevent the workflow manager from running after the web server is restarted. Also, no validation checks are performed on the user-provided values. Proceed with caution!At the bottom of the properties table is the "Save Properties" button. The number of modified properties is shown in parentheses. Clicking the button will make the necessary changes to the properties file on the file system, but the changes will not take effect until the workflow manager is restarted. The saved properties will be colored blue and a blue icon will be displayed to the right of the property name. Additionally, a notification will appear at the top of the page alerting all system users that a restart is required:![Properties Settings Page](img/mpf-adm-property-settings-change.png "Properties Settings Page")## Component RegistrationThis page allows an admin user to add and remove non-default components to and from the system:![Component Registration Page](img/mpf-adm-component.png "Component Registration Page")A component package takes the form of a tar.gz file. An admin user can either drag and drop the file onto the "Upload a new component" dropzone area or click the dropzone area to open a file browser and select the file that way. In either case, the component will begin to be uploaded to the system. If the admin user dragged and dropped the file onto the dropzone area then the upload progress will be shown in that area. Once uploaded, the workflow manager will automatically attempt to register the component. Notification messages will appear in the upper right side of the screen to indicate success or failure if an error occurs. The "Current Components" table will display the component status.![Component Registration Page](img/mpf-adm-component2.png "Component Registration Page")If for some reason the component package upload succeeded but the component registration failed then the admin user will be able to click the "Register" button again to try to another registration attempt. For example, the admin user may do this after reviewing the workflow manager logs and resolving any issues that prevented the component from successfully registering the first time. One reason may be that a component with the same name already exists on the system. Note that an error will also occur if the top-level directory of the component package, once extracted, already exists in the /opt/mpf/plugins directory on the system.Once registered, an admin user has the option to remove the component. This will unregister it and completely remove any configured services, as well as the uploaded file and its extracted contents, from the system. Also, the component algorithm as well as any actions, tasks, and pipelines specified in the component's descriptor file will be removed when the component is removed.WARNING: Any actions, tasks, or pipelines created through the Create Custom Pipelines page that make use of the algorithm, actions, or tasks specified in the descriptor file of the component being removed will also be removed. This is to prevent pipelines from not working properly once the component is removed.# User ConfigurationEvery time the Workflow Manager starts it will attempt to create accounts for the users listed in the `user.properties` file. At runtime this file is extracted to `$MPF_HOME/config` on the machine running the Workflow Manager. For every user listed in that file, the Workflow Manager will create that user account if a user with the same name doesn't already exists in the SQL database. By default, that file contains two entries, one for the "admin" user with the "mpfadm" password, and one for a non-admin "mpf" user with the "mpf123" password.We highly recommend modifying the `user.properties` file with your own user entries before attempting to start the Workflow Manager for the first time. This will ensure that the default user accounts are not created. The official way to deploy OpenMPF is to use the Docker container platform. If you are using Docker, please follow the instructions in the openmpf-docker [README](https://github.com/openmpf/openmpf-docker/blob/master/README.md#optional-configure-users) that explain how to use a `docker secret` for your custom `user.properties` file.
1+
**NOTICE:** This software (or technical data) was produced for the U.S. Government under contract, and is subject to theRights in Data-General Clause 52.227-14, Alt. IV (DEC 2007). Copyright 2021 The MITRE Corporation. All Rights Reserved.<div style="background-color:orange"><p style="color:white; padding:5px"><b>WARNING:</b> Please refer to the <a href="#user-configuration">User Configuration</a> section for changing the default user passwords.</p></div># Web UIThe login procedure, as well as all of the pages accessible through the Workflow Manager sidebar, are the same for admin and non-admin users. Refer to the [User Guide](/User-Guide/index.html) for more information. The default account for an admin user has the username "admin" and password "mpfadm". We highly recommend changing the default username and password settings for any environment which is exposed on a network, especially production environments. The default settings are public knowledge, which could be a security risk. Please refer to the [User Configuration](#user-configuration) section below.This document will cover the additional functionality permitted to admin users through the Admin Console pages.## DashboardThe landing page for an admin user is the Job Status page:![Admin Landing Page](img/mpf-adm-landing.png "Admin Landing Page")The Job Status page displays a summary of the status for all jobs run by any user in the past. The current status and progress of any running job can be monitored from this view, which is updated automatically.## Properties SettingsThis page allows an admin user to view and edit various OpenMPF properties:![Properties Settings Page](img/mpf-adm-property-settings.png "Properties Settings Page")An admin user can click inside of the "Value" field for any of the properties and type a new value. Doing so will change the color of the property to orange and display an orange icon to the right of the property name.Note that if the admin user types in the original value of the property, or clicks the "Reset" button, then it will return back to the normal coloration.<div style="background-color:orange"><p style="color:white; padding:5px"><b>WARNING:</b> Changing the value of these properties can prevent the workflow manager from running after the web server is restarted. Also, no validation checks are performed on the user-provided values. Proceed with caution!</p></div>At the bottom of the properties table is the "Save Properties" button. The number of modified properties is shown in parentheses. Clicking the button will make the necessary changes to the properties file on the file system, but the changes will not take effect until the workflow manager is restarted. The saved properties will be colored blue and a blue icon will be displayed to the right of the property name. Additionally, a notification will appear at the top of the page alerting all system users that a restart is required:![Properties Settings Page](img/mpf-adm-property-settings-change.png "Properties Settings Page")# User ConfigurationEvery time the Workflow Manager starts it will attempt to create accounts for the users listed in the `user.properties` file. At runtime this file is extracted to `$MPF_HOME/config` on the machine running the Workflow Manager. For every user listed in that file, the Workflow Manager will create that user account if a user with the same name doesn't already exists in the SQL database. By default, that file contains two entries, one for the "admin" user with the "mpfadm" password, and one for a non-admin "mpf" user with the "mpf123" password.We highly recommend modifying the `user.properties` file with your own user entries before attempting to start the Workflow Manager for the first time. This will ensure that the default user accounts are not created. The official way to deploy OpenMPF is to use the Docker container platform. If you are using Docker, please follow the instructions in the openmpf-docker [README](https://github.com/openmpf/openmpf-docker/blob/master/README.md#optional-configure-users) that explain how to use a `docker secret` for your custom `user.properties` file.

docs/docs/CPP-Batch-Component-API.md

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
1-
> **NOTICE:** This software (or technical data) was produced for the U.S. Government under contract, and is subject to the Rights in Data-General Clause 52.227-14, Alt. IV (DEC 2007). Copyright 2021 The MITRE Corporation. All Rights Reserved.
1+
**NOTICE:** This software (or technical data) was produced for the U.S. Government under contract, and is subject to the
2+
Rights in Data-General Clause 52.227-14, Alt. IV (DEC 2007). Copyright 2021 The MITRE Corporation. All Rights Reserved.
23

34
# API Overview
45

56
In OpenMPF, a **component** is a plugin that receives jobs (containing media), processes that media, and returns results.
67

7-
The OpenMPF Batch Component API currently supports the development of **detection components**, which are used detect objects in image, video, audio, or other (generic) files that reside on disk.
8+
The OpenMPF Batch Component API currently supports the development of **detection components**, which are used to detect objects in image, video, audio, or other (generic) files that reside on disk.
89

910
Using this API, detection components can be built to provide:
1011

@@ -54,15 +55,16 @@ Detection components are implemented by:
5455
5556
1. Extending [`MPFDetectionComponent`](#detection-component-interface).
5657
2. Building the component into a shared object library. (See [HelloWorldComponent CMakeLists.txt](https://github.com/openmpf/openmpf-cpp-component-sdk/blob/master/detection/examples/HelloWorldComponent/CMakeLists.txt)).
57-
3. Packaging the component into an OpenMPF-compliant .tar.gz file. (See [Component Packaging](#component-packaging)).
58-
4. Registering the component with OpenMPF. (See [Packaging and Registering a Component](Packaging-and-Registering-a-Component/index.html)).
58+
3. Creating a component Docker image. (See the [README](https://github.com/openmpf/openmpf-docker/tree/master/components/cpp_executor#overview)).
5959
6060
# API Specification
6161
6262
The figure below presents a high-level component diagram of the C++ Batch Component API:
6363
6464
![OpenMPF Component Diagram](img/component_diagram_cpp_batch.png "OpenMPF Component Diagram")
6565
66+
The Node Manager is only used in a non-Docker deployment. In a Docker deployment the Component Executor is started by the Docker container itself.
67+
6668
The API consists of *Component Interfaces*, which provide interfaces and abstract classes for developing components; *Job Definitions*, which define the work to be performed by a component; *Job Results*, which define the results generated by the component; [*Component Adapters*](#convenience-adapters), which provide default implementations of several of the [`MPFDetectionComponent`](#detection-component-interface) interface functions; and [*Component Utilities*](#utility-classes), which perform actions such as image rotation, and cropping.
6769
6870
**Component Interface**
@@ -439,7 +441,7 @@ MPFJob(
439441
|---|---|---|
440442
| job_name <a name="job-name"></a> | `const string &` | A specific name given to the job by the OpenMPF framework. This value may be used, for example, for logging and debugging purposes. |
441443
| data_uri <a name="data-uri"></a> | `const string &` | The URI of the input media file to be processed. Currently, this is a file path. For example, "/opt/mpf/share/remote-media/test-file.avi". |
442-
| job_properties <a name="job-properties"></a> | `const Properties &` | Contains a map of `<string, string>` which represents the property name and the property value. The key corresponds to the property name specified in the component descriptor file described in [Packaging and Registering a Component](Packaging-and-Registering-a-Component/index.html). Values are determined when creating a pipeline or when submitting a job. <br/><br/> Note: The job_properties map may not contain the full set of job properties. For properties not contained in the map, the component must use a default value. |
444+
| job_properties <a name="job-properties"></a> | `const Properties &` | Contains a map of `<string, string>` which represents the property name and the property value. The key corresponds to the property name specified in the component descriptor file described in the [Component Descriptor Reference](Component-Descriptor-Reference/index.html). Values are determined when creating a pipeline or when submitting a job. <br/><br/> Note: The job_properties map may not contain the full set of job properties. For properties not contained in the map, the component must use a default value. |
443445
| media_properties <a name="media-properties"></a> | `const Properties &` | Contains a map of `<string, string>` of metadata about the media associated with the job. The entries in the map vary depending on the type of media. Refer to the type-specific job structures below. |
444446
445447
@@ -1048,7 +1050,7 @@ OpenMPF components should be stateless in operation and give identical output fo
10481050
10491051
For components that want to take advantage of NVIDA GPU processors, please read the [GPU Support Guide](GPU-Support-Guide/index.html). Also ensure that your build environment has the NVIDIA CUDA Toolkit installed, as described in the [Build Environment Setup Guide](Build-Environment-Setup-Guide/index.html).
10501052
1051-
## Component Packaging
1053+
## Component Structure for non-Docker Deployments
10521054
It is recommended that C++ components are organized according to the following directory structure:
10531055
10541056
```
@@ -1066,10 +1068,10 @@ Once built, components should be packaged into a .tar.gz containing the contents
10661068
## Logging
10671069
It is recommended to use [Apache log4cxx](https://logging.apache.org/log4cxx/index.html) for
10681070
OpenMPF Component logging. Components using log4cxx should not configure logging themselves.
1069-
The component executor will configure log4cxx globally. Components should call
1071+
The Component Executor will configure log4cxx globally. Components should call
10701072
`log4cxx::Logger::getLogger("<componentName>")` to a get a reference to the logger. If you
10711073
are using a different logging framework, you should make sure its behavior is similar to how
1072-
the component executor configures log4cxx as described below.
1074+
the Component Executor configures log4cxx as described below.
10731075
10741076
The following log LEVELs are supported: `FATAL, ERROR, WARN, INFO, DEBUG, TRACE`.
10751077
The `LOG_LEVEL` environment variable can be set to one of the log levels to change the logging

0 commit comments

Comments
 (0)