Skip to content

README.md

Latest commit

 

History

History
320 lines (237 loc) · 15.1 KB

README.md

File metadata and controls

320 lines (237 loc) · 15.1 KB

Stream Path Matcher

This example shows how to exercise the path matching functionality of the Location Library in a streaming environment.

The example map-matches data coming from the sample-streaming-layer that contains SDII messages. The catalog that contains this layer is defined in the sdii-catalog entry in config/here/pipeline-config.conf. The example writes the result of the computation to the out-data streaming layer in the catalog you will create.

The StreamPathMatcherExample class does the following:

  • Get the Flink StreamExecutionEnvironment
  • Add a SdiiMessageMapFunction, the Flink SourceFunction that provides data from the input streaming layer and deserializes it
  • Add a PathMatcherMapFunction, the Flink SourceFunction that does the path matching
  • Put the result descriptions into the catalog you will create, as well as print them to the standard output

Prerequisites

Before you execute the instructions in the next sections of this document, read the Prerequisites for the Location Library examples.

To run the example, you need access to the following catalogs:

Get Your Credentials

To run this example, you need two sets of credentials:

  • Platform credentials: To get access to platform data and resources.
  • Repository credentials: To download HERE Data SDK for Java & Scala libraries and Maven archetypes to your environment.

For more details on how to set up your credentials, see the Identity & Access Management Developer Guide.

For more details on how to verify that your platform credentials are configured correctly, see the Verify Your Credentials tutorial.

Build and Run the Compiler

In the commands that follow, replace the variable placeholders with the following values:

  • $PROJECT_HRN is your project's HRN (returned by the olp project create command).
  • $OLP_EMAIL is a single contact e-mail address for the pipeline.
  • $COVERAGE is a two-letter code for country and region (in this case, DE for Germany)
  • $INPUT_SDII_CATALOG is the HRN of the public sdii-catalog catalog in your pipeline configuration (HERE environment.
  • $INPUT_OPTIMIZED_MAP_CATALOG is the HRN of the public optimized-map-catalog catalog in your pipeline configuration (HERE environment.

Note: We recommend that you set values to variables, so that you can easily copy and execute the following commands.

Run the Application Locally

Create a Local Output Catalog

As mentioned above we will use two public input catalogs, however we need to create our own output catalog to store the results of the Stream Path Matcher example.

To run this compiler locally, use a local output catalog as described below. For more information about local catalogs, see the SDK tutorial about local development and testing and the OLP CLI documentation.

  1. Use the olp local catalog create command to create a local catalog.
olp local catalog create path-matcher-java path-matcher-java --summary "Output catalog for Stream Path Matcher example" \
            --description "Output catalog for Stream Path Matcher example"

The local catalog will have the HRN hrn:local:data:::path-matcher-java.

  1. Use the olp local catalog layer add command to add one stream layer to your catalog:
olp local catalog layer add hrn:local:data:::path-matcher-java out-data out-data --stream --summary "Layer for output partitions" \
            --description "Layer for output partitions" --coverage $COVERAGE

Run the Application from the Command Line

First, we're going to run the example using a Flink local environment, suitable for local development and debugging.

  1. Compile and execute the example.

For the HERE platform environment:

To run your Flink application locally with Java 17, you should provide --add-opens=java.base/java.util=ALL-UNNAMED to the command arguments.

mvn compile exec:exec \
-Dexec.args="--add-opens=java.base/java.util=ALL-UNNAMED -cp %classpath -Dpipeline-config.file=config/here/local-pipeline-config.conf -Dpipeline-job.file=config/here/pipeline-job.conf com.here.platform.example.location.java.flink.StreamPathMatcherExample"
  1. Open a different terminal and let a few partitions stream out of the layer. They consist of small, one-line messages. The following command prints 10 messages specified by the --limit parameter or messages coming within 900 seconds separated by --delimeter=\\n parameter.
olp local catalog layer stream get hrn:local:data:::path-matcher-java out-data --delimiter=\\n --limit=10 --timeout=900

The command should return the following results:

Result for id 818bad2e-6d18-43ae-9396-86d9a8c9ab84: matched 4 points out of 4
Result for id f319318f-a161-4e65-8017-db0588c5470b: matched 69 points out of 70
Result for id 5a095919-49c1-48ef-8be0-d867d176170b: matched 6 points out of 6
Result for id f99bca1a-18a8-4f3d-81df-7ae7742b68b1: matched 92 points out of 92
Result for id 05aa970f-4222-4708-a2a4-3b20b5943d9c: matched 103 points out of 103
Result for id 5f19487e-0390-458d-8e71-1b0d3c2c6050: matched 96 points out of 98
Result for id e315d58c-f64a-42fe-928d-96851ad8c975: matched 7 points out of 7
Result for id 4cddbdbc-152d-4c14-8c19-0deb077026a5: matched 199 points out of 199
Result for id 6417cb50-267b-4ec3-8256-47d6ea661d1f: matched 69 points out of 70
Result for id f1185134-9af1-4514-b3e3-dae7708f810f: matched 6 points out of 6
Result for id bd69c6c6-4d0b-419d-9ae2-51fb0d7365c7: matched 72 points out of 72
  1. End stream example process after partitions were retrieved.

Run this Application as a HERE Platform Pipeline

Configure a Project

To follow this example, you will need a project. A project is a collection of platform resources (catalogs, pipelines, and schemas) with controlled access. You can create a project through the platform portal.

Alternatively, use the OLP CLI olp project create command to create the project:

olp project create $PROJECT_ID $PROJECT_NAME

The command returns the HERE Resource Name (HRN) of your new project. Note down this HRN as you will need it later in this tutorial.

Note

You do not have to provide a --scope parameter if your app has a default scope. For details on how to set a default project scope for an app, see the Specify a default Project for Apps chapter of the Identity & Access Management Developer Guide.

For more information on how to work with projects, see the Organize your work in projects tutorial.

Create an Output Catalog

The catalog you need to create is used to store the results of the Stream Path Matcher example.

Use the HERE platform portal to create the output catalog in your project and add the following layers:

Layer ID Layer Type Content Type Content Encoding Coverage
out-data Stream application/x-protobuf uncompressed DE

Alternatively, you can use the OLP CLI to create the catalog and the corresponding layers.

  1. Use the olp catalog create command to create the catalog. Make sure to note down the HRN returned by the following command for later use:
olp catalog create $CATALOG_ID $CATALOG_ID --summary "Output catalog for Stream Path Matcher example" \
            --description "Output catalog for Stream Path Matcher example" \
            --scope $PROJECT_HRN
  1. Use the olp catalog layer add command to add a stream layer to your catalog:
olp catalog layer add $CATALOG_HRN out-data out-data --stream --summary "Layer for output partitions" \
            --description "Layer for output partitions" --coverage $COVERAGE \
            --scope $PROJECT_HRN

Note::

If a billing tag is required in your realm, use the --billing-tags: "YOUR_BILLING_TAG" parameter.

  1. Update the output catalog HRN in the pipeline-config.conf file

The config/here/pipeline-config.conf (for the HERE platform environment) file contains the permanent configuration of the data sources for the example.

Pick the file that corresponds to your platform environment and replace YOUR_OUTPUT_CATALOG_HRN placeholder with the HRN of your Stream Path Matcher catalog.

To find the HRN, in the HERE platform portal, navigate to your catalog. The HRN is displayed in the upper left corner of the page.

  1. Use the olp project resource link command to link the HERE Sample SDII Messages - Berlin and HERE Optimized Map for Location Library catalog to your project.
olp project resource link $PROJECT_HRN $INPUT_SDII_CATALOG
olp project resource link $PROJECT_HRN $INPUT_OPTIMIZED_MAP_CATALOG

Generate a Fat JAR file

Generate a "fat jar" for StreamPathMatcherExample that will be sent to the platform later

mvn -Pplatform package

Deploy Fat JAR to a Pipeline

You can use the OLP CLI to create pipeline components and activate the pipeline version with the following commands:

  1. Create pipeline components:
olp pipeline create $COMPONENT_NAME_Pipeline --email $OLP_EMAIL --scope $PROJECT_HRN
olp pipeline template create $COMPONENT_NAME_Template stream-7.0 $PATH_TO_JAR \
                com.here.platform.example.location.java.flink.StreamPathMatcherExample \
                --input-catalog-ids="$PATH_TO_CONFIG_FOLDER/pipeline-config.conf" \
                --scope $PROJECT_HRN
olp pipeline version create $COMPONENT_NAME_version $PIPELINE_ID $PIPELINE_TEMPLATE_ID \
                "$PATH_TO_CONFIG_FOLDER/pipeline-config.conf" \
                --scope $PROJECT_HRN
  • Make sure logs are produced with info level
olp pipeline version log level set $PIPELINE_ID $PIPELINE_VERSION_ID --root info --scope $PROJECT_HRN

If the operation is successful, you should be able to see the log level you just set:

olp pipeline version log level get $PIPELINE_ID $PIPELINE_VERSION_ID \
--scope $PROJECT_HRN
  1. Activate the pipeline version:
olp pipeline version activate $PIPELINE_ID $PIPELINE_VERSION_ID --input-catalogs "$PATH_TO_CONFIG_FOLDER/pipeline-job.conf" --scope $PROJECT_HRN

Since this is a Flink application, this means that it runs until you stop it. In order to stop the application after you have checked its operation, execute the following command:

olp pipeline version cancel $PIPELINE_ID $PIPELINE_VERSION_ID --scope $PROJECT_HRN

Let a few partitions stream out of the layer

Partitions consist of small, one-line messages.

The following command prints 10 messages specified by the --limit parameter or messages coming within 900 seconds separated by --delimeter=\\n parameter.

olp catalog layer stream get $CATALOG_HRN out-data --delimiter=\\n --limit=10 --timeout=900 --scope $PROJECT_HRN

The command should return the following results:

Result for id 818bad2e-6d18-43ae-9396-86d9a8c9ab84: matched 4 points out of 4
Result for id f319318f-a161-4e65-8017-db0588c5470b: matched 69 points out of 70
Result for id 5a095919-49c1-48ef-8be0-d867d176170b: matched 6 points out of 6
Result for id f99bca1a-18a8-4f3d-81df-7ae7742b68b1: matched 92 points out of 92
Result for id 05aa970f-4222-4708-a2a4-3b20b5943d9c: matched 103 points out of 103
Result for id 5f19487e-0390-458d-8e71-1b0d3c2c6050: matched 96 points out of 98
Result for id e315d58c-f64a-42fe-928d-96851ad8c975: matched 7 points out of 7
Result for id 4cddbdbc-152d-4c14-8c19-0deb077026a5: matched 199 points out of 199
Result for id 6417cb50-267b-4ec3-8256-47d6ea661d1f: matched 69 points out of 70
Result for id f1185134-9af1-4514-b3e3-dae7708f810f: matched 6 points out of 6
Result for id bd69c6c6-4d0b-419d-9ae2-51fb0d7365c7: matched 72 points out of 72