resource-info-fetcher.adoc

Resource info fetcher

The Resource info fetcher allows for additional information to be obtained from the configured backend (for example, DataHub). You can then write Rego rules for OpenPolicyAgent which make an HTTP request to the Resource info fetcher and make use of the additional information returned for the resource, such as Trino tables or Apache Kafka topics.

You can enable the Resource info fetcher sidecar as follows:

apiVersion: opa.stackable.tech/v1alpha2
kind: OpaCluster
metadata:
  name: opa
spec:
  image:
    productVersion: 1.0.0
  clusterConfig:
    resourceInfo: # (1)
      backend:
        dataHub:
          hostname: datahub-gms.my-namespace.svc.cluster.local
          port: 8080
          tls:
            verification:
              server:
                caCert:
                  secretClass: tls # (2)
          credentialsSecretName: resource-info-fetcher-credentials # (3)
          env: PROD # (4)
      cache: # optional, enabled by default
        entryTimeToLive: 60s # optional, defaults to 60s
  servers:
    roleGroups:
      default: {}
---
apiVersion: v1
kind: Secret
metadata:
  name: resource-info-fetcher-credentials
stringData:
  token: <datahub-personal-access-token> # (3)

1. Enable the resource-info-fetcher sidecar

2. Enable TLS verification using the CA from the tls SecretClass.

3. Authenticate to DataHub with a Personal Access Token (PAT) read from the specified Secret. The Secret must have a token entry. See the DataHub documentation on Personal Access Tokens for how to create one, and make sure Metadata Service Authentication is enabled on your DataHub.

4. The DataHub environment (fabric) to query, e.g. PROD or DEV. Defaults to PROD, which is also what DataHub’s ingestion sources default to.

Currently the following backends are supported:

• DataHub

Backends

The initial version is tightly modeled to the only supported backend, namely DataHub. The API might change as soon as we add support for other backends.

DataHub

DataHub is currently only supported backend, so the general documentation from above applies.

Resource info fetcher API

Resource information can be retrieved from regorules using the functions in data.stackable.opa.resourceinfo.v1. There is one function per kind of resource:

databaseResourceInfo(system, instance, database)
schemaResourceInfo(system, instance, database, schema)
tableResourceInfo(system, instance, database, schema, table)
streamResourceInfo(system, instance, queue)
dashboardResourceInfo(system, instance, id)
chartResourceInfo(system, instance, id)

rawIdentifierResourceInfo(identifier)

The naming is intentionally product-agnostic, so that one function serves the equivalent resource of every product. databaseResourceInfo addresses what Trino calls a catalog, and streamResourceInfo what Apache Kafka calls a topic.

rawIdentifierResourceInfo is the escape hatch for resources the functions above do not cover: it passes the identifier to the backend as-is. For DataHub that is a URN, such as urn:li:chart:(superset,my-namespace/my-superset.1).

The first two arguments are the same everywhere:

• system is the kind of product the resource lives in, for example trino, kafka or superset. DataHub calls this the data platform.

• instance identifies which deployment of that product, for example my-namespace/my-trino. DataHub calls this the platform instance, and the value must match the platform_instance of the ingestion source that produced the metadata.

The DataHub environment (fabric) is deliberately not an argument: it describes how the catalog was populated rather than the resource being authorized, so it is configured once on the OpaCluster (see env above) instead of being passed in by every Rego rule.

An example of the returned structure:

{
  "dataProducts": [],
  "domain": null,
  "owners": {
    "urn:li:ownershipType:__system__business_owner": {
      "groups": [
        {
          "description": "Customer Service/Analytics (mirrored from the Keycloak demo realm)",
          "displayName": "Customer Service/Analytics",
          "urn": "urn:li:corpGroup:customer-service-analytics"
        }
      ],
      "ownershipTypeName": "Business Owner",
      "users": []
    }
  },
  "tags": [
    {
      "name": "PII",
      "urn": "urn:li:tag:PII"
    }
  ],
  "urn": "urn:li:container:c8531e5a52cacf56768d0bf77ca8787c"
}

Debug request

To debug the resource-info-fetcher you can curl its API for a given resource. Every Rego function above maps to a GET /metadata/<resource type> endpoint that takes its arguments as query parameters. To achieve this shell into the opa container and execute

curl 'localhost:9477/metadata/schema?system=trino&instance=my-namespace/my-trino&database=lakehouse&schema=customer_analytics' | jq

curl 'localhost:9477/metadata/chart?system=superset&instance=my-namespace/my-superset&id=1' | jq

curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:chart:(superset,my-namespace/my-superset.1)' | jq

Rego rule library

The HTTP API exposed by the resource-info-fetcher can be called directly using the rego function http.send. However, we provide a convenience rego rule library, which we ship with OpaClusters by default.

For example, the following rule allows access to tables tagged as public:

package test

import data.stackable.opa.resourceinfo.v1 as resourceinfo

default allow := false

allow if {
    table := resourceinfo.tableResourceInfo("trino", "my-namespace/my-trino", input.catalog, input.schema, input.table)
    some tag in table.tags
    tag.urn == "urn:li:tag:public"
}

A resource the backend does not know about is not reported as an error: the resource-info-fetcher returns a record with empty tags, owners and dataProducts and a null domain. Prefer rules that require a positive signal, like the one above, which denies access in that case. A rule that merely excludes a tag would instead grant access to every resource missing from the backend.