feat: Add resource-info-fetcher

Cargo.toml

@@ -26,7 +26,8 @@ futures = { version = "0.3" }
 hyper = "1.4"
 indoc = "2.0"
 ldap3 = { version = "0.12", features = ["gssapi", "tls"] }
-moka = { version = "0.12", features = ["future"] }
+md5 = "0.8"
+moka = { version = "0.12", default-features = false, features = ["future"] }
 native-tls = "0.2.12"
 pin-project = "1.1"
 # `default-features = false` drops reqwest 0.13's `default-tls = ["rustls"]`; we select
@@ -47,6 +48,7 @@ tar = "0.4"
 tokio = { version = "1.52", features = ["full"] }
 tracing = "0.1"
 url = "2.5"
+urlencoding = "2.1"
 uuid = "1.10"
 wiremock = "0.6"
 

_TEST/CURLS.sh

@@ -0,0 +1,13 @@
+curl 'localhost:9477/metadata/database?system=trino&instance=kuttl-secure-hermit-wf84/my-trino&database=tpch' | jq
+curl 'localhost:9477/metadata/schema?system=trino&instance=kuttl-secure-hermit-wf84/my-trino&database=tpch&schema=sf1' | jq
+curl 'localhost:9477/metadata/table?system=trino&instance=kuttl-secure-hermit-wf84/my-trino&database=tpch&schema=sf1&table=customer' | jq
+curl 'localhost:9477/metadata/stream?system=kafka&instance=kuttl-secure-hermit-wf84/test-kafka&queue=orders' | jq
+curl 'localhost:9477/metadata/dashboard?system=superset&instance=kuttl-secure-hermit-wf84/my-superset&id=1' | jq
+curl 'localhost:9477/metadata/chart?system=superset&instance=kuttl-secure-hermit-wf84/my-superset&id=1' | jq
+
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:container:792ce6aace288712a1ef036fbca2bb37' | jq
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:container:ae5d10cb199e26e8df70c7563e6d1c58' | jq
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:dataset:(urn:li:dataPlatform:trino,kuttl-secure-hermit-wf84/my-trino.tpch.sf1.customer,PROD)' | jq
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:dataset:(urn:li:dataPlatform:kafka,kuttl-secure-hermit-wf84/test-kafka.orders,PROD)' | jq
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:dashboard:(superset,kuttl-secure-hermit-wf84/my-superset.1)' | jq
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:chart:(superset,kuttl-secure-hermit-wf84/my-superset.1)' | jq

_TEST/clientId

@@ -0,0 +1 @@
+__datahub_system

_TEST/clientSecret

@@ -0,0 +1 @@
+datahub-demo-system-secret

_TEST/rif.json

@@ -0,0 +1,11 @@
+{
+    "cache": {
+        "entryTimeToLive": "0s"
+    },
+    "backend": {
+        "dataHub": {
+            "hostname": "localhost",
+            "port": 8080
+        }
+    }
+}

docs/modules/opa/pages/usage-guide/resource-info-fetcher.adoc

@@ -0,0 +1,165 @@
+= Resource info fetcher
+:description: Resource Info Fetcher for OPA retrieves data from backends like DataHub. Integrate extra resource details into Rego rules for enhanced policy management.
+
+The _Resource info fetcher_ allows for additional information to be obtained from the configured backend (for example, DataHub).
+You can then write Rego rules for OpenPolicyAgent which make an HTTP request to the Resource info fetcher and make use of the additional information returned for the resource, such as Trino tables or Apache Kafka topics.
+
+You can enable the Resource info fetcher sidecar as follows:
+
+[source,yaml]
+----
+apiVersion: opa.stackable.tech/v1alpha2
+kind: OpaCluster
+metadata:
+  name: opa
+spec:
+  image:
+    productVersion: 1.0.0
+  clusterConfig:
+    resourceInfo: # <1>
+      backend:
+        dataHub:
+          hostname: datahub-gms.my-namespace.svc.cluster.local
+          port: 8080
+          tls:
+            verification:
+              server:
+                caCert:
+                  secretClass: tls # <2>
+          credentialsSecretName: resource-info-fetcher-credentials # <3>
+          env: PROD # <4>
+      cache: # optional, enabled by default
+        entryTimeToLive: 60s # optional, defaults to 60s
+  servers:
+    roleGroups:
+      default: {}
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: resource-info-fetcher-credentials
+stringData:
+  token: <datahub-personal-access-token> # <3>
+----
+
+<1> Enable the `resource-info-fetcher` sidecar
+<2> Enable TLS verification using the CA from the `tls` SecretClass.
+<3> Authenticate to DataHub with a Personal Access Token (PAT) read from the specified Secret. The Secret must have a `token` entry. See the DataHub documentation on https://docs.datahub.com/docs/authentication/personal-access-tokens[Personal Access Tokens] for how to create one, and make sure https://docs.datahub.com/docs/authentication/introducing-metadata-service-authentication[Metadata Service Authentication] is enabled on your DataHub.
+<4> The DataHub environment (fabric) to query, e.g. `PROD` or `DEV`. Defaults to `PROD`, which is also what DataHub's ingestion sources default to.
+
+Currently the following backends are supported:
+
+* xref:#backend-data-hub[]
+
+[#backends]
+== Backends
+
+The initial version is tightly modeled to the only supported backend, namely DataHub.
+The API might change as soon as we add support for other backends.
+
+
+[#backend-data-hub]
+=== DataHub
+
+DataHub is currently only supported backend, so the general documentation from above applies.
+
+== Resource info fetcher API
+
+Resource information can be retrieved from regorules using the functions in `data.stackable.opa.resourceinfo.v1`.
+There is one function per kind of resource:
+
+[source,rego]
+----
+databaseResourceInfo(system, instance, database)
+schemaResourceInfo(system, instance, database, schema)
+tableResourceInfo(system, instance, database, schema, table)
+streamResourceInfo(system, instance, queue)
+dashboardResourceInfo(system, instance, id)
+chartResourceInfo(system, instance, id)
+
+rawIdentifierResourceInfo(identifier)
+----
+
+The naming is intentionally product-agnostic, so that one function serves the equivalent resource of every product.
+`databaseResourceInfo` addresses what Trino calls a catalog, and `streamResourceInfo` what Apache Kafka calls a topic.
+
+`rawIdentifierResourceInfo` is the escape hatch for resources the functions above do not cover: it passes the identifier to the backend as-is.
+For DataHub that is a URN, such as `urn:li:chart:(superset,my-namespace/my-superset.1)`.
+
+The first two arguments are the same everywhere:
+
+* `system` is the kind of product the resource lives in, for example `trino`, `kafka` or `superset`. DataHub calls this the _data platform_.
+* `instance` identifies _which_ deployment of that product, for example `my-namespace/my-trino`. DataHub calls this the _platform instance_, and the value must match the `platform_instance` of the ingestion source that produced the metadata.
+
+The DataHub environment (fabric) is deliberately *not* an argument: it describes how the catalog was populated rather than the resource being authorized, so it is configured once on the OpaCluster (see `env` above) instead of being passed in by every Rego rule.
+
+An example of the returned structure:
+
+[source,json]
+----
+{
+  "dataProducts": [],
+  "domain": null,
+  "owners": {
+    "urn:li:ownershipType:__system__business_owner": {
+      "groups": [
+        {
+          "description": "Customer Service/Analytics (mirrored from the Keycloak demo realm)",
+          "displayName": "Customer Service/Analytics",
+          "urn": "urn:li:corpGroup:customer-service-analytics"
+        }
+      ],
+      "ownershipTypeName": "Business Owner",
+      "users": []
+    }
+  },
+  "tags": [
+    {
+      "name": "PII",
+      "urn": "urn:li:tag:PII"
+    }
+  ],
+  "urn": "urn:li:container:c8531e5a52cacf56768d0bf77ca8787c"
+}
+----
+
+=== Debug request
+
+To debug the resource-info-fetcher you can `curl` its API for a given resource.
+Every Rego function above maps to a `GET /metadata/<resource type>` endpoint that takes its arguments as query parameters.
+To achieve this shell into the `opa` container and execute
+
+[source,bash]
+----
+curl 'localhost:9477/metadata/schema?system=trino&instance=my-namespace/my-trino&database=lakehouse&schema=customer_analytics' | jq
+
+curl 'localhost:9477/metadata/chart?system=superset&instance=my-namespace/my-superset&id=1' | jq
+
+curl 'localhost:9477/metadata/rawIdentifier?identifier=urn:li:chart:(superset,my-namespace/my-superset.1)' | jq
+----
+
+=== Rego rule library
+
+The HTTP API exposed by the resource-info-fetcher can be called directly using the rego function `http.send`.
+However, we provide a convenience rego rule library, which we ship with `OpaClusters` by default.
+
+For example, the following rule allows access to tables tagged as `public`:
+
+[source,rego]
+----
+package test
+
+import data.stackable.opa.resourceinfo.v1 as resourceinfo
+
+default allow := false
+
+allow if {
+    table := resourceinfo.tableResourceInfo("trino", "my-namespace/my-trino", input.catalog, input.schema, input.table)
+    some tag in table.tags
+    tag.urn == "urn:li:tag:public"
+}
+----
+
+A resource the backend does not know about is not reported as an error: the resource-info-fetcher returns a record with empty `tags`, `owners` and `dataProducts` and a `null` `domain`.
+Prefer rules that require a positive signal, like the one above, which denies access in that case.
+A rule that merely excludes a tag would instead grant access to every resource missing from the backend.

docs/modules/opa/pages/usage-guide/user-info-fetcher.adoc

@@ -46,7 +46,7 @@ stringData:
 
 <1> Enable the `user-info-fetcher` sidecar
 <2> Enable TLS verification using the CA from the `tls` SecretClass.
-<3> Obtain Keycloak API credentials from the specified secret. The Secret must have `clientId` and `clientSecret` entries.
+<3> Obtain Keycloak API credentials from the specified Secret. The Secret must have `clientId` and `clientSecret` entries.
 <4> Refer to the applicable realm in your Keycloak server.
 
 Currently the following backends are supported: