[BUG] Opa sometimes 401s unexpectedly

[daisieh]

From slack messages:
authz sends opa a PUT request for /v1/data/opa_token :

{
    "client_addr": "127.0.0.1:57932",
    "level": "info",
    "msg": "Received request.",
    "req_id": 3520,
    "req_method": "PUT",
    "req_path": "/v1/data/opa_token",
    "time": "2026-04-15T15:28:24Z"
}

OPA responds with a 401 status every time:

{
    "client_addr": "127.0.0.1:57932",
    "level": "info",
    "msg": "Sent response.",
    "req_id": 3520,
    "req_method": "PUT",
    "req_path": "/v1/data/opa_token",
    "resp_bytes": 87,
    "resp_duration": 0.467167,
    "resp_status": 401,
    "time": "2026-04-15T15:28:24Z"
}

(these show up as request rejected by administrative policy in the authz log)
This seems to occur right after authz renews the OPA approle token.

As far as I can tell, this can only happen if the file /permissions_engine/opa_secret is not present or has an incorrect value in the Opa container, but the only time that happens is on initial setup.

[v-rocheleau]

Issue just occured again, with authz being deployed with a replica count of 1.

This seems to invalidate the hypothesis that multiple replicas were responsible for the issue.

[v-rocheleau]

Getting back to this issue!

Rubber ducking here to validate if I understand correctly what is going on.

1. app/entrypoint.sh

Authz container entrypoint, invokes scripts in a loop:

• /app/renew_token.sh: renews the Authz AppRole token
  • Enables authentication between authz => OpenBao by making sure the token does not expire
• /app/refresh_stores.py: I don't know OPA enough to fully understand what is going on here
  • (my uneducated guess) gives OPA a token so that it can communicate with OpenBao (OPA => OpenBao) ?

2. /app/refresh_stores.py

Runs after /app/renew_token.sh.

Seemingly the script where the issue comes up:

• Calls the endpoint that returns a 401:
  • PUT /v1/data/opa_token on the OPA container (only call to this endpoint I am seeing in the codebase)
  • Payload: {"token": get_vault_token_for_service("opa", role_id=opa_role_id)}
    • opa_role_id: value read from /home/pcgl/opa-roleid or OPA_ROLEID env var
  • Headers:
    • X-Opa: value read from /permissions_engine/opa_secret.json

Question: what is the general purpose of this script, and what is the difference between the payload token and the header secret?

It is still unclear to me what the OPA AppRole does concretely, and how it differs from the other AppRole.

3. app/permissions_engine/authz.rego

From my understanding, the value fed to X-Opa never changes and the error request rejected by administrative policy seems to indicate that OPA's authorization rejects the request.

The OPA policy for the /v1/data/opa_token endpoint is:

# Opa should be able to store its vault token
allow if {
	input.path == ["v1", "data", "opa_token"]
	input.method == "PUT"
	input.headers["X-Opa"][_] == data.opa_secret
}

We can safely assume that the path and method are always correct, which leaves input.headers["X-Opa"][_] == data.opa_secret as the possible culprit.

For whatever reason, there seems to be a mismatch between authz and OPA for this value.
But this feels unlikely given that /permissions_engine/opa_secret.json is only written when the container first starts.

Question: how does OPA read this value to evaluate it (data.opa_secret)?

Further debugging

I'm creating a PR for the dev environment to set OPA's log level to debug, hopefully we can get more useful info with this.

[v-rocheleau]

@daisieh created the PR for the OPA log level

[daisieh]

The approle system is meant to make sure that only authorized services are able to communicate with each other in predefined ways (https://developer.hashicorp.com/vault/docs/auth/approle). I originally used it in CanDIG because we do have several services in different docker containers that communicate with OPA and it made sense as a security measure. It is rotated through refresh_stores.py so that in case of a breach, it won't stay the same forever.

I'm not completely sure it's necessary in PCGL, but I didn't see the harm in keeping it. Maybe it's unnecessary though?

And yes, your interpretation of the problem is the same as mine. The value of data.opa_secret is loaded into OPA with the PUT request on line 24 of refresh_stores.py; after that, OPA can read it at the path data/opa_token.

[v-rocheleau]

Thanks for the details, clears things up for me!

Looks like we have a (somewhat) related issue at the moment in dev, made a PR to try to solve it. Except this time it fails on start which is more worrying.

[v-rocheleau]

Read a bunch of OPA docs this week and dove deeper in the authz <=> OPA interactions, think I have identified some possible problems we can resolve.

I think the issues we are having stem from circular dependencies between the 2 containers, this seems to lead to deadlocks.

We've had the issue where authz runs fine for some time, then it becomes unable to call OPA's /v1/data/opa_token endpoint.

This week we had another similar issue in dev, this time during startup. Issue was fixed in this PR, by adding the -w flag to the OPA run command so that the files in /permissions_engine are reloaded when they are written by the authz container.

To me this indicates there is a race condition caused by the tight coupling.
Will pitch fixes in a following comment.

[v-rocheleau]

Root cause

Curent integration of authz and OPA creates circular dependencies:

• Authz
  • Shares /permissions_engine volume with OPA
    • Rego policies are modified with sed for some env vars substitutions
    • Writes modified Rego policies to the volume on startup
  • Generates random runtime OPA API secret at startup, written to /permissions_engine/opa_secret.json
  • Calls OPA API with generated secret to update OpenBao token on an interval
• OPA
  • Shares /permissions_engine volume with OPA
  • Rego files loaded from shared volume
  • Must have loaded /permissions_engine/opa_secret.json (created by authz on startup) to authorize authz container requests

This makes startup order, file reload timing and runtime state tightly coupled, which can all lead to races and deadlocks.

We can take some actions to reduce this coupling, I have some recommendations bellow.

Recomendations

1. Create OPA secret outside of authz runtime

Now: authz creates this secrets on startup then opa has to load it
Recommended: create secret outside of runtime and inject it into both containers

Removes potential secret race, since the secret exists before runtime starts and is provided to the containers that need it.

Requires modifying or creating a new authz container entrypoint.

2. Stop using a shared volume for rego policies

Now: authz and opa share a volume on /permissions_engine, authz writes to it and opa reads from it on startup
Recommended: 1 volume for Rego files, only used by the opa container

Removes rego files race on startup. opa and authz containers can start in any order.

Requires modifying the authz entrypoint, and a mechanism to inject rego files in the opa container.

Note

Can be achieved in Docker-Compose and Kubernetes via "init" containers (we can use the authz image here since it has the versionned policies in it):

• Docker-Compose: pre_start init container for opa, writes policies to /permissions_engine
• Kubernetes: same idea, but via pod.spec.initContainers

3. Package rego policies as an OPA bundle (if possible, optional)

Now: rego files are not bundled and are therefore mutable
Recommended: make the policies immutable by packaging them in an OPA bundle (.tar.gz)

Turns policies into a single immutable artifact that can be versionned with authz.

[daisieh]

Working on implementing these changes in #86