SQL-414: Do allowed_roles check on internal endpoints

[SangJunBak]

There exists a security bug where self managed environments with password/oidc auth can have
any non-system user access our internal HTTP routes. These include injecting audit events,
leader promotion, and etc. This PR fixes this by extending the schema of our server listener configurations.

Motivation

Fixes https://linear.app/materializeinc/issue/SQL-414/http-endpoints-only-authenticate-but-dont-authorize

Description

To review, I'd recommend reading the plan in the commit "Outline the problem and solution"
769b7bd (this PR) to understand the problem. The solution proposed there is quite 1:1 with the implementation. The implementation aims to simplify the authorization code in our web server handling, so much of the code added in this PR are simple refactors. I'd recommend reviewing commit by commit.

Verification

We base it off the regression test Dennis initially created and make sure it passes our existing test suite.

[def-]

Nightly failures look very related! https://buildkite.com/materialize/nightly/builds/16892

[def-]

We're on v26.32.0-dev, rebase and bump

[def-]

Are we ok with this file allowing full access to internal routes for the emulator? Edit:

diff --git a/test/http-auth/mzcompose.py b/test/http-auth/mzcompose.py
index 74cb1a914c..ba5f737f23 100644
--- a/test/http-auth/mzcompose.py
+++ b/test/http-auth/mzcompose.py
@@ -111,6 +111,12 @@ def workflow_default(c: Composition) -> None:

     check_livez_coordinator_coupling(c)

+    check_emulator_baked_in_password_config(c)
+
+
+def workflow_emulator_password_config(c: Composition) -> None:
+    check_emulator_baked_in_password_config(c)
+

 def check_livez_coordinator_coupling(c: Composition) -> None:
     """Regression test for SQL-343: previously having group_claim_for above
@@ -198,3 +204,36 @@ def check_internal_endpoints_require_authorization(
             f"/api/catalog/inject-audit-events: status={r.status_code}, "
             f"audit rows={forged_rows}"
         )
+
+
+def check_emulator_baked_in_password_config(c: Composition) -> None:
+    with c.override(
+        Materialized(
+            listeners_config_path=f"{MZ_ROOT}/src/materialized/ci/listener_configs/v26_31_0/password.json"
+        )
+    ):
+        c.up("materialized")
+        ext = f"http://localhost:{c.port('materialized', 6876)}"
+
+        # mz_system authenticates with the baked-in login password
+        # (MZ_EXTERNAL_LOGIN_PASSWORD_MZ_SYSTEM) to provision a normal login role.
+        admin = ("mz_system", "password")
+        normal = ("sql414_normal", "sql414_normal_pw")
+        r = requests.post(
+            f"{ext}/api/sql",
+            auth=admin,
+            json={"query": f"CREATE ROLE {normal[0]} LOGIN PASSWORD '{normal[1]}'"},
+        )
+        assert r.status_code == 200, f"role setup failed: {r.status_code}: {r.text}"
+
+        # `/api/catalog/dump` and `/api/coordinator/dump` are the `internal`
+        # group, `/prof/` the `profiling` group. Coordinator dump carries pgwire
+        # cancellation secret keys.
+        for path in ("/api/catalog/dump", "/api/coordinator/dump", "/prof/"):
+            slug = path.strip("/").replace("/", "_")
+            with c.test_case(f"emulator_password_{slug}_blocks_normal_user"):
+                r = requests.get(f"{ext}{path}", auth=normal)
+                assert r.status_code in (401, 403), (
+                    f"normal role {normal[0]} reached {path} on the external "
+                    f"emulator port: status={r.status_code}, {len(r.text)} bytes"
+                )

Running bin/mzcompose --find http-auth run emulator-password-config fails with:

==> Running test case emulator_password_prof_blocks_normal_user
==> mzcompose: test case emulator_password_prof_blocks_normal_user failed: builtins.AssertionError: normal role sql414_normal reached /prof/ on the external emulator port: status=200, 1564 bytes
Traceback (most recent call last):
  File "/home/deen/git/materialize/misc/python/materialize/mzcompose/composition.py", line 740, in test_case
    yield
  File "/home/deen/git/materialize/test/http-auth/mzcompose.py", line 250, in check_emulator_baked_in_password_config
    assert r.status_code in (401, 403), (
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: normal role sql414_normal reached /prof/ on the external emulator port: status=200, 1564 bytes

[SangJunBak]

Hmm I can see what happens if I make it more strict, but I essentially copied over the same configuration as the old listener config for password.json.

[SangJunBak]

I may follow up for this however in a followup PR given my changes doesn't change the existent behavior. Also I can see how it'd be convenient to relax the access control in the listener configs for our tests

[SangJunBak]

Converting into draft to address nightlies

[SangJunBak]

@def- I looked into why many of the nightlies (scalability, feature, workload replay) are failing, and it seems to be because each are comparing the branch's version against current main. However because main is at the same dev version (with a different sha) but doesn't contain my listener config changes, my version guard fails and the "ancestor" version panics. It seems to me like this is self healing however once this PR gets merged since future dev versions of v26.32.0 will know how to support the new listener config.

For fast lookup, this my commit for version guarding my change in Materialized: 27f5f57 (this PR)

Other nightlies (like the platform checks) I've fixed as well.