Skip to content

Commit 07724d5

Browse files
aicamclaude
andauthored
fix(access-control-service): include port in computing unit pod URI and use Envoy Gateway for distributed CUs (#5629)
### What changes were proposed in this PR? Make the in-cluster address of a computing unit come from a single source of truth — the URI recorded when its pod is created — and ensure that URI is complete (includes the port). This lets the gateway route a user to a computing unit located **anywhere it can reach** (in the local cluster, another cluster, or an external host), instead of being limited to a reconstructed in-cluster address. See #5630. Two related changes: **1. Include the port in the generated pod URI** (`computing-unit-managing-service`) `KubernetesClient.generatePodURI` builds the address stored as the computing unit's `uri` (via `setUri` in `ComputingUnitManagingResource`) and returned to clients as `nodeAddresses`. The pod's container listens on `KubernetesConfig.computeUnitPortNumber` (declared with `withContainerPort(...)` in the same file), but the generated URI omitted the port, so the persisted address was not directly connectable. The port is now appended: ```scala s"...svc.cluster.local:${KubernetesConfig.computeUnitPortNumber}" ``` **2. Route using the recorded URI** (`access-control-service`) `AccessControlResource` rebuilt the computing unit's address from `KubernetesConfig` on every authorization request, duplicating the construction logic in `generatePodURI` and pinning every CU to the local cluster. It now reads the URI recorded for the unit and returns it as the `Host` for the gateway to route to. If no URI has been recorded, the unit is not routable and the request is **refused with `403`** (no in-cluster fallback, per review). ### Routing flow The access-control service is the gateway's external authorizer; the `Host` it returns is the upstream Envoy forwards the (upgraded) connection to. Because that host comes from the unit's recorded URI, the same gateway can reach computing units in different locations: ```mermaid flowchart LR FE["Frontend<br/>(/wsapi?cuid=N)"] --> GW["Envoy Gateway"] GW -. "ext-auth: authorize + get Host" .-> ACS["access-control-service"] ACS -- "read recorded uri for CU N" --> DB[("workflow_computing_unit")] ACS -- "Host = recorded uri<br/>(or 403 if none)" --> GW GW == "dynamic forward proxy<br/>to returned Host" ==> R{Where the CU lives} R --> CU1["In-cluster CU pod<br/>computing-unit-N...svc.cluster.local:port"] R --> CU2["CU in another cluster"] R --> CU3["External / remote CU host:port"] ``` ### Any related issues, documentation, discussions? - Closes #5630. - Builds on the Envoy Gateway / ext-auth routing introduced in #4191 (unified Envoy Gateway) and #3598 (access-control-service as the ext-auth service for computing-unit traffic). ### How was this PR tested? On live deployment. <img width="1835" height="960" alt="Screenshot from 2026-06-13 13-31-00" src="https://github.com/user-attachments/assets/d56a48f9-b99d-4d36-827a-0a4ce54995fd" /> ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 990b7cb commit 07724d5

3 files changed

Lines changed: 82 additions & 14 deletions

File tree

access-control-service/src/main/scala/org/apache/texera/service/resource/AccessControlResource.scala

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,10 @@ import jakarta.ws.rs.{Consumes, DELETE, GET, POST, PUT, Path, Produces}
2727
import org.apache.texera.auth.JwtParser.parseToken
2828
import org.apache.texera.auth.SessionUser
2929
import org.apache.texera.auth.util.{ComputingUnitAccess, HeaderField}
30-
import org.apache.texera.common.config.{GuiConfig, KubernetesConfig, LLMConfig}
30+
import org.apache.texera.common.config.{GuiConfig, LLMConfig}
31+
import org.apache.texera.dao.SqlServer
3132
import org.apache.texera.dao.jooq.generated.enums.PrivilegeEnum
33+
import org.apache.texera.dao.jooq.generated.tables.daos.WorkflowComputingUnitDao
3234

3335
import java.net.URLDecoder
3436
import java.nio.charset.StandardCharsets
@@ -136,12 +138,25 @@ object AccessControlResource extends LazyLogging {
136138
}
137139

138140
// Dynamic Routing Logic
139-
val workflowComputingUnitPoolName = KubernetesConfig.computeUnitPoolName
140-
val workflowComputingUnitPoolNamespace = KubernetesConfig.computeUnitPoolNamespace
141-
val workflowComputingUnitPoolPort = KubernetesConfig.computeUnitPortNumber
142-
143-
val targetHost =
144-
s"computing-unit-$cuidInt.$workflowComputingUnitPoolName-svc.$workflowComputingUnitPoolNamespace.svc.cluster.local:$workflowComputingUnitPoolPort"
141+
// Route to the URI recorded for the computing unit (written by the managing
142+
// service when the pod is created). This recorded URI is the single source
143+
// of truth for where the unit is reachable, allowing units to live anywhere
144+
// the gateway can route to. If no URI has been recorded, the unit is not
145+
// routable and the connection is refused.
146+
val cuDao = new WorkflowComputingUnitDao(
147+
SqlServer.getInstance().createDSLContext().configuration()
148+
)
149+
val unit = cuDao.fetchOneByCuid(cuidInt)
150+
val recordedUri = Option(unit).flatMap(u => Option(u.getUri)).map(_.trim).filter(_.nonEmpty)
151+
152+
val targetHost = recordedUri match {
153+
case Some(uri) =>
154+
logger.info(s"Routing CU $cuidInt to recorded host: $uri")
155+
uri
156+
case None =>
157+
logger.warn(s"Refusing CU $cuidInt: no URI recorded for the computing unit")
158+
return Response.status(Response.Status.FORBIDDEN).build()
159+
}
145160

146161
Response
147162
.ok()

access-control-service/src/test/scala/org/apache/texera/AccessControlResourceSpec.scala

Lines changed: 59 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,11 @@ class AccessControlResourceSpec
5555
private val testURI: String = "http://localhost:8080/"
5656
private val testPath: String = "/api/executions/1/stats/1"
5757

58+
// The host:port the managing service records for a computing unit when it
59+
// creates the pod. The access-control-service routes to this recorded URI.
60+
private val testRecordedUri: String =
61+
"computing-unit-2.compute-unit-svc.default.svc.cluster.local:8888"
62+
5863
private val testUser1: User = {
5964
val user = new User()
6065
user.setUid(1)
@@ -81,6 +86,31 @@ class AccessControlResourceSpec
8186
cu.setType(WorkflowComputingUnitTypeEnum.kubernetes)
8287
cu.setCuid(2)
8388
cu.setName("test-cu")
89+
cu.setUri(testRecordedUri)
90+
cu
91+
}
92+
93+
// A computing unit the user can access but for which no URI was ever recorded
94+
// (e.g. the pod was never created). Such a unit is not routable and must be
95+
// refused.
96+
private val testCUNoUri: WorkflowComputingUnit = {
97+
val cu = new WorkflowComputingUnit()
98+
cu.setUid(2)
99+
cu.setType(WorkflowComputingUnitTypeEnum.kubernetes)
100+
cu.setCuid(3)
101+
cu.setName("test-cu-no-uri")
102+
cu
103+
}
104+
105+
// A computing unit whose recorded URI is blank/whitespace-only — also treated
106+
// as "no URI recorded" and refused.
107+
private val testCUBlankUri: WorkflowComputingUnit = {
108+
val cu = new WorkflowComputingUnit()
109+
cu.setUid(2)
110+
cu.setType(WorkflowComputingUnitTypeEnum.kubernetes)
111+
cu.setCuid(4)
112+
cu.setName("test-cu-blank-uri")
113+
cu.setUri(" ")
84114
cu
85115
}
86116

@@ -96,12 +126,18 @@ class AccessControlResourceSpec
96126
userDao.insert(testUser1)
97127
userDao.insert(testUser2)
98128
computingUnitDao.insert(testCU)
99-
100-
val cuAccess = new ComputingUnitUserAccess()
101-
cuAccess.setUid(testUser1.getUid)
102-
cuAccess.setCuid(testCU.getCuid)
103-
cuAccess.setPrivilege(PrivilegeEnum.WRITE)
104-
computingUnitOfUserDao.insert(cuAccess)
129+
computingUnitDao.insert(testCUNoUri)
130+
computingUnitDao.insert(testCUBlankUri)
131+
132+
// Grant testUser1 WRITE access to every test computing unit so the routing
133+
// logic (not the access check) is what each routing test exercises.
134+
Seq(testCU, testCUNoUri, testCUBlankUri).foreach { cu =>
135+
val cuAccess = new ComputingUnitUserAccess()
136+
cuAccess.setUid(testUser1.getUid)
137+
cuAccess.setCuid(cu.getCuid)
138+
cuAccess.setPrivilege(PrivilegeEnum.WRITE)
139+
computingUnitOfUserDao.insert(cuAccess)
140+
}
105141

106142
val claims = JwtAuth.jwtClaims(testUser1, 1)
107143
token = JwtAuth.jwtToken(claims)
@@ -232,6 +268,23 @@ class AccessControlResourceSpec
232268
response.getHeaderString(HeaderField.UserId) shouldBe testUser1.getUid.toString
233269
response.getHeaderString(HeaderField.UserName) shouldBe testUser1.getName
234270
response.getHeaderString(HeaderField.UserEmail) shouldBe testUser1.getEmail
271+
// Envoy routes by the rewritten Host header, which must be the URI recorded
272+
// for the computing unit.
273+
response.getHeaderString("Host") shouldBe testRecordedUri
274+
}
275+
276+
it should "refuse the connection when no URI is recorded for the computing unit" in {
277+
val (uri, headers) = mockRequest(testPath, Some(testCUNoUri.getCuid.toString))
278+
val response = new AccessControlResource().authorizeGet(uri, headers)
279+
280+
response.getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode
281+
}
282+
283+
it should "refuse the connection when the recorded URI is blank" in {
284+
val (uri, headers) = mockRequest(testPath, Some(testCUBlankUri.getCuid.toString))
285+
val response = new AccessControlResource().authorizeGet(uri, headers)
286+
287+
response.getStatus shouldBe Response.Status.FORBIDDEN.getStatusCode
235288
}
236289

237290
private def mockRequest(

computing-unit-managing-service/src/main/scala/org/apache/texera/service/util/KubernetesClient.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ object KubernetesClient {
3535
private val podNamePrefix = "computing-unit"
3636

3737
def generatePodURI(cuid: Int): String = {
38-
s"${generatePodName(cuid)}.${KubernetesConfig.computeUnitServiceName}.$namespace.svc.cluster.local"
38+
s"${generatePodName(cuid)}.${KubernetesConfig.computeUnitServiceName}.$namespace.svc.cluster.local:${KubernetesConfig.computeUnitPortNumber}"
3939
}
4040

4141
def generatePodName(cuid: Int): String = s"$podNamePrefix-$cuid"

0 commit comments

Comments
 (0)