Skip to content

feat: support Filesystem Wasm code source via EnvoyProxy allowlist#9530

Open
daanvinken wants to merge 4 commits into
envoyproxy:mainfrom
daanvinken:feat/wasm-filesystem-source-9448
Open

feat: support Filesystem Wasm code source via EnvoyProxy allowlist#9530
daanvinken wants to merge 4 commits into
envoyproxy:mainfrom
daanvinken:feat/wasm-filesystem-source-9448

Conversation

@daanvinken

@daanvinken daanvinken commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Description

Adds a Filesystem WASM code source so Envoy can load modules from a local path
without the control-plane fetch/cache path used by HTTP and Image sources.

Design matches the discussion in #9448 (DynamicModule-style allowlist):

  • Register modules on EnvoyProxy.spec.wasmModules (name + absolute path)
  • Reference them from EnvoyExtensionPolicy with code.type: Filesystem and
    filesystem.name only (no path on the policy)
  • Translator emits AsyncDataSource.local; HTTP/Image behavior is unchanged

Fixes #9448

API note

The Wasm code.type enum is now HTTP, Image, and Filesystem. The
previous ConfigMap enum value was never implemented and has been removed.

Testing

  • Unit test for local vs remote Wasm code source in the xDS translator
  • Translator fixtures for EnvoyExtensionPolicy with Filesystem Wasm
    (registered module resolves to a path; unknown module name is rejected)
  • xDS fixture that emits code.local.filename for a Filesystem module
  • CEL validation cases for Filesystem code source and EnvoyProxy wasmModules

If desired, I can test it on a real cluster, need to upgrade Envoy Gateway first...

Generated with Grok Build 4.5.

@daanvinken
daanvinken requested a review from a team as a code owner July 20, 2026 12:30
@netlify

netlify Bot commented Jul 20, 2026

Copy link
Copy Markdown

Deploy Preview for cerulean-figolla-1f9435 ready!

Name Link
🔨 Latest commit edb360c
🔍 Latest deploy log https://app.netlify.com/projects/cerulean-figolla-1f9435/deploys/6a5e3cf7ead20e000873a55d
😎 Deploy Preview https://deploy-preview-9530--cerulean-figolla-1f9435.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@daanvinken

Copy link
Copy Markdown
Contributor Author

@zhaohuabing if you have some time 🙏

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 53e53febd3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

wasms []ir.Wasm
wasmFailOpen bool
)
if wasms, wasmError, wasmFailOpen = t.buildWasms(policy, resources, gtwCtx.envoyProxy); wasmError != nil {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Scope filesystem Wasm errors per ancestor

When an HTTPRoute is attached to multiple managed Gateways, this call now resolves the filesystem module against each Gateway's EnvoyProxy but appends any failure to one errs value returned to processEnvoyExtensionPolicyForRoute, which then sets the translation error on all ancestor refs. If one Gateway allowlists the module and another does not, the successful ancestor is still marked translation-failed even though its routes get the filter; please report/apply the error per parentRef instead of aggregating it across all ancestors.

Useful? React with 👍 / 👎.

@daanvinken
daanvinken force-pushed the feat/wasm-filesystem-source-9448 branch from 53e53fe to 089e532 Compare July 20, 2026 12:43
@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.84211% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.56%. Comparing base (2646c11) to head (fddff27).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
internal/gatewayapi/envoyextensionpolicy.go 78.94% 4 Missing and 4 partials ⚠️
internal/xds/translator/wasm.go 94.73% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #9530   +/-   ##
=======================================
  Coverage   75.55%   75.56%           
=======================================
  Files         252      252           
  Lines       41661    41755   +94     
=======================================
+ Hits        31479    31552   +73     
- Misses       8057     8071   +14     
- Partials     2125     2132    +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@daanvinken
daanvinken force-pushed the feat/wasm-filesystem-source-9448 branch 2 times, most recently from f4fe92e to fddff27 Compare July 20, 2026 14:21
@zhaohuabing

zhaohuabing commented Jul 20, 2026

Copy link
Copy Markdown
Member
  1. Unnecessary wasm_cluster emitted for filesystem-only configs (minor, worth fixing)

containsWasm (internal/xds/translator/globalresources.go:144) returns true for any Wasm, so createWasmHTTPServiceCluster still emits the wasm_cluster (the EG HTTP serving cluster, plus its upstream TLS transportSocket) even when every Wasm is a Filesystem source that never contacts it. This is visible in the PR's own fixture wasm-filesystem.clusters.yaml, which contains a full wasm_cluster that no filter references.

Suggest gating on the presence of a remote source:

if route.EnvoyExtensions != nil {
    for _, w := range route.EnvoyExtensions.Wasms {
        if w.Code != nil { // HTTP/Image only
            return true
        }
    }
}

This avoids shipping a dead cluster (and its client-cert secret dependency) to proxies that only use filesystem modules — which is precisely the "skip the control-plane path" benefit the PR advertises.

  1. buildWasms moved inside the per-parentRef loop — cache side effect now repeats (minor)

In translateEnvoyExtensionPolicyForRoute, buildWasms moved from once-per-policy to once-per-parentRef (necessary, since the filesystem path resolves against gtwCtx.envoyProxy, which can differ per Gateway). This matches the existing buildLuas/buildDynamicModules placement, so structurally it's fine.

The wrinkle unique to Wasm: for HTTP/Image sources this now calls t.WasmCache.Get() N times for a route with N parentRefs, where it previously ran once. For IfNotPresent this is a cheap cache hit, but with pullPolicy: Always it can trigger repeated fetches. Not a correctness bug, but a behavioral change worth a comment or a second look — a small memoization keyed by resolved config would eliminate it if it matters.

  1. Enum value ConfigMap silently removed (call out in PR description)

The field-level annotation changed from Enum=HTTP;Image;ConfigMap to Enum=HTTP;Image;Filesystem (wasm_types.go:41). ConfigMap was never implemented (no code path, and the type alias only listed HTTP;Image), so this is stale cleanup rather than a real removal. Still, it's technically a narrowing of the accepted API surface — worth an explicit mention in the PR description/release note in case anyone had it set (it would have been non-functional regardless).

Register Wasm modules on EnvoyProxy.spec.wasmModules and reference them
by name from EnvoyExtensionPolicy with type Filesystem. Envoy loads the
local file via AsyncDataSource.local and skips the control-plane
fetch/cache path used by HTTP and Image sources.

Includes generated CRD/helm fixtures, API docs, and the jwt DnsCluster
fixture update needed for gen-check.

Fixes envoyproxy#9448

Signed-off-by: daanvinken <daanvinken@tythus.com>
@daanvinken
daanvinken force-pushed the feat/wasm-filesystem-source-9448 branch from fddff27 to 3e206c5 Compare July 20, 2026 14:56
Only create the control-plane wasm HTTP service cluster (and gate the
client certificate) when a route uses a remote HTTP/Image code source.
Filesystem modules load a local path and never use that cluster.

Signed-off-by: daanvinken <daanvinken@tythus.com>
Filesystem resolution is Gateway-scoped, so buildWasms runs per parentRef
like Lua and DynamicModules. HTTP/Image may call WasmCache.Get once per
parent; IfNotPresent hits the cache, Always may re-fetch.

Signed-off-by: daanvinken <daanvinken@tythus.com>
ConfigMap was never implemented; the code source enum is now HTTP, Image,
and Filesystem only.

Signed-off-by: daanvinken <daanvinken@tythus.com>
@daanvinken

Copy link
Copy Markdown
Contributor Author

Super helpful, thanks @zhaohuabing

I've addressed your comments. I'll see when I can carve out some time to deploy this on an actual cluster and test.
Feel free to leave any comments in the meantime.

@zhaohuabing

zhaohuabing commented Jul 20, 2026

Copy link
Copy Markdown
Member

Should we mirror DynamicModuleSource to leave room for defining HTTP/OCI WASM extensions in the EnvoyProxy(not now)? cc @envoyproxy/gateway-maintainers

type WasmModuleEntry struct {
    Name   string           `json:"name"`
    Source WasmModuleSource `json:"source"`
}

type WasmModuleSource struct {
    // +kubebuilder:default=Local
    // +unionDiscriminator
    Type  *WasmModuleSourceType  `json:"type,omitempty"`   // Local (+ Remote/Image later)
    Local *LocalWasmModuleSource `json:"local,omitempty"`  // { path }
}

wasmModules:
- name: security-filter
  source:
    type: Local          # defaulted; can be omitted
    local:
      path: /var/lib/envoy/security-filter.wasm
const EnvoyProxyModuleWasmCodeSourceType WasmCodeSourceType = "EnvoyProxyModule"

type EnvoyProxyModuleWasmCodeSource struct {
    // Name references a module in EnvoyProxy.spec.wasmModules.
    Name string `json:"name"`
}

wasm:
- code:
    type: EnvoyProxyModule
    envoyProxyModule:
      name: security-filter        # local or remote — resolved from EnvoyProxy

@zhaohuabing

Copy link
Copy Markdown
Member

I'll see when I can carve out some time to deploy this on an actual cluster and test

You should be able to add an e2e test for this, following the dynamic module example. You can use this WASM file. for the test.

@jukie

jukie commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

for defining HTTP/OCI WASM extensions in the EnvoyProxy(not now)?

I agree with the suggested shape for filesystem WASM modules but long-term handling of image/http sources might get awkward so we should discuss that as well.
It doesn't need to be wired up here but maybe we should deprecate the existing image/http source fields and offer operators a RuntimeFlag to control whether EnvoyProxy registration is required or optional?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support a local/filesystem Wasm code source in EnvoyExtensionPolicy

3 participants