Improve static file and SPA hosting in the YARP container image

[davidfowl]

Improve static file and SPA hosting in the YARP container image

Context

The YARP container image (src/Application) can serve as a static file host and reverse proxy, but it's missing features that every major static hosting platform provides out of the box. Today, a team building an Astro, React, or Angular site behind the YARP container has to accept bare 404s, no compression, no cache control, and no redirect support — all things that nginx, Caddy, Netlify, Vercel, Azure Static Web Apps, and Firebase Hosting handle automatically.

Scenario: Astro docs site behind YARP (like aspire.dev)

aspire.dev is a production Astro Starlight site served by a custom ASP.NET Core static host. Its build produces:

• index.html per page (static site generation)
• _astro/ directory with hashed assets (_astro/page.C3x9k2.css, _astro/hoisted.B7dK1p.js)
• 404.html custom error page
• Static assets (favicon, images, fonts)

Using this as a baseline, here's what works and what doesn't when serving the same output through the YARP container:

Request	nginx	Caddy	YARP today
GET / → index.html	✅	✅	✅
GET /docs/getting-started/ → directory default document	✅	✅	✅
GET /nonexistent/ → 404.html with status 404	✅ error_page 404	✅ handle_errors	❌ bare empty 404
GET /_astro/page.C3x9k2.css → Cache-Control: immutable	✅ location-based	✅ header directive	❌ no cache headers
GET /index.html → Cache-Control: no-cache	✅	✅	❌ no cache headers
Pre-compressed .br/.gz sidecar files served when client accepts	✅ gzip_static/brotli_static	✅ precompressed br gzip	❌
On-the-fly response compression	✅ gzip on	✅ encode gzip zstd	❌
HSTS + HTTPS redirect in production	✅ configurable	✅ automatic	❌
Declarative redirects (/old → /new)	✅ rewrite/return 301	✅ redir	❌
SPA fallback with path exclusions	✅ via location blocks	✅ via matchers	⚠️ fallback exists, no exclusions
Custom response headers per path	✅ add_header	✅ header directive	❌
Trailing slash normalization	✅ rewrite	✅ automatic	❌
Clean URLs (/about → about.html)	✅ try_files $uri.html	✅ try_files {path}.html	❌

What this looks like in nginx and Caddy

Here's a complete production config for the same Astro site on each platform — covering all the features above:

nginx

server {
    listen 443 ssl http2;
    server_name docs.example.com;
    root /var/www/site;

    # HTTPS / HSTS
    ssl_certificate /etc/ssl/cert.pem;
    ssl_certificate_key /etc/ssl/key.pem;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;

    # Compression: on-the-fly + pre-compressed sidecars
    gzip on;
    gzip_types text/plain text/css application/javascript application/json image/svg+xml;
    gzip_static on;
    brotli_static on;

    # Security headers (all responses)
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-Frame-Options "DENY" always;

    # Custom 404 page
    error_page 404 /404.html;
    location = /404.html {
        internal;
        add_header Cache-Control "no-cache" always;
    }

    # Redirects
    location = /old-page     { return 301 /new-page; }
    location = /get-started  { return 302 /docs/quickstart/; }
    location = /install.sh   { return 302 https://aka.ms/install.sh; }

    # Hashed assets: cache forever
    location /_astro/ {
        add_header Cache-Control "public, max-age=31536000, immutable" always;
        try_files $uri =404;
    }

    # HTML files: never cache (so deployments take effect)
    location ~* \.html$ {
        add_header Cache-Control "no-cache" always;
    }

    # API routes: proxy to backend (no SPA fallback)
    location /api/ {
        proxy_pass http://backend:5000;
    }

    # Clean URLs + SPA fallback (excluding /api/ and file extensions)
    location / {
        try_files $uri $uri.html $uri/ /index.html;
    }

    # Trailing slash: redirect /about/ → /about (if not a directory)
    rewrite ^(.+)/$ $1 permanent;
}

# HTTP → HTTPS redirect
server {
    listen 80;
    server_name docs.example.com;
    return 301 https://$host$request_uri;
}

Caddy

docs.example.com {
    root * /var/www/site

    # Compression: on-the-fly + pre-compressed sidecars
    encode gzip zstd br
    file_server {
        precompressed br gzip
    }

    # HSTS (Caddy handles HTTPS + certs automatically)
    header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

    # Security headers (all responses)
    header X-Content-Type-Options "nosniff"
    header X-Frame-Options "DENY"

    # Hashed assets: cache forever
    header /_astro/* Cache-Control "public, max-age=31536000, immutable"

    # HTML files: never cache
    header /*.html Cache-Control "no-cache"

    # Redirects
    redir /old-page /new-page 301
    redir /get-started /docs/quickstart/ 302
    redir /install.sh https://aka.ms/install.sh 302

    # Trailing slash: strip (redirect /about/ → /about)
    @trailing path_regexp trailing ^(.+)/$
    redir @trailing {re.trailing.1} 301

    # API routes: proxy to backend (before SPA fallback)
    handle /api/* {
        reverse_proxy backend:5000
    }

    # Clean URLs + SPA fallback (excluding /api/ and file extensions)
    handle {
        try_files {path} {path}.html {path}/ /index.html
        file_server
    }

    # Custom 404 page
    handle_errors {
        @404 expression {http.error.status_code} == 404
        rewrite @404 /404.html
        header Cache-Control "no-cache"
        file_server
    }
}

Both configs represent the same production-ready static site with all 9 features. The YARP container should be able to replicate this behavior through configuration alone.

Features

1. Custom 404 page

Serve wwwroot/404.html (if it exists) when no route matches, keeping the 404 status code. Today the YARP container returns a bare empty 404 body.

What others do:

• nginx: error_page 404 /404.html;
• Caddy: handle_errors { rewrite * /404.html; file_server }
• Aspire StaticHost: UseStatusCodePages + response.SendFileAsync("404.html") with no-cache headers
• Netlify: automatically serves 404.html from publish directory

2. Cache control for hashed vs unhashed assets

Every SPA/static site build tool (Vite, webpack, Astro, Angular CLI) produces content-hashed filenames like app.a1b2c3.js. These should be cached aggressively. index.html and other unhashed files should not be cached so deployments take effect immediately.

Expected behavior:

• Files with hash-like patterns in the filename (or under _astro/, assets/, etc.): Cache-Control: public, max-age=31536000, immutable
• index.html and other HTML files: Cache-Control: no-cache

What others do:

• nginx: separate location blocks with different Cache-Control headers
• Caddy: header directive with path matchers
• Aspire StaticHost: MSBuild targets set immutable on _astro/ files at publish time

This could be convention-based (detect hashed filenames) or configurable via patterns.

3. Response compression

Enable on-the-fly gzip/brotli compression for text-based responses (HTML, CSS, JS, JSON, SVG). SPA bundles are large and highly compressible — serving them uncompressed is a significant performance penalty.

What others do:

• nginx: gzip on; gzip_types text/plain text/css application/javascript ...
• Caddy: encode gzip zstd br (one line)

Since the YARP container is a pre-built image where users mount their own wwwroot at runtime, MapStaticAssets() (which requires build-time manifests) is not an option. UseResponseCompression() is the right approach here.

4. Pre-compressed sidecar file support

If the wwwroot contains pre-compressed sidecar files (e.g., app.js.br, app.js.gz alongside app.js), serve the compressed version when the client's Accept-Encoding supports it. This avoids the CPU cost of on-the-fly compression.

What others do:

• nginx: gzip_static on; brotli_static on; — native, zero-config
• Caddy: file_server { precompressed br gzip } — native, one line

Many static site generators and build tools can produce these sidecar files at build time. This is the highest-performance path for static asset delivery.

5. HTTPS redirection and HSTS

Enable UseHsts() and UseHttpsRedirection() in production (non-Development) environments as a security baseline. Opt-out via configuration.

This must be forwarded-header aware — the YARP container will often sit behind a CDN, load balancer, or Kubernetes ingress that terminates TLS. Without checking X-Forwarded-Proto, the container would redirect in a loop. UseForwardedHeaders() should be enabled when HTTPS redirect is on.

What others do:

• nginx: add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload";
• Caddy: automatic with HTTPS (enabled by default)
• Aspire StaticHost: UseHsts() + UseHttpsRedirection() with 180-day max-age

6. Declarative redirects

Support configurable redirect rules (301/302) via JSON configuration for common scenarios like renamed pages, vanity URLs, and external link shortcuts — without requiring code changes.

Example config:

{
  "Redirects": {
    "/old-page": { "Destination": "/new-page", "StatusCode": 301 },
    "/get-started": { "Destination": "/docs/quickstart/", "StatusCode": 302 }
  }
}

What others do:

• nginx: rewrite ^/old$ /new permanent; or return 301 /new;
• Caddy: redir /old /new 301
• Netlify: _redirects file with pattern matching
• Vercel: vercel.json redirects array

Note: Declarative redirects cover the common case. Use cases that require custom per-request logic (e.g., telemetry tracking on redirects, as aspire.dev does for install scripts) would still need a custom YARP route or middleware.

7. Custom response headers per path

Allow setting custom HTTP headers on responses based on path patterns, configurable via JSON. This covers CORS, security headers, custom caching, and anything else a site needs to set per-route.

Example config:

{
  "Headers": {
    "/*": {
      "X-Content-Type-Options": "nosniff",
      "X-Frame-Options": "DENY"
    },
    "/api/*": {
      "Access-Control-Allow-Origin": "*"
    },
    "/_astro/*": {
      "Cache-Control": "public, max-age=31536000, immutable"
    }
  }
}

What others do:

• nginx: add_header per location block
• Caddy: header directive with path matchers
• Netlify: _headers file with path patterns
• Vercel: vercel.json headers array

This subsumes the cache control feature (#2) — cache headers become one use case of the general custom headers capability.

8. Trailing slash normalization

Consistently handle trailing slashes by redirecting to the canonical form. Static site generators like Astro have a trailingSlash config (always, never, ignore) and expect the host to enforce it.

Expected behavior (configurable):

• always: /about → 301 to /about/
• never: /about/ → 301 to /about

What others do:

• nginx: rewrite ^(.+)/$ $1 permanent; or vice versa
• Caddy: handles automatically based on file existence
• Netlify: trailingSlash option in netlify.toml
• Azure Static Web Apps: trailingSlash in staticwebapp.config.json (always, never, auto)
• Astro: trailingSlash in astro.config.mjs (build-time only, expects host to enforce)

9. Clean URLs

Serve /about from /about.html without requiring the .html extension in the URL. This is the default behavior on Netlify and GitHub Pages, and is how most static site generators expect their output to be served when not using directory-based routing.

Expected behavior:

• GET /about → serves about.html if it exists (without redirect)
• GET /about.html → optionally redirect to /about for canonical URLs

What others do:

• nginx: try_files $uri $uri.html $uri/ =404;
• Caddy: try_files {path} {path}.html {path}/
• Netlify: enabled by default (Pretty URLs)
• Vercel: cleanUrls: true in vercel.json
• Firebase: cleanUrls: true in firebase.json

Interaction with trailing slash: These features must be ordered to avoid redirect loops. For example, if cleanUrls is on and trailingSlash is never, then /about/ should redirect to /about (not try to serve /about/.html). The resolution order should be: trailing slash normalization first, then clean URL resolution.

10. SPA fallback with path exclusions

The current SPA fallback (MapFallbackToFile("index.html")) is all-or-nothing — it catches every unmatched non-file request. In a combined proxy + static host, this swallows misrouted API calls and other paths that should return 404.

Support configurable exclusion patterns so specific paths bypass the fallback:

Example config:

{
  "NavigationFallback": {
    "Rewrite": "/index.html",
    "Exclude": ["/api/*", "/.well-known/*", "/healthz"]
  }
}

What others do:

• Azure Static Web Apps: navigationFallback.exclude array in staticwebapp.config.json
• nginx: separate location /api/ block with proxy_pass (no fallback)
• Caddy: handle /api/* block before the fallback handle block
• Netlify: SPA rewrite /* /index.html 200 is ordered after explicit routes
• Vercel: rewrites are processed in order; earlier rules win

Request resolution order

These features form a small rule system, not isolated switches. The resolution order matters — both for correctness and to avoid surprising interactions with YARP's reverse proxy routes.

Proposed order:

1. HTTPS redirect (if enabled and request is HTTP)
2. Trailing slash normalization (redirect to canonical form)
3. Declarative redirects (exact match or pattern)
4. Static file (exact match in wwwroot)
5. Clean URL resolution (try .html extension)
6. Directory default document (try index.html in directory)
7. YARP reverse proxy routes
8. SPA fallback to index.html (if enabled and path not excluded)
9. Custom 404 page (serve 404.html if it exists)

Key principle: These features apply to the static hosting pipeline only — they must not accidentally mutate proxied traffic. Custom headers, cache control, compression, and SPA fallback should not apply to responses that YARP proxies to a backend. YARP's own route configuration handles those.

Non-goals

• MapStaticAssets() — requires a build-time manifest, which doesn't work for a pre-packaged container where users mount their own wwwroot at runtime