Add HTTP forwarding proxy mode for dynamic upstream origins.

Parse embedded http(s) URLs from PSS forwarding paths, align journal lookup
with the file plugin, and document example configs for browser-facing proxies.

Assisted-by: Cursor:Composer-2.5 CursorAI

Author: apeters1971

src/XrdApps/CMakeLists.txt

@@ -79,6 +79,7 @@ if(BUILD_HTTP AND NOT XRDCL_ONLY)
 
   add_library(${XrdClJournalCacheHttpExt} MODULE
     XrdClJournalCachePlugin/http/JournalCacheHttpExt.cc
+    XrdClJournalCachePlugin/http/ForwardingUrl.cc
     XrdClJournalCachePlugin/file/CacheHeaders.cc
     XrdClJournalCachePlugin/file/Digest.cc
   )

src/XrdApps/XrdClJournalCachePlugin/README.md

@@ -413,6 +413,66 @@ Per-file operational CGIs (append to the HTTP URL or XRootD open URL):
 /store/file.root?xrd.journalcache.clean=1
 ```
 
+## 7.7 HTTP forwarding proxy (dynamic origins)
+
+For a browser-facing **forwarding proxy** where each request names its own HTTP(S) upstream, use PSS forwarding mode together with `forwarding = 1` in the HTTP ext config.
+
+**Request form** (destination embedded in the path):
+
+```
+GET /https://cdn.example.org/store/file.dat HTTP/1.1
+Host: cache-proxy.example
+```
+
+**Server config** (see `http/journalcache-forwarding.cf`):
+
+```ini
+pss.origin =http,https
+all.export /
+ofs.osslib libXrdPss.so
+
+# Required: restrict allowed upstream hosts
+pss.permit /* .example.org
+
+http.exthandler journalcache libXrdClJournalCacheHttpExt-5.so \
+  /etc/xrootd/journalcache-http-forwarding.ext.conf
+```
+
+**HTTP ext config** (`http/journalcache-http-forwarding.ext.conf`):
+
+```ini
+forwarding = 1
+cache = /var/tmp/journalcache/
+flat = 0
+prefix = /
+```
+
+**Client plugins** for the xrootd process (`http/journalcache-forwarding-client.conf`):
+
+```ini
+url = *
+lib = libXrdClJournalCachePlugin-5.so
+enable = true
+cache = /var/tmp/journalcache/
+
+url = http*
+lib = libXrdClHttp.so
+enable = true
+
+url = https*
+lib = libXrdClHttp.so
+enable = true
+```
+
+In forwarding mode the ext handler:
+
+- Parses the embedded `http://` / `https://` URL from each path
+- Resolves journal files using the same cache-key rules as the file plugin
+- Issues HTTP **HEAD** to that upstream for cache metadata (when libcurl is available)
+- Skips local `root://` xattr lookups (`server` / `http_origin` are ignored)
+
+PSS turns `/https://host/path` into a client open of `https://host/path`; **XrdClHttp** performs the HTTP fetch; JournalCache journals the bytes.
+
 # 8 JournalCache in a Proxy server
 
 To run a proxy server with JournalCache you create a usual proxy configuration file:

src/XrdApps/XrdClJournalCachePlugin/html/index.html

@@ -513,6 +513,19 @@ <h3>HTTP ext handler config</h3>
       <p>Origin files can publish cache policy by setting xattrs, for example:</p>
       <pre><code>xrdfs setxattr /store/data/file.root http.cache-control 'public, s-maxage=3600'
 xrdfs setxattr /store/data/file.root http.etag '"abc123"'</code></pre>
+
+      <h3>HTTP forwarding proxy (dynamic origins)</h3>
+      <p>For browser clients that embed the upstream URL in the path (<code>/https://host/path</code>), enable PSS forwarding and set <code>forwarding = 1</code> in the HTTP ext config. Example files live under <code>http/</code>: <code>journalcache-forwarding.cf</code>, <code>journalcache-http-forwarding.ext.conf</code>, <code>journalcache-forwarding-client.conf</code>.</p>
+      <pre><code># xrootd server
+pss.origin =http,https
+pss.permit /* .example.org
+http.exthandler journalcache libXrdClJournalCacheHttpExt-5.so \
+  /etc/xrootd/journalcache-http-forwarding.ext.conf
+
+# ext handler
+forwarding = 1
+cache = /var/tmp/journalcache/</code></pre>
+      <p>Load <strong>XrdClHttp</strong> alongside JournalCache in the xrootd process client plugins. PSS fetches via HTTP(S); the ext handler parses each embedded URL for journal lookup, 304, and response headers.</p>
     </section>
 
     <section id="proxy">

src/XrdApps/XrdClJournalCachePlugin/html/index.md

@@ -289,6 +289,24 @@ xrdfs setxattr /store/data/file.root http.etag '"abc123"'
 xrdfs setxattr /store/data/file.root http.last-modified 'Wed, 21 Oct 2015 07:28:00 GMT'
 ```
 
+### HTTP forwarding proxy (dynamic origins)
+
+For browser clients that embed the upstream URL in the path (`/https://host/path`), enable PSS forwarding and set `forwarding = 1` in the HTTP ext config. Example files: `http/journalcache-forwarding.cf`, `http/journalcache-http-forwarding.ext.conf`, `http/journalcache-forwarding-client.conf`.
+
+```ini
+# xrootd server
+pss.origin =http,https
+pss.permit /* .example.org
+http.exthandler journalcache libXrdClJournalCacheHttpExt-5.so \
+  /etc/xrootd/journalcache-http-forwarding.ext.conf
+
+# ext handler
+forwarding = 1
+cache = /var/tmp/journalcache/
+```
+
+Load **XrdClHttp** alongside JournalCache in the xrootd process client plugins. PSS fetches via HTTP(S); the ext handler parses each embedded URL for journal lookup, 304, and response headers.
+
 ---
 
 ## Proxy and HTTP front-end setup

src/XrdApps/XrdClJournalCachePlugin/http/ForwardingUrl.cc

@@ -0,0 +1,112 @@
+#include "http/ForwardingUrl.hh"
+#include "file/Digest.hh"
+
+#include "XrdCl/XrdClURL.hh"
+
+namespace JournalCache {
+namespace {
+
+bool startsWith(const std::string &value, const std::string &prefix) {
+  return value.size() >= prefix.size() &&
+         value.compare(0, prefix.size(), prefix) == 0;
+}
+
+std::string stripQuery(const std::string &path) {
+  const auto pos = path.find('?');
+  if (pos == std::string::npos) {
+    return path;
+  }
+  return path.substr(0, pos);
+}
+
+} // namespace
+
+namespace {
+
+std::string normalizeRemotePath(const std::string &path) {
+  if (path.empty()) {
+    return "/";
+  }
+  if (path[0] == '/') {
+    return path;
+  }
+  return "/" + path;
+}
+
+} // namespace
+
+EmbeddedFileUrl parseEmbeddedFileUrl(const std::string &path) {
+  EmbeddedFileUrl result;
+  std::string rest = stripQuery(path);
+  if (!rest.empty() && rest.front() == '/') {
+    rest.erase(0, 1);
+  }
+
+  if (!startsWith(rest, "https://") && !startsWith(rest, "http://")) {
+    return result;
+  }
+
+  XrdCl::URL parsed(rest);
+  if (!parsed.IsValid()) {
+    return result;
+  }
+
+  XrdCl::URL clean;
+  clean.SetProtocol(parsed.GetProtocol());
+  clean.SetHostName(parsed.GetHostName());
+  clean.SetPort(parsed.GetPort());
+  clean.SetPath(parsed.GetPath());
+  result.fileUrl = clean.GetURL();
+  result.valid = !result.fileUrl.empty();
+  return result;
+}
+
+std::string resolveJournalDirWithSettings(const std::string &cacheRoot,
+                                          const std::string &serverUrl,
+                                          const std::string &remotePath,
+                                          bool flatHierarchy,
+                                          const std::string &basePath) {
+  const std::string normPath = normalizeRemotePath(remotePath);
+  if (flatHierarchy) {
+    return cacheRoot + computeSHA256(serverUrl + normPath);
+  }
+
+  if (!basePath.empty()) {
+    const size_t pos = normPath.find(basePath);
+    if (pos != std::string::npos) {
+      return cacheRoot + normPath.substr(pos);
+    }
+    XrdCl::URL url(serverUrl);
+    const size_t urlPos = url.GetPath().find(basePath);
+    if (urlPos != std::string::npos) {
+      return cacheRoot + normPath;
+    }
+  }
+
+  XrdCl::URL url(serverUrl);
+  const std::string host =
+      url.GetHostName() + ":" + std::to_string(url.GetPort());
+  return cacheRoot + host + normPath;
+}
+
+std::string resolveJournalPathFromCacheKey(const std::string &cacheRoot,
+                                           const std::string &cacheKeyUrl,
+                                           bool flatHierarchy,
+                                           const std::string &basePath) {
+  if (cacheRoot.empty() || cacheKeyUrl.empty()) {
+    return {};
+  }
+
+  std::string journalDir;
+  if (flatHierarchy) {
+    journalDir = cacheRoot + computeSHA256(cacheKeyUrl);
+  } else {
+    XrdCl::URL url(cacheKeyUrl);
+    journalDir = resolveJournalDirWithSettings(cacheRoot, cacheKeyUrl,
+                                               url.GetPath(), flatHierarchy,
+                                               basePath);
+  }
+  return journalDir + "/journal";
+}
+
+} // namespace JournalCache

src/XrdApps/XrdClJournalCachePlugin/http/ForwardingUrl.hh

@@ -0,0 +1,28 @@
+#pragma once
+
+#include <string>
+
+namespace JournalCache {
+
+//! Parsed upstream URL embedded in a PSS forwarding-proxy path.
+struct EmbeddedFileUrl {
+  std::string fileUrl;
+  bool valid = false;
+};
+
+//! Parse `/https://host/path` or `/http://host/path` into a canonical file URL.
+EmbeddedFileUrl parseEmbeddedFileUrl(const std::string &path);
+
+std::string resolveJournalDirWithSettings(const std::string &cacheRoot,
+                                          const std::string &serverUrl,
+                                          const std::string &remotePath,
+                                          bool flatHierarchy,
+                                          const std::string &basePath);
+
+//! Resolve on-disk journal path using the same rules as the file plugin.
+std::string resolveJournalPathFromCacheKey(const std::string &cacheRoot,
+                                           const std::string &cacheKeyUrl,
+                                           bool flatHierarchy,
+                                           const std::string &basePath);
+
+} // namespace JournalCache