Skip to content

Commit 267aa2d

Browse files
joseegman-idoneeajoseegarciaclaude
authored
Load API specifications from a remote URL with authentication (Apicurio Registry / HTTP) (#385)
* Support loading specifications from a remote URL (Apicurio / HTTP) `specFile.filePath` may now be an http/https/ftp/file URL; the spec is fetched from that URL at generation time instead of the filesystem/classpath. This enables consuming specs published in an Apicurio Registry (artifacts served over HTTP) and any URL-addressable source, for both the OpenAPI and AsyncAPI generators. - PathUtil.isRemoteUri detects URL-scheme paths. - SchemaUtil.readFile fetches remote specs directly (shared main-spec reader for both generators). - New RemoteFileLocation resolves external $refs relative to a remote spec's URL. - AsyncApiGenerator / BaseAsyncApiHandler resolve remote spec locations. - ApiTool.nodeFromFile handles URL references. - OpenApiGenerator uses the URL parent as the $ref base when the spec is remote. Adds PathUtilTest and a SchemaUtil URL-loading test (file:// URL, network-free). Documents the Apicurio/URL usage in the README. Bumps version 6.5.1 -> 6.6.0 (new feature). Closes joseegman-idoneea#7 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Harden remote spec loading: timeouts and narrower scheme allowlist - Add bounded connect (15s) / read (30s) timeouts to all URL fetches via PathUtil.openUrlStream, so an unresponsive remote spec server cannot hang the build. Timeouts are harmless for file:/jar: schemes. - Drop `ftp` from the accepted URL schemes (plaintext, effectively unused for API specs) — remote specs are now http/https/file only, reducing attack surface. - Document TLS expectations for https registries with internal/self-signed certs (must be trusted by the build JVM; validation is not disabled). Full multiapi-engine suite: 113/113 (mvn clean test). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Authenticate against protected remote registries (Apicurio) Adds authentication for remote spec fetching, resolving credentials from system properties (preferred) or environment variables - never from build files: bearer token, basic auth, and an arbitrary header (e.g. X-Registry-ApiKey). Headers are applied only to http/https fetches. Optional host scoping (scs.multiapi.remote.host) sends credentials only to the configured host, so a token is never leaked to a different host via an external $ref or cross-host redirect. Credentials are never logged; TLS validation is unchanged. Adds PathUtilTest coverage and documents the setup in the README. Full multiapi-engine suite: 118/118 (mvn clean test). Closes joseegman-idoneea#8 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix Codacy CodeStyle issues in remote-spec loading - Replace deprecated new URL(String) with URI.create(...).toURL() (ApiTool, SchemaUtil) - Remove fully-qualified names (java.net.URL, StandardCharsets in test) - Route AsyncApi remote read through PathUtil.openUrlStream (timeouts + shorter line) - Wrap long lines Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix Codacy markdownlint issues in README (MD013 line length, MD032 list spacing) The 17 Codacy issues were all markdownlint on the new README section: lines over 80 chars and a list not surrounded by blank lines. Rewrap prose to <=80, replace the wide auth table with a wrapped bullet list, and add blank lines around lists. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: joseegarcia <jose.garcia@disashop.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent d6a2cff commit 267aa2d

13 files changed

Lines changed: 362 additions & 10 deletions

File tree

‎README.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -834,3 +834,69 @@ from a local JAR containing `contracts/event-api.yml` in its resources:
834834
</dependencies>
835835
</plugin>
836836
```
837+
838+
## Loading specifications from a remote URL (Apicurio Registry, HTTP)
839+
840+
`filePath` also accepts a remote URL (`http`, `https` or `file` scheme). When
841+
it does, the spec is downloaded at generation time instead of being read from
842+
the filesystem or the classpath. This works for both OpenAPI and AsyncAPI
843+
specs. Remote fetches use bounded connect/read timeouts so an unresponsive
844+
server cannot hang the build.
845+
846+
This is the mechanism to consume a spec published in an **Apicurio Registry**,
847+
whose artifacts are served over HTTP. Point `filePath` at the artifact's
848+
content endpoint, for example:
849+
850+
```xml
851+
<specFile>
852+
<filePath>https://my-apicurio-host/apis/registry/v2/groups/default/artifacts/my-api</filePath>
853+
<apiPackage>com.sngular.apigenerator.openapi.api</apiPackage>
854+
<modelPackage>com.sngular.apigenerator.openapi.model</modelPackage>
855+
</specFile>
856+
```
857+
858+
Gradle:
859+
860+
```groovy
861+
filePath = 'https://my-apicurio-host/apis/registry/v2/groups/default/artifacts/my-api'
862+
```
863+
864+
### Authenticated registries
865+
866+
For a protected registry (for example an Apicurio Registry with security
867+
enabled), credentials are read from **system properties** (preferred) or
868+
**environment variables** — never from the build files — and sent as request
869+
headers on `http`/`https` fetches. Each mechanism has a system property and an
870+
equivalent environment variable:
871+
872+
- **Bearer token**: `scs.multiapi.remote.token` /
873+
`SCS_MULTIAPI_REMOTE_TOKEN`.
874+
- **Basic auth**: `scs.multiapi.remote.user` + `scs.multiapi.remote.password`
875+
(env `SCS_MULTIAPI_REMOTE_USER` / `SCS_MULTIAPI_REMOTE_PASSWORD`).
876+
- **Custom header**: `scs.multiapi.remote.header.name` +
877+
`scs.multiapi.remote.header.value` (env `SCS_MULTIAPI_REMOTE_HEADER_NAME` /
878+
`SCS_MULTIAPI_REMOTE_HEADER_VALUE`), e.g. an `X-Registry-ApiKey`.
879+
- **Restrict credentials to a host**: `scs.multiapi.remote.host`
880+
(env `SCS_MULTIAPI_REMOTE_HOST`).
881+
882+
A bearer token takes precedence over basic auth; the custom header is
883+
additive. Example (bearer token from the CI environment):
884+
885+
```bash
886+
export SCS_MULTIAPI_REMOTE_TOKEN="$APICURIO_TOKEN"
887+
export SCS_MULTIAPI_REMOTE_HOST="my-apicurio-host" # optional but recommended
888+
mvn generate-sources
889+
```
890+
891+
Notes:
892+
893+
- Set `scs.multiapi.remote.host` to the registry host so the token is sent
894+
**only** to that host and never leaked to a different host reached through
895+
an external `$ref` or a cross-host redirect.
896+
- Provide credentials via CI secrets / environment variables; they are never
897+
logged.
898+
- External `$ref`s are resolved relative to the spec's URL when the spec is
899+
remote.
900+
- For an `https` registry using an internally-issued or self-signed
901+
certificate, the certificate must be trusted by the JVM running the build
902+
(for example imported into its truststore); validation is not disabled.

‎multiapi-engine/pom.xml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
<groupId>com.sngular</groupId>
66
<artifactId>multiapi-engine</artifactId>
7-
<version>6.5.3</version>
7+
<version>6.6.0</version>
88
<packaging>jar</packaging>
99

1010
<properties>

‎multiapi-engine/src/main/java/com/sngular/api/generator/plugin/asyncapi/AsyncApiGenerator.java‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
import com.sngular.api.generator.plugin.common.files.ClasspathFileLocation;
2727
import com.sngular.api.generator.plugin.common.files.DirectoryFileLocation;
2828
import com.sngular.api.generator.plugin.common.files.FileLocation;
29+
import com.sngular.api.generator.plugin.common.files.RemoteFileLocation;
2930
import com.sngular.api.generator.plugin.common.tools.PathUtil;
3031
import lombok.extern.slf4j.Slf4j;
3132
import org.apache.commons.lang3.tuple.ImmutablePair;
@@ -85,6 +86,16 @@ public final void processFileSpec(final List<SpecFile> specsListFile) {
8586

8687
private static Pair<InputStream, FileLocation> resolveYmlLocation(final String ymlFilePath) throws FileNotFoundException {
8788
log.debug("Resolving YAML file location:{}", ymlFilePath);
89+
if (PathUtil.isRemoteUri(ymlFilePath)) {
90+
log.debug("Loading spec from remote URL");
91+
try {
92+
final URI uri = URI.create(ymlFilePath);
93+
final InputStream remoteStream = PathUtil.openUrlStream(uri.toURL());
94+
return new ImmutablePair<>(remoteStream, new RemoteFileLocation(uri.resolve(".")));
95+
} catch (final IOException e) {
96+
throw new FileNotFoundException("Could not open remote YAML file: " + ymlFilePath);
97+
}
98+
}
8899
final InputStream classPathInput = AsyncApiGenerator.class.getClassLoader().getResourceAsStream(ymlFilePath);
89100
final InputStream ymlFile;
90101
final FileLocation ymlParentPath;

‎multiapi-engine/src/main/java/com/sngular/api/generator/plugin/asyncapi/handler/BaseAsyncApiHandler.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@
2121
import com.sngular.api.generator.plugin.asyncapi.parameter.SpecFile;
2222
import com.sngular.api.generator.plugin.asyncapi.template.TemplateFactory;
2323
import com.sngular.api.generator.plugin.common.files.ClasspathFileLocation;
24+
import com.sngular.api.generator.plugin.common.files.RemoteFileLocation;
25+
import com.sngular.api.generator.plugin.common.tools.PathUtil;
2426
import com.sngular.api.generator.plugin.common.files.DirectoryFileLocation;
2527
import com.sngular.api.generator.plugin.common.files.FileLocation;
2628
import com.sngular.api.generator.plugin.common.model.CommonSpecFile;
@@ -108,6 +110,9 @@ protected BaseAsyncApiHandler(
108110
}
109111

110112
protected static FileLocation resolveYmlLocation(final String ymlFilePath) throws IOException, URISyntaxException {
113+
if (PathUtil.isRemoteUri(ymlFilePath)) {
114+
return new RemoteFileLocation(URI.create(ymlFilePath).resolve("."));
115+
}
111116
final var classPathInput = BaseAsyncApiHandler.class.getClassLoader().getResource(ymlFilePath);
112117
if (Objects.nonNull(classPathInput)) {
113118
return new ClasspathFileLocation(getParentUri(classPathInput.toURI()));

‎multiapi-engine/src/main/java/com/sngular/api/generator/plugin/common/files/RemoteFileLocation.java‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
/*
2+
* This Source Code Form is subject to the terms of the Mozilla Public
3+
* * License, v. 2.0. If a copy of the MPL was not distributed with this
4+
* * file, You can obtain one at https://mozilla.org/MPL/2.0/.
5+
*/
6+
7+
package com.sngular.api.generator.plugin.common.files;
8+
9+
import java.io.IOException;
10+
import java.io.InputStream;
11+
import java.net.URI;
12+
import java.net.URL;
13+
14+
import com.sngular.api.generator.plugin.common.tools.PathUtil;
15+
16+
/**
17+
* {@link FileLocation} for specifications loaded from a remote URL (e.g. an Apicurio Registry
18+
* artifact served over HTTP). Relative sibling files (external {@code $ref}s) are resolved
19+
* against the spec's base URI and fetched over the same protocol.
20+
*/
21+
public class RemoteFileLocation implements FileLocation {
22+
23+
private final URI baseUri;
24+
25+
public RemoteFileLocation(final URI baseUri) {
26+
this.baseUri = baseUri;
27+
}
28+
29+
@Override
30+
public InputStream getFileAtLocation(final String filename) throws IOException {
31+
final URL target = baseUri.resolve(filename).toURL();
32+
return PathUtil.openUrlStream(target);
33+
}
34+
35+
@Override
36+
public URI path() {
37+
return baseUri;
38+
}
39+
}

‎multiapi-engine/src/main/java/com/sngular/api/generator/plugin/common/tools/ApiTool.java‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
import java.io.InputStream;
66
import java.math.BigDecimal;
77
import java.math.BigInteger;
8+
import java.net.URI;
89
import java.util.ArrayList;
910
import java.util.Arrays;
1011
import java.util.Collections;
@@ -453,8 +454,10 @@ public static boolean hasComponents(final JsonNode node) {
453454

454455
public static JsonNode nodeFromFile(final FileLocation ymlParent, final String filePath, final FactoryTypeEnum factoryTypeEnum) throws IOException {
455456
final InputStream file;
456-
// Check if path is absolute first
457-
if (PathUtil.isAbsolutePath(filePath)) {
457+
// Remote references (http/https/ftp/file URLs) are fetched directly from their URL.
458+
if (PathUtil.isRemoteUri(filePath)) {
459+
file = PathUtil.openUrlStream(URI.create(filePath).toURL());
460+
} else if (PathUtil.isAbsolutePath(filePath)) {
458461
// For absolute paths, open directly
459462
file = new FileInputStream(filePath);
460463
} else if (filePath.startsWith(PACKAGE_SEPARATOR_STR) || filePath.matches("^\\w.*$")) {

‎multiapi-engine/src/main/java/com/sngular/api/generator/plugin/common/tools/PathUtil.java‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,16 @@
66

77
package com.sngular.api.generator.plugin.common.tools;
88

9+
import java.io.IOException;
10+
import java.io.InputStream;
11+
import java.net.URL;
12+
import java.net.URLConnection;
13+
import java.nio.charset.StandardCharsets;
914
import java.nio.file.InvalidPathException;
1015
import java.nio.file.Paths;
16+
import java.util.Base64;
17+
import java.util.LinkedHashMap;
18+
import java.util.Map;
1119

1220
import org.apache.commons.lang3.StringUtils;
1321

@@ -16,6 +24,12 @@
1624
*/
1725
public final class PathUtil {
1826

27+
/** Connection timeout (ms) applied when fetching a remote specification. */
28+
public static final int REMOTE_CONNECT_TIMEOUT_MS = 15_000;
29+
30+
/** Read timeout (ms) applied when fetching a remote specification. */
31+
public static final int REMOTE_READ_TIMEOUT_MS = 30_000;
32+
1933
private PathUtil() {
2034
// Utility class
2135
}
@@ -37,4 +51,92 @@ public static boolean isAbsolutePath(final String filePath) {
3751
return false;
3852
}
3953
}
54+
55+
/**
56+
* Checks whether a spec path points to a remote/URL location that must be fetched via a
57+
* {@link java.net.URL} stream rather than the filesystem or classpath. Enables loading
58+
* specifications from HTTP(S) endpoints such as an Apicurio Registry artifact.
59+
*
60+
* @param filePath the spec path to check
61+
* @return true if the path uses an http, https or file scheme
62+
*/
63+
public static boolean isRemoteUri(final String filePath) {
64+
return StringUtils.isNotEmpty(filePath)
65+
&& filePath.matches("^(?i)(https?|file)://.*");
66+
}
67+
68+
/**
69+
* Opens a stream to a URL with bounded connect/read timeouts, so an unresponsive remote
70+
* spec server (e.g. an Apicurio Registry) cannot hang the build indefinitely. Timeouts are
71+
* harmless for non-network schemes such as {@code file:} and {@code jar:}. For {@code http}/
72+
* {@code https} URLs any configured authentication headers (see {@link #remoteAuthHeaders})
73+
* are applied.
74+
*
75+
* @param url the URL to open
76+
* @return the input stream
77+
* @throws IOException if the connection cannot be established or times out
78+
*/
79+
public static InputStream openUrlStream(final URL url) throws IOException {
80+
final URLConnection connection = url.openConnection();
81+
connection.setConnectTimeout(REMOTE_CONNECT_TIMEOUT_MS);
82+
connection.setReadTimeout(REMOTE_READ_TIMEOUT_MS);
83+
final String protocol = url.getProtocol();
84+
if ("http".equalsIgnoreCase(protocol) || "https".equalsIgnoreCase(protocol)) {
85+
remoteAuthHeaders(url.getHost()).forEach(connection::setRequestProperty);
86+
}
87+
return connection.getInputStream();
88+
}
89+
90+
/**
91+
* Resolves the authentication headers to attach when fetching a remote spec, from system
92+
* properties (preferred) or environment variables. Credentials are never read from the build
93+
* files. Supported schemes (first match wins for {@code Authorization}):
94+
* <ul>
95+
* <li>Bearer token: {@code scs.multiapi.remote.token} / {@code SCS_MULTIAPI_REMOTE_TOKEN}</li>
96+
* <li>Basic auth: {@code scs.multiapi.remote.user}(+{@code .password}) /
97+
* {@code SCS_MULTIAPI_REMOTE_USER}(+{@code _PASSWORD})</li>
98+
* </ul>
99+
* plus an optional arbitrary header ({@code scs.multiapi.remote.header.name}/{@code .value} or
100+
* {@code SCS_MULTIAPI_REMOTE_HEADER_NAME}/{@code _VALUE}), e.g. an {@code X-Registry-ApiKey}.
101+
*
102+
* <p>When {@code scs.multiapi.remote.host} / {@code SCS_MULTIAPI_REMOTE_HOST} is set, credentials
103+
* are only sent to that host, so a token is never leaked to a different host reached through an
104+
* external {@code $ref} or a cross-host redirect.
105+
*
106+
* @param host the host of the URL being fetched
107+
* @return the headers to apply (empty if none configured, or scoped out by host)
108+
*/
109+
public static Map<String, String> remoteAuthHeaders(final String host) {
110+
final Map<String, String> headers = new LinkedHashMap<>();
111+
112+
final String scopedHost = config("scs.multiapi.remote.host", "SCS_MULTIAPI_REMOTE_HOST");
113+
if (StringUtils.isNotBlank(scopedHost) && !scopedHost.equalsIgnoreCase(host)) {
114+
return headers;
115+
}
116+
117+
final String token = config("scs.multiapi.remote.token", "SCS_MULTIAPI_REMOTE_TOKEN");
118+
final String user = config("scs.multiapi.remote.user", "SCS_MULTIAPI_REMOTE_USER");
119+
if (StringUtils.isNotBlank(token)) {
120+
headers.put("Authorization", "Bearer " + token);
121+
} else if (StringUtils.isNotBlank(user)) {
122+
final String password = StringUtils.defaultString(
123+
config("scs.multiapi.remote.password", "SCS_MULTIAPI_REMOTE_PASSWORD"));
124+
final String credentials = user + ":" + password;
125+
final String basic = Base64.getEncoder().encodeToString(credentials.getBytes(StandardCharsets.UTF_8));
126+
headers.put("Authorization", "Basic " + basic);
127+
}
128+
129+
final String headerName = config("scs.multiapi.remote.header.name", "SCS_MULTIAPI_REMOTE_HEADER_NAME");
130+
final String headerValue = config("scs.multiapi.remote.header.value", "SCS_MULTIAPI_REMOTE_HEADER_VALUE");
131+
if (StringUtils.isNotBlank(headerName) && StringUtils.isNotBlank(headerValue)) {
132+
headers.put(headerName, headerValue);
133+
}
134+
135+
return headers;
136+
}
137+
138+
private static String config(final String systemProperty, final String environmentVariable) {
139+
final String fromProperty = System.getProperty(systemProperty);
140+
return StringUtils.isNotBlank(fromProperty) ? fromProperty : System.getenv(environmentVariable);
141+
}
40142
}

‎multiapi-engine/src/main/java/com/sngular/api/generator/plugin/common/tools/SchemaUtil.java‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,12 @@ private static String readFile(final URI rootFilePath, final String filePath) th
137137
throw new IllegalArgumentException("File Path cannot be empty");
138138
}
139139

140+
// Remote specifications (http/https/ftp/file URLs, e.g. an Apicurio Registry artifact) are
141+
// fetched directly from their URL, bypassing classpath and filesystem resolution.
142+
if (PathUtil.isRemoteUri(filePath)) {
143+
return readFromUrl(URI.create(filePath).toURL());
144+
}
145+
140146
// Normalize the incoming filePath: remove leading './' and replace backslashes with forward slashes
141147
final String cleaned = cleanUpPath(filePath).replace('\\', '/');
142148

@@ -161,8 +167,12 @@ private static String readFile(final URI rootFilePath, final String filePath) th
161167
}
162168
}
163169
}
170+
return readFromUrl(fileURL);
171+
}
172+
173+
private static String readFromUrl(final URL fileURL) {
164174
final var sb = new StringBuilder();
165-
try (BufferedReader reader = new BufferedReader(new InputStreamReader(fileURL.openStream()))) {
175+
try (BufferedReader reader = new BufferedReader(new InputStreamReader(PathUtil.openUrlStream(fileURL)))) {
166176
String inputLine;
167177
while ((inputLine = reader.readLine()) != null) {
168178
sb.append(inputLine).append(System.lineSeparator());

‎multiapi-engine/src/main/java/com/sngular/api/generator/plugin/openapi/OpenApiGenerator.java‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88

99
import java.io.File;
1010
import java.io.IOException;
11+
import java.net.URI;
1112
import java.nio.file.Path;
1213
import java.util.ArrayList;
1314
import java.util.HashSet;
@@ -23,6 +24,7 @@
2324
import com.sngular.api.generator.plugin.common.tools.ApiTool;
2425
import com.sngular.api.generator.plugin.common.tools.MapperContentUtil;
2526
import com.sngular.api.generator.plugin.common.tools.MapperUtil;
27+
import com.sngular.api.generator.plugin.common.tools.PathUtil;
2628
import com.sngular.api.generator.plugin.exception.GeneratorTemplateException;
2729
import com.sngular.api.generator.plugin.openapi.exception.DuplicateModelClassException;
2830
import com.sngular.api.generator.plugin.openapi.model.AuthObject;
@@ -99,7 +101,10 @@ private void processFile(final SpecFile specFile) {
99101

100102
final JsonNode openAPI = OpenApiUtil.getPojoFromSpecFile(baseDir, specFile);
101103
OpenApiUtil.mergeWebhooksIntoPaths(openAPI);
102-
OpenApiUtil.solvePathRefs(openAPI, baseDir.resolve(specFile.getFilePath()).getParent().toUri());
104+
final URI specBaseUri = PathUtil.isRemoteUri(specFile.getFilePath())
105+
? URI.create(specFile.getFilePath()).resolve(".")
106+
: baseDir.resolve(specFile.getFilePath()).getParent().toUri();
107+
OpenApiUtil.solvePathRefs(openAPI, specBaseUri);
103108
final String clientPackage = specFile.getClientPackage();
104109

105110
if (specFile.isCallMode()) {

0 commit comments

Comments
 (0)