Escaping For setBody (#2810)

* fix: compute URL expressions in the target after request flow in the http client interceptor

* refactor(core): use `getAuthority` instead of `getHost` for improved URL handling and modify target configurations

- Introduced `computeDestinationExpressions` to streamline target URL modification logic.
- Improved `DispatchingInterceptor` handling of base path resolution with illegal characters.
- Enhanced `urlEncode` and `pathSeg` utility functions for safety and clarity.
- Refactored `HTTPClientInterceptor` to adjust the target flow and error messages.
- Added extensive URI tests, including custom-parsed and relative path resolution scenarios.
- Updated `applyTargetModifications` for streamlined behavior during request processing.

* feat(core): add URL encoding support for template evaluation in target URLs

- Introduced `TemplateUtil` with a method to check for template markers.
- Enhanced `TemplateExchangeExpression` to support URL encoding for dynamic templates.
- Updated `Target` to skip unnecessary evaluation for URLs without template markers.
- Added extensive tests to verify correct URL encoding across various languages (Groovy, SpEL, XPath, JSONPath).

* docs(roadmap): update URL encoding details for template evaluation in target URLs

- Documented the change to URL encode expressions inside `${}` in target URLs.
- Updated tests to improve validation coverage and logging for clearer output.

* feat(core): optimize expression handling for dynamic URLs with caching

- Introduced a `templateExpressionCache` in `Target` to avoid redundant parsing and compilation of frequently used expressions.
- Improved token parsing and evaluation logic in `TemplateExchangeExpression`, enhancing clarity and performance.
- Renamed test helper method `extracted` to `testExpression` for better readability.
- Updated test cases across multiple classes to cover new enhancements and ensure consistent behavior.

* refactor(core): remove template expression caching from `Target` and simplify logic

- Eliminated `templateExpressionCache` to reduce complexity and improve maintainability.
- Updated `computeDestinationExpressions` to use `Collectors.toList()` for a mutable list.
- Simplified `evaluateTemplate` by removing cache-related logic and comments on minor performance differences.

* feat(core): introduce `Escaping` enum for customizable URL encoding in `Target`

- Added `Escaping` enum with options `NONE`, `URL`, and `SEGMENT` to enhance placeholder escaping in URLs.
- Updated `Target` to include configurable escaping behavior and a helper method `getEscapingFunction()`.
- Extended test cases to validate new escaping options in various scenarios.

* refactor(core): enhance URI handling and extend test coverage

- Updated `URI` to leverage `var` for better readability and added JavaDoc for `resolve` method.
- Improved `ResolverMap` handling of URI schemes, including `classpath:` and `internal:`.
- Added new tests in `HTTPClientInterceptorTest` for complete path computation, including URL encoding.

* feat(core): improve handling of illegal characters in URLs and enhance template evaluation

- Added `UriIllegalCharacterDetector` for validating URI components with RFC 3986 compliance and optional extensions.
- Updated `Target` to skip template evaluation for URLs without placeholders and log warnings for insecure configurations.
- Enhanced `TemplateUtil.containsTemplateMarker` with null safety.
- Improved `RewriteInterceptor` to handle query parsing errors with more informative logging.
- Refactored tests and added new ones to validate changes.

* refactor(core): simplify `URI` logic and enhance test readability

- Refactored `removeLastSegment` method for concise logic using `Math.max()` to avoid redundant checks.
- Replaced `assertEquals(false, ...)` with `assertFalse(...)` for improved readability in tests.
- Cleaned up unused imports in `ResolverMapCombineTest`.

* feat(core): enhance URI handling and replace `pathSeg` with `pathEncode`

- Replaced `pathSeg` method with `pathEncode` for improved naming clarity and functionality in line with RFC 3986.
- Introduced `IPv6Util` for robust validation of IPv6 address text, ensuring only allowed characters are accepted.
- Refactored `Target` to include URL encoding strategy through a dedicated `escapingFunction` and removed redundant logic.
- Streamlined URI combination logic in `ResolverMap` with utility methods for handling slashes and file URIs.
- Added extensive tests covering URL path encoding, IPv6 address validation, and robustification of URI logic for edge cases.

* refactor(core): improve code readability and consistency in URI tests and utilities

- Adjusted visibility modifiers of constants to `final` for better immutability (`URITest`).
- Reordered equality assertion arguments for better readability (`TargetTest`).
- Replaced redundant comments with concise `<p/>` tags in JavaDocs for uniformity.
- Refactored tests to leverage `assertNull` for null comparisons (`FileUtilTest`).
- Removed unused method `stripIpv6BracketsIfNeeded` and redundant imports for cleaner code.
- Enhanced URI combination logic and streamlined utility imports (`DispatchingInterceptor`).

* feat(core): introduce `URIFactory` for enhanced URI creation and add comprehensive RFC 3986 compliance tests

- Added `URIFactory` to support flexible URI creation with options for illegal character handling and backslash escaping.
- Implemented extensive

* test(core): extend URI test cases for edge scenarios in IPv4 and IPv6 parsing

- Added edge case assertions for parsing IPv6 with empty ports.
- Introduced additional checks for IPv4 host and port parsing scenarios.
- Renamed test methods to reflect broader case coverage.

* feat(core): enhance template marker detection and URL evaluation in `Target` and `CallInterceptor`

- Replaced `evaluateExpressions` with `urlIsTemplate` for better semantics and clarity.
- Improved `Target` and `CallInterceptor` to skip URL evaluation when no template markers are present.
- Introduced stricter validation for configurations involving template markers and illegal characters.
- Updated relevant tests to validate changes and added new test cases for URL template evaluation.

* feat(core): add dynamic escaping to `setBody` and improve expression evaluation

- Introduced JSON escaping support for `setBody` based on `contentType`.
- Enhanced `ExchangeExpression` to centralize router retrieval logic.
- Updated tests to include validation logging for transformation scenarios.
- Fixed minor formatting inconsistencies and improved code readability.

* feat(core): add XML escaping support and enhance `getEscapingFunction`

- Introduced XML escaping using `StringEscapeUtils::escapeXml11`.
- Updated `getEscapingFunction` to handle XML content type.
- Added `XML` escaping type to `Escaping` enum.
- Enhanced utility imports and updated dependencies for XML handling.

* test(core): add JSON and Groovy null value handling tests in `SetBodyInterceptorTest`

- Added tests to verify behavior for null values in JSONPath and Groovy expressions.
- Enhanced `SetBodyInterceptorTest` with additional scenarios for content escaping.
- Updated imports for improved code readability and consistency.

* test(core): improve null value handling tests for JSONPath and Groovy in `SetBodyInterceptorTest`

- Refactored tests to consolidate null value handling for JSONPath and Groovy.
- Enhanced `SetBodyInterceptor` to handle `Object` evaluation results, ensuring compatibility and proper type handling.
- Updated helper methods to restructure and reuse test setup across scenarios.

* test(tutorials): log validation errors in `SoapArrayToJsonTutorialTest`

- Added `.log().ifValidationFails()` to improve debugging of failing test scenarios.

---------

Co-authored-by: Christian Gördes <118011644+christiangoerdes@users.noreply.github.com>

Author: predic8

core/src/main/java/com/predic8/membrane/core/interceptor/lang/SetBodyInterceptor.java

@@ -16,19 +16,26 @@
 
 import com.predic8.membrane.annot.*;
 import com.predic8.membrane.core.exchange.*;
+import com.predic8.membrane.core.http.*;
 import com.predic8.membrane.core.interceptor.*;
+import com.predic8.membrane.core.lang.*;
 import com.predic8.membrane.core.util.*;
+import com.predic8.membrane.core.util.uri.*;
+import org.jetbrains.annotations.*;
 import org.slf4j.*;
 
+import java.util.function.*;
+
 import static com.predic8.membrane.core.exceptions.ProblemDetails.*;
 import static com.predic8.membrane.core.interceptor.Interceptor.Flow.*;
 import static com.predic8.membrane.core.interceptor.Outcome.*;
 import static com.predic8.membrane.core.interceptor.Outcome.ABORT;
+import static com.predic8.membrane.core.util.uri.EscapingUtil.Escaping.NONE;
+import static com.predic8.membrane.core.util.uri.EscapingUtil.getEscapingFunction;
 import static java.nio.charset.StandardCharsets.*;
 
 /**
- * @description
- * sets the content of the HTTP message body to the specified value. The value
+ * @description sets the content of the HTTP message body to the specified value. The value
  * can be a static string, or it can be dynamically generated by an expression.
  * Different languages such as SpEL, Groovy, XMLPath or JsonPath are supported.
  * setBody does not support conditional processing or loops. When you need these features,
@@ -58,16 +65,19 @@ private Outcome handleInternal(Exchange exchange, Flow flow) {
         var msg = exchange.getMessage(flow);
         try {
             // The value is typically set from YAML there we can assume UTF-8
-            var result = exchangeExpression.evaluate(exchange, flow, String.class);
+            var result = exchangeExpression.evaluate(exchange, flow, Object.class);
             if (result == null) {
                 result = "null";
             }
-            msg.setBodyContent(result.getBytes(UTF_8));
-
-            if (contentType!=null) {
+            if (result instanceof String s) {
+                msg.setBodyContent(s.getBytes(UTF_8));
+            } else {
+                // Hope that all possibles return types are covered
+                msg.setBodyContent(result.toString().getBytes(UTF_8));
+            }
+            if (contentType != null) {
                 msg.getHeader().setContentType(contentType);
             }
-
             return CONTINUE;
         } catch (Exception e) {
             var root = ExceptionUtil.getRootCause(e);
@@ -84,6 +94,14 @@ private Outcome handleInternal(Exchange exchange, Flow flow) {
         }
     }
 
+    protected ExchangeExpression getExchangeExpression() {
+        return TemplateExchangeExpression.newInstance(this, language, expression, router,
+                getEscapingFunction(contentType).orElseGet(() -> {
+            log.warn("Turning off escaping for 'setBody'. No escaping found for content type {}. To enable escaping set 'contentType' on setBody.", contentType);
+            return getEscapingFunction(NONE);
+        }));
+    }
+
     /**
      * Sets the expression to be evaluated for modifying the HTTP message body.
      * The provided string value can represent a static text or a dynamic expression

core/src/main/java/com/predic8/membrane/core/lang/ExchangeExpression.java

@@ -23,6 +23,7 @@
 import com.predic8.membrane.core.lang.spel.*;
 import com.predic8.membrane.core.lang.xpath.*;
 import com.predic8.membrane.core.router.*;
+import org.jetbrains.annotations.*;
 
 /**
  * Language expression that takes an exchange as input
@@ -59,15 +60,18 @@ enum Language {GROOVY, SPEL, XPATH, JSONPATH}
      * @return
      */
     static ExchangeExpression expression(Interceptor interceptor, Language language, String expression) {
-        var router = interceptor != null ? interceptor.getRouter() : null;
         return switch (language) {
-            case GROOVY -> new GroovyExchangeExpression(expression, router);
-            case SPEL -> new SpELExchangeExpression(expression,null, router); // parserContext is null on purpose ${} or #{} are not needed here
-            case XPATH -> new XPathExchangeExpression(interceptor,expression, router);
-            case JSONPATH -> new JsonpathExchangeExpression(expression, router);
+            case GROOVY -> new GroovyExchangeExpression(expression, getRouter(interceptor));
+            case SPEL -> new SpELExchangeExpression(expression,null, getRouter(interceptor)); // parserContext is null on purpose ${} or #{} are not needed here
+            case XPATH -> new XPathExchangeExpression(interceptor,expression, getRouter(interceptor));
+            case JSONPATH -> new JsonpathExchangeExpression(expression, getRouter(interceptor));
         };
     }
 
+    private static @Nullable Router getRouter(Interceptor interceptor) {
+        return interceptor != null ? interceptor.getRouter() : null;
+    }
+
     /**
      * Allows to pass an Interceptor as an argument where there is no interceptor e.g. Target
      */

core/src/main/java/com/predic8/membrane/core/util/uri/EscapingUtil.java

@@ -14,9 +14,19 @@
 
 package com.predic8.membrane.core.util.uri;
 
+import com.predic8.membrane.core.http.*;
+import com.predic8.membrane.core.lang.*;
+import groovy.json.*;
+import org.apache.commons.text.StringEscapeUtils;
+import org.jetbrains.annotations.*;
+
 import java.net.*;
+import java.util.*;
 import java.util.function.*;
 
+import static com.predic8.membrane.core.http.MimeType.isJson;
+import static com.predic8.membrane.core.http.MimeType.isXML;
+import static com.predic8.membrane.core.util.uri.EscapingUtil.Escaping.*;
 import static java.lang.Character.*;
 import static java.nio.charset.StandardCharsets.*;
 
@@ -30,20 +40,36 @@ public class EscapingUtil {
      * - {@code NONE}: No escaping is applied. Strings are returned as-is.
      * - {@code URL}: Encodes strings for safe inclusion in a URL, replacing spaces and
      *   other special characters with their percent-encoded counterparts (e.g., SPACE -> +).
+     * - {@code JSON}: Escapes strings for safe inclusion in a JSON context.
+     * - {@code XML}: Escapes strings for safe inclusion in an XML context using XML 1.1 rules.
      * - {@code SEGMENT}: Encodes strings as safe URI path segments, ensuring they do not introduce
      *   path separators, query delimiters, or other unsafe characters, as per RFC 3986.
      */
     public enum Escaping {
         NONE,
         URL,
-        SEGMENT
+        SEGMENT,
+        JSON,
+        XML
+    }
+
+    public static Optional<Function<String, String>> getEscapingFunction(String mimeType) {
+        if (isJson(mimeType)) {
+            return Optional.of(getEscapingFunction(JSON));
+        }
+        if (isXML(mimeType)) {
+            return Optional.of(getEscapingFunction(XML));
+        }
+        return Optional.empty();
     }
 
     public static Function<String, String> getEscapingFunction(Escaping escaping) {
         return switch (escaping) {
             case NONE -> Function.identity();
             case URL -> s -> URLEncoder.encode(s, UTF_8);
             case SEGMENT -> EscapingUtil::pathEncode;
+            case JSON -> CommonBuiltInFunctions::toJSON; // Alternative: StringEscapeUtils::escapeJson ?
+            case XML -> StringEscapeUtils::escapeXml11;
         };
     }
 

core/src/test/java/com/predic8/membrane/core/interceptor/lang/SetBodyInterceptorTest.java

@@ -15,13 +15,16 @@
 package com.predic8.membrane.core.interceptor.lang;
 
 import com.predic8.membrane.core.exchange.*;
-import com.predic8.membrane.core.http.*;
+import com.predic8.membrane.core.lang.*;
 import com.predic8.membrane.core.router.*;
 import org.junit.jupiter.api.*;
 
 import java.net.*;
 
-import static com.predic8.membrane.core.http.Response.notImplemented;
+import static com.predic8.membrane.core.http.MimeType.*;
+import static com.predic8.membrane.core.http.Request.*;
+import static com.predic8.membrane.core.http.Response.*;
+import static com.predic8.membrane.core.lang.ExchangeExpression.Language.*;
 import static org.junit.jupiter.api.Assertions.*;
 
 /**
@@ -37,7 +40,7 @@ class SetBodyInterceptorTest {
     void setup() throws URISyntaxException {
         sbi = new SetBodyInterceptor();
 
-        exc = Request.get("/foo").buildExchange();
+        exc = get("/foo").buildExchange();
         exc.setResponse(notImplemented().body("bar").build());
     }
 
@@ -57,7 +60,43 @@ void evalOfSimpleExpression() {
         assertEquals("/foo", exc.getRequest().getBodyAsStringDecoded());
     }
 
+    /**
+     * When inserting a value from JSONPath into a JSON document like:
+     * { "a": ${.a} }
+     * and the value is null, the document should be:
+     * { "a": null }
+     */
+    @Nested
+    class Null {
+
+        @Test
+        void escapeNullJsonPath() throws URISyntaxException {
+            callSetBody(JSONPATH, "${$.a}");
+        }
+
         @Test
+        void escapeNullGroovy() throws URISyntaxException {
+            callSetBody(GROOVY, "${fn.jsonPath('$.a')}");
+        }
+
+        private void callSetBody(ExchangeExpression.Language language, String expression) throws URISyntaxException {
+            exc = setJsonSample();
+            sbi.setLanguage(language);
+            sbi.setContentType(APPLICATION_JSON_UTF8);
+            sbi.setValue(expression);
+            sbi.init(new DefaultRouter());
+            sbi.handleRequest(exc);
+            assertEquals("null", exc.getRequest().getBodyAsStringDecoded());
+        }
+
+        private Exchange setJsonSample() throws URISyntaxException {
+            return post("/foo").json("""
+                    {"a":null}
+                    """).buildExchange();
+        }
+    }
+
+    @Test
     void response() {
         sbi.setValue("SC: ${statusCode}");
         sbi.init(new DefaultRouter());

core/src/test/java/com/predic8/membrane/core/interceptor/templating/TemplateInterceptorTest.java

@@ -17,7 +17,6 @@
 import com.fasterxml.jackson.databind.*;
 import com.predic8.membrane.core.exchange.*;
 import com.predic8.membrane.core.http.*;
-import com.predic8.membrane.core.resolver.*;
 import com.predic8.membrane.core.router.*;
 import com.predic8.membrane.core.security.*;
 import com.predic8.membrane.core.util.*;
@@ -29,6 +28,7 @@
 import javax.xml.parsers.*;
 import javax.xml.xpath.*;
 import java.io.*;
+import java.net.*;
 import java.nio.file.*;
 import java.util.*;
 
@@ -51,7 +51,7 @@ public class TemplateInterceptorTest {
     Router router;
 
     @BeforeEach
-    void setUp(){
+    void setUp() {
         router = new DummyTestRouter();
         ti = new TemplateInterceptor();
         exc = new Exchange(null);
@@ -84,23 +84,23 @@ void accessJson() throws Exception {
     @SuppressWarnings("unchecked")
     @Test
     void createJson() throws Exception {
-        Exchange exchange = Request.put("/foo").contentType(APPLICATION_JSON).buildExchange();
+        var exchange = Request.put("/foo").contentType(APPLICATION_JSON).buildExchange();
 
         invokeInterceptor(exchange, """
                 {"foo":7,"bar":"baz"}
                 """, APPLICATION_JSON);
 
         assertEquals(APPLICATION_JSON, exchange.getRequest().getHeader().getContentType());
 
-        Map<String,Object> m = om.readValue(exchange.getRequest().getBodyAsStringDecoded(),Map.class);
-        assertEquals(7,m.get("foo"));
-        assertEquals("baz",m.get("bar"));
+        Map<String, Object> m = om.readValue(exchange.getRequest().getBodyAsStringDecoded(), Map.class);
+        assertEquals(7, m.get("foo"));
+        assertEquals("baz", m.get("bar"));
     }
 
     @Test
     void accessBindings() throws Exception {
         Exchange exchange = post("/foo?a=1&b=2").contentType(TEXT_PLAIN).body("vlinder").buildExchange();
-        exchange.setProperty("baz",7);
+        exchange.setProperty("baz", 7);
 
         invokeInterceptor(exchange, """
                 <% for(h in header.allHeaderFields) { %>
@@ -123,7 +123,7 @@ void accessBindings() throws Exception {
                     <%= p.key %> : <%= p.value %>
                 <% } %>
                 """, APPLICATION_JSON);
-        
+
         String body = exchange.getRequest().getBodyAsStringDecoded();
         assertTrue(body.contains("/foo"));
         assertTrue(body.contains("Flow: \"REQUEST\""));
@@ -201,15 +201,15 @@ void contentTypeTestOther() {
     @Test
     void contentTypeTestJson() {
         setAndHandleRequest("json/template_test.json");
-        assertEquals(APPLICATION_JSON,exc.getRequest().getHeader().getContentType());
+        assertEquals(APPLICATION_JSON, exc.getRequest().getHeader().getContentType());
     }
 
     @Test
     void contentTypeTestNoXml() {
         ti.setSrc("normal text");
         ti.init(router);
         ti.handleRequest(exc);
-        assertEquals(TEXT_PLAIN,exc.getRequest().getHeader().getContentType());
+        assertEquals(TEXT_PLAIN, exc.getRequest().getHeader().getContentType());
     }
 
     @Test
@@ -223,7 +223,7 @@ void testPrettify() {
 
         ti.setContentType(APPLICATION_JSON);
         ti.setSrc(inputJson);
-        ti.setPretty( TRUE);
+        ti.setPretty(TRUE);
         ti.init();
         assertArrayEquals(expectedPrettyJson.getBytes(UTF_8), ti.prettify(inputJson.getBytes(UTF_8)));
     }
@@ -246,14 +246,40 @@ void builtInFunctions() {
         exc.setProperty(SECURITY_SCHEMES, List.of(new BasicHttpSecurityScheme().username("alice")));
         ti.setContentType(APPLICATION_JSON);
         ti.setSrc("""
-        { "foo": ${user()} }
-        """);
+                { "foo": ${user()} }
+                """);
         ti.init(router);
         ti.handleRequest(exc);
 
         assertTrue(exc.getRequest().getBodyAsStringDecoded().contains("alice"));
     }
 
+    /**
+     * When inserting a value from JSONPath into a JSON document like:
+     * { "a": ${.a} }
+     * and the value is null, the document should be:
+     * { "a": null }
+     */
+    @Nested
+    class Null {
+
+        @Test
+        void escapeNull() throws URISyntaxException {
+            exc = setJsonSample();
+            ti.setContentType(APPLICATION_JSON_UTF8);
+            ti.setSrc("${fn.jsonPath('$.a')}");
+            ti.init(new DefaultRouter());
+            ti.handleRequest(exc);
+            assertEquals("null", exc.getRequest().getBodyAsStringDecoded());
+        }
+
+        private Exchange setJsonSample() throws URISyntaxException {
+            return post("/foo").json("""
+                    {"a":null}
+                    """).buildExchange();
+        }
+    }
+
     private void setAndHandleRequest(String location) {
         ti.setLocation(Paths.get("src/test/resources/" + location).toString());
         ti.init(router);

core/src/test/java/com/predic8/membrane/core/util/xml/NormalizeXMLForJsonUtilTest.java

@@ -1,3 +1,17 @@
+/* Copyright 2026 predic8 GmbH, www.predic8.com
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License. */
+
 package com.predic8.membrane.core.util.xml;
 
 import com.fasterxml.jackson.databind.*;

distribution/src/test/java/com/predic8/membrane/tutorials/soap/WSDLRewriterTutorialTest.java

@@ -59,6 +59,7 @@ void soapCall() throws IOException {
         .when()
             .post("http://localhost:%d/my-service".formatted(getPort()))
         .then()
+            .log().ifValidationFails()
             .body("Envelope.Body.getCityResponse.population", equalTo("34665600"));
          // @formatter:on
     }

distribution/src/test/java/com/predic8/membrane/tutorials/transformation/RestGetToSoapTutorialTest.java

@@ -33,6 +33,7 @@ void restGetIsConvertedToSoapAndBackToJson() {
         .when()
             .get("http://localhost:2000/cities/Bielefeld")
         .then()
+            .log().ifValidationFails()
             .statusCode(200)
             .body("country", equalTo("Germany"))
             .body("population", greaterThan(0));