|
| 1 | +--- |
| 2 | +name: new-java-e2e-test-yaml-and-test |
| 3 | +description: "Use this skill when creating a new Java E2E integration test (failsafe IT) that requires a new replay proxy YAML snapshot file in test/snapshots/" |
| 4 | +--- |
| 5 | + |
| 6 | +# Creating a New Java E2E Test with a Replay Proxy YAML Snapshot |
| 7 | + |
| 8 | +This skill covers the complete workflow for adding a new Java failsafe |
| 9 | +integration test backed by a handcrafted YAML snapshot for the replay proxy. |
| 10 | + |
| 11 | +## Overview |
| 12 | + |
| 13 | +The Java E2E tests use a **replay proxy** (`test/harness/replayingCapiProxy.ts`) |
| 14 | +that intercepts HTTP calls to the Copilot API and returns pre-recorded responses |
| 15 | +from YAML snapshot files. This avoids needing real authentication in CI. |
| 16 | + |
| 17 | +**Key constraint:** Java's `CapiProxy.java` always sets `GITHUB_ACTIONS=true` |
| 18 | +(line 104), which forces the replay proxy into read-only mode. You **cannot** |
| 19 | +record snapshots by running Java tests — you must handcraft the YAML. |
| 20 | + |
| 21 | +## Step-by-Step Workflow |
| 22 | + |
| 23 | +### Step 1: Choose a snapshot category and test name |
| 24 | + |
| 25 | +- Category = a directory under `test/snapshots/` (e.g., `system_message_sections`) |
| 26 | +- Test name = the Java method name, converted to snake_case (e.g., |
| 27 | + `shouldUseReplacedIdentitySectionInResponse` → `should_use_replaced_identity_section_in_response`) |
| 28 | +- Resulting file: `test/snapshots/<category>/<snake_case_test_name>.yaml` |
| 29 | + |
| 30 | +### Step 2: Create the YAML snapshot file |
| 31 | + |
| 32 | +The format is: |
| 33 | + |
| 34 | +```yaml |
| 35 | +models: |
| 36 | + - claude-sonnet-4.5 |
| 37 | +conversations: |
| 38 | + - messages: |
| 39 | + - role: system |
| 40 | + content: ${system} |
| 41 | + - role: user |
| 42 | + content: <the exact prompt your test will send> |
| 43 | + - role: assistant |
| 44 | + content: <the response the proxy will return> |
| 45 | +``` |
| 46 | +
|
| 47 | +**Rules:** |
| 48 | +- `${system}` is a placeholder that matches ANY system message content |
| 49 | +- `${workdir}` in tool arguments is substituted with the actual temp workDir |
| 50 | +- Each conversation entry represents one request-response exchange |
| 51 | +- For multi-turn, add multiple conversation entries |
| 52 | +- For tool calls, include `tool_calls` on assistant messages and `role: tool` for results |
| 53 | +- The user content must **exactly match** what your test sends (after normalization) |
| 54 | + |
| 55 | +### Step 3: Create the Java IT test class |
| 56 | + |
| 57 | +Place it in `java/src/test/java/com/github/copilot/` with an `IT` suffix |
| 58 | +(e.g., `MyFeatureIT.java`). The failsafe plugin picks up `*IT.java` files. |
| 59 | + |
| 60 | +**Template:** |
| 61 | + |
| 62 | +```java |
| 63 | +package com.github.copilot; |
| 64 | +
|
| 65 | +import static org.junit.jupiter.api.Assertions.*; |
| 66 | +
|
| 67 | +import java.util.concurrent.TimeUnit; |
| 68 | +
|
| 69 | +import org.junit.jupiter.api.AfterAll; |
| 70 | +import org.junit.jupiter.api.BeforeAll; |
| 71 | +import org.junit.jupiter.api.Test; |
| 72 | +
|
| 73 | +import com.github.copilot.generated.AssistantMessageEvent; |
| 74 | +import com.github.copilot.rpc.MessageOptions; |
| 75 | +import com.github.copilot.rpc.PermissionHandler; |
| 76 | +import com.github.copilot.rpc.SessionConfig; |
| 77 | +// ... other imports as needed |
| 78 | +
|
| 79 | +class MyFeatureIT { |
| 80 | +
|
| 81 | + private static E2ETestContext ctx; |
| 82 | +
|
| 83 | + @BeforeAll |
| 84 | + static void setUp() throws Exception { |
| 85 | + ctx = E2ETestContext.create(); |
| 86 | + } |
| 87 | +
|
| 88 | + @AfterAll |
| 89 | + static void tearDown() throws Exception { |
| 90 | + if (ctx != null) { |
| 91 | + ctx.close(); |
| 92 | + } |
| 93 | + } |
| 94 | +
|
| 95 | + @Test |
| 96 | + void myTestMethod() throws Exception { |
| 97 | + // 1. Configure the proxy to use your snapshot |
| 98 | + ctx.configureForTest("my_category", "my_test_method"); |
| 99 | +
|
| 100 | + // 2. Create a client (uses fake token + proxy automatically) |
| 101 | + try (CopilotClient client = ctx.createClient()) { |
| 102 | +
|
| 103 | + // 3. Create a session with desired config |
| 104 | + CopilotSession session = client.createSession(new SessionConfig() |
| 105 | + .setOnPermissionRequest(PermissionHandler.APPROVE_ALL)) |
| 106 | + .get(30, TimeUnit.SECONDS); |
| 107 | +
|
| 108 | + try { |
| 109 | + // 4. Send the prompt (must match YAML exactly) |
| 110 | + AssistantMessageEvent response = session |
| 111 | + .sendAndWait(new MessageOptions().setPrompt("Your prompt here"), 60_000) |
| 112 | + .get(90, TimeUnit.SECONDS); |
| 113 | +
|
| 114 | + // 5. Assert on the response |
| 115 | + assertNotNull(response); |
| 116 | + String content = response.getData().content(); |
| 117 | + assertTrue(content.contains("expected text")); |
| 118 | + } finally { |
| 119 | + session.close(); |
| 120 | + } |
| 121 | + } |
| 122 | + } |
| 123 | +} |
| 124 | +``` |
| 125 | + |
| 126 | +### Step 4: Verify |
| 127 | + |
| 128 | +```sh |
| 129 | +cd java |
| 130 | +mvn spotless:apply |
| 131 | +mvn failsafe:integration-test -Dit.test="MyFeatureIT#myTestMethod" -Denforcer.skip=true |
| 132 | +``` |
| 133 | + |
| 134 | +Then run the full build to confirm no regressions: |
| 135 | + |
| 136 | +```sh |
| 137 | +mvn clean verify |
| 138 | +``` |
| 139 | + |
| 140 | +## Key Classes and Files |
| 141 | + |
| 142 | +| What | Where | |
| 143 | +|------|-------| |
| 144 | +| Test context (manages proxy, workDir, CLI) | `java/src/test/java/com/github/copilot/E2ETestContext.java` | |
| 145 | +| Java proxy wrapper | `java/src/test/java/com/github/copilot/CapiProxy.java` | |
| 146 | +| Replay proxy (TypeScript) | `test/harness/replayingCapiProxy.ts` | |
| 147 | +| Proxy server entry point | `test/harness/server.ts` | |
| 148 | +| Snapshot files | `test/snapshots/<category>/<name>.yaml` | |
| 149 | +| Existing IT tests for reference | `java/src/test/java/com/github/copilot/*IT.java` | |
| 150 | + |
| 151 | +## How the Proxy Matches Requests |
| 152 | + |
| 153 | +1. The proxy normalizes the incoming request's messages |
| 154 | +2. It compares against each conversation in the YAML: |
| 155 | + - System message matches if YAML has `${system}` (wildcard) |
| 156 | + - User messages are compared by content (exact text match) |
| 157 | + - Tool results are compared after normalizing `${workdir}` paths |
| 158 | +3. If a match is found, the **last assistant message** in that conversation is returned |
| 159 | +4. If no match, in CI mode (`GITHUB_ACTIONS=true`) it errors with "No cached response found" |
| 160 | + |
| 161 | +## YAML Format for Tool Calls |
| 162 | + |
| 163 | +If your test involves tool use: |
| 164 | + |
| 165 | +```yaml |
| 166 | +conversations: |
| 167 | + # First exchange: model wants to call a tool |
| 168 | + - messages: |
| 169 | + - role: system |
| 170 | + content: ${system} |
| 171 | + - role: user |
| 172 | + content: Read the file test.txt |
| 173 | + - role: assistant |
| 174 | + content: I'll read that file. |
| 175 | + tool_calls: |
| 176 | + - id: toolcall_0 |
| 177 | + type: function |
| 178 | + function: |
| 179 | + name: view |
| 180 | + arguments: '{"path":"${workdir}/test.txt"}' |
| 181 | + # Second exchange: after tool result is provided, model gives final answer |
| 182 | + - messages: |
| 183 | + - role: system |
| 184 | + content: ${system} |
| 185 | + - role: user |
| 186 | + content: Read the file test.txt |
| 187 | + - role: assistant |
| 188 | + content: I'll read that file. |
| 189 | + tool_calls: |
| 190 | + - id: toolcall_0 |
| 191 | + type: function |
| 192 | + function: |
| 193 | + name: view |
| 194 | + arguments: '{"path":"${workdir}/test.txt"}' |
| 195 | + - role: tool |
| 196 | + tool_call_id: toolcall_0 |
| 197 | + content: "1. Hello world!" |
| 198 | + - role: assistant |
| 199 | + content: The file test.txt contains "Hello world!" |
| 200 | +``` |
| 201 | + |
| 202 | +**Important:** When the model calls tools like `view`, the CLI actually executes |
| 203 | +them locally. The file must exist in the test's workDir. Create it in your test |
| 204 | +before sending the prompt: |
| 205 | + |
| 206 | +```java |
| 207 | +Files.writeString(ctx.getWorkDir().resolve("test.txt"), "Hello world!\n"); |
| 208 | +``` |
| 209 | + |
| 210 | +## Common Pitfalls |
| 211 | + |
| 212 | +1. **Prompt mismatch** — The user content in YAML must exactly match what |
| 213 | + `session.sendAndWait(new MessageOptions().setPrompt("..."))` sends. |
| 214 | +2. **Forgetting `${system}`** — Always use `${system}` for the system role content |
| 215 | + unless testing a specific system message matching scenario. |
| 216 | +3. **Tool execution** — If the snapshot has the model calling `view` or other |
| 217 | + built-in tools, the CLI will actually execute those tools. Files must exist. |
| 218 | +4. **Snake_case conversion** — `configureForTest("category", "myMethodName")` |
| 219 | + converts `myMethodName` to `my_method_name` for the filename lookup. |
| 220 | +5. **Cannot record via Java** — `CapiProxy.java` forces `GITHUB_ACTIONS=true`. |
| 221 | + Always handcraft snapshots or use the Node.js proxy directly for recording. |
0 commit comments