Commit f7f9450
Expand E2E test coverage across all 4 SDKs (#1186)
* Greatly expand E2E test coverage and harden tests across all 4 SDKs
Adds comprehensive end-to-end coverage of the SDK + Copilot CLI surface area
across C#, Python, TypeScript, and Go, with the goal that every public API
and every JSON-RPC method has at least one E2E test. Reorganizes the C#
and Python test layouts to clearly separate unit and E2E tests, and adds
`E2E` suffixes to test class/file names for clarity.
What's added
------------
E2E coverage (parity across all 4 SDKs):
- Session lifecycle: connect/disconnect, dispose semantics, multi-client
scenarios, resume, force-stop, idle-then-suspend.
- Session config: model selection (vision-enabled/disabled transitions),
agent selection, allowed/denied tool sets, working directory,
environment variables, system prompts, MCP servers.
- RPC surface: agent (get/getCurrent/list/reload), session state
(capabilities/getCurrent/getMetadata/list/reset), trust, model
(get/getCurrent/list/setCurrent), permission (request/respond/list),
hooks, plan, telemetry, command execute/elicit/respond.
- Streaming: assistant.message_delta + reasoning_delta ordering and
matching message IDs across delta and final events.
- Suspend RPC: suspend during pending permission, suspend during pending
external tool, suspend idle session, resume + continue conversation
after suspend.
- Hooks: pre/post tool, pre/post session, deny verification (asserts
target file is unchanged after a deny).
- Permissions: per-session auth tokens with auto-token opt-out for tests
that explicitly verify the unauthenticated path.
- GitHub references, attachments, custom request headers, trace context
propagation.
- Tool routing, command handlers, elicitation flow.
Unit coverage:
- Forward-compatibility for unknown discriminators and unknown event
envelope types so unrecognized future events round-trip safely.
Snapshot harness:
- 100+ new YAML snapshots under `test/snapshots/` covering the new
scenarios; existing snapshots normalized for portability.
Test layout:
- C# tests split into `dotnet/test/E2E/` and `dotnet/test/Unit/`
folders; xUnit collection serializes E2E execution to avoid CLI
process contention (with explanatory comment at the attribute site).
- Python E2E tests live under `python/e2e/` with a shared harness
proxy/context. `E2E` suffix added to classes/methods.
- Go E2E tests use `_e2e_test.go` suffix and an `E2E` test-function
suffix for clarity.
- TypeScript E2E tests use `.e2e.test.ts` suffix; `createSdkTestContext`
registers Vitest hooks that auto-stop `CopilotClient` and `CapiProxy`.
What's fixed
------------
Snapshot drift root cause:
- `test/harness/replayingCapiProxy.ts`: `writeCapturesToDisk` was
called from both `updateConfig` and `stop` regardless of CI mode.
When a test exercised only a subset of a multi-conversation snapshot,
the file was silently rewritten with that subset, breaking later runs.
Both writes are now guarded by `process.env.GITHUB_ACTIONS !== "true"`.
`gh` CLI environment-dependent help text:
- Added a `normalizeGhAuthMessages` tool result normalizer so the two
forms of "auth required" help text emitted by `gh` (GitHub Actions
vs. local dev) both map to a stable `\` placeholder
before snapshot match.
PerSessionAuth auto-token leak:
- `dotnet/test/Harness/E2ETestContext.cs`: added an
`autoInjectGitHubToken` parameter so `CreateAuthTestClient` can
opt out of the CI auto-token injection that was silently authenticating
tests verifying the unauthenticated path.
Go SDK lifecycle and codegen:
- Polling-based `waitForCapability` helper in `go/session_test.go`
to replace race-prone fixed sleeps.
- Telemetry marker constant aligned with snapshot.
- Codegen / lifecycle bugs surfaced while porting C# E2E coverage.
TypeScript cleanup hangs:
- Replaced fixed `setTimeout` waits in lifecycle/session tests with
bounded polling helpers (`waitFor` / `getLastSessionId` /
`getSessionMetadata` polls).
- Bumped the `should stop cleanly` test timeout to 60s to absorb
slow-machine variability.
Python:
- `_wait_for` polling helper in `test_commands_and_elicitation.py`
replaces flaky fixed sleeps.
Verification
------------
Five consecutive full-suite runs across all 4 SDKs, all clean (the final
run was executed with all 4 suites in parallel to confirm the fixes hold
under contention):
- C#: 319 passed / 0 failed / 4 skipped
- Python: 356 passed / 0 failed / 8 skipped
- TS: 328 passed / 0 failed / 9 skipped
- Go: all packages ok
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: resolve CI failures in expanded E2E coverage
- Format Node E2E test files with prettier (4 files: client_lifecycle,
commands, session_fs, session_lifecycle).
- Format/lint Python files with ruff: switched Callable import from
typing to collections.abc in test_commands_and_elicitation.py; split a
long comment in conftest.py to satisfy E501; removed unnecessary "r"
open mode in test_hooks_e2e.py.
- Resolve macOS /var -> /private/var symlinks in test harnesses so the
paths match what spawned subprocesses see when they resolve their cwd:
* Go: filepath.EvalSymlinks on home_dir / work_dir.
* .NET: P/Invoke libc realpath (with CA2101-compliant marshaling)
and a Windows fallback to Path.GetFullPath.
* Python: os.path.realpath wrapping tempfile.mkdtemp in the shared
E2ETestContext and in the per-test multi-client harnesses
(test_commands_e2e, test_multi_client_e2e,
test_ui_elicitation_multi_client_e2e).
- Loosen per-session "no token" assertions in .NET and Python so they
match Node/Go: in CI the process-level fake GITHUB_TOKEN can make
IsAuthenticated true even without a per-session token, and the Login
may surface as null/None on Linux/macOS but as an empty string on
Windows.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: round 2 of CI fixes for expanded E2E coverage
- StreamingFidelityE2ETests: lock around the events list in the three
remaining streaming tests. Without locking, the event handler runs on
the dispatcher thread and continues to append after SendAndWaitAsync
returns, racing with the test thread's enumeration and producing
"Collection was modified; enumeration operation may not execute".
This mirrors the lock-and-snapshot pattern already used in the
Should_Emit_AssistantMessageStart_Before_Deltas_With_Matching_MessageId
test in the same file.
- python/pyproject.toml: add opentelemetry-sdk to dev dependencies so
test_telemetry_e2e.py::test_restores_w3c_trace_context can construct
a real TracerProvider. The opentelemetry-api package alone provides
the namespace package "opentelemetry" but not "opentelemetry.sdk".
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address CodeQL findings on E2E tests and harness
- Bind awaited results in async helpers to silence `no-effect''
warnings (Python `await x` -> `_ = await x`).
- Add explanatory comments to intentional `except ExceptionGroup: pass`
blocks in test_rpc_server_e2e.py.
- HookLifecycleAndOutputE2ETests: remove unused errorInputs container.
- ClientOptionsE2ETests: dispose TcpListener via `using`.
- RpcShellAndFleetE2ETests: simplify `Contains(...) == true` to a
null-coalescing form.
- SessionConfigE2ETests/SessionE2ETests: use `catch (Exception)` in
`DisposeAsync` cleanup blocks instead of bare `catch`.
- E2ETestContext.ResolveSymlinks: replace libc realpath/free P/Invoke
with managed `DirectoryInfo.ResolveLinkTarget`-based component walk;
preserves macOS `/var -> /private/var` resolution behavior. Also
switches the fallback `catch` to typed exceptions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use single-underscore discard for awaited results
CodeQL flagged the named `_result`/`_captured_permission_request`
discards as unused locals after the prior commit. Switch to the
single-underscore convention, which CodeQL's py/unused-local-variable
rule and ruff F841 both treat as an intentional discard.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use Path.Join instead of Path.Combine to avoid silent-drop warnings
CodeQL flags Path.Combine because if a later argument is rooted, it
silently discards the earlier ones. Path.Join concatenates segments
literally and is the safer choice for these cases.
- E2ETestContext.ResolveSymlinks: use Path.Join when walking path
components (the loop appends user-controlled directory names that
realistically never start with a path separator on these temp dirs,
but the safer API removes the risk regardless).
- HooksE2ETests: use Path.Join for the protected-file lookup.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add cross-SDK E2E parity tests
Add missing E2E coverage so suspend RPC, event fidelity, built-in tool smoke tests, error resilience, and multi-turn scenarios are represented across all SDKs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>1 parent f8cf846 commit f7f9450
235 files changed
Lines changed: 21754 additions & 1542 deletions
File tree
- dotnet
- src
- Generated
- test
- E2E
- Harness
- Unit
- go
- internal/e2e
- testharness
- rpc
- nodejs/test
- e2e
- harness
- python
- copilot
- e2e
- testharness
- scripts/codegen
- test
- harness
- snapshots
- agent_and_compact_rpc
- client_api
- client_options
- hooks_extended
- multi_client
- pending_work_resume
- permissions
- rpc_session_state
- rpc_shell_and_fleet
- session_config
- session_fs
- session
- streaming_fidelity
- suspend
- telemetry
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1775 | 1775 | | |
1776 | 1776 | | |
1777 | 1777 | | |
| 1778 | + | |
1778 | 1779 | | |
1779 | 1780 | | |
1780 | 1781 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2787 | 2787 | | |
2788 | 2788 | | |
2789 | 2789 | | |
| 2790 | + | |
2790 | 2791 | | |
2791 | 2792 | | |
2792 | 2793 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
Lines changed: 2 additions & 89 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
| 11 | + | |
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| |||
148 | 148 | | |
149 | 149 | | |
150 | 150 | | |
151 | | - | |
152 | | - | |
153 | | - | |
154 | | - | |
155 | | - | |
156 | | - | |
157 | | - | |
158 | | - | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
165 | | - | |
166 | | - | |
167 | | - | |
168 | | - | |
169 | | - | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
177 | | - | |
178 | | - | |
179 | | - | |
180 | | - | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
186 | | - | |
187 | | - | |
188 | | - | |
189 | | - | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
196 | | - | |
197 | | - | |
198 | | - | |
199 | | - | |
200 | | - | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
205 | | - | |
206 | | - | |
207 | | - | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
213 | | - | |
214 | | - | |
215 | | - | |
216 | | - | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
223 | | - | |
224 | | - | |
225 | | - | |
226 | | - | |
227 | | - | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
232 | | - | |
233 | | - | |
234 | | - | |
235 | | - | |
236 | | - | |
237 | | - | |
238 | 151 | | |
239 | 152 | | |
240 | 153 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
0 commit comments