Test results aren't updated when tests are run in the debugger

[malsyned]

Steps to reproduce:

1. Change a test or some code under test, such that a passing test should fail or a failing test should pass
2. In the Test Explorer, click the 🐞 ("Debug this test") icon next to the test whose result should change
3. Allow the debugging session to run to completion and exit

Expected result: The status icon to the left of the test name changes from a red "X" to a green check mark, or vice versa
Actual result: The status icon stays the same, until the ▶ ("Run tests") button is pressed to run the test outside of the debugger

[bneumann]

Just tried this here in my example code:

1. build test code (with a failing test in there)
2. setup extension pointing to the executable
3. running tests in extension (test is red)
4. fixing code and rebuilding
5. running tests in extension again
   The result is the test being green now.

So maybe it is only in a debugger?

[malsyned]

Correct, only in a debugger. As far as I can tell from the code, the plugin doesn't try to capture the output of the debugger or convert it into test adapter events at all.

[malsyned]

I'm not actually sure it's possible to capture or spy on the output from a debugging session. I'm digging around the docs now.

[malsyned]

I don't see any API that fits the bill. I'm a little confused by CppUTestContainer.ts:DebugTest() though -- it seems to be using undocumented fields of DebugConfiguration to modify the arguments passed to the launcher. Is it actually the case that any field you can use in a launch.json configuration can be used as a member of a DebugConfiguration? And if so, maybe there's something we could do to capture the output by modifying the configuration to redirect somehow?

If I got a -o filename option into CppUTest to redirect output to a file, maybe we could use args to point it to a named pipe?

[bneumann]

Hmm yes I see. The problem would be that your debugger "swallows" the output information so we can't get them out.
Other test runners seem to just indicate another state. I checked and VSCode offers these here, but nothing like "debugged, didn't run":

state: 'running' | 'passed' | 'failed' | 'skipped' | 'errored';

[malsyned]

The Test Adapter documentation makes frequent mention of a "RetireEvent" which can be used to mark a test result as "outdated". I don't know what that looks like in the UI. Might that be a path to victory here?

[bneumann]

That is the old test adapter. When I started this project, I forked from the test explorer api repo and build on top of that. Since then a lot changed and VSCode has it's own test explorer API. I still use the old one which wraps the native implementation because I didn't want to rewrite everything.
I will take a look into it. Maybe I was overlooking something

[bneumann]

I read through the new documentation and found that they have a completely different approach on how to handle states. They have a dedicated debug, run, coverage etc. profile which you execute and the result state will be handled by the UI then. sooo, I think finally porting it to the new API might solve our little problem here.
Autumn is coming, and maybe I will find some downtime to get it done :)