System tests of main are broken

[304NotModified]

I see the tests are broken on main. I don't if this is the result of merging this PR

@clrudolphi any idea? I havent found time to check the root cause of the test failures

The connector tests are dynamically building a few of the Reqnroll example sample projects from the ReqnrollExploratoryTestProjects repo, such as the SpecFlowCompatibilityProject. Those projects have hard-wired references to Reqnroll 3.3.4-local:

 		<PackageReference Include="Reqnroll.SpecFlowCompatibility" Version="3.3.4-local" />

Version 3.3.4 final was released two days ago and so a reference to 3.3.4-local isn't valid.
I am not sure whether those should be updated to the latest released build or to a local build of version 4.0.0 (the next build coming).

Originally posted by @clrudolphi in #167 (comment)

[304NotModified]

The previous commit of main also fails indeed

https://github.com/reqnroll/Reqnroll.VisualStudio/actions/runs/23549959814

[304NotModified]

@clrudolphi could you please check reqnroll/Reqnroll.ExploratoryTestProjects#2?

[gasparnagy]

Sorry for causing a confusion.

In general the idea of the https://github.com/reqnroll/Reqnroll.ExploratoryTestProjects repo was to have a set of projects that can be used for local exploratory (manual) tests. I wanted to keep the projects as real-looking as possible (to be able to detect real integration problems), therefore I did not wanted to use a global Directory.props based version setting.

As the projects are used mostly to test the local versions, the projects were generally set to the current local versions and I usually updated that (with the update-versions.ps1 it is easy) and pushed. Sometimes you need to update the versions for local testing anyway.

In order to improve the VS discovery coverage, I introduced a new set of "smoke tests" that clones and goes through the projects in the ExploratoryTestProjects repo and tries to load them to VS. We cannot assume a correct version in the cloned projects, so the test should update the versions to the expected one by calling the ps1 script. This should be done by these lines: https://github.com/reqnroll/Reqnroll.VisualStudio/blob/main/Tests/Connector/Reqnroll.VisualStudio.ReqnrollConnector.Tests/SampleProjectTestBase.cs#L206-L210

So "normally" this should work, even with an old and "-local" versions in the ExploratoryTestProjects repo. And it was at least at the time of the last release.

Something has probably changed, or I have a bug in the connector tests that causes this error now. I will look at it and come back with the details.

[304NotModified]

Sometimes you need to update the versions for local testing anyway.

I think this isn't a problem. But I prefer to have it working on the CI, then it's easier to analyze when it isn't working locally:)

[gasparnagy]

Sometimes you need to update the versions for local testing anyway.

I think this isn't a problem. But I prefer to have it working on the CI, then it's easier to analyze when it isn't working locally:)

Just for clarification: you mean that we should have the "released" versions in the pushed repo, and use the "-local" locally, right?

[304NotModified]

Yes indeed :)

[gasparnagy]

Analyzed the problem and the build works as expected, it updates the versions of the exploratory test projects to the current released version before running the tests. The build failure comes from the Reqnroll issue reqnroll/Reqnroll#1068 - the missing Reqnroll.SpecFlowCompatibility 3.3.4 package.

Since the plan is to discontinue the support for Reqnroll.SpecFlowCompatibility, I think the most paratactical immediate solution is to exclude those tests from the connector smoke test suite.

In addition to this, I will change the CI build for the VisualStudio project to run weekly as well in addition to running per commit. There is not that much activity on that repo, so we might not realize a problem, like this on time.

I will react on the note of @304NotModified about the pushed version in a separate comment, because that is a slightly different topic.