[Bug]: Playwright MSTest.Sdk Uses MSTest Runner, Ignoring runsettings Configuration / Microsoft.Testing.Platform

[terius7]

Version

1.50.0

Steps to reproduce

When using Playwright version 1.50.0 with MSTest.Sdk version 3.8.1, the MSTest runner is used instead of the expected VSTest runner. As a result, the Playwright configuration section in the runsettings file is ignored. Key settings such as BrowserName, ExpectedTimeout, Headless, and Channel do not work.

However, when <UseVSTest>true</UseVSTest> is explicitly set in the .csproj file, and the VSTest runner is used, the Playwright settings in the runsettings file work as expected.

Expected behavior

• Playwright should respect the runsettings file when running tests with MSTest.Sdk.
• Playwright-specific settings such as BrowserName, ExpectedTimeout, Headless, and Channel should be applied correctly, regardless of whether MSTest or VSTest is used.

Actual behavior

• With MSTest runner, the Playwright settings in runsettings are ignored.
• With VSTest runner (<UseVSTest>true</UseVSTest>), the settings work as expected.

Additional context

This issue suggests that MSTest.Sdk is defaulting to the MSTest runner instead of the VSTest runner, which does not properly load the Playwright settings from runsettings. It would be helpful if Playwright could support runsettings consistently across different test runners.

Would appreciate any insights or fixes on this issue. Thanks!

Environment

- Playwright Version: 1.50.0
- MSTest.Sdk Version: 3.8.1
- .NET Version: net8.0

[mxschmitt]

Sounds similar to the issues we are encountering with #3093. We probably need to change how the config is getting parsed from a TestAdapter to something else. Or plumb the values through the other process somehow.

[0xced]

Sounds like another argument in favor of #3081. 😉

[Youssef1313]

You can probably plug your own configuration provider via TestingPlatformBuilderHook. I think you should be able to get it to work with MTP that way.

[alexaka1]

@Youssef1313 Can you elaborate on that? A quick search of the docs, I didn't find anything I could start with.

[Youssef1313]

It's supposed to be in https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-architecture-extensions but looks like the doc is missing some stuff, partially tracked by microsoft/testfx#5552.

In general though, TestingPlatformBuilderHook is an MSBuild item that lets Microsoft.Testing.Platform entry point generation inject extension registration.

Then for a custom configuration provider, you can do something similar to this:

https://github.com/microsoft/testfx/blob/093c672c4dc15c0d4cbbd84c85e0a43774aa82c6/src/Platform/Microsoft.Testing.Extensions.VSTestBridge/Helpers/TestApplicationBuilderExtensions.cs#L36, or it might even be better to just get the IConfiguration from IServiceProvider and read settings from it right away, without providing a whole custom configuration provider. But really depends on your actual needs and I didn't look deeply in it.

[jeffreyrivor]

Given that recommended way to use MSTest is to use the MSTest.Sdk MSBuild project SDK and the latest MSTest v4 uses Microsoft.Testing.Platform instead of VSTest by default and removed the UseVSTest MSBuild property for switching back, requiring an extra step of referencing Microsoft.NET.Test.Sdk package to regain VSTest support, it seems that either fixing the RunSettings support or creating a new mechanism to pass equivalent configuration for Playwright is needed soon (before the next major LTS of .NET, which is the first likely point at which support for VSTest will begin to be deprecated as part of the semantic version bump).

[Youssef1313]

UseVSTest property is still present if you want only VSTest support and don't want MTP support. However, if you want your project to support both VSTest and MTP, that's when you need the package reference.

[Youssef1313]

at which support for VSTest will begin to be deprecated

We never announced anywhere that VSTest will begin to be deprecated.

[jeffreyrivor]

To be precise, I did not say "at which support for VSTest will begin to be deprecated" as if it was a sure thing; I said, "the first likely point at which support for VSTest will begin to be deprecated" to mean that eventually there will likely be a point where MS will not want to commit finite engineering resources to supporting two different test execution platforms and will commit to the newer one (similar to the push from .NET Framework to .NET), and my speculation that such a question (if it were to come up) would be timed with the next .NET LTS. If that's too speculative or pessimistic, I apologize.

I just do not think we need to wait until that time comes to make RunSettings or an equivalent work for MTP; this issue has been sitting for almost a year and is still "collecting feedback" so I just wanted to put my two cents in that from my perspective, MSTest on MTP is the current "front-runner" in terms of being supported for the foreseeable future.

Or in other words, while we can just document that Playwright settings don't work via MTP on MSTest.Sdk and the workaround is to set UseVSTest, it seems reasonable to get it working on MTP sooner rather than later particularly since it's a prominently documented feature of MSTest.Sdk: https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-sdk#test-with-playwright (aside: perhaps this section should also have a disclaimer that Playwright settings won't work without UseVSTest).

UseVSTest property is still present if you want only VSTest support and don't want MTP support. However, if you want your project to support both VSTest and MTP, that's when you need the package reference.

Sorry, the docs were not very precise with this so maybe I misunderstood. https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-migration-v3-v4#mstestsdk-no-longer-adds-microsoftnettestsdk-reference-when-using-microsofttestingplatform says the following:

By default, MSTest.Sdk uses Microsoft.Testing.Platform. If the UseVSTest MSBuild property is set to true, it will use VSTest instead. In MSTest 3.x, the SDK added a reference to Microsoft.NET.Test.Sdk (which brings VSTest support) even when using Microsoft.Testing.Platform. This package reference is unnecessary when running with Microsoft.Testing.Platform and has been removed in MSTest v4.

If you still want to have VSTest supported (for example, if you want to run with vstest.console), you need to manually add a package reference to Microsoft.NET.Test.Sdk NuGet package to your project.

I took the second part to mean "you have to add an extra Microsoft.NET.Test.Sdk reference to get VSTest support back in MSTest v4." I don't understand how that squares with the first paragraph.

[Vampire2008]

I use xUnit. While Playwright not implemented providing options other way than .runsettings I maked simple workaround.

using System.Xml;
using MyApp.Tests;
using Microsoft.Playwright.TestAdapter;

[assembly: AssemblyFixture(typeof(PlaywrightSettingsLoader))]
namespace MyApp.Tests;

public class PlaywrightSettingsLoader 
{
    public PlaywrightSettingsLoader() 
    {
        if(!File.Exists(".runsettings")) return;

        var reader = XmlReader.Create(".runsettings", new XmlReaderSettings 
        {
            IgnoreComments = true,
            IgnoreWhitespace = true
        });

        while (reader.Read()) 
        {
            if (reader is { NodeType: XmlNodeType.Element, Name: "Playwright" }) 
            {
                new PlaywrightSettingsProvider().Load(reader);
                break;
            }
        }
    }
}