Generate BSF CSVs both with and without /third_party tests

[yulunwu2]

Adds an --include-third-party flag to browser-specific-failures.js
(off by default), and extends build.sh to produce four CSVs per run:
the existing {stable,experimental}-browser-specific-failures.csv files
now exclude /third_party/ tests while the new -with-third-party.csv
will include them.

/third_party/ introduces Temporal tests as large number of individual tests instead of
using the subtest WPT convention. These test failures misrepresent the interoperability
of the web by causing the single browser failure graph to be completely dominated
by a feature not yet shipped by all browsers.

Charts including /third_party/ results will be added to wpt.fyi in a separate PR.

[jgraham]

No objection to landing this change. However I don't think that continuing to show a metric that's known to be broken is a good long term approach.

[yulunwu2]

@jgraham @foolip could you take a look at this change? Thank you!

[stubbornella]

@jgraham we could also move/remove the graph. @yulunwu2 created a separate PR for that option.
web-platform-tests/wpt.fyi#4906

[yulunwu2]

@jgraham thank you for your support on this matter! If you don't find this objectionable to land in the interim as we figure what we want to do regarding the BSF graph on the front page, could I get a r+1?

[foolip]

I think that the current BSF metric should be supplemented with more different views and ways of weighing the data. IIUC Apple is working on a proposal for BSF updates and a wpt.fyi landing page, which I've wanted for a long time and very much look forward to.

In the spirit of making as much data as possible available, I'd like suggest a tweak to this PR. Rather than simply excluding Test262, instead compute the metric with and without third_party/. Then let a checkbox (default unchecked) control whether third party test suites are included. Later one we might be able to scrap this in favor of filtering by spec/WG/feature/etc..

I'd also like @dandclark to review this PR before we go ahead.

[gsnedders]

It would be good to add some tests to test/browser-specific.js for the flag, verify it's doing the right thing — make sure it does exclude /third_party/test.html, and that it doesn't exclude /foo/third_party/test.html, etc.

[yulunwu2]

[yulunwu2]

adding screenshot for proposed UI changes for wpt.fyi

[foolip]

Thanks for the preview, @yulunwu2! It looks like this will make the component taller as the text is now taller than the graph itself. I think we can omit the "By default" paragraph. If that's not enough, then also omit the "Last updated WPT revision", or merge it into a single line.

For the new toggle, how about "WPT" and "WPT + Test262" as the options instead of first/third party? That'll be accurate at least while third_party is just Test262, hopefully long enough to get the next iteration of graphs done.

And if you have a wpt.fyi PR I can review there instead :)

[foolip]

This LGTM together with a wpt.fyi change along the lines that @yulunwu2 showed in the screenshot. I've also asked @DanielRyanSmith to review this.

[yulunwu2]

@foolip Thanks for reviewing this change and your comments on the wpt.fyi change!

I've removed the third_party paragraph and updated the toggle name.

The PR is here:

web-platform-tests/wpt.fyi#4921

[yulunwu2]

@jgraham could I get a +1 for this PR?

[DanielRyanSmith]

Sorry for the delay here. The logic here looks good to me!

[jgraham]

Approved, but reiterating for future reference that I don't think continuing to publish this metric is a good long term approach.