@AkA84 -- Just to sure, can you confirm you have set "engine": "puppeteer" in your config?

@garris Yes absolutely, we've been using Puppeteer for a while now. Although I think it's important to point out that these reliability issues begun after we switched to Headless Chrome (first with Chromy and now with Puppeteer), while with PhantomJS we never encountered them.

I completely understand this retry request from a practical view. But I am also conflicted about this feature
[...]
I really don't want to add a feature to compensate for the fact that the backstop is broken.

Yes I completely understand you. I'm far from wanting to promote a hit-it-with-a-wrench-until-it-works kind of approach!

This to me isn't to be intended as a way to dismiss performance/reliability issues with the tool itself, but as i said it's a fallback mechanism stemmed from a pragmatic approach: with a tool that depends on so many different variables, things can go wrong for many different reasons, some of which can be outside of the user's control. It is pretty bad to see a ~20min test run fail because of some random timeout error that never happened before and that doesn't seem to be due to problems in the script or the app/site being tested, especially if this is done in the context of CI.

Just to be sure I tried parallel capturing again after your reply, and unfortunately I still see random errors popping up here and there, in every run :( Also I want to stress again that we see these random errors also in single capture, although much more rarely than when using parallel capture.

Finally I thought that maybe this is a problem that we have only with our app, so i set up a simple test with these scenarios

{
  "label": "Google 1",
  "url": "https://google.com"
},
{
  "label": "Google / Type 1",
  "url": "https://google.com",
  "onReadyScript": "type.js"
},
// ...
{
  "label": "Google 99",
  "url": "https://google.com"
},
{
  "label": "Google / Type 99",
  "url": "https://google.com",
  "onReadyScript": "type.js"
}

and first ran reference with no parallel capturing, then ran test with "asyncCaptureLimit": 7. In the first case it all went smoothly, in the second case I got errors on dozens of scenarios. I got errors also with "asyncCaptureLimit": 2,, although not as many (please let me know if you need some details about my set up)

Thanks as usual!

[New Feature Request] Retry Function #761

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions