Pipelines doesn't support on-demand cloud workers

[tmbgreaves]

Issue #12 comments:

A critical issue is that Pipelines doesn't allow use of higher specification workers or workers with non-standard hardware, except in the case where users link their own existing hardware into pipelines. Doing this removes a key benefit of researchers not having to maintain the service and hardware themselves.

A good solution would be for pipelines to support on-demand workers provisioned from existing Azure instances and billed to an existing subscription on established hourly rates. Getting the PIpelines team to implement this could have a very significant impact on research software testing, allowing researchers to run longer, larger, and more exotic tests without having to invest time into a custom testing service.

[ggorman]

If I understand correctly we would also need this feature if we want to test on multiple architectures - eg GPU's.

[ggorman]

Is this not a duplication of #5 ?

[tmbgreaves]

No - they're looking at different environments. #5 refers to the problem that in an AKS cluster you can only have one nodepool and hence only one type of VM, relevant to a custom Jenkins instance. #6 refers to the inability to get anything other than a 'standard worker' on pipelines, the inability to bill extra workers to a subscription, and the inability to request a particular worker type only for jobs that explicitly request it. I'm assuming none of these are a result of AKS nodepool limitations :-)

[tmbgreaves]

If I understand correctly we would also need this feature if we want to test on multiple architectures - eg GPU's.

Agreed - I've updated the initial issue text to reflect this.