Make the preprocessing tasks for workflow files cacheable

[Vampire]

No description provided.

[Vampire]

• Make the preprocessing tasks for workflow files cacheable #2157 👈 (View in Graphite)
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.90%. Comparing base (5d220ca) to head (cf896a5).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2157      +/-   ##
============================================
- Coverage     81.91%   81.90%   -0.01%     
- Complexity     4732     4735       +3     
============================================
  Files           463      463              
  Lines         14767    14767              
  Branches       1869     1869              
============================================
- Hits          12096    12095       -1     
- Misses         1980     1981       +1     
  Partials        691      691              

see 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[leonard84]

🤔 This does not work, iirc Gradle wants a dedicated directory for output caching to work.

https://ge.spockframework.org/s/ybqsxrfvj6d7q/timeline?details=6lhcybvrqmfrk

Overlapping outputs: Gradle does not know how file '.github/workflows/docs-pr.yaml' was created (output property 'workflowFile'). Task output caching requires exclusive access to output paths to guarantee correctness (i.e. multiple tasks are not allowed to produce output in the same location).

And I think you meant to use inputs here.

Suggested change

	tasks.preprocessBranchesAndPrsWorkflow {
	outputs.file('.github/codecov.yml').withPropertyName('codecovConfig')
	}
	tasks.preprocessReleaseWorkflow {
	outputs.file('.github/codecov.yml').withPropertyName('codecovConfig')
	}
	tasks.preprocessBranchesAndPrsWorkflow {
	inputs.file('.github/codecov.yml').withPropertyName('codecovConfig')
	}
	tasks.preprocessReleaseWorkflow {
	inputs.file('.github/codecov.yml').withPropertyName('codecovConfig')
	}

[Vampire]

Hm, I'm a bit confused now.
The part you quoted complains about docs-pr.yaml which does not make sense to me.
The part you linked to complains about codecov.yml which does make sense to me because I was stupid.

I did not mean to declare it as input.
Actually it is input and output, as the file is read, the number updated and then written again.
Which per-se is sub-optimal, especially for a cacheable task.
And the main problem is, that two different workflows do this and so both have it as output.

I think I'll change it, so that both tasks write to separate different files and then have finalizer tasks that move them over the actual checked in file to decouple the input and output and have no overlapping outputs.

This will only make the task cached after the second run if the codecov file changed, but that should be an acceptable trade-off.

But the question remains where you copied the warning with docs-pr.yaml from.

[leonard84]

You still have overlapping outputs for all workflow tasks:

• https://ge.spockframework.org/s/n5ofs5ftv6czg/timeline?details=3yt6nuz2vwgza
• https://ge.spockframework.org/s/n5ofs5ftv6czg/timeline?details=6lhcybvrqmfrk
• https://ge.spockframework.org/s/n5ofs5ftv6czg/timeline?details=6ykqsd4xqsblq
• https://ge.spockframework.org/s/n5ofs5ftv6czg/timeline?details=uvtxjtpy2mnnc

But the question remains where you copied the warning with docs-pr.yaml from.

From the preprocessDocsPrWorkflow, as all had the same issue, I just picked one task at random.

[Vampire]

Huh, somehow this does not make any sense.

I made a non-sense change before pushing, so that if the file is coming from cache, the codecov file is not updated, I'll make a new push with that fixed like I had it before that last change.

Besides that the cache key is also bad as I forgot to define path sensitivity.

But either way, there are no overlapping outputs anymore.
Especially the file it complains about is dedicated to that one task and there should not be any overlap.
Locally the result is taken from cache just fine: https://ge.spockframework.org/s/4phvtkur47e74/timeline?details=3yt6nuz2vwgza&expanded=WyIxIl0

I even added

allprojects { tasks.all {} }
tasks.register('foo') {
  doLast {
    allprojects*.tasks*.all { task ->
      outputs.files.files.findAll { it.absolutePath.contains('.github') }.each {
        println("FOO: ${task.name} | ${it.absolutePath}")
      }
    }
  }
}

and all I got was the expected

FOO: preprocessBranchesAndPrsWorkflow | D:\Sourcecode\other\spock\.github\workflows\branches-and-prs.yaml
FOO: preprocessCodeqlAnalysisWorkflow | D:\Sourcecode\other\spock\.github\workflows\codeql-analysis.yaml
FOO: preprocessCommonWorkflow | D:\Sourcecode\other\spock\.github\workflows\common.yaml
FOO: preprocessDocsPrWorkflow | D:\Sourcecode\other\spock\.github\workflows\docs-pr.yaml
FOO: preprocessReleaseWorkflow | D:\Sourcecode\other\spock\.github\workflows\release.yaml

:-/

[Vampire]

Same on GHA: https://ge.spockframework.org/s/vnf6tebn2ygd4/console-log?page=1#L100 while the complaint stays the same: https://ge.spockframework.org/s/vnf6tebn2ygd4/timeline?details=3yt6nuz2vwgza

[Vampire]

Also using the ghActionsBuild task locally works just fine here: https://ge.spockframework.org/s/y6x7rzl55yags/timeline?details=3yt6nuz2vwgza, now even the cache key is the same as on GHA.

[Vampire]

Argh, it is because it is a clean checkout where the file already exists and Gradle does not know that itself was the one creating it. 🙈

[Vampire]

Ok, with that knowledge now finally the tasks are also cacheable on GHA