ci: GPU/OptiX full build and test#1992
Conversation
At long last, turn on full execution and testing for OptiX in our CI, using the GitHub GPU-enabled runners. This should greatly reduce the possibility of build breaks from not actually testing OptiX rendering, not to mention making it possible to make changes on the OptiX code path and have it confirmed by our usual CI. Previously, we were building against the OptiX headers, so at least we wouldn't make an obvious build break, but we weren't executing the testsuite on GPUs. So it was still possible to merge code that broke that code path in practice. For PRs that looked obviously risky, I would pull them to my work machine, build and test with real GPUs to confirm. But this was a big pain, extra trouble for me, and very error prone -- sometimes I didn't recognize that a PR needed that extra testing, or sometimes I forgot entirely, and merged broken code. Note that this depends on PAID GitHub runners with GPUs. The bill is paid by ASWF, but it means those runners are only available on the ASWF GitHub account -- that is, they will run for PRs once submitted, but not when you merely push to your own account's fork, sorry. When running on a non-ASWF account, it will select an ordinary free runner and build OSL against the OptiX headers, but will not run the testsuite since there is no GPU. Signed-off-by: Larry Gritz <lg@larrygritz.com>
|
I feel like the very short amount of changes in the end does not adequately convey the volume of tinkering it took (not to mention essential help from Luca and @jfpanisset) over several days and a few nights to figure out how to make it all work in practice. |
fpsunflower
left a comment
fpsunflower
left a comment
There was a problem hiding this comment.
Very exciting to see this working!
Changes themselves make it look easy - but I did see the long thread on slack that led up to this, so kudos to all involved.
|
Since I am guessing that the GPU runners are special machines whose time is valuable, do we want to maybe disable the CPU tests when running on that runner? Or do we think its worth making sure the GPU build can also successfully run the CPU tests? |
That's a really interesting question! I think it's hard to judge how much money we're spending until we see how much use it gets in the first month. How many PRs does this project have per month (times how many CI runs per PR)? If it costs $1 or so for each one, is that enough to justify potentially less testing? It's one pool of money for all the projects, but ASWF gets a breakdown of which repos are responsible for how much, so we'll know pretty quickly if we are eating up more than our fair share. (There's a cap, so we can't really bankrupt the organization, at worst we'll hit the monthly cap and maybe inconvenience OpenVDB until the end of the month.) If it turns out that we're spending too much, there may be a few ways to cut back. One is to try to exclude the CPU tests, as you suggest. I hadn't thought of that before, actually. Maybe we should do enough CPU tests just to prove to ourselves that having the optix mode present in the build doesn't somehow break the CPU modes, but exclude some of the more expensive ones? Another way to reduce the time we burn on those machines is just to run the tests less often. Can we get away with only running the GPU tests when the PR alters certain source files but not others? I guess if only oslc is changed, maybe it's safe to skip GPU? But once you start slicing into liboslexec or testshade/testrender, or maybe most of the header files, it seems like there are all sorts of ways to break optix, so you want to test. This approach seems like it's splitting hairs. Another strategy is to speed up the build. In the course of doing the work of this PR, I realized that the ccache saving itself between jobs seems to not actually be working. So I need to look into why that is, which may speed up the build quite a bit most of the time (but will not speed up the tests). JF is looking into an option to store the images on GitHub's machines themselves somehow, rather than pulling from DockerHub every time. It's unclear if there's a speedup from that, but if there is, it could be significant -- it takes us nearly as long just to pull the docker image and instantiate it as it does to do the build of OSL. |
|
So unless you think I should tackle the throttling as fully as I can in this initial PR, I'm tempted to want to call this done for now, merge and then watch for a few days at least to see how rapidly we are spending money. At that point, if there is any concern about the burn rate, I can implement your suggestion try to cut way back on the number of CPU tests we do (for this one job that exists to test GPU). |
|
Next up, we just need to get the Windows tests fully working! |
|
Sounds good. Lets get this in and look at real data. If resources are not an issue, then it is nice to know that the presence of the GPU backend doesn't break the CPU side. |
At long last, turn on full execution and testing for OptiX in our CI, using the GitHub GPU-enabled runners. This should greatly reduce the possibility of build breaks from not actually testing OptiX rendering, not to mention making it possible to make changes on the OptiX code path and have it confirmed by our usual CI. Previously, we were building against the OptiX headers, so at least we wouldn't make an obvious build break, but we weren't executing the testsuite on GPUs. So it was still possible to merge code that broke that code path in practice. For PRs that looked obviously risky, I would pull them to my work machine, build and test with real GPUs to confirm. But this was a big pain, extra trouble for me, and very error prone -- sometimes I didn't recognize that a PR needed that extra testing, or sometimes I forgot entirely, and merged broken code. Note that this depends on PAID GitHub runners with GPUs. The bill is paid by ASWF, but it means those runners are only available on the ASWF GitHub account -- that is, they will run for PRs once submitted, but not when you merely push to your own account's fork, sorry. When running on a non-ASWF account, it will select an ordinary free runner and build OSL against the OptiX headers, but will not run the testsuite since there is no GPU. Signed-off-by: Larry Gritz <lg@larrygritz.com>
2 participants
At long last, turn on full execution and testing for OptiX in our CI, using the GitHub GPU-enabled runners.
This should greatly reduce the possibility of build breaks from not actually testing OptiX rendering, not to mention making it possible to make changes on the OptiX code path and have it confirmed by our usual CI.
Previously, we were building against the OptiX headers, so at least we wouldn't make an obvious build break, but we weren't executing the testsuite on GPUs. So it was still possible to merge code that broke that code path in practice. For PRs that looked obviously risky, I would pull them to my work machine, build and test with real GPUs to confirm. But this was a big pain, extra trouble for me, and very error prone -- sometimes I didn't recognize that a PR needed that extra testing, or sometimes I forgot entirely, and merged broken code.
Note that this depends on PAID GitHub runners with GPUs. The bill is paid by ASWF, but it means those runners are only available on the ASWF GitHub account -- that is, they will run for PRs once submitted, but not when you merely push to your own account's fork, sorry. When running on a non-ASWF account, it will select an ordinary free runner and build OSL against the OptiX headers, but will not run the testsuite since there is no GPU.