Ways we can speed up CI

[CodingWithSaksham]

Currently, the CI is quite slow, taking around 27 minutes on average across all matrix jobs.

(Thank you @pandafy for the image.)

Of that time, roughly 1.5 minutes are spent downloading dependencies, while nearly 16 minutes are spent actually running the tests. I have been experimenting with a potential solution: creating a pre-baked Docker image within the repository's packages and splitting the test suite into three groups (sample app, Selenium, and regular tests).

I tested this approach on my fork and reduced the average CI runtime across all matrix jobs from approximately 27 minutes to 16 minutes.

While the results are promising, one significant downside (as pointed out by @pandafy) is the maintenance burden of the Docker image. Keeping the image up to date and ensuring it remains compatible with our dependencies could become a substantial ongoing responsibility.

I'd like to open a discussion on this topic and explore whether there are better or more sustainable approaches to improving CI performance.

[nemesifier]

I am also not in favour of adding more complexity and would rather look for small wins that allow keeping the current setup.
What you propose is also hard to replicate in the other modules we maintain, leaving openwisp-controller as some sort of a special case, but the inconsistency can cause maintenance problems: we need to explain these differences to all the contributors who will ever touch the CI.

The install dependencies step should be already cached, maybe there's something not working 100% there?

However, the tests take a long time to run, some tests are slow and can be optimized, some other aspects of tests could be improved to reduce the total time.

[CodingWithSaksham]

That can be done as well, but finding slow tests can be a PITA, even with coverage telling is the list of slow tests, I believe that other solutions should be explored first.

[nemesifier]

If you don't want to address slow tests, just pick other issues. This is not for you.

[CodingWithSaksham]

One thing I noticed in the CI is that runtest executes the sample app tests, Selenium tests, and regular tests sequentially. While this makes sense on a developer's machine due to limited CPU resources, the same constraint doesn't apply in GitHub Actions.

We could consider splitting these into three separate test categories. Instead of the current 11 matrices (Python × Django versions), we'd have 33 matrices (Python × Django × test type), where the test types are sample app, Selenium, and regular tests. This would allow GitHub Actions to run them in parallel, which could significantly reduce the overall CI runtime.

It might be worth exploring this approach.

[nemesifier]

Still not addressing the core issue: some tests are excessively slow and need to be optimized. Github actions won't run 33 builds at the same time and in periods of high activity, we we'll probably end up with a very large queue.