cmake: Introduce ctest labels for testing by wjwithagen · Pull Request #29844 · ceph/ceph

wjwithagen · 2019-08-23T10:35:53Z

cmake: Introduce ctest TEST labels for testing and COST

Tests in CMake/Ctest can have labels on which can be filtered

  ctest  --print-labels
  Test project /home/wjw/work/ceph/build
  All Labels:
    freebsd
    long
    tox
    unittest
    vstart

Which allows to run certain subsets of by using RE filters
in ctest.

COST will determine the order in which the tests are executed.
Current settings will start with long running tests.
Like:

        Start 118: unittest_erasure_code_shec_all
        Start  31: unittest_bufferlist
        Start 106: readable.sh
        Start  13: run-cli-tests
        Start 105: check-generated.sh
        Start 140: safe-to-destroy.sh
        Start 130: ceph_test_object_map
        Start  15: smoke.sh
        Start  12: rbd-ggate.sh
        Start 169: unittest_rbd_mirror
        Start 154: test_ceph_argparse.py
        Start 134: unittest_fastbmap_allocator
        Start 141: unittest_osdmap
        Start   1: run-tox-mgr-dashboard
        Start   2: run-tox-mgr-insights
        Start   3: run-tox-mgr-ansible

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard backend
jenkins test docs
jenkins render docs

epuertat

Looks ok. Just a few suggestions/questions left over there.

tchaikov · 2019-08-30T04:02:34Z

i'd recommend s/build:/cmake:/ in the title of your commit message. we could be more specific here.

epuertat

Looks like this is improving even further! Thanks @wjwithagen! Just a few suggestions: mostly related to the hardcoded values and perhaps explaining this in Ceph Developer's docs.

epuertat · 2019-09-02T10:35:51Z

BTW, with the help of @alfredodeza we've created a Jenkins Build Analyzer parser for CTest Failures, so the output in case of failure (like the above "make check fail") would look like the following:

CTest Failure

Label Time Summary:
long        = 3576.34 sec*proc (14 tests)
tox         = 183.55 sec*proc (6 tests)
unittest    = 5227.38 sec*proc (161 tests)

Total Test time (real) = 4063.85 sec

The following tests FAILED:
	174 - unittest_seastar_buffer (Failed)
	176 - unittest_seastar_socket (Failed)
	177 - unittest_seastar_messenger (Failed)
	178 - unittest_seastar_thread_pool (Timeout)
	179 - unittest_seastar_perfcounters (Failed)```

wjwithagen · 2019-09-02T10:56:40Z

You can supress the total label runtime if wanted with ctest --no-label-summary
And perhaps @tchaikov could/would mark his seastar test with a label as well.

epuertat

Looks great. Just added a minor suggestion to improve the docs already provided. Thanks a lot for this @wjwithagen !

stale · 2019-11-26T23:32:21Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

epuertat · 2019-11-27T10:16:18Z

Keep away from this, bad stale bot!

I think this is valuable in order to speed-up/fine-tune CI.

github-actions · 2022-07-07T17:42:09Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

anthonyeleven · 2022-07-10T00:51:47Z

+               Labels can be selected with ctest -L or -LE
+
+Since COST is used to determine the testing order, values can be chosen rather
+arbitrarily. Currently long running programs are given a value equal to the


Zac may have a different view, but I'd make it long-running.

anthonyeleven · 2022-07-10T00:57:37Z

+execution time in a OpenStack VM. The idea is to first run the long standing 
+tests, and then fill up free cores with programs with shorter execution time.
+Turning this into a bin-packing challenge, So the exact value is not important,
+it is about the relative COST value.


an OpenStack. But conceptually I'm a bit unclear, how is OpenStack involved? Wouldn't runtime in an OpenStack instance be a function of the aggregate's overcommit policy, the instance flavor, and and the underlying CPU model? Where does the numerical COST come from? Historical data? Enquiring minds want to know.

"The idea is to first run the long-running tests, and then fill unused cores with tests with shorter execution time.
This is effectively a bin-packing challenge, so the exact values are not important, bur rather relative COST value.

an OpenStack. But conceptually I'm a bit unclear, how is OpenStack involved? Wouldn't runtime in an OpenStack instance be a function of the aggregate's overcommit policy, the instance flavor, and and the underlying CPU model? Where does the numerical COST come from? Historical data? Enquiring minds want to know.

"The idea is to first run the long-running tests, and then fill unused cores with tests with shorter execution time. This is effectively a bin-packing challenge, so the exact values are not important, bur rather relative COST value.

The main purpose here is to prevent that run in a multi-core environment and try to schedule to very long running jobs at the end of test-run, since that could extend the test-run time with like 5 - 10 minutes.

'Openstack ' refers to the fact that I was running these builds in an OpenStack system. But like I indicated COST is only used to steer the bin-packing algorithm, no use in getting exact numbers, order of magnitude will already work.
And I used the times that I found in my runs.

github-actions · 2022-09-13T14:09:15Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2022-11-12T15:01:39Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2023-01-12T15:02:04Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2023-02-11T16:01:17Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

jmundack · 2025-03-07T13:19:37Z

@wjwithagen - i am unsure if you are still working on ceph BUT was hoping/wondering if we can/should revive this PR.

We are trying to improve the general builds, tests - developer experience around this and I think would really help with that endeavor

epuertat · 2025-05-14T20:34:03Z

Recovering this PR in the context of CI improvements, as this would enable both:

Improved scheduling of tests,
Conditional running of tests based on git changed files. This requires further work to create mappings between modified source code directories and ctest labels (e.g.: in .gitattributes).

idryomov · 2025-05-14T21:32:09Z

Conditional running of tests based on git changed files. This requires further work to create mappings between modified source code directories and ctest labels (e.g.: in .gitattributes).

@cbodley is trying out an approach to do this based on parsing CODEOWNERS file inside of ceph-build: ceph/ceph-build#2361

github-actions · 2025-07-13T22:01:46Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2025-08-12T23:01:28Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

wjwithagen requested a review from a team as a code owner August 23, 2019 10:35

wjwithagen requested a review from tchaikov August 23, 2019 10:36

epuertat approved these changes Aug 23, 2019

View reviewed changes

Comment thread cmake/modules/AddCephTest.cmake Outdated

Comment thread cmake/modules/AddCephTest.cmake Outdated

Comment thread src/pybind/mgr/dashboard/CMakeLists.txt Outdated

cbodley added the build/ops label Aug 23, 2019

tchaikov reviewed Aug 30, 2019

View reviewed changes

wjwithagen changed the title ~~build: Introduce ctest labels for testing~~ cmake: Introduce ctest labels for testing Sep 1, 2019

wjwithagen force-pushed the wjw-wip-cmake-labels branch 5 times, most recently from 59f9a1c to e9e5f6f Compare September 2, 2019 08:21

epuertat reviewed Sep 2, 2019

View reviewed changes

Comment thread cmake/modules/AddCephTest.cmake Outdated

Comment thread cmake/modules/AddCephTest.cmake Outdated

Comment thread cmake/modules/AddCephTest.cmake Outdated

Comment thread src/test/CMakeLists.txt Outdated

wjwithagen force-pushed the wjw-wip-cmake-labels branch 4 times, most recently from bb5b582 to 46e677f Compare September 7, 2019 14:26

epuertat approved these changes Sep 9, 2019

View reviewed changes

Comment thread doc/dev/developer_guide/index.rst Outdated

tchaikov self-requested a review September 9, 2019 11:11

wjwithagen force-pushed the wjw-wip-cmake-labels branch from 46e677f to f125dcc Compare September 9, 2019 20:12

wjwithagen force-pushed the wjw-wip-cmake-labels branch 4 times, most recently from a854a58 to 169d9f6 Compare September 27, 2019 22:36

stale Bot added the stale label Nov 26, 2019

stale Bot removed the stale label Nov 27, 2019

github-actions Bot added the stale label Jul 7, 2022

djgalloway changed the base branch from master to main July 9, 2022 00:00

djgalloway requested review from a team as code owners July 9, 2022 00:00

anthonyeleven requested changes Jul 10, 2022

View reviewed changes

github-actions Bot removed the stale label Jul 15, 2022

github-actions Bot added the stale label Sep 13, 2022

epuertat removed the stale label Sep 13, 2022

github-actions Bot added stale and removed stale labels Nov 12, 2022

github-actions Bot added the stale label Jan 12, 2023

github-actions Bot closed this Feb 11, 2023

epuertat reopened this May 14, 2025

github-actions Bot added core dashboard documentation mgr pybind labels May 14, 2025

epuertat mentioned this pull request May 15, 2025

ceph-pull-requests: omit run-rbd-unit-tests unless pr matches @ceph/rbd codeowner ceph/ceph-build#2361

Open

Conversation

wjwithagen commented Aug 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

epuertat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tchaikov commented Aug 30, 2019

Uh oh!

epuertat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

epuertat commented Sep 2, 2019

Uh oh!

wjwithagen commented Sep 2, 2019

Uh oh!

epuertat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

stale Bot commented Nov 26, 2019

Uh oh!

epuertat commented Nov 27, 2019

Uh oh!

github-actions Bot commented Jul 7, 2022

Uh oh!

anthonyeleven Jul 10, 2022

Choose a reason for hiding this comment

Uh oh!

anthonyeleven Jul 10, 2022

Choose a reason for hiding this comment

Uh oh!

wjwithagen Nov 13, 2022

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 13, 2022

Uh oh!

github-actions Bot commented Nov 12, 2022

Uh oh!

github-actions Bot commented Jan 12, 2023

Uh oh!

github-actions Bot commented Feb 11, 2023

Uh oh!

jmundack commented Mar 7, 2025

Uh oh!

epuertat commented May 14, 2025

Uh oh!

idryomov commented May 14, 2025

Uh oh!

github-actions Bot commented Jul 13, 2025

Uh oh!

github-actions Bot commented Aug 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

wjwithagen commented Aug 23, 2019 •

edited

Loading