-
Notifications
You must be signed in to change notification settings - Fork 272
GTest development
This is a list of recommendations for GTests development (either fixing current tests or writing new tests and porting from the old test infrastructure).
-
TEST_P([TESTSUITE_NAME], [TEST_INFO]) INSTANTIATE_TEST_SUITE_P([TEST_TYPE], [TESTSUITE_NAME], ...);TEST_TYPE = {Smoke|Full|Perf}
-
Smokeis for small set of short running tests quickly verifying basic functionality -
Fullis for the main set of tests -
Perfis for dedicated performance tests
TESTSUITE_NAME = [HW_TYPE]_[NAME]_[CONTEXT]
HW_TYPE = {CPU|GPU}-
CPUis for the tests which do not require GPU and do not verify kernels
CONTEXT = {FP8..64|BFP16|BFP8|I64|I32|I16|I8|NONE|SYS}
- Defines the primary type for datatype-specific tests
-
SYSis for tests which operate at the system level -
NONEis for the tests which are datatype agnostic, they must haveCPUprefix, like db tests or similar
NAME = {ALGORITHM_DIRECTION|AnyOtherTestNamePattern}
TEST_INFO = {Test|Regression|AnyOtherAdditionalTestInfo}Example:
TEST_P(GPU_Mha_fwd_FP32, Test) INSTANTIATE_TEST_SUITE_P(Smoke, GPU_Mha_fwd_FP32, testing::ValuesIn(GenSmokeTestCases())); INSTANTIATE_TEST_SUITE_P(Full, GPU_Mha_fwd_FP32, testing::ValuesIn(GenFullTestCases()));Will result in the following test names:
Smoke/GPU_Mha_fwd_FP32.Test/0 ... Smoke/GPU_Mha_fwd_FP32.Test/100 Full/GPU_Mha_fwd_FP32.Test/0 ... Full/GPU_Mha_fwd_FP32.Test/10 -
-
All the test parameters must have an appropriate test name generator:
Either by the
operator<<friend std::ostream& operator<<(std::ostream& os, const TestCaseType& tc) { return os << "Param1: " << tc.param1; }or by the explicit INSTANTIATE_TEST_SUITE_P test name generator:
INSTANTIATE_TEST_SUITE_P(..., ..., testing::ValuesIn(GenFullTestCases()), [](const auto& case){ return "Param1: " + std::to_string(case.param.param1); }); -
All the tests must not use env variables to adjust it's state. Separated test must be created with appropriate naming instead of using env variables.
-
TestSuite/TestCase must detect compatible HW and automatically skip non-compatible. See #2675 and #2605
-
Use
EXPECT_macros for non-fatal checks. For the cases when we want to continue the test even if the check failed (for example, various numeric checks). -
Use
ASSERT_macros for fatal check. ASSERT_* aborts the current test case. For the cases when we can't continue the test execution or the test execution does not make sense (for example, failed handle creation). -
Exit from the test as early as possible. Any test skipping functionality should be done before any major allocations such as
tensorinstantiations. For example, right now 198 AddLayerNorm tests take around 20s to skip, but if we move skip routine at the beginning of theSetUp, it takes 1ms.void SetUp() overrideis a good place; however, be aware thatGTEST_SKIP()only callsreturnfrom the current function, and does not abort the test. It is up to the developer to detect the skip later and prevent running code that is unneeded and/or invalid due to skipped initialization. -
Do not use headers if there is no intention to include it into multiple cpp files.
-
Use anonymous namespace for all the helper functions in cpp file.
- NOTE: however, due to an apparent bug in cppcheck, most gtest macros must be outside the anonymous namespace.
-
Appropriately use
testing::Values,testing::ValuesIn, etc. See https://google.github.io/googletest/reference/testing.html#param-generators -
Use EXPECT_PRED* instead of EXPECT_TRUE if possible. See https://google.github.io/googletest/reference/assertions.html#predicates
-
Use testsuite shared resources as much as possible. See https://google.github.io/googletest/advanced.html#sharing-resources-between-tests-in-the-same-test-suite
NOTE: If your test fixture defines SetUpTestSuite() or TearDownTestSuite() they must be declared public rather than protected in order to use TEST_P. -
Reset PRNG everywhere to make the tests order-agnostic:
void SetUp() override{ prng::reset_seed(); ... }