GTest development

This is a list of recommendations for GTests development (either fixing current tests or writing new tests and porting from the old test infrastructure).

• Test naming convention:

  TEST_P([TESTSUITE_NAME], [TEST_INFO])
  INSTANTIATE_TEST_SUITE_P([TEST_TYPE], [TESTSUITE_NAME], ...);
  

  TEST_TYPE = {Smoke|Full|Perf}
  

  • Smoke is for small set of short running tests quickly verifying basic functionality
  • Full is for the main set of tests
  • Perf is for dedicated performance tests

  TESTSUITE_NAME = [HW_TYPE]_[NAME]_[CONTEXT]
  HW_TYPE = {CPU|GPU}
  

  • CPU is for the tests which do not require GPU and do not verify kernels

  CONTEXT = {FP8..64|BFP16|BFP8|I64|I32|I16|I8|NONE|SYS}
  

  • Defines the primary type for datatype-specific tests
  • SYS is for tests which operate at the system level
  • NONE is for the tests which are datatype agnostic, they must have CPU prefix, like db tests or similar

  NAME = {ALGORITHM_DIRECTION|AnyOtherTestNamePattern}
  TEST_INFO = {Test|Regression|AnyOtherAdditionalTestInfo}
  

  Example:

  TEST_P(GPU_Mha_fwd_FP32, Test)
  INSTANTIATE_TEST_SUITE_P(Smoke, GPU_Mha_fwd_FP32, testing::ValuesIn(GenSmokeTestCases()));
  INSTANTIATE_TEST_SUITE_P(Full, GPU_Mha_fwd_FP32, testing::ValuesIn(GenFullTestCases()));
  

  Will result in the following test names:

  Smoke/GPU_Mha_fwd_FP32.Test/0
  ...
  Smoke/GPU_Mha_fwd_FP32.Test/100
  Full/GPU_Mha_fwd_FP32.Test/0
  ...
  Full/GPU_Mha_fwd_FP32.Test/10
  

• All the test parameters must have an appropriate test name generator:

  Either by the operator<<

  friend std::ostream& operator<<(std::ostream& os, const TestCaseType& tc) {
      return os << "Param1: " << tc.param1;
  }
  

  or by the explicit INSTANTIATE_TEST_SUITE_P test name generator:

  INSTANTIATE_TEST_SUITE_P(..., ..., testing::ValuesIn(GenFullTestCases()), [](const auto& case){
      return "Param1: " + std::to_string(case.param.param1);
  });
  

  See https://google.github.io/googletest/advanced.html#specifying-names-for-value-parameterized-test-parameters

• All the tests must not use env variables to adjust it's state. Separated test must be created with appropriate naming instead of using env variables.

• TestSuite/TestCase must detect compatible HW and automatically skip non-compatible. See #2675 and #2605

• Use EXPECT_ macros for non-fatal checks. For the cases when we want to continue the test even if the check failed (for example, various numeric checks).

• Use ASSERT_ macros for fatal check. ASSERT_* aborts the current test case. For the cases when we can't continue the test execution or the test execution does not make sense (for example, failed handle creation).

• Exit from the test as early as possible. Any test skipping functionality should be done before any major allocations such as tensor instantiations. For example, right now 198 AddLayerNorm tests take around 20s to skip, but if we move skip routine at the beginning of the SetUp, it takes 1ms. void SetUp() override is a good place; however, be aware that GTEST_SKIP() only calls return from the current function, and does not abort the test. It is up to the developer to detect the skip later and prevent running code that is unneeded and/or invalid due to skipped initialization.

• Do not use headers if there is no intention to include it into multiple cpp files.

• Use anonymous namespace for all the helper functions in cpp file.

  • NOTE: however, due to an apparent bug in cppcheck, most gtest macros must be outside the anonymous namespace.

• Appropriately use testing::Values, testing::ValuesIn, etc. See https://google.github.io/googletest/reference/testing.html#param-generators

• Use EXPECT_PRED* instead of EXPECT_TRUE if possible. See https://google.github.io/googletest/reference/assertions.html#predicates

• Use testsuite shared resources as much as possible. See https://google.github.io/googletest/advanced.html#sharing-resources-between-tests-in-the-same-test-suite
  NOTE: If your test fixture defines SetUpTestSuite() or TearDownTestSuite() they must be declared public rather than protected in order to use TEST_P.

• Reset PRNG everywhere to make the tests order-agnostic: void SetUp() override{ prng::reset_seed(); ... }

• Reset internal environment values.