Mismatched exception tag in function template

[m7913d]

If the the (number of) branches depend on the template argument a mismatched exception tag may be generated as lcov assumes that a single line has the same branches on all invocations.

Example

Consider the following example:

#include <exception>

struct Throw {Throw () {throw std::exception();}};
struct NoThrow {NoThrow () {}};

static int a;

template<typename T>
bool test()
{
  return new T() && a > 0 ? 1 : 0; // <-- this is the line lcov is complaining about
}

int main()
{
  try {test<Throw>();} catch (...) {}
  test<NoThrow>();
}

Generating branch coverage fails with the following error:

mismatched exception tag for id 3, 3: '0' -> '1'

I think this is an lcov issue, and not a gcov one as the output of gcov make sense:

...
{
  "line_number": 11,
  "function_name": "_Z4testI7NoThrowEbv",
  "count": 3,
  "unexecuted_block": true,
  "branches": [
    {
      "count": 3,
      "throw": false,
      "fallthrough": true
    },
    {
      "count": 0,
      "throw": true,
      "fallthrough": false
    },
    {
      "count": 3,
      "throw": false,
      "fallthrough": true
    },
    {
      "count": 0,
      "throw": false,  // <-- 4th branch is NOT an exception branch for test<NoThrow> function
      "fallthrough": false
    },
   ... 6 more branches
  ]
},
{
  "line_number": 11,
  "function_name": "_Z4testI5ThrowEbv",
  "count": 3,
  "unexecuted_block": true,
  "branches": [
    {
      "count": 3,
      "throw": false,
      "fallthrough": true
    },
    {
      "count": 0,
      "throw": true,
      "fallthrough": false
    },
    {
      "count": 0,
      "throw": false,
      "fallthrough": true
    },
    {
      "count": 3,
      "throw": true, // <-- 4th branch is an exception branch for test<Throw> function
      "fallthrough": false
    },
   ... 8 more branches
  ]
},
...

test<Throw> contains more (exception) branches than test<NoThrow> because the Throw constructor throws an exception and the NoThrow constructor doesn't.

Possible solutions

• Create a separate coverage report per set of function template arguments. Disadvantage is that it may result in a very long coverage report in case the template is used with many different arguments.
• When merging branch coverage for template instantiations with a different set of template arguments (i.e. a different function_name), merge the exception and non-exception branches separately, i.e. merge the 1th/2nd/3th/... exception branch with the 1th/2nd/3th/... exception branch and the merge the 1th/2nd/3th/... non-exception branch with the 1th/2nd/3th/... non-exception branch. This is not 100% correct either, but is at least better than the current approach.

[henry2cox]

Not exactly a solution to the issue you bring up...just a bit of an explanation and a workaround.

As you point out: lcov has a pretty deeply held belief that branch blocks on the same line, which contain the same number of branch expressions are the same and should be merged.
In this particular case: it also complains because the attributes of one of the branch expressions in one block does not match the attributes of the corresponding branch in the other block. (If you add the flag --ignore mismatch to your command line: then lcov will overlook the difference and merge anyway.)
For example: if the number of expressions in the blocks was different, then lcov would know that the blocks were not the same and would not try to merge them.

In this specific case: merging only if the branch attributes also match would avoid the issue - but this is not a solution. It is easy to invent another example where attributes match and we merge when we should not.
In addition: I have seen (many) cases in which the gcov 'exception' attribute seems to be incorrect/inaccurate/misleading. In those cases, both "merged when we should not" and "didn't merge when we should have" errors would occur. (It is possible that newer gcc versions have fixed some or all of those issues...I have not investigated recently.)

The underlying problem is that there is no way for lcov to know whether the branch expressions really are the same or are merely similar.

What would be nice (and is available with certain other tools/other languages such as Perl or Verilog) is an explicit indication of the expression corresponding to each branch - for example "! new T", "new T && a>0", .etc.
That information needs to come from the compiler/toolchain though...it is not possible to derive from the gcov data or from the source code (simple cases: yes. Real code: no).

If the expressions are available, then lcov will only merge blocks if the blocks are in the same order and all the expressions match.
In their absence: lcov uses the branch index as the identifier - and happily (if erroneously) merges blocks with the same number of branches.

It turns out that gcc's implementation of MC/DC metrics is closer to what you want (in this case, at least).
In particular: gcc produces 2 MC/DC blocks, with 2 and 4 conditions, respectively (though we still have no indication to which expression the condition indices correspond):

$ g++ -fcondition-coverage --coverage example.cpp 
$ /a.out
$ lcov --capture --branch --mcdc -d . -o gcc.info --ignore mismatch
$ genhtml -o gcc --branch --mcdc gcc.info --flat # <- open brower to 'gcc/index.html'

It isn't hard to create a case where MC/DC blocks have identical numbering but are not the same - such that lcov will merge their metrics when it should not.

In this example, LLVM is a bit less informative.
In particular: it generates only one MC/DC block with 4 conditions and fails to notice that some conditions are sensitized (see the discussion in #234 for some of the reasons that we see what we see)

$ clang++ -fprofile-instr-generate -fcoverage-mapping -fcoverage-mcdc example.cpp -lstdc++
$ ./a.out`
$ llvm-profdata merge --sparse *.profraw -o llvm.profdata`
$ llvm-cov export -format=test -instr-profile=llvm.profdata ./a.out > llvm.json`
$ llvm2lcov --branch --mcdc llvm.json -o llvm.info`
$ genhtml -o llvm --branch --mcdc llvm.info --flat # <- open browser to 'llvm/index.html'

(We can generate a differential coverage report, comparing the gcc to LLVM results - but the example is small enough to diff by eye and the comparison it isn't super interesting in this case anyway).

The summary/takeaways:

• to really fix the expression aliasing issue, lcov needs the toolchain to insert unique identifiers into the generated data.
  From the user's perspective: it would be most helpful if those identifiers corresponded to the user's source code (not compiler internals) - as that would make the result much easier to understand and much more actionable.
• you may find MC/DC metrics to be more useful. (But yes: branch and MC/DC measure different things - and both might be important.)
• I will experiment with using expression attributes in the merge compare - and see what it does to some larger examples.

[m7913d]

@henry2cox Thanks for your elaborative response.

As you point out: lcov has a pretty deeply held belief that branch blocks on the same line, which contain the same number of branch expressions are the same and should be merged.
For example: if the number of expressions in the blocks was different, then lcov would know that the blocks were not the same and would not try to merge them.

Do you mean that lcov will not generate an error (even if --ignore mismatch is not specified) if the number of branches is different? In my example, the number of branches returned by gcov is different. In my (limited) experience, lcov happily merges this case (probably incorrectly), as long as the "throw" property matches between corresponding indexes (for the mutual indexes).

In this specific case: merging only if the branch attributes also match would avoid the issue - but this is not a solution.

I agree it's not an ideal solution, but I think it is at least better than the current approach and would avoid the need for --ignore mismatch. I would prefer that lcov only generates an error if it effectively detects really inconsistent entries and not when lcov just doesn't have enough information to reliably interpret the entries. At most a (non-blocking) warning should suffice in that case.

It is easy to invent another example where attributes match and we merge when we should not.

I know. I always take the branch coverage with a grain of salt and often (mostly) ignore the exception branches.

It turns out that gcc's implementation of MC/DC metrics is closer to what you want (in this case, at least).

I will have a look at MC/DC. I haven't used it before.

I will experiment with using expression attributes in the merge compare

I'm looking forward to the result of your experiments.