fix(bb): skip BatchMergeTests.TooManySubtablesFails under _GLIBCXX_DEBUG

[AztecBot]

Summary

Unblock the Nightly Debug Build workflow on next. Latest failing run: https://github.com/AztecProtocol/aztec-packages/actions/runs/25539203050 (run #80, head_sha f23aa82c52, exit 134, 8m15s).

Skip BatchMergeTests.TooManySubtablesFails under _GLIBCXX_DEBUG (i.e. when NDEBUG is not defined). The verifier-rejection contract is still exercised by release builds. Same approach as the existing commitment_key.test.cpp:DISABLED_PippengerZeroCountRegression.

Root cause

TooManySubtablesFails calls BB_DISABLE_ASSERTS() so it can drive the verifier with N = max_subtables + 1 subtables. With BB asserts demoted to warnings, control flow falls past the BB_ASSERT_LTE(N, M, ...) guard at the top of BatchMergeProver::construct_proof and reaches compute_degree_check_polynomial:

for (size_t idx = 0; idx < flattened_columns.size(); ++idx) {
    reversed_column_spans.emplace_back(reversed_columns[idx]);
    scalars.push_back(degree_check_challenges[idx]);  // OOB when idx >= challenges.size()
}

With M = 9, NUM_WIRES = 4, N = 10:

• flattened_columns.size() = NUM_WIRES + N * NUM_WIRES = 44
• degree_check_challenges.size() = (M + 1) * NUM_WIRES = 40

In release this is silent UB and the verifier rejects the malformed proof. In debug, libstdc++'s _GLIBCXX_DEBUG makes std::vector::operator[] bounds-checked and the OOB at idx == 40 traps before the verifier runs:

Error: attempt to subscript container with out-of-bounds index 40,
       but container only holds 40 elements.

BB_DISABLE_ASSERTS() cannot suppress this — the bounds check is libstdc++'s, not bb's.

Why the previously merged fix (#23019) is dead code on this path

#23019 modifies TweakableBatchMergeProver::construct_proof. But BatchMergeTests::prove_and_verify only routes through TweakableBatchMergeProver when a FaultMode is supplied; TooManySubtablesFails calls prove_and_verify(op_queue) with no fault mode, so it goes through the base BatchMergeProver::construct_proof (production code path) which is unchanged by #23019. Merge-train CI never exposed this because it runs ci-barretenberg (release) only — ci-barretenberg-debug is nightly-only and runs on next per barretenberg-nightly-debug-build.yml.

Verification

Reproduced and verified inside the ClaudeBox container on next f23aa82c52 + this fix, with the same flags as the nightly:

cd barretenberg/cpp
CXXFLAGS="-gdwarf-4 -D_GLIBCXX_DEBUG" CC=clang-20 CXX=clang++-20 \
  cmake -S . -B build-debug -G Ninja \
        -DCMAKE_BUILD_TYPE=Debug -DENABLE_ASAN=OFF -DDISABLE_ASM=OFF -DENABLE_STACKTRACES=ON
cd build-debug && ninja goblin_tests
./bin/goblin_tests

• Pre-fix on the same commit: aborts on BatchMergeTests/2.TooManySubtablesFails with the OOB above (exit 134).
• Post-fix: 66 passed, 9 skipped (4 of them the now-skipped TooManySubtablesFails parameterizations), 0 failed; full suite runs in 328 s.

Relationship to #23018

#23018 (still open, draft) proposed the same _GLIBCXX_DEBUG skip on b30fe8f4.... This PR re-applies the same fix on top of current next so it can land without further rebase. #23018 can be closed once this merges.

Detailed analysis: https://gist.github.com/AztecBot/22747657c0f363ff0278db37024d6db1
ClaudeBox log: https://claudebox.work/s/5470e5f058610dce?run=1

ClaudeBox log: https://claudebox.work/s/5470e5f058610dce?run=1

[AztecBot]

Automatically closing this stale claudebox draft PR (no updates for 5+ days). Re-open if still needed.