tools: Add adaptive Bazel wrapper

tools/bazel_adaptive.py runs Bazel with a concrete integer --jobs value,
monitors progress output and memory pressure, stops and resumes bazel
build processes as needed under memory pressure, and if that does not
relieve memory pressure, restarts with lower or higher parallelism as
memory allows. Integer --jobs values are used directly; Bazel-style
HOST_CPUS and HOST_RAM values with optional multipliers, such as
HOST_CPUS*.5 and HOST_RAM*.0002, are resolved once as the maximum
adaptive jobs cap.

The wrapper rewrites bazel output to indicate the number of paused jobs
(if any).  "(13 actions, 12 running)" is shown as "(13 actions, 2 paused,
10 running)" when 2 of the running actions have been paused by the
wrapper.

This wrapper is only modestly faster than builds with carefully tuned
--jobs count and ample available swap space. The main benefit of this
wrapper is that it removes the tuning work and automatically restarts
bazel when if fails for out-of-memory or job termination.

To allow Bazel to get through memory bottlenecks during the build without
unnecessary restarts the build environment should have swap
available. This wrapper was tested on a 24 GB VM with 16 GB swap.

Running the wrapper with BAZEL_ADAPTIVE_DISABLE_PTY=1 defined allows more
of the diagnostic lines to remain visible even when running in an
interactive terminal. Each diagnostic line starts with
"[bazel-adaptive/<XXX>s]", where "<XXX>" is the number of seconds since
the wrapper started.

This commit changes the Makefile to use the wrapper by default.

Written by Codex through an iterative session where codex has run the
included Python tests and has observed the behavior of live runs to
observe the runtime behavior.

Signed-off-by: Jarno Rajahalme <jarno@isovalent.com>

Author: jrajahalme

Makefile

@@ -23,15 +23,15 @@ ENVOY_TESTS = bazel-bin/tests/*_test
 BUILD_DEP_FILES = ENVOY_VERSION WORKSPACE .bazelrc envoy.bazelrc bazel/toolchains/BUILD bazel/toolchains/cc_toolchain_config.bzl
 
 SHELL=/bin/bash -o pipefail
-BAZEL ?= $(QUIET) bazel
+BAZEL ?= $(QUIET) tools/bazel_adaptive.py
 BAZEL_FILTER ?=
 BAZEL_OPTS ?=
 BAZEL_BUILD_OPTS ?=
 ifdef BAZEL_REMOTE_CACHE
   BAZEL_BUILD_OPTS += --remote_cache=$(BAZEL_REMOTE_CACHE)
 endif
 
-BAZEL_TEST_OPTS ?= --jobs=HOST_RAM*.0002 --test_timeout=100 --local_test_jobs=1 --flaky_test_attempts=3
+BAZEL_TEST_OPTS ?= --test_timeout=100
 BAZEL_TEST_OPTS += --test_output=errors
 
 BUILDARCH := $(subst aarch64,arm64,$(subst x86_64,amd64,$(shell uname -m)))