chore: add more documentation

Author: apiarian-datadog

.github/workflows/release-serverless-init.yml

@@ -55,9 +55,11 @@ jobs:
 
       # Pin QEMU to a known-good version. The default (binfmt:latest) has broken
       # arm64 emulation multiple times due to QEMU segfaults in libc-bin triggers:
-      #   - Feb 2025: qemu-v9.2.0 (https://github.com/tonistiigi/binfmt/issues/240)
-      #   - Mar 2026: qemu-v10.2.1
-      # Use test-qemu-versions.yml to sweep versions before bumping this pin.
+      #   - Feb 2025: qemu-v9.2.0 — PR #571 pinned, PR #581 reverted to :latest
+      #   - Mar 2026: qemu-v10.2.1 — binfmt:latest updated Feb 18, broke builds
+      # See .github/workflows/test-qemu-versions.yml to sweep versions before
+      # bumping this pin. A sweep pass is necessary but not sufficient — always
+      # verify with a real build, as some failures are load-dependent.
       - name: Set up QEMU
         uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 #v3.7.0
         with:

.github/workflows/test-qemu-versions.yml

@@ -1,40 +1,60 @@
 # Test QEMU/binfmt versions for arm64 cross-compilation on x86_64 runners.
 #
-# The serverless-init release build uses QEMU (via docker/setup-qemu-action) to
-# emulate arm64 during multi-platform Docker builds. The QEMU version provided by
-# tonistiigi/binfmt:latest has broken arm64 emulation multiple times, causing
-# segfaults during "Processing triggers for libc-bin" in the ubuntu:22.04
-# compresser stage of our Dockerfiles.
+# The serverless-init release build (release-serverless-init.yml) uses QEMU via
+# docker/setup-qemu-action to emulate arm64 during multi-platform Docker builds.
+# The QEMU version provided by tonistiigi/binfmt:latest has broken arm64 emulation
+# multiple times, causing segfaults during "Processing triggers for libc-bin" in
+# the ubuntu:22.04 compresser stage of our Dockerfiles (Dockerfile.build and
+# Dockerfile.alpine.build). Because of this, release-serverless-init.yml pins a
+# specific binfmt image version rather than using :latest.
 #
-# Known broken versions:
-#   - qemu-v9.2.0:  Feb 2025, https://github.com/tonistiigi/binfmt/issues/240
-#   - qemu-v8.1.5:  also segfaults in simple docker run tests
-#   - qemu-v8.1.4:  also segfaults in simple docker run tests
-#   - qemu-v10.2.1: Mar 2026, passes simple tests but segfaults under real build
-#                    load (parallel Go compilation + compresser stage under QEMU)
+# History of QEMU breakage in this repo:
 #
-# Known working versions (as of Mar 2026):
-#   - qemu-v10.1.3: released Feb 17 2026, passes all tests
-#   - qemu-v10.0.4: passes all tests
-#   - qemu-v9.2.2:  has QEMU-side fix for segfault (actions/runner-images#11662)
-#   - qemu-v8.0.4:  passes all tests
-#   - qemu-v7.0.0-28: long-standing known-good baseline
+#   Feb 2025 — qemu-v9.2.0 caused segfaults in arm64 emulation.
+#     - https://github.com/tonistiigi/binfmt/issues/240
+#     - https://github.com/DataDog/datadog-agent/pull/33988 (pin in datadog-agent)
+#     - https://github.com/DataDog/datadog-lambda-extension/pull/571 (pin here)
+#     - https://github.com/DataDog/datadog-lambda-extension/pull/581 (reverted pin
+#       after upstream claimed fix — this left us on :latest for ~12 months)
 #
-# Relevant issues:
+#   Mar 2026 — qemu-v10.2.1 (shipped in binfmt:latest on Feb 18 2026) caused the
+#     same segfault pattern. The last successful build was Feb 17 2026, one day
+#     before the new image was published. This time we pin permanently.
+#
+# Test results (Mar 2026, two runs to check for flakiness):
+#
+#   Version       | Run 1 | Run 2 | Notes
+#   --------------|-------|-------|------
+#   qemu-v10.2.1  |  pass |  pass | Passes simple tests but FAILS in real build
+#   qemu-v10.1.3  |  pass |  pass | Current pin target
+#   qemu-v10.0.4  |  pass |  pass |
+#   qemu-v9.2.2   |  pass |  pass | Has QEMU-side fix (actions/runner-images#11662)
+#   qemu-v9.2.0   |  FAIL |  FAIL | Consistently broken
+#   qemu-v8.1.5   |  FAIL |  FAIL | Consistently broken
+#   qemu-v8.1.4   |  FAIL |  pass | Flaky
+#   qemu-v8.0.4   |  pass |  pass |
+#   qemu-v7.0.0-28|  pass |  pass | Long-standing known-good baseline
+#
+# Important caveats:
+#   - Some versions are non-deterministically broken (v8.1.4 flipped between runs).
+#   - v10.2.1 passes all tests here but segfaults in the real build, likely because
+#     the real build runs Go compilation and the compresser stage concurrently under
+#     QEMU, creating enough load to trigger the flaky segfault.
+#   - Therefore: a version passing this sweep is necessary but NOT sufficient.
+#     Always verify with an actual release build before merging a pin change.
+#
+# Relevant upstream issues:
 #   - https://github.com/tonistiigi/binfmt/issues/215
 #   - https://github.com/tonistiigi/binfmt/issues/240
 #   - https://github.com/tonistiigi/binfmt/issues/245
 #   - https://github.com/actions/runner-images/issues/11662
 #
-# The pinned version in release-serverless-init.yml should be updated to the
-# newest version that passes here AND succeeds in a real build. Note that this
-# test workflow may not catch all failures — the real build's parallel multi-stage
-# Dockerfile (Go compilation + compresser running concurrently under QEMU) can
-# trigger segfaults that simpler tests miss (as seen with v10.2.1).
-#
-# Usage: dispatch this workflow from the Actions tab, then check which versions
-# pass/fail. Use the newest passing version as the pin in release-serverless-init.yml,
-# and verify with an actual release build before merging.
+# Usage:
+#   1. Dispatch this workflow from the Actions tab to sweep all versions.
+#   2. Run it twice to check for flakiness.
+#   3. Pick the newest consistently passing version.
+#   4. Update the pin in release-serverless-init.yml.
+#   5. Do a real serverless-init release build to confirm before merging.
 
 name: Test QEMU versions for arm64 emulation
 
@@ -77,7 +97,7 @@ jobs:
         uses: docker/setup-buildx-action@v3
 
       # Simple docker run tests — catches deterministically broken versions
-      # (e.g. v9.2.0, v8.1.5, v8.1.4 all fail here).
+      # (v9.2.0 and v8.1.5 consistently fail here).
       - name: "Test: docker run - ubuntu:22.04 apt-get (simple)"
         run: |
           docker run --rm --platform linux/arm64 ubuntu:22.04 \