fix(ci): auto-retry transient Windows compile/clone failures via exit 75 (#4023)

* fix(ci): exit 75 on transient Windows compile/clone failures for auto-retry

Windows compile jobs (and clone/submodule fetch) intermittently fail on
transient network errors such as 'Could not resolve host: index.crates.io'
during cargo's crates.io index fetch, surfacing as NMAKE U1077 / exit 2.

Convert the failure guards in the Windows compile job and the shared
windows_git_setup() helper to 'exit 75'. 75 is in default.retry.exit_codes
(generate-common.php) with max: 2, so GitLab auto-retries these jobs.

git checkout (deterministic) keeps its native exit. Real compile breaks
still surface as failures; they merely retry twice first, which cannot
produce a false pass on a compile job.

* fix(ci): grep build log so only network failures exit 75 on Windows compile

Per review: make the Windows compile steps selective instead of blanket
exit 75. Tee each nmake build's output to a log and exit 75 only when a
transient network signature is present (crates.io DNS, spurious network
error, failed to download, name-resolution/connection timeouts);
otherwise propagate the real exit code so genuine compile breaks fail
fast. The libdatadog target move reverts to native exit (local, not
network). windows_git_setup() clone/submodule guards stay blanket 75
(those steps are network by nature).

* fix(ci): narrow Windows build network-grep to the observed crates.io failure

Drop speculative patterns; match only signatures present in the actual
failure (Could not resolve host / spurious network error / failed to
download).

* fix(ci): don't let build stderr abort the Windows compile capture

The '2>&1 | Tee-Object' capture, under GitLab's default $ErrorActionPreference='Stop',
turned the build's normal native stderr (cargo warnings) into a terminating
NativeCommandError, aborting the job with exit 1 before nmake finished.

Set $ErrorActionPreference='Continue' around the captured docker exec so merged
stderr is emitted rather than thrown, snapshot $LASTEXITCODE, then restore 'Stop'
before the network-signature classification.

Author: Leiyks

.gitlab/generate-common.php

@@ -73,17 +73,17 @@ function windows_git_setup() {
     git config --global core.symlinks true
     git clone --branch $env:CI_COMMIT_REF_NAME $env:CI_REPOSITORY_URL .
     if ($LASTEXITCODE -ne 0) {
-        Write-Host "ERROR: git clone failed. Remaining workspace contents:"
+        Write-Host "ERROR: git clone failed (likely transient network); exiting 75 so GitLab auto-retries. Remaining workspace contents:"
         Get-ChildItem -Force | Select-Object Name
-        exit $LASTEXITCODE
+        exit 75
     }
     git checkout $env:CI_COMMIT_SHA
     if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
 
     # Initialize submodules
     Write-Host "Initializing submodules..."
     git submodule update --init --recursive
-    if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
+    if ($LASTEXITCODE -ne 0) { exit 75 }  # transient network (submodule fetch); 75 triggers default retry
     Write-Host "Git setup complete."
 <?php
 }

.gitlab/generate-package.php

@@ -544,17 +544,29 @@
     # Start the container
     docker run -v ${pwd}:C:\Users\ContainerAdministrator\app -d --name ${CONTAINER_NAME} ${IMAGE} ping -t localhost
 
-    # Build nts (fail fast on any step)
-    docker exec ${CONTAINER_NAME} powershell.exe -Command "`$ErrorActionPreference='Stop'; `$PSNativeCommandUseErrorActionPreference=`$true; cd app; switch-php nts; & 'C:\\php\\SDK\\phpize.bat'; if (`$LASTEXITCODE -ne 0) { exit `$LASTEXITCODE }; .\\configure.bat --enable-debug-pack; if (`$LASTEXITCODE -ne 0) { exit `$LASTEXITCODE }; nmake; if (`$LASTEXITCODE -ne 0) { exit `$LASTEXITCODE }; Move-Item x64\\Release\\php_ddtrace.dll extensions_x86_64\\php_ddtrace-${ABI_NO}.dll -ErrorAction Stop; Move-Item x64\\Release\\php_ddtrace.pdb extensions_x86_64_debugsymbols\\php_ddtrace-${ABI_NO}.pdb -ErrorAction Stop"
-    if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
+    # Build nts (fail fast on any step); capture combined output for failure classification.
+    # ErrorActionPreference=Continue so the build's native stderr (e.g. cargo warnings) is not
+    # turned into a terminating NativeCommandError by the 2>&1 capture under GitLab's default Stop.
+    $ErrorActionPreference = 'Continue'
+    docker exec ${CONTAINER_NAME} powershell.exe -Command "`$ErrorActionPreference='Stop'; `$PSNativeCommandUseErrorActionPreference=`$true; cd app; switch-php nts; & 'C:\\php\\SDK\\phpize.bat'; if (`$LASTEXITCODE -ne 0) { exit `$LASTEXITCODE }; .\\configure.bat --enable-debug-pack; if (`$LASTEXITCODE -ne 0) { exit `$LASTEXITCODE }; nmake; if (`$LASTEXITCODE -ne 0) { exit `$LASTEXITCODE }; Move-Item x64\\Release\\php_ddtrace.dll extensions_x86_64\\php_ddtrace-${ABI_NO}.dll -ErrorAction Stop; Move-Item x64\\Release\\php_ddtrace.pdb extensions_x86_64_debugsymbols\\php_ddtrace-${ABI_NO}.pdb -ErrorAction Stop" 2>&1 | Tee-Object -FilePath nts-build.log
+    $ntsCode = $LASTEXITCODE
+    $ErrorActionPreference = 'Stop'
+    # Only transient network failures (e.g. crates.io DNS) get exit 75 for GitLab auto-retry; real compile breaks keep their native code and fail fast.
+    if ($ntsCode -ne 0) { if (Select-String -Path nts-build.log -Pattern 'Could not resolve host','spurious network error','failed to download' -Quiet) { Write-Host "Transient network failure during nts build; exiting 75 so GitLab auto-retries (see default retry.exit_codes in generate-common.php)"; exit 75 } else { exit $ntsCode } }
 
     # Reuse libdatadog build (fail if move fails)
     docker exec ${CONTAINER_NAME} powershell.exe -Command "`$ErrorActionPreference='Stop'; `$PSNativeCommandUseErrorActionPreference=`$true; New-Item -ItemType Directory -Force -Path 'app\\x64\\Release_TS' | Out-Null; Move-Item 'app\\x64\\Release\\target' 'app\\x64\\Release_TS\\target' -ErrorAction Stop"
-    if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
-
-    # Build zts (fail fast on any step)
-    docker exec ${CONTAINER_NAME} powershell.exe -Command "`$ErrorActionPreference='Stop'; `$PSNativeCommandUseErrorActionPreference=`$true; cd app; switch-php zts; & 'C:\\php\\SDK\\phpize.bat'; if (`$LASTEXITCODE -ne 0) { exit `$LASTEXITCODE }; .\\configure.bat --enable-debug-pack; if (`$LASTEXITCODE -ne 0) { exit `$LASTEXITCODE }; nmake; if (`$LASTEXITCODE -ne 0) { exit `$LASTEXITCODE }; Move-Item x64\\Release_TS\\php_ddtrace.dll extensions_x86_64\\php_ddtrace-${ABI_NO}-zts.dll -ErrorAction Stop; Move-Item x64\\Release_TS\\php_ddtrace.pdb extensions_x86_64_debugsymbols\\php_ddtrace-${ABI_NO}-zts.pdb -ErrorAction Stop"
-    if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
+    if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }  # local file move, not network — fail fast (no retry)
+
+    # Build zts (fail fast on any step); capture combined output for failure classification.
+    # ErrorActionPreference=Continue so the build's native stderr (e.g. cargo warnings) is not
+    # turned into a terminating NativeCommandError by the 2>&1 capture under GitLab's default Stop.
+    $ErrorActionPreference = 'Continue'
+    docker exec ${CONTAINER_NAME} powershell.exe -Command "`$ErrorActionPreference='Stop'; `$PSNativeCommandUseErrorActionPreference=`$true; cd app; switch-php zts; & 'C:\\php\\SDK\\phpize.bat'; if (`$LASTEXITCODE -ne 0) { exit `$LASTEXITCODE }; .\\configure.bat --enable-debug-pack; if (`$LASTEXITCODE -ne 0) { exit `$LASTEXITCODE }; nmake; if (`$LASTEXITCODE -ne 0) { exit `$LASTEXITCODE }; Move-Item x64\\Release_TS\\php_ddtrace.dll extensions_x86_64\\php_ddtrace-${ABI_NO}-zts.dll -ErrorAction Stop; Move-Item x64\\Release_TS\\php_ddtrace.pdb extensions_x86_64_debugsymbols\\php_ddtrace-${ABI_NO}-zts.pdb -ErrorAction Stop" 2>&1 | Tee-Object -FilePath zts-build.log
+    $ztsCode = $LASTEXITCODE
+    $ErrorActionPreference = 'Stop'
+    # Only transient network failures (e.g. crates.io DNS) get exit 75 for GitLab auto-retry; real compile breaks keep their native code and fail fast.
+    if ($ztsCode -ne 0) { if (Select-String -Path zts-build.log -Pattern 'Could not resolve host','spurious network error','failed to download' -Quiet) { Write-Host "Transient network failure during zts build; exiting 75 so GitLab auto-retries (see default retry.exit_codes in generate-common.php)"; exit 75 } else { exit $ztsCode } }
 
     # Try to stop the container, don't care if we fail
     try { docker stop -t 5 ${CONTAINER_NAME} } catch { }