Fix upload-result-diff and azure-login pipeline failures (#197) (#218)

* Fix upload-result-diff and azure-login pipeline failures

- Skip Azure Login, Download, Upload, and comparison steps on
  pull_request events where OIDC secrets are unavailable
- Separate the "Upload result diff" step from the "Fail if diff
  detected" step so uploads succeed independently and failures
  have clear error messages
- Apply fixes to both test-query-health and
  test-codeql-latest-vs-current jobs

Agent-Logs-Url: https://github.com/microsoft/Windows-Driver-Developer-Supplemental-Tools/sessions/9c85d93e-42c1-4451-81ad-2a71ece7bc2e

Co-authored-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>

* Fix empty-string arg passed to test script on PR runs

Building the python command with an empty $compareFlag variable
caused PowerShell to pass an empty string as a positional arg,
producing: "unrecognized arguments: ". Switched to building an
argument array and splatting it so the flag is only included when
present.

Applied to both test-query-health and test-codeql-latest-vs-current.

Agent-Logs-Url: https://github.com/microsoft/Windows-Driver-Developer-Supplemental-Tools/sessions/a5ce6b58-e7aa-4b94-b731-d0242b47b40e

Co-authored-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>

* Surface msbuild errors from codeql database create and fail loudly on test failures

Agent-Logs-Url: https://github.com/microsoft/Windows-Driver-Developer-Supplemental-Tools/sessions/1a592ad2-4448-490f-ae16-a9c34d72bf79

Co-authored-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>

* Remove stray .pyc accidentally committed in previous commit

Agent-Logs-Url: https://github.com/microsoft/Windows-Driver-Developer-Supplemental-Tools/sessions/1a592ad2-4448-490f-ae16-a9c34d72bf79

Co-authored-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>

* Optimize pipeline for faster PR runs

Removed unnecessary build steps for CA ported queries (covered by the build-all step) and adjusted dependencies in workflow to speed up PR runs (test steps will build as part of work.)

Signed-off-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>

* Add Directory.Build.props to wire WDK/SDK NuGet packages into driver test projects

Agent-Logs-Url: https://github.com/microsoft/Windows-Driver-Developer-Supplemental-Tools/sessions/7795692b-bc9c-4c2e-a3d3-18f00aed9bcb

Co-authored-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>

* Fix stampinf, ApiValidator, and Dvl.exe failures in CI tests

- Remove <Inf> items from KMDFTestTemplate.vcxproj and CppKMDFTestTemplate.vcxproj
  so the StampInf MSBuild task is not triggered (stampinf.exe is not available in
  a NuGet-only WDK environment).  These templates exist solely for CodeQL analysis,
  not for building a production driver, so INF stamping is not needed.

- Add <ApiValidatorEnabled>false</ApiValidatorEnabled> to Directory.Build.props.
  ApiValidator.exe is not included in the WDK NuGet packages and is not available
  in the CI environment.  Disabling it allows the ApplicationForDriversTestTemplate
  (WindowsApplicationForDrivers10.0 toolset) to compile successfully for CodeQL.

- Update dvl_tests.ps1 to locate Dvl.exe dynamically from the NuGet packages
  directory instead of assuming the traditional system WDK install path
  C:\Program Files (x86)\Windows Kits\10\Tools\dvl\Dvl.exe.  Falls back to the
  system path for developer machines with a full WDK install, and gracefully skips
  the dvl command-type tests when Dvl.exe is not found in either location.

- Fix typo -test_emtpy -> -test_empty in the second Test-DVL "dvl" call inside
  Test-Driver (the typo was previously unreachable because the test exited earlier
  due to the Dvl.exe failure).

Agent-Logs-Url: https://github.com/microsoft/Windows-Driver-Developer-Supplemental-Tools/sessions/8484509b-2924-427f-bd9a-63e365e4b404

Co-authored-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>

* plan: fix ApiValidator and parallelize test runner

Agent-Logs-Url: https://github.com/microsoft/Windows-Driver-Developer-Supplemental-Tools/sessions/71e6036e-2c98-4bb7-85ab-2270cbce9c68

Co-authored-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>

* Fix ApiValidator failures, harden Dvl.exe check, parallelize test runner

- Add src/drivers/test/Directory.Build.targets that overrides the WDK's
  ApiValidator MSBuild target with an empty target.  The WDK NuGet packages
  reference ApiValidator.exe in WindowsDriver.common.targets but do not ship
  the binary itself, so the post-build step always failed with MSB3721 for
  builds using the Universal driver target platform.  Replacing the target
  with a no-op cleanly suppresses the step; API validation is irrelevant
  for the CodeQL static analysis these tests perform.  This is the cause of
  the 6 still-failing tests (UnsafeCallInGlobalInit, MultithreadedAVCondition,
  StaticInitializer, DeviceInitApi, FloatSafeExit, FloatUnsafeExit).

- Remove the previous <ApiValidatorEnabled>false</ApiValidatorEnabled>
  workaround from Directory.Build.props -- the WDK targets do not honor that
  property name, so it had no effect.

- dvl_tests.ps1: when Dvl.exe cannot be located in either the NuGet packages
  directory or the system WDK install path, exit with a clear failure instead
  of silently skipping the dvl command-type tests.  Skipping let regressions
  go undetected.

- build_create_analyze_test.py: parallelize the test runner.  Each ql_test
  uses isolated working/<name>, TestDB/<name>, and AnalysisFiles/<name>.sarif
  paths, so multiple tests are safe to execute concurrently.  Use
  multiprocessing.pool.ThreadPool (already imported) with a worker count
  controlled by a new -j/--jobs flag, defaulting to os.cpu_count().  Pass
  --jobs 1 to fall back to the legacy sequential behaviour.  Print output
  is already protected by print_mutex; added results_mutex around the shared
  health_df / detailed_health_df DataFrame writes.  Refactored the body of
  the per-test loop into _run_single_test for use by the pool's
  imap_unordered.

Agent-Logs-Url: https://github.com/microsoft/Windows-Driver-Developer-Supplemental-Tools/sessions/71e6036e-2c98-4bb7-85ab-2270cbce9c68

Co-authored-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>

* Parallelize test-query-health and test-codeql-latest-vs-current jobs

Agent-Logs-Url: https://github.com/microsoft/Windows-Driver-Developer-Supplemental-Tools/sessions/694e9d3d-f942-4564-8e96-8af21abeb87a

Co-authored-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>

* Fix LNK1318 mspdbsrv race; wire --jobs into workflow

Agent-Logs-Url: https://github.com/microsoft/Windows-Driver-Developer-Supplemental-Tools/sessions/694e9d3d-f942-4564-8e96-8af21abeb87a

Co-authored-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>

* Pipeline tweaks: progress log, parallel query compile, scoped pack-version gate, broader publish needs

Agent-Logs-Url: https://github.com/microsoft/Windows-Driver-Developer-Supplemental-Tools/sessions/96f4dbad-dab0-45e8-9a14-94efd2c96d27

Co-authored-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>

---------

Signed-off-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: NateD-MSFT <34494373+NateD-MSFT@users.noreply.github.com>

Author: Copilot

.github/workflows/build-codeql.yaml

@@ -52,23 +52,18 @@ jobs:
 
     - name: Build must-fix driver suite
       shell: cmd
-      run: .\codeql-cli\codeql.cmd query compile --check-only mustfix.qls
+      run: .\codeql-cli\codeql.cmd query compile --check-only --threads=0 mustfix.qls
 
     - name: Build recommended driver suite
       shell: cmd
-      run: .\codeql-cli\codeql.cmd query compile --check-only recommended.qls
-    
-    - name: Build CA ported queries
-      shell: cmd
-      run: .\codeql-cli\codeql.cmd query compile --check-only ported_driver_ca_checks.qls
+      run: .\codeql-cli\codeql.cmd query compile --check-only --threads=0 recommended.qls
 
     - name: Build all Windows queries
       shell: cmd
-      run: .\codeql-cli\codeql.cmd query compile --check-only .\src
+      run: .\codeql-cli\codeql.cmd query compile --check-only --threads=0 .\src
 
   test-query-health:
     runs-on: windows-latest
-    needs: build
     permissions:
       contents: read
       packages: write
@@ -111,13 +106,15 @@ jobs:
     - name: Add msbuild to PATH
       uses: microsoft/setup-msbuild@v2
     - name: Azure Login
+      if: github.event_name != 'pull_request'
       uses: azure/login@v2
       with:
         client-id: ${{ secrets.AZURE_CLIENT_ID }}
         tenant-id: ${{ secrets.AZURE_TENANT_ID }}
         subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
         enable-AzPSSession: true
     - name: Download previous results
+      if: github.event_name != 'pull_request'
       uses: azure/powershell@v2
       with:
         azPSVersion: latest
@@ -127,24 +124,40 @@ jobs:
           Get-AzStorageFileContent -ShareName "$env:SHARE_NAME" -Path "detailedfunctiontestresults.xlsx" -Destination $destination -Context $context
     - name: Run test script
       shell: pwsh
-      run: python src\drivers\test\build_create_analyze_test.py --codeql_path .\codeql-cli\codeql.exe --no_build --compare_results -v
+      run: |
+        # Run per-test build/analyze in parallel inside the script. Default is
+        # one worker per logical CPU (--jobs <N>); each worker is isolated to
+        # its own working/, TestDB/, and AnalysisFiles/<name>.sarif paths.
+        $pyArgs = @('src\drivers\test\build_create_analyze_test.py', '--codeql_path', '.\codeql-cli\codeql.exe', '--no_build', '-v', '--jobs', "$env:NUMBER_OF_PROCESSORS")
+        if ("${{ github.event_name }}" -ne "pull_request") {
+          $pyArgs += '--compare_results'
+        }
+        python @pyArgs
     - name: Upload result diff
-      if: ${{ hashFiles('diffdetailedfunctiontestresults.xlsx') != '' }} # Only upload if there are changes
+      if: github.event_name != 'pull_request' && hashFiles('diffdetailedfunctiontestresults.xlsx') != ''
       uses: azure/powershell@v2
       with:
         azPSVersion: latest
         inlineScript: |
           Update-AzConfig -DisplayBreakingChangeWarning $false
           $context = New-AzStorageContext -StorageAccountName "$env:ACCOUNT_NAME" -UseConnectedAccount -EnableFileBackupRequestIntent
           Set-AzStorageFileContent -ShareName "$env:SHARE_NAME" -Source "diffdetailedfunctiontestresults.xlsx" -Path "health-diffdetailedfunctiontestresults.xlsx" -Context $context -Force
-          exit 1 
+    - name: Fail if result diff detected
+      if: github.event_name != 'pull_request' && hashFiles('diffdetailedfunctiontestresults.xlsx') != ''
+      shell: pwsh
+      run: |
+        Write-Host "::error::Test results differ from the stored baseline. The diff has been uploaded to Azure Storage as 'health-diffdetailedfunctiontestresults.xlsx'. Please review."
+        exit 1
     
 
   test-codeql-latest-vs-current:
-  # Tests if the latest codeql version produces the same results as the current version. 
+  # Tests if the latest codeql version produces the same results as the current version.
+  # Runs in parallel with `test-query-health` (no `needs:` dependency) to halve the
+  # pipeline's wall-clock time. It is independent: it downloads its own (latest)
+  # CodeQL CLI and runs the same per-test build/analyze cycle. `continue-on-error`
+  # below means failures here never block the workflow regardless of order.
     runs-on: windows-latest
     continue-on-error: true # Allow script to return non-zero exit code
-    needs: [build,test-query-health]
     permissions:
       contents: read
       packages: write
@@ -153,10 +166,6 @@ jobs:
       ACCOUNT_NAME: ${{ secrets.ACCOUNT_NAME }}
       SHARE_NAME: ${{ secrets.SHARE_NAME }}
     steps:
-    - name: Check Prev Job
-      if: ${{ needs.test-query-health.result == 'failure' }}
-      shell: pwsh
-      run: exit 1  
     - name: Enable long git paths
       shell: cmd
       run: git config --global core.longpaths true
@@ -194,13 +203,15 @@ jobs:
     - name: Add msbuild to PATH
       uses: microsoft/setup-msbuild@v2
     - name: Azure Login
+      if: github.event_name != 'pull_request'
       uses: azure/login@v2
       with:
         client-id: ${{ secrets.AZURE_CLIENT_ID }}
         tenant-id: ${{ secrets.AZURE_TENANT_ID }}
         subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
         enable-AzPSSession: true
     - name: Download previous results
+      if: github.event_name != 'pull_request'
       uses: azure/powershell@v2
       with:
         azPSVersion: latest
@@ -210,16 +221,29 @@ jobs:
           Get-AzStorageFileContent -ShareName "$env:SHARE_NAME" -Path "detailedfunctiontestresults.xlsx" -Destination $destination -Context $context
     - name: Run test script
       shell: pwsh
-      run: python src\drivers\test\build_create_analyze_test.py --codeql_path .\codeql-cli\codeql.exe --no_build --compare_results -v
+      run: |
+        # Run per-test build/analyze in parallel inside the script. Default is
+        # one worker per logical CPU (--jobs <N>); each worker is isolated to
+        # its own working/, TestDB/, and AnalysisFiles/<name>.sarif paths.
+        $pyArgs = @('src\drivers\test\build_create_analyze_test.py', '--codeql_path', '.\codeql-cli\codeql.exe', '--no_build', '-v', '--jobs', "$env:NUMBER_OF_PROCESSORS")
+        if ("${{ github.event_name }}" -ne "pull_request") {
+          $pyArgs += '--compare_results'
+        }
+        python @pyArgs
     - name: Upload result diff
-      if: ${{ hashFiles('diffdetailedfunctiontestresults.xlsx') != '' }} # Only upload if there are changes
+      if: github.event_name != 'pull_request' && hashFiles('diffdetailedfunctiontestresults.xlsx') != ''
       uses: azure/powershell@v2
       with:
         azPSVersion: latest
         inlineScript: |
           $context = New-AzStorageContext -StorageAccountName "$env:ACCOUNT_NAME" -UseConnectedAccount -EnableFileBackupRequestIntent
           Set-AzStorageFileContent -ShareName "$env:SHARE_NAME" -Source "diffdetailedfunctiontestresults.xlsx" -Path "version-diffdetailedfunctiontestresults.xlsx" -Context $context -Force
-          exit 1 
+    - name: Fail if result diff detected
+      if: github.event_name != 'pull_request' && hashFiles('diffdetailedfunctiontestresults.xlsx') != ''
+      shell: pwsh
+      run: |
+        Write-Host "::error::Test results from latest CodeQL version differ from the stored baseline. The diff has been uploaded to Azure Storage as 'version-diffdetailedfunctiontestresults.xlsx'. Please review."
+        exit 1
     - name: Save Latest Version
       if: ${{ hashFiles('diffdetailedfunctiontestresults.xlsx') == '' }} # Only if there were no differences
       uses: actions/upload-artifact@v4
@@ -230,7 +254,13 @@ jobs:
           
   test-pack-version-update:
     runs-on: windows-latest
-    needs: build
+    # Only enforce qlpack version bumps when the change is actually heading to
+    # `main`. We routinely stage many commits in `development` and bump the
+    # qlpack version once when promoting to `main`, so requiring a bump on
+    # every `development`-targeted PR/push is noise.
+    if: |
+      (github.event_name == 'pull_request' && github.base_ref == 'main') ||
+      (github.event_name != 'pull_request' && github.ref == 'refs/heads/main')
     permissions:
       contents: read
       packages: write
@@ -272,7 +302,6 @@ jobs:
         }
   test-create-dvl:
     runs-on: windows-latest
-    needs: build
     permissions:
       contents: read
       packages: write
@@ -319,7 +348,19 @@ jobs:
   publish:
     runs-on: windows-latest
     continue-on-error: true 
-    needs: [build, test-pack-version-update, test-query-health]
+    needs: [build, test-pack-version-update, test-query-health, test-codeql-latest-vs-current, test-create-dvl]
+    # Run when all required gates pass. `test-pack-version-update` is skipped
+    # for non-`main` targets (see its `if:` above), so allow `success` *or*
+    # `skipped`. `test-codeql-latest-vs-current` is `continue-on-error: true`,
+    # which already produces a `success` result for `needs`, so we don't need
+    # special handling for it here -- listing it in `needs` just makes publish
+    # wait for it to finish before running.
+    if: |
+      always() &&
+      needs.build.result == 'success' &&
+      needs.test-query-health.result == 'success' &&
+      needs.test-create-dvl.result == 'success' &&
+      (needs.test-pack-version-update.result == 'success' || needs.test-pack-version-update.result == 'skipped')
     permissions:
       contents: read
       packages: write

src/drivers/test/Directory.Build.props

@@ -0,0 +1,23 @@
+<Project>
+  <!--
+    Auto-imported by MSBuild for every project under src\drivers\test\ (both the
+    TestTemplates\ originals and the generated working\ copies used by
+    build_create_analyze_test.py). Wires the WDK and Windows SDK CPP NuGet
+    packages restored by the workflow's `nuget restore ... -PackagesDirectory
+    .\packages\` step into the driver template projects so they can find wdm.h,
+    ntifs.h, wdf.h, etc.
+
+    Mirrors the pattern used by microsoft/Windows-driver-samples
+    (see https://github.com/microsoft/Windows-driver-samples/blob/main/Directory.Build.props),
+    keeping the package set and versions in sync with packages.config in this
+    same directory.
+  -->
+  <PropertyGroup>
+    <WdkNuGetPackagesDir>$(MSBuildThisFileDirectory)..\..\..\packages\</WdkNuGetPackagesDir>
+  </PropertyGroup>
+  <Import Project="$(WdkNuGetPackagesDir)Microsoft.Windows.WDK.x64.10.0.26100.6584\build\native\Microsoft.Windows.WDK.x64.props" Condition="Exists('$(WdkNuGetPackagesDir)Microsoft.Windows.WDK.x64.10.0.26100.6584\build\native\Microsoft.Windows.WDK.x64.props') and '$(Platform)' == 'x64'" />
+  <Import Project="$(WdkNuGetPackagesDir)Microsoft.Windows.WDK.arm64.10.0.26100.6584\build\native\Microsoft.Windows.WDK.arm64.props" Condition="Exists('$(WdkNuGetPackagesDir)Microsoft.Windows.WDK.arm64.10.0.26100.6584\build\native\Microsoft.Windows.WDK.arm64.props') and '$(Platform)' == 'ARM64'" />
+  <Import Project="$(WdkNuGetPackagesDir)Microsoft.Windows.SDK.CPP.x64.10.0.26100.6584\build\native\Microsoft.Windows.SDK.cpp.x64.props" Condition="Exists('$(WdkNuGetPackagesDir)Microsoft.Windows.SDK.CPP.x64.10.0.26100.6584\build\native\Microsoft.Windows.SDK.cpp.x64.props') and '$(Platform)' == 'x64'" />
+  <Import Project="$(WdkNuGetPackagesDir)Microsoft.Windows.SDK.CPP.arm64.10.0.26100.6584\build\native\Microsoft.Windows.SDK.cpp.arm64.props" Condition="Exists('$(WdkNuGetPackagesDir)Microsoft.Windows.SDK.CPP.arm64.10.0.26100.6584\build\native\Microsoft.Windows.SDK.cpp.arm64.props') and '$(Platform)' == 'ARM64'" />
+  <Import Project="$(WdkNuGetPackagesDir)Microsoft.Windows.SDK.CPP.10.0.26100.6584\build\native\Microsoft.Windows.SDK.cpp.props" Condition="Exists('$(WdkNuGetPackagesDir)Microsoft.Windows.SDK.CPP.10.0.26100.6584\build\native\Microsoft.Windows.SDK.cpp.props')" />
+</Project>

src/drivers/test/Directory.Build.targets

@@ -0,0 +1,28 @@
+<Project>
+  <!--
+    Auto-imported by MSBuild AFTER each project's main targets (including the
+    WDK targets pulled in by Directory.Build.props), for every project under
+    src\drivers\test\.
+
+    Override the WDK's `ApiValidator` post-build target with an empty target.
+    The WDK NuGet packages restored by the CI workflow do not ship the
+    `ApiValidator.exe` binary even though they reference its path in
+    `WindowsDriver.common.targets`.  Without this override the target fails
+    with:
+
+        error MSB3721: The command "...\ApiValidator.exe -DriverPackagePath:..."
+        exited with code 1
+
+    breaking the `codeql database create` invocation for tests built with the
+    Universal driver target platform (UnsafeCallInGlobalInit,
+    MultithreadedAVCondition, StaticInitializer, DeviceInitApi, FloatSafeExit,
+    FloatUnsafeExit, ...).
+
+    API validation is irrelevant to the CodeQL static analysis these tests
+    perform, so replacing the target with a no-op is safe here.  Setting
+    `ApiValidatorExePath` to an empty value would still leave the target
+    running an empty exec; replacing the target itself is the most reliable
+    suppression.
+  -->
+  <Target Name="ApiValidator" />
+</Project>

src/drivers/test/TestTemplates/CppKMDFTestTemplate/CppKMDFTestTemplate.vcxproj

@@ -33,9 +33,6 @@
     <ClInclude Include="Queue.h" />
     <ClInclude Include="Trace.h" />
   </ItemGroup>
-  <ItemGroup>
-    <Inf Include="CppKMDFTestTemplate.inf" />
-  </ItemGroup>
   <PropertyGroup Label="Globals">
     <ProjectGuid>{0B59834A-7319-449C-822B-09B4CFAC9752}</ProjectGuid>
     <TemplateGuid>{8c0e3d8b-df43-455b-815a-4a0e72973bc6}</TemplateGuid>

src/drivers/test/TestTemplates/KMDFTestTemplate/KMDFTestTemplate.vcxproj

@@ -33,9 +33,6 @@
     <ClInclude Include="Queue.h" />
     <ClInclude Include="Trace.h" />
   </ItemGroup>
-  <ItemGroup>
-    <Inf Include="KMDFTestTemplate.inf" />
-  </ItemGroup>
   <PropertyGroup Label="Globals">
     <ProjectGuid>{AD97E1A9-DDBC-4BC2-B3B8-95D11062B471}</ProjectGuid>
     <TemplateGuid>{8c0e3d8b-df43-455b-815a-4a0e72973bc6}</TemplateGuid>