Skip to content

ci: run integration tests on all three OSes (ubuntu, macos, windows) #79

Description

@SebTardif

Problem

The integration-test job in ci.yml currently only runs on macos-latest. Since VS Code is a cross-platform editor and our extension interacts with OS-level features (binary discovery, file system, PATH resolution), integration tests should cover all three supported platforms.

Current test matrix:

Job Ubuntu macOS Windows
unit-test yes yes yes
integration-test no yes no

Proposed changes

  1. Convert integration-test to a matrix job with os: [ubuntu-latest, macos-latest, windows-latest]
  2. Ubuntu: wrap VS Code test commands with xvfb-run to provide a virtual display (standard pattern for headless GUI testing)
  3. Windows: verify ExTester and @vscode/test-electron work on Windows runners; adjust paths if needed
  4. macOS: keep existing setup, including hide-test-vscode.sh (which already no-ops on non-Darwin)

Additional cleanup

Remove step-security/harden-runner from the integration-test job. It only supports Ubuntu VMs and prints a confusing warning on macOS:

This job is not running in a GitHub Actions Hosted Runner Ubuntu VM. Harden Runner is only supported on Ubuntu VM. This job will not be monitored.

Since it does nothing on macOS (or Windows), it should be removed from this job entirely. If the job is expanded to a matrix, harden-runner can be conditionally included only for the Ubuntu leg:

- name: Harden runner
  if: runner.os == 'Linux'
  uses: step-security/harden-runner@...
  with:
    egress-policy: audit

Cost

All runners are free for public repos. No billing impact.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions