[DRAFT] chore: drive acceptance skips from CircleCI context (CLI-1464)

.circleci/config.yml

@@ -763,6 +763,7 @@ workflows:
             - nodejs-install
             - team_hammerhead-cli
             - devex_cli_docker_hub
+            - team-cli-workflow-context
           filters:
             branches:
               ignore:
@@ -785,6 +786,7 @@ workflows:
             - nodejs-install
             - team_hammerhead-cli
             - devex_cli_docker_hub
+            - team-cli-workflow-context
           filters:
             branches:
               ignore:
@@ -807,6 +809,7 @@ workflows:
             - nodejs-install
             - team_hammerhead-cli
             - devex_cli_docker_hub
+            - team-cli-workflow-context
           filters:
             branches:
               ignore:
@@ -832,6 +835,7 @@ workflows:
             - nodejs-install
             - team_hammerhead-cli
             - devex_cli_docker_hub
+            - team-cli-workflow-context
           filters:
             branches:
               ignore:
@@ -851,6 +855,7 @@ workflows:
           context:
             - nodejs-install
             - team_hammerhead-cli
+            - team-cli-workflow-context
           filters:
             branches:
               ignore:
@@ -874,6 +879,7 @@ workflows:
           context:
             - nodejs-install
             - team_hammerhead-cli
+            - team-cli-workflow-context
           filters:
             branches:
               ignore:
@@ -896,6 +902,7 @@ workflows:
           context:
             - nodejs-install
             - team_hammerhead-cli
+            - team-cli-workflow-context
           filters:
             branches:
               ignore:
@@ -916,6 +923,7 @@ workflows:
           context:
             - nodejs-install
             - team_hammerhead-cli
+            - team-cli-workflow-context
           filters:
             branches:
               ignore:

.eslintrc.json

@@ -62,7 +62,8 @@
             "additionalTestBlockFunctions": [
               "testIf",
               "describeIf",
-              "testIf.each"
+              "testIf.each",
+              "acceptanceIt"
             ]
           }
         ]

CONTRIBUTING.md

@@ -158,6 +158,38 @@ You can run acceptance tests with:
 npm run test:acceptance -- --selectProjects coreCli
 ```
 
+#### Skipping acceptance spec files via CircleCI context (`TEST_SNYK_IGNORE_LIST`)
+
+When needed (for example blocking failures outside the CLI), CI can exclude specific acceptance specs by path without changing repo code. Set the environment variable **`TEST_SNYK_IGNORE_LIST`** to a comma-separated list of **regex fragments** that Jest merges into `testPathIgnorePatterns` (same semantics as Jest’s ignore patterns). Empty entries are ignored after trimming. Each fragment must compile as a JavaScript **`RegExp`** source; malformed fragments are skipped and listed on stderr under **`[acceptance ignore]`** so Jest still starts.
+
+- **CircleCI:** add the variable on context **`team-cli-workflow-context`** (name `TEST_SNYK_IGNORE_LIST`, value is only the pattern text—for example `snyk-code-user-journey\.spec\.ts`—not `TEST_SNYK_IGNORE_LIST=...`). Those workflows attach that context to **`acceptance-tests`** jobs so the env is available there.
+- **Precedence:** for paths that match a fragment, **`TEST_SNYK_IGNORE_LIST` wins over `TEST_SNYK_DONT_SKIP_ANYTHING`** (the file is not collected). `TEST_SNYK_DONT_SKIP_ANYTHING` still applies to specs that remain in the run.
+- **Observability:** **`console.warn`** on **stderr** with prefix **`[acceptance ignore]`**: skipped invalid fragments (if any), then applied fragments and precedence vs **`TEST_SNYK_DONT_SKIP_ANYTHING`**—each summary logged once per worker; avoid relying on stdout for this signal.
+
+`testPathIgnorePatterns` applies to whole files; it cannot skip a single `it()` inside a spec.
+
+#### Skipping individual acceptance tests via CircleCI context (`TEST_SNYK_SKIP_TEST_IDS`)
+
+CI can skip selected `it()` blocks inside specs that Jest still collects, using comma-separated **stable ids** (empty entries ignored after trimming). **Ids must not contain commas.**
+
+- **CircleCI:** add the variable on context **`team-cli-workflow-context`** (name `TEST_SNYK_SKIP_TEST_IDS`, value is only the id list—for example `snyk-code-user-journey:golang-native:ignored-issues:severity-threshold,snyk-code-user-journey:golang-native:ignored-issues:single-file`—not `TEST_SNYK_SKIP_TEST_IDS=...`). Those workflows attach that context to **`acceptance-tests`** jobs so the env is available there.
+- **Wiring:** specs use **`acceptanceIt(id)`** from [`test/jest/util/acceptanceTestSkipById.ts`](test/jest/util/acceptanceTestSkipById.ts); stable ids for the Snyk Code user-journey are **`SnykCodeUserJourneyContextSkipIds`** in that file.
+- **Precedence:** if a path matches **`TEST_SNYK_IGNORE_LIST`**, Jest never collects that spec—**`TEST_SNYK_SKIP_TEST_IDS` has no effect** on tests inside it. **`TEST_SNYK_DONT_SKIP_ANYTHING`** does **not** unsuspend **`acceptanceIt`** skips; clear **`TEST_SNYK_SKIP_TEST_IDS`** instead.
+- **Observability:** **`console.warn`** on **stderr** with prefix **`[acceptance skip tests]`**, listing ids—once per worker; avoid relying on stdout for this signal.
+
+Stable ids for the golang/native ignored-issues journeys:
+
+- `snyk-code-user-journey:golang-native:ignored-issues:severity-threshold`
+- `snyk-code-user-journey:golang-native:ignored-issues:include-ignores`
+- `snyk-code-user-journey:golang-native:ignored-issues:single-file`
+
+Local repro:
+
+```
+TEST_SNYK_SKIP_TEST_IDS='snyk-code-user-journey:golang-native:ignored-issues:severity-threshold' \
+npm run test:acceptance -- --selectProjects coreCli --runTestsByPath test/jest/acceptance/snyk-code/snyk-code-user-journey.spec.ts
+```
+
 ### Smoke Tests
 
 Smoke tests typically don't run on branches unless the branch is specifically prefixed with `smoke/`. They usually run

test/createJestConfig.js

@@ -1,4 +1,55 @@
-const createJestConfig = (config) => {
+/**
+ * Comma-separated env list: trim segments, drop empties. For literal string sets (no RegExp checks).
+ * @param {string} raw
+ * @returns {string[]}
+ */
+function parseCommaSeparatedEnvList(raw) {
+  if (typeof raw !== 'string' || raw.trim() === '') {
+    return [];
+  }
+  return raw
+    .split(',')
+    .map((s) => s.trim())
+    .filter(Boolean);
+}
+
+/**
+ * Comma-separated env list (`TEST_SNYK_IGNORE_LIST`): trim, drop empties,
+ * validate each fragment as a JavaScript RegExp source (same rule as Jest path patterns).
+ * @param {string} raw
+ * @returns {{ valid: string[], invalid: string[] }}
+ */
+function parseSnykIgnoreFragments(raw) {
+  if (typeof raw !== 'string' || raw.trim() === '') {
+    return { valid: [], invalid: [] };
+  }
+  const pieces = raw
+    .split(',')
+    .map((s) => s.trim())
+    .filter(Boolean);
+  const valid = [];
+  const invalid = [];
+  for (const f of pieces) {
+    try {
+      // Validate fragment as a JavaScript RegExp source (same as Jest path patterns).
+      // eslint-disable-next-line no-new
+      new RegExp(f);
+      valid.push(f);
+    } catch {
+      invalid.push(f);
+    }
+  }
+  return { valid, invalid };
+}
+
+function getSkipTestList() {
+  const raw = process.env.TEST_SNYK_IGNORE_LIST;
+  return parseSnykIgnoreFragments(typeof raw === 'string' ? raw : '');
+}
+
+let ignoreFragmentsWarned = false;
+
+const createJestConfig = (config = {}) => {
   const ignorePatterns = [
     '/node_modules/',
     '/dist/',
@@ -9,17 +60,44 @@ const createJestConfig = (config) => {
     '<rootDir>/pysrc/',
   ];
 
+  const { valid, invalid } = getSkipTestList();
+
+  if ((valid.length > 0 || invalid.length > 0) && !ignoreFragmentsWarned) {
+    ignoreFragmentsWarned = true;
+    if (invalid.length > 0) {
+      console.warn(
+        '[acceptance ignore]',
+        'Skipping invalid TEST_SNYK_IGNORE_LIST fragments (must compile as JavaScript RegExp sources):',
+        invalid,
+      );
+    }
+    if (valid.length > 0) {
+      console.warn(
+        '[acceptance ignore]',
+        valid,
+        'TEST_SNYK_IGNORE_LIST overrides TEST_SNYK_DONT_SKIP_ANYTHING for matching files.',
+      );
+    }
+  }
+
+  const { testPathIgnorePatterns, ...restConfig } = config;
+  const extraPathIgnores = Array.isArray(testPathIgnorePatterns)
+    ? testPathIgnorePatterns
+    : [];
+
   return {
     preset: 'ts-jest',
     testRegex: '\\.spec\\.ts$',
-    testPathIgnorePatterns: [...ignorePatterns],
+    testPathIgnorePatterns: [...ignorePatterns, ...valid, ...extraPathIgnores],
     modulePathIgnorePatterns: [...ignorePatterns],
     coveragePathIgnorePatterns: [...ignorePatterns],
     transformIgnorePatterns: [...ignorePatterns],
-    ...config,
+    ...restConfig,
   };
 };
 
 module.exports = {
   createJestConfig,
+  parseCommaSeparatedEnvList,
+  parseSnykIgnoreFragments,
 };

test/jest/acceptance/resilience.spec.ts

@@ -57,6 +57,55 @@ interface ResilienceScenario {
   skip?: string[]; // Commands to skip for this scenario (not yet consistent)
 }
 
+describe('TEST_SNYK_IGNORE_LIST → testPathIgnorePatterns', () => {
+  // eslint-disable-next-line @typescript-eslint/no-require-imports
+  const { createJestConfig } = require('../../createJestConfig') as {
+    createJestConfig: (config?: object) => {
+      testPathIgnorePatterns: string[];
+    };
+  };
+  let previousIgnoreList: string | undefined;
+
+  beforeEach(() => {
+    previousIgnoreList = process.env.TEST_SNYK_IGNORE_LIST;
+  });
+
+  afterEach(() => {
+    if (previousIgnoreList === undefined) {
+      delete process.env.TEST_SNYK_IGNORE_LIST;
+    } else {
+      process.env.TEST_SNYK_IGNORE_LIST = previousIgnoreList;
+    }
+  });
+
+  it('when unset or empty, does not add fragments from TEST_SNYK_IGNORE_LIST', () => {
+    for (const value of [undefined as string | undefined, '']) {
+      if (value === undefined) {
+        delete process.env.TEST_SNYK_IGNORE_LIST;
+      } else {
+        process.env.TEST_SNYK_IGNORE_LIST = value;
+      }
+      const ignorePathPatterns = createJestConfig({}).testPathIgnorePatterns;
+      expect(ignorePathPatterns).toContain('/node_modules/');
+      expect(ignorePathPatterns).not.toContain('happy-path-one');
+    }
+  });
+
+  it('single comma-separated pattern is merged', () => {
+    process.env.TEST_SNYK_IGNORE_LIST = 'happy-path-one';
+    expect(createJestConfig({}).testPathIgnorePatterns).toContain(
+      'happy-path-one',
+    );
+  });
+
+  it('two comma-separated patterns are merged', () => {
+    process.env.TEST_SNYK_IGNORE_LIST = 'happy-path-a, happy-path-b';
+    const ignorePathPatterns = createJestConfig({}).testPathIgnorePatterns;
+    expect(ignorePathPatterns).toContain('happy-path-a');
+    expect(ignorePathPatterns).toContain('happy-path-b');
+  });
+});
+
 const RESILIENCE_SCENARIOS: ResilienceScenario[] = [
   // Scenario 1
   {

test/jest/acceptance/snyk-code/snyk-code-user-journey.spec.ts

@@ -16,6 +16,10 @@ import {
   createFilepaths,
   deleteFilepaths,
 } from '../../../jest/util/fileIgnoreRulesFixture';
+import {
+  acceptanceIt,
+  SnykCodeUserJourneyContextSkipIds,
+} from '../../util/acceptanceTestSkipById';
 
 expect.extend(matchers);
 jest.setTimeout(1000 * 300);
@@ -774,8 +778,10 @@ describe('snyk code test', () => {
                   }
                 });
 
-                // TODO: Temporarily skipped - investigate persistent CI failures
-                it.skip('with --severity-threshold', async () => {
+                // `TEST_SNYK_SKIP_TEST_IDS` (CircleCI context) → `acceptanceIt(<id>)`
+                acceptanceIt(
+                  SnykCodeUserJourneyContextSkipIds.GOLANG_NATIVE_IGNORED_ISSUES_SEVERITY_THRESHOLD,
+                )('with --severity-threshold', async () => {
                   const { stdout, stderr, code } = await runSnykCLI(
                     `code test ${pathToTest} --severity-threshold=high --sarif-file-output=${sarifFile}`,
                     {
@@ -805,8 +811,9 @@ describe('snyk code test', () => {
                   expect(levels.length).toBe(0);
                 });
 
-                // TODO: Temporarily skipped - investigate persistent CI failures
-                it.skip('with --include-ignores', async () => {
+                acceptanceIt(
+                  SnykCodeUserJourneyContextSkipIds.GOLANG_NATIVE_IGNORED_ISSUES_INCLUDE_IGNORES,
+                )('with --include-ignores', async () => {
                   const { stdout, stderr, code } = await runSnykCLI(
                     `code test ${pathToTest} --include-ignores --sarif-file-output=${sarifFile}`,
                     {
@@ -831,7 +838,9 @@ describe('snyk code test', () => {
           );
 
           describe(`with ignored issues`, () => {
-            it.skip('test a single file', async () => {
+            acceptanceIt(
+              SnykCodeUserJourneyContextSkipIds.GOLANG_NATIVE_IGNORED_ISSUES_SINGLE_FILE,
+            )('test a single file', async () => {
               const { stderr, code } = await runSnykCLI(
                 `code test ${localPath}/routes/index.js --sarif-file-output=${sarifFile}`,
                 {

test/jest/unit/acceptanceTestSkipById.spec.ts

@@ -0,0 +1,78 @@
+describe('acceptanceTestSkipById (TEST_SNYK_SKIP_TEST_IDS)', () => {
+  afterEach(() => {
+    delete process.env.TEST_SNYK_SKIP_TEST_IDS;
+    jest.resetModules();
+    jest.restoreAllMocks();
+  });
+
+  async function loadModule() {
+    jest.resetModules();
+    return import('../util/acceptanceTestSkipById');
+  }
+
+  it('returns global `it` when TEST_SNYK_SKIP_TEST_IDS is unset', async () => {
+    delete process.env.TEST_SNYK_SKIP_TEST_IDS;
+    const { acceptanceIt } = await loadModule();
+    expect(acceptanceIt('any-stable-id')).toBe(it);
+  });
+
+  it('returns global `it` when TEST_SNYK_SKIP_TEST_IDS is empty or whitespace-only', async () => {
+    process.env.TEST_SNYK_SKIP_TEST_IDS = '   ';
+    const { acceptanceIt } = await loadModule();
+    expect(acceptanceIt('any-stable-id')).toBe(it);
+  });
+
+  it('returns `it.skip` when the id is listed', async () => {
+    jest.spyOn(console, 'warn').mockImplementation(() => {});
+    process.env.TEST_SNYK_SKIP_TEST_IDS = 'stable-id-one';
+    const { acceptanceIt } = await loadModule();
+    expect(acceptanceIt('stable-id-one')).toBe(it.skip);
+    expect(acceptanceIt('other-id')).toBe(it);
+  });
+
+  it('parses comma-separated ids with trimming and ignores empty segments', async () => {
+    jest.spyOn(console, 'warn').mockImplementation(() => {});
+    process.env.TEST_SNYK_SKIP_TEST_IDS = ' a , , b ';
+    const { acceptanceIt } = await loadModule();
+    expect(acceptanceIt('a')).toBe(it.skip);
+    expect(acceptanceIt('b')).toBe(it.skip);
+    expect(acceptanceIt('c')).toBe(it);
+  });
+
+  it('treats stable ids as literal strings, not RegExp sources (e.g. `[` in the id)', async () => {
+    jest.spyOn(console, 'warn').mockImplementation(() => {});
+    const id = 'suite:case:with[bracket-unclosed';
+    process.env.TEST_SNYK_SKIP_TEST_IDS = id;
+    const { acceptanceIt } = await loadModule();
+    expect(acceptanceIt(id)).toBe(it.skip);
+    expect(acceptanceIt('other-id')).toBe(it);
+  });
+
+  it('logs [acceptance skip tests] once per module load when ids are present', async () => {
+    const warn = jest.spyOn(console, 'warn').mockImplementation(() => {});
+    process.env.TEST_SNYK_SKIP_TEST_IDS = 'x,y';
+    const { acceptanceIt } = await loadModule();
+    acceptanceIt('x');
+    acceptanceIt('y');
+    expect(warn).toHaveBeenCalledTimes(1);
+    expect(warn.mock.calls[0][0]).toBe('[acceptance skip tests]');
+    expect(warn.mock.calls[0][1]).toEqual(['x', 'y']);
+  });
+
+  it('exports documented stable id constants for the Snyk Code user journey', async () => {
+    const { SnykCodeUserJourneyContextSkipIds } = await loadModule();
+    expect(
+      SnykCodeUserJourneyContextSkipIds.GOLANG_NATIVE_IGNORED_ISSUES_SEVERITY_THRESHOLD,
+    ).toBe(
+      'snyk-code-user-journey:golang-native:ignored-issues:severity-threshold',
+    );
+    expect(
+      SnykCodeUserJourneyContextSkipIds.GOLANG_NATIVE_IGNORED_ISSUES_INCLUDE_IGNORES,
+    ).toBe(
+      'snyk-code-user-journey:golang-native:ignored-issues:include-ignores',
+    );
+    expect(
+      SnykCodeUserJourneyContextSkipIds.GOLANG_NATIVE_IGNORED_ISSUES_SINGLE_FILE,
+    ).toBe('snyk-code-user-journey:golang-native:ignored-issues:single-file');
+  });
+});