Skip to content

Commit d43c37c

Browse files
committed
Add testing harness for grammar
1 parent 809e22f commit d43c37c

12 files changed

+683
-1
lines changed

language/README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Language Grammars And Syntax Tests
2+
3+
Workflow grammar assets live in `language/syntaxes/`.
4+
5+
For syntax-highlighting triage guidance and fixture-based regression test patterns, see:
6+
7+
- `src/workflow/syntax/README.md`

language/syntaxes/expressions.tmGrammar.json

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@
7272
]
7373
},
7474
"if-expression": {
75-
"match": "^\\s*\\b(if:) (.*?)$",
75+
"match": "\\b(if:)\\s+((?:'(?:''|[^'])*'|[^#\\n])+?)(\\s+#.*)?$",
7676
"contentName": "meta.embedded.block.github-actions-expression",
7777
"captures": {
7878
"1": {
@@ -88,6 +88,13 @@
8888
"include": "#expression"
8989
}
9090
]
91+
},
92+
"3": {
93+
"patterns": [
94+
{
95+
"include": "source.github-actions-workflow"
96+
}
97+
]
9198
}
9299
}
93100
},

src/secrets/index.test.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import {describe, expect, it} from "@jest/globals";
12
import libsodium from "libsodium-wrappers";
23
import {encodeSecret} from "./index";
34

src/workflow/syntax/README.md

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# Workflow Syntax Highlighting Triage & Tests
2+
3+
This note documents a lightweight process for triaging syntax-highlighting bugs in workflow files and turning them into fixture-based regression tests.
4+
5+
## What this covers
6+
7+
This is for **TextMate grammar / tokenization** issues in:
8+
9+
- `language/syntaxes/yaml.tmLanguage.json`
10+
- `language/syntaxes/expressions.tmGrammar.json`
11+
- workflow syntax injection grammars (for embedded languages)
12+
13+
This is **not** the right path for:
14+
15+
- parser/validation diagnostics from language services
16+
- schema/completion issues
17+
- runtime extension behavior
18+
19+
## Triage checklist (quick)
20+
21+
1. Reproduce in `GitHub Actions Workflow` language mode.
22+
2. Run `Developer: Inspect Editor Tokens and Scopes`.
23+
3. Check whether the bug is:
24+
- wrong token scopes/colors (grammar bug)
25+
- a diagnostic/problem message (language service/parser bug)
26+
4. Identify likely grammar file:
27+
- inline `${{ }}` / `if:` expression behavior: `language/syntaxes/expressions.tmGrammar.json`
28+
- general YAML tokenization/comments/keys/scalars: `language/syntaxes/yaml.tmLanguage.json`
29+
- embedded JS/shell/etc: injection grammar(s)
30+
5. Add a fixture and a focused regression test before patching.
31+
32+
## What to ask for in a bug report
33+
34+
If the issue is syntax highlighting, ask for:
35+
36+
- a minimal workflow snippet (`.yml`)
37+
- exact line/token that looks wrong
38+
- screenshot (optional but helpful)
39+
- token inspector output for the wrong token (`textmate scopes`)
40+
- expected behavior (what scope/color should have happened)
41+
42+
Ideally, contributors can include a minimal repro snippet that can be copied directly into a fixture file.
43+
44+
## Current test utilities
45+
46+
Shared helpers live in:
47+
48+
- `src/workflow/syntax/syntax-test-utils.ts`
49+
50+
Current tests live in:
51+
52+
- `src/workflow/syntax/*.test.ts`
53+
54+
Current fixture files live in:
55+
56+
- `src/workflow/syntax/fixtures/`
57+
58+
The helpers are intentionally lightweight and focus on grammar-regression behavior (not VS Code integration tests).
59+
60+
## Which helper to use
61+
62+
- `readJson(relativePath)`
63+
- Use to load grammar JSON files from `language/syntaxes/`.
64+
- `readFixture(relativePath)`
65+
- Use to load YAML fixture files from `src/workflow/syntax/fixtures/`.
66+
- `analyzeSingleOuterEmbeddedBlockFixture(...)`
67+
- Use when grammar has one outer context and one embedded block rule inside it (for example `github-script` + `with.script`).
68+
- `analyzeTopLevelInjectionContexts(...)`
69+
- Use when grammar has multiple top-level included contexts (for example `run` + `shell` per-shell contexts).
70+
- `findGithubActionsInlineExpression(line)`
71+
- Use in expression-regression tests that need to ensure `${{ ... }}` does not terminate on `}}` inside quoted strings (for example `#223`).
72+
73+
## Fixture naming
74+
75+
Use behavior-based, kebab-case fixture names:
76+
77+
- format: `<behavior>.yml`
78+
- examples:
79+
- `if-comment-after-string.yml`
80+
- `expression-nested-braces.yml`
81+
- `run-shell-embedded.yml`
82+
83+
Avoid issue-number-only names in fixture filenames. Issue references should live in test comments or fixture comments.
84+
85+
## Adding a new grammar regression test
86+
87+
1. Add a minimal fixture file under `src/workflow/syntax/fixtures/`
88+
2. Add/extend a Jest test in `src/workflow/syntax/*.test.ts`
89+
3. Keep assertions narrow (what should be embedded, what should not be consumed, header/body boundaries, etc.)
90+
4. Run `npm test`
91+
92+
## Example: `#531`-style triage (inline comment after `if:`)
93+
94+
Issue type:
95+
96+
- likely grammar tokenization bug in `language/syntaxes/expressions.tmGrammar.json`
97+
- symptom: `if: ... 'string' # comment` does not highlight the comment as a YAML comment
98+
99+
Suggested test plan:
100+
101+
1. Add a fixture with lines like:
102+
103+
```yaml
104+
jobs:
105+
test:
106+
if: matrix.os != 'macos-latest' # Cache causes errors on macOS
107+
```
108+
109+
1. Add a focused test for the `if-expression` rule behavior in `expressions.tmGrammar.json`
110+
2. Verify the expression matcher does not swallow the trailing comment, while preserving `#` inside quoted strings
111+
112+
Note:
113+
114+
- This kind of issue may need a new small helper in `syntax-test-utils.ts` for line/capture-level grammar matching, in addition to the embedded-block helpers already present.
115+
116+
## Proposed pattern for community-submitted failing tests
117+
118+
For syntax-highlighting bugs in this area, contributors can submit:
119+
120+
1. A fixture file in `src/workflow/syntax/fixtures/`
121+
2. A failing Jest assertion in `src/workflow/syntax/*.test.ts`
122+
3. A short comment linking the issue number and describing the expected scopes/behavior
123+
124+
That gives maintainers a reproducible regression case even before a fix is implemented.
Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
/* eslint-disable @typescript-eslint/no-non-null-assertion --
2+
* Test code intentionally dereferences values after explicit existence/null checks
3+
* to keep expectations concise and readable.
4+
*/
5+
import {describe, expect, it} from "@jest/globals";
6+
import {
7+
findGithubActionsExpressionInText,
8+
findGithubActionsInlineExpression,
9+
readFixture,
10+
readJson
11+
} from "./syntax-test-utils";
12+
13+
type ExpressionsGrammar = {
14+
repository: {
15+
"block-inline-expression": {
16+
begin: string;
17+
end: string;
18+
patterns: Array<{include: string}>;
19+
};
20+
expression: {
21+
patterns: Array<{include: string}>;
22+
};
23+
"if-expression": {
24+
match: string;
25+
};
26+
"block-if-expression": {
27+
begin: string;
28+
beginCaptures: {
29+
"6": {
30+
patterns: Array<{include: string}>;
31+
};
32+
};
33+
};
34+
"op-comparison": {
35+
match: string;
36+
};
37+
"op-logical": {
38+
match: string;
39+
};
40+
};
41+
};
42+
43+
function collectInlineExpressions(line: string): string[] {
44+
const result: string[] = [];
45+
let offset = 0;
46+
47+
while (offset < line.length) {
48+
const chunk = line.slice(offset);
49+
const extracted = findGithubActionsInlineExpression(chunk);
50+
if (!extracted) {
51+
break;
52+
}
53+
result.push(extracted);
54+
const localIndex = chunk.indexOf(extracted);
55+
offset += localIndex + extracted.length;
56+
}
57+
58+
return result;
59+
}
60+
61+
describe("workflow expression syntax highlighting", () => {
62+
// Regression test for https://github.com/github/vscode-github-actions/issues/531
63+
it("does not swallow trailing YAML comments on if: lines after quoted strings (#531)", () => {
64+
const grammar = readJson<ExpressionsGrammar>("language/syntaxes/expressions.tmGrammar.json");
65+
const fixture = readFixture("src/workflow/syntax/fixtures/if-comment-after-string.yml");
66+
const ifLine = fixture.split(/\r?\n/).find(line => line.includes("if: matrix.os != 'macos-latest' #"));
67+
68+
expect(ifLine).toBeDefined();
69+
70+
const ifExpression = new RegExp(grammar.repository["if-expression"].match);
71+
const match = ifExpression.exec(ifLine!);
72+
73+
expect(match).not.toBeNull();
74+
expect(match![1]).toBe("if:");
75+
76+
// Desired behavior: the expression capture should stop before the YAML comment.
77+
// Current bug (#531): capture 2 includes the trailing "# ...", preventing comment tokenization.
78+
expect(match![2]).toBe("matrix.os != 'macos-latest'");
79+
expect(match![3]).toBe(" # Cache causes errors on macOS");
80+
});
81+
82+
// Regression test for https://github.com/github/vscode-github-actions/issues/223
83+
it("does not terminate ${{ }} expression early when }} appears inside quoted strings", () => {
84+
const grammar = readJson<ExpressionsGrammar>("language/syntaxes/expressions.tmGrammar.json");
85+
const fixture = readFixture("src/workflow/syntax/fixtures/expression-nested-braces.yml");
86+
const matrixLine = fixture.split(/\r?\n/).find(line => line.includes("matrix: ${{"));
87+
88+
expect(matrixLine).toBeDefined();
89+
expect(grammar.repository["block-inline-expression"]).toBeDefined();
90+
expect(grammar.repository.expression.patterns).toContainEqual({include: "#string"});
91+
92+
const extracted = findGithubActionsInlineExpression(matrixLine!);
93+
expect(extracted).toBe("${{ fromJSON(format('{{\"linting\":[\"{0}\"]}}', 'ubuntu-latest')).linting }}");
94+
});
95+
96+
it("supports block if-expression syntax (if: | / if: >)", () => {
97+
const grammar = readJson<ExpressionsGrammar>("language/syntaxes/expressions.tmGrammar.json");
98+
const fixture = readFixture("src/workflow/syntax/fixtures/if-block-expression.yml");
99+
const ifLiteralLine = fixture.split(/\r?\n/).find(line => /^\s*if:\s*\|(?:\s+#.*)?$/.test(line));
100+
const ifFoldedLine = fixture.split(/\r?\n/).find(line => /^\s*if:\s*>(?:\s+#.*)?$/.test(line));
101+
102+
expect(ifLiteralLine).toBeDefined();
103+
expect(ifFoldedLine).toBeDefined();
104+
expect(grammar.repository["block-if-expression"]).toBeDefined();
105+
106+
const blockIfBegin = new RegExp(grammar.repository["block-if-expression"].begin);
107+
expect(blockIfBegin.test(ifLiteralLine!)).toBe(true);
108+
expect(blockIfBegin.test(ifFoldedLine!)).toBe(true);
109+
});
110+
111+
it("supports multi-line inline expressions", () => {
112+
const grammar = readJson<ExpressionsGrammar>("language/syntaxes/expressions.tmGrammar.json");
113+
const fixture = readFixture("src/workflow/syntax/fixtures/expression-multiline.yml");
114+
115+
expect(grammar.repository["block-inline-expression"]).toBeDefined();
116+
expect(grammar.repository["block-inline-expression"].begin).toContain("\\$\\{\\{");
117+
expect(grammar.repository["block-inline-expression"].end).toContain("\\}\\}");
118+
119+
const extracted = findGithubActionsExpressionInText(fixture);
120+
expect(extracted).not.toBeNull();
121+
expect(extracted).toContain("${{ format(");
122+
expect(extracted).toContain("github.ref");
123+
expect(extracted).toContain("github.sha");
124+
expect(extracted).toContain(") }}");
125+
});
126+
127+
it("supports logical and comparison operators in expression patterns", () => {
128+
const grammar = readJson<ExpressionsGrammar>("language/syntaxes/expressions.tmGrammar.json");
129+
130+
expect(grammar.repository["op-comparison"]).toBeDefined();
131+
expect(grammar.repository["op-logical"]).toBeDefined();
132+
133+
const comparison = new RegExp(grammar.repository["op-comparison"].match, "g");
134+
const logical = new RegExp(grammar.repository["op-logical"].match, "g");
135+
const sample = "github.ref == 'refs/heads/main' && github.event_name != 'pull_request' || false";
136+
137+
expect(sample.match(comparison)).toEqual(["==", "!="]);
138+
expect(sample.match(logical)).toEqual(["&&", "||"]);
139+
});
140+
141+
it("keeps # inside quoted strings and still separates trailing comments on if: lines", () => {
142+
const grammar = readJson<ExpressionsGrammar>("language/syntaxes/expressions.tmGrammar.json");
143+
const fixture = readFixture("src/workflow/syntax/fixtures/if-inline-edge-cases.yml");
144+
const lines = fixture.split(/\r?\n/);
145+
const hashLine = lines.find(line => line.includes("if: contains(github.ref, '#main') #"));
146+
const escapedLine = lines.find(line => line.includes("if: github.ref == 'it''s-main' #"));
147+
148+
expect(hashLine).toBeDefined();
149+
expect(escapedLine).toBeDefined();
150+
151+
const ifExpression = new RegExp(grammar.repository["if-expression"].match);
152+
const hashMatch = ifExpression.exec(hashLine!);
153+
const escapedMatch = ifExpression.exec(escapedLine!);
154+
155+
expect(hashMatch).not.toBeNull();
156+
expect(escapedMatch).not.toBeNull();
157+
expect(hashMatch![2]).toBe("contains(github.ref, '#main')");
158+
expect(escapedMatch![2]).toBe("github.ref == 'it''s-main'");
159+
expect(hashMatch![3]).toBe(" # trailing comment");
160+
expect(escapedMatch![3]).toBe(" # escaped quote case");
161+
});
162+
163+
it("handles multiple inline expressions on one line", () => {
164+
const fixture = readFixture("src/workflow/syntax/fixtures/inline-multiple-expressions.yml");
165+
const combinedLine = fixture.split(/\r?\n/).find(line => line.includes("COMBINED: "));
166+
167+
expect(combinedLine).toBeDefined();
168+
const extracted = collectInlineExpressions(combinedLine!);
169+
170+
expect(extracted).toEqual(["${{ github.ref }}", "${{ github.sha }}"]);
171+
});
172+
});
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
name: test
2+
3+
# Regression-style fixture for multi-line inline expressions
4+
on: workflow_dispatch
5+
6+
jobs:
7+
demo:
8+
runs-on: ubuntu-latest
9+
env:
10+
CACHE_KEY: ${{ format(
11+
'{0}-{1}',
12+
github.ref,
13+
github.sha
14+
) }}
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
name: test
2+
3+
# Regression fixture for https://github.com/github/vscode-github-actions/issues/223
4+
on: workflow_dispatch
5+
6+
jobs:
7+
demo:
8+
runs-on: ubuntu-latest
9+
strategy:
10+
matrix: ${{ fromJSON(format('{{"linting":["{0}"]}}', 'ubuntu-latest')).linting }}
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
name: test
2+
3+
# Regression-style fixture for block if-expression syntax
4+
on: workflow_dispatch
5+
6+
jobs:
7+
literal:
8+
runs-on: ubuntu-latest
9+
if: | # literal condition
10+
github.ref == 'refs/heads/main' &&
11+
github.event_name != 'pull_request'
12+
steps:
13+
- run: echo "ok"
14+
folded:
15+
runs-on: ubuntu-latest
16+
if: > # folded condition
17+
github.ref == 'refs/heads/main' &&
18+
github.event_name != 'pull_request'
19+
steps:
20+
- run: echo "ok"

0 commit comments

Comments
 (0)