Skip to content

Commit f68ca16

Browse files
johnsamuelwritesjohnsamuelwrites
committed
Ensure Linguist compatability
1 parent d882e74 commit f68ca16

8 files changed

Lines changed: 127 additions & 211 deletions

File tree

‎.github/QUICK_REFERENCE.md‎

Lines changed: 39 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
# Quick Reference: Keyword Sync Workflow
1+
# Quick Reference: Keyword Sync Workflow
22

33
## Before You Commit
44

5-
`bash
5+
```bash
66
# 1. Install deps (one-time)
77
pip install pyyaml
88

@@ -18,11 +18,11 @@ python .github/scripts/check-sync-results.py sync-report.json
1818

1919
# 4. View details
2020
cat sync-report.json | jq .
21-
`
21+
```
2222

2323
## Updating Keywords Interactively
2424

25-
`bash
25+
```bash
2626
# Dry-run: see what would change
2727
python .github/scripts/update-keywords-from-reference.py \
2828
--reference ../multilingual/multilingualprogramming/resources/usm/keywords.json \
@@ -35,78 +35,62 @@ python .github/scripts/update-keywords-from-reference.py \
3535
--target data/keywords.yaml \
3636
--interactive \
3737
--backup
38-
`
38+
```
3939

4040
## GitHub Workflow Feedback
4141

42-
When you push/PR with keyword changes:
42+
When you push or open a PR with keyword changes:
4343

44-
✓ Workflow runs automatically
45-
✓ Creates sync-report.json
46-
✓ Posts comment on PR with results
47-
✓ Artifacts available for download
44+
- The workflow runs automatically
45+
- It creates `sync-report.json`
46+
- It can post a PR comment with results
47+
- The report is uploaded as an artifact
4848

49-
**Example PR Comment:**
50-
`
49+
Example PR comment:
50+
51+
```text
5152
## Keyword Registry Sync Report
5253
53-
✅ All keywords are in sync with the reference implementation.
54+
All keywords are in sync with the reference implementation.
55+
56+
Coverage: 100%
57+
```
5458

55-
**Coverage**: 100%
56-
`
59+
Example failure comment:
5760

58-
**Or if issues:**
59-
`
61+
```text
6062
## Keyword Registry Sync Report
6163
62-
⚠️ Keyword registry sync issues detected.
64+
Keyword registry sync issues detected.
6365
6466
### Language Mismatches
65-
- if (ja): Expected 'もし', got '如果'
66-
- class (fr): Expected 'classe', got 'class'
67+
- if (ja): expected 'もし', got '如果'
68+
- class (fr): expected 'classe', got 'class'
6769
68-
**Coverage**: 95%
69-
`
70+
Coverage: 95%
71+
```
7072

7173
## Common Issues
7274

73-
**"Coverage below 95%"**
74-
→ Add missing constructs to keywords.yaml or lower threshold
75+
**Coverage below 95%**
76+
Add missing constructs to `keywords.yaml` or lower the threshold.
7577

76-
**"Language mismatches detected"**
77-
→ Update values in keywords.yaml to match reference
78+
**Language mismatches detected**
79+
Update values in `keywords.yaml` to match the reference implementation.
7880

79-
**"Missing language support"**
80-
→ Add language columns (sv, da, fi, etc.) to keywords.yaml
81+
**Missing language support**
82+
Add missing language columns to `keywords.yaml`.
8183

8284
**Unicode errors**
83-
→ Ensure keywords.yaml is UTF-8 encoded
85+
Ensure `keywords.yaml` is saved as UTF-8.
8486

8587
## File Locations
8688

87-
- Workflow: .github/workflows/sync-keywords.yml
88-
- Sync script: .github/scripts/sync-keywords.py
89-
- Check script: .github/scripts/check-sync-results.py
90-
- Update script: .github/scripts/update-keywords-from-reference.py
91-
- Keywords: data/keywords.yaml
92-
- Reference: ../multilingual/multilingualprogramming/resources/usm/keywords.json
93-
- Report: sync-report.json (generated)
94-
- Docs: .github/SYNC_WORKFLOW.md
95-
96-
## Configuration
97-
98-
In .github/workflows/sync-keywords.yml:
99-
100-
`yaml
101-
# Minimum coverage threshold
102-
check-sync-results.py sync-report.json --min-coverage 95
103-
104-
# Strict mode (fail on warnings)
105-
check-sync-results.py sync-report.json --strict
106-
`
107-
108-
## Resources
109-
110-
- 🔗 Multilingual Repo: https://github.com/multilingualprogramming/multilingual
111-
- 📖 Full Docs: .github/SYNC_WORKFLOW.md
112-
- 🐛 Issues: .github/scripts/ (check source code)
89+
- Workflow: `.github/workflows/sync-keywords.yml`
90+
- Sync script: `.github/scripts/sync-keywords.py`
91+
- Check script: `.github/scripts/check-sync-results.py`
92+
- Update script: `.github/scripts/update-keywords-from-reference.py`
93+
- Keywords: `data/keywords.yaml`
94+
- Reference: `../multilingual/multilingualprogramming/resources/usm/keywords.json`
95+
- Report: `sync-report.json`
96+
- Docs: `.github/SYNC_WORKFLOW.md`

‎.github/SYNC_WORKFLOW.md‎

Lines changed: 61 additions & 139 deletions
Original file line numberDiff line numberDiff line change
@@ -1,191 +1,113 @@
1-
# Keyword Registry Sync Workflow
1+
# Keyword Registry Sync Workflow
22

33
## Overview
44

5-
This GitHub workflow automatically validates that the keyword registry in ree-sitter-multilingual is kept in sync with the reference implementation in multilingual.
6-
7-
The workflow:
8-
1. Clones both repositories
9-
2. Compares the keyword definitions
10-
3. Reports any mismatches, missing constructs, or language gaps
11-
4. Posts detailed feedback on pull requests
12-
5. Generates a JSON report artifact
13-
14-
## How It Works
15-
16-
### Trigger Events
17-
18-
The workflow runs automatically on:
19-
- **Push** to main/dev branches when data/keywords.yaml changes
20-
- **Pull request** to main when data/keywords.yaml changes
21-
- Manual trigger via GitHub Actions interface
22-
23-
### Validation Checks
24-
25-
The workflow performs these checks:
26-
27-
1. **Missing Constructs** — Constructs defined in the reference but not in tree-sitter-multilingual
28-
2. **Extra Constructs** — Constructs in tree-sitter-multilingual not in the reference
29-
3. **Keyword Mismatches** — Values that differ between reference and target
30-
4. **Language Gaps** — Languages supported by reference but not in tree-sitter-multilingual
31-
5. **Coverage Calculation** — Overall percentage of keywords properly synced
32-
33-
## Reference Implementation Mapping
34-
35-
The workflow maps semantic IDs from the reference keywords.json to construct names:
36-
37-
| Semantic ID | Construct Name |
38-
|---|---|
39-
| COND_IF | if |
40-
| COND_ELSE | else |
41-
| COND_ELIF | elif |
42-
| LOOP_FOR | for |
43-
| LOOP_WHILE | while |
44-
| LOOP_BREAK | break |
45-
| LOOP_CONTINUE | continue |
46-
| FUNC_DEF | def |
47-
| CLASS_DEF | class |
48-
| RETURN | return |
49-
| LET_VAR | let |
50-
| IMPORT | import |
51-
| FROM | from |
52-
| AS | as |
53-
| PASS | pass |
54-
| TRUE | true |
55-
| FALSE | false |
56-
| NONE | none |
57-
| AND | and |
58-
| OR | or |
59-
| NOT | not |
60-
| IN | in |
61-
| IS | is |
62-
| PRINT | print |
63-
| RANGE | range |
64-
| LEN | len |
65-
| TYPE | type |
66-
67-
## PR Feedback
68-
69-
When you open a pull request with keyword changes, the workflow posts a comment like:
70-
71-
\\\
5+
This workflow validates that `data/keywords.yaml` stays in sync with the reference implementation in `multilingualprogramming/multilingual`.
6+
7+
It checks:
8+
9+
1. Missing constructs
10+
2. Extra constructs
11+
3. Keyword mismatches
12+
4. Missing language coverage
13+
5. Overall sync coverage
14+
15+
## Trigger Events
16+
17+
The workflow runs on:
18+
19+
- Pushes to `main` or `dev` when `data/keywords.yaml` changes
20+
- Pull requests to `main` when `data/keywords.yaml` changes
21+
- Manual execution from GitHub Actions
22+
23+
## What It Compares
24+
25+
The workflow compares this repository's `data/keywords.yaml` against the reference file:
26+
27+
- `multilingual/multilingualprogramming/resources/usm/keywords.json`
28+
29+
It maps semantic IDs from the reference implementation to construct names used in this grammar.
30+
31+
## Example PR Feedback
32+
33+
Success:
34+
35+
```text
7236
## Keyword Registry Sync Report
7337
74-
✅ All keywords are in sync with the reference implementation.
38+
All keywords are in sync with the reference implementation.
7539
76-
**Coverage**: 100%
77-
\\\
40+
Coverage: 100%
41+
```
7842

79-
If there are issues:
43+
Failure:
8044

81-
\\\
45+
```text
8246
## Keyword Registry Sync Report
8347
84-
⚠️ Keyword registry sync issues detected.
48+
Keyword registry sync issues detected.
8549
8650
### Missing Constructs
8751
- break
8852
- continue
8953
9054
### Language Mismatches
91-
Found 3 mismatches:
92-
- if (ja): Expected 'もし', got '如果'
93-
- class (fr): Expected 'classe', got 'class'
94-
- ... and 1 more
55+
- if (ja): expected 'もし', got '如果'
56+
- class (fr): expected 'classe', got 'class'
9557
96-
**Coverage**: 95%
97-
\\\
58+
Coverage: 95%
59+
```
9860

9961
## Running Locally
10062

101-
To validate keywords locally before pushing:
102-
103-
\\\ash
104-
# Install dependencies
63+
```bash
10564
pip install pyyaml
10665

107-
# Clone reference implementation
10866
git clone https://github.com/multilingualprogramming/multilingual ../multilingual
10967

110-
# Run sync validation
11168
python .github/scripts/sync-keywords.py \
11269
--reference ../multilingual/multilingualprogramming/resources/usm/keywords.json \
11370
--target data/keywords.yaml \
11471
--report sync-report.json
11572

116-
# Check results
11773
python .github/scripts/check-sync-results.py sync-report.json
11874

119-
# View detailed report
12075
cat sync-report.json | jq .
121-
\\\
122-
123-
## Updating Keywords
124-
125-
When updating data/keywords.yaml:
126-
127-
1. **Check the reference** — Look at the reference implementation in multilingual/multilingualprogramming/resources/usm/keywords.json for the canonical keywords
128-
2. **Sync across languages** — Ensure all 17 supported languages have values
129-
3. **Use correct case** — Match the capitalization from the reference (e.g., True not rue)
130-
4. **Support variants** — The reference may list multiple accepted forms; consider supporting these
131-
5. **Test locally** — Run the sync script before pushing
76+
```
13277

133-
Example:
78+
## Updating Keywords Safely
13479

135-
\\\yaml
136-
# Reference has:
137-
# "COND_IF": {"en": "if", "fr": "si", "de": "wenn", ...}
80+
When editing `data/keywords.yaml`:
13881

139-
# So in keywords.yaml:
140-
if:
141-
en: if
142-
fr: si
143-
de: wenn
144-
# ... all 17 languages
145-
\\\
82+
1. Check the canonical values in the reference implementation.
83+
2. Keep language coverage aligned across supported constructs.
84+
3. Match capitalization exactly.
85+
4. Preserve UTF-8 encoding.
86+
5. Run the local sync tools before pushing.
14687

14788
## Troubleshooting
14889

149-
### Coverage below 95%
150-
Some constructs may be missing or incomplete. Check the workflow artifacts for details.
90+
### Coverage below threshold
15191

152-
**Fix:** Add missing languages or constructs to data/keywords.yaml based on the reference.
92+
Add missing constructs or missing language values to `data/keywords.yaml`.
15393

15494
### Language mismatches
155-
Keywords don't match the reference implementation.
15695

157-
**Fix:** Update the values in data/keywords.yaml to match the reference. Run the local sync script to verify.
96+
Update the values in `data/keywords.yaml` to match the reference implementation.
15897

15998
### Extra constructs
160-
You have constructs in tree-sitter-multilingual that aren't in the reference.
161-
162-
**Note:** This is typically OK — you may support additional constructs beyond the reference. Use --strict mode in CI if you want to prevent this.
163-
164-
## CI Configuration
165-
166-
### Minimum Coverage Threshold
167-
Set minimum coverage in the workflow (currently 95%):
168-
169-
\\\yaml
170-
check-sync-results.py sync-report.json --min-coverage 95
171-
\\\
172-
173-
### Strict Mode
174-
Enable strict mode to fail on any warnings:
17599

176-
\\\yaml
177-
check-sync-results.py sync-report.json --strict
178-
\\\
100+
Extra constructs may be acceptable if this grammar intentionally supports more than the reference implementation.
179101

180102
## File Locations
181103

182-
- **Workflow definition:** .github/workflows/sync-keywords.yml
183-
- **Sync script:** .github/scripts/sync-keywords.py
184-
- **Check script:** .github/scripts/check-sync-results.py
185-
- **Keyword registry:** data/keywords.yaml
186-
- **Reference implementation:** https://github.com/multilingualprogramming/multilingual
104+
- Workflow definition: `.github/workflows/sync-keywords.yml`
105+
- Sync script: `.github/scripts/sync-keywords.py`
106+
- Check script: `.github/scripts/check-sync-results.py`
107+
- Update script: `.github/scripts/update-keywords-from-reference.py`
108+
- Keyword registry: `data/keywords.yaml`
109+
- Reference implementation: `https://github.com/multilingualprogramming/multilingual`
187110

188111
## Related
189112

190113
- [Multilingual Repository](https://github.com/multilingualprogramming/multilingual)
191-
- [Universal Semantic Model (USM) Documentation](https://multilingual.readthedocs.io/)

0 commit comments

Comments
 (0)