Skip to content

Commit 8f2d44f

Browse files
committed
update workflows
1 parent 11c9a2c commit 8f2d44f

2 files changed

Lines changed: 186 additions & 8 deletions

File tree

.github/aw/actions-lock.json

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,10 +35,10 @@
3535
"version": "v7.0.0",
3636
"sha": "bbbca2ddaa5d8feaa63e36b76fdaad77386f024f"
3737
},
38-
"github/gh-aw-actions/setup@v0.68.1": {
38+
"github/gh-aw-actions/setup@v0.68.3": {
3939
"repo": "github/gh-aw-actions/setup",
40-
"version": "v0.68.1",
41-
"sha": "2fe53acc038ba01c3bbdc767d4b25df31ca5bdfc"
40+
"version": "v0.68.3",
41+
"sha": "ba90f2186d7ad780ec640f364005fa24e797b360"
4242
},
4343
"github/gh-aw/actions/setup@v0.66.1": {
4444
"repo": "github/gh-aw/actions/setup",

workflows/lean-squad.md

Lines changed: 183 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ description: |
1515
8. Aeneas Extraction (optional, Rust only) — use Charon+Aeneas to auto-generate Lean from Rust
1616
9. CI Automation — set up and maintain CI workflows that verify proofs on every PR
1717
10. Project Report — create and incrementally maintain REPORT.md with mermaid diagrams
18+
11. Conference Paper — write and maintain an IEEE conference submission (11-page limit) on the formal verification work
1819
1920
Phases are sequentially weighted: Task 1 dominates until research is done,
2021
then Task 2 rises, and so on up to proofs. Each run builds on prior runs
@@ -101,10 +102,11 @@ steps:
101102
[ -f ".github/workflows/lean-ci.yml" ] && echo 1 > /tmp/gh-aw/has_lean_ci.txt || echo 0 > /tmp/gh-aw/has_lean_ci.txt
102103
[ -f ".github/workflows/aeneas-generate.yml" ] && echo 1 > /tmp/gh-aw/has_aeneas_ci.txt || echo 0 > /tmp/gh-aw/has_aeneas_ci.txt
103104
104-
# Detect CORRESPONDENCE.md, CRITIQUE.md, and REPORT.md
105+
# Detect CORRESPONDENCE.md, CRITIQUE.md, REPORT.md, and PAPER.md
105106
[ -f "formal-verification/CORRESPONDENCE.md" ] && echo 1 > /tmp/gh-aw/has_correspondence.txt || echo 0 > /tmp/gh-aw/has_correspondence.txt
106107
[ -f "formal-verification/CRITIQUE.md" ] && echo 1 > /tmp/gh-aw/has_critique.txt || echo 0 > /tmp/gh-aw/has_critique.txt
107108
[ -f "formal-verification/REPORT.md" ] && echo 1 > /tmp/gh-aw/has_report.txt || echo 0 > /tmp/gh-aw/has_report.txt
109+
[ -f "formal-verification/PAPER.md" ] && echo 1 > /tmp/gh-aw/has_paper.txt || echo 0 > /tmp/gh-aw/has_paper.txt
108110
109111
# Detect formal-verification directory
110112
[ -d "formal-verification" ] && echo 1 > /tmp/gh-aw/fv_dir.txt || echo 0 > /tmp/gh-aw/fv_dir.txt
@@ -135,6 +137,7 @@ steps:
135137
has_correspondence = int(open('/tmp/gh-aw/has_correspondence.txt').read().strip() or 0)
136138
has_critique = int(open('/tmp/gh-aw/has_critique.txt').read().strip() or 0)
137139
has_report = int(open('/tmp/gh-aw/has_report.txt').read().strip() or 0)
140+
has_paper = int(open('/tmp/gh-aw/has_paper.txt').read().strip() or 0)
138141
fv_dir = int(open('/tmp/gh-aw/fv_dir.txt').read().strip() or 0)
139142
fv_docs = int(open('/tmp/gh-aw/fv_docs.txt').read().strip() or 0)
140143
fv_issues = json.load(open('/tmp/gh-aw/fv_issues.json'))
@@ -154,6 +157,7 @@ steps:
154157
8: 'Aeneas Extraction (Rust only)',
155158
9: 'CI Automation',
156159
10: 'Project Report',
160+
11: 'Conference Paper',
157161
}
158162
159163
# Phase progress heuristics derived from repo state
@@ -177,6 +181,7 @@ steps:
177181
8: (3.0 if has_lean_specs else 1.0) if (has_rust and has_research) else 0.0, # aeneas: only for Rust codebases with research done
178182
9: 12.0 if (has_lean_specs and not has_ci) else 2.0, # CI: critical when lean files exist but no CI; regular check otherwise
179183
10: (8.0 if not has_report else 3.0) if has_proofs else (2.0 if has_lean_specs else 0.0), # report: important when proofs exist but no report; available once lean specs exist
184+
11: (8.0 if not has_paper else 3.0) if has_proofs else 0.0, # paper: important when proofs exist but no paper; only available once proofs exist
180185
}
181186
182187
run_id = int(os.environ.get('GITHUB_RUN_ID', '0'))
@@ -204,7 +209,7 @@ steps:
204209
f'lean_specs={has_lean_specs}, impl={has_impl}, proofs={has_proofs}, '
205210
f'rust={has_rust}, ci={has_ci}, '
206211
f'correspondence={bool(has_correspondence)}, critique={bool(has_critique)}, '
207-
f'report={bool(has_report)}')
212+
f'report={bool(has_report)}, paper={bool(has_paper)}')
208213
print()
209214
print('Task weights:')
210215
for t, w in weights.items():
@@ -228,6 +233,7 @@ steps:
228233
'has_correspondence': bool(has_correspondence),
229234
'has_critique': bool(has_critique),
230235
'has_report': bool(has_report),
236+
'has_paper': bool(has_paper),
231237
},
232238
'task_names': task_names,
233239
'weights': {str(k): round(v, 2) for k, v in weights.items()},
@@ -581,7 +587,14 @@ This is a reflective task. The goal is not to prove more things, but to evaluate
581587
- **Strength**: is the property tight (captures exactly the right behaviour) or weak (too easy to satisfy, even by incorrect implementations)?
582588
3. For unproved / `sorry`-guarded theorems, assess whether they are worth proving or should be revised.
583589
4. Identify the **highest-value gaps**: which properties, if proved, would give the most confidence in the codebase? Are there important invariants or safety properties that have not yet been attempted?
584-
5. Write or update `formal-verification/CRITIQUE.md`:
590+
5. **(Optional) Review the conference paper**: if `formal-verification/PAPER.md` exists, read it and assess it as a critical reviewer would:
591+
- **Accuracy**: are all claims in the paper supported by the actual Lean proofs and FV artifacts? Are results overstated or understated?
592+
- **Completeness**: does the paper cover all significant findings, including negative results and limitations?
593+
- **Intellectual honesty**: are modelling approximations and their impact on proof validity clearly disclosed?
594+
- **Clarity**: is the methodology well-explained? Would a reader unfamiliar with the codebase understand the contribution?
595+
- **Missing content**: are there important results, findings, or limitations not yet reflected in the paper?
596+
Include a `## Paper Review` section in CRITIQUE.md with specific, actionable feedback for improving the paper. Note any claims that need revision based on the current state of the proofs.
597+
6. Write or update `formal-verification/CRITIQUE.md`:
585598
- **Always** include a `## Last Updated` section at the top with the current UTC date/time and the HEAD commit SHA:
586599
```
587600
## Last Updated
@@ -593,8 +606,9 @@ This is a reflective task. The goal is not to prove more things, but to evaluate
593606
- **Gaps and recommendations**: what should be proved next and why — prioritised by impact.
594607
- **Concerns**: any theorems that look proved but may be vacuous due to model approximations (cross-reference CORRESPONDENCE.md).
595608
- **Positive findings**: highlight any case where FV revealed or confirmed something non-obvious.
596-
6. Create a PR with the updated CRITIQUE.md.
597-
7. Update memory: record the critique findings, flag high-priority gaps for future runs.
609+
- **Paper review** (if PAPER.md was reviewed in step 5): specific, actionable feedback on the conference paper — claims to revise, missing content, clarity issues.
610+
7. Create a PR with the updated CRITIQUE.md.
611+
8. Update memory: record the critique findings, flag high-priority gaps for future runs.
598612
599613
---
600614
@@ -1076,6 +1090,170 @@ timeline
10761090

10771091
---
10781092

1093+
### Task 11: Conference Paper
1094+
1095+
**Goal**: Write and maintain `formal-verification/PAPER.md` — an IEEE conference submission (11-page limit) on formal verification of the codebase, suitable for a conference on formal verification or software engineering. The paper covers the methodology, findings, proof architecture, modelling choices, and lessons learned.
1096+
1097+
This task produces a living document. Each run updates the paper to reflect the current state of the project rather than rewriting it from scratch.
1098+
1099+
> **Applicability gate**: This task only applies when meaningful proof work exists (proofs attempted or completed). If no Lean proofs exist yet, skip this task and substitute the most logically prior incomplete task.
1100+
1101+
1. Read all existing FV artifacts: Lean files, informal specs, CORRESPONDENCE.md, CRITIQUE.md, REPORT.md, TARGETS.md, RESEARCH.md, memory, open issues, and merged PRs. Read the existing PAPER.md if it exists.
1102+
2. **Create or update** `formal-verification/PAPER.md` following standard IEEE conference paper structure. The document must fit within an **11-page limit** when rendered (use judgment to keep content appropriately scoped and concise).
1103+
1104+
#### Paper Structure
1105+
1106+
```markdown
1107+
# <Title>: Formal Verification of <Repository/Component> with Lean 4
1108+
1109+
> 🔬 *Lean Squad — automated formal verification for `<owner>/<repo>`.*
1110+
1111+
## Abstract
1112+
1113+
{150–250 words. Summarise the contribution: what was verified, the approach taken
1114+
(Lean 4, Mathlib, the modelling strategy), key results (theorems proved, bugs found,
1115+
coverage achieved), and conclusions. State the headline numbers: N theorems, M files,
1116+
S sorry remaining.}
1117+
1118+
## 1. Introduction
1119+
1120+
{Motivate the work: why formal verification of this codebase matters, what properties
1121+
are safety-critical or high-value, and what prior assurance existed (tests, code review).
1122+
State the research questions or goals. Outline the paper structure.}
1123+
1124+
## 2. Background
1125+
1126+
### 2.1 The Target Codebase
1127+
1128+
{Describe the repository: purpose, language(s), architecture, size, key modules.
1129+
Focus on the aspects relevant to formal verification.}
1130+
1131+
### 2.2 Lean 4 and Mathlib
1132+
1133+
{Brief introduction to Lean 4 as a proof assistant and programming language.
1134+
Describe relevant Mathlib libraries used. Cite appropriately.}
1135+
1136+
### 2.3 Related Work
1137+
1138+
{Survey related formal verification efforts in the same domain or using similar
1139+
tools. Compare approaches: what has been verified formally in similar systems?
1140+
What tools were used? How does this work differ?}
1141+
1142+
## 3. Methodology
1143+
1144+
### 3.1 Target Selection
1145+
1146+
{Describe how FV-amenable targets were identified: criteria used, prioritisation
1147+
approach, how the codebase was surveyed. Reference RESEARCH.md and TARGETS.md.}
1148+
1149+
### 3.2 Specification Strategy
1150+
1151+
{Describe the two-phase specification approach: informal specs extracted from code
1152+
and documentation, then formalised into Lean 4. Discuss how ambiguities were
1153+
resolved and what design choices were made.}
1154+
1155+
### 3.3 Modelling Choices
1156+
1157+
{Describe how implementation code was translated into Lean 4 functional models.
1158+
Be explicit about what is modelled faithfully, what is abstracted, and what is
1159+
omitted (I/O, error handling, memory layout, concurrency, etc.). This is the
1160+
most important section for intellectual honesty — the value of proofs depends
1161+
on model fidelity. Cross-reference CORRESPONDENCE.md.}
1162+
1163+
### 3.4 Proof Approach
1164+
1165+
{Describe the proof strategies used: decidable propositions, tactic-based proofs,
1166+
automation (simp, omega, decide, aesop), structural induction, etc. Discuss what
1167+
worked well and what required significant manual effort.}
1168+
1169+
### 3.5 Aeneas Extraction (if applicable)
1170+
1171+
{If Aeneas was used, describe the extraction pipeline: Charon → LLBC → Aeneas → Lean.
1172+
Discuss what extracted successfully, what failed, and any bugs encountered in the
1173+
toolchain. Describe bridging theorems between generated and hand-written models.}
1174+
1175+
## 4. Results
1176+
1177+
### 4.1 Proof Inventory
1178+
1179+
{Enumerate what was proved: list key theorems with brief descriptions of what each
1180+
establishes. Include a summary table:}
1181+
1182+
| Theorem | File | Property | Status | Tactics |
1183+
|---------|------|----------|--------|---------|
1184+
| `name` | `File.lean` | Description | ✅ Proved / 🔄 sorry | Key tactics used |
1185+
1186+
### 4.2 Bugs and Findings
1187+
1188+
{Detail any bugs discovered through formal verification — these are the most
1189+
compelling results. For each: the property expected to hold, the counterexample
1190+
or proof failure, root cause, severity, and resolution. If no bugs were found,
1191+
discuss what this means (implementation is correct w.r.t. verified properties,
1192+
or the model was too abstract to catch bugs).}
1193+
1194+
### 4.3 Coverage Assessment
1195+
1196+
{Assess what fraction of the codebase's correctness-critical behaviour is covered
1197+
by the current proofs. What remains unverified? What are the highest-value gaps?
1198+
Cross-reference CRITIQUE.md.}
1199+
1200+
## 5. Discussion
1201+
1202+
### 5.1 Proof Utility
1203+
1204+
{Critically assess: are the proved properties meaningful? Would real bugs cause
1205+
theorem failures? Or are the models too abstract to catch practical issues?
1206+
Discuss the tension between model fidelity and proof tractability.}
1207+
1208+
### 5.2 Automation and Effort
1209+
1210+
{Discuss the level of automation achieved. How much was automatic (decide, simp)
1211+
vs. manual proof engineering? What was the approximate effort per theorem?
1212+
Discuss implications for scaling formal verification.}
1213+
1214+
### 5.3 Limitations
1215+
1216+
{Be explicit about limitations: model approximations, unverified assumptions,
1217+
properties not yet attempted, toolchain limitations. This section strengthens
1218+
the paper by demonstrating intellectual honesty.}
1219+
1220+
### 5.4 Lessons Learned
1221+
1222+
{Practical lessons from the verification effort: what worked, what didn't,
1223+
what would you do differently, advice for others attempting similar work.}
1224+
1225+
## 6. Conclusion
1226+
1227+
{Summarise contributions, key findings, and future work directions.
1228+
Restate headline numbers and most important results.}
1229+
1230+
## References
1231+
1232+
{Include references to: Lean 4, Mathlib, Aeneas/Charon (if used), related
1233+
formal verification work, the target codebase documentation, any textbook
1234+
algorithms or protocols that were verified against.}
1235+
```
1236+
1237+
3. **Content quality requirements**:
1238+
- Write in formal academic style appropriate for an IEEE conference submission.
1239+
- All claims must be supported by evidence from the actual Lean proofs and FV artifacts.
1240+
- Be intellectually honest: clearly distinguish between what is proved and what is assumed, between exact models and approximations.
1241+
- Include concrete examples: show key theorem statements in Lean syntax, show counterexamples for bugs found, show representative proof fragments for interesting cases.
1242+
- The paper should be self-contained: a reader unfamiliar with the repository should understand the contribution.
1243+
- Respect the **11-page limit**: be concise. Prioritise depth on methodology, results, and discussion over exhaustive listing. Move detailed inventories to an appendix or reference REPORT.md.
1244+
4. **Incremental updates**: read the existing PAPER.md (if any), update sections that have changed (new theorems, new findings, revised assessments), and maintain consistency throughout. Do not delete prior content unless it has become incorrect.
1245+
5. **Always** include a `## Last Updated` section near the top with the current UTC date/time and the HEAD commit SHA:
1246+
```
1247+
## Last Updated
1248+
- **Date**: YYYY-MM-DD HH:MM UTC
1249+
- **Commit**: `<SHA>`
1250+
```
1251+
6. Cross-reference CORRESPONDENCE.md, CRITIQUE.md, and REPORT.md for modelling choices, proof utility assessments, and detailed inventories.
1252+
7. Create a PR with the updated PAPER.md.
1253+
8. Update memory: note that the paper exists and what state it covers.
1254+
1255+
---
1256+
10791257
### Task Final: Update Lean Squad Status Issue *(ALWAYS DO THIS EVERY RUN)*
10801258

10811259
Maintain a single open issue titled `[Lean Squad] Formal Verification Status` as a continuously-updated dashboard for maintainers.

0 commit comments

Comments
 (0)