Skip to content

Commit f7a7cac

Browse files
GiggleLiuclaude
andcommitted
refactor(skills): separate Expected Outcome from Example Instance and add alias check
Split the combined "example with known optimal solution" into distinct Example Instance (problem data only) and Expected Outcome (solution/value) sections across issue templates, check-issue, add-model, propose, and final-review skills. Differentiates satisfaction (valid config + justification) from optimization (optimal config + objective value) throughout. Also adds an alias sanity check to final-review's model completeness checklist. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent f81e52f commit f7a7cac

6 files changed

Lines changed: 36 additions & 14 deletions

File tree

.claude/skills/add-model/SKILL.md

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,9 +26,15 @@ Before any implementation, collect all required information. If called from `iss
2626
| 9 | **Best known exact algorithm** | Complexity with variable definitions | "O(1.1996^n) by Xiao & Nagamochi (2017), where n = \|V\|" |
2727
| 10 | **Solving strategy** | How it can be solved | "BruteForce works; ILP reduction available" |
2828
| 11 | **Category** | Which sub-module under `src/models/` | `graph`, `formula`, `set`, `algebraic`, `misc` |
29+
| 12 | **Expected outcome from the issue** | Concrete outcome for the issue's example instance | Optimization: one optimal solution + optimal value. Satisfaction: one valid/satisfying solution + why it is valid |
2930

3031
If any item is missing, ask the user to provide it. Do NOT proceed until the checklist is complete.
3132

33+
The issue's **Expected Outcome** section is the source of truth for the implementation-facing example.
34+
- For optimization problems, use the issue's optimal solution and optimal objective value.
35+
- For satisfaction problems, use the issue's valid / satisfying solution and its justification.
36+
- Do not invent or replace the expected outcome during implementation unless the issue is corrected first.
37+
3238
### Associated Rule Check
3339

3440
Before implementation, verify that at least one reduction rule exists or is planned for this problem — otherwise it will be an orphan node in the reduction graph.
@@ -185,14 +191,14 @@ Required tests:
185191
- `test_<name>_direction` -- verify optimization direction (if optimization problem)
186192
- `test_<name>_serialization` -- round-trip serde test (optional but recommended)
187193
- `test_<name>_solver` -- verify brute-force solver finds correct solutions
188-
- `test_<name>_paper_example` -- **use the same instance from the paper example** (Step 6), verify the claimed solution is valid/optimal and the solution count matches
194+
- `test_<name>_paper_example` -- **use the same instance from the paper example** (Step 6), verify the issue's expected outcome is valid/optimal and the solution count matches
189195

190196
The `test_<name>_paper_example` test is critical for consistency between code and paper. It must:
191197
1. Construct the exact same instance shown in the paper's example figure
192-
2. Evaluate the solution shown in the paper and assert it is valid (and optimal for optimization problems)
198+
2. Evaluate the solution from the issue's **Expected Outcome** section as shown in the paper and assert it is valid (and optimal for optimization problems)
193199
3. Use `BruteForce` to find all optimal/satisfying solutions and assert the count matches the paper's claim
194200

195-
This test should be written **after** Step 6 (paper entry), once the example instance and solution are finalized. If writing tests before the paper, use the same instance you plan to use in the paper and come back to verify consistency.
201+
This test should be written **after** Step 6 (paper entry), once the example instance and expected outcome are finalized. If writing tests before the paper, use the issue's Example Instance + Expected Outcome as the source of truth and come back to verify consistency.
196202

197203
Link the test file via `#[cfg(test)] #[path = "..."] mod tests;` at the bottom of the model file.
198204

.claude/skills/check-issue/SKILL.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -306,7 +306,8 @@ Check all template sections are present and substantive:
306306
| Schema | Type name, variants, field table |
307307
| Complexity | Best known algorithm with citation **and** a concrete complexity expression in terms of problem parameters (e.g., `q^n`, `2^{0.8765n}`) |
308308
| How to solve | At least one solver method checked |
309-
| Example Instance | Concrete instance with known solution |
309+
| Example Instance | Concrete instance that exercises the core structure |
310+
| Expected Outcome | Satisfaction: one valid / satisfying solution with brief justification. Optimization: one optimal solution with the optimal objective value |
310311

311312
Missing or placeholder sections → list them as **Fail** items.
312313

@@ -328,7 +329,9 @@ The formal definition must be **precise and implementable**:
328329

329330
- **Non-trivial**: Enough vertices/variables to exercise constraints meaningfully (not just a triangle)
330331
- **Exercises core structure**: Examples must use the defining features of the problem. For instance, a "MultivariateQuadratic" example that only has linear terms does not exercise the quadratic structure → **Fail**. If the problem's name or definition highlights a specific structural feature (quadratic, k-colorable, bipartite, etc.), at least one example must exercise that feature.
331-
- **Known optimal solution provided**: Must state the optimal value, not just the instance
332+
- **Expected outcome provided**:
333+
- Satisfaction problems must include a concrete valid / satisfying solution and say why it is valid
334+
- Optimization problems must include a concrete optimal solution and the optimal objective value
332335
- **Detailed enough for paper**: This example will appear in the paper — it needs to be illustrative
333336
- **Round-trip testable**: The example must be complex enough that a round-trip test (construct instance → solve → verify) can catch implementation bugs. A too-simple instance (e.g., 2 vertices, a single clause) may have a trivially correct solution that passes even with a wrong implementation. The example should have multiple feasible configurations with different objective values (for optimization) or a mix of satisfying and non-satisfying configurations (for satisfaction problems), so that correctness is meaningfully tested. Rule of thumb: the instance should have at least 2 suboptimal feasible solutions in addition to the optimal one.
334337

.claude/skills/final-review/SKILL.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,7 @@ Verify the PR includes all required components. Check:
184184
- [ ] Paper section in `docs/paper/reductions.typ` (`problem-def` entry)
185185
- [ ] `display-name` entry in paper
186186
- [ ] `trait_consistency.rs` entry in `src/unit_tests/trait_consistency.rs` (`test_all_problems_implement_trait_correctly`, plus `test_direction` for optimization)
187+
- [ ] Aliases: if provided, verify they are standard literature abbreviations (not made up); if empty, confirm no well-known abbreviation is missing; check no conflict with existing aliases
187188

188189
**For [Rule] PRs:**
189190
- [ ] Reduction implementation (`src/rules/...`)

.claude/skills/propose/SKILL.md

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -312,7 +312,9 @@ Work through these topics in order, using `AskUserQuestion` where multiple-choic
312312

313313
If the user picks "Generate new batch", create 3 new examples with different sizes/structures and re-present.
314314

315-
After the user picks a concrete example, provide a complete instance with its known optimal solution.
315+
After the user picks a concrete example, provide a complete instance with its expected outcome.
316+
- For optimization problems: give at least one optimal solution and the optimal objective value
317+
- For satisfaction problems: give at least one valid / satisfying solution and explain briefly why it is valid
316318
- Must exercise the problem's core structure
317319
- Must be small enough to verify by hand
318320

@@ -488,7 +490,7 @@ If the reduction is well-known, use the literature to **pre-fill** answers in St
488490
- Must define all symbols before using them
489491
- Must be detailed enough that someone could implement it
490492

491-
3. **Explanation** — Present a correctness argument explaining why the reduction preserves optimal solutions, then ask for feedback via `AskUserQuestion`:
493+
3. **Explanation** — Present a correctness argument explaining why the reduction preserves feasibility (for satisfaction problems) or optimality (for optimization problems), then ask for feedback via `AskUserQuestion`:
492494
```
493495
AskUserQuestion:
494496
question: "How does this explanation look?"
@@ -525,7 +527,8 @@ If the reduction is well-known, use the literature to **pre-fill** answers in St
525527

526528
If the user picks "Generate new batch", create 3 new examples with different sizes/structures and re-present.
527529

528-
After the user picks a concrete example, fully work out the example: show source instance, each construction step, resulting target instance, and the optimal solution.
530+
After the user picks a concrete example, fully work out the example: show source instance, each construction step, and the resulting target instance.
531+
- Do not ask the user to provide solved witnesses manually
529532
- Must be non-trivial but hand-verifiable
530533
- Must exercise the core structure of the reduction
531534

@@ -637,14 +640,17 @@ If proposing a model + rules, present all drafts together:
637640
- Reduction Rule Crossref (linking to companion rule issues or noting planned rules)
638641
- How to solve (brute-force, ILP, or other — if ILP/QUBO, must cross-reference rule issue)
639642
- Example Instance
643+
- Expected Outcome
644+
- Optimization problems: optimal solution + optimal objective value
645+
- Satisfaction problems: valid / satisfying solution + brief justification
640646
- BibTeX (include the BibTeX entry for the complexity/definition reference at the end of the issue)
641647

642648
**For rules**, the draft must include:
643649
- Source, Target, Motivation, Reference (with BibTeX)
644650
- Reduction Algorithm (numbered steps, all symbols defined)
645651
- Size Overhead (table with target metrics and formulas)
646652
- Validation Method
647-
- Example (fully worked)
653+
- Example (fully worked: source instance, construction, target instance)
648654
- BibTeX (include the BibTeX entry for the reference at the end of the issue)
649655

650656
---
@@ -665,7 +671,7 @@ Apply all 4 checks from `/check-issue` against the draft content:
665671
1. **Usefulness:** `pred show <name>` must fail (problem doesn't exist). At least one reduction planned.
666672
2. **Non-trivial:** Not isomorphic to existing problem.
667673
3. **Correctness:** Complexity expression verified against literature.
668-
4. **Well-written:** All template sections present, symbols consistent, example exercises core structure.
674+
4. **Well-written:** All template sections present, symbols consistent, example exercises core structure, and Expected Outcome matches the problem type (valid solution for satisfaction, optimal solution/value for optimization).
669675

670676
**If any check fails:** Fix the draft automatically if possible. If user input is needed, ask. Loop back to Step 4 with the corrected draft.
671677

.github/ISSUE_TEMPLATE/problem.md

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -83,13 +83,20 @@ Solver is required for reduction rule verification purpose.
8383
## Example Instance
8484

8585
<!--
86-
A small but non-trivial instance with known optimal solution, for testing and the paper.
86+
A small but non-trivial instance for testing and the paper.
8787
Should be large enough to exercise the problem's constraints meaningfully (avoid trivial cases like triangle graphs).
88-
E.g. "Petersen graph: |V|=10, |E|=15, 3-regular. Optimal IS size = 4, and more details.."
88+
E.g. "Petersen graph: |V|=10, |E|=15, 3-regular."
8989
9090
This example will be shown in our paper, where you could find some references.
9191
-->
9292

93+
## Expected Outcome
94+
95+
<!--
96+
Optimization: provide one optimal configuration and its objective value.
97+
Satisfaction: provide one valid / satisfying configuration and a brief justification.
98+
-->
99+
93100
## BibTeX
94101

95102
<!-- Machine-readable citation for the definition/complexity references. E.g.

.github/ISSUE_TEMPLATE/rule.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -62,12 +62,11 @@ Structure your example as follows:
6262
1. **Source instance:** Describe the input (e.g. graph, formula, sequence).
6363
2. **Construction:** Show how each step of the reduction algorithm transforms it.
6464
3. **Target instance:** Show the resulting target problem data (e.g. QUBO matrix, ILP constraints).
65-
4. **Optimal solution:** Solve the target, extract back to source, verify optimality.
6665
6766
Must be small enough for brute-force solving, but large enough to exercise the reduction meaningfully.
6867
Please provide as many details as possible, because
6968
1. this example will appear in the paper.
70-
2. AI needs this information to generate example code, run it, and try to compare with what you provided.
69+
2. AI needs this information to generate example code and derive round-trip tests from it. You do **not** need to provide a solved witness manually.
7170
7271
Please check existing examples in our paper for references.
7372
-->

0 commit comments

Comments
 (0)