Skip to content

Commit 846db71

Browse files
rengzhengcodesrengzhengcodes
committed
LLM tests
1 parent 68079b9 commit 846db71

38 files changed

Lines changed: 715 additions & 0 deletions

File tree

‎tests/llm/README.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
This directory contains LLM-authored test case manifests organized by subfield.
2+
3+
Subfield layout mirrors `tests/`:
4+
- `tests/llm/mapper`
5+
- `tests/llm/model`
6+
- `tests/llm/toll`
7+
- `tests/llm/plotting`
8+
- `tests/llm/mapping_viz`
9+
- `tests/llm/notebooks`
10+
- `tests/llm/isl/mapper/*`
11+
- `tests/llm/isl/distributed/*`
12+
- `tests/llm/vibe_see_readme_in_this_dir/*`
13+
14+
Each subfield directory has a `cases.yaml` manifest with LLM test cases.
15+
16+
`tests/llm/mapper/ground_truth.json` is generated from live runs in the `pyenv`
17+
`accelforge` environment and is used by
18+
`tests/llm/mapper/test_mapper_ground_truth.py` as golden output for:
19+
- access counts
20+
- energy
21+
- latency
22+
- area
23+
- per-component and per-einsum energy/latency checks

‎tests/llm/__init__.py‎

Whitespace-only changes.

‎tests/llm/case_index.yaml‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
subfields:
2+
- name: isl/distributed/binding
3+
cases_file: tests/llm/isl/distributed/binding/cases.yaml
4+
- name: isl/distributed/binding_application
5+
cases_file: tests/llm/isl/distributed/binding_application/cases.yaml
6+
- name: isl/distributed/multicast
7+
cases_file: tests/llm/isl/distributed/multicast/cases.yaml
8+
- name: isl/mapper/isl_functions
9+
cases_file: tests/llm/isl/mapper/isl_functions/cases.yaml
10+
- name: isl/mapper/mapping_to_isl
11+
cases_file: tests/llm/isl/mapper/mapping_to_isl/cases.yaml
12+
- name: isl/mapper/spatial_reuse_analysis
13+
cases_file: tests/llm/isl/mapper/spatial_reuse_analysis/cases.yaml
14+
- name: isl/mapper/temporal_reuse_analysis
15+
cases_file: tests/llm/isl/mapper/temporal_reuse_analysis/cases.yaml
16+
- name: mapping_viz
17+
cases_file: tests/llm/mapping_viz/cases.yaml
18+
- name: model
19+
cases_file: tests/llm/model/cases.yaml
20+
- name: notebooks
21+
cases_file: tests/llm/notebooks/cases.yaml
22+
- name: plotting
23+
cases_file: tests/llm/plotting/cases.yaml
24+
- name: toll
25+
cases_file: tests/llm/toll/cases.yaml
26+
- name: vibe_see_readme_in_this_dir/api_gaps
27+
cases_file: tests/llm/vibe_see_readme_in_this_dir/api_gaps/cases.yaml
28+
- name: vibe_see_readme_in_this_dir/arch_flattening
29+
cases_file: tests/llm/vibe_see_readme_in_this_dir/arch_flattening/cases.yaml
30+
- name: vibe_see_readme_in_this_dir/component_fields
31+
cases_file: tests/llm/vibe_see_readme_in_this_dir/component_fields/cases.yaml
32+
- name: vibe_see_readme_in_this_dir/config_variables_model
33+
cases_file: tests/llm/vibe_see_readme_in_this_dir/config_variables_model/cases.yaml
34+
- name: vibe_see_readme_in_this_dir/evaluation
35+
cases_file: tests/llm/vibe_see_readme_in_this_dir/evaluation/cases.yaml
36+
- name: vibe_see_readme_in_this_dir/mapping
37+
cases_file: tests/llm/vibe_see_readme_in_this_dir/mapping/cases.yaml
38+
- name: vibe_see_readme_in_this_dir/parsed_values
39+
cases_file: tests/llm/vibe_see_readme_in_this_dir/parsed_values/cases.yaml
40+
- name: vibe_see_readme_in_this_dir/renames
41+
cases_file: tests/llm/vibe_see_readme_in_this_dir/renames/cases.yaml
42+
- name: vibe_see_readme_in_this_dir/rendering
43+
cases_file: tests/llm/vibe_see_readme_in_this_dir/rendering/cases.yaml
44+
- name: vibe_see_readme_in_this_dir/spec
45+
cases_file: tests/llm/vibe_see_readme_in_this_dir/spec/cases.yaml
46+
- name: vibe_see_readme_in_this_dir/workload_model
47+
cases_file: tests/llm/vibe_see_readme_in_this_dir/workload_model/cases.yaml
48+
- name: vibe_see_readme_in_this_dir/workload_parser
49+
cases_file: tests/llm/vibe_see_readme_in_this_dir/workload_parser/cases.yaml
50+
- name: vibe_see_readme_in_this_dir/workload_parsing
51+
cases_file: tests/llm/vibe_see_readme_in_this_dir/workload_parsing/cases.yaml
52+
- name: vibe_see_readme_in_this_dir/yaml_and_expressions
53+
cases_file: tests/llm/vibe_see_readme_in_this_dir/yaml_and_expressions/cases.yaml
54+
- name: mapper
55+
cases_file: tests/llm/mapper/cases.yaml
56+
ground_truth_file: tests/llm/mapper/ground_truth.json

‎tests/llm/isl/__init__.py‎

Whitespace-only changes.

‎tests/llm/isl/distributed/__init__.py‎

Whitespace-only changes.

‎tests/llm/isl/distributed/binding/cases.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
subfield: binding
2+
cases:
3+
- id: binding_smoke_parse
4+
prompt: "Load representative inputs for binding and ensure parsing or object construction succeeds."
5+
expected: "no_error"
6+
- id: binding_smoke_execute
7+
prompt: "Execute one representative code path for binding and ensure it returns a non-empty result."
8+
expected: "no_error"

‎tests/llm/isl/distributed/binding_application/cases.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
subfield: binding_application
2+
cases:
3+
- id: binding_application_smoke_parse
4+
prompt: "Load representative inputs for binding_application and ensure parsing or object construction succeeds."
5+
expected: "no_error"
6+
- id: binding_application_smoke_execute
7+
prompt: "Execute one representative code path for binding_application and ensure it returns a non-empty result."
8+
expected: "no_error"

‎tests/llm/isl/distributed/multicast/cases.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
subfield: multicast
2+
cases:
3+
- id: multicast_smoke_parse
4+
prompt: "Load representative inputs for multicast and ensure parsing or object construction succeeds."
5+
expected: "no_error"
6+
- id: multicast_smoke_execute
7+
prompt: "Execute one representative code path for multicast and ensure it returns a non-empty result."
8+
expected: "no_error"

‎tests/llm/isl/mapper/__init__.py‎

Whitespace-only changes.

‎tests/llm/isl/mapper/isl_functions/cases.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
subfield: isl_functions
2+
cases:
3+
- id: isl_functions_smoke_parse
4+
prompt: "Load representative inputs for isl_functions and ensure parsing or object construction succeeds."
5+
expected: "no_error"
6+
- id: isl_functions_smoke_execute
7+
prompt: "Execute one representative code path for isl_functions and ensure it returns a non-empty result."
8+
expected: "no_error"

0 commit comments

Comments
 (0)