Skip to content

Commit 086a1f8

Browse files
tenpercenttenpercent
authored
Add LLM-agnostic Docker and build analysis tools (#3576)
This commit introduces utility tools for building, testing, and analyzing Composable Kernel. The tools are designed to be LLM-agnostic and can be used with any AI assistant or directly from the command line. Tools Added: ============ 1. ck-docker - Docker container management - Start/stop ROCm-enabled containers - Build targets with CMake + Ninja - Run tests with gtest filters - Auto-detect GPU targets (gfx950, gfx942, etc.) - Per-user, per-branch container naming to avoid conflicts 2. ck-build-analysis - Build time profiling - Uses Clang's -ftime-trace for compilation analysis - Aggregates statistics across multiple trace files - Identifies template instantiation bottlenecks - Generates detailed Markdown reports with: * Compilation phase breakdown * Top expensive instantiations * Template family analysis * Data-driven optimization recommendations - Configurable granularity (1µs to 500µs) - PEP 723 compliant Python script with auto-dependency management via uv Key Features: ============= - LLM-agnostic design (works with any AI assistant) - Zero-configuration setup with automatic dependency installation - Comprehensive documentation in script/tools/README*.md - Security hardening (input validation, no command injection) - Multi-file trace aggregation for accurate build analysis - Jinja2-based report generation for customizable output Implementation: =============== - script/tools/ck-docker - Main Docker orchestration script - script/tools/ck-build-analysis - Build analysis orchestration - script/tools/common.sh - Shared utilities (container mgmt, GPU detection) - script/tools/analyze_build_trace.py - PEP 723 compliant Python analyzer - script/tools/templates/ - Jinja2 templates for report generation - script/tools/README*.md - Comprehensive documentation Directory Structure: ==================== script/tools/ ├── README.md # Main overview ├── README_ck-docker.md # ck-docker documentation ├── README_ck-build-analysis.md # ck-build-analysis documentation ├── ck-docker # Docker orchestration script ├── ck-build-analysis # Build analysis orchestration ├── common.sh # Shared utilities ├── analyze_build_trace.py # Python analyzer (PEP 723) └── templates/ └── build_analysis_report.md.jinja # Report template The tools follow Unix philosophy: do one thing well, compose easily, and work from both CLI and programmatic contexts.
1 parent f573956 commit 086a1f8

8 files changed

Lines changed: 1426 additions & 0 deletions

File tree

‎script/tools/README.md‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# Composable Kernel Tools
2+
3+
This directory contains utility tools for building, testing, and analyzing Composable Kernel.
4+
5+
These tools are designed to be LLM-agnostic and can be used with any AI assistant or directly from the command line.
6+
7+
## Available Tools
8+
9+
### ck-docker
10+
11+
Build and test composable_kernel in Docker with ROCm support.
12+
13+
See [README_ck-docker.md](README_ck-docker.md) for details.
14+
15+
**Quick start:**
16+
```bash
17+
# Add to PATH
18+
export PATH="$PATH:$PWD/script/tools"
19+
20+
# Start container and build
21+
ck-docker start
22+
ck-docker build test_amdgcn_mma
23+
ck-docker test test_amdgcn_mma
24+
```
25+
26+
### ck-build-analysis
27+
28+
Analyze Composable Kernel build times using Clang's -ftime-trace profiler.
29+
30+
See [README_ck-build-analysis.md](README_ck-build-analysis.md) for details.
31+
32+
**Quick start:**
33+
```bash
34+
# Add to PATH
35+
export PATH="$PATH:$PWD/script/tools"
36+
37+
# Analyze build time
38+
ck-build-analysis example_convnd_fwd_xdl_fp8
39+
```
40+
41+
## LLM Assistant Integration
42+
43+
These tools can be used as-is with any LLM assistant by providing the tool documentation to the assistant. The assistant can then invoke these tools on your behalf.
44+
45+
For example, you can ask:
46+
- "Start the docker container"
47+
- "Build and test test_amdgcn_mma"
48+
- "Analyze build time for example_convnd_fwd_xdl_fp8"
49+
50+
The assistant will translate your natural language request into the appropriate tool invocation.
51+
52+
## Dependencies
53+
54+
- **ck-docker**: Requires Docker and ROCm-capable GPU (for running tests)
55+
- **ck-build-analysis**: Requires Docker, automatically installs Python dependencies (jinja2) via `uv`
56+
57+
## Directory Structure
58+
59+
```
60+
script/tools/
61+
├── README.md # This file
62+
├── README_ck-docker.md # Documentation for ck-docker
63+
├── README_ck-build-analysis.md # Documentation for ck-build-analysis
64+
├── ck-docker # Docker container management tool
65+
├── ck-build-analysis # Build time analysis tool
66+
├── common.sh # Shared utilities for bash scripts
67+
├── analyze_build_trace.py # Python script for trace analysis (PEP 723 compliant)
68+
└── templates/
69+
└── build_analysis_report.md.jinja # Jinja2 template for analysis reports
70+
```
71+
72+
## Contributing
73+
74+
When adding new tools to this directory:
75+
1. Keep them LLM-agnostic (avoid hardcoding references to specific AI assistants)
76+
2. Provide clear command-line usage documentation
77+
3. Include examples for both CLI and LLM assistant usage
78+
4. Follow the existing naming convention and structure

‎script/tools/README_ck-build-analysis.md‎

Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
# ck-build-analysis
2+
3+
Analyze Composable Kernel build times using Clang's -ftime-trace profiler.
4+
5+
## Terminal Usage
6+
7+
Direct command-line usage:
8+
9+
```bash
10+
# From composable_kernel directory
11+
script/tools/ck-build-analysis example_convnd_fwd_xdl_fp8
12+
script/tools/ck-build-analysis example_convnd_fwd_xdl_fp8 --granularity=1
13+
script/tools/ck-build-analysis example_convnd_fwd_xdl_fp8 --granularity=1 --output=my_report.md
14+
15+
# Or add to PATH
16+
export PATH="$PATH:$PWD/script/tools"
17+
ck-build-analysis example_convnd_fwd_xdl_fp8
18+
```
19+
20+
## LLM Assistant Integration
21+
22+
If using an LLM assistant, you can ask in natural language:
23+
- "Analyze build time for example_convnd_fwd_xdl_fp8"
24+
- "Profile the compilation of test_amdgcn_mma with 1us granularity"
25+
- "Generate a build time report for example_gemm_xdl"
26+
27+
## Commands
28+
29+
```
30+
ck-build-analysis <target> [options]
31+
32+
Options:
33+
--granularity=N Time trace granularity in microseconds (default: 1)
34+
--output=FILE Output report filename (default: build_time_analysis_report.md)
35+
--name=NAME Docker container name (default: from CK_CONTAINER_NAME or auto-generated)
36+
--no-reconfigure Skip CMake reconfiguration if build exists
37+
--help Show this help message
38+
```
39+
40+
## What It Does
41+
42+
1. **Configures CMake** with `-ftime-trace` and custom granularity
43+
2. **Builds the target** using Ninja in Docker
44+
3. **Analyzes the trace** JSON file for template instantiation patterns
45+
4. **Generates a report** with:
46+
- Compilation phase breakdown
47+
- Top expensive individual instantiations
48+
- Template families ranked by total time and count
49+
- Key insights and optimization recommendations
50+
- Complete statistics
51+
52+
## Configuration
53+
54+
- **Container**: Uses ck-docker container (auto-starts if needed)
55+
- **Granularity**: Default 1us (100% template coverage, best balance)
56+
- **Output**: Markdown report in project root
57+
58+
## Environment
59+
60+
```bash
61+
export CK_CONTAINER_NAME=my_build # Override container name
62+
export CK_BUILD_ANALYSIS_GRANULARITY=1 # Default granularity in microseconds
63+
```
64+
65+
## Examples
66+
67+
```bash
68+
# Complete template analysis with default granularity (1us - recommended)
69+
ck-build-analysis example_convnd_fwd_xdl_fp8
70+
71+
# Quick daily check (10us granularity, captures most expensive templates)
72+
ck-build-analysis example_convnd_fwd_xdl_fp8 --granularity=10
73+
74+
# Maximum detail (0us granularity, includes LLVM internals)
75+
ck-build-analysis example_convnd_fwd_xdl_fp8 --granularity=0
76+
77+
# High-level overview (500us granularity, major bottlenecks only)
78+
ck-build-analysis example_convnd_fwd_xdl_fp8 --granularity=500
79+
80+
# Custom output filename
81+
ck-build-analysis example_convnd_fwd_xdl_fp8 --output=fp8_conv_analysis.md
82+
83+
# Analyze test target
84+
ck-build-analysis test_amdgcn_mma
85+
86+
# Use existing build (skip reconfigure)
87+
ck-build-analysis example_convnd_fwd_xdl_fp8 --no-reconfigure
88+
```
89+
90+
## Output
91+
92+
The report includes:
93+
- **Executive Summary**: Total time, events, instantiations, unique templates
94+
- **Compilation Phases**: InstantiateFunction, Frontend, Backend, Optimizer, etc.
95+
- **Top 30 Individual Instantiations**: Most expensive single templates
96+
- **Template Families**: Grouped by total time and instantiation count
97+
- **Key Insights**: What's slow and why
98+
- **Optimization Recommendations**: Short, medium, and long-term strategies
99+
- **Detailed Statistics**: Averages, medians, distributions
100+
101+
## Granularity Trade-offs
102+
103+
| Granularity | Template Coverage | Use Case |
104+
|-------------|-------------------|----------|
105+
| **0us** | All templates + sub-us compiler internals | LLVM internals debugging, very large files, higher overhead |
106+
| **1us (default)** | **All templates** | **Default: Complete template analysis with low overhead** |
107+
| **10us** | Most expensive templates | Daily quick checks, smaller files, minimal overhead |
108+
| **50-100us** | Top bottlenecks | Balanced detail/size, suitable for CI/CD |
109+
| **500us** | High-level phases only | Not recommended for template analysis |
110+
111+
**Recommended default**: 1us captures all template instantiations with minimal overhead
112+
113+
## Notes
114+
115+
- **0us and 1us capture all templates** - 0us adds sub-microsecond compiler internals
116+
- **1us is the sweet spot**: complete template coverage, filters noise, low overhead
117+
- **10us is practical** for daily use: captures most expensive templates, smaller files
118+
- **500us loses most template instantiation data** - only use for high-level phase breakdown
119+
- Finer granularity = more events = larger files + higher build time overhead
120+
- For template-heavy C++ codebases like CK: **use 1us for analysis, 10us for daily checks**
121+
122+
## Implementation Details
123+
124+
### PEP 723 Compliance with Automatic Dependency Management
125+
126+
The analysis script (`analyze_build_trace.py`) is PEP 723 compliant with inline dependency metadata:
127+
128+
```python
129+
# /// script
130+
# requires-python = ">=3.8"
131+
# dependencies = [
132+
# "jinja2>=3.0.0",
133+
# ]
134+
# ///
135+
```
136+
137+
**The tool automatically installs and uses `uv`**, which provides:
138+
- ✅ Zero-configuration dependency management
139+
- ✅ Automatic installation of jinja2 from PEP 723 metadata
140+
- ✅ Isolated dependency environment (no system pollution)
141+
- ✅ Fast caching for subsequent runs
142+
143+
**No manual setup required!** The first time you run the tool, it will:
144+
1. Detect if `uv` is installed in the container
145+
2. If not, automatically install it via Ubuntu packages (pipx install uv)
146+
3. Use `uv run` to execute the analysis with auto-managed dependencies
147+
148+
On subsequent runs, `uv` will already be available and dependencies will be cached.
149+
150+
Installation is done through Ubuntu's package manager for security and reliability.
151+
152+
### Components
153+
154+
- **ck-build-analysis** - Main bash script that orchestrates Docker, CMake, and analysis
155+
- **analyze_build_trace.py** - PEP 723 compliant Python script for trace analysis
156+
- **templates/build_analysis_report.md.jinja** - Jinja2 template for report generation
157+
158+
### Standalone Usage
159+
160+
The Python script can also be run independently:
161+
162+
```bash
163+
# With uv (recommended - auto-installs dependencies from PEP 723 metadata)
164+
uv run script/tools/analyze_build_trace.py trace.json report.md target 100 22 templates/
165+
166+
# With pipx (alternative - also auto-installs dependencies)
167+
pipx run script/tools/analyze_build_trace.py trace.json report.md target 100 22 templates/
168+
```

‎script/tools/README_ck-docker.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# ck-docker
2+
3+
Build and test composable_kernel in Docker with ROCm support.
4+
5+
## Terminal Usage
6+
7+
Direct command-line usage:
8+
9+
```bash
10+
# From composable_kernel directory
11+
script/tools/ck-docker start
12+
script/tools/ck-docker build test_amdgcn_mma
13+
script/tools/ck-docker test test_amdgcn_mma --gtest_filter=*Fp16*
14+
script/tools/ck-docker status
15+
script/tools/ck-docker shell
16+
17+
# Or add to PATH
18+
export PATH="$PATH:$PWD/script/tools"
19+
ck-docker start
20+
```
21+
22+
## LLM Assistant Integration
23+
24+
If using an LLM assistant, you can ask in natural language:
25+
- "Start the docker container"
26+
- "Build test_amdgcn_mma"
27+
- "Run test_amdgcn_mma with filter *Fp16*"
28+
- "Check container status"
29+
- "Open a shell in the container"
30+
31+
## Commands
32+
33+
```
34+
ck-docker start [name] Start Docker container
35+
ck-docker build [target] [--reconfigure] Build target (optionally reconfigure CMake)
36+
ck-docker test <name> [options] Run test
37+
ck-docker shell [name] Interactive shell
38+
ck-docker status [name] Check status
39+
ck-docker stop [name] Stop container
40+
```
41+
42+
## Configuration
43+
44+
- **Image**: rocm/composable_kernel:ck_ub24.04_rocm7.0.1
45+
- **GPU**: Auto-detected via rocminfo (fallback: gfx950)
46+
- **Compiler**: /opt/rocm/llvm/bin/clang++
47+
- **Build**: Ninja + CMake (Release)
48+
- **Mount**: Current directory → /workspace
49+
- **Container Name**: Auto-generated as `ck_<username>_<branch>` to avoid clashes
50+
51+
## Environment
52+
53+
```bash
54+
export CK_CONTAINER_NAME=my_build # Override default container name
55+
export CK_DOCKER_IMAGE=rocm/composable_kernel:ck_ub24.04_rocm7.0.1 # Override Docker image
56+
export GPU_TARGET=gfx942 # Override GPU target detection
57+
```
58+
59+
## Examples
60+
61+
```bash
62+
# Start container
63+
ck-docker start
64+
65+
# Build and run test
66+
ck-docker build test_amdgcn_mma
67+
ck-docker test test_amdgcn_mma
68+
69+
# Force clean CMake reconfiguration and build
70+
ck-docker build --reconfigure test_amdgcn_mma
71+
72+
# Custom container
73+
ck-docker start my_build
74+
ck-docker build test_amdgcn_mma --name my_build
75+
ck-docker test test_amdgcn_mma --name my_build
76+
77+
# Debug
78+
ck-docker shell
79+
ck-docker status
80+
```

0 commit comments

Comments
 (0)