Skip to content

Commit b156533

Browse files
dsymedsyme
committed
add dataops pattern
1 parent d07932e commit b156533

2 files changed

Lines changed: 261 additions & 0 deletions

File tree

‎docs/astro.config.mjs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -183,6 +183,7 @@ export default defineConfig({
183183
{ label: 'IssueOps', link: '/patterns/issueops/' },
184184
{ label: 'LabelOps', link: '/patterns/labelops/' },
185185
{ label: 'ProjectOps', link: '/patterns/projectops/' },
186+
{ label: 'DataOps', link: '/patterns/dataops/' },
186187
{ label: 'TaskOps', link: '/patterns/taskops/' },
187188
{ label: 'MultiRepoOps', link: '/patterns/multirepoops/' },
188189
{ label: 'SideRepoOps', link: '/patterns/siderepoops/' },

‎docs/src/content/docs/patterns/dataops.md‎

Lines changed: 260 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,260 @@
1+
---
2+
title: DataOps
3+
description: Deterministic data extraction in steps, followed by agentic analysis and reporting
4+
sidebar:
5+
badge: { text: 'Hybrid', variant: 'caution' }
6+
---
7+
8+
DataOps combines deterministic data extraction with agentic analysis. Shell commands in `steps:` collect and prepare data, then the AI agent in the markdown body analyzes results and produces safe outputs like discussions or comments.
9+
10+
## When to Use DataOps
11+
12+
- **Data aggregation** - Collect metrics from APIs, logs, or repositories
13+
- **Report generation** - Analyze data and produce human-readable summaries
14+
- **Trend analysis** - Process historical data and identify patterns
15+
- **Auditing** - Gather evidence and generate audit reports
16+
17+
## The DataOps Pattern
18+
19+
### Separation of Concerns
20+
21+
DataOps separates two distinct phases:
22+
23+
1. **Deterministic extraction** (`steps:`) - Shell commands that reliably fetch, filter, and structure data. These run before the agent and produce predictable, reproducible results.
24+
25+
2. **Agentic analysis** (markdown body) - The AI agent reads the prepared data, interprets patterns, and generates insights. The agent has access to the data files created by the steps.
26+
27+
This separation ensures data collection is fast, reliable, and cacheable, while the AI focuses on interpretation and communication.
28+
29+
### Basic Structure
30+
31+
```aw wrap
32+
---
33+
on:
34+
schedule: daily
35+
workflow_dispatch:
36+
37+
steps:
38+
- name: Collect data
39+
run: |
40+
# Deterministic data extraction
41+
gh api ... > /tmp/gh-aw/data.json
42+
43+
safe-outputs:
44+
create-discussion:
45+
category: "reports"
46+
---
47+
48+
# Analysis Workflow
49+
50+
Analyze the data at `/tmp/gh-aw/data.json` and create a summary report.
51+
```
52+
53+
## Example: PR Activity Summary
54+
55+
This workflow collects statistics from recent pull requests and generates a weekly summary:
56+
57+
````aw wrap
58+
---
59+
name: Weekly PR Summary
60+
description: Summarizes pull request activity from the last week
61+
on:
62+
schedule: weekly
63+
workflow_dispatch:
64+
65+
permissions:
66+
contents: read
67+
pull-requests: read
68+
69+
engine: copilot
70+
strict: true
71+
72+
network:
73+
allowed:
74+
- defaults
75+
- github
76+
77+
safe-outputs:
78+
create-discussion:
79+
title-prefix: "[weekly-summary] "
80+
category: "announcements"
81+
max: 1
82+
close-older-discussions: true
83+
84+
tools:
85+
bash: ["*"]
86+
87+
steps:
88+
- name: Fetch recent pull requests
89+
env:
90+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
91+
run: |
92+
mkdir -p /tmp/gh-aw/pr-data
93+
94+
# Fetch last 100 PRs with key metadata
95+
gh pr list \
96+
--repo "${{ github.repository }}" \
97+
--state all \
98+
--limit 100 \
99+
--json number,title,state,author,createdAt,mergedAt,closedAt,additions,deletions,changedFiles,labels \
100+
> /tmp/gh-aw/pr-data/recent-prs.json
101+
102+
echo "Fetched $(jq 'length' /tmp/gh-aw/pr-data/recent-prs.json) PRs"
103+
104+
- name: Compute summary statistics
105+
run: |
106+
cd /tmp/gh-aw/pr-data
107+
108+
# Generate statistics summary
109+
jq '{
110+
total: length,
111+
merged: [.[] | select(.state == "MERGED")] | length,
112+
open: [.[] | select(.state == "OPEN")] | length,
113+
closed: [.[] | select(.state == "CLOSED")] | length,
114+
total_additions: [.[].additions] | add,
115+
total_deletions: [.[].deletions] | add,
116+
total_files_changed: [.[].changedFiles] | add,
117+
authors: [.[].author.login] | unique | length,
118+
top_authors: ([.[].author.login] | group_by(.) | map({author: .[0], count: length}) | sort_by(-.count) | .[0:5])
119+
}' recent-prs.json > stats.json
120+
121+
echo "Statistics computed:"
122+
cat stats.json
123+
124+
timeout-minutes: 10
125+
---
126+
127+
# Weekly Pull Request Summary
128+
129+
Generate a summary of pull request activity for the repository.
130+
131+
## Available Data
132+
133+
The following data has been prepared for your analysis:
134+
135+
- `/tmp/gh-aw/pr-data/recent-prs.json` - Last 100 PRs with full metadata
136+
- `/tmp/gh-aw/pr-data/stats.json` - Pre-computed statistics
137+
138+
## Your Task
139+
140+
1. **Read the prepared data** from the files above
141+
2. **Analyze the statistics** to identify:
142+
- Overall activity levels
143+
- Merge rate and velocity
144+
- Most active contributors
145+
- Code churn (additions vs deletions)
146+
3. **Generate a summary report** as a GitHub discussion with:
147+
- Key metrics in a clear format
148+
- Notable trends or observations
149+
- Top contributors acknowledgment
150+
151+
## Report Format
152+
153+
Create a discussion with this structure:
154+
155+
```markdown
156+
# Weekly PR Summary - [Date Range]
157+
158+
## Key Metrics
159+
- **Total PRs**: X
160+
- **Merged**: X (Y%)
161+
- **Open**: X
162+
- **Code Changes**: +X / -Y lines across Z files
163+
164+
## Top Contributors
165+
1. @author1 - X PRs
166+
2. @author2 - Y PRs
167+
...
168+
169+
## Observations
170+
[Brief insights about activity patterns]
171+
```
172+
173+
Keep the report concise and factual. Focus on the numbers and let them tell the story.
174+
````
175+
176+
## Data Caching
177+
178+
For workflows that run frequently or process large datasets, use caching to avoid redundant API calls:
179+
180+
```aw wrap
181+
---
182+
cache:
183+
- key: pr-data-${{ github.run_id }}
184+
path: /tmp/gh-aw/pr-data
185+
restore-keys: |
186+
pr-data-
187+
188+
steps:
189+
- name: Check cache and fetch only new data
190+
run: |
191+
if [ -f /tmp/gh-aw/pr-data/recent-prs.json ]; then
192+
echo "Using cached data"
193+
else
194+
gh pr list --limit 100 --json ... > /tmp/gh-aw/pr-data/recent-prs.json
195+
fi
196+
---
197+
```
198+
199+
## Advanced: Multi-Source Data
200+
201+
Combine data from multiple sources before analysis:
202+
203+
```aw wrap
204+
---
205+
steps:
206+
- name: Fetch PR data
207+
run: gh pr list --json ... > /tmp/gh-aw/prs.json
208+
209+
- name: Fetch issue data
210+
run: gh issue list --json ... > /tmp/gh-aw/issues.json
211+
212+
- name: Fetch workflow runs
213+
run: gh run list --json ... > /tmp/gh-aw/runs.json
214+
215+
- name: Combine into unified dataset
216+
run: |
217+
jq -s '{prs: .[0], issues: .[1], runs: .[2]}' \
218+
/tmp/gh-aw/prs.json \
219+
/tmp/gh-aw/issues.json \
220+
/tmp/gh-aw/runs.json \
221+
> /tmp/gh-aw/combined.json
222+
---
223+
224+
# Repository Health Report
225+
226+
Analyze the combined data at `/tmp/gh-aw/combined.json` covering:
227+
- Pull request velocity and review times
228+
- Issue response rates and resolution times
229+
- CI/CD success rates and flaky tests
230+
```
231+
232+
## Best Practices
233+
234+
**Keep steps deterministic** - Avoid randomness or time-dependent logic in steps. The same inputs should produce the same outputs.
235+
236+
**Pre-compute aggregations** - Use `jq`, `awk`, or Python in steps to compute statistics. This reduces agent token usage and improves reliability.
237+
238+
**Structure data clearly** - Output JSON with clear field names. Include a summary file alongside raw data.
239+
240+
**Document data locations** - Tell the agent exactly where to find the prepared data and what format to expect.
241+
242+
**Use safe outputs** - Always use `safe-outputs` for agent actions. Discussions are ideal for reports since they support threading and reactions.
243+
244+
## Comparison with Other Patterns
245+
246+
| Pattern | Trigger | Data Source | Output |
247+
|---------|---------|-------------|--------|
248+
| **DataOps** | Schedule/manual | External APIs, logs | Reports, discussions |
249+
| [DailyOps](/gh-aw/patterns/dailyops/) | Schedule | Repository analysis | Incremental PRs |
250+
| [IssueOps](/gh-aw/patterns/issueops/) | Issue events | Issue content | Issue comments |
251+
| [ChatOps](/gh-aw/patterns/chatops/) | Comments | User commands | Inline responses |
252+
253+
DataOps is distinguished by its emphasis on **deterministic data preparation** before agentic processing.
254+
255+
## Additional Resources
256+
257+
- [Steps Reference](/gh-aw/reference/steps/) - Shell step configuration
258+
- [Safe Outputs Reference](/gh-aw/reference/safe-outputs/) - Validated GitHub operations
259+
- [Cache Configuration](/gh-aw/reference/cache/) - Caching data between runs
260+
- [DailyOps](/gh-aw/patterns/dailyops/) - Scheduled improvement workflows

0 commit comments

Comments
 (0)