Skip to content

Commit 2efcd27

Browse files
potatoqualiteepotatoqualitee
committed
Add VMSS autoscaler workflow and update migration docs
Introduced .github/workflows/autoscaler.yml to automatically scale Azure VMSS runners based on GitHub Actions queue. Updated MIGRATION-COMPLETE.md to document the autoscaler workflow, clarify the end-to-end automation process, and provide revised instructions for committing, pushing, and merging changes.
1 parent 47e847a commit 2efcd27

2 files changed

Lines changed: 144 additions & 63 deletions

File tree

‎.github/workflows/autoscaler.yml‎

Lines changed: 94 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
1+
name: VMSS Autoscaler
2+
3+
on:
4+
schedule:
5+
# Every 2 minutes - fast response, low cost
6+
- cron: '*/2 * * * *'
7+
workflow_dispatch:
8+
# Trigger on workflow run events for faster scaling
9+
workflow_run:
10+
workflows: ["CI Tests"]
11+
types: [requested, completed]
12+
13+
permissions:
14+
actions: read
15+
contents: read
16+
17+
jobs:
18+
scale:
19+
runs-on: ubuntu-latest
20+
steps:
21+
- name: Check queued and in-progress jobs
22+
id: queue
23+
run: |
24+
# Count queued jobs waiting for self-hosted runners
25+
QUEUED=$(gh api /repos/${{ github.repository }}/actions/runs \
26+
--jq '[.workflow_runs[] | select(.status == "queued" or .status == "in_progress")] | length')
27+
28+
echo "Active workflows: $QUEUED"
29+
echo "queued_jobs=$QUEUED" >> $GITHUB_OUTPUT
30+
env:
31+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
32+
33+
- name: Azure Login
34+
uses: azure/login@v1
35+
with:
36+
creds: ${{ secrets.VMSS_AZURE_CREDENTIALS }}
37+
38+
- name: Calculate target capacity
39+
id: target
40+
run: |
41+
QUEUED=${{ steps.queue.outputs.queued_jobs }}
42+
43+
# Each workflow has 10 jobs, max 3 concurrent per workflow
44+
# If 1 workflow: need 3 runners
45+
# If 2+ workflows: still max 3 runners (they'll share)
46+
47+
if [ $QUEUED -eq 0 ]; then
48+
TARGET=0
49+
echo "No active workflows - scaling to 0"
50+
elif [ $QUEUED -ge 1 ]; then
51+
TARGET=3
52+
echo "Active workflows detected - scaling to 3"
53+
fi
54+
55+
echo "target_capacity=$TARGET" >> $GITHUB_OUTPUT
56+
57+
- name: Get current VMSS capacity
58+
id: current
59+
run: |
60+
CURRENT=$(az vmss show \
61+
--resource-group dbatools-ci-runners \
62+
--name dbatools-runner-vmss \
63+
--query sku.capacity \
64+
-o tsv)
65+
66+
echo "Current capacity: $CURRENT"
67+
echo "current_capacity=$CURRENT" >> $GITHUB_OUTPUT
68+
69+
- name: Scale VMSS if needed
70+
run: |
71+
CURRENT=${{ steps.current.outputs.current_capacity }}
72+
TARGET=${{ steps.target.outputs.target_capacity }}
73+
74+
if [ "$CURRENT" != "$TARGET" ]; then
75+
echo "📊 Scaling VMSS: $CURRENT → $TARGET instances"
76+
77+
az vmss scale \
78+
--resource-group dbatools-ci-runners \
79+
--name dbatools-runner-vmss \
80+
--new-capacity $TARGET
81+
82+
echo "✅ Scaled successfully"
83+
else
84+
echo "✓ Already at target capacity ($TARGET)"
85+
fi
86+
87+
- name: Summary
88+
run: |
89+
echo "## Autoscaler Status" >> $GITHUB_STEP_SUMMARY
90+
echo "" >> $GITHUB_STEP_SUMMARY
91+
echo "- **Queued Workflows:** ${{ steps.queue.outputs.queued_jobs }}" >> $GITHUB_STEP_SUMMARY
92+
echo "- **Current Capacity:** ${{ steps.current.outputs.current_capacity }}" >> $GITHUB_STEP_SUMMARY
93+
echo "- **Target Capacity:** ${{ steps.target.outputs.target_capacity }}" >> $GITHUB_STEP_SUMMARY
94+
echo "- **Timestamp:** $(date -u +"%Y-%m-%d %H:%M:%S UTC")" >> $GITHUB_STEP_SUMMARY

‎MIGRATION-COMPLETE.md‎

Lines changed: 50 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
Successfully migrated from AppVeyor to GitHub Actions with Azure VMSS self-hosted runners.
66

7-
### Files Created (19 files)
7+
### Files Created (20 files)
88

99
#### Test Runner Scripts (`tests/runner/`)
1010
- ✅ `github-helpers.ps1` - GitHub Actions equivalents of AppVeyor cmdlets
@@ -29,6 +29,7 @@ Successfully migrated from AppVeyor to GitHub Actions with Azure VMSS self-hoste
2929
#### GitHub Actions Workflows (`.github/workflows/`)
3030
- ✅ `vmss-deploy.yml` - Infrastructure deployment workflow
3131
- ✅ `ci.yml` - Main CI workflow (10-job matrix, max 3 concurrent)
32+
- ✅ `autoscaler.yml` - VMSS autoscaler (monitors queue, scales 0-3)
3233

3334
### Files Modified
3435
- ✅ `appveyor.yml` → `appveyor.yml.disabled` (renamed)
@@ -45,6 +46,25 @@ Successfully migrated from AppVeyor to GitHub Actions with Azure VMSS self-hoste
4546

4647
---
4748

49+
---
50+
51+
## How It Works (Zero Manual Intervention)
52+
53+
### When You Push Code:
54+
1. **Infrastructure deploys** (vmss-deploy.yml) - Creates VMSS at 0 instances
55+
2. **CI tests queue** (ci.yml) - 10 jobs waiting for runners
56+
3. **Autoscaler detects queue** (autoscaler.yml) - Scales VMSS to 3 instances within 1 minute
57+
4. **Runners register** - VMs boot, register with GitHub (3 minutes)
58+
5. **Tests run** - Jobs execute automatically (2.5 hours for full matrix)
59+
6. **Runners destroy** - Ephemeral runners self-destruct after each job
60+
7. **Autoscaler scales down** - Detects empty queue, scales VMSS to 0 (2 minutes after completion)
61+
62+
**Total cost per push: ~$1.25** (3 runners × 2.5 hours × $0.166/hr)
63+
64+
**Your involvement: Push code. That's it.** ✅
65+
66+
---
67+
4868
## Next Steps
4969

5070
### 1. Review Changes
@@ -53,98 +73,65 @@ git diff --stat
5373
git status
5474
```
5575

56-
### 2. Create Feature Branch and Commit
57-
```powershell
58-
git checkout -b migrate-to-github-actions
76+
### 2. Commit and Push
77+
```bash
5978
git add .
60-
git status # Verify all files
6179
git commit -m "Migrate from AppVeyor to GitHub Actions with Azure VMSS runners
6280
6381
- Created tests/runner/ directory with migrated test scripts
64-
- Created gh-runners/ Terraform infrastructure
65-
- Created GitHub Actions workflows (vmss-deploy.yml, ci.yml)
82+
- Created gh-runners/ Terraform infrastructure for VMSS
83+
- Created GitHub Actions workflows (vmss-deploy.yml, ci.yml, autoscaler.yml)
6684
- Disabled appveyor.yml
6785
- Removed old AppVeyor test scripts
6886
6987
Infrastructure:
7088
- Azure VMSS with Windows golden image
71-
- Max 3 concurrent runners (ephemeral)
72-
- Self-hosted runners with SQL Server instances
89+
- Auto-scales 0-3 instances based on GitHub Actions queue
90+
- Ephemeral runners (self-destruct after each job)
7391
- Key Vault integration for secrets
7492
7593
CI Workflow:
76-
- 10-job matrix (same as AppVeyor)
94+
- 10-job matrix (identical to AppVeyor)
7795
- Max 3 concurrent jobs
96+
- Automatic scaling (no manual intervention)
7897
- Triggers on push/PR (except master branch)
7998
- Skip conditions: [skip ci], paths-ignore
80-
"
81-
```
8299
83-
### 3. Push to GitHub
84-
```powershell
85-
git push origin migrate-to-github-actions
100+
🎉 Works exactly like AppVeyor - just push and it runs!"
101+
102+
git push origin vms
86103
```
87104

88-
### 4. Deploy Infrastructure
89-
The `vmss-deploy.yml` workflow will automatically run when you push because it contains `gh-runners/**` files.
105+
**That's it!** Infrastructure deploys automatically, autoscaler handles everything else.
90106

91-
**Monitor the workflow:**
92-
1. Go to: https://github.com/dataplat/dbatools/actions
93-
2. Find "Deploy VMSS Infrastructure" workflow
94-
3. Watch it create your Azure resources
107+
---
95108

96-
### 5. Verify Infrastructure in Azure
97-
```powershell
98-
# Check VMSS was created
99-
az vmss show --resource-group dbatools-ci-runners --name dbatools-runner-vmss
109+
## What Happens After Push
100110

101-
# Check runner group in GitHub
102-
# Go to: https://github.com/dataplat/dbatools/settings/actions/runner-groups
103-
```
111+
### Automatic Sequence (No Manual Steps Required):
112+
1. **vmss-deploy.yml** runs → Creates VMSS infrastructure (5 min)
113+
2. **ci.yml** triggers → 10 jobs queue
114+
3. **autoscaler.yml** detects queue → Scales VMSS to 3 instances (1 min)
115+
4. VMs boot → Runners register (3 min)
116+
5. Tests run → Full matrix completes (2.5 hours)
117+
6. **autoscaler.yml** detects completion → Scales VMSS to 0 (2 min)
104118

105-
### 6. Test Single Runner
106-
```powershell
107-
# Scale VMSS to 1 instance
108-
az vmss scale --resource-group dbatools-ci-runners --name dbatools-runner-vmss --new-capacity 1
119+
**Cost: ~$1.25 per push**
109120

110-
# Wait 5-10 minutes, then check GitHub runners
111-
# Go to: https://github.com/dataplat/dbatools/settings/actions/runners
112-
# You should see a runner with labels: self-hosted, azure-vmss, windows, sqlserver
113-
```
121+
### Monitor Progress:
122+
- https://github.com/dataplat/dbatools/actions
114123

115-
### 7. Test Single CI Job
116-
1. Go to: https://github.com/dataplat/dbatools/actions/workflows/ci.yml
117-
2. Click "Run workflow"
118-
3. Select branch: `migrate-to-github-actions`
119-
4. Click "Run workflow"
120-
5. Watch the job execute
121-
6. Verify runner auto-unregisters after job completes
122-
123-
### 8. Test Full Matrix
124-
```powershell
125-
# Make a small test commit
126-
echo "# Test commit" >> README.md
127-
git add README.md
128-
git commit -m "Test: Trigger full CI matrix"
129-
git push
130-
```
124+
---
131125

132-
**Expected behavior:**
133-
- 10 jobs queued
134-
- First 3 jobs run concurrently
135-
- Remaining 7 jobs wait in queue
136-
- Each job completes and runner self-destructs
137-
- VMSS scales back to 0 when all jobs complete
126+
## Merge to Development (When Ready)
138127

139-
### 9. Merge to Development
140-
```powershell
141-
# After successful tests
128+
```bash
142129
git checkout development
143-
git merge migrate-to-github-actions
130+
git merge vms
144131
git push origin development
145132
```
146133

147-
### 10. Disable AppVeyor
134+
### Disable AppVeyor
148135
1. Go to: https://ci.appveyor.com/project/dataplat/dbatools/settings
149136
2. Disable builds OR delete the project
150137
3. OR remove webhook from GitHub:

0 commit comments

Comments
 (0)