Skip to content

Commit 2c2cf47

Browse files
devakoneclaude
andcommitted
feat(analysis): dampen initial/bulk commits in Automation axis
Add weighted averaging for the Automation Heaviness axis to prevent initial project commits and bulk operations from skewing results: - First commit: weight = 0.25 (likely scaffolding) - Bulk commits (>50% of max files, when max > 20): weight = 0.5 - Normal commits: weight = 1.0 This addresses feedback where users with small repos got high Automation scores due to a single large initial commit. Documentation updated: - docs/how-vibed-works.md - docs/architecture/vibe-metrics-v2.md - apps/web/src/app/methodology/page.tsx Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent 3d9918a commit 2c2cf47

4 files changed

Lines changed: 84 additions & 9 deletions

File tree

apps/web/src/app/methodology/page.tsx

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ export default async function MethodologyPage() {
5454
id: "A",
5555
name: "Automation",
5656
description:
57-
"Large, wide changes: high files-changed per commit, big commits, big PRs.",
57+
"Large, wide changes: high files-changed per commit, big commits, big PRs. Initial/bulk commits are dampened to avoid skewing small repos.",
5858
},
5959
{
6060
id: "B",
@@ -186,8 +186,12 @@ export default async function MethodologyPage() {
186186
copy/paste, refactors between commits), so we infer from what lands in Git.
187187
</li>
188188
<li>
189-
Different projects can pull you into different modes; aggregation may “average you
190-
out”.
189+
Different projects can pull you into different modes; aggregation may "average you
190+
out".
191+
</li>
192+
<li>
193+
We dampen initial and bulk commits in the Automation axis, but repos with very few
194+
commits (&lt;10) may still show unusual scores.
191195
</li>
192196
</ul>
193197
</section>

docs/architecture/vibe-metrics-v2.md

Lines changed: 18 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,11 +41,26 @@ How "agentic" the workflow looks.
4141

4242
**Score formula:**
4343
```
44-
40% - Commit chunkiness (avg_files_changed, commit_size_p90)
45-
40% - PR chunkiness (pr_files_changed_p90, commits_per_pr_p90)
46-
20% - Text templating score
44+
50% - Commit chunkiness (weighted_avg_files_changed, commit_size_p90)
45+
30% - p90 commit size
46+
20% - PR chunkiness (pr_files_changed_p90)
4747
```
4848

49+
**Initial/Bulk Commit Dampening:**
50+
51+
To prevent initial project commits and bulk operations from skewing the Automation score, we apply weight dampening when calculating `weighted_avg_files_changed`:
52+
53+
```typescript
54+
// Weight assignment per commit:
55+
weight = 1.0; // default
56+
if (isFirstCommit) weight = 0.25; // scaffolding likely
57+
else if (files_changed > maxFiles * 0.5 && maxFiles > 20) weight = 0.5; // bulk op
58+
59+
// Then: weighted_avg = sum(files * weight) / sum(weight)
60+
```
61+
62+
This ensures small repos with large initial commits (e.g., committing an entire create-react-app scaffold) don't get marked as "AI-heavy" based on that single data point.
63+
4964
**Evidence examples:**
5065
- "Top 10% PRs change 40+ files"
5166
- "Median commit touches 6 files"

docs/how-vibed-works.md

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,13 +76,27 @@ Six deterministic axes (0-100 scores) capture your workflow style:
7676

7777
| Axis | What It Measures |
7878
|------|------------------|
79-
| **Automation Heaviness** | How much you rely on AI agents and tools for generating code |
79+
| **Automation Heaviness** | How much you rely on AI agents and tools for generating code (initial/bulk commits dampened) |
8080
| **Guardrail Strength** | How closely tests, CI, and docs follow your AI-generated changes |
8181
| **Iteration Loop Intensity** | How often you run prompt-fix-run loops to refine AI output |
8282
| **Planning Signal** | How much structure you define before prompting AI to generate code |
8383
| **Surface Area per Change** | How many parts of the codebase your typical prompt or change touches |
8484
| **Shipping Rhythm** | Your coding session pattern — steady output vs intense vibe sessions |
8585

86+
#### Automation Axis: Initial Commit Dampening
87+
88+
The Automation Heaviness axis measures how "agentic" your workflow looks based on commit size and breadth. However, initial project commits (scaffolding an entire codebase at once) and bulk operations can skew this metric, especially for small repos.
89+
90+
To address this, we apply **commit weight dampening**:
91+
92+
| Commit Type | Weight | Rationale |
93+
|-------------|--------|-----------|
94+
| First commit | 0.25 | Often includes project scaffolding, framework setup |
95+
| Bulk commits (>50% of max files, when max > 20) | 0.5 | Large refactors, dependency updates |
96+
| Normal commits | 1.0 | Typical development work |
97+
98+
This means a repo with 7 commits where the first commit touched 800 files won't be marked as "AI-heavy" just because of the initial setup. The weighted average better reflects actual day-to-day patterns.
99+
86100
### Step 5: Detect Persona
87101
Based on your axes, we match you to one of 7 Vibe Personas:
88102

packages/core/src/vibe.ts

Lines changed: 44 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -603,9 +603,51 @@ export function computeVibeAxes(input: ComputeVibeAxesInput): ComputeVibeAxesOut
603603
const episodes = input.episodes ?? [];
604604

605605
// --- Axis A: Automation Heaviness ---
606-
const totalCommits = metrics.total_commits ?? input.commits?.length ?? 1;
606+
// We dampen the impact of initial/bulk commits to avoid skewing results for small repos.
607+
// - First commit: weight = 0.25 (often includes project scaffolding)
608+
// - Bulk commits (>50% of max files, when max > 20): weight = 0.5
609+
// - Normal commits: weight = 1.0
610+
const commits = input.commits ?? [];
611+
const totalCommits = metrics.total_commits ?? commits.length ?? 1;
607612
const totalFilesChanged = metrics.total_files_changed ?? 0;
608-
const avgFiles = totalFilesChanged / Math.max(1, totalCommits);
613+
614+
const avgFiles = (() => {
615+
if (commits.length === 0) {
616+
return totalFilesChanged / Math.max(1, totalCommits);
617+
}
618+
619+
// Sort by date to identify first commit
620+
const sortedCommits = [...commits].sort(
621+
(a, b) => new Date(a.committer_date).getTime() - new Date(b.committer_date).getTime()
622+
);
623+
624+
// Find max files changed to detect bulk commits
625+
const maxFilesChanged = Math.max(...commits.map((c) => c.files_changed), 1);
626+
const bulkThreshold = maxFilesChanged * 0.5;
627+
628+
let weightedFilesSum = 0;
629+
let totalWeight = 0;
630+
631+
for (let i = 0; i < sortedCommits.length; i++) {
632+
const commit = sortedCommits[i];
633+
let weight = 1.0;
634+
635+
// First commit gets dampened weight (likely project scaffolding)
636+
if (i === 0) {
637+
weight = 0.25;
638+
}
639+
// Bulk commits (>50% of max files) get reduced weight, but only if max is substantial
640+
else if (commit.files_changed > bulkThreshold && maxFilesChanged > 20) {
641+
weight = 0.5;
642+
}
643+
644+
weightedFilesSum += commit.files_changed * weight;
645+
totalWeight += weight;
646+
}
647+
648+
return totalWeight > 0 ? weightedFilesSum / totalWeight : totalFilesChanged / Math.max(1, totalCommits);
649+
})();
650+
609651
const p90Commit = Number(metrics.commit_size_p90 ?? 0);
610652

611653
const prChangedFiles = prs

0 commit comments

Comments
 (0)