feat: implement concurrent trigram index building

[yanurag-dev]

• Introduced a worker pool and collector pattern in Builder.Build for parallel file processing.
• Updated benchcmd to use the concurrent approach for accurate performance measurement.
• Improved indexing speed on Go stdlib corpus by ~1.9x.

Summary by CodeRabbit

• Chores
  • Upgraded Go toolchain from version 1.20 to 1.25.0
  • Updated and promoted key dependencies to direct requirements

[coderabbitai]

Warning

Rate limit exceeded

@yanurag-dev has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 54 minutes and 38 seconds before requesting another review.

Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 54 minutes and 38 seconds.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 67bc1900-8308-4b1a-9213-84ab899cf09c

📥 Commits

Reviewing files that changed from the base of the PR and between 7cc9fb6 and b1e2d6f.

⛔ Files ignored due to path filters (1)

• go.sum is excluded by !**/*.sum

📒 Files selected for processing (2)

• .github/workflows/go.yml
• go.mod

📝 Walkthrough

Walkthrough

The Go toolchain version is bumped from 1.20 to 1.25.0, and two dependencies are promoted to direct requirements. File processing in the index builder and benchmark tools is refactored from single-threaded sequential operations to a producer/worker/consumer concurrency pattern using channels and goroutines.

Changes

Cohort / File(s)	Summary
Dependency Management go.mod	Go toolchain updated from 1.20 to 1.25.0; github.com/spf13/cobra and golang.org/x/sys moved from indirect to direct dependencies.
Concurrent Indexing internal/index/builder.go, scripts/benchcmd/main.go	File processing refactored from single-threaded sequential operations to producer/worker/consumer concurrency patterns. Directory walking enqueues file paths; worker goroutines validate files and extract trigrams; consumer goroutines serially assign file IDs and update builder state via channels.

Sequence Diagram(s)

sequenceDiagram
    participant Main as Main Goroutine
    participant Walker as Directory Walker
    participant PathCh as paths Channel
    participant Workers as Worker Goroutines
    participant ResultCh as results Channel
    participant Consumer as Consumer Goroutine
    participant Builder as Builder State

    Main->>Walker: Start directory walk
    Walker->>PathCh: Enqueue file paths
    PathCh->>Workers: Send path
    Workers->>Workers: Validate file (UTF-8, size)
    Workers->>Workers: Extract trigrams
    Workers->>ResultCh: Send extractResult
    ResultCh->>Consumer: Receive result
    Consumer->>Builder: Assign fileID<br/>(stable, based on b.Files length)
    Consumer->>Builder: Append file path
    Consumer->>Builder: Batch-add trigram mappings
    Main->>Main: Wait for all workers & consumer
    Main->>Builder: Call Posts.Finalize()

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐰 With channels dancing and goroutines at play,
Files now process in a concurrent way!
Workers extract while consumers align,
Each trigram mapped in perfect design.
From 1.20 springs forth to 1.25,
This indexing code feels so alive! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The pull request title 'feat: implement concurrent trigram index building' directly and accurately describes the main change—introducing concurrent processing to the trigram index builder via a worker pool and collector pattern.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feature/concurrent-index-build

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}