Skip to content

collect_github_messages wildly inefficient with API calls for large repos #419

Description

@MoralCode

How did this take so long to identify...

collect_github_messages decides whether a repo is "small" (by checking if issue_pr_sum < 10) and then switches how it collects messages (comments i think)

if is_repo_small(repo_id):
message_data = fast_retrieve_all_pr_and_issue_messages(repo_git, logger, manifest.key_auth, task_name, core_data_last_collected)
if message_data:
process_messages(message_data, task_name, repo_id, logger, db_session)
else:
logger.info(f"{owner}/{repo} has no messages")
else:
process_large_issue_and_pr_message_collection(repo_id, repo_git, logger, manifest.key_auth, task_name, db_session, core_data_last_collected)

The "fast" path uses a call to /repos/{owner}/{repo}/issues/comments with the since= parameter

the slow path does an individual call per issue or comment:

for comment_url in comment_urls:
try:
messages = list(github_data_access.paginate_resource(comment_url))
all_data += messages

additionally, according to Sonnet 4.6 medium:

The large repo path filters by parent issue/PR updated_at in the DB query

The unified endpoint supports ?since= which filters by comment's own updated_at.

These semantics differ: if someone changes a label on a 3-year-old issue (updating the issue's updated_at), the large repo path fetches ALL of that issue's old comments again. The unified endpoint with since would correctly fetch only comments that were actually new or edited since the cutoff.

The large repo approach is actually less correct for incremental collection, not more correct — it re-fetches stale comment data that's already in the DB whenever a parent issue is touched for any reason.

This is very likely significantly contributing to #391

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions