Proposal: Stop overloading of notes column

[lukas2510]

Hi IDoFT maintainers,

Context

• Motivation: I started programmatic analysis on the dataset, focusing on tests that have a fix to infer the origin of flakiness from the fix commit. While doing so I ran into inconsistent evidence formats.
• I analyzed only rows marked as “fixed” (DeveloperFixed, Accepted, InspiredAFix, FixedOrder) across pr-data.csv, gr-data.csv, py-data.csv.
• Goal: verify the presence/quality of fix evidence using the existing “PR Link” and “Notes” fields for automation (e.g., to identify the origin of flakiness from the fix commit).
• Reproducibility: the scripts and generated CSVs are available on my fork/branch: https://github.com/lukas2510/idoft/tree/origin-of-flakiness

Artifacts (full, reproducible samples)
Here is the overview over the datasets: edge_cases_report.csv

• Fork/branch with scripts and outputs: https://github.com/lukas2510/idoft/tree/origin-of-flakiness
• data_transformation/output/edge_cases/edge_case_pr_link_and_notes_samples.csv
• data_transformation/output/edge_cases/edge_case_old_dataset_repo_link_samples.csv
• data_transformation/output/edge_cases/edge_case_idoft_repo_link_samples.csv
• data_transformation/output/edge_cases/edge_case_no_evidence_samples.csv
• data_transformation/output/edge_cases/edge_case_bare_commit_sha_samples.csv
• data_transformation/output/edge_cases/edge_case_multiple_links_in_notes_samples.csv
• data_transformation/output/edge_cases/edge_case_pr_commit_url_samples.csv
• data_transformation/output/edge_cases/edge_case_fork_link_samples.csv
• data_transformation/output/edge_cases/edge_case_branch_tree_link_samples.csv

Why this matters

• The Notes column is currently overloaded (issues, PRs, commits, branches, external references). This makes automation brittle and increases manual review.
• A small, consistent structure would make it much easier to automate origin-of-flakiness analysis and keep the dataset uniform.

Minimal change proposal (grouped for clarity)

1. Fix commit column & link format unification
   Example of a confusing row here

• Add a single optional column: Commit Link (or Fix_Commit_Link). Keep existing PR Link.
• Evidence hierarchy: Commit Link (best) > PR Link (second-best). If both exist, Commit Link is authoritative.
• Accepted commit evidence formats (one of):
  • Full URL: https://github.com/<org>/<repo>/commit/<sha>
  • PR commit view: https://github.com/<org>/<repo>/pull/<num>/commits/<sha> (normalize/store as /commit/<sha>)
  • Bare 40-char SHA (should be resolved/expanded to full URL where possible)
• Move any commit links currently embedded in Notes into the new Commit Link column.

2. Untangle idoft-wrapped / self-referential links
   Example of a confusing row here

• Pattern: Notes contains an idoft issue linking onward to the actual PR or commit.
• Proposed rule: Store the final PR/commit in Commit Link / PR Link; keep the idoft issue only as contextual provenance in Notes.
• Remove links to TestingResearchIllinois/flaky-test-dataset because it does not exist anymore.
• Result: Notes becomes lighter

3. Cross-repo redirection
   Example of a confusing row here

• Pattern: Initial PR (original repo) in PR Link; final accepted fix in a different repo only in Notes.
• Proposed rule: PR Link (or Commit Link) should point to the authoritative PR/commit where the fix landed (target repo). Initial exploratory/redirected PR kept in Notes.
• Benefit: Downstream automation can fetch the diff from the correct repository without extra heuristics.

Questions for maintainers

• In fixed-status rows with neither a PR link nor a commit link (currently 47 cases), is data missing or are these valid (e.g., private/internal fixes)? The cases can be found here
• What do you think of my approach to remove the overloading of the notes column? I am open for other suggestions as well.

Thanks for your work on IDoFT! I’m happy to prepare a PR if this direction sounds good.

[winglam]

@lukas2510 Thank you for your interest and suggestions to improve the dataset.

We won't be accepting any big changes (e.g., adding a new column) before the semester finishes in mid-December, but we're open to discussing all proposed changes until then.

For the proposed changes (2) and (3), would you be open to putting together separate pull requests to address them?

For the 47 cases where the PR link is empty, some of these cases appear similar to the ones you highlighted in (2). Namely, there are issue links that then point to the developer fixed PR/Commit link. For the remaining ones, we know only that the test is no longer flaky at the latest version, but we do not know the PR or commit in which the test is fixed.

Lastly, for the changes proposed in (1), the Commit link column sounds interesting. We can revisit this change closer to mid-December.

[darko-marinov]

Thank you for your interest in IDoFT, @lukas2510. You're right that Notes has been used for all kinds of purposes. As Wing suggested, it'd be great if you can open some PR(s) to fix issues (2) and (3) in the current format.

We can revisit (1) when IDoFT is not actively used by many students. Even for that, we may use Notes in a more consistent way, e.g., I looked at the first case that you listed (apache/activemq#1129), and the developer left a clear note "I'm doing a manual merge to squash all commits in one." (apache/activemq#1129 (comment)), but the Notes field does not explicitly label that the listed commit URL is the fix commit.

[lukas2510]

Thank you for the quick and thoughtful responses. I understand the constraint regarding major changes (like adding a column) until mid-December.

Update on Proposed Changes

Change (3): Cross-Repo Redirection
This is straightforward. I will prepare a separate Pull Request (PR) shortly to update the PR Link to the authoritative fix location for these few cases, as agreed.

Change (2): Untangle idoft-wrapped / Self-Referential Links
I found that many of these IDoFT issues link directly to a commit SHA rather than a Pull Request, often because the fix was merged without an explicit PR. This complicates fixing the evidence solely within the existing PR Link column. I believe a dedicated Commit Link column, as proposed in (1), would be the most robust solution for standardizing this evidence.

Change (1): Fix Commit Column & Link Format Unification
As this is on hold, I will probably proceed with my analysis using a commit link column in my own fork/scripts to automate the origin-of-flakiness analysis. If I find this structure highly beneficial during my work, I will share the results to inform your decision about integrating it into the main dataset later.

I will keep you updated on my progress.

[darko-marinov]

Thanks again for your suggestions. For (2), in most cases of Accepted, the commit link is simply the last commit in the PR link, whenever the GitHub PR status is "Merged". If the PR status is "Closed" (or "Open"), we may require an easy-to-parse note, e.g., fix in <commit_link>, and we could enforce it by the checker, without adding a new column.