[GR-21590] Import update

PullRequest: graalpython/4306

Author: timfel

.agents/skills/graalpython-rota/SKILL.md

@@ -0,0 +1,83 @@
+---
+name: graalpython-rota
+description: Run GraalPy ROTA maintenance workflows for (1) import update pull requests and (2) triage of recent periodic job failures in Jira. Use when asked to perform or guide recurring ROTA tasks from `docs/contributor/ROTA.md`, including branch setup, `mx` update commands, PR creation with reviewers/gates via `ol-cli bitbucket`, and date-bounded periodic-failure issue triage via `ol-cli jira`.
+---
+
+# GraalPy ROTA
+
+## Overview
+Execute recurring GraalPy ROTA tasks with exact commands and strict output structure. Prefer the procedures in this skill, and use `docs/contributor/ROTA.md` as the detailed source text.
+
+## Source Of Truth
+- Primary: `docs/contributor/ROTA.md`
+- If this skill workflow differs from the primary source, follow `docs/contributor/ROTA.md`.
+
+## Choose Workflow
+- Use `Import update` when asked to refresh imports and open the standard PR.
+- Use `Recent periodic issues` when asked to triage periodic job failures in Jira.
+
+## Import Update Workflow
+1. Create a branch from latest `master`:
+```bash
+git checkout master
+git pull --ff-only
+git checkout -b "update/GR-21590/$(date +%d%m%y)"
+```
+2. Update graal import:
+```bash
+mx python-update-import
+```
+3. Update CPython unittest whitelist and inspect diff for plausibility. Expect mostly additions, not removals:
+```bash
+mx --dy /graalpython-enterprise python-update-unittest-tags
+```
+4. Create PR with description `[GR-21590] Import update`.
+5. Use `ol-cli bitbucket` to create PR, start gates, and set reviewers:
+- `tim.felgentreff@oracle.com`
+- `michael.simacek@oracle.com`
+- `stepan.sindelar@oracle.com`
+6. Fix gate failures and push updates until gates pass.
+
+## Recent Periodic Issues Workflow
+1. Verify creator identity mapping:
+- Treat `ol-automation_ww` as Jira username `olauto`.
+- If query returns zero results, test both identities, then keep `creator = olauto` once verified.
+
+2. Filter to recent periodic job failures, excluding in progress or closed.
+- Default to the last 14 days unless user specifies otherwise.
+- Always state concrete start/end calendar dates in the response.
+```bash
+ol-cli jira search --json --max 100 \
+  -f key,summary,creator,created,status,labels,components,assignee \
+  -jql "project = GR AND component = Python AND creator = olauto AND labels = periodic-job-failures AND created >= -14d AND status != Closed AND status != 'In Progress' ORDER BY created DESC"
+```
+
+3. Fetch shortlisted issue details with `get-issue`:
+```bash
+ol-cli jira get-issue --json -id GR-XXXX \
+  | jq '{key, summary:.fields.summary, status:.fields.status.name, created:.fields.created, labels:.fields.labels, assignee:(.fields.assignee.name // null), description:.fields.description, comments:(.fields.comment.comments | map({author:.author.name, created, body}))}'
+```
+
+7. Convert findings into an implementation-ready plan per issue:
+- Extract failing job name, error signature, and log clue.
+- Map probable source area in repo.
+- Propose first verification command.
+- Define exit criteria to close ticket.
+- Prepare temporary git worktree per issue with branch naming based on Jira key plus very short hyphenated description.
+
+## Output Contract For Periodic Triage
+Return exactly:
+1. Query scope used (component, creator, time window, status filter).
+2. Count summary (total recent automation issues vs periodic failures).
+3. Issue list with key, created date, summary, status.
+4. Per-issue plan with:
+- Hypothesis
+- First code locations to inspect
+- First reproducibility command
+- Exit criteria for closing ticket
+5. Recommended implementation order.
+
+## Guardrails
+- State concrete dates for recency windows.
+- Prefer `--json` and explicit `-f` fields in searches.
+- Use `get-issue` only for shortlisted issues to keep output small.

.agents/skills/pr-gate-check/SKILL.md

@@ -0,0 +1,59 @@
+---
+name: pr-gate-check
+description: Check gate status for a Bitbucket PR by resolving the PR head commit, finding the gate merge commit, inspecting builds on that merge commit, and summarizing root-cause failures with actionable next steps.
+---
+
+# PR Gate Check
+
+## Overview
+Use this workflow when asked for gate status of a PR. Usually the builds are tied to a merge commit generated on Bitbucket, so this skill goes through finding the remote merge commit.
+
+## Workflow
+1. Get PR commits and identify PR head commit (first commit in `ol-cli bitbucket commits` output):
+```bash
+ol-cli bitbucket commits --project=G --repo=graalpython --pullrequest=<PR_ID> --all --json
+```
+
+2. Fetch refs and locate merge commit whose parent includes PR head:
+```bash
+git ls-remote origin 'refs/pull-requests/<PR_ID>/*'
+git fetch --no-tags origin '+refs/heads/*:refs/remotes/origin/*' --prune
+git rev-list --all --parents | rg ' <PR_HEAD_SHA>( |$)'
+```
+Pick the merge commit where one parent is `<PR_HEAD_SHA>` and the other is the target branch tip at merge time.
+
+3. Check builds on that merge commit:
+```bash
+ol-cli bitbucket get-builds --commit=<MERGE_SHA> --all --format=key,state,url
+```
+
+4. Separate root failures from fan-out failures:
+- `FAILED` + `/builders/.../builds/...` URL: executed failed build (root failure candidate).
+- `FAILED` + `build_request?brid=` URL: usually not-run/downstream due to earlier failure.
+
+5. Inspect root failed build logs and extract exact failing test/error:
+- Open build URL and `Run executor` stdio log.
+- Capture failing test id, traceback/assertion, and command context.
+
+6. Report back:
+- PR head SHA
+- merge SHA + parents (target parent + PR parent)
+- build summary counts
+- root cause failure(s)
+- fix options and next action question
+
+## Output Template
+1. `PR head:` `<sha>`
+2. `Gate merge commit:` `<sha>` (`parent1=<target_sha>`, `parent2=<pr_sha>`)
+3. `Builds:` `<total>` total, `<success>` successful, `<failed>` failed
+4. `Root failure(s):`
+- `<build key>`: `<error summary>`
+- `<failing test/path>`
+- `<build url>`
+5. `Proposed fixes:` short list
+6. Ask user what to do next.
+
+## Guardrails
+- Do not conclude from PR commit statuses alone; always resolve and inspect merge-commit builds.
+- If many builds are failed but only one executed failure exists, treat that one as primary cause.
+- Keep proposed fixes minimal and scoped to observed failure.

ci/graal/common.json

@@ -4,7 +4,7 @@
     "Jsonnet files should not include this file directly but use ci/common.jsonnet instead."
   ],
 
-  "mx_version": "7.69.0",
+  "mx_version": "7.71.0",
 
   "COMMENT.jdks": "When adding or removing JDKs keep in sync with JDKs in ci/common.jsonnet",
   "jdks": {

graalpython/com.oracle.graal.python.test/src/tests/test_fcntl.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2020, 2024, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2020, 2026, Oracle and/or its affiliates. All rights reserved.
 # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
 #
 # The Universal Permissive License (UPL), Version 1.0
@@ -65,6 +65,7 @@ def log(msg):
 def python_flock_blocks_sh_flock(python_flock_type, sh_flock_type):
     os.close(os.open(TEST_FILENAME_FULL_PATH, os.O_WRONLY | os.O_CREAT))
     file = os.open(TEST_FILENAME_FULL_PATH, os.O_WRONLY)
+    p = None
     try:
         fcntl.flock(file, python_flock_type)
         p = subprocess.Popen("flock -%s %s -c 'exit 42'" % (sh_flock_type, TEST_FILENAME_FULL_PATH), shell=True)
@@ -74,10 +75,13 @@ def python_flock_blocks_sh_flock(python_flock_type, sh_flock_type):
         log("unlocking the file...")
         fcntl.flock(file, fcntl.LOCK_UN) # release the lock
         log("checking the retcode...")
-        time.sleep(0.25)
-        assert p.poll() == 42
-        log(f"{p.returncode=}")
+        retcode = p.wait(timeout=5)
+        assert retcode == 42
+        log(f"{retcode=}")
     finally:
+        if p is not None and p.poll() is None:
+            p.terminate()
+            p.wait(timeout=2)
         fcntl.flock(file, fcntl.LOCK_UN)
         os.close(file)
 

graalpython/lib-python/3/test/test_concurrent_futures/executor.py

@@ -59,9 +59,11 @@ def test_map_timeout(self):
         # GraalPy change: submit some dummy work first, so the next map call doesn't time out in the worker start up
         list(self.executor.map(time.sleep, [0]))
         try:
+            # GraalPy change: larger timeout for CI flakiness
+            # for i in self.executor.map(time.sleep, [0, 0, 6], timeout=5):
             for i in self.executor.map(time.sleep,
-                                       [0, 0, 6],
-                                       timeout=5):
+                                       [0, 0, 8],
+                                       timeout=3):
                 results.append(i)
         except futures.TimeoutError:
             pass

mx.graalpython/mx_graalpython.py

@@ -2347,7 +2347,7 @@ def python_coverage(args):
             '--strict-mode',
             '--tags', args.tags,
         ] + jacoco_args, env=env)
-        run_mx([
+        jacoco_report_cmd = [
             '--strict-compliance',
             '--kill-with-sigquit',
             'jacocoreport',
@@ -2356,7 +2356,14 @@ def python_coverage(args):
             'coverage',
             '--generic-paths',
             '--exclude-src-gen',
-        ], env=env)
+        ]
+        # CI can occasionally leave transiently truncated execution data; retry once to reduce flakiness.
+        try:
+            run_mx(jacoco_report_cmd, env=env)
+        except Exception:  # pylint: disable=broad-except
+            mx.warn("jacocoreport failed, retrying once after a short delay")
+            time.sleep(5)
+            run_mx(jacoco_report_cmd, env=env)
 
     if args.mode == 'truffle':
         executable = graalpy_standalone_jvm()

mx.graalpython/mx_graalpython_python_benchmarks.py

@@ -77,6 +77,8 @@
     "bench_core.CountNonzero.time_count_nonzero_multi_axis(2, 1000000, <class 'str'>)",  # Times out
     "bench_core.CountNonzero.time_count_nonzero_multi_axis(3, 1000000, <class 'str'>)",  # Times out
     "bench_linalg.LinalgSmallArrays.time_det_small_array",  # TODO fails with numpy.linalg.LinAlgError
+    "bench_indexing.IndexingSeparate.time_mmap_fancy_indexing",  # Hangs in periodic job GR-73912
+    "bench_indexing.IndexingStructured0D.time_array_slice",  # Hangs in periodic job GR-73912
 ]
 
 DEFAULT_PANDAS_BENCHMARKS = [
@@ -98,6 +100,8 @@
     "reshape.Explode.time_explode",  # Transient failure GR-61245, exit code -11
 ]
 
+SETUPTOOLS_PIN = "77.0.1"
+
 DEFAULT_PYPERFORMANCE_BENCHMARKS = [
     # "2to3",
     # "chameleon",
@@ -564,7 +568,7 @@ class NumPySuite(PySuite):
 
     BENCHMARK_REQ = [
         "asv==0.5.1",
-        "setuptools==70.3.0",
+        f"setuptools=={SETUPTOOLS_PIN}",
         "distlib==0.3.6",
         "filelock==3.8.0",
         "platformdirs==2.5.2",
@@ -671,7 +675,7 @@ class PandasSuite(PySuite):
 
     BENCHMARK_REQ = [
         "asv==0.5.1",
-        "setuptools==70.3.0",
+        f"setuptools=={SETUPTOOLS_PIN}",
         "distlib==0.3.6",
         "filelock==3.8.0",
         "platformdirs==2.5.2",
@@ -762,8 +766,7 @@ def _vmRun(self, vm, workdir, command, benchmarks, bmSuiteArgs):
             vm.run(workdir, ["-m", "venv", join(workdir, vm_venv)])
             pip = join(workdir, vm_venv, "bin", "pip")
             with tempfile.NamedTemporaryFile('w') as constraints:
-                # Constrain the version of setuptools used to build pandas
-                constraints.write('setuptools==70.3.0\n')
+                constraints.write(f"setuptools=={SETUPTOOLS_PIN}\n")
                 constraints.flush()
                 env = os.environ.copy()
                 env['PIP_CONSTRAINT'] = constraints.name

mx.graalpython/suite.py

@@ -53,15 +53,15 @@
             },
             {
                 "name": "tools",
-                "version": "bb17fd7e8ec441c087b63300c2d75e06828b8dde",
+                "version": "fcbc5553eb65a0d6c88495efc24e998606dd5fa6",
                 "subdir": True,
                 "urls": [
                     {"url": "https://github.com/oracle/graal", "kind": "git"},
                 ],
             },
             {
                 "name": "regex",
-                "version": "bb17fd7e8ec441c087b63300c2d75e06828b8dde",
+                "version": "fcbc5553eb65a0d6c88495efc24e998606dd5fa6",
                 "subdir": True,
                 "urls": [
                     {"url": "https://github.com/oracle/graal", "kind": "git"},