Merge branch 'main' into zgu/refactor-vmstructs

Author: zhengyu123

.claude/commands/contribute-upstream.md

@@ -0,0 +1,169 @@
+---
+description: Analyze divergences from upstream async-profiler, propose grouped PRs, and open draft PRs.
+---
+
+# Contribute Upstream Workflow
+
+You are orchestrating the contribution of java-profiler divergences back to the upstream async-profiler project. These are all changes in our repo relative to upstream — not just uncommitted local modifications. Follow these steps precisely.
+
+## Configuration
+
+- **Fork repo**: `git@github.com:DataDog/async-profiler.git`
+- **Upstream repo**: `async-profiler/async-profiler` (for PR target)
+- **Upstream branch**: `master`
+- **Analysis script**: `utils/check_contribution_candidates.sh`
+- **Report dir**: `build/contribution-reports/`
+
+## Step 1: Run Analysis
+
+Execute the analysis script to generate reports:
+
+```bash
+./utils/check_contribution_candidates.sh
+```
+
+If it fails, diagnose and report the error to the user. Do not proceed.
+
+## Step 2: Parse Results
+
+Find the most recent JSON report in `build/contribution-reports/` (highest timestamp). Read it to get the list of files with contributable hunks.
+
+Also read the corresponding markdown report to understand the actual diff hunks for each file.
+
+If there are zero candidates, tell the user and stop.
+
+## Step 3: Filter Out Existing PRs
+
+Before proposing new PRs, check what's already open from the DataDog fork against upstream. Note: `--author DataDog` does not work because fork PRs are authored by the pushing user, not the org. Instead, query the API and filter by head repo:
+
+```bash
+gh api 'repos/async-profiler/async-profiler/pulls?state=open&per_page=100' \
+  --jq '.[] | select(.head.repo.full_name == "DataDog/async-profiler") | {number, title}'
+```
+
+Then for each matching PR, fetch the files it touches:
+
+```bash
+gh api 'repos/async-profiler/async-profiler/pulls/<number>/files' --jq '.[].filename'
+```
+
+For each open PR, extract the list of files it touches. Then cross-reference with the candidate files from Step 2:
+
+- If a candidate file is already fully covered by an open PR (i.e., the PR touches that file and addresses the same type of change), **exclude it** from proposals
+- If a candidate file is only partially covered (the open PR addresses some hunks but not others), keep the uncovered hunks as candidates
+- When presenting proposals in Step 4, mention any skipped files and the existing PR that covers them, so the user has full visibility
+
+Analyze the remaining contributable hunks across candidate files and group them into logical PR proposals. Guidelines:
+
+- **Related changes go together**: e.g., a bug fix touching `stackWalker.cpp` and `vmStructs.cpp` for the same issue = one PR
+- **Independent changes are separate**: unrelated fixes in different files = separate PRs
+- **Each PR should be self-contained**: it should make sense on its own, compile on its own, and have a clear rationale
+- **Keep PRs small**: prefer multiple small PRs over one large one
+
+For each proposed PR, prepare:
+- A descriptive title (e.g., "Fix null pointer check in stackWalker", "Add bounds validation in VMStruct")
+- The list of files and hunks it covers
+- A brief rationale explaining why this change benefits upstream
+
+## Step 5: Present Proposals to User
+
+Show the user a numbered list of proposed PRs with:
+- Title
+- Files involved
+- Brief description of the change
+- Number of hunks
+
+Use `AskUserQuestion` with `multiSelect: true` to let the user pick which PRs to create. Offer all proposals as options.
+
+If the user selects none, stop.
+
+## Step 6: Create Selected PRs
+
+For each selected PR, perform the following:
+
+### 6a. Clone the Fork (once)
+
+Clone `git@github.com:DataDog/async-profiler.git` to a temp directory. Reuse this clone for all PRs.
+
+```bash
+FORK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/async-profiler-fork.XXXXXX")
+git clone git@github.com:DataDog/async-profiler.git "$FORK_DIR"
+cd "$FORK_DIR"
+git remote add upstream https://github.com/async-profiler/async-profiler.git
+git fetch upstream
+git checkout -b temp upstream/master
+git branch -D master 2>/dev/null || true
+git checkout -b master
+git branch -D temp
+```
+
+### 6b. Create Feature Branch
+
+For each PR, create a branch from upstream master:
+
+```bash
+cd "$FORK_DIR"
+git checkout master
+BRANCH_NAME="contribute/<slug>-$(date +%Y%m%d)"
+git checkout -b "$BRANCH_NAME"
+```
+
+Where `<slug>` is a short kebab-case description derived from the PR title.
+
+### 6c. Port Changes
+
+Apply only the relevant contributable hunks for this PR to the upstream files:
+
+1. For each file in the PR, find the corresponding upstream file in `src/` of the fork
+2. Apply the contributable hunks using careful manual editing (use the Edit tool)
+3. **Critical**: Ensure NO Datadog-specific references leak through (DD_, ddprof, Datadog, datadog, DDPROF, context.h, counters.h, tagger, QueueItem)
+4. If a hunk cannot be cleanly applied because the upstream file diverged, skip it and note it for the user
+
+### 6d. Verify Build
+
+Attempt a basic build check:
+
+```bash
+cd "$FORK_DIR"
+make
+```
+
+If the build fails, analyze the error. If it's a simple fix (e.g., missing include), fix it. If it's complex, note it for the user and proceed anyway (the PR is draft).
+
+### 6e. Commit and Push
+
+```bash
+cd "$FORK_DIR"
+git add -A
+git commit -m "<concise description of the change>"
+git push origin "$BRANCH_NAME"
+```
+
+### 6f. Open Draft PR
+
+Use `gh` to create a draft PR against upstream. **Important**: Before creating the first PR, fetch the target project's PR template from the upstream repo (`gh api repos/async-profiler/async-profiler/contents/.github/PULL_REQUEST_TEMPLATE.md` and decode the base64 content). The PR body **must** follow that template exactly — use all its sections, checkboxes, and footer verbatim. Fill in each section with the relevant content for this change.
+
+```bash
+gh pr create \
+  --repo async-profiler/async-profiler \
+  --base master \
+  --head "DataDog:$BRANCH_NAME" \
+  --draft \
+  --title "<PR title>" \
+  --body "<body following the upstream PR template>"
+```
+
+## Step 7: Report
+
+After all selected PRs are created, show the user:
+- A summary table of created PRs with their URLs
+- Any hunks that could not be applied
+- Any build issues encountered
+- The temp directory path in case manual follow-up is needed
+
+## Error Handling
+
+- If `gh` is not authenticated, tell the user to run `gh auth login` and stop
+- If the fork clone fails, check SSH key setup and report
+- If a branch already exists on the fork, append a counter suffix (e.g., `-2`)
+- Always clean up on fatal errors (remove temp directory)

README.md

@@ -433,6 +433,20 @@ For detailed documentation, see [doc/RemoteSymbolication.md](doc/plans/RemoteSym
 4. Run tests: `./gradlew test`
 5. Submit a pull request
 
+## Utility Scripts
+
+The [`utils/`](utils/) directory contains helper scripts for common workflows. See [`utils/README.md`](utils/README.md) for full documentation.
+
+| Script | Description |
+|--------|-------------|
+| `release.sh` | Trigger a validated release (major/minor/patch) via GitHub Actions |
+| `backport-pr.sh` | Cherry-pick a merged PR onto a release branch and open a backport PR |
+| `patch-dd-java-agent.sh` | Patch `dd-java-agent.jar` with a local ddprof build for quick testing |
+| `run-docker-tests.sh` | Run tests in Docker containers (musl/glibc, multiple JDKs) |
+| `check_upstream_changes.sh` | Check for upstream async-profiler changes locally |
+| `track_upstream_changes.sh` | Track upstream changes and generate reports |
+| `generate_tracked_files.sh` | Generate the list of files tracked from upstream |
+
 ## License
 This project is licensed under the Apache License 2.0 - see the LICENSE file for details.
 

ddprof-lib/src/main/cpp/javaApi.cpp

@@ -440,9 +440,6 @@ Java_com_datadoghq_profiler_JVMAccess_healthCheck0(JNIEnv *env,
   return true;
 }
 
-// Static variable to track the current published context
-static otel_process_ctx_result* current_published_context = nullptr;
-
 extern "C" DLLEXPORT void JNICALL
 Java_com_datadoghq_profiler_OTelContext_setProcessCtx0(JNIEnv *env,
                                                          jclass unused,
@@ -460,16 +457,19 @@ Java_com_datadoghq_profiler_OTelContext_setProcessCtx0(JNIEnv *env,
   JniString version_str(env, version);
   JniString tracer_version_str(env, tracer_version);
 
+  const char *host_name_attrs[] = {"host.name", hostname_str.c_str(), NULL};
+
   otel_process_ctx_data data = {
-    .deployment_environment_name = const_cast<char*>(env_str.c_str()),
-    .host_name = const_cast<char*>(hostname_str.c_str()),
-    .service_instance_id = const_cast<char*>(runtime_id_str.c_str()),
-    .service_name = const_cast<char*>(service_str.c_str()),
-    .service_version = const_cast<char*>(version_str.c_str()),
-    .telemetry_sdk_language = const_cast<char*>("java"),
-    .telemetry_sdk_version = const_cast<char*>(tracer_version_str.c_str()),
-    .telemetry_sdk_name = const_cast<char*>("dd-trace-java"),
-    .resources = NULL // TODO: Arbitrary tags not supported yet for Java
+    .deployment_environment_name = env_str.c_str(),
+    .service_instance_id = runtime_id_str.c_str(),
+    .service_name = service_str.c_str(),
+    .service_version = version_str.c_str(),
+    .telemetry_sdk_language = "java",
+    .telemetry_sdk_version = tracer_version_str.c_str(),
+    .telemetry_sdk_name = "dd-trace-java",
+    .resource_attributes = host_name_attrs,
+    .extra_attributes = NULL,
+    .thread_ctx_config = NULL
   };
 
   otel_process_ctx_result result = otel_process_ctx_publish(&data);
@@ -491,8 +491,6 @@ Java_com_datadoghq_profiler_OTelContext_readProcessCtx0(JNIEnv *env, jclass unus
   // Convert C strings to Java strings
   jstring jDeploymentEnvironmentName = result.data.deployment_environment_name ?
     env->NewStringUTF(result.data.deployment_environment_name) : nullptr;
-  jstring jHostName = result.data.host_name ?
-    env->NewStringUTF(result.data.host_name) : nullptr;
   jstring jServiceInstanceId = result.data.service_instance_id ?
     env->NewStringUTF(result.data.service_instance_id) : nullptr;
   jstring jServiceName = result.data.service_name ?
@@ -505,7 +503,17 @@ Java_com_datadoghq_profiler_OTelContext_readProcessCtx0(JNIEnv *env, jclass unus
     env->NewStringUTF(result.data.telemetry_sdk_version) : nullptr;
   jstring jTelemetrySdkName = result.data.telemetry_sdk_name ?
     env->NewStringUTF(result.data.telemetry_sdk_name) : nullptr;
-  // TODO: result.data.resources not supported yet for Java
+
+  // Extract host.name from resource_attributes
+  jstring jHostName = nullptr;
+  if (result.data.resource_attributes != NULL) {
+    for (int i = 0; result.data.resource_attributes[i] != NULL; i += 2) {
+      if (strcmp(result.data.resource_attributes[i], "host.name") == 0 && result.data.resource_attributes[i + 1] != NULL) {
+        jHostName = env->NewStringUTF(result.data.resource_attributes[i + 1]);
+        break;
+      }
+    }
+  }
 
   otel_process_ctx_read_drop(&result);