use latest nightly to test by usamasaqib · Pull Request #49869 · DataDog/datadog-agent

usamasaqib · 2026-04-24T09:53:40Z

What does this PR do?

Motivation

Describe how you validated your changes

Additional Notes

dd-octo-sts · 2026-04-24T09:58:33Z

Gitlab CI Configuration Changes

Modified Jobs

.single-machine-performance-run_regression_detector

  .single-machine-performance-run_regression_detector:
    artifacts:
      expire_in: 1 weeks
      paths:
      - submission_metadata
      - outputs/report.md
      - outputs/regression_signal.json
      - outputs/bounds_check_signal.json
      - outputs/junit.xml
      - outputs/report.json
      - outputs/decision_record.md
      when: always
    before_script:
    - echo "BASELINE_SHA=${BASELINE_SHA}" > regression_detector.env
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$CI_IMAGE_DOCKER_X64_SUFFIX:$CI_IMAGE_DOCKER_X64
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    script:
    - DATADOG_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DATADOG_API_KEY
    - datadog-ci tag --level job --tags smp_failure_mode:"timeout"
    - mkdir outputs
    - AWS_NAMED_PROFILE="single-machine-performance"
    - "SMP_ACCOUNT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT account_id)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - SMP_ECR_URL=${SMP_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com
    - "SMP_TEAM_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT \"${SMP_TEAM_NAME}_team_id\"\
      ) || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_API=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $SMP_API_URL)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_LOGIN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_TOKEN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - aws configure set aws_access_key_id "$SMP_BOT_ID" --profile ${AWS_NAMED_PROFILE}
    - aws configure set aws_secret_access_key "$SMP_BOT_KEY" --profile ${AWS_NAMED_PROFILE}
    - aws configure set region us-west-2 --profile ${AWS_NAMED_PROFILE}
    - "aws --profile single-machine-performance s3 cp s3://smp-cli-releases/${SMP_VERSION}/x86_64-unknown-linux-musl/smp\
      \ smp || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-fetch-cli\"\n  exit $exit_code\n}\n"
    - chmod +x smp
    - source regression_detector.env
    - echo "Baseline SHA is ${BASELINE_SHA}"
    - echo -n "${BASELINE_SHA}" > "${CI_COMMIT_SHA}-baseline_sha"
    - aws s3 cp --profile single-machine-performance --only-show-errors "${CI_COMMIT_SHA}-baseline_sha"
      "s3://${SMP_TEAM_ID}-smp-artifacts/information/"
    - BASELINE_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${BASELINE_SHA}-7-full-amd64
    - echo "${BASELINE_SHA} | ${BASELINE_IMAGE}"
    - COMPARISON_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${CI_COMMIT_SHA}-7-full-amd64
    - echo "${CI_COMMIT_SHA} | ${COMPARISON_IMAGE}"
    - SMP_TAGS="ci_pipeline_id=${PARENT_PIPELINE_ID},ci_job_id=${CI_JOB_ID},ci_commit_branch=${CI_COMMIT_BRANCH},purpose=agent_ci"
    - echo "Tags passed through SMP are ${SMP_TAGS}"
    - RUST_LOG="info,aws_config::profile::credentials=error"
    - RUST_LOG_DEBUG="debug,aws_config::profile::credentials=error"
+   - if [ "$RUN_CONFIG_ONLY" == "true" ]; then
+   - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
+     \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit-experimental-config-only\
+     \ \\\n--target-image ${COMPARISON_IMAGE} \\\n--target-sha ${CI_COMMIT_SHA} \\\n\
+     --target-config-dir ${CONFIG_DIR} \\\n--submission-metadata submission_metadata\
+     \ \\\n--tags ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp job submit command\
+     \ failed with code $exit_code\"\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
+     job-submission\"\n  exit $exit_code\n} \n"
+   - else
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit \\\n--baseline-image\
      \ ${BASELINE_IMAGE} \\\n--comparison-image ${COMPARISON_IMAGE} \\\n--baseline-sha\
      \ ${BASELINE_SHA} \\\n--comparison-sha ${CI_COMMIT_SHA} \\\n--target-config-dir\
-     \ test/regression/ \\\n--submission-metadata submission_metadata \\\n--tags ${SMP_TAGS}\
?       ^^^^^^^^^^^^^^^^
+     \ ${CONFIG_DIR} \\\n--submission-metadata submission_metadata \\\n--tags ${SMP_TAGS}\
?       ^^^^^^^^^^^^^
      \ || {\n  exit_code=$?\n  echo \"smp job submit command failed with code $exit_code\"\
      \n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\n  exit\
-     \ $exit_code\n}\n"
+     \ $exit_code\n} \n"
?                    +
+   - fi
    - SMP_JOB_ID=$(jq -r '.jobId' submission_metadata)
    - echo "SMP Job Id is ${SMP_JOB_ID}"
    - datadog-ci tag --level job --tags smp_job_id:${SMP_JOB_ID}
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob status \\\n--wait \\\n--wait-delay-seconds\
      \ 60 \\\n--submission-metadata submission_metadata || {\n  exit_code=$?\n  echo\
      \ \"smp job status command failed with code $exit_code\"\n  datadog-ci tag --level\
      \ job --tags smp_failure_mode:\"job-status\"\n  exit $exit_code\n}\n"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob sync \\\n--submission-metadata\
      \ submission_metadata \\\n--output-path outputs || {\n  exit_code=$?\n  echo \"\
      smp job sync command failed with code $exit_code\"\n  datadog-ci tag --level job\
      \ --tags smp_failure_mode:\"job-sync\"\n  exit $exit_code\n}\n"
    - cat outputs/report.md | sed "s/^\$/$(echo -ne '\uFEFF\u00A0\u200B')/g"
    - datadog-ci junit upload --service datadog-agent outputs/junit.xml
    - datadog-ci tag --level job --tags smp_failure_mode:"none"
    - datadog-ci tag --level job --tags smp_optimization_goal:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal regression-detector || {\n  exit_code=$?\n  echo\
      \ \"smp regression detector has detected a regression\"\n  datadog-ci tag --level\
      \ job --tags smp_optimization_goal:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_bounds_check:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal bounds-check || {\n  exit_code=$?\n  echo \"smp\
      \ regression detector has detected a failed bounds check\"\n  datadog-ci tag --level\
      \ job --tags smp_bounds_check:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"failed"
    - "python3 <<'EOF'\nimport json\nimport sys\n\ntry:\n    with open('outputs/report.json')\
      \ as f:\n        data = json.load(f)\nexcept FileNotFoundError:\n    print(\"\
      Machine readable report not found.\")\n    sys.exit(1)\nexcept json.JSONDecodeError\
      \ as e:\n    print(f\"Error parsing JSON report: {e}\")\n    sys.exit(1)\n\nexperiments\
      \ = data.get('experiments', {})\nfailed = False\ndecision_record = []\n\nfor exp_name,\
      \ exp_data in experiments.items():\n    if exp_name.startswith('quality_gate_'):\n\
      \        bounds_checks = exp_data.get('bounds_checks', {})\n        for check_name,\
      \ check_data in bounds_checks.items():\n            results = check_data.get('results',\
      \ {})\n            comparison = results.get('comparison', [])\n            num_total\
      \ = len(comparison)\n            failed_replicates = [\n                replicate\
      \ for replicate in comparison if not replicate.get('passed', False)\n        \
      \    ]\n            num_failed = len(failed_replicates)\n            num_passed\
      \ = num_total - num_failed\n            if failed_replicates:\n              \
      \  decision_record.append(\n                    f\"- **{exp_name}**, bounds check\
      \ **{check_name}**: {num_passed}/{num_total} replicas passed. Failed {num_failed}\
      \ which is > 0. Gate **FAILED**.\"\n                )\n                failed\
      \ = True\n            else:\n                decision_record.append(\n       \
      \             f\"- **{exp_name}**, bounds check **{check_name}**: {num_passed}/{num_total}\
      \ replicas passed. Gate passed.\"\n                )\n\nwith open('outputs/decision_record.md',\
      \ 'w') as f:\n    # Extra newline since this is appended to another report\n \
      \   f.write('\\n\\n## CI Pass/Fail Decision\\n\\n')\n    if failed:\n        f.write('\u274C\
      \ **Failed.** Some Quality Gates were violated.\\n\\n')\n        f.write('\\n'.join(decision_record))\n\
      \    else:\n        f.write('\u2705 **Passed.** All Quality Gates passed.\\n\\\
      n')\n        f.write('\\n'.join(decision_record))\n\nif failed:\n    print(\"\
      Quality gate failed, see decision record\")\n    sys.exit(1)\nelse:\n    print(\"\
      Quality gate passed.\")\n    sys.exit(0)\nEOF\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"passed"
    stage: functional_test
    tags:
    - arch:amd64
    - specific:true
    timeout: 1h10m

single-machine-performance-metal-runners-regression_detector

  single-machine-performance-metal-runners-regression_detector:
    allow_failure: true
    artifacts:
      expire_in: 1 weeks
      paths:
      - submission_metadata
      - outputs/report.md
      - outputs/regression_signal.json
      - outputs/bounds_check_signal.json
      - outputs/junit.xml
      - outputs/report.json
      - outputs/decision_record.md
      when: always
    before_script:
    - echo "BASELINE_SHA=${BASELINE_SHA}" > regression_detector.env
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$CI_IMAGE_DOCKER_X64_SUFFIX:$CI_IMAGE_DOCKER_X64
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    script:
    - DATADOG_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DATADOG_API_KEY
    - datadog-ci tag --level job --tags smp_failure_mode:"timeout"
    - mkdir outputs
    - AWS_NAMED_PROFILE="single-machine-performance"
    - "SMP_ACCOUNT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT account_id)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - SMP_ECR_URL=${SMP_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com
    - "SMP_TEAM_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT \"${SMP_TEAM_NAME}_team_id\"\
      ) || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_API=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $SMP_API_URL)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_LOGIN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_TOKEN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - aws configure set aws_access_key_id "$SMP_BOT_ID" --profile ${AWS_NAMED_PROFILE}
    - aws configure set aws_secret_access_key "$SMP_BOT_KEY" --profile ${AWS_NAMED_PROFILE}
    - aws configure set region us-west-2 --profile ${AWS_NAMED_PROFILE}
    - "aws --profile single-machine-performance s3 cp s3://smp-cli-releases/${SMP_VERSION}/x86_64-unknown-linux-musl/smp\
      \ smp || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-fetch-cli\"\n  exit $exit_code\n}\n"
    - chmod +x smp
    - source regression_detector.env
    - echo "Baseline SHA is ${BASELINE_SHA}"
    - echo -n "${BASELINE_SHA}" > "${CI_COMMIT_SHA}-baseline_sha"
    - aws s3 cp --profile single-machine-performance --only-show-errors "${CI_COMMIT_SHA}-baseline_sha"
      "s3://${SMP_TEAM_ID}-smp-artifacts/information/"
    - BASELINE_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${BASELINE_SHA}-7-full-amd64
    - echo "${BASELINE_SHA} | ${BASELINE_IMAGE}"
    - COMPARISON_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${CI_COMMIT_SHA}-7-full-amd64
    - echo "${CI_COMMIT_SHA} | ${COMPARISON_IMAGE}"
    - SMP_TAGS="ci_pipeline_id=${PARENT_PIPELINE_ID},ci_job_id=${CI_JOB_ID},ci_commit_branch=${CI_COMMIT_BRANCH},purpose=agent_ci"
    - echo "Tags passed through SMP are ${SMP_TAGS}"
    - RUST_LOG="info,aws_config::profile::credentials=error"
    - RUST_LOG_DEBUG="debug,aws_config::profile::credentials=error"
+   - if [ "$RUN_CONFIG_ONLY" == "true" ]; then
+   - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
+     \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit-experimental-config-only\
+     \ \\\n--target-image ${COMPARISON_IMAGE} \\\n--target-sha ${CI_COMMIT_SHA} \\\n\
+     --target-config-dir ${CONFIG_DIR} \\\n--submission-metadata submission_metadata\
+     \ \\\n--tags ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp job submit command\
+     \ failed with code $exit_code\"\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
+     job-submission\"\n  exit $exit_code\n} \n"
+   - else
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit \\\n--baseline-image\
      \ ${BASELINE_IMAGE} \\\n--comparison-image ${COMPARISON_IMAGE} \\\n--baseline-sha\
      \ ${BASELINE_SHA} \\\n--comparison-sha ${CI_COMMIT_SHA} \\\n--target-config-dir\
-     \ test/regression/ \\\n--submission-metadata submission_metadata \\\n--tags ${SMP_TAGS}\
?       ^^^^^^^^^^^^^^^^
+     \ ${CONFIG_DIR} \\\n--submission-metadata submission_metadata \\\n--tags ${SMP_TAGS}\
?       ^^^^^^^^^^^^^
      \ || {\n  exit_code=$?\n  echo \"smp job submit command failed with code $exit_code\"\
      \n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\n  exit\
-     \ $exit_code\n}\n"
+     \ $exit_code\n} \n"
?                    +
+   - fi
    - SMP_JOB_ID=$(jq -r '.jobId' submission_metadata)
    - echo "SMP Job Id is ${SMP_JOB_ID}"
    - datadog-ci tag --level job --tags smp_job_id:${SMP_JOB_ID}
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob status \\\n--wait \\\n--wait-delay-seconds\
      \ 60 \\\n--submission-metadata submission_metadata || {\n  exit_code=$?\n  echo\
      \ \"smp job status command failed with code $exit_code\"\n  datadog-ci tag --level\
      \ job --tags smp_failure_mode:\"job-status\"\n  exit $exit_code\n}\n"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob sync \\\n--submission-metadata\
      \ submission_metadata \\\n--output-path outputs || {\n  exit_code=$?\n  echo \"\
      smp job sync command failed with code $exit_code\"\n  datadog-ci tag --level job\
      \ --tags smp_failure_mode:\"job-sync\"\n  exit $exit_code\n}\n"
    - cat outputs/report.md | sed "s/^\$/$(echo -ne '\uFEFF\u00A0\u200B')/g"
    - datadog-ci junit upload --service datadog-agent outputs/junit.xml
    - datadog-ci tag --level job --tags smp_failure_mode:"none"
    - datadog-ci tag --level job --tags smp_optimization_goal:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal regression-detector || {\n  exit_code=$?\n  echo\
      \ \"smp regression detector has detected a regression\"\n  datadog-ci tag --level\
      \ job --tags smp_optimization_goal:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_bounds_check:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal bounds-check || {\n  exit_code=$?\n  echo \"smp\
      \ regression detector has detected a failed bounds check\"\n  datadog-ci tag --level\
      \ job --tags smp_bounds_check:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"failed"
    - "python3 <<'EOF'\nimport json\nimport sys\n\ntry:\n    with open('outputs/report.json')\
      \ as f:\n        data = json.load(f)\nexcept FileNotFoundError:\n    print(\"\
      Machine readable report not found.\")\n    sys.exit(1)\nexcept json.JSONDecodeError\
      \ as e:\n    print(f\"Error parsing JSON report: {e}\")\n    sys.exit(1)\n\nexperiments\
      \ = data.get('experiments', {})\nfailed = False\ndecision_record = []\n\nfor exp_name,\
      \ exp_data in experiments.items():\n    if exp_name.startswith('quality_gate_'):\n\
      \        bounds_checks = exp_data.get('bounds_checks', {})\n        for check_name,\
      \ check_data in bounds_checks.items():\n            results = check_data.get('results',\
      \ {})\n            comparison = results.get('comparison', [])\n            num_total\
      \ = len(comparison)\n            failed_replicates = [\n                replicate\
      \ for replicate in comparison if not replicate.get('passed', False)\n        \
      \    ]\n            num_failed = len(failed_replicates)\n            num_passed\
      \ = num_total - num_failed\n            if failed_replicates:\n              \
      \  decision_record.append(\n                    f\"- **{exp_name}**, bounds check\
      \ **{check_name}**: {num_passed}/{num_total} replicas passed. Failed {num_failed}\
      \ which is > 0. Gate **FAILED**.\"\n                )\n                failed\
      \ = True\n            else:\n                decision_record.append(\n       \
      \             f\"- **{exp_name}**, bounds check **{check_name}**: {num_passed}/{num_total}\
      \ replicas passed. Gate passed.\"\n                )\n\nwith open('outputs/decision_record.md',\
      \ 'w') as f:\n    # Extra newline since this is appended to another report\n \
      \   f.write('\\n\\n## CI Pass/Fail Decision\\n\\n')\n    if failed:\n        f.write('\u274C\
      \ **Failed.** Some Quality Gates were violated.\\n\\n')\n        f.write('\\n'.join(decision_record))\n\
      \    else:\n        f.write('\u2705 **Passed.** All Quality Gates passed.\\n\\\
      n')\n        f.write('\\n'.join(decision_record))\n\nif failed:\n    print(\"\
      Quality gate failed, see decision record\")\n    sys.exit(1)\nelse:\n    print(\"\
      Quality gate passed.\")\n    sys.exit(0)\nEOF\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"passed"
    stage: functional_test
    tags:
    - arch:amd64
    - specific:true
    timeout: 1h10m
    variables:
      BOT_LOGIN: bot_login_metal-runners
      BOT_TOKEN: bot_token_metal-runners
+     CONFIG_DIR: test/regression/ebpf
      SMP_API_URL: metal-runners_api_url
      SMP_TEAM_NAME: metal-runners
-     SMP_VERSION: dev-pr3796-e7f7a988
+     SMP_VERSION: v0.27.0-nightly.20260423T145046Z
    when: manual

single-machine-performance-regression_detector

  single-machine-performance-regression_detector:
    allow_failure: true
    artifacts:
      expire_in: 1 weeks
      paths:
      - submission_metadata
      - outputs/report.md
      - outputs/regression_signal.json
      - outputs/bounds_check_signal.json
      - outputs/junit.xml
      - outputs/report.json
      - outputs/decision_record.md
      when: always
    before_script:
    - echo "BASELINE_SHA=${BASELINE_SHA}" > regression_detector.env
    image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$CI_IMAGE_DOCKER_X64_SUFFIX:$CI_IMAGE_DOCKER_X64
    retry:
      exit_codes:
      - 42
      max: 2
      when:
      - runner_system_failure
      - stuck_or_timeout_failure
      - unknown_failure
      - api_failure
      - scheduler_failure
      - stale_schedule
      - data_integrity_failure
    script:
    - DATADOG_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
      token)" || exit $?; export DATADOG_API_KEY
    - datadog-ci tag --level job --tags smp_failure_mode:"timeout"
    - mkdir outputs
    - AWS_NAMED_PROFILE="single-machine-performance"
    - "SMP_ACCOUNT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT account_id)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - SMP_ECR_URL=${SMP_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com
    - "SMP_TEAM_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT \"${SMP_TEAM_NAME}_team_id\"\
      ) || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_API=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $SMP_API_URL)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_LOGIN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - "SMP_BOT_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_TOKEN)\
      \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-secrets\"\n  exit $exit_code\n}\n"
    - aws configure set aws_access_key_id "$SMP_BOT_ID" --profile ${AWS_NAMED_PROFILE}
    - aws configure set aws_secret_access_key "$SMP_BOT_KEY" --profile ${AWS_NAMED_PROFILE}
    - aws configure set region us-west-2 --profile ${AWS_NAMED_PROFILE}
    - "aws --profile single-machine-performance s3 cp s3://smp-cli-releases/${SMP_VERSION}/x86_64-unknown-linux-musl/smp\
      \ smp || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
      setup-fetch-cli\"\n  exit $exit_code\n}\n"
    - chmod +x smp
    - source regression_detector.env
    - echo "Baseline SHA is ${BASELINE_SHA}"
    - echo -n "${BASELINE_SHA}" > "${CI_COMMIT_SHA}-baseline_sha"
    - aws s3 cp --profile single-machine-performance --only-show-errors "${CI_COMMIT_SHA}-baseline_sha"
      "s3://${SMP_TEAM_ID}-smp-artifacts/information/"
    - BASELINE_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${BASELINE_SHA}-7-full-amd64
    - echo "${BASELINE_SHA} | ${BASELINE_IMAGE}"
    - COMPARISON_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${CI_COMMIT_SHA}-7-full-amd64
    - echo "${CI_COMMIT_SHA} | ${COMPARISON_IMAGE}"
    - SMP_TAGS="ci_pipeline_id=${PARENT_PIPELINE_ID},ci_job_id=${CI_JOB_ID},ci_commit_branch=${CI_COMMIT_BRANCH},purpose=agent_ci"
    - echo "Tags passed through SMP are ${SMP_TAGS}"
    - RUST_LOG="info,aws_config::profile::credentials=error"
    - RUST_LOG_DEBUG="debug,aws_config::profile::credentials=error"
+   - if [ "$RUN_CONFIG_ONLY" == "true" ]; then
+   - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
+     \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit-experimental-config-only\
+     \ \\\n--target-image ${COMPARISON_IMAGE} \\\n--target-sha ${CI_COMMIT_SHA} \\\n\
+     --target-config-dir ${CONFIG_DIR} \\\n--submission-metadata submission_metadata\
+     \ \\\n--tags ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp job submit command\
+     \ failed with code $exit_code\"\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
+     job-submission\"\n  exit $exit_code\n} \n"
+   - else
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit \\\n--baseline-image\
      \ ${BASELINE_IMAGE} \\\n--comparison-image ${COMPARISON_IMAGE} \\\n--baseline-sha\
      \ ${BASELINE_SHA} \\\n--comparison-sha ${CI_COMMIT_SHA} \\\n--target-config-dir\
-     \ test/regression/ \\\n--submission-metadata submission_metadata \\\n--tags ${SMP_TAGS}\
?       ^^^^^^^^^^^^^^^^
+     \ ${CONFIG_DIR} \\\n--submission-metadata submission_metadata \\\n--tags ${SMP_TAGS}\
?       ^^^^^^^^^^^^^
      \ || {\n  exit_code=$?\n  echo \"smp job submit command failed with code $exit_code\"\
      \n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\n  exit\
-     \ $exit_code\n}\n"
+     \ $exit_code\n} \n"
?                    +
+   - fi
    - SMP_JOB_ID=$(jq -r '.jobId' submission_metadata)
    - echo "SMP Job Id is ${SMP_JOB_ID}"
    - datadog-ci tag --level job --tags smp_job_id:${SMP_JOB_ID}
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob status \\\n--wait \\\n--wait-delay-seconds\
      \ 60 \\\n--submission-metadata submission_metadata || {\n  exit_code=$?\n  echo\
      \ \"smp job status command failed with code $exit_code\"\n  datadog-ci tag --level\
      \ job --tags smp_failure_mode:\"job-status\"\n  exit $exit_code\n}\n"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob sync \\\n--submission-metadata\
      \ submission_metadata \\\n--output-path outputs || {\n  exit_code=$?\n  echo \"\
      smp job sync command failed with code $exit_code\"\n  datadog-ci tag --level job\
      \ --tags smp_failure_mode:\"job-sync\"\n  exit $exit_code\n}\n"
    - cat outputs/report.md | sed "s/^\$/$(echo -ne '\uFEFF\u00A0\u200B')/g"
    - datadog-ci junit upload --service datadog-agent outputs/junit.xml
    - datadog-ci tag --level job --tags smp_failure_mode:"none"
    - datadog-ci tag --level job --tags smp_optimization_goal:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal regression-detector || {\n  exit_code=$?\n  echo\
      \ \"smp regression detector has detected a regression\"\n  datadog-ci tag --level\
      \ job --tags smp_optimization_goal:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_bounds_check:"passed"
    - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
      \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
      \ submission_metadata --signal bounds-check || {\n  exit_code=$?\n  echo \"smp\
      \ regression detector has detected a failed bounds check\"\n  datadog-ci tag --level\
      \ job --tags smp_bounds_check:\"failed\"\n}\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"failed"
    - "python3 <<'EOF'\nimport json\nimport sys\n\ntry:\n    with open('outputs/report.json')\
      \ as f:\n        data = json.load(f)\nexcept FileNotFoundError:\n    print(\"\
      Machine readable report not found.\")\n    sys.exit(1)\nexcept json.JSONDecodeError\
      \ as e:\n    print(f\"Error parsing JSON report: {e}\")\n    sys.exit(1)\n\nexperiments\
      \ = data.get('experiments', {})\nfailed = False\ndecision_record = []\n\nfor exp_name,\
      \ exp_data in experiments.items():\n    if exp_name.startswith('quality_gate_'):\n\
      \        bounds_checks = exp_data.get('bounds_checks', {})\n        for check_name,\
      \ check_data in bounds_checks.items():\n            results = check_data.get('results',\
      \ {})\n            comparison = results.get('comparison', [])\n            num_total\
      \ = len(comparison)\n            failed_replicates = [\n                replicate\
      \ for replicate in comparison if not replicate.get('passed', False)\n        \
      \    ]\n            num_failed = len(failed_replicates)\n            num_passed\
      \ = num_total - num_failed\n            if failed_replicates:\n              \
      \  decision_record.append(\n                    f\"- **{exp_name}**, bounds check\
      \ **{check_name}**: {num_passed}/{num_total} replicas passed. Failed {num_failed}\
      \ which is > 0. Gate **FAILED**.\"\n                )\n                failed\
      \ = True\n            else:\n                decision_record.append(\n       \
      \             f\"- **{exp_name}**, bounds check **{check_name}**: {num_passed}/{num_total}\
      \ replicas passed. Gate passed.\"\n                )\n\nwith open('outputs/decision_record.md',\
      \ 'w') as f:\n    # Extra newline since this is appended to another report\n \
      \   f.write('\\n\\n## CI Pass/Fail Decision\\n\\n')\n    if failed:\n        f.write('\u274C\
      \ **Failed.** Some Quality Gates were violated.\\n\\n')\n        f.write('\\n'.join(decision_record))\n\
      \    else:\n        f.write('\u2705 **Passed.** All Quality Gates passed.\\n\\\
      n')\n        f.write('\\n'.join(decision_record))\n\nif failed:\n    print(\"\
      Quality gate failed, see decision record\")\n    sys.exit(1)\nelse:\n    print(\"\
      Quality gate passed.\")\n    sys.exit(0)\nEOF\n"
    - datadog-ci tag --level job --tags smp_quality_gates:"passed"
    stage: functional_test
    tags:
    - arch:amd64
    - specific:true
    timeout: 1h10m
    variables:
      BOT_LOGIN: bot_login
      BOT_TOKEN: bot_token
+     CONFIG_DIR: test/regression
      SMP_API_URL: api_url
      SMP_TEAM_NAME: agent
      SMP_VERSION: v0.25.1

Added Jobs

.smp_config_only_job_submit

.smp_config_only_job_submit:
- "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API} --aws-named-profile\
  \ ${AWS_NAMED_PROFILE} \\\njob submit-experimental-config-only \\\n--target-image\
  \ ${COMPARISON_IMAGE} \\\n--target-sha ${CI_COMMIT_SHA} \\\n--target-config-dir\
  \ ${CONFIG_DIR} \\\n--submission-metadata submission_metadata \\\n--tags ${SMP_TAGS}\
  \ || {\n  exit_code=$?\n  echo \"smp job submit command failed with code $exit_code\"\
  \n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\n  exit\
  \ $exit_code\n} \n"

.smp_job_submit

.smp_job_submit:
- "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API} --aws-named-profile\
  \ ${AWS_NAMED_PROFILE} \\\njob submit \\\n--baseline-image ${BASELINE_IMAGE} \\\n\
  --comparison-image ${COMPARISON_IMAGE} \\\n--baseline-sha ${BASELINE_SHA} \\\n--comparison-sha\
  \ ${CI_COMMIT_SHA} \\\n--target-config-dir ${CONFIG_DIR} \\\n--submission-metadata\
  \ submission_metadata \\\n--tags ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp\
  \ job submit command failed with code $exit_code\"\n  datadog-ci tag --level job\
  \ --tags smp_failure_mode:\"job-submission\"\n  exit $exit_code\n} \n"

single-machine-performance-metal-runners-regression_detector_cfg

single-machine-performance-metal-runners-regression_detector_cfg:
  allow_failure: true
  artifacts:
    expire_in: 1 weeks
    paths:
    - submission_metadata
    - outputs/report.md
    - outputs/regression_signal.json
    - outputs/bounds_check_signal.json
    - outputs/junit.xml
    - outputs/report.json
    - outputs/decision_record.md
    when: always
  before_script:
  - echo "BASELINE_SHA=${BASELINE_SHA}" > regression_detector.env
  image: registry.ddbuild.io/ci/datadog-agent-buildimages/docker_x64$CI_IMAGE_DOCKER_X64_SUFFIX:$CI_IMAGE_DOCKER_X64
  retry:
    exit_codes:
    - 42
    max: 2
    when:
    - runner_system_failure
    - stuck_or_timeout_failure
    - unknown_failure
    - api_failure
    - scheduler_failure
    - stale_schedule
    - data_integrity_failure
  script:
  - DATADOG_API_KEY="$("$CI_PROJECT_DIR"/tools/ci/fetch_secret.sh "$AGENT_API_KEY_ORG2"
    token)" || exit $?; export DATADOG_API_KEY
  - datadog-ci tag --level job --tags smp_failure_mode:"timeout"
  - mkdir outputs
  - AWS_NAMED_PROFILE="single-machine-performance"
  - "SMP_ACCOUNT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT account_id)\
    \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
    setup-secrets\"\n  exit $exit_code\n}\n"
  - SMP_ECR_URL=${SMP_ACCOUNT_ID}.dkr.ecr.us-west-2.amazonaws.com
  - "SMP_TEAM_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT \"${SMP_TEAM_NAME}_team_id\"\
    ) || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
    setup-secrets\"\n  exit $exit_code\n}\n"
  - "SMP_API=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $SMP_API_URL)\
    \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
    setup-secrets\"\n  exit $exit_code\n}\n"
  - "SMP_BOT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_LOGIN)\
    \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
    setup-secrets\"\n  exit $exit_code\n}\n"
  - "SMP_BOT_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SMP_ACCOUNT $BOT_TOKEN)\
    \ || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
    setup-secrets\"\n  exit $exit_code\n}\n"
  - aws configure set aws_access_key_id "$SMP_BOT_ID" --profile ${AWS_NAMED_PROFILE}
  - aws configure set aws_secret_access_key "$SMP_BOT_KEY" --profile ${AWS_NAMED_PROFILE}
  - aws configure set region us-west-2 --profile ${AWS_NAMED_PROFILE}
  - "aws --profile single-machine-performance s3 cp s3://smp-cli-releases/${SMP_VERSION}/x86_64-unknown-linux-musl/smp\
    \ smp || {\n  exit_code=$?\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
    setup-fetch-cli\"\n  exit $exit_code\n}\n"
  - chmod +x smp
  - source regression_detector.env
  - echo "Baseline SHA is ${BASELINE_SHA}"
  - echo -n "${BASELINE_SHA}" > "${CI_COMMIT_SHA}-baseline_sha"
  - aws s3 cp --profile single-machine-performance --only-show-errors "${CI_COMMIT_SHA}-baseline_sha"
    "s3://${SMP_TEAM_ID}-smp-artifacts/information/"
  - BASELINE_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${BASELINE_SHA}-7-full-amd64
  - echo "${BASELINE_SHA} | ${BASELINE_IMAGE}"
  - COMPARISON_IMAGE=${SMP_ECR_URL}/${SMP_TEAM_ID}-agent:${CI_COMMIT_SHA}-7-full-amd64
  - echo "${CI_COMMIT_SHA} | ${COMPARISON_IMAGE}"
  - SMP_TAGS="ci_pipeline_id=${PARENT_PIPELINE_ID},ci_job_id=${CI_JOB_ID},ci_commit_branch=${CI_COMMIT_BRANCH},purpose=agent_ci"
  - echo "Tags passed through SMP are ${SMP_TAGS}"
  - RUST_LOG="info,aws_config::profile::credentials=error"
  - RUST_LOG_DEBUG="debug,aws_config::profile::credentials=error"
  - if [ "$RUN_CONFIG_ONLY" == "true" ]; then
  - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
    \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit-experimental-config-only\
    \ \\\n--target-image ${COMPARISON_IMAGE} \\\n--target-sha ${CI_COMMIT_SHA} \\\n\
    --target-config-dir ${CONFIG_DIR} \\\n--submission-metadata submission_metadata\
    \ \\\n--tags ${SMP_TAGS} || {\n  exit_code=$?\n  echo \"smp job submit command\
    \ failed with code $exit_code\"\n  datadog-ci tag --level job --tags smp_failure_mode:\"\
    job-submission\"\n  exit $exit_code\n} \n"
  - else
  - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
    \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob submit \\\n--baseline-image\
    \ ${BASELINE_IMAGE} \\\n--comparison-image ${COMPARISON_IMAGE} \\\n--baseline-sha\
    \ ${BASELINE_SHA} \\\n--comparison-sha ${CI_COMMIT_SHA} \\\n--target-config-dir\
    \ ${CONFIG_DIR} \\\n--submission-metadata submission_metadata \\\n--tags ${SMP_TAGS}\
    \ || {\n  exit_code=$?\n  echo \"smp job submit command failed with code $exit_code\"\
    \n  datadog-ci tag --level job --tags smp_failure_mode:\"job-submission\"\n  exit\
    \ $exit_code\n} \n"
  - fi
  - SMP_JOB_ID=$(jq -r '.jobId' submission_metadata)
  - echo "SMP Job Id is ${SMP_JOB_ID}"
  - datadog-ci tag --level job --tags smp_job_id:${SMP_JOB_ID}
  - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
    \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob status \\\n--wait \\\n--wait-delay-seconds\
    \ 60 \\\n--submission-metadata submission_metadata || {\n  exit_code=$?\n  echo\
    \ \"smp job status command failed with code $exit_code\"\n  datadog-ci tag --level\
    \ job --tags smp_failure_mode:\"job-status\"\n  exit $exit_code\n}\n"
  - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
    \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\njob sync \\\n--submission-metadata\
    \ submission_metadata \\\n--output-path outputs || {\n  exit_code=$?\n  echo \"\
    smp job sync command failed with code $exit_code\"\n  datadog-ci tag --level job\
    \ --tags smp_failure_mode:\"job-sync\"\n  exit $exit_code\n}\n"
  - cat outputs/report.md | sed "s/^\$/$(echo -ne '\uFEFF\u00A0\u200B')/g"
  - datadog-ci junit upload --service datadog-agent outputs/junit.xml
  - datadog-ci tag --level job --tags smp_failure_mode:"none"
  - datadog-ci tag --level job --tags smp_optimization_goal:"passed"
  - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
    \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
    \ submission_metadata --signal regression-detector || {\n  exit_code=$?\n  echo\
    \ \"smp regression detector has detected a regression\"\n  datadog-ci tag --level\
    \ job --tags smp_optimization_goal:\"failed\"\n}\n"
  - datadog-ci tag --level job --tags smp_bounds_check:"passed"
  - "RUST_LOG=\"${RUST_LOG}\" ./smp --team-id ${SMP_TEAM_ID} --api-base ${SMP_API}\
    \ --aws-named-profile ${AWS_NAMED_PROFILE} \\\n  job result \\\n  --submission-metadata\
    \ submission_metadata --signal bounds-check || {\n  exit_code=$?\n  echo \"smp\
    \ regression detector has detected a failed bounds check\"\n  datadog-ci tag --level\
    \ job --tags smp_bounds_check:\"failed\"\n}\n"
  - datadog-ci tag --level job --tags smp_quality_gates:"failed"
  - "python3 <<'EOF'\nimport json\nimport sys\n\ntry:\n    with open('outputs/report.json')\
    \ as f:\n        data = json.load(f)\nexcept FileNotFoundError:\n    print(\"\
    Machine readable report not found.\")\n    sys.exit(1)\nexcept json.JSONDecodeError\
    \ as e:\n    print(f\"Error parsing JSON report: {e}\")\n    sys.exit(1)\n\nexperiments\
    \ = data.get('experiments', {})\nfailed = False\ndecision_record = []\n\nfor exp_name,\
    \ exp_data in experiments.items():\n    if exp_name.startswith('quality_gate_'):\n\
    \        bounds_checks = exp_data.get('bounds_checks', {})\n        for check_name,\
    \ check_data in bounds_checks.items():\n            results = check_data.get('results',\
    \ {})\n            comparison = results.get('comparison', [])\n            num_total\
    \ = len(comparison)\n            failed_replicates = [\n                replicate\
    \ for replicate in comparison if not replicate.get('passed', False)\n        \
    \    ]\n            num_failed = len(failed_replicates)\n            num_passed\
    \ = num_total - num_failed\n            if failed_replicates:\n              \
    \  decision_record.append(\n                    f\"- **{exp_name}**, bounds check\
    \ **{check_name}**: {num_passed}/{num_total} replicas passed. Failed {num_failed}\
    \ which is > 0. Gate **FAILED**.\"\n                )\n                failed\
    \ = True\n            else:\n                decision_record.append(\n       \
    \             f\"- **{exp_name}**, bounds check **{check_name}**: {num_passed}/{num_total}\
    \ replicas passed. Gate passed.\"\n                )\n\nwith open('outputs/decision_record.md',\
    \ 'w') as f:\n    # Extra newline since this is appended to another report\n \
    \   f.write('\\n\\n## CI Pass/Fail Decision\\n\\n')\n    if failed:\n        f.write('\u274C\
    \ **Failed.** Some Quality Gates were violated.\\n\\n')\n        f.write('\\n'.join(decision_record))\n\
    \    else:\n        f.write('\u2705 **Passed.** All Quality Gates passed.\\n\\\
    n')\n        f.write('\\n'.join(decision_record))\n\nif failed:\n    print(\"\
    Quality gate failed, see decision record\")\n    sys.exit(1)\nelse:\n    print(\"\
    Quality gate passed.\")\n    sys.exit(0)\nEOF\n"
  - datadog-ci tag --level job --tags smp_quality_gates:"passed"
  stage: functional_test
  tags:
  - arch:amd64
  - specific:true
  timeout: 1h10m
  variables:
    BOT_LOGIN: bot_login_metal-runners
    BOT_TOKEN: bot_token_metal-runners
    CONFIG_DIR: test/regression/ebpf/config-only
    RUN_CONFIG_ONLY: 'true'
    SMP_API_URL: metal-runners_api_url
    SMP_TEAM_NAME: metal-runners
    SMP_VERSION: v0.27.0-nightly.20260423T145046Z
  when: manual

Changes Summary

Removed	Modified	Added	Renamed
0	3	3	0

ℹ️ Diff available in the job log.

dd-octo-sts · 2026-04-24T10:18:34Z

Files inventory check summary

File checks results against ancestor 33a5258b:

Results for datadog-agent_7.80.0~devel.git.200.a85d39b.pipeline.109897344-1_amd64.deb:

No change detected

cit-pr-commenter-54b7da · 2026-04-24T11:12:09Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: cf81a761-9f28-4f02-b00f-07d0cd81cc43

Baseline: 33a5258
Comparison: e94cf4e
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+2.33	[-0.74, +5.40]	1	Logs

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+2.33	[-0.74, +5.40]	1	Logs
➖	quality_gate_metrics_logs	memory utilization	+0.45	[+0.19, +0.72]	1	Logs bounds checks dashboard
➖	tcp_syslog_to_blackhole	ingress throughput	+0.27	[+0.11, +0.43]	1	Logs
➖	ddot_metrics_sum_delta	memory utilization	+0.22	[+0.02, +0.42]	1	Logs
➖	ddot_logs	memory utilization	+0.21	[+0.16, +0.27]	1	Logs
➖	otlp_ingest_logs	memory utilization	+0.14	[+0.03, +0.24]	1	Logs
➖	ddot_metrics	memory utilization	+0.09	[-0.11, +0.29]	1	Logs
➖	quality_gate_idle	memory utilization	+0.06	[+0.01, +0.11]	1	Logs bounds checks dashboard
➖	file_to_blackhole_500ms_latency	egress throughput	+0.01	[-0.40, +0.41]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	+0.00	[-0.20, +0.21]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.00	[-0.11, +0.11]	1	Logs
➖	uds_dogstatsd_to_api_v3	ingress throughput	-0.00	[-0.21, +0.21]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	-0.02	[-0.45, +0.42]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	-0.04	[-0.15, +0.07]	1	Logs
➖	docker_containers_memory	memory utilization	-0.08	[-0.18, +0.02]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	-0.10	[-0.60, +0.41]	1	Logs
➖	quality_gate_logs	% cpu utilization	-0.14	[-1.70, +1.43]	1	Logs bounds checks dashboard
➖	quality_gate_idle_all_features	memory utilization	-0.14	[-0.17, -0.10]	1	Logs bounds checks dashboard
➖	ddot_metrics_sum_cumulativetodelta_exporter	memory utilization	-0.16	[-0.40, +0.08]	1	Logs
➖	uds_dogstatsd_20mb_12k_contexts_20_senders	memory utilization	-0.22	[-0.27, -0.17]	1	Logs
➖	otlp_ingest_metrics	memory utilization	-0.38	[-0.53, -0.22]	1	Logs
➖	file_tree	memory utilization	-0.46	[-0.50, -0.42]	1	Logs
➖	ddot_metrics_sum_cumulative	memory utilization	-0.51	[-0.67, -0.35]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	observed_value	links
✅	docker_containers_cpu	simple_check_run	10/10	716 ≥ 26
✅	docker_containers_memory	memory_usage	10/10	242.33MiB ≤ 370MiB
✅	docker_containers_memory	simple_check_run	10/10	720 ≥ 26
✅	file_to_blackhole_0ms_latency	memory_usage	10/10	0.16GiB ≤ 1.20GiB
✅	file_to_blackhole_0ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10	0.20GiB ≤ 1.20GiB
✅	file_to_blackhole_1000ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_100ms_latency	memory_usage	10/10	0.16GiB ≤ 1.20GiB
✅	file_to_blackhole_100ms_latency	missed_bytes	10/10	0B = 0B
✅	file_to_blackhole_500ms_latency	memory_usage	10/10	0.18GiB ≤ 1.20GiB
✅	file_to_blackhole_500ms_latency	missed_bytes	10/10	0B = 0B
✅	quality_gate_idle	intake_connections	10/10	3 ≤ 4	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	137.76MiB ≤ 147MiB	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	3 ≤ 4	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	461.61MiB ≤ 495MiB	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	4 ≤ 6	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	176.94MiB ≤ 195MiB	bounds checks dashboard
✅	quality_gate_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard
✅	quality_gate_metrics_logs	cpu_usage	10/10	347.55 ≤ 2000	bounds checks dashboard
✅	quality_gate_metrics_logs	intake_connections	10/10	3 ≤ 6	bounds checks dashboard
✅	quality_gate_metrics_logs	memory_usage	10/10	370.86MiB ≤ 430MiB	bounds checks dashboard
✅	quality_gate_metrics_logs	missed_bytes	10/10	0B = 0B	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.

use latest nightly to test

f94e1d3

dd-octo-sts Bot added internal Identify a non-fork PR team/agent-devx labels Apr 24, 2026

github-actions Bot added the short review PR is simple enough to be reviewed quickly label Apr 24, 2026

usamasaqib added 2 commits April 24, 2026 12:25

add ebpf experiments

1c17cfe

allow selecting config-only experiments

6bb1537

github-actions Bot added long review PR is complex, plan time to review it and removed short review PR is simple enough to be reviewed quickly labels Apr 24, 2026

usamasaqib added 3 commits April 24, 2026 22:01

use CONFIG_DIR for normal job sumbission as well

67d2e50

enable ebpf-check and collect system-probe telemetry

e94cf4e

set optimization goal for ebpf experiments to none

a85d39b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use latest nightly to test#49869

use latest nightly to test#49869
usamasaqib wants to merge 6 commits intomainfrom
usama.saqib/wib

usamasaqib commented Apr 24, 2026

Uh oh!

dd-octo-sts Bot commented Apr 24, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Apr 24, 2026 •

edited

Loading

Uh oh!

cit-pr-commenter-54b7da Bot commented Apr 24, 2026 •

edited

Loading

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

usamasaqib commented Apr 24, 2026

What does this PR do?

Motivation

Describe how you validated your changes

Additional Notes

Uh oh!

dd-octo-sts Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Gitlab CI Configuration Changes

Modified Jobs

Added Jobs

Changes Summary

Uh oh!

dd-octo-sts Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Files inventory check summary

Results for datadog-agent_7.80.0~devel.git.200.a85d39b.pipeline.109897344-1_amd64.deb:

Uh oh!

cit-pr-commenter-54b7da Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CI Pass/Fail Decision

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

dd-octo-sts Bot commented Apr 24, 2026 •

edited

Loading

dd-octo-sts Bot commented Apr 24, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Apr 24, 2026 •

edited

Loading