Reports regzbot tracking (#1925)

* reports: drop leading blank lines from report emails

base.txt opened with a blank line and an empty header block, so every report rendered with several blank lines before the first content line. Trim the top of base.txt so reports start at their first line, and refresh the metrics example fixture to match.

Signed-off-by: Ben Copeland <ben.copeland@linaro.org>

* notifications: find last good build/test commit for regzbot range

Add kcidb_last_build_without_issue() to fetch the most recent passing build of the same config (filtered by start_time) that does not carry the issue, and set last_pass_commit on build incidents. For standalone test reports, derive the last passing commit from the status history (passing the required group_size). This lets the mails emit a real last_good..HEAD introduced range.

Signed-off-by: Ben Copeland <ben.copeland@linaro.org>

* reports: reorder regression mails and add regzbot tags

Rework the regression report layout so the details humans need come
first and the machine-readable tags come last: drop the "Hello,"
greeting, move the log excerpt up next to the issue description, and
list the dashboard details below the attribution with a lead-in
explaining where the full logs live.

Replace the tag request with the patch submission rules wording, and
generate the Message-ID before rendering so each report can carry a
Link: to its own thread on lore.kernel.org next to the Reported-by tag.

Add a regzbot tracking block (introduced/link) in its own
blank-line-delimited paragraph at the very end of the body together
with the #kernelci marker. The title command is left out so regzbot
takes the mail subject, and the block is only emitted when a last known
good commit is available, so regzbot is never handed a single
unbisected commit it cannot act on.

Signed-off-by: Ben Copeland <ben.copeland@linaro.org>

* notifications: drop the new-issues report Cc

The regressions list is already added in code for mainline/next, so the explicit Cc is not needed.

Signed-off-by: Ben Copeland <ben.copeland@linaro.org>

* tests: add golden-file tests for the report templates

Render the issue and test report templates with fixed data and compare
the output against committed example files, so template changes show
up as reviewable diffs. Add a refresh_report_examples management
command to regenerate the fixtures when a template change is
intentional.

Signed-off-by: Ben Copeland <ben.copeland@linaro.org>

---------

Signed-off-by: Ben Copeland <ben.copeland@linaro.org>

Author: bhcopeland

backend/kernelCI/settings.py

@@ -175,7 +175,6 @@ def get_json_env_var(name, default):
                 "--monitoring-id=notifications_new_issues",
                 "--action=new_issues",
                 "--to=kernelci-results@groups.io",
-                "--cc=gus@collabora.com",
                 "--send",
                 "--yes",
             ],

backend/kernelCI_app/helpers/email.py

@@ -11,7 +11,15 @@ def smtp_setup_connection():
 
 
 def smtp_send_email(
-    connection, sender_email, to, subject, message_text, cc, reply_to, in_reply_to=None
+    connection,
+    sender_email,
+    to,
+    subject,
+    message_text,
+    cc,
+    reply_to,
+    in_reply_to=None,
+    message_id=None,
 ):
     """Send an email using Django's SMTP backend.
 
@@ -24,6 +32,9 @@ def smtp_send_email(
         cc: CC recipients
         reply_to: Reply-to address
         in_reply_to: Message ID to set in In-Reply-To header (optional)
+        message_id: Message ID to use for the email, generated when not given.
+          Pass one when the message body references its own Message-ID
+          (e.g. a lore.kernel.org link).
 
     Returns:
         Message ID of the sent email
@@ -45,7 +56,8 @@ def smtp_send_email(
         )
 
         # Generate a unique message ID for tracking
-        message_id = make_msgid()
+        if not message_id:
+            message_id = make_msgid()
         email.extra_headers["Message-ID"] = message_id
 
         if in_reply_to:

backend/kernelCI_app/management/commands/helpers/common.py

@@ -91,6 +91,7 @@ def send_email_report(
     git_url: Optional[str] = None,
     signup_folder: Optional[str] = None,
     recipients: Optional[list[str]] = None,
+    message_id: Optional[str] = None,
 ):
     """Sets up the email arguments and sends the report.
 
@@ -105,6 +106,9 @@ def send_email_report(
           Is only used if git_url is set and default recipients are not ignored.
         recipients: a direct list of email recipients to use,
           this parameter overrides the use of git_url + signup_folder.
+        message_id: the Message-ID to send the email with, generated by the
+          smtp helper when not given. Pass one when the report body references
+          its own Message-ID (e.g. a lore.kernel.org link).
     """
     sender_email = "KernelCI bot <bot@kernelci.org>"
     subject = report["title"]
@@ -167,4 +171,5 @@ def send_email_report(
         cc,
         reply_to,
         email_args.in_reply_to,
+        message_id,
     )

backend/kernelCI_app/management/commands/notifications.py

@@ -2,6 +2,7 @@
 import sys
 from collections import defaultdict
 from datetime import datetime, timedelta, timezone
+from email.utils import make_msgid
 from types import SimpleNamespace
 from typing import Optional
 from urllib.parse import quote_plus
@@ -39,6 +40,7 @@
     get_metrics_data,
     kcidb_build_incidents,
     kcidb_issue_details,
+    kcidb_last_build_without_issue,
     kcidb_last_test_without_issue,
     kcidb_new_issues,
     kcidb_test_incidents,
@@ -153,10 +155,14 @@ def ask_ignore_issue():
             print("Please enter 'y' or 'n'.")
 
 
-def generate_build_issue_report(issue, incidents):
+def generate_build_issue_report(issue, incidents, message_id=None):
     template = setup_jinja_template("issue_build.txt.j2")
     report = {}
-    report["content"] = template.render(issue=issue, builds=incidents)
+    report["content"] = template.render(
+        issue=issue,
+        builds=incidents,
+        message_id=message_id.strip("<>") if message_id else None,
+    )
     snippet = (
         issue["comment"]
         if len(issue["comment"]) <= 70
@@ -168,10 +174,14 @@ def generate_build_issue_report(issue, incidents):
     return report
 
 
-def generate_boot_issue_report(issue, incidents):
+def generate_boot_issue_report(issue, incidents, message_id=None):
     template = setup_jinja_template("issue_boot.txt.j2")
     report = {}
-    report["content"] = template.render(issue=issue, boots=incidents)
+    report["content"] = template.render(
+        issue=issue,
+        boots=incidents,
+        message_id=message_id.strip("<>") if message_id else None,
+    )
     snippet = (
         issue["comment"]
         if len(issue["comment"]) <= 70
@@ -218,11 +228,18 @@ def generate_issue_report(
     if isinstance(issue["misc"], str):
         issue["misc"] = json.loads(issue["misc"])
 
+    # Generated upfront so the report body can link to its own mail on lore
+    message_id = make_msgid(domain="kernelci.org")
+
     issue_type: PossibleIssueType
     if issue["build_id"]:
         issue_type = "build"
         incidents = kcidb_build_incidents(issue_id)
-        report = generate_build_issue_report(issue, incidents)
+        for incident in incidents:
+            last_build = kcidb_last_build_without_issue(issue, incident)
+            if last_build:
+                incident["last_pass_commit"] = last_build[0]["git_commit_hash"]
+        report = generate_build_issue_report(issue, incidents, message_id)
     elif issue["test_id"]:
         incidents = kcidb_test_incidents(issue_id)
         issue_type = "boot" if is_boot(path=incidents[0]["path"]) else "test"
@@ -231,7 +248,7 @@ def generate_issue_report(
             incident["last_pass"] = last_test[0]["start_time"]
             incident["last_pass_commit"] = last_test[0]["git_commit_hash"]
             incident["last_pass_id"] = last_test[0]["id"]
-        report = generate_boot_issue_report(issue, incidents)
+        report = generate_boot_issue_report(issue, incidents, message_id)
     else:
         print(f"unable to generate issue report for {issue_id}", file=sys.stderr)
         sys.exit(-1)
@@ -244,6 +261,7 @@ def generate_issue_report(
         email_args=email_args,
         signup_folder=signup_folder,
         git_url=issue["git_repository_url"],
+        message_id=message_id,
     )
 
     if msg_id and email_args.update:
@@ -566,6 +584,7 @@ def generate_test_report(*, service, test_id, email_args, signup_folder):
         test_start_time=test["start_time"],
         config_name=test["config_name"],
         field_timestamp=test["_timestamp"],
+        group_size=10,
     )
 
     status_symbols = []
@@ -581,8 +600,22 @@ def generate_test_report(*, service, test_id, email_args, signup_folder):
     status_symbols.reverse()
     test["status_history"] = " → ".join(status_symbols)
 
+    # Most recent passing commit before the failure, used to bound the
+    # regression range for regzbot (history is ordered most recent first).
+    test["last_pass_commit"] = None
+    for entry in history:
+        if (
+            entry["status"] == "PASS"
+            and entry["build__checkout__git_commit_hash"] != test["git_commit_hash"]
+        ):
+            test["last_pass_commit"] = entry["build__checkout__git_commit_hash"]
+            break
+
+    # Generated upfront so the report body can link to its own mail on lore
+    message_id = make_msgid(domain="kernelci.org")
+
     template = setup_jinja_template("test_report.txt.j2")
-    report_content = template.render(test=test)
+    report_content = template.render(test=test, message_id=message_id.strip("<>"))
 
     if is_boot(path=test["path"]):
         subject_prefix = "[BOOT REGRESSION]"
@@ -615,6 +648,7 @@ def generate_test_report(*, service, test_id, email_args, signup_folder):
         email_args=email_args,
         git_url=test["git_repository_url"],
         signup_folder=signup_folder,
+        message_id=message_id,
     )
 
 

backend/kernelCI_app/management/commands/refresh_report_examples.py

@@ -0,0 +1,34 @@
+from django.core.management.base import BaseCommand
+
+from kernelCI_app.tests.unitTests.commands.fixtures.report_notifications_data import (
+    ISSUE_BOOT_REPORT_EXAMPLE_FILEPATH,
+    ISSUE_BUILD_REPORT_EXAMPLE_FILEPATH,
+    TEST_REPORT_EXAMPLE_FILEPATH,
+)
+from kernelCI_app.tests.unitTests.commands.report_notifications_test import (
+    TestReportTemplates,
+)
+
+
+class Command(BaseCommand):
+    help = (
+        "Refreshes the example report fixtures used by the report notification "
+        "unit tests. Run this manually when a report template changes."
+    )
+
+    def handle(self, *args, **options):
+        outputs = [
+            (
+                ISSUE_BUILD_REPORT_EXAMPLE_FILEPATH,
+                TestReportTemplates.render_build_report(),
+            ),
+            (
+                ISSUE_BOOT_REPORT_EXAMPLE_FILEPATH,
+                TestReportTemplates.render_boot_report(),
+            ),
+            (TEST_REPORT_EXAMPLE_FILEPATH, TestReportTemplates.render_test_report()),
+        ]
+        for filepath, content in outputs:
+            with open(filepath, "w") as f:
+                f.write(content)
+            self.stdout.write(self.style.SUCCESS(f"{filepath} updated."))

backend/kernelCI_app/management/commands/templates/base.txt

@@ -1,5 +1,4 @@
-
-{% block header %}{% endblock %}
+{% block header %}{% endblock -%}
 {% block content %}{% endblock %}
 {% block footer %}
 --

backend/kernelCI_app/management/commands/templates/issue_boot.txt.j2

@@ -1,34 +1,33 @@
-
 {% extends "base.txt" %}
 {% block header %}{% endblock %}
-{% block content %}
-
-Hello,
-
+{% block content -%}
 New boot regression found on {{ issue["tree_name"] }}/{{ issue["git_repository_branch"] }}:
 
 ---
 {{ issue["comment"]}}
 ---
-
-- dashboard: https://d.kernelci.org/i/{{ issue["id"] }}
-- giturl: {{ issue["git_repository_url"]}}
-- commit HEAD:  {{ issue["git_commit_hash"] }}
-{% if issue["git_commit_tags"] -%}
-- tags: {% for tag in issue["git_commit_tags"] %}{{tag}}{% if not loop.last %},{% endif %}{% endfor %}
-{%- endif %}
-
-Please include the KernelCI tag when submitting a fix:
-
-Reported-by: kernelci.org bot <bot@kernelci.org>
-
 {% if "misc" in issue.keys() and "logspec" in issue["misc"].keys() %}
 Log excerpt:
 =====================================================
 {{ issue["misc"]["logspec"]["error"]["log_excerpt"] }}
 =====================================================
 {% endif %}
+When fixing this issue, please attribute this report according to the
+patch submission rules:
+
+Reported-by: kernelci.org bot <bot@kernelci.org>
+{% if message_id -%}
+Link: https://lore.kernel.org/all/{{ message_id }}/
+{% endif %}
+For complete logs and details about the environment used, see the issue
+on the KernelCI dashboard:
 
+- dashboard: https://d.kernelci.org/i/{{ issue["id"] }}
+- giturl: {{ issue["git_repository_url"]}}
+- commit HEAD:  {{ issue["git_commit_hash"] }}
+{% if issue["git_commit_tags"] -%}
+- tags: {% for tag in issue["git_commit_tags"] %}{{tag}}{% if not loop.last %},{% endif %}{% endfor %}
+{% endif %}
 # Hardware platforms affected:
 {% for boot in boots %}
 ## {{boot["platform"]}}
@@ -46,7 +45,10 @@ Log excerpt:
     - commit hash:  {{ boot["last_pass_commit"] }}
     - test id:  {{ boot["last_pass_id"] }}
 {% endfor %}
+{% if boots and boots[0]["last_pass_commit"] -%}
+#regzbot introduced: {{ boots[0]["last_pass_commit"] }}..{{ issue["git_commit_hash"] }}
+#regzbot link: https://d.kernelci.org/i/{{ issue["id"] }}
 
+{% endif -%}
 #kernelci issue {{ issue["id"] }}
-
 {%- endblock -%}

backend/kernelCI_app/management/commands/templates/issue_build.txt.j2

@@ -1,42 +1,44 @@
-
 {% extends "base.txt" %}
 {% block header %}{% endblock %}
-{% block content %}
-Hello,
-
+{% block content -%}
 New build issue found on {{ issue["tree_name"] }}/{{ issue["git_repository_branch"] }}:
 
 ---
 {{ issue["comment"]}}
 ---
-
-- dashboard: https://d.kernelci.org/i/{{ issue["id"] }}
-- giturl: {{ issue["git_repository_url"]}}
-- commit HEAD:  {{ issue["git_commit_hash"] }}
-{% if issue["git_commit_tags"] -%}
-- tags: {% for tag in issue["git_commit_tags"] %}{{tag}}{% if not loop.last %},{% endif %}{% endfor %}
-{%- endif %}
-
-Please include the KernelCI tag when submitting a fix:
-
-Reported-by: kernelci.org bot <bot@kernelci.org>
-
 {% if "misc" in issue.keys() and "logspec" in issue["misc"].keys() %}
 Log excerpt:
 =====================================================
 {{ issue["misc"]["logspec"]["error"]["log_excerpt"] }}
 =====================================================
 {% endif %}
+When fixing this issue, please attribute this report according to the
+patch submission rules:
 
+Reported-by: kernelci.org bot <bot@kernelci.org>
+{% if message_id -%}
+Link: https://lore.kernel.org/all/{{ message_id }}/
+{% endif %}
+For complete logs and details about the environment used, see the issue
+on the KernelCI dashboard:
+
+- dashboard: https://d.kernelci.org/i/{{ issue["id"] }}
+- giturl: {{ issue["git_repository_url"]}}
+- commit HEAD:  {{ issue["git_commit_hash"] }}
+{% if issue["git_commit_tags"] -%}
+- tags: {% for tag in issue["git_commit_tags"] %}{{tag}}{% if not loop.last %},{% endif %}{% endfor %}
+{% endif %}
 # Builds where the incident occurred:
 {% for build in builds %}
 ## {{ build["config_name"] }} on ({{ build["architecture"] }}):
 - compiler: {{ build["compiler"] }}
 - config: {{ build["config_url"] }}
 - dashboard: https://d.kernelci.org/build/{{build["id"]}}
 {% endfor %}
+{% if builds and builds[0]["last_pass_commit"] -%}
+#regzbot introduced: {{ builds[0]["last_pass_commit"] }}..{{ issue["git_commit_hash"] }}
+#regzbot link: https://d.kernelci.org/i/{{ issue["id"] }}
 
+{% endif -%}
 #kernelci issue {{ issue["id"] }}
-
-
 {%- endblock -%}