fix(jira): _raw_data_*-Spalten in _tool_jira_sprint_reports nachruesten

Integriert den Inhalt von PR apache#9015 in den lokalen
Integrations-Branch.

Die Sprint-Report-Migration 20260722_add_sprint_report_table.go legt
_tool_jira_sprint_reports aus einem Struct an, das common.NoPKModel
nicht einbettet, waehrend das Laufzeitmodell models.JiraSprintReport es
einbettet. Die Spalten _raw_data_params / _raw_data_table /
_raw_data_id / _raw_data_remark (plus created_at, updated_at) fehlten
daher, sodass die Cleanup-Query des ApiExtractors

    WHERE _raw_data_table = ? AND _raw_data_params = ?

den Subtask extractSprintReport mit "Error 1054 (42S22): Unknown column
'_raw_data_table' in 'where clause'" abbrechen liess.

Neue, additive Migration statt Aenderung der alten: Migrationsskripte
sind append-only, und ein Edit wuerde Datenbanken nicht reparieren, die
die Version bereits protokolliert haben.

Zwei Schema-Drift-Guards, die die ECHTEN Migrationsskripte ausfuehren
(statt das Laufzeitmodell zu AutoMigrate-n, was genau diese Drift
verdecken wuerde):

  * plugins/jira/e2e/migration_schema_test.go - Jira-spezifisch.
  * plugins/schema_e2e/migration_schema_test.go - plugin-uebergreifend
    fuer alle Go-Plugins, inkl. TestAllGoPluginsListed.

Der uebergreifende Guard deckte drei bestehende Drifts derselben Klasse
auf, je mit eigener additiver Migration behoben:

  * _tool_taiga_scope_configs      - type_mappings fehlte
  * _tool_teambition_scope_configs - id, created_at, updated_at fehlten
  * _tool_testmo_scope_configs     - connection_id, name fehlten

Verifiziert gegen MySQL 8.4.10 und PostgreSQL 17.2: neue Spalten werden
nullable angelegt, doppelte NULLs sind im uniqueIndex zulaessig, und die
AUTO_INCREMENT-PK laesst sich auf der PK-losen Teambition-Tabelle
nachruesten (kein Error 1075). Alle vier Migrationen wurden zusaetzlich
auf der bestehenden lokalen Datenbank angewendet.

Ausserdem: plugins/schema_e2e in scripts/build-plugins.sh ausgeschlossen
(kein Plugin, kein main-Package -> "make build-plugin" scheiterte mit
"-buildmode=plugin requires exactly one main package").

AGENTS.md um die Schema-Drift-Guards und die zugehoerigen Fallstricke
ergaenzt (fork-only, nicht Teil des Upstream-PRs).

Signed-off-by: DoDiODev <DoDiDev@proton.me>

Author: DoDiODev

AGENTS.md

@@ -162,6 +162,32 @@ helper.ConnectLocalServer(t, &helper.LocalClientConfig{
 ### Model Validation
 Run `make migration-script-lint` from `backend/` to validate all migration scripts follow the correct format (`YYYYMMDD_description.go`).
 
+### Schema-Drift Guard (migration vs. model)
+Two e2e checks assert that the schema produced by the **real migration scripts**
+still matches what the **runtime GORM models** expect (they run the migrations
+instead of `AutoMigrate`-ing the model, which would hide drift):
+- `backend/plugins/jira/e2e/migration_schema_test.go` — Jira-specific regression
+  test for the Sprint Report `_raw_data_table` bug.
+- `backend/plugins/schema_e2e/migration_schema_test.go` — cross-plugin guard for
+  **every** built-in Go plugin, plus `TestAllGoPluginsListed`, which fails if a
+  new plugin's `impl` package is not registered in `allGoPlugins()`.
+
+Both require `E2E_DB_URL` and run under `make e2e-test-go-plugins`. A common
+cause of failure: a migration struct that forgets to embed `common.NoPKModel`,
+so the `_raw_data_*` columns are missing while the model declares them.
+
+Notes:
+- `backend/plugins/schema_e2e/` is **not** a plugin (no `main` package); it is
+  excluded in `backend/scripts/build-plugins.sh`, otherwise `make build-plugin`
+  fails with "-buildmode=plugin requires exactly one main package".
+- Both tests set a fallback `ENCRYPTION_SECRET` and call `dalgorm.Init(...)`;
+  without it migrations fail with `invalid encKey` / `invalid serializer type
+  encdec` (`runner.CreateBasicRes` does not register the serializer, only
+  `CreateAppBasicRes` does).
+- Migration scripts must not import `core/models/common` (enforced by
+  `make migration-script-lint`) — use `core/models/migrationscripts/archived`.
+
+
 ## Python Plugins
 Located in `backend/python/plugins/`. Use Poetry for dependencies. See [backend/python/README.md](backend/python/README.md).
 Python is pinned to **3.11** (Pydantic v1 + `dbt-mysql` 1.7 block 3.12+).
@@ -177,7 +203,9 @@ Python is pinned to **3.11** (Pydantic v1 + `dbt-mysql` 1.7 block 3.12+).
 
 ## Common Pitfalls
 - Forgetting to add models to `GetTablesInfo()` fails `plugins/table_info_test.go`
+- Migration structs that create a `_tool_*` table must embed `common.NoPKModel` (or `common.RawDataOrigin`) so the `_raw_data_*` columns exist — otherwise the ApiExtractor's cleanup query fails at runtime with `Unknown column '_raw_data_table'`. Guarded by the schema-drift e2e checks (see Testing → Schema-Drift Guard).
 - Migration scripts must be added to `All()` in `register.go` AND follow `YYYYMMDD_description.go` naming (validate with `make migration-script-lint`)
+- Never edit a migration script that has already shipped/run; fix schema issues with a **new** migration (append-only), since executed versions never re-run.
 - API changes require running `make swag` to update Swagger docs (this runs `make mock` first)
 - Python plugins require `libgit2` for gitextractor functionality
 - New Plugin interfaces like `PluginInit` or `PluginOpenApiSpec` are optional - only implement if needed

backend/plugins/jira/e2e/migration_schema_test.go

@@ -0,0 +1,112 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package e2e
+
+import (
+	"sync"
+	"testing"
+
+	"github.com/apache/incubator-devlake/core/dal"
+	"github.com/apache/incubator-devlake/core/migration"
+	coreMigration "github.com/apache/incubator-devlake/core/models/migrationscripts"
+	"github.com/apache/incubator-devlake/core/plugin"
+	"github.com/apache/incubator-devlake/core/runner"
+	"github.com/apache/incubator-devlake/helpers/e2ehelper"
+	"github.com/apache/incubator-devlake/impls/dalgorm"
+	"github.com/apache/incubator-devlake/plugins/jira/impl"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+	"gorm.io/gorm/schema"
+)
+
+// TestMigrationSchema guards against schema drift between Jira's migration
+// scripts and its runtime GORM models.
+//
+// Regression test for the Sprint Report bug (introduced by PR #8967/#9010):
+// the migration that created `_tool_jira_sprint_reports` used a struct that did
+// NOT embed common.NoPKModel, so the `_raw_data_table` / `_raw_data_params` /
+// `_raw_data_id` / `_raw_data_remark` columns were missing. The runtime model
+// DID embed common.NoPKModel, so the ApiExtractor's cleanup query
+// (`WHERE _raw_data_table = ? AND _raw_data_params = ?`) failed at runtime with
+// "Error 1054: Unknown column '_raw_data_table' in 'where clause'".
+//
+// The test runs the REAL migration scripts (framework + jira) to build the
+// schema exactly the way a production install would — deliberately NOT via
+// AutoMigrate on the runtime model, which would silently hide such drift — and
+// then asserts that every column each runtime model expects actually exists in
+// the migrated table. Any future migration that forgets to embed
+// common.NoPKModel (or otherwise omits a column) will fail this test.
+//
+// Requires E2E_DB_URL (runs under `make e2e-test` / `make e2e-test-go-plugins`).
+func TestMigrationSchema(t *testing.T) {
+	var pluginInstance impl.Jira
+	dataflowTester := e2ehelper.NewDataFlowTester(t, "jira", pluginInstance)
+
+	// Some jira migration scripts refuse to run without an encryption secret
+	// (20220716: "jira v0.11 invalid encKey"); CI does not necessarily provide
+	// one, so fall back to a deterministic test value. dalgorm.Init registers
+	// the `encdec` GORM serializer used by connection models — without it the
+	// migrations fail with "invalid serializer type encdec".
+	if dataflowTester.Cfg.GetString(plugin.EncodeKeyEnvStr) == "" {
+		dataflowTester.Cfg.Set(plugin.EncodeKeyEnvStr, "jira-e2e-test-encryption-secret")
+	}
+	dalgorm.Init(dataflowTester.Cfg.GetString(plugin.EncodeKeyEnvStr))
+
+	// Apply the real migration scripts so the schema matches a production install.
+	basicRes := runner.CreateBasicRes(dataflowTester.Cfg, dataflowTester.Log, dataflowTester.Db)
+	migrator, err := migration.NewMigrator(basicRes)
+	require.NoError(t, err)
+	migrator.Register(coreMigration.All(), "Framework")
+	migrator.Register(pluginInstance.MigrationScripts(), "jira")
+	require.NoError(t, migrator.Execute())
+
+	keepAll := func(dal.ColumnMeta) bool { return true }
+
+	for _, table := range pluginInstance.GetTablesInfo() {
+		table := table
+		t.Run(table.TableName(), func(t *testing.T) {
+			// Columns the runtime GORM model expects.
+			sch, err := schema.Parse(table, &sync.Map{}, schema.NamingStrategy{})
+			require.NoErrorf(t, err, "unable to parse schema for %T", table)
+
+			// Columns that actually exist in the migrated table.
+			actualColumns, err := dal.GetColumnNames(dataflowTester.Dal, table, keepAll)
+			if err != nil || len(actualColumns) == 0 {
+				// No migration creates this table (e.g. API response models
+				// that are listed in GetTablesInfo but never persisted) —
+				// there is no schema to drift from.
+				t.Skipf("table %q not present after migrations", table.TableName())
+			}
+			existing := make(map[string]struct{}, len(actualColumns))
+			for _, c := range actualColumns {
+				existing[c] = struct{}{}
+			}
+
+			for _, field := range sch.Fields {
+				if field.DBName == "" || field.IgnoreMigration {
+					continue
+				}
+				_, ok := existing[field.DBName]
+				assert.Truef(t, ok,
+					"table %q is missing column %q expected by model %T — "+
+						"did a migration script forget to embed common.NoPKModel (raw-data columns) or add the field?",
+					table.TableName(), field.DBName, table)
+			}
+		})
+	}
+}

backend/plugins/jira/models/migrationscripts/20260727_add_raw_data_columns_to_sprint_report.go

@@ -0,0 +1,65 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package migrationscripts
+
+import (
+	"github.com/apache/incubator-devlake/core/context"
+	"github.com/apache/incubator-devlake/core/errors"
+	"github.com/apache/incubator-devlake/core/models/migrationscripts/archived"
+	"github.com/apache/incubator-devlake/helpers/migrationhelper"
+)
+
+// jiraSprintReport20260727 mirrors the JiraSprintReport model. The original
+// migration (20260722) created _tool_jira_sprint_reports without embedding
+// common.NoPKModel, so the _raw_data_table / _raw_data_params / _raw_data_id /
+// _raw_data_remark columns (and created_at / updated_at) were missing. The
+// runtime model expects them, which made the ApiExtractor's cleanup query
+// (WHERE _raw_data_table = ? AND _raw_data_params = ?) fail with
+// "Unknown column '_raw_data_table' in 'where clause'". Re-running
+// AutoMigrateTables adds the missing columns without dropping existing data.
+type jiraSprintReport20260727 struct {
+	archived.NoPKModel
+	ConnectionId uint64 `gorm:"primaryKey"`
+	BoardId      uint64 `gorm:"primaryKey"`
+	SprintId     uint64 `gorm:"primaryKey"`
+	IssueId      uint64 `gorm:"primaryKey"`
+
+	IssueKey                 string `gorm:"type:varchar(255)"`
+	Bucket                   string `gorm:"type:varchar(32);index"`
+	Done                     bool
+	StoryPointsAtSprintStart *float64
+	StoryPointsAtSprintEnd   *float64
+}
+
+func (jiraSprintReport20260727) TableName() string {
+	return "_tool_jira_sprint_reports"
+}
+
+type addRawDataColumnsToSprintReport struct{}
+
+func (script *addRawDataColumnsToSprintReport) Up(basicRes context.BasicRes) errors.Error {
+	return migrationhelper.AutoMigrateTables(basicRes, &jiraSprintReport20260727{})
+}
+
+func (*addRawDataColumnsToSprintReport) Version() uint64 {
+	return 20260727000000
+}
+
+func (*addRawDataColumnsToSprintReport) Name() string {
+	return "add missing _raw_data_* columns to _tool_jira_sprint_reports"
+}

backend/plugins/jira/models/migrationscripts/register.go

@@ -59,5 +59,6 @@ func All() []plugin.MigrationScript {
 		new(changeFixVersionsToText20260707),
 		new(addExtraJQLToScopeConfig),
 		new(addSprintReportTable),
+		new(addRawDataColumnsToSprintReport),
 	}
 }