Merge branch 'dev' into fix-18074-sql-parameter-type-passing

Author: SbloodyS

.github/ISSUE_TEMPLATE/bug-report.yml

@@ -104,6 +104,7 @@ body:
         - 3.3.2
         - 3.4.0
         - 3.4.1
+        - 3.4.2
     validations:
       required: true
 

.gitignore

@@ -60,3 +60,4 @@ ds_schema_check_test
 docs/superpowers/
 .claude/worktrees
 CLAUDE.local.md
+AGENT.local.md

AGENT.md

@@ -0,0 +1,161 @@
+# AGENT.md - Apache DolphinScheduler
+
+Apache DolphinScheduler is a distributed, visual DAG workflow-scheduling platform. This is the monorepo: backend servers (master / worker / api / alert), a Vue 3 frontend, plugin families for tasks / datasources / storage / alerting / scheduling, and the release tooling.
+
+**This file is an agent-facing project index, adapted from `CLAUDE.md`.** Module-specific details currently live in each module's `CLAUDE.md`; use those files as the source of truth and do not duplicate module contents here.
+
+---
+
+## Tech stack (project-wide)
+
+- **Java 1.8** (do not assume 11+ APIs; `dolphinscheduler-api-test` is the only Java 11 island).
+- **Spring Boot 2.6.1** across servers, **Jetty** (Tomcat is excluded transitively).
+- **MyBatis-Plus** for ORM; **HikariCP** for the metadata DB pool, **Druid** inside user-facing datasource plugins.
+- **Quartz** for cron scheduling (via `scheduler-plugin`).
+- **Netty / gRPC** for inter-server RPC (see `extract-base`).
+- **Vue 3 + Vite + TypeScript + Naive UI** for the frontend.
+- **Maven** multi-module reactor (26 modules in root `pom.xml` + 2 test modules).
+- **Zookeeper 3.8** by default for the registry (Etcd and JDBC also supported).
+
+## Runnable services
+
+A production deployment runs **four independent services** (plus an external registry and metadata DB). A fifth entry point, `StandaloneServer`, embeds all four in one JVM for development.
+
+| Service | Module | Main class | Default ports |
+|---------|--------|------------|---------------|
+| **API** | [`dolphinscheduler-api`](dolphinscheduler-api/CLAUDE.md) | `org.apache.dolphinscheduler.api.ApiApplicationServer` | `12345` (HTTP / UI + REST) |
+| **Master** | [`dolphinscheduler-master`](dolphinscheduler-master/CLAUDE.md) | `org.apache.dolphinscheduler.server.master.MasterServer` | `5679` (RPC) |
+| **Worker** | [`dolphinscheduler-worker`](dolphinscheduler-worker/CLAUDE.md) | `org.apache.dolphinscheduler.server.worker.WorkerServer` | `1235` (RPC) |
+| **Alert** | [`dolphinscheduler-alert`](dolphinscheduler-alert/CLAUDE.md) (to `-alert-server`) | `org.apache.dolphinscheduler.alert.AlertServer` | `50053` (HTTP), `50052` (RPC) |
+| Standalone (dev only) | [`dolphinscheduler-standalone-server`](dolphinscheduler-standalone-server/CLAUDE.md) | `org.apache.dolphinscheduler.StandaloneServer` | `12345` + `50052` (API + alert; master/worker use in-JVM calls) |
+
+Every service is a `@SpringBootApplication` on Jetty and implements `IStoppable`. Scale Master / Worker / Alert horizontally; coordination happens via the registry (Zookeeper by default). API is stateless and also scales horizontally behind a load balancer.
+
+Ports are overridable via `server.port` / service-specific keys in each service's `application.yaml`.
+
+## Build & run
+
+```bash
+# Full build (release profile; produces dist tarball)
+./mvnw clean install -Prelease
+
+# Zookeeper 3.4 legacy
+./mvnw clean install -Prelease -Dzk-3.4
+
+# Skip UI build (faster iteration on backend only)
+./mvnw -pl '!dolphinscheduler-ui' clean install
+
+# Build one module (+ its required siblings)
+./mvnw -pl dolphinscheduler-master -am clean install
+
+# Format (Spotless is configured)
+./mvnw spotless:apply
+
+# Standalone server (after building)
+cd dolphinscheduler-standalone-server/target && ./bin/start.sh
+```
+
+Binary artifact: `dolphinscheduler-dist/target/apache-dolphinscheduler-*-bin.tar.gz`.
+
+## Test
+
+```bash
+# Unit tests for one module
+./mvnw -pl dolphinscheduler-master test
+
+# API integration tests (separate reactor, requires Docker)
+mvn -pl dolphinscheduler-api-test/dolphinscheduler-api-test-case test
+
+# E2E browser tests (Selenium + Docker)
+mvn -pl dolphinscheduler-e2e/dolphinscheduler-e2e-case test
+
+# Apple Silicon: add -Dm1_chip=true to the Docker-driven suites
+```
+
+---
+
+## Module index
+
+Click into a module's `CLAUDE.md` for details. Each description is one line here on purpose.
+
+### Core execution
+
+- [`dolphinscheduler-master`](dolphinscheduler-master/CLAUDE.md) - workflow orchestration engine; consumes `Command`s, runs the DAG state machine, dispatches to workers.
+- [`dolphinscheduler-worker`](dolphinscheduler-worker/CLAUDE.md) - runs physical tasks dispatched from master; hosts task plugins.
+- [`dolphinscheduler-task-executor`](dolphinscheduler-task-executor/CLAUDE.md) - reusable task-lifecycle framework embedded by the worker.
+- [`dolphinscheduler-alert`](dolphinscheduler-alert/CLAUDE.md) - alert server + channel plugins (email, Feishu, DingTalk, ...).
+
+### API layer
+
+- [`dolphinscheduler-api`](dolphinscheduler-api/CLAUDE.md) - REST API server (entry point for UI, Python SDK, external clients).
+- [`dolphinscheduler-api-test`](dolphinscheduler-api-test/CLAUDE.md) - integration tests against the REST API (Docker Compose + Testcontainers).
+- [`dolphinscheduler-authentication`](dolphinscheduler-authentication/CLAUDE.md) - Actuator-endpoint auth + AWS credential helpers (NOT the main login path).
+
+### Shared libraries
+
+- [`dolphinscheduler-common`](dolphinscheduler-common/CLAUDE.md) - foundation utilities (everything depends on this).
+- [`dolphinscheduler-dao`](dolphinscheduler-dao/CLAUDE.md) - MyBatis DAO layer + SQL migration scripts.
+- [`dolphinscheduler-service`](dolphinscheduler-service/CLAUDE.md) - business logic between DAO and the servers.
+- [`dolphinscheduler-spi`](dolphinscheduler-spi/CLAUDE.md) - Service-Provider Interface root (every plugin depends on this).
+- [`dolphinscheduler-extract`](dolphinscheduler-extract/CLAUDE.md) - RPC interface contracts between servers.
+- [`dolphinscheduler-eventbus`](dolphinscheduler-eventbus/CLAUDE.md) - in-process event-bus abstractions.
+- [`dolphinscheduler-registry`](dolphinscheduler-registry/CLAUDE.md) - pluggable registry (Zookeeper / Etcd / JDBC).
+- [`dolphinscheduler-meter`](dolphinscheduler-meter/CLAUDE.md) - metrics (Prometheus) + server load-protection primitives.
+
+### Plugin families
+
+- [`dolphinscheduler-task-plugin`](dolphinscheduler-task-plugin/CLAUDE.md) - task-type plugins (shell, SQL, Spark, Flink, K8s, EMR, ...). 33 concrete plugins.
+- [`dolphinscheduler-datasource-plugin`](dolphinscheduler-datasource-plugin/CLAUDE.md) - user-facing datasource plugins (MySQL, Hive, Trino, Snowflake, ...). 28 concrete plugins.
+- [`dolphinscheduler-storage-plugin`](dolphinscheduler-storage-plugin/CLAUDE.md) - resource storage (S3, HDFS, OSS, GCS, ABS, OBS, COS).
+- [`dolphinscheduler-scheduler-plugin`](dolphinscheduler-scheduler-plugin/CLAUDE.md) - cron scheduler (Quartz today).
+- [`dolphinscheduler-dao-plugin`](dolphinscheduler-dao-plugin/CLAUDE.md) - metadata-DB dialect support (MySQL / PostgreSQL / H2).
+
+### Build, ops, tools
+
+- [`dolphinscheduler-bom`](dolphinscheduler-bom/CLAUDE.md) - Maven BOM; central dependency version pinning.
+- [`dolphinscheduler-dist`](dolphinscheduler-dist/CLAUDE.md) - assembles the release tarball + Docker images.
+- [`dolphinscheduler-standalone-server`](dolphinscheduler-standalone-server/CLAUDE.md) - all-in-one JVM with H2 (dev / smoke tests).
+- [`dolphinscheduler-tools`](dolphinscheduler-tools/CLAUDE.md) - CLIs for schema upgrade + resource / lineage migration.
+- [`dolphinscheduler-microbench`](dolphinscheduler-microbench/CLAUDE.md) - JMH micro-benchmarks.
+- [`dolphinscheduler-yarn-aop`](dolphinscheduler-yarn-aop/CLAUDE.md) - AspectJ weaver capturing YARN ApplicationIds.
+
+### Frontend & E2E
+
+- [`dolphinscheduler-ui`](dolphinscheduler-ui/CLAUDE.md) - Vue 3 frontend.
+- [`dolphinscheduler-e2e`](dolphinscheduler-e2e/CLAUDE.md) - Selenium browser tests.
+
+---
+
+## Architecture overview
+
+A **user** hits the UI, which calls the API server. The API server writes to the **metadata DB** and, for runtime operations (start / kill / pause workflow), talks to the **master** over RPC. The master consumes `t_ds_command` rows, runs the workflow state machine, and dispatches tasks to **workers**. Workers execute task plugins (shell, SQL, Spark, ...) and stream lifecycle events back to master. Failures and SLA breaches flow to the **alert server**, which fans out through alert plugins. **Registry** (Zookeeper / Etcd / JDBC) provides service discovery, leader election, and distributed locks. **Storage plugins** back the resource center and distributed-task artifacts. **Quartz** (via scheduler plugin) fires scheduled workflows, which become new `Command` rows.
+
+## Where things live (quick lookup)
+
+| Looking for... | Start here |
+|----------------|------------|
+| A REST endpoint | `dolphinscheduler-api/src/main/java/.../api/controller/` |
+| Workflow execution logic | `dolphinscheduler-master/src/main/java/.../server/master/engine/` |
+| Task execution logic | `dolphinscheduler-worker` + the specific `task-plugin/<type>` |
+| How "X" is stored | `dolphinscheduler-dao/src/main/java/.../dao/entity/` |
+| SQL schema / upgrade | `dolphinscheduler-dao/src/main/resources/sql/` |
+| RPC contract between servers | `dolphinscheduler-extract/dolphinscheduler-extract-<role>` |
+| UI page source | `dolphinscheduler-ui/src/views/<feature>/` |
+| API call in the UI | `dolphinscheduler-ui/src/service/modules/<resource>.ts` |
+| Version of a dependency | `dolphinscheduler-bom/pom.xml` |
+
+## Project-wide conventions
+
+- **Formatting**: Run `./mvnw spotless:apply` before every commit/push. Spotless covers Java sources, `pom.xml`, and Markdown files; CI runs `./mvnw spotless:check` and will fail PRs that are not formatted. Java imports are ordered; license headers are enforced.
+- **Commit style**: `[Type-ISSUE_ID][Scope] Subject`, e.g. `[Fix-18168][Worker] ...`. All types except `Chore` require an issue ID. See [commit-message.md](docs/docs/en/contribute/join/commit-message.md) for the full convention.
+- **Branching**: `dev` is the main integration branch (not `main`/`master`).
+- **PRs must link a GitHub issue** and keep their scope tight: one module / one concern. For `Chore` commits, no issue ID is required by the commit convention.
+- **Do not break wire / DB compatibility** silently. Changes to `extract-*` RPC interfaces, `dao` entities, enum values, and `spi.DbType` ripple to deployed clusters mid-upgrade.
+- **Only one registry / storage / DB dialect is active at runtime**. Code paths that check "which one" belong inside the plugin SPI, not sprinkled through services.
+
+## External references
+
+- Release docs (version-specific): https://dolphinscheduler.apache.org/en-us/docs
+- GitHub issues: https://github.com/apache/dolphinscheduler/issues
+- Python SDK: https://dolphinscheduler.apache.org/python/main/index.html
+- Contribution guide: [`docs/docs/en/contribute/join/contribute.md`](docs/docs/en/contribute/join/contribute.md)

AGENTS.md

@@ -0,0 +1,29 @@
+<!--
+SPDX-License-Identifier: Apache-2.0
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Agent Guide for dolphinscheduler
+
+This file is read by automated agents (security scanners, code
+analyzers, AI assistants) operating on this repository.
+
+## Security
+
+Security model: [SECURITY.md](./SECURITY.md)
+
+Agents that scan this repository should consult `SECURITY.md` and the
+threat model it links before reporting issues.
+
+DolphinScheduler's security model defines user roles, trust boundaries, and known non-findings.

SECURITY.md

@@ -0,0 +1,13 @@
+# Security Policy
+
+## Reporting a Vulnerability
+
+`apache/dolphinscheduler` follows the [Apache Software Foundation security process](https://www.apache.org/security/). Please report suspected
+vulnerabilities privately to `security@apache.org`; do not open public
+GitHub issues or pull requests for security reports.
+
+## Threat Model
+
+What the project treats as in scope and out of scope, the security
+properties it provides and disclaims, the adversary model, and how
+findings are triaged are documented in https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/security-model.md.

docs/configs/index.md.jsx

@@ -68,6 +68,8 @@ import docs330Config from '../../../site_config/docs3-3-0-alpha';
 import docs331Config from '../../../site_config/docs3-3-1';
 import docs332Config from '../../../site_config/docs3-3-2';
 import docs340Config from '../../../site_config/docs3-4-0';
+import docs341Config from '../../../site_config/docs3-4-1';
+import docs342Config from '../../../site_config/docs3-4-2';
 import docsDevConfig from '../../../site_config/docsdev';
 
 const docsSource = {
@@ -113,6 +115,7 @@ const docsSource = {
   '3.3.2': docs332Config,
   '3.4.0': docs340Config,
   '3.4.1': docs341Config,
+  '3.4.2': docs342Config,
   dev: docsDevConfig,
 };
 

docs/configs/site.js

@@ -24,7 +24,7 @@ export default {
   port: 8080,
   domain: 'dolphinscheduler.apache.org',
   copyToDist: ['asset', 'img', 'file', '.asf.yaml', 'sitemap.xml', '.nojekyll', '.htaccess', 'googled0df7b96f277a143.html'],
-  docsLatest: '3.4.1',
+  docsLatest: '3.4.2',
   defaultSearch: 'google', // default search engine
   defaultLanguage: 'en-us',
   'en-us': {
@@ -45,7 +45,7 @@ export default {
         children: [
           {
             key: 'docs0',
-            text: 'latest(3.4.1)',
+            text: 'latest(3.4.2)',
             link: '/en-us/docs/latest/user_doc/about/introduction.html',
           },
           {
@@ -173,7 +173,7 @@ export default {
         children: [
           {
             key: 'docs0',
-            text: '最新版本latest(3.4.1)',
+            text: '最新版本latest(3.4.2)',
             link: '/zh-cn/docs/latest/user_doc/about/introduction.html',
           },
           {

docs/docs/en/faq.md

@@ -238,7 +238,7 @@ export PATH=$HADOOP_HOME/bin:$SPARK_HOME/bin:$PYTHON_LAUNCHER:$JAVA_HOME/bin:$HI
 
 ## Q：Worker Task will generate a child process through sudo -u tenant sh xxx.command, will kill when kill
 
-A： We will add the kill task in 1.0.4 and kill all the various child processes generated by the task.
+A: We will add the kill task in 1.0.4 and kill all the various child processes generated by the task.
 
 ---
 
@@ -370,7 +370,7 @@ A: The license of mysql jdbc connector is not compatible with apache v2 license,
 <p align="center">
    <img src="https://user-images.githubusercontent.com/16174111/81312485-476e9380-90b9-11ea-9aad-ed009db899b1.png" width="60%" />
  </p>
-A： This bug have fix in dev and in Requirement/TODO list.
+A: This bug has been fixed in dev and is in the Requirement/TODO list.
 
 ---
 
@@ -636,7 +636,7 @@ sed -i 's/Defaults    requirett/#Defaults    requirett/g' /etc/sudoers
 
 ## Q：Deploy for multiple YARN clusters
 
-A：By deploying different worker in different yarn clusters，the steps are as follows(eg: AWS EMR):
+A: By deploying different worker in different yarn clusters, the steps are as follows(eg: AWS EMR):
 
 1. Deploying the worker server on the master node of the EMR cluster
 
@@ -648,7 +648,7 @@ A：By deploying different worker in different yarn clusters，the steps are as
 
 ## Q：Update process definition error: Duplicate key TaskDefinition
 
-A：Before DS 2.0.4 (after 2.0.0-alpha), there may be a problem of duplicate keys TaskDefinition due to version switching, which may cause the update workflow to fail; you can refer to the following SQL to delete duplicate data, taking MySQL as an example: (Note: Before operating, be sure to back up the original data, the SQL from pr[#8408](https://github.com/apache/dolphinscheduler/pull/8408))
+A: Before DS 2.0.4 (after 2.0.0-alpha), there may be a problem of duplicate keys TaskDefinition due to version switching, which may cause the update workflow to fail; you can refer to the following SQL to delete duplicate data, taking MySQL as an example: (Note: Before operating, be sure to back up the original data, the SQL from pr[#8408](https://github.com/apache/dolphinscheduler/pull/8408))
 
 ```SQL
 DELETE FROM t_ds_process_task_relation_log WHERE id IN
@@ -736,7 +736,7 @@ DELETE FROM t_ds_task_definition_log WHERE id IN
 
 ## Q：Upgrade from 2.0.1 to 2.0.5 using PostgreSQL database failed
 
-A：The repair can be completed by executing the following SQL in the database:
+A: The repair can be completed by executing the following SQL in the database:
 
 ```SQL
 update t_ds_version set version='2.0.1';
@@ -746,7 +746,7 @@ update t_ds_version set version='2.0.1';
 
 ## Q：Can not find python-gateway-server in distribute package
 
-A：After version 3.0.0-alpha, Python gateway server integrate into API server, and Python gateway service will start when you
+A: After version 3.0.0-alpha, Python gateway server integrate into API server, and Python gateway service will start when you
 start API server. If you want disabled when Python gateway service you could change API server configuration in path
 `api-server/conf/application.yaml` and change attribute `python-gateway.enabled : false`.