2026.0.0#962
Open
qianmoQ wants to merge 21 commits into
Open
Conversation
- 新增 SPI: RowCallback / BatchWriter / JdbcStreamAdapter，JDBC 插件默认获得流式读 + PreparedStatement 批量写能力 - PluginService 增加 supportsStreaming / executeStream / openBatchWriter 默认方法，旧插件零改动 - LocalExecutorService 两端 JDBC 时走流式路径，源端 fetchSize 拉取、目标端 batchSize 批量 flush，全程不物化结果集 - 回退路径修复 NULL/类型/转义错误（替换已废弃的 commons-lang2 escapeSql），按 batch 切片提交不再拼巨大字符串 - 新增 datacap.executor.local.fetchSize / batchSize 配置项，默认 1000
- 引入 datacap-logger，按 SeaTunnel 同款方式在 workHome 下生成 {taskName}.log
- 每 10000 行打印一次进度（已读 / 已 commit / 耗时 / 行/秒）
- 同步开始 / 结束 / 异常均输出到该任务专属日志
- pom 加 maven-dependency-plugin:copy-dependencies，确保 IDE 开发模式下 PluginClassLoader 能在 target/dependencies 找到依赖 jar
- ExecutorService SPI 加 default name() 方法，默认去掉类名末尾的 "ExecutorService"，保留大小写
- DataSetServiceImpl 用 executorService.name().toLowerCase() 作为 workHome 中的 executor 类型段
- 路径格式调整为 dataset/executor/{executor type}/{taskName}
…base
- introduce ExecutorPlugin base in datacap-executor-spi: auto-derives
getName() by stripping the "Executor" suffix from the subclass name
and fixes getType() to EXECUTOR, so each plugin needs zero boilerplate
- rename SeatunnelExecutorPlugin -> SeatunnelExecutor, LocalExecutorPlugin
-> LocalExecutor; both now extend ExecutorPlugin with empty bodies
- drop the redundant ExecutorService.name() default; Plugin.getName() is
the single source of truth for the executor short name
- WorkflowServiceImpl / DataSetServiceImpl: derive the property key from
plugin.getName().toLowerCase() so datacap.executor.<name>.home actually
resolves; previously the lookup used the long form and silently
returned null, producing "null/bin/start-seatunnel-..." commands
- UI: align hardcoded executor identifiers ('SeatunnelExecutor' ->
'Seatunnel', 'LocalExecutor' -> 'Local') in FlowEditor, DatasetInfo
and WorkflowInfo
- configure/etc/conf/executor/category.yaml and datacap.sql default value
switched to the short form
- add 2026.0.0/schema.sql migration to rewrite existing datacap_dataset
and datacap_workflow rows to the short form
- enable -Xjvm-default=all in the root kotlin-maven-plugin so Kotlin
interface defaults compile as JVM 8 default methods and Java
implementations inherit them without AbstractMethodError
- update executor pluginName tests to the short form
- schema 2026.0.0/schema.sql: datacap_dataset_history 新增 total_count / processed_count / progress - DatasetHistoryEntity 同步加 totalCount / processedCount / progress 字段 - executor-spi 新增 ExecutorProgressListener，ExecutorRequest 加 preCount + progressListener - LocalExecutorService 支持 pre-count + 每 1000 行回调进度（PROGRESS_INTERVAL 从 10000 调小） - DataSetServiceImpl 进度回调实时更新 history 与 dataset.totalRows；totalSize 走独立 daemon 线程异步刷 ClickHouse system.parts，避免阻塞 MySQL 流式 cursor 触发 net_write_timeout - datacap.executor.local.preCount 默认 false：SELECT COUNT(*) FROM (userQuery) 在 MySQL/InnoDB 上会物化整个 derived table，慢于真正的同步 - DatasetHistory.vue 增加进度条 slot，DatasetUtils.ts 加 3 列，i18n 加 dataset.history.totalCount / processedCount / progress 三个 key
- schema 2026.0.0/schema.sql: datacap_dataset_history 新增 task_name / work_home 两列，用于定位 executor 写出的独立日志文件
- DatasetHistoryEntity 同步加 taskName / workHome 字段
- DataSetServiceImpl 创建 history 时把 taskName / workHome 落库；新增 getHistoryLog(id) 读取 {workHome}/{taskName}.log，含路径穿越防御
- DataSetController 加 GET /api/v1/dataset/history/log/{id}
- 新增 DatasetHistoryLogger.vue，复用 SeaTunnel 用的 ShadcnLogger；RUNNING 任务默认开启 3 秒自动刷新
- DatasetHistory.vue 加查看日志按钮列；DatasetUtils.ts historyHeaders 加 action
- i18n: dataset.history.viewLog / logger / autoRefresh / refreshInterval（en + zh-cn）
- RunState 新增 STOPPING 中间态：用户点击停止到任务真正退出之间显示"停止中"
- RowCallback 新增 onStatement(Statement) 回调，JdbcStreamAdapter 暴露 JDBC Statement
- LocalExecutorService:
- 静态 runningTasks: ConcurrentHashMap<String, TaskHandle> 索引活跃任务
- stop() 设置 cancelled 标志 + 异步调用 Statement.cancel() 立即 abort 源端流式 cursor
- 行循环每行检查 cancelled；post-loop guard 兜底处理驱动 cancel 后 rs.next()=false 的情况
- 停止事件写入任务专属日志（Stop requested by user / Source JDBC statement cancelled）
- catch 区分 TaskCancelledException 与 JDBC cancel 引发的 SQLException，均归入 STOPPED
- DataSetService.stopHistory(id) + Controller PUT /api/v1/dataset/history/stop/{id}
- stopHistory 在 executor.stop 成功后立即把 history.state 写成 STOPPING；进度回调期间维持 STOPPING 不会被覆盖回 RUNNING
- 同步主流程 finally 块从 stoppingHistoryIds 移除；遇到 STOPPED 状态跳过 Preconditions / flushTableMetadata，保留已写入数据
- UI: 停止按钮仅在 RUNNING/CREATED 显示；STOPPING 状态加橙色 + 国际化（state.common.stopping / dataset.history.stop / stopRequested）
- RunState 新增 INTERRUPTED：服务异常退出后留下的 RUNNING / CREATED / STOPPING 历史在下次启动时被标记为此状态 - DatasetHistoryRepository 加 findAllByStateIn(Collection<RunState>) 供恢复扫描 - 新增 DatasetHistoryRecoveryRunner（CommandLineRunner，Order=10，早于 SchedulerRunner）： - 启动时批量把非终态历史改为 INTERRUPTED - message 带时间戳与原状态，便于审计 - UI 增加紫色标签 + 国际化 state.common.interrupted；DatasetHistory.vue hover-card 同时支持 FAILURE 与 INTERRUPTED 展示 message
- messages_en / zh-cn 增加 11 个 dataset.state.* key，与 DataSetState 枚举一一对应 - common.ts useUtil 暴露 getDatasetStateText(code)，未知 code 回落原文 - DatasetHome.vue 状态列改走 getDatasetStateText，不再直显枚举名 - DatasetState.vue 重写：按数组顺序渲染所有阶段（含 DATA_* / COMPLETE_* 原版缺失分支），并按 SUCCESS / FAILED / START 加色圆点
- datacap-plugin 新增 PluginFieldType 与 PluginConfigureField；Plugin.configures() 默认空，子类可重写 - ExecutorPlugin 不再持有配置 schema，直接继承通用机制 - LocalExecutor.configures() 声明 fetchSize / batchSize / preCount（tunable=true） - SeatunnelExecutor.configures() 声明 home / startScript / engine / way / mode / data（tunable=false 仅管理员可改） - ExecutorService 同步加 name() 方法（去掉 ExecutorService 后缀）
- ConfigureEntity 继承 BaseEntity，加 category / configure (JSON TEXT) / description - 唯一键 (category, name)，例：EXECUTOR/Local、EXECUTOR/Seatunnel、DATASET/Default - ConfigureRepository: findByCategoryAndName / findAllByCategoryOrderByName / existsByCategoryAndName
- 接口提供 getEffective / getRaw / list / save / seedIfAbsent - getEffective 用 schema 默认值兜底，永远返回完整 map - 与现有 ConfigureService（读 executor YAML 元数据）语义分开，不冲突 - 常量 CATEGORY_EXECUTOR / CATEGORY_DATASET 统一范畴 key
- 新增 DatasetTargetSchema：以 PluginConfigureField 声明 8 个 dataset 目标字段（全部 tunable=false） - 新增 RuntimeConfigureSeedRunner（@order=5）：启动遍历 ExecutorPlugin 并 seedIfAbsent，把 SPI 默认值落 DB - DataSetConfigure 移除 @ConfigurationProperties，改在 @PostConstruct 中 seed + 从 RuntimeConfigureService 读 effective；提供 reload() 供管理员改完热刷新 - DataSetServiceImpl / WorkflowServiceImpl 不再读 environment.getProperty("datacap.executor.*")，改用 runtimeConfigureService.getEffective(EXECUTOR, plugin.getName(), plugin.configures()) - 保留 DataSetConfigure 的 getter 签名 → 其余 30+ 调用方零改动
- DatasetHistoryEntity 加 executor_configure TEXT 字段，存合并后 effective 配置 JSON - DataSetServiceImpl 同步开始时把 runtimeConfigureService.getEffective(...) 结果 Jackson 序列化写入 history - DatasetHistory.vue 加"查看配置"按钮（仅当行有数据时显示），弹窗 pretty-printed JSON - i18n: dataset.history.viewConfigure / configureTitle（en + zh-cn） - DDL 列添加 ALTER TABLE 留到 6.9 一并落到 2026.0.0/schema.sql
- DataSetService 增 syncData(code, overrides) 重载 + getSyncFields(code)
- DataSetServiceImpl 通过 ThreadLocal 把本次 overrides 透传给异步同步体；只接受 schema 中 tunable=true 的字段，其余安全忽略；最终合并后 JSON 仍会写入 history.executor_configure
- Controller: PUT /syncData/{code} 接受可选 Map body；GET /syncFields/{code} 返回当前数据集 executor 的 tunable 字段（defaultValue 已替换为 effective 值供 UI 预填）
- Frontend service 加 getSyncFields + syncData(code, overrides)
- DatasetSync.vue 重写：按字段 type 动态渲染 input / number / switch / password；提交时携带 overrides
- i18n: dataset.sync.overrideTitle（en + zh-cn）
后端 - RuntimeConfigureController 提供 GET list / GET detail / PUT save 三个端点，路径 /api/v1/configure/runtime - save 时按 schema 过滤脏字段；写完 DATASET/Default 时主动调 DataSetConfigure.reload() 热刷新内存值 - resolveSchema：EXECUTOR 走 PluginManager.getPluginInfos() 找 plugin.configures()，DATASET 走 DatasetTargetSchema.fields() 前端 - RuntimeConfigureService 封装 list / detail / save - ConfigureHome.vue：左侧分类 + 条目列表，右侧按 schema 动态表单；按字段 type 渲染 input / number / switch / password；非 tunable 字段加"仅管理员"标记 - router 加 /system/configure 路由 - i18n: configure.runtime.title / categoryExecutor / categoryDataset / adminOnly / empty / selectHint（en + zh-cn） 注：菜单入口可在系统菜单管理页面手动添加；URL 直接访问也可用
- application.properties × 2: 移除 datacap.executor.* 与 datacap.dataset.* 全部条目（含 SeaTunnel Spark/Flink 三种模板） - 保留板块占位注释，指向新位置（/system/configure）+ 自动 seed 机制说明 - docs install.md / install.en.md: 重写"执行器配置"章节，说明 2026.0.0 起从 datacap_configure 表读取；删除全部 datacap.executor.* / datacap.dataset.* 示例
….executor_configure - datacap_dataset_history 新增 executor_configure TEXT，对应 6.5 落历史的本次生效 JSON - 新表 datacap_configure(id, name, code, active, category, configure TEXT, description, create_time, update_time) - 唯一键 uk_configure_category_name (category, name)，例：EXECUTOR/Local、EXECUTOR/Seatunnel、DATASET/Default - 表存在则不重建（CREATE TABLE IF NOT EXISTS），便于已运行版本平滑升级；初始默认行由 RuntimeConfigureSeedRunner 在启动时按 SPI schema 自动 seed
- datacap_menu 加 RUNTIME_CONFIGURE 行，挂在管理员系统菜单下（parent=8），url=/system/configure，i18n_key=configure.runtime.title，icon=Settings2 - datacap_role_menu_relation 把该菜单绑定到 admin 角色（role_id=1） - 两条 INSERT 都通过 NOT EXISTS 保证幂等，重复执行迁移不会重复插入
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog category (leave one)
Changelog entry (Details of this change)
Affected version
latest version