[improvement](build) Improve Hive docker startup refresh, idempotency, and metadata backend

### What problem does this PR solve?

Related Issue: #62101

Related PR: #62102

Problem Summary:
This PR overhauls Hive thirdparty startup in docker/thirdparties to make startup and refresh predictable, faster, and repeatable in local and CI workflows.

Main changes:
- add structured Hive startup modes: --hive-mode fast|refresh|rebuild
- add module-scoped refresh: --hive-modules
- persist and reuse Hive state (HDFS/PostgreSQL/state dirs) and introduce baseline/module SHA tracking for incremental refresh
- optimize healthy refresh path to skip unnecessary compose rebuild/up steps
- reduce startup log noise (xtrace gated by HIVE_DEBUG=1, cleaner staged refresh logs, obsolete compose version removal)
- refactor Hive bootstrap scripts and HQL to be idempotent (drop-then-create style for repeated reruns)
- remove redundant startup-heavy operations in refresh path
- switch Hive JuiceFS default metadata backend to Hive metastore PostgreSQL and remove auto-MySQL dependency from Hive startup
- add Hive README documenting component segmentation, startup modes/modules, idempotency expectations, and troubleshooting

### Release note

None

### Check List (For Author)

- Test: Manual test
    - Ran ./docker/thirdparties/run-thirdparties-docker.sh -c hive3 --hive-mode refresh
    - Ran ./docker/thirdparties/run-thirdparties-docker.sh -c hive3 --hive-mode refresh --hive-modules preinstalled_hql
    - Verified healthy refresh path, module refresh behavior, and JuiceFS metadata initialization with PostgreSQL backend
- Behavior changed: Yes
    - Hive startup now follows mode/module-based refresh semantics
    - Default Hive JuiceFS metadata backend is PostgreSQL (still overrideable by JFS_CLUSTER_META)
- Does this need documentation: No

Author: suxiaogang223

docker/thirdparties/docker-compose/hive/README.md

@@ -0,0 +1,169 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either implied.  See the License for the specific
+language governing permissions and limitations
+under the License.
+-->
+
+# Hive Docker Environment
+
+This directory contains Hive2/Hive3 Docker Compose templates and bootstrap scripts used by Doris thirdparty startup.
+
+## Components
+
+- `hive-server`: HiveServer2 endpoint
+- `hive-metastore`: Hive Metastore service
+- `hive-metastore-postgresql`: metastore backend database
+- `namenode` / `datanode`: HDFS services for Hive test data
+
+## Component Segmentation
+
+Hive startup can be understood in 3 layers:
+
+### 1) Docker Service Layer
+
+- Compute/SQL entry:
+  - `hive-server`
+- Metadata:
+  - `hive-metastore`
+  - `hive-metastore-postgresql`
+- Storage:
+  - `namenode`
+  - `datanode`
+
+### 2) Refresh Module Layer (`--hive-modules`)
+
+- `default`: basic default-db external tables
+- `multi_catalog`: multi-format and multi-path external table cases
+- `partition_type`: partition type coverage cases
+- `statistics`: table stats and empty-table stats cases
+- `tvf`: tvf data/bootstrap cases
+- `regression`: special regression datasets (serde, delimiters, etc.)
+- `test`: lightweight smoke test datasets
+- `preinstalled_hql`: centralized preinstalled HQL scripts (`create_preinstalled_scripts/*.hql`)
+- `view`: view bootstrap (`create_view_scripts/create_view.hql`)
+
+### 3) Bootstrap Group Layer
+
+- `common`: shared items for hive2/hive3
+- `hive2_only`: hive2-only items
+- `hive3_only`: hive3-only items
+
+By default:
+
+- Hive2 uses: `common,hive2_only`
+- Hive3 uses: `common,hive3_only`
+
+This grouping controls which files are selected during `run.sh`/HQL refresh.
+
+## Start/Stop
+
+```bash
+# Start Hive3
+./docker/thirdparties/run-thirdparties-docker.sh -c hive3
+
+# Start Hive2
+./docker/thirdparties/run-thirdparties-docker.sh -c hive2
+
+# Stop Hive3
+./docker/thirdparties/run-thirdparties-docker.sh -c hive3 --stop
+```
+
+## Startup Modes
+
+Use `--hive-mode` to control startup behavior:
+
+- `fast`: reuse existing state as much as possible
+- `refresh` (default): refresh only changed modules by SHA
+- `rebuild`: force reset and rebuild hive state
+
+Examples:
+
+```bash
+# Default mode (refresh)
+./docker/thirdparties/run-thirdparties-docker.sh -c hive3
+
+# Explicit refresh
+./docker/thirdparties/run-thirdparties-docker.sh -c hive3 --hive-mode refresh
+
+# Full rebuild
+./docker/thirdparties/run-thirdparties-docker.sh -c hive3 --hive-mode rebuild
+```
+
+## Module Refresh
+
+Use `--hive-modules` to limit refresh scope:
+
+- `default,multi_catalog,partition_type,statistics,tvf,regression,test,preinstalled_hql,view`
+- `all` means all modules
+
+Examples:
+
+```bash
+# Refresh only preinstalled HQL scripts
+./docker/thirdparties/run-thirdparties-docker.sh -c hive3 --hive-mode refresh --hive-modules preinstalled_hql
+
+# Refresh selected data modules
+./docker/thirdparties/run-thirdparties-docker.sh -c hive3 --hive-mode refresh --hive-modules default,multi_catalog
+```
+
+## Idempotency Rules
+
+To keep `refresh` repeatable:
+
+- `run.sh` scripts should be idempotent
+- HQL should use `DROP ... IF EXISTS` then `CREATE ...`
+- avoid relying on `CREATE ... IF NOT EXISTS` for table/view recreation
+
+## JuiceFS Metadata Backend
+
+Hive now defaults JuiceFS metadata to PostgreSQL (Hive metastore DB), so Hive startup no longer auto-requires MySQL.
+
+- Hive2 default (in `hive-2x_settings.env`):
+  - `postgres://postgres@127.0.0.1:${PG_PORT}/juicefs_meta?sslmode=disable`
+- Hive3 default (in `hive-3x_settings.env`):
+  - `postgres://postgres@127.0.0.1:${PG_PORT}/juicefs_meta?sslmode=disable`
+
+If your environment still needs MySQL metadata, override before startup:
+
+```bash
+export JFS_CLUSTER_META="mysql://root:123456@(127.0.0.1:3316)/juicefs_meta"
+./docker/thirdparties/run-thirdparties-docker.sh -c hive3
+```
+
+## Logs and Debug
+
+- Hive3 startup log: `docker/thirdparties/logs/start_hive3.log`
+- Hive2 startup log: `docker/thirdparties/logs/start_hive2.log`
+
+By default, helper scripts keep xtrace off to reduce log noise.
+Enable debug trace when needed:
+
+```bash
+export HIVE_DEBUG=1
+./docker/thirdparties/run-thirdparties-docker.sh -c hive3 --hive-mode refresh
+```
+
+## Common Troubleshooting
+
+- Metastore health check fails:
+  - check `hive-metastore-postgresql` is healthy
+  - inspect `start_hive3.log` or `start_hive2.log`
+- JuiceFS format/status fails:
+  - verify `JFS_CLUSTER_META` is reachable
+  - ensure target metadata database exists (startup script auto-creates for local MySQL/PostgreSQL)
+- Refresh is unexpectedly slow:
+  - confirm `--hive-mode refresh` is used
+  - use `--hive-modules` to narrow refresh scope

docker/thirdparties/docker-compose/hive/hive-2x.yaml.tpl

@@ -16,8 +16,6 @@
 #
 
 
-version: "3.8"
-
 services:
   namenode:
     image: bde2020/hadoop-namenode:2.0.0-hadoop2.7.4-java8
@@ -35,6 +33,8 @@ services:
       interval: 5s
       timeout: 120s
       retries: 120
+    volumes:
+      - ${HIVE_STATE_ROOT}/namenode:/hadoop/dfs/name
     network_mode: "host"
 
   datanode:
@@ -52,6 +52,8 @@ services:
       interval: 5s
       timeout: 60s
       retries: 120
+    volumes:
+      - ${HIVE_STATE_ROOT}/datanode:/hadoop/dfs/data
     network_mode: "host"
 
   hive-server:
@@ -61,9 +63,12 @@ services:
     environment:
       HIVE_CORE_CONF_javax_jdo_option_ConnectionURL: "jdbc:postgresql://${IP_HOST}:${PG_PORT}/metastore"
       SERVICE_PRECONDITION: "${IP_HOST}:${HMS_PORT}"
+      HIVE_SITE_CONF_hive_aux_jars_path: "file:///mnt/scripts/auxlib/json-serde-1.3.9-SNAPSHOT-jar-with-dependencies.jar"
     container_name: ${CONTAINER_UID}hive2-server
     expose:
       - "${HS_PORT}"
+    volumes:
+      - ./scripts:/mnt/scripts
     depends_on:
       datanode:
         condition: service_healthy
@@ -81,7 +86,7 @@ services:
     image: bde2020/hive:2.3.2-postgresql-metastore
     env_file:
       - ./hadoop-hive-2x.env
-    command: /bin/bash /mnt/scripts/hive-metastore.sh
+    command: /bin/bash /mnt/scripts/start-hive-metastore.sh
     environment:
       SERVICE_PRECONDITION: "${IP_HOST}:50070 ${IP_HOST}:50075 ${IP_HOST}:${PG_PORT}"
       HMS_PORT: "${HMS_PORT}"
@@ -90,6 +95,7 @@ services:
       - "${HMS_PORT}"
     volumes:
       - ./scripts:/mnt/scripts
+      - ${HIVE_STATE_ROOT}/state:/mnt/state
     depends_on:
       hive-metastore-postgresql:
         condition: service_healthy
@@ -105,6 +111,8 @@ services:
     container_name: ${CONTAINER_UID}hive2-metastore-postgresql
     ports:
       - "${PG_PORT}:5432"
+    volumes:
+      - ${HIVE_STATE_ROOT}/pgdata:/var/lib/postgresql/data
     healthcheck:
       test: ["CMD-SHELL", "pg_isready -U postgres"]
       interval: 5s

docker/thirdparties/docker-compose/hive/hive-2x_settings.env

@@ -26,7 +26,7 @@ export HS_PORT=10000 # should be same as hive2ServerPort in regression-conf.groo
 export PG_PORT=5432 # should be same as hive2PgPort in regression-conf.groovy
 
 # JuiceFS metadata endpoint for property `juicefs.cluster.meta`.
-# CI can override this env, e.g.:
+# CI can override this env, e.g. to point at the docker-published mysql_57 port:
 #   export JFS_CLUSTER_META="mysql://user:pwd@(127.0.0.1:3316)/juicefs_meta"
-# default to mysql_57 (3316) because external pipeline always starts mysql, but not redis.
-export JFS_CLUSTER_META="${JFS_CLUSTER_META:-mysql://root:123456@(127.0.0.1:3316)/juicefs_meta}"
+# default to hive metastore postgresql to avoid external mysql dependency.
+export JFS_CLUSTER_META="${JFS_CLUSTER_META:-postgres://postgres@127.0.0.1:${PG_PORT}/juicefs_meta?sslmode=disable}"

docker/thirdparties/docker-compose/hive/hive-3x.yaml.tpl

@@ -16,8 +16,6 @@
 #
 
 
-version: "3.8"
-
 services:
   namenode:
     image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8
@@ -35,6 +33,8 @@ services:
       interval: 5s
       timeout: 120s
       retries: 120
+    volumes:
+      - ${HIVE_STATE_ROOT}/namenode:/hadoop/dfs/name
     network_mode: "host"
 
   datanode:
@@ -52,6 +52,8 @@ services:
       interval: 5s
       timeout: 60s
       retries: 120
+    volumes:
+      - ${HIVE_STATE_ROOT}/datanode:/hadoop/dfs/data
     network_mode: "host"
 
   hive-server:
@@ -63,9 +65,12 @@ services:
       HIVE_CORE_CONF_javax_jdo_option_ConnectionURL: "jdbc:postgresql://${IP_HOST}:${PG_PORT}/metastore"
       SERVICE_PRECONDITION: "${IP_HOST}:${HMS_PORT}"
       JVM_OPTS: -Xmx2g
+      HIVE_SITE_CONF_hive_aux_jars_path: "file:///mnt/scripts/auxlib/json-serde-1.3.9-SNAPSHOT-jar-with-dependencies.jar"
     container_name: ${CONTAINER_UID}hive3-server
     expose:
       - "${HS_PORT}"
+    volumes:
+      - ./scripts:/mnt/scripts
     depends_on:
       datanode:
         condition: service_healthy
@@ -83,7 +88,7 @@ services:
     image: doristhirdpartydocker/hive:3.1.2-postgresql-metastore
     env_file:
       - ./hadoop-hive-3x.env
-    command: /bin/bash /mnt/scripts/hive-metastore.sh
+    command: /bin/bash /mnt/scripts/start-hive-metastore.sh
     environment:
       SERVICE_PRECONDITION: "${IP_HOST}:9870 ${IP_HOST}:9864 ${IP_HOST}:${PG_PORT}"
       HMS_PORT: "${HMS_PORT}"
@@ -92,6 +97,7 @@ services:
       - "${HMS_PORT}"
     volumes:
       - ./scripts:/mnt/scripts
+      - ${HIVE_STATE_ROOT}/state:/mnt/state
       - /tmp/jfs-bucket:/tmp/jfs-bucket
     depends_on:
       hive-metastore-postgresql:
@@ -108,6 +114,8 @@ services:
     container_name: ${CONTAINER_UID}hive3-metastore-postgresql
     ports:
       - "${PG_PORT}:5432"
+    volumes:
+      - ${HIVE_STATE_ROOT}/pgdata:/var/lib/postgresql/data
     healthcheck:
       test: ["CMD-SHELL", "pg_isready -U postgres"]
       interval: 5s

docker/thirdparties/docker-compose/hive/hive-3x_settings.env

@@ -26,9 +26,10 @@ export HS_PORT=13000 # should be same as hive3ServerPort in regression-conf.groo
 export PG_PORT=5732 # should be same as hive3PgPort in regression-conf.groovy
 
 # JuiceFS metadata endpoint for property `juicefs.cluster.meta`.
-# CI can override this env, e.g.:
+# Default to hive metastore postgresql to avoid external mysql dependency.
+# CI can still override this env, e.g.:
 #   export JFS_CLUSTER_META="mysql://user:pwd@(127.0.0.1:3316)/juicefs_meta"
-export JFS_CLUSTER_META="${JFS_CLUSTER_META:-mysql://root:123456@(127.0.0.1:3316)/juicefs_meta}"
+export JFS_CLUSTER_META="${JFS_CLUSTER_META:-postgres://postgres@127.0.0.1:${PG_PORT}/juicefs_meta?sslmode=disable}"
 
 # prepare for paimon hms test,control load paimon hms data or not
 export enablePaimonHms="false"