[improvement](build) Speed up Hive thirdparty startup with named volumes, OSS baseline, and parallel HQL refresh

[suxiaogang223]

What problem does this PR solve?

Related Issue: #62101

Problem Summary:

Hive thirdparty startup in docker/thirdparties was slow, stateful in hard-to-reason ways, and hard to iterate on in CI. This PR restructures it so that:

• a fresh CI host can restore a ready-to-use Hive stack from a pre-built baseline in under a minute, instead of paying the full HDFS + HMS + HQL bootstrap cost every run,
• repeated local runs only re-execute the HQL / run.sh files that actually changed,
• Hive2 and Hive3 no longer share any on-disk state, so the two versions can be started, rebuilt, or refreshed independently.

Main changes

1. Docker named volumes replace bind mounts. Hive NameNode, DataNode, Postgres metastore, and the Hive state dir now live in four Docker named volumes per version (<CONTAINER_UID><hive_version>-{namenode,datanode,pgdata,state}). This makes volume lifecycle explicit (docker volume create/rm), avoids host-path permission issues, and lets us snapshot/restore volumes atomically.

2. OSS baseline snapshot + auto-download. On first start, if the volumes are empty, the script downloads a pre-tarred baseline (<hive_version>-baseline-<arch>.tar.gz) from a configurable OSS prefix (defaults to doris-thirdparty.oss-cn-beijing.aliyuncs.com/thirdparties/hive-baseline) and unpacks it directly into the four volumes via a single alpine tar container. Tarballs are cached under /tmp/hive-baseline-cache/ and keyed per Hive version. snapshot-hive-baseline.sh is the inverse operation used to produce the tarball.

3. Structured startup modes (--hive-mode fast|refresh|rebuild).

   • fast: skip compose up if the stack is already healthy; skip data refresh entirely.
   • refresh (default): skip compose up if healthy; re-run only modules / HQL files whose SHA changed.
   • rebuild: tear down compose, wipe the four volumes, full cold start.

4. Module-scoped, SHA-based incremental refresh. Nine modules (default, multi_catalog, partition_type, statistics, tvf, regression, test, preinstalled_hql, view) each track a content SHA under /mnt/state/modules/. Only modules with a SHA mismatch get re-run. --hive-modules limits refresh to a subset.

5. Parallel preinstalled_hql execution. The ~77 HQL files in create_preinstalled_scripts/ are now refreshed in two phases — serial SHA check, then xargs -P ${LOAD_PARALLEL} for the files that actually changed. Combined with a beeline-backed hive shim on PATH (avoids the per-call Hive CLI JVM cold-start), this is the single largest wall-clock win.

6. Bootstrap groups (common, hive2_only, hive3_only). Files that only belong to one Hive version are listed under scripts/bootstrap/<group>.<kind>.list and skipped for the other version, so Hive2 no longer runs Hive3-only HQL and vice versa.

7. HQL idempotency. All create_preinstalled_scripts/*.hql and module create_table.hql files are rewritten to DROP TABLE IF EXISTS + CREATE, so reruns are safe whether the baseline is fresh or already populated.

8. Logs and observability. Per-phase timing (compose up, init-hive-baseline, refresh-hive-modules, per-HQL BEGIN/END took=Ns) is printed to the component log. Verbose xtrace is gated by HIVE_DEBUG=1.

9. Documentation. New Hive docker README covers architecture, modes, modules, bootstrap groups, how to add test data, and troubleshooting.

How to use

# Default (refresh mode, all modules)
./docker/thirdparties/run-thirdparties-docker.sh -c hive3

# Fast mode — stack is healthy, no data changes
./docker/thirdparties/run-thirdparties-docker.sh -c hive3 --hive-mode fast

# Rebuild from scratch (wipes volumes; re-downloads baseline if available)
./docker/thirdparties/run-thirdparties-docker.sh -c hive3 --hive-mode rebuild

# Refresh only one module
./docker/thirdparties/run-thirdparties-docker.sh -c hive3 \
  --hive-mode refresh --hive-modules preinstalled_hql

Release note

None

Check List (For Author)

• Test: Manual test on aliyun dev machine
  • --hive-mode refresh (healthy stack, no change) — data refresh ~6-7s
  • --hive-mode refresh (HQL content change) — only changed files re-run in parallel
  • --hive-mode rebuild — full cold start from empty volumes
  • --hive-mode fast — stack reuse, ~1-2s
  • Baseline restore from OSS for both hive2 and hive3, then healthy compose up
• Behavior changed: Yes
  • Hive state now lives in Docker named volumes, not bind mounts
  • Startup follows --hive-mode / --hive-modules semantics
• Does this need documentation: No (README included)

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[suxiaogang223]

run external

[suxiaogang223]

run external

[suxiaogang223]

run external

[suxiaogang223]

run buildall

[suxiaogang223]

run external

[suxiaogang223]

run external

[suxiaogang223]

run external