|
| 1 | +--- |
| 2 | +{ |
| 3 | + "title": "Doris Operator Lifecycle Management", |
| 4 | + "sidebar_label": "Lifecycle Management", |
| 5 | + "language": "en", |
| 6 | + "description": "Explains the main behavior of Doris Operator during cluster creation, scaling, configuration changes, rolling updates, and deletion.", |
| 7 | + "keywords": ["Doris Operator lifecycle", "scale out", "scale in", "rolling update", "configuration change", "DorisCluster", "DorisDisaggregatedCluster"] |
| 8 | +} |
| 9 | +--- |
| 10 | + |
| 11 | +Doris Operator manages Doris clusters through the Reconcile loop. After a Doris custom resource is changed, the Operator updates underlying Kubernetes resources according to the new desired state and, when needed, performs Doris metadata-level actions. |
| 12 | + |
| 13 | +This document explains the operational semantics behind those changes. |
| 14 | + |
| 15 | +## Cluster creation |
| 16 | + |
| 17 | +When you create a `DorisCluster` or `DorisDisaggregatedCluster`, the Operator creates the underlying Kubernetes resources according to the component configuration in the resource. |
| 18 | + |
| 19 | +For compute-storage integrated clusters, FE is usually created first: |
| 20 | + |
| 21 | + |
| 22 | + |
| 23 | +For compute-storage decoupled clusters, MetaService is created first: |
| 24 | + |
| 25 | + |
| 26 | + |
| 27 | +If dependencies are not ready, the Operator keeps waiting and retries in later Reconcile loops. |
| 28 | + |
| 29 | +## Scale out |
| 30 | + |
| 31 | +Scaling out is usually done by increasing a component's `replicas` field. For example: |
| 32 | + |
| 33 | +```yaml |
| 34 | +spec: |
| 35 | + beSpec: |
| 36 | + replicas: 5 |
| 37 | +``` |
| 38 | +
|
| 39 | +After the Operator detects the change, it updates the corresponding `StatefulSet`. Kubernetes creates the new Pods, and the Operator continues checking readiness and updating CR `status`. |
| 40 | + |
| 41 | +For compute-storage decoupled clusters, scaling out a ComputeGroup is done in the same way: |
| 42 | + |
| 43 | +```yaml |
| 44 | +spec: |
| 45 | + computeGroups: |
| 46 | + - uniqueId: adhoc-query |
| 47 | + replicas: 5 |
| 48 | +``` |
| 49 | + |
| 50 | +Pay attention to: |
| 51 | + |
| 52 | +- Whether the cluster has enough CPU, memory, and storage. |
| 53 | +- Whether new Pods can be scheduled. |
| 54 | +- Whether PVCs can be bound. |
| 55 | +- Whether new nodes can register with the Doris cluster. |
| 56 | + |
| 57 | +## Scale in |
| 58 | + |
| 59 | +Scale-in is riskier than scale-out because it may affect data replicas, metadata quorum, node roles, or service capacity. |
| 60 | + |
| 61 | +:::caution Caution |
| 62 | +Before scaling in a production cluster, confirm Doris replica status, business traffic, and rollback options. |
| 63 | +::: |
| 64 | + |
| 65 | +### Compute-storage integrated clusters |
| 66 | + |
| 67 | +Scale-in risks differ by component: |
| 68 | + |
| 69 | +| Component | Main concern | |
| 70 | +| --- | --- | |
| 71 | +| FE | Follower and Observer roles can affect metadata quorum | |
| 72 | +| BE | Replica migration and data availability must be considered | |
| 73 | +| CN | No data replicas, but scale-in affects compute capacity and cache | |
| 74 | +| Broker | Check whether external access tasks still depend on it | |
| 75 | + |
| 76 | +For FE, the replica count cannot be lower than the number of election nodes. For scale-in in this mode, evaluate cluster topology and Doris-level risk separately. |
| 77 | + |
| 78 | +### Compute-storage decoupled clusters |
| 79 | + |
| 80 | +When scaling in a ComputeGroup, Doris metadata-level actions may be required. The behavior depends on `enableDecommission`: |
| 81 | + |
| 82 | +| Configuration | Behavior | |
| 83 | +| --- | --- | |
| 84 | +| `enableDecommission: true` | Run decommission before scale-in and wait for safe removal | |
| 85 | +| `enableDecommission: false` | Directly drop the corresponding node | |
| 86 | + |
| 87 | + |
| 88 | + |
| 89 | +Before scaling in, confirm current data distribution and business traffic. |
| 90 | + |
| 91 | +## Configuration changes |
| 92 | + |
| 93 | +Doris startup configuration is usually mounted through `ConfigMap`. Whether a change requires a restart depends on the configuration type and Operator settings. |
| 94 | + |
| 95 | +For compute-storage integrated clusters: |
| 96 | + |
| 97 | +```yaml |
| 98 | +spec: |
| 99 | + enableRestartWhenConfigChange: true |
| 100 | +``` |
| 101 | + |
| 102 | +When this is enabled, core ConfigMap changes can trigger a rolling restart. |
| 103 | + |
| 104 | +Check the following when changing configuration: |
| 105 | + |
| 106 | +- Whether ConfigMap keys match component requirements, such as `fe.conf`. |
| 107 | +- Whether configured directories match PVC mount paths. |
| 108 | +- Whether ports, FQDN, and authentication settings match the Kubernetes network model. |
| 109 | +- Whether the change takes effect only after component restart. |
| 110 | + |
| 111 | +## Rolling updates |
| 112 | + |
| 113 | +Changing component images, some Pod template fields, or configuration hashes can trigger a StatefulSet rolling update. |
| 114 | + |
| 115 | +Recommended practice: |
| 116 | + |
| 117 | +- Perform the update during off-peak hours. |
| 118 | +- Make sure the cluster has no unresolved failures first. |
| 119 | +- Confirm client retry behavior. |
| 120 | +- Follow Doris version upgrade documentation for upgrade order. |
| 121 | + |
| 122 | +## Cluster deletion |
| 123 | + |
| 124 | +After the Doris custom resource is deleted, the Operator enters cleanup flow and removes Kubernetes resources that it manages. |
| 125 | + |
| 126 | +Before deletion, confirm: |
| 127 | + |
| 128 | +- Whether PVCs should be retained. |
| 129 | +- Whether object storage, FoundationDB, or other external dependencies are shared. |
| 130 | +- Whether data and metadata backups are needed. |
| 131 | +- Whether clients are still connected. |
| 132 | + |
| 133 | +:::caution Caution |
| 134 | +Deleting the CR can remove the corresponding Kubernetes resources. Confirm cleanup scope and backup strategy before proceeding. |
| 135 | +::: |
| 136 | + |
| 137 | +## Observing lifecycle operations |
| 138 | + |
| 139 | +Lifecycle completion should be judged using CR `status`, Kubernetes resource state, and Doris component state together. |
| 140 | + |
| 141 | +```shell |
| 142 | +kubectl get dcr -n ${namespace} |
| 143 | +kubectl describe dcr ${cluster_name} -n ${namespace} |
| 144 | +kubectl get ddc -n ${namespace} |
| 145 | +kubectl describe ddc ${cluster_name} -n ${namespace} |
| 146 | +kubectl get pod,sts,svc,pvc -n ${namespace} |
| 147 | +``` |
| 148 | + |
| 149 | +If status does not converge for a long time, continue with Operator logs and Kubernetes Events to determine whether the problem is in scheduling, storage binding, configuration mounting, node registration, or Doris metadata operations. |
0 commit comments