Skip to content

Commit 9bc830e

Browse files
authored
[GLUTEN-12058][VL] Skip stage ResourceProfile adjustment rule when off-heap memory is disabled (#12060)
1 parent cad164a commit 9bc830e

2 files changed

Lines changed: 8 additions & 1 deletion

File tree

docs/get-started/VeloxStageResourceAdj.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,11 @@ To address this issue, Apache Gluten introduces a stage-level resource auto-adju
2020
```properties
2121
spark.dynamicAllocation.enabled=true
2222
```
23-
3. **Resource Scheduler Compatibility**:
23+
3. ** Enable Off-Heap Memory**:
24+
```properties
25+
spark.memory.offHeap.enabled=true
26+
```
27+
4. **Resource Scheduler Compatibility**:
2428
Ensure the underlying cluster resource manager (e.g., YARN, Kubernetes) supports dynamic resource allocation.
2529

2630
### **Key Configurations**

gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,9 @@ case class GlutenAutoAdjustStageResourceProfile(glutenConf: GlutenConfig, spark:
5858
if (!SQLConf.get.adaptiveExecutionEnabled) {
5959
return plan
6060
}
61+
if (!sparkConf.getBoolean(GlutenCoreConfig.SPARK_OFFHEAP_ENABLED_KEY, defaultValue = false)) {
62+
return plan
63+
}
6164
// Starting here, the resource profile may differ between stages. Configure resource settings
6265
// using the default profile to prevent any impact from the previous stage. If a new resource
6366
// profile is applied, the settings will be updated accordingly.

0 commit comments

Comments
 (0)