Skip to content

spark-webui-executors.adoc

Latest commit

 

History

History
58 lines (42 loc) · 1.97 KB

spark-webui-executors.adoc

File metadata and controls

58 lines (42 loc) · 1.97 KB

Executors Tab

Executors tab in web UI shows…​FIXME

spark webui executors
Figure 1. Executors Tab in web UI (local mode)

The Executors tab is available under /executors URL, i.e. http://localhost:4040/executors.

Internally, the Executors tab is represented by ExecutorsTab.

Note

What’s interesting in how Storage Memory is displayed in the Executors tab is that the default UnifiedMemoryManager calculates the maximum memory in a way that is different from what the page displays (using the custom JavaScript formatBytes function in utils.js).

// local mode with spark.driver.memory 2g
// ./bin/spark-shell --conf spark.driver.memory=2g
// UnifiedMemoryManager reports 912MB
// You can see it after enabling INFO messages for BlockManagerMasterEndpoint

INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.8:54503 with 912.3 MB RAM, BlockManagerId(driver, 192.168.1.8, 54503, None)

// custom JavaScript `formatBytes` function (from utils.js) reports...956.6MB
// See https://github.com/apache/spark/blob/master/core/src/main/resources/org/apache/spark/ui/static/utils.js#L40-L48
def formatBytes(bytes: Double) = {
  val k = 1000
  val i = math.floor(math.log(bytes) / math.log(k))
  val maxMemoryWebUI = bytes / math.pow(k, i)
  f"$maxMemoryWebUI%1.1f"
}
scala> println(formatBytes(maxMemory))
956.6

getExecInfo Method

getExecInfo(
  listener: ExecutorsListener,
  statusId: Int,
  isActive: Boolean): ExecutorSummary

getExecInfo creates a ExecutorSummary.

Caution
 FIXME
Note
 getExecInfo is used when…​FIXME

Settings

spark.ui.threadDumpsEnabled

spark.ui.threadDumpsEnabled (default: true) is to enable (true) or disable (false) ExecutorThreadDumpPage.