spark-webui-WebUI.adoc

WebUI — Base Web UI

WebUI is the base of the web UIs in Apache Spark:

• Active Spark applications

• Spark History Server

• Spark Standalone cluster manager

• Spark on Mesos cluster manager

Note	Spark on YARN uses a different web framework for the web UI.

WebUI is used as the parent of WebUITabs.

package org.apache.spark.ui

abstract class WebUI {
  // only required methods that have no implementation
  // the others follow
  def initialize(): Unit
}

Note	WebUI is a private[spark] contract.

Table 1. (Subset of) WebUI Contract

Method	Description
initialize	Used in implementations only to let them initialize their web components Note initialize does not add anything special to the Scala type hierarchy but a common name to use across WebUIs (that could also be possible without it). In other words, initialize does not participate in any design pattern or a type hierarchy.

WebUI is a Scala abstract class and cannot be created directly, but only as one of the web UIs.

Table 2. WebUIs

WebUI	Description
HistoryServer	Used in Spark History Server
MasterWebUI	Used in Spark Standalone cluster manager
MesosClusterUI	Used in Spark on Mesos cluster manager
SparkUI	WebUI of a Spark application
WorkerWebUI	Used in Spark Standalone cluster manager

Once bound to a Jetty HTTP server, WebUI is available at an HTTP port (and is used in the web URL as boundPort).

WebUI is available at a web URL, i.e. http://[publicHostName]:[boundPort]. The publicHostName is…​FIXME and the boundPort is the port that the port the Jetty HTTP Server bound to.

Table 3. WebUI’s Internal Properties (e.g. Registries, Counters and Flags)

Name	Description
tabs	WebUITabs Used when…​FIXME
handlers	ServletContextHandlers Used when…​FIXME
pageToHandlers	ServletContextHandlers per WebUIPage Used when…​FIXME
serverInfo	Optional ServerInfo (default: None) Used when…​FIXME
publicHostName	Host name of the UI publicHostName is either SPARK_PUBLIC_DNS environment variable or spark.driver.host configuration property. Defaults to the following if defined (in order): SPARK_LOCAL_HOSTNAME environment variable Host name of SPARK_LOCAL_IP environment variable Utils.findLocalInetAddress Used exclusively when WebUI is requested for the web URL
className	Used when…​FIXME

Tip	Enable INFO or ERROR logging level for the corresponding loggers of the WebUIs, e.g. org.apache.spark.ui.SparkUI, to see what happens inside. Add the following line to conf/log4j.properties: log4j.logger.org.apache.spark.ui=INFO Refer to Logging.

Creating WebUI Instance

WebUI takes the following when created:

• SecurityManager

• SSLOptions

• Port number

• SparkConf

• basePath (default: empty)

• Name (default: empty)

WebUI initializes the internal registries and counters.

Note	WebUI is a Scala abstract class and cannot be created directly, but only as one of the implementations.

Detaching Page And Associated Handlers from UI — detachPage Method

detachPage(page: WebUIPage): Unit

detachPage…​FIXME

Note	detachPage is used when…​FIXME

Detaching Tab And Associated Pages from UI — detachTab Method

detachTab(tab: WebUITab): Unit

detachTab…​FIXME

Note	detachTab is used when…​FIXME

Detaching Handler — detachHandler Method

detachHandler(handler: ServletContextHandler): Unit

detachHandler…​FIXME

Note	detachHandler is used when…​FIXME

Detaching Handler At Path — detachHandler Method

detachHandler(path: String): Unit

detachHandler…​FIXME

Note	detachHandler is used when…​FIXME

Attaching Page to UI — attachPage Method

attachPage(page: WebUIPage): Unit

Internally, attachPage creates the path of the WebUIPage that is / (forward slash) followed by the prefix of the page.

attachPage creates a HTTP request handler…​FIXME

Note	attachPage is used when: WebUI is requested to attach a WebUITab (the WebUIPages actually) HistoryServer, Spark Standalone’s MasterWebUI and WorkerWebUI, Spark on Mesos' MesosClusterUI are requested to initialize

Attaching Tab And Associated Pages to UI — attachTab Method

attachTab(tab: WebUITab): Unit

attachTab attaches every WebUIPage of the input WebUITab.

In the end, attachTab adds the input WebUITab to WebUITab tabs.

Note	attachTab is used when…​FIXME

Attaching Static Handler — addStaticHandler Method

addStaticHandler(resourceBase: String, path: String): Unit

addStaticHandler…​FIXME

Note	addStaticHandler is used when…​FIXME

Attaching Handler to UI — attachHandler Method

attachHandler(handler: ServletContextHandler): Unit

attachHandler simply adds the input Jetty ServletContextHandler to handlers registry and requests the ServerInfo to addHandler (only if the ServerInfo is defined).

Note

attachHandler is used when: web UIs (i.e. HistoryServer, Spark Standalone’s MasterWebUI and WorkerWebUI, Spark on Mesos' MesosClusterUI, SparkUI) are requested to initialize WebUI is requested to attach a page to web UI and addStaticHandler SparkContext is created (and attaches the driver metrics servlet handler to the web UI) HistoryServer is requested to attachSparkUI Spark Standalone’s Master and Worker are requested to onStart (and attach their metrics servlet handlers to the web ui)

getBasePath Method

getBasePath: String

getBasePath simply returns the base path.

Note	getBasePath is used exclusively when WebUITab is requested for the base path.

Requesting Header Tabs — getTabs Method

getTabs: Seq[WebUITab]

getTabs simply returns the registered tabs.

Note	getTabs is used exclusively when WebUITab is requested for the header tabs.

Requesting Handlers — getHandlers Method

getHandlers: Seq[ServletContextHandler]

getHandlers simply returns the registered handlers.

Note	getHandlers is used when…​FIXME

Binding UI to Jetty HTTP Server on Host — bind Method

bind(): Unit

bind…​FIXME

Note	bind is used when…​FIXME

Stopping UI — stop Method

stop(): Unit

stop…​FIXME

Note	stop is used when…​FIXME