spark-webui-SparkUI.adoc

SparkUI — Web UI of Spark Application

SparkUI is the web UI of a Spark application (aka Application UI).

SparkUI is created along with the following:

• SparkContext (for a live Spark application with spark.ui.enabled configuration property enabled)

• FsHistoryProvider is requested for the application UI (for a live or completed Spark application)

Figure 1. Creating SparkUI for Live Spark Application

When created (while SparkContext is created for a live Spark application), SparkUI gets the following:

• Live AppStatusStore (with a ElementTrackingStore using an InMemoryStore and a live AppStatusListener)

• Name of the Spark application that is exactly the value of spark.app.name configuration property

• Empty base path

When started, SparkUI binds to appUIAddress address that you can control using SPARK_PUBLIC_DNS environment variable or spark.driver.host Spark property.

Note	With spark.ui.killEnabled configuration property turned on, SparkUI allows to kill jobs and stages (subject to SecurityManager.checkModifyPermissions permissions).

SparkUI gets an AppStatusStore that is then used for the following:

• Initializing tabs, i.e. JobsTab, StagesTab, StorageTab, EnvironmentTab

• AbstractApplicationResource is requested for jobsList, oneJob, executorList, allExecutorList, rddList, rddData, environmentInfo

• StagesResource is requested for stageList, stageData, oneAttemptData, taskSummary, taskList

• SparkUI is requested for the current Spark user

• Creating Spark SQL’s SQLTab (when SQLHistoryServerPlugin is requested to setupUI)

• Spark Streaming’s BatchPage is created

Table 1. SparkUI’s Internal Properties (e.g. Registries, Counters and Flags)

Name	Description
appId

Tip	Enable INFO logging level for org.apache.spark.ui.SparkUI logger to see what happens inside. Add the following line to conf/log4j.properties: log4j.logger.org.apache.spark.ui.SparkUI=INFO Refer to Logging.

Assigning Unique Identifier of Spark Application — setAppId Method

setAppId(id: String): Unit

setAppId sets the internal appId.

Note	setAppId is used exclusively when SparkContext is initialized.

Stopping SparkUI — stop Method

stop(): Unit

stop stops the HTTP server and prints the following INFO message to the logs:

INFO SparkUI: Stopped Spark web UI at [appUIAddress]

Note	appUIAddress in the above INFO message is the result of appUIAddress method.

appUIAddress Method

appUIAddress: String

appUIAddress returns the entire URL of a Spark application’s web UI, including http:// scheme.

Internally, appUIAddress uses appUIHostPort.

Accessing Spark User — getSparkUser Method

getSparkUser: String

getSparkUser returns the name of the user a Spark application runs as.

Internally, getSparkUser requests user.name System property from EnvironmentListener Spark listener.

Note	getSparkUser is used…​FIXME

createLiveUI Method

createLiveUI(
  sc: SparkContext,
  conf: SparkConf,
  listenerBus: SparkListenerBus,
  jobProgressListener: JobProgressListener,
  securityManager: SecurityManager,
  appName: String,
  startTime: Long): SparkUI

createLiveUI creates a SparkUI for a live running Spark application.

Internally, createLiveUI simply forwards the call to create.

Note	createLiveUI is called when SparkContext is created (and spark.ui.enabled is enabled).

createHistoryUI Method

Caution

FIXME

appUIHostPort Method

appUIHostPort: String

appUIHostPort returns the Spark application’s web UI which is the public hostname and port, excluding the scheme.

Note	appUIAddress uses appUIHostPort and adds http:// scheme.

getAppName Method

getAppName: String

getAppName returns the name of the Spark application (of a SparkUI instance).

Note	getAppName is used when…​FIXME

Creating SparkUI Instance — create Factory Method

create(
  sc: Option[SparkContext],
  store: AppStatusStore,
  conf: SparkConf,
  securityManager: SecurityManager,
  appName: String,
  basePath: String = "",
  startTime: Long,
  appSparkVersion: String = org.apache.spark.SPARK_VERSION): SparkUI

create creates a SparkUI backed by a AppStatusStore.

Internally, create simply creates a new SparkUI (with the predefined Spark version).

Note	create is used when: SparkContext is created (for a running Spark application) FsHistoryProvider is requested to getAppUI (for a Spark application that already finished)

Creating SparkUI Instance

SparkUI takes the following when created:

• AppStatusStore

• SparkContext

• SparkConf

• SecurityManager

• Application name

• basePath

• Start time

• appSparkVersion

SparkUI initializes the internal registries and counters and the tabs and handlers.

Attaching Tabs and Context Handlers — initialize Method

initialize(): Unit

Note	initialize is part of WebUI Contract to initialize web components.

initialize creates and attaches the following tabs (with the reference to the SparkUI and its AppStatusStore):

1. JobsTab

2. StagesTab

3. StorageTab

4. EnvironmentTab

5. ExecutorsTab

In the end, initialize creates and attaches the following ServletContextHandlers:

1. Creates a static handler for serving files from a static directory, i.e. /static to serve static files from org/apache/spark/ui/static directory (on CLASSPATH)

2. Creates a redirect handler to redirect / to /jobs/ (and so the Jobs tab is the welcome tab when you open the web UI)

3. Creates the /api/* context handler for the Status REST API

4. Creates a redirect handler to redirect /jobs/job/kill to /jobs/ and request the JobsTab to execute handleKillRequest before redirection

5. Creates a redirect handler to redirect /stages/stage/kill to /stages/ and request the StagesTab to execute handleKillRequest before redirection