spark-core-AppStatusStore.adoc

AppStatusStore

AppStatusStore is…​FIXME

Figure 1. AppStatusStore in Spark Application

AppStatusStore is created when:

• SparkContext is created (that triggers creating a live store (i.e. an AppStatusStore for an active Spark application))

• FsHistoryProvider is requested to create a LoadedAppUI

When created for a live Spark application, AppStatusStore creates a AppStatusListener that is later used to get the active stages.

streamBlocksList Method

streamBlocksList(): Seq[StreamBlockData]

streamBlocksList…​FIXME

Note	streamBlocksList is used when…​FIXME

activeStages Method

activeStages(): Seq[v1.StageData]

activeStages…​FIXME

Note	activeStages is used when…​FIXME

Creating AppStatusStore Instance

AppStatusStore takes the following when created:

• KVStore

• Optional AppStatusListener (default: None)

Creating Live Store (AppStatusStore For Active Spark Application) — createLiveStore Factory Method

createLiveStore(conf: SparkConf): AppStatusStore

createLiveStore creates a fully-initialized AppStatusStore.

Internally, createLiveStore creates a ElementTrackingStore (with a new InMemoryStore and the input SparkConf).

createLiveStore creates a AppStatusListener (with the ElementTrackingStore created, the input SparkConf and the live flag enabled).

In the end, createLiveStore creates an AppStatusStore (with the ElementTrackingStore and AppStatusListener just created).

Note	createLiveStore is used exclusively when SparkContext is created.

Closing AppStatusStore — close Method

close(): Unit

close simply requests KVStore to close.

Note	close is used when…​FIXME

rddList Method

rddList(cachedOnly: Boolean = true): Seq[v1.RDDStorageInfo]

rddList requests the KVStore for a view over RDDStorageInfoWrapper entities.

In the end, rddList takes RDDStorageInfos with at least one partition cached (when cachedOnly flag is on) or all RDDStorageInfos (when cachedOnly flag is off).

Note	cachedOnly flag is on and therefore rddList gives RDDs cached only.

Note	rddList is used when: StoragePage is requested to render itself AbstractApplicationResource is requested to handle storage/rdd REST API StagePagedTable is requested to makeDescription