spark-taskscheduler-taskmetrics.adoc

TaskMetrics

TaskMetrics is a collection of metrics tracked during execution of a Task.

TaskMetrics uses accumulators to represent the metrics and offers "increment" methods to increment them.

Note	The local values of the accumulators for a task (as accumulated while the task runs) are sent from the executor to the driver when the task completes (and DAGScheduler re-creates TaskMetrics).

Table 1. Metrics

Property	Name	Type	Description
_memoryBytesSpilled	internal.metrics.memoryBytesSpilled	LongAccumulator	Used in memoryBytesSpilled, incMemoryBytesSpilled
_updatedBlockStatuses	internal.metrics.updatedBlockStatuses	CollectionAccumulator[(BlockId, BlockStatus)]	Used in updatedBlockStatuses, recording updated BlockStatus for a Block, setUpdatedBlockStatuses

Table 2. TaskMetrics’s Internal Registries and Counters

Name	Description
nameToAccums	Internal accumulators indexed by their names. Used when TaskMetrics re-creates TaskMetrics from AccumulatorV2s, …​FIXME NOTE: nameToAccums is a transient and lazy value.
internalAccums	Collection of internal AccumulatorV2 objects. Used when…​FIXME NOTE: internalAccums is a transient and lazy value.
externalAccums	Collection of external AccumulatorV2 objects. Used when TaskMetrics re-creates TaskMetrics from AccumulatorV2s, …​FIXME NOTE: externalAccums is a transient and lazy value.

accumulators Method

Caution

FIXME

mergeShuffleReadMetrics Method

Caution

FIXME

memoryBytesSpilled Method

Caution

FIXME

updatedBlockStatuses Method

Caution

FIXME

setExecutorCpuTime Method

Caution

FIXME

setResultSerializationTime Method

Caution

FIXME

setJvmGCTime Method

Caution

FIXME

setExecutorRunTime Method

Caution

FIXME

setExecutorDeserializeCpuTime Method

Caution

FIXME

setExecutorDeserializeTime Method

Caution

FIXME

setUpdatedBlockStatuses Method

Caution

FIXME

Re-Creating TaskMetrics From AccumulatorV2s — fromAccumulators Method

fromAccumulators(accums: Seq[AccumulatorV2[_, _]]): TaskMetrics

fromAccumulators creates a new TaskMetrics and registers accums as internal and external task metrics (using nameToAccums internal registry).

Internally, fromAccumulators creates a new TaskMetrics. It then splits accums into internal and external task metrics collections (using nameToAccums internal registry).

For every internal task metrics, fromAccumulators finds the metrics in nameToAccums internal registry (of the new TaskMetrics instance), copies metadata, and merges state.

In the end, fromAccumulators adds the external accumulators to the new TaskMetrics instance.

Note	fromAccumulators is used exclusively when DAGScheduler gets notified that a task has finished (and re-creates TaskMetrics).

Recording Memory Bytes Spilled — incMemoryBytesSpilled Method

incMemoryBytesSpilled(v: Long): Unit

incMemoryBytesSpilled adds v to _memoryBytesSpilled task metrics.

Note

incMemoryBytesSpilled is used when: Aggregator updates task metrics CoGroupedRDD computes a Partition BlockStoreShuffleReader reads combined key-value records for a reduce task ShuffleExternalSorter frees execution memory by spilling to disk ExternalSorter writes the records into a temporary partitioned file in the disk store UnsafeExternalSorter spills current records due to memory pressure SpillableIterator spills records to disk JsonProtocol creates TaskMetrics from JSON

Recording Updated BlockStatus For Block — incUpdatedBlockStatuses Method

incUpdatedBlockStatuses(v: (BlockId, BlockStatus)): Unit

incUpdatedBlockStatuses adds v in _updatedBlockStatuses internal registry.

Note	incUpdatedBlockStatuses is used exclusively when BlockManager does addUpdatedBlockStatusToTaskMetrics.

Registering Internal Accumulators — register Method

register(sc: SparkContext): Unit

register registers the internal accumulators (from nameToAccums internal registry) with countFailedValues enabled (true).

Note	register is used exclusively when Stage is requested for its new attempt.