spark-taskmetrics-ShuffleWriteMetrics.adoc

ShuffleWriteMetrics

ShuffleWriteMetrics is a collection of accumulators that represents task metrics about writing shuffle data.

ShuffleWriteMetrics tracks the following task metrics:

1. Shuffle Bytes Written

2. Shuffle Write Time

3. Shuffle Records Written

Note	Accumulators allow tasks (running on executors) to communicate with the driver.

Table 1. ShuffleWriteMetrics’s Accumulators

Name	Description
_bytesWritten	Accumulator to track how many shuffle bytes were written in a shuffle task. Used when ShuffleWriteMetrics is requested the shuffle bytes written and to increment or decrement it. NOTE: _bytesWritten is available as internal.metrics.shuffle.write.bytesWritten (internally shuffleWrite.BYTES_WRITTEN) in TaskMetrics.
_writeTime	Accumulator to track shuffle write time (as 64-bit integer) of a shuffle task. Used when ShuffleWriteMetrics is requested the shuffle write time and to increment it. NOTE: _writeTime is available as internal.metrics.shuffle.write.writeTime (internally shuffleWrite.WRITE_TIME) in TaskMetrics.
_recordsWritten	Accumulator to track how many shuffle records were written in a shuffle task. Used when ShuffleWriteMetrics is requested the shuffle records written and to increment or decrement it. NOTE: _recordsWritten is available as internal.metrics.shuffle.write.recordsWritten (internally shuffleWrite.RECORDS_WRITTEN) in TaskMetrics.

decRecordsWritten Method

Caution

FIXME

decBytesWritten Method

Caution

FIXME

writeTime Method

Caution

FIXME

recordsWritten Method

Caution

FIXME

Returning Number of Shuffle Bytes Written — bytesWritten Method

bytesWritten: Long

bytesWritten represents the shuffle bytes written metrics of a shuffle task.

Internally, bytesWritten returns the sum of _bytesWritten internal accumulator.

Note

bytesWritten is used when: ShuffleWriteMetricsUIData is created In decBytesWritten StatsReportListener intercepts stage completed events to show shuffle bytes written ShuffleExternalSorter does writeSortedFile (to incDiskBytesSpilled) JsonProtocol converts ShuffleWriteMetrics to JSON ExecutorsListener intercepts task end events to update executor metrics JobProgressListener updates stage and executor metrics

Incrementing Shuffle Bytes Written Metrics — incBytesWritten Method

incBytesWritten(v: Long): Unit

incBytesWritten simply adds v to _bytesWritten internal accumulator.

Note	incBytesWritten is used when: UnsafeShuffleWriter does mergeSpills DiskBlockObjectWriter does updateBytesWritten JsonProtocol creates TaskMetrics from JSON

Incrementing Shuffle Write Time Metrics — incWriteTime Method

incWriteTime(v: Long): Unit

incWriteTime simply adds v to _writeTime internal accumulator.

Note

incWriteTime is used when: SortShuffleWriter stops. BypassMergeSortShuffleWriter writes records (i.e. when it initializes DiskBlockObjectWriter partition writers) and later when concatenates per-partition files into a single file. UnsafeShuffleWriter does mergeSpillsWithTransferTo. DiskBlockObjectWriter does commitAndGet (but only when syncWrites flag is enabled that forces outstanding writes to disk). JsonProtocol creates TaskMetrics from JSON TimeTrackingOutputStream does its operation (after all it is an output stream to track shuffle write time).

Incrementing Shuffle Records Written Metrics — incRecordsWritten Method

incRecordsWritten(v: Long): Unit

incRecordsWritten simply adds v to _recordsWritten internal accumulator.

Note	incRecordsWritten is used when: ShuffleExternalSorter does writeSortedFile DiskBlockObjectWriter does recordWritten JsonProtocol creates TaskMetrics from JSON