spark-ShuffleReader.adoc

ShuffleReader

ShuffleReader is a contract of shuffle readers to read combined key-value records for a reduce task.

package org.apache.spark.shuffle

trait ShuffleReader[K, C] {
  def read(): Iterator[Product2[K, C]]
}

Note	ShuffleReader is a private[spark] contract.

Table 1. ShuffleReader Contract

Method	Description
read	Reading combined key-value records for a reduce task Used when: CoGroupedRDD, ShuffledRDD, and SubtractedRDD are requested to compute a partition (for a ShuffleDependency dependency) Spark SQL’s ShuffledRowRDD is requested to compute a partition

Note	BlockStoreShuffleReader is the one and only known ShuffleReader in Apache Spark.