spark-streaming-BlockRDD.adoc

BlockRDD

BlockRDD is an RDD that is created when Spark Streaming’s ReceiverInputDStream is requested to compute and createBlockRDD.

Spark Streaming calls BlockRDD.removeBlocks() while clearing metadata.

Note	It appears that BlockRDD is used in Spark Streaming exclusively.

Computing Partition (in TaskContext) — compute Method

compute(split: Partition, context: TaskContext): Iterator[T]

Note	compute is part of RDD Contract to compute a partition (in a TaskContext).

compute…​FIXME

getPartitions Method

getPartitions: Array[Partition]

Note	getPartitions is part of RDD Contract to…​FIXME.

getPartitions…​FIXME

getPreferredLocations Method

getPreferredLocations(split: Partition): Seq[String]

Note	getPreferredLocations is part of RDD Contract to…​FIXME.

getPreferredLocations…​FIXME

Creating BlockRDD Instance

BlockRDD takes the following when created:

• SparkContext

• Collection of BlockIds

BlockRDD initializes the internal registries and counters.