spark-BlockTransferService.adoc

BlockTransferService — Pluggable Block Transfers (To Fetch and Upload Blocks)

BlockTransferService is the base for ShuffleClients that can fetch and upload blocks of data synchronously or asynchronously.

package org.apache.spark.network

abstract class BlockTransferService extends ShuffleClient {
  // only required methods that have no implementation
  // the others follow
  def init(blockDataManager: BlockDataManager): Unit
  def close(): Unit
  def port: Int
  def hostName: String
  def fetchBlocks(
    host: String,
    port: Int,
    execId: String,
    blockIds: Array[String],
    listener: BlockFetchingListener,
    tempFileManager: TempFileManager): Unit
  def uploadBlock(
    hostname: String,
    port: Int,
    execId: String,
    blockId: BlockId,
    blockData: ManagedBuffer,
    level: StorageLevel,
    classTag: ClassTag[_]): Future[Unit]
}

Note	BlockTransferService is a private[spark] contract.

Table 1. (Subset of) BlockTransferService Contract

Method	Description
init	Used when…​FIXME
close	Used when…​FIXME
port	Used when…​FIXME
hostName	Used when…​FIXME
fetchBlocks	Fetches a sequence of blocks from a remote node asynchronously Used exclusively when BlockTransferService is requested to fetch only one block (in a blocking fashion) Note fetchBlocks is part of ShuffleClient Contract to…​FIXME.
uploadBlock	Used exclusively when BlockTransferService is requested to upload a single block to a remote node (in a blocking fashion).

Note	NettyBlockTransferService is the one and only known implementation of BlockTransferService Contract.

Note	BlockTransferService was introduced in SPARK-3019 Pluggable block transfer interface (BlockTransferService) and is available since Spark 1.2.0.

fetchBlockSync Method

fetchBlockSync(
  host: String,
  port: Int,
  execId: String,
  blockId: String,
  tempFileManager: TempFileManager): ManagedBuffer

fetchBlockSync…​FIXME

Synchronous (and hence blocking) fetchBlockSync to fetch one block blockId (that corresponds to the ShuffleClient parent’s asynchronous fetchBlocks).

fetchBlockSync is a mere wrapper around fetchBlocks to fetch one blockId block that waits until the fetch finishes.

Note	fetchBlockSync is used when…​FIXME

Uploading Single Block to Remote Node (Blocking Fashion) — uploadBlockSync Method

uploadBlockSync(
  hostname: String,
  port: Int,
  execId: String,
  blockId: BlockId,
  blockData: ManagedBuffer,
  level: StorageLevel,
  classTag: ClassTag[_]): Unit

uploadBlockSync…​FIXME

uploadBlockSync is a mere blocking wrapper around uploadBlock that waits until the upload finishes.

Note	uploadBlockSync is used exclusively when BlockManager is requested to replicate (when a replication level is greater than 1).