spark-ShuffleClient.adoc

ShuffleClient — Contract to Fetch Shuffle Blocks

ShuffleClient is the contract of clients that can fetch shuffle block files.

ShuffleClient can optionally be initialized with an appId (that actually does nothing by default)

ShuffleClient has shuffle-related Spark metrics that are used when BlockManager is requested for a shuffle-related Spark metrics source (only when Executor is created for a non-local / cluster mode).

package org.apache.spark.network.shuffle;

abstract class ShuffleClient implements Closeable {
  // only required methods that have no implementation
  // the others follow
  abstract void fetchBlocks(
      String host,
      int port,
      String execId,
      String[] blockIds,
      BlockFetchingListener listener,
      TempFileManager tempFileManager);
}

Table 1. (Subset of) ShuffleClient Contract

Method	Description
fetchBlocks	Fetches a sequence of blocks from a remote block manager node asynchronously Used exclusively when ShuffleBlockFetcherIterator is requested to sendRequest

Table 2. ShuffleClients

ShuffleClient	Description
BlockTransferService
ExternalShuffleClient

init Method

void init(String appId)

init does nothing by default.

Note	init is used when: BlockManager is requested to initialize Spark on Mesos' MesosCoarseGrainedSchedulerBackend is requested to registered

Requesting Shuffle-Related Spark Metrics — shuffleMetrics Method

MetricSet shuffleMetrics()

shuffleMetrics returns an empty Dropwizard Metrics' MetricSet by default.

Note	shuffleMetrics is used exclusively when BlockManager is requested for a shuffle-related Spark metrics source (only when Executor is created for a non-local / cluster mode).