spark-RetryingBlockFetcher.adoc

RetryingBlockFetcher

RetryingBlockFetcher is…​FIXME

RetryingBlockFetcher is created and immediately started when:

• NettyBlockTransferService is requested to fetchBlocks (when maxIORetries is greater than 0 which it is by default)

• ExternalShuffleClient is requested to fetchBlocks (when maxIORetries is greater than 0 which it is by default)

RetryingBlockFetcher uses a BlockFetchStarter to createAndStart when requested to start and later initiateRetry.

RetryingBlockFetcher uses outstandingBlocksIds internal registry of outstanding block IDs to fetch that is initially the block IDs to fetch when created.

At initiateRetry, RetryingBlockFetcher prints out the following INFO message to the logs (with the number of outstandingBlocksIds):

Retrying fetch ([retryCount]/[maxRetries]) for [size] outstanding blocks after [retryWaitTime] ms

On onBlockFetchSuccess and onBlockFetchFailure, RetryingBlockFetchListener removes the block ID from outstandingBlocksIds.

RetryingBlockFetcher uses a RetryingBlockFetchListener to remove block IDs from the outstandingBlocksIds internal registry.

Creating RetryingBlockFetcher Instance

RetryingBlockFetcher takes the following when created:

• TransportConf

• BlockFetchStarter

• Block IDs to fetch

• BlockFetchingListener

Starting RetryingBlockFetcher — start Method

void start()

start simply fetchAllOutstanding.

Note	start is used when: NettyBlockTransferService is requested to fetchBlocks (when maxIORetries is greater than 0 which it is by default) ExternalShuffleClient is requested to fetchBlocks (when maxIORetries is greater than 0 which it is by default)

initiateRetry Internal Method

synchronized void initiateRetry()

initiateRetry…​FIXME

Note	initiateRetry is used when: RetryingBlockFetcher is requested to fetchAllOutstanding RetryingBlockFetchListener is requested to onBlockFetchFailure

fetchAllOutstanding Internal Method

void fetchAllOutstanding()

fetchAllOutstanding requests BlockFetchStarter to createAndStart for the outstandingBlocksIds.

Note	fetchAllOutstanding is used when RetryingBlockFetcher is requested to start and initiateRetry.

RetryingBlockFetchListener

RetryingBlockFetchListener is a BlockFetchingListener that RetryingBlockFetcher uses to remove block IDs from the outstandingBlocksIds internal registry.

onBlockFetchSuccess Method

void onBlockFetchSuccess(String blockId, ManagedBuffer data)

Note	onBlockFetchSuccess is part of BlockFetchingListener Contract to…​FIXME.

onBlockFetchSuccess…​FIXME

onBlockFetchFailure Method

void onBlockFetchFailure(String blockId, Throwable exception)

Note	onBlockFetchFailure is part of BlockFetchingListener Contract to…​FIXME.

onBlockFetchFailure…​FIXME