spark-BlockInfo.adoc

BlockInfo — Metadata of Memory Block

BlockInfo is a metadata of memory block (aka memory page) — the memory block’s size, the number of readers and the id of the writer task.

BlockInfo has a StorageLevel, ClassTag and tellMaster flag.

Size — size Attribute

size attribute is the size of the memory block. It starts with 0.

It represents the number of bytes that BlockManager saved or BlockManager.doPutIterator.

Reader Count — readerCount Counter

readerCount counter is the number of readers of the memory block, i.e. the number of read locks. It starts with 0.

readerCount is incremented when a read lock is acquired and decreases when the following happens:

• The memory block is unlocked

• All locks for the memory block obtained by a task are released.

• The memory block is removed

• Clearing the current state of BlockInfoManager.

Writer Task — writerTask Attribute

writerTask attribute is the task that owns the write lock for the memory block.

A writer task can be one of the three possible identifiers:

• NO_WRITER (i.e. -1) to denote no writers and hence no write lock in use.

• NON_TASK_WRITER (i.e. -1024) for non-task threads, e.g. by a driver thread or by unit test code.

• the task attempt id of the task which currently holds the write lock for this block.

The writer task is assigned in the following scenarios:

• A write lock is requested for a memory block (with no writer and readers)

• A memory block is unlocked

• All locks obtained by a task are released

• A memory block is removed

• Clearing the current state of BlockInfoManager.