Streams Application Master

The Streams Application Master is a Thrift server that registers with the Resource Manager once it starts operation. Post-launch it starts a timer thread which periodically (every streams.yarn.am.heartbeat) pings the Resource Manager via the Streams Container Manager. In addition, it contains a Streams Instance Manager object (indexed by the instance ID) for each Streams instance running within YARN. Finally, for each instance it invokes corresponding methods from the Streams Instance Manager to start, stop, list, add a host to, and remove a host from a Streams instance.

Container Manager

The Container Manager implements the IContainerManager interface (See Streams Instance Manager) and communicates with the Resource Manager via AMRMClientImpl and with Node Managers via NMClientImpl. It internally maintains state in 4 data structures:

1. allocatedContainerSet: Set of the IDs of all allocated containers
2. containerMap: Contains the state of each allocated container indexed by container ID
3. pendingRequests: List of all pending container requests (sent to RM)
4. requestQueue: Queue of all container requests (not sent to RM)

Container requests are added via AMRMClientImpl#addContainerRequest while containers are removed via AMRMClientImpl#releaseAssignedContainer.

The Container Manager periodically pings the Resource Manager via AMRMClientImpl#allocate and takes actions based on AllocateResponse. If it is allocated new containers, it matches them against pendingRequests and deploys Streams services within them via NMClientImpl#startContainer.