Clarification on deployment configuration

[tonyf]

It would be helpful to have further documentation on deployment recommendations in a production setting.

For example:

• Should the parameter server / lighthouse server be colocated for performance? Is it necessary to have high speed interconnect between the lighthouse server and the worker nodes?
• Can a single lighthouse server be shared amongst multiple training jobs? If so, how are the instances/jobs distinguished from each other?
• What kind of minimum specs are recommended for the lighthouse / parameter servers? How does this relate to model size?

[d4l3k]

1. What do you mean by parameter server? The lighthouse is just for metadata + quorum so while you should have a stable low latency connection, infiniband etc is not required. The Lighthouse only communicates over TCP using grpc.
2. A single lighthouse instance can only be used by one training job. There's some prototype PRs that support sharing a Lighthouse but not in the current state. Happy to discuss this if you would find that useful
3. Lighthouse is very lightweight and can run on a single core. If using cloud VMs we recommend running on an instance that has dedicated CPU and not a shared instance to minimize latency and maximize reliability.