Add self-play/multi-agent support

[cswinter]

Ideally, we eventually add support for the full PettingZoo multi-agent API. As a much simpler first step that already covers a lot of interesting environments, we could add support for multi-agent environments with simultaneous moves, and a fixed number of actors. The main new feature this would require is a version of Environment which accepts an entire list of actions, and similarly returns a list of observations. This multi-agent environment can the be simply treated as multiple normal environments by ListEnv or ParallelListEnv.

[Bam4d]

I will need this for the grafter environment so I can do this as part of that integration

[cswinter]

Going through required changes in more detail:

• Add a version of Environment class that takes in multiple actions and returns multiple observations. Maybe the simplest solution would be to just modify the Environment class to optionally returns lists of everything, so e.g. type of act changes to Either[Mapping[ActionName, Action], List[Mapping[ActionName, Action]] -> Either[Observation, List[Observation]] and also add another method that returns the number of players. But maybe having separate class would be cleaner.

entity-gym/entity_gym/env/environment.py

Line 199 in 7df3dd0

class Environment(ABC):

• Second step is to modify EnvList (or create a corresponding MultiEnvList) that implements VecEnv for a list of multi-player environments. Should be fairly straightforward to adapt.

entity-gym/entity_gym/env/env_list.py

Line 27 in 7df3dd0

class EnvList(VecEnv):

There's probably a few other places that need small fixes to fully support, enn-trainer needs a method that accepts a multi-player environment and code that instantiates environments needs to be modified a bit to account for that fact that each environment instance yields effectively multiple environments.