Motivation

When using the Hydra config system to instantiate collectors with policies that include
exploration modules (e.g., AdditiveGaussianModule), the exploration module cannot be
properly configured because it requires an action_spec at construction time. However,
in config-based workflows, the policy is instantiated before the environment is created
inside the collector, making the spec unavailable.

The workflow that should be possible:

1. Policy configs (including exploration modules) are defined in YAML
2. Hydra instantiates the policy from configs
3. The collector creates environments internally
4. The exploration module needs the env's action_spec to clip/bound noisy actions

Currently, this fails for several reasons:

1. AdditiveGaussianModule.__init__ requires that spec is not None.
2. The exploration module's spec requires access to the environment, but the environment is instantiated inside the collector. There is currently no way to set the spec on the policy from inside the collector.

Solution

1. Allow AdditiveGaussianModule (and other exploration modules) to accept spec=None at construction
2. Add a spec property setter for delayed initialization
3. Inside Collector.__init__, instantiate the environment and set the spec property on any exploration modules which exist in the policy.

Alternatives

Additional context

Checklist

• I have checked that there is no similar issue in the repo (required)