Generalize SkillEnv to support real/sim skill backends sharing skill names

[tomsilver]

Goal

Enable real-to-sim-to-real workflows where the interface between real and sim is the high-level parameterized-skill language (ParameterizedSkillReference), with sim and real each having their own skill implementations under shared names (e.g., sim Pick uses waypoints, real Pick uses visual servoing).

What already exists

• KinDERParameterizedSkillEnv (kinder-models/src/kinder_models/utils.py) — wraps an ObjectCentricKinDEREnv so the action space is ParameterizedSkillReference(name, objects, params). Tested.
• LiftedParameterizedController / GroundParameterizedController / ParameterizedController in bilevel_planning.structs — the controller interface skill implementations satisfy.
• Concrete sim skill implementations per env in kinder-models (e.g., GroundPickController in dynamic2d/dyn_obstruction2d/parameterized_skills.py).

What's missing for the vision

1. Backend-agnostic SkillEnv. KinDERParameterizedSkillEnv hardcodes sim: ObjectCentricKinDEREnv. The wrapper body only uses reset / step / render / observation_space, so loosening to gymnasium.Env (or extracting a base class) unlocks using the same wrapper around a real env.

2. One skill name, multiple implementations. Pattern: LiftedParameterizedController(name="Pick", controller_cls=SimPickController, ...) and LiftedParameterizedController(name="Pick", controller_cls=RealPickController, ...). The agent emits ParameterizedSkillReference("Pick", ...); the surrounding SkillEnv is constructed with whichever implementation set matches its backend. A Skill registry (name -> {backend: controller_cls}) is a nice-to-have but not required day-one.

3. At least one real implementation of a skill (e.g., a visual-servoing Pick) that operates on TidyBotObservation and emits real actions. This is the actual research work; the rest is scaffolding.

Concrete tasks

• Decide where the generalized SkillEnv lives. Candidates: bilevel-planning/src/bilevel_planning/skill_env.py (natural home for skill machinery; depends on prpl_utils already), or keep it in kinder-models and just drop the type bound. Leaning toward bilevel-planning.
• Extract a backend-agnostic ParameterizedSkillEnv (parent of KinDERParameterizedSkillEnv or replacement).
• Write a real Pick controller as the first concrete real skill (or whatever skill is most useful to start with).
• Wire ParameterizedSkillEnv(real_tidybot_env, real_skills) and run a ParameterizedSkillReference("Pick", ...) on hardware.

Depends on

• prpl-mono issue: Refactor prpl-tidybot for low-level real-to-sim-to-real round-trip (Refactor prpl-tidybot for low-level real-to-sim-to-real round-trip prpl-mono#467). The real env needs to actually execute commands before any real skill can run.

Out of scope

• Perceiver in the agent loop (only needed if the agent reasons over ObjectCentricState between skill calls — additive, defer).