More progress on coffee.

Author: ashayathalye

predicators/approaches/grammar_search_invention_approach.py

@@ -490,6 +490,8 @@ def enumerate(self) -> Iterator[Tuple[Predicate, float]]:
         "JugInMachine0",
         "Holding0",
         "HandEmpty0",
+        "JugFilled0",
+        "MachineOn0",
     ]
 }
 _DEBUG_VLM_PREDICATES = defaultdict(list, _DEBUG_VLM_PREDICATES)

predicators/datasets/generate_atom_trajs_with_vlm.py

@@ -113,7 +113,6 @@ def _sample_vlm_atom_proposals_from_trajectories(
     """Given a list of ImageOptionTrajectories, query a VLM to generate a list
     of names of ground atoms from which we can extract predicates that might be
     relevant for planning to recreate these trajectories."""
-    import pdb; pdb.set_trace()
     aggregated_vlm_output_strs = []
     all_vlm_queries_list = []
     for traj in trajectories:
@@ -751,6 +750,7 @@ def _generate_ground_atoms_with_vlm_pure_visual_preds(
         vlm: VisionLanguageModel) -> List[List[Set[GroundAtom]]]:
     """Given a collection of ImageOptionTrajectories, query a VLM to convert
     these into ground atom trajectories."""
+    import pdb; pdb.set_trace()
     if not CFG.grammar_search_vlm_atom_proposal_use_debug:
         logging.info("Querying VLM for candidate atom proposals...")
         atom_strs_proposals_list = _sample_vlm_atom_proposals_from_trajectories(

predicators/datasets/vlm_input_data_prompts/atom_labelling/double_check_prompt_prev_labels.txt

@@ -1 +1 @@
-Sometimes your reasoning about the value of a predicate at the current timestep uses an incorrect value of that predicate in the previous timestep. Below, I give you give you the values of the predicates at the previous timestep once again. Please check your reasoning and provide a corrected version of your previous answer, if it needs correcting. Regardless of whether or not it needs correctly, your reply should be formatted exactly the same as the previous answer.
+Sometimes your reasoning about the value of a predicate at the current timestep uses an incorrect value of that predicate in the previous timestep. Below, I give you give you the values of the predicates at the previous timestep once again. Please check your reasoning and provide a corrected version of your previous answer, if it needs correcting. Regardless of whether or not it needs correctly, your reply should be formatted exactly the same as the previous answer. Furthermore, make sure you provide labels for all predicates we requested labels for.

predicators/datasets/vlm_input_data_prompts/atom_labelling/img_option_diffs_label_history.txt

@@ -1,6 +1,8 @@
 You are a vision system for a robot provided with two images: a before image showing the state before a skill is executed, an after image showing the state after the skill is executed. You are given a list of predicates below, and you are given the values of these predicates in the image before the skill is executed. Your job is to output the values of the following predicates in the image after the skill is executed. Pay careful attention to the visual changes between the two images to figure out which predicates change and which predicates do not change. Note that some or all of the predicates don't necessary have to change. First, output a description of what changes you expect to happen based on the skill that was just run, explicitly noting the skill that was run. Second, output a description of what visual changes you see happen between the before and after images, looking specifically at the objects involved in the skill's arguments, noting what objects these are. From these two descriptions, for each predicate labeled in the previous timestep, note whether you expect its value to change or stay the same. Next, for each predicate given in the list of predicates to label, output each predicate value in the after image as a bulleted list (use '*' for the bullets) with each predicate and value on a different line. Ensure there is a period ('.') after the truth value of the predicate. For each predicate value, provide an explanation as to why you labelled this predicate as having this particular value, and note what value this predicate had in the previous timestep, which is given to you in the prompt. Use the format: `* <predicate>: <truth_value>. <explanation>`. When labeling the value of a predicate, if you don't see the objects involved in that predicate, retain its truth value from the previous timestep. Also, if your description of changes you expect to happen, and your description of visual changes you saw happen, have nothing to do with the predicate you are trying to label, retain its truth value from the previous timestep. For example, if in the previous timestep I paint an object, and in the current timestamp I sit on it, we don't expect its color to change after sitting on it. 
 
-Note that the coffee machine is black and the table is white. 
+Our object detection system tells us that the table is white, the coffee machine is black, the jug is white, and coffee is brown.
+
+Double check your answer and make sure that you have provided labels for all predicates we requested labels for.
 
 Your response should have three sections. Here is an outline of what your response should look like:
 [START OULTLINE]

predicators/datasets/vlm_input_data_prompts/atom_labelling/per_scene_naive.txt

@@ -2,6 +2,8 @@ You are a vision system for a robot. Your job is to output the values of the fol
 
 Before output predicate values, give a description of the scene and explain why you think the predicate values are what they are.
 
-Note that the coffee machine is black and the table is white.
+Our object detection system tells us that the table is white, the coffee machine is black, the jug is white, and coffee is brown.
+
+Double check your answer and make sure that you have provided labels for all predicates we requested labels for.
 
 Predicates:

predicators/datasets/vlm_input_data_prompts/atom_proposal/options_labels_whole_traj_diverse_coffee.txt

@@ -0,0 +1 @@
+You are a robotic vision system whose job is to output a structured set of predicates useful for describing important concepts in the following demonstration of a task. You will be provided with a list of actions used during the task, as well as images of states before and after every action execution. Please provide predicates in terms of the following objects: {objs}. For each predicate, output it in the following format: predicate_name(obj1, obj2, obj3...). Start by generating predicates that capture visual changes before and after each action. After this, generate any other predicates that perhaps do not change but are still important to describing the demonstration shown. For each predicate you generate, also generate some predicates that are synonyms and antonyms so that any predicate that is tangentially relevant to the demonstrations is generated. Generate a concise list of predicates you think would be the most important for solving the task: consider those that characterize the preconditions and effects of the actions being taken in the most general way. Don't generate anything too similar to the given predicates, which are: CupFilled, RobotAboveCup, JugAboveCup, NotAboveCup, JugAboveCup, PressingButton, Twisting, NotSameCup, JugPickable, to avoid redundancy.

predicators/envs/coffee.py

@@ -189,7 +189,7 @@ def __init__(self, use_gui: bool = True) -> None:
                                       self._JugPickable_holds)
 
         # Static objects (always exist no matter the settings).
-        self._robot = Object("robby", self._robot_type)
+        self._robot = Object("robot", self._robot_type)
         self._jug = Object("jug", self._jug_type)
         self._machine = Object("coffee_machine", self._machine_type)
         self._table = Object("table", self._table_type)

predicators/envs/pybullet_coffee.py

@@ -150,8 +150,10 @@ def jug_handle_height(cls) -> float:
     jug_init_rot_ub: ClassVar[float] = 2 * np.pi / 3
     # jug_color: ClassVar[Tuple[float, float, float, float]] =\
     #     (0.5,1,0,0.5) # Green
+    # jug_color: ClassVar[Tuple[float, float, float, float]] =\
+    #             (1,1,1,0.5) # White
     jug_color: ClassVar[Tuple[float, float, float, float]] =\
-                (1,1,1,0.5) # White
+                (1,1,1,0.3) # White
     # Dispense area settings.
     dispense_area_x: ClassVar[float] = machine_x
     dispense_area_y: ClassVar[float] = machine_y - 5 * jug_radius
@@ -301,8 +303,9 @@ def get_vlm_debug_atom_strs(self,
         #                                      atom_strs_by_task_type)
         # atom_strs = atom_strs_by_task_type[CFG.burger_no_move_task_type]
         # return [[a] for a in atom_strs]
-        return [["JugInMachine(jug, coffee_machine)"], ["Holding(robby, jug)"], ["HandEmpty(robby)"]]
-        
+        # return [["JugInMachine(jug, coffee_machine)"], ["Holding(robby, jug)"], ["HandEmpty(robby)"]]
+        return [["JugFilled(jug)"], ["JugInMachine(jug, coffee_machine)"], ["Holding(robot, jug)"], ["HandEmpty(robot)"], ["MachineOn(coffee_machine)"]]
+        # return [["JugFilled(jug)"], ["JugInMachine(jug, coffee_machine)"]]
 
     @property
     def oracle_proposed_predicates(self) -> Set[Predicate]:

predicators/ground_truth_models/coffee/nsrts.py

@@ -54,11 +54,10 @@ def get_nsrts(env_name: str, types: Dict[str, Type],
             TwistJug = options["TwistJug"]
         PickJug = options["PickJug"]
         PlaceJugInMachine = options["PlaceJugInMachine"]
-        TurnMachineOn = options["TurnMachineOn"]
-        Pour = options["Pour"]
+        TurnMachineOn = options["TurnMachineOnAndFill"]
+        Pour = options["PourSomeLiquid"]
 
         nsrts = set()
-
         if CFG.coffee_machine_has_plug:
             # PlugIn
             plug = Variable("?plug", plug_type)

predicators/ground_truth_models/coffee/options.py

@@ -155,7 +155,7 @@ def _TurnMachineOn_terminal(state: State, memory: Dict,
             return MachineOn.holds(state, [machine])
 
         TurnMachineOn = ParameterizedOption(
-            "TurnMachineOn",
+            "TurnMachineOnAndFill",
             types=[robot_type, machine_type],
             params_space=Box(0, 1, (0, )),
             policy=cls._create_turn_machine_on_policy(),
@@ -175,7 +175,7 @@ def _Pour_terminal(state: State, memory: Dict,
             _, _, cup = objects
             return CupFilled.holds(state, [cup])
 
-        Pour = ParameterizedOption("Pour",
+        Pour = ParameterizedOption("PourSomeLiquid",
                                    types=[robot_type, jug_type, cup_type],
                                    params_space=Box(0, 1, (0, )),
                                    policy=cls._create_pour_policy(),
@@ -634,7 +634,7 @@ def _MoveBackAfterPlaceOrPush_terminal(state: State, memory: Dict,
             )
 
             _TurnMachineOn = utils.get_parameterized_option_by_name(
-                options, "TurnMachineOn")
+                options, "TurnMachineOnAndFill")
             _PlaceJugInMachine = utils.get_parameterized_option_by_name(
                 options, "PlaceJugInMachine")
             assert _TurnMachineOn is not None
@@ -646,7 +646,7 @@ def _MoveBackAfterPlaceOrPush_terminal(state: State, memory: Dict,
                 "PlaceJugInMachine", [_PlaceJugInMachine, MoveBackAfterPlace])
 
             TurnMachineOn = utils.LinearChainParameterizedOption(
-                "TurnMachineOn", [_TurnMachineOn, MoveBackAfterPush])
+                "TurnMachineOnAndFill", [_TurnMachineOn, MoveBackAfterPush])
             options.add(PlaceJugInMachine)
             options.add(TurnMachineOn)