Add GRL Sokoban Resource Server

[yixinhuang48]

GymnasiumServer Refactor Notes (PR #1072 alignment)

• Refactored resources_servers/grl_sokoban to resources_servers/base_gymnasium.GymnasiumServer lifecycle (/reset, /step) and session-aware env metadata handling.
• Updated resources_servers/grl_sokoban/configs/grl_sokoban.yaml to use responses_api_agents/gymnasium_agent.
• Updated example prompt and generator for <action>...</action> format and generated example rollouts at resources_servers/grl_sokoban/data/example_rollouts.jsonl (5 rows).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yixinhuang48]

@cmunley1 @bxyu-nvidia this is the updated PR for the one that I closed (#261).

[cmunley1]

can you reupload the dataset generation script or point to a huggingface dataset in the readme? I removed this by accident on my branch.

otherwise this looks pretty good to me

[yixinhuang48]

Just uploaded the dataset generation script. The generation script includes different kinds of configurations for room size and box numbers for reward profiling, but for the training validation, I just used 6x6 room configuration and 1 box.

[cmunley1]

can you regenerate example.jsonl so it include this instruction in your latest dataset generation script? "IMPORTANT: First call the step tool with an empty array [] to see the initial puzzle state. Example: step({"actions": []})"

I run the existing example and get a response like
"text": "I cannot solve the Sokoban puzzle without specific tool observations or the initial state of the puzzle. Please provide the details of the puzzle (e.g., layout, box positions, player position) so I can assist further.",

Also, with updated example, I can run rollouts that look okay, but i do get lots of these messages in ng_run logs, do you see this too, is expected?

INFO:     127.0.0.1:56362 - "POST /step HTTP/1.1" 422 Unprocessable Entity
Hit validation exception! Errors: [
    {
        "type": "model_attributes_type",
        "loc": [
            "body"
        ],
        "msg": "Input should be a valid dictionary or object to extract fields from",
        "input": [
            "Right"
        ]
    }
]
Full body: [
    "Right"
]

[cmunley1]

@bxyu-nvidia can u please look at simple_agent changes? I tested offline rollouts with other resources servers, seems unaffected, but want to make sure with you.

[yixinhuang48]

@cmunley1 I've updated the example.jsonl and example_rollouts.jsonl files, and fixed the rollout message issue we encountered.

Problem

The /step endpoint was receiving 422 Unprocessable Entity errors when the model sometimes sent actions in the format ["Right"] instead of the expected {"actions": ["Right"]} format. This caused rollouts to fail with validation errors.

Solution

Updated the /step endpoint in app.py to handle both formats:

• {"actions": ["Right"]} (expected format)
• ["Right"] (array-only format that was causing errors)

The endpoint now automatically detects the format and normalizes it to the expected structure, making it more robust to handle cases where the model sends just the array directly. Hopefully this should eliminate the 422 errors you were seeing in the ng_run logs.

[cmunley1]

looks alright to me, tested offline rollouts, seems ok.

lets wait for @bxyu-nvidia to review simple_agent changes, at least

[cmunley1]

Considering refactoring this PR to fit into GymnasiumServer #1072, in case you are interested to help, or I may open a PR based on this one.

[yixinhuang48]

Yeah I can help out

[yixinhuang48]

@cmunley1 just refactored this PR to fit into the GymnasiumServer