Fix async grpo picklable reward functions

[siddiksawani]

What does this PR do?

Fixes #6133

AsyncGRPO supports picklable reward callables in its spawned rollout worker,
including functools.partial values and callable class instances. The worker
previously assumed every reward exposed __name__, which caused initialization
to fail for valid unnamed callables.

This PR resolves metric names by unwrapping partials, preserving any existing
__name__, and falling back to the callable class name only when no name exists.

Before submitting

• This PR fixes a typo or improves the docs.
• Did you read the contributor guideline, Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

AI writing disclosure

• AI-assisted: some parts were suggested or improved by AI, but the PR was written and reviewed by a human.

Who can review?

Community reviewers are welcome.

---

Note

Low Risk
Narrow change to reward metric naming at worker init; reward scoring behavior is unchanged.

Overview
Fixes rollout worker startup when reward functions are valid picklable callables that do not expose __name__ (e.g. unnamed functools.partial or callable class instances).

The async rollout worker now derives per-reward metric keys via _get_callable_name: unwrap partial objects (using an explicit __name__ when set), otherwise use __name__, and fall back to the callable’s class name. reward_func_names is built with this helper instead of assuming every reward has __name__.

Adds a parametrized test covering partials, named partials, callable instances, and builtins like operator.add.

^{Reviewed by Cursor Bugbot for commit 03bf68a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[siddiksawani]

Follow-up to #5293, which introduced AsyncGRPO and the rollout-worker reward metric naming path.

cc @qgallouedec , since this affects the original AsyncGRPO implementation. Also cc @AmineDiro because #5749 recently documented support for picklable reward callables such as functools.partial, which exposes this incompatibility.

The current reward metric-name initialization assumes every callable has name, but functools.partial objects and common callable class instances do not.

[Copilot]

Pull request overview

This PR fixes AsyncGRPO rollout-worker initialization when reward functions are valid picklable callables that don’t expose __name__ (e.g., functools.partial and callable class instances), by introducing a helper to derive stable metric names and adding regression tests for the naming behavior.

Changes:

• Add _get_callable_name to resolve reward metric names for partials and callable instances.
• Use _get_callable_name when building reward_func_names in the spawned rollout worker.
• Add focused regression tests covering partials, callable instances (named/unnamed), and builtin callables.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
trl/experimental/async_grpo/async_rollout_worker.py	Adds a callable-name resolver and uses it to build reward metric keys without assuming __name__ exists.
tests/experimental/test_async_grpo_trainer.py	Adds regression tests ensuring callable-name resolution works for documented picklable reward callable types.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[siddiksawani]

Related to #6134, which addresses the same issue. This implementation additionally preserves an explicitly assigned name on functools.partial values. That allows users to distinguish metrics for multiple configured partial rewards wrapping the same underlying function, instead of collapsing them under one metric key.

[siddiksawani]

Could I please get a quick review on this short PR whenever you have a moment?
cc: @AmineDiro @albertvillanova

[siddiksawani]

Could I please get a quick review on this short PR whenever you have a moment?
cc: @AmineDiro @albertvillanova

[qgallouedec]

thanks, this one is superseded by #6313