[BugFix] Fix GAE compact path bias on recurrent value nets at internal truncations

[vmoens]

Stack from ghstack (oldest at bottom):

• [Doc] Migrate shifted=True callers to legacy/compact + docstring polish #3776
• [Example] Expose compact GAE cat dimension #3775
• [Example] Add Isaac RNN PPO rollout mode flags #3769
• [Example] Isaac Lab RNN PPO with compact memory + knowledge-base notes #3763
• -> [BugFix] Fix GAE compact path bias on recurrent value nets at internal truncations #3771

_call_value_net_compact previously built data_in = [root[0:T], boundary]
along the time dim and read value_[t] = V(data_in[t+1]). For every internal
done[t]=True (t < T-1), that read was V(root_obs[t+1]) -- the
post-reset first observation of the next episode -- rather than the
env-returned ("next", obs)[t] (the true pre-reset truncation observation).
GAE bootstraps with (1 - terminated), so on envs that truncate without
terminating (Isaac-Ant, Pendulum-on-timeout, ...) the wrong
next_state_value was not masked out and flowed straight into the value
target. With recurrent value nets the wrong observation also corrupted the
LSTM hidden state going forward, cascading into downstream slots. The wandb
runs g71pk34w / x05igvw7 (shifted='compact') trailed sln6yf2a /
6c8ihgh7 (shifted=False) by ~20% end-of-traj reward at iter 1000 for
exactly this reason.

The fix replaces the T+1 interleave with a fused batched call: the root
and ("next", ...) streams are concatenated along a non-time batch dim
into a constant-shape [2*B, T, *F] tensor and the value net is invoked
once. Reads of value and value_ are simple batch-half slices. The
("final", k) collector contract still overrides the next side at slot
T-1. For recurrent value nets ("next", "is_init") |= root_is_init is
applied so the LSTM resets at every trajectory boundary, exactly matching
the shifted=False reference (verified byte-exact on the regression
fixture). _fill_missing_next_inputs handles compact_obs=True rollouts
that don't populate ("next", k). A time_idx == 0 guard keeps 1D
rollouts correct by unsqueezing a batch dim before the cat.

Shape stays constant across calls in a training run (no .item() syncs,
no Python branching on tensor values), so torch.compile and vmap
remain happy.

The regression test test_gae_recurrent_shifted_compact_matches_unshifted_isaac_shape
asserts compact matches shifted=False to within rel < 0.05 on an
Isaac-shaped multi-trajectory rollout (B=4, T=16, truncations every 4
steps, compact_obs=False semantics). Pre-fix LSTM/GRU rel-err: 0.52 /
0.52. Post-fix: < 1e-7 (FP noise) for both. All other 1972 tests in
test/objectives/test_values.py continue to pass.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/3771

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• Run pull request jobs on OSDC runners in shadow mode

❌ 1 New Failure, 1 Cancelled Job, 2 Unrelated Failures

As of commit 45c9ef4 with merge base 5d11fa3 ():

NEW FAILURE - The following job has failed:

• Libs Tests on Linux / unittests-sklearn (3.10, 12.8) / linux-job (gh)
  test/libs/test_datasets.py::TestOpenML::test_data[mushroom_onehot]

CANCELLED JOB - The following job was cancelled. Please retry:

• Unit-tests on Windows / unittests-cpu (3.10, windows.4xlarge, cpu) / windows-job (gh)

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Build Windows Wheels / pytorch/rl / build-wheel-py3_10-cpu (gh) (trunk failure)
  Process completed with exit code 1.
• Build Windows Wheels / pytorch/rl / upload / upload-wheel-py3_10-cpu (gh) (trunk failure)
  Unable to download artifact(s): Artifact not found for name: pytorch_rl__3.10_cpu_x64

This comment was automatically generated by Dr. CI and updates every 15 minutes.