[Question] Terrain curriculum issue: Terrain level does not exceed 6

[H-Hisamichi]

Question

Hello everyone,

I have been trying to resolve an ongoing issue where the terrain level does not go above 6 when using the terrain curriculum. This problem occurs not only in my custom environment but also in the default RL demo.

In Fig. 1, the black line represents the terrain level progression for "Isaac-Velocity-Rough-Anymal-D-v0" using the default parameters. The red line shows the progression when the terrain sampling range is halved. As you can see, the level converges to 6 regardless of the terrain's difficulty. Furthermore, the graph does not look like simple hard clipping; rather, it exhibits a curve similar to a PD controller's response.

Fig. 1:

I have reviewed the terrain_levels_vel implementation but could not find any incorrect processing. Considering the possibility of the RL agent falling into a local optimum, I also tried adjusting PPO parameters such as entropy_coef and learning_rate, but observed no improvement.

Is it the expected behavior for the terrain level to be restricted to a maximum of 6 in the terrain curriculum?

Thank you in advance for your insights.

Build Info

Describe the versions that you are currently using:

• Isaac Lab Version: [2.3.1]
• Isaac Sim Version: [5.1.0]

[RandomOakForest]

Thank you for posting this. I moved your post to our Discussions for follow-up. You are not misusing the curriculum, but its current design makes mid‑level saturation a very plausible outcome—so your observation is consistent with known behaviour, even though it is not the ideal target behaviour.

Practical workarounds you can try now

If you want terrain difficulty to keep increasing beyond level 6 in your experiments, consider:

• Increasing commanded speeds and biasing command sampling toward forward heading velocities so the robot reliably exits its current grid cell during episodes.
• Reducing episode length or reset conditions that bring the robot back near the origin before it has traversed enough distance to trigger a level increase.
• Implementing or swapping in a custom curriculum function that:
  • Uses velocity tracking error or success metrics instead of L2 distance, or
  • Uses cumulative signed distance along commanded heading instead of absolute position, so oscillating commands don’t cancel out progress.

These do require editing or replacing terrain_levels_vel in your task config but do not require changes to PPO itself.

[pmorerio]

Hi,
I noticed that given how the update function is implemented, all robots reaching level 9 are in the long term reassigned uniformly across levels

IsaacLab/source/isaaclab/isaaclab/terrains/terrain_importer.py

Line 325 in f4aa17f

torch.randint_like(self.terrain_levels[env_ids], self.max_terrain_level),

It is easy to prove mathematically that the long term expectation is exactly 6, under the mild assumption that the transition probability across levels is the same.

If you want all robots to stay in level 9 once they reach it, simply replace the torch.where block in the function with a simple clip:

-        self.terrain_levels[env_ids] = torch.where(
-            self.terrain_levels[env_ids] >= self.max_terrain_level,
-            torch.randint_like(self.terrain_levels[env_ids], self.max_terrain_level),
-            torch.clip(self.terrain_levels[env_ids], 0),
-        )
+        self.terrain_levels[env_ids] = torch.clip(self.terrain_levels[env_ids], 0, self.max_terrain_level-1)

[pmorerio]

Proof

1. Setting Up the Balance Equations

Let $\pi_i$ be the long-term probability of the random variable $x$ being in state $i$, where $i \in {0, 1, 2, \dots, 9}$. In a steady state, the probability "flowing" out of a state must equal the probability "flowing" into it.

• For state 0: The only way to enter state 0 is if the variable was at 9 and got uniformly reassigned to 0. The uniform reassignment happens with probability $p$, and since there are 10 possible numbers $[0,9]$, the chance of landing specifically on 0 is $\frac{p}{10}$. The variable leaves state 0 when it increases to 1 (with probability $p$).
  $$\pi_0 \cdot p = \pi_9 \cdot \frac{p}{10}$$
  Dividing by $p$, we get:
  $$\pi_0 = \frac{\pi_9}{10}$$

• For state 1: The variable can enter state 1 either by increasing from 0, or by being reassigned from 9. It leaves state 1 by increasing to 2.
  $$\pi_1 \cdot p = \pi_0 \cdot p + \pi_9 \cdot \frac{p}{10}$$
  Dividing by $p$ and substituting $\pi_0$:
  $$\pi_1 = \pi_0 + \frac{\pi_9}{10} = \frac{\pi_9}{10} + \frac{\pi_9}{10} = \frac{2\pi_9}{10}$$

• For state $i$ (up to 8): Following this exact same pattern of flow (entering from the previous state $i-1$ or being reassigned from 9), we find a linear sequence:
  $$\pi_i = \pi_{i-1} + \frac{\pi_9}{10}$$
  Which resolves to:
  $$\pi_i = \frac{(i + 1)\pi_9}{10}$$

2. Finding the Stationary Distribution

To find the exact probabilities, we use the fact that the sum of all state probabilities must equal 1:
$$\sum_{i=0}^{9} \pi_i = 1$$

Substitute our formula $\pi_i = \frac{(i+1)\pi_9}{10}$ into the sum:
$$\sum_{i=0}^{9} \frac{(i + 1)\pi_9}{10} = \frac{\pi_9}{10} \sum_{i=0}^{9} (i + 1)$$

The sum of the first 10 integers ($1 + 2 + \dots + 10$) is 55.
$$\frac{\pi_9}{10} \cdot 55 = 1$$
$$5.5 \pi_9 = 1 \implies \pi_9 = \frac{1}{5.5} = \frac{2}{11}$$

Now we can define the general probability for any state $i$:
$$\pi_i = \frac{i+1}{10} \cdot \frac{2}{11} = \frac{i+1}{55}$$

3. Calculating the Expected Value

The long-term expectation $E[x]$ is the sum of each value multiplied by its long-term probability:
$$E[x] = \sum_{i=0}^{9} i \cdot \pi_i = \sum_{i=0}^{9} i \cdot \left(\frac{i+1}{55}\right)$$

Factor out the constant $\frac{1}{55}$:
$$E[x] = \frac{1}{55} \sum_{i=0}^{9} (i^2 + i)$$

Now, evaluate the sums of $i^2$ and $i$ from 0 to 9:

• Sum of $i$: $0 + 1 + 2 + \dots + 9 = 45$
• Sum of $i^2$: $0^2 + 1^2 + 2^2 + \dots + 9^2 = 285$

Add them together and divide by 55:
$$E[x] = \frac{1}{55} (285 + 45)$$
$$E[x] = \frac{330}{55} = 6$$

The long-term expectation of your random variable is exactly 6.

[H-Hisamichi]

Hi everyone,

Thank you so much for all the advice. I’m going to look into these suggestions and conduct some tests. I’ll make sure to share the results with you all later.

Thanks again!

[H-Hisamichi]

Hello everyone,

The issue where the terrain curriculum would not progress beyond level 6 has been resolved (it has now reached level 8.8 out of a 0–9 range). I found @pmorerio’s approach to be the most effective so far.
Regarding the issue where command resampling was causing the actors' final L2 distance to decrease, I addressed this by disabling resampling during the episodes. While this fixed the distance issue, it has hindered the learning of steering skills during locomotion, so I am now considering using cumulative travel distance as an alternative metric.

Thank you all for your valuable advice!