the regressive loss of QCNeXt

[AdamQLMeng]

Hi, Dr Zhou, i am working on reproducing the results of QCNeXt, but i encountered a problem on the regressive loss function of the proposal trajectories and refinement trajectories. In my ablation experiments, using a joint loss function will lead to a worse performance on the marginal prediction, which is contrary compared with the results in Tab. 2 of QCNeXt. Would you kindly open the source code of the loss function? It will be really helpful to my ongoing work, thanks sincerely!

[caiocj1]

Hello, I also tried to reproduce QCNeXt and ran into some problems. Were you able to get good performance on the joint task itself?

[AdamQLMeng]

haven't reproduced the same results as the paper. My results on joint prediction are just slightly better than qcnet. I found the metrics of refine traj is the same as propose traj, and haven't solved it yet. Do you incur the same problem?

…

---Original--- From: "Caio ***@***.***> Date: Tue, Oct 14, 2025 15:37 PM To: ***@***.***>; Cc: "Adam QL. ***@***.******@***.***>; Subject: Re: [ZikangZhou/QCNet] the regressive loss of QCNeXt (Issue #52) caiocj1 left a comment (ZikangZhou/QCNet#52) Hello, I also tried to reproduce QCNeXt and ran into some problems. Were you able to get good performance on the joint task itself? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[caiocj1]

For me it's a bit different, when using the joint loss even the joint results seem to be worse than qcnet. I think my version of the joint loss is causing mode collapse, though I don't know why. Are your predictions diverse? Could you share how you did the joint loss when you can? I'd appreciate it.

Because I'm having this problem it might not be worth much but my refined trajs perform a bit better then the proposed ones.

[AdamQLMeng]

For me it's a bit different, when using the joint loss even the joint results seem to be worse than qcnet. I think my version of the joint loss is causing mode collapse, though I don't know why. Are your predictions diverse? Could you share how you did the joint loss when you can? I'd appreciate it.

Because I'm having this problem it might not be worth much but my refined trajs perform a bit better then the proposed ones.

The data is not aligned, data within a batch is concatenated into one tensor with a huge number of agents, thus, pi should be [B, K], traj_best should be {[Ai, T, 4]}, i=1, ..., B. This is the problem that i didn't notice at the first time. Hope it can help you.

[caiocj1]

For me the problem is there even taking this into account.

What I'm doing is basically:

1. Get best joint mode using proposal trajectories:
   1.1. calculate total L2 dists per-agent -> [N, K];
   1.2. aggregate using scatter and data["agent"]["batch"] as indices -> [B, K];
   1.3. then .argmin(-1) -> [B].

2. Get joint regression loss for proposal and refined joint trajs:
   2.1. for proposal trajs, get reg_nll -> [N, K, T, 2];
   2.2. sum last dim and across time using valid timesteps mask for each agent -> [N, K];
   2.3. aggregate like in step 1 -> [B, K];
   2.4. select best mode from step 1 -> [B];
   2.5. .mean() for the loss, repeat for refined trajs.

3. Get joint classification loss:
   3.1. using detached refined trajs, calculate for each agent the NLL of last point like QCNet is doing -> [N, K];
   3.2. aggregate -> [B, K];
   3.3. do -torch.logsumexp(log_pi - cls_nll, dim=-1) again like QCNet is already doing but using scene-level scores -> [B];
   3.4. .mean() for the loss.

Let me know if you're doing anything differently whenever you have the time. @ZikangZhou if you could confirm this is how it's supposed to go too, that'd be great. Really appreciate the help!

[AdamQLMeng]

Did you use mask to exclude the trajs of the agents don't need be predicted? i.e. multiply the mask and divide the mask.sum(0) as the author does. After all, you shouldn't calculate mean of all agents in each batch (data["agent"]["batch"]). And calculate NLL of all trajs is not necessary，use .argmin(-1) to select the trajs for computing NLL ([N, T, 4]) is more efficient. Or you can use NLL to select best trajs directly, which will get the same results as L2. But the first one is more computational efficient, because L2 is more efficient than NLL.

…

---Original--- From: "Caio ***@***.***> Date: Thu, Oct 16, 2025 20:34 PM To: ***@***.***>; Cc: "Adam QL. ***@***.******@***.***>; Subject: Re: [ZikangZhou/QCNet] the regressive loss of QCNeXt (Issue #52) caiocj1 left a comment (ZikangZhou/QCNet#52) For me the problem is there even taking this into account. What I'm doing is basically: Get best joint mode using proposal trajectories: 1.1. calculate total L2 dists per-agent -> [N, K]; 1.2. aggregate using scatter and data["agent"]["batch"] as indices -> [B, K]; 1.3. then .argmin(-1) -> [B]. Get joint regression loss for proposal and refined joint trajs: 2.1. for proposal trajs, get reg_nll -> [N, K, T, 2]; 2.2. sum last dim and across time using valid timesteps mask for each agent -> [N, K]; 2.3. aggregate like in step 1 -> [B, K]; 2.4. select best mode from step 1 -> [B]; 2.5. .mean() for the loss, repeat for refined trajs. Get joint classification loss: 3.1. using detached refined trajs, calculate for each agent the NLL of last point like QCNet is doing -> [N, K]; 3.2. aggregate -> [B, K]; 3.3. do -torch.logsumexp(log_pi - cls_nll, dim=-1) again like QCNet is already doing but using scene-level scores -> [B]; 3.4. .mean() for the loss. Let me know if you're doing anything differently whenever you have the time. @ZikangZhou if you could confirm this is how it's supposed to go too, that'd be great. Really appreciate the help! — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[caiocj1]

Yes, I'm making sure to filter out unnecessary agents, you can consider N in my previous comment the number of agents after filtering. Tried a few different ways to do this filtering (like 1. using just agents that havid a valid current timestep and any valid future timestep, or 2. using just the agents that you evaluate on), but the problem remains there.

And true about not having to calculate NLL of all trajs, thanks.

[AdamQLMeng]

Follow the author's idea, the mask should multiply with the nll [A, K, T] (not filter out the valid agents, nll should be A, which is the original number of agents), because the predicted agents might exit at any future time step. You can calculate sum of data["agent"]["predict_mask"] on the time dim. You will see the diverse number of valid steps.
I encounter the problem that the brier doesn't converge, there lacks the details of attentive pooling in the paper. Would you mind sharing the pooling you used in scene score module? And which is the paper you referenced?

[caiocj1]

I was thinking from the QCNeXt report that for the NLL you use A' <= A, no? And the way you get the A' I think is by dropping agents not present at the current timestep or agents of static types, etc, that's what I meant by filtering out unnecessary agents. I'm taking into account that predicted agents (the A') can exit, by multiplying the NLLs and L2 dists by the mask of valid steps.

For the scene scoring, I got from reading this blog:
https://benjaminwarner.dev/2022/07/14/tinkering-with-attention-pooling#learned-aggregation
though it's not exactly the implementation they give. I wanted a simple version since the goal is to just have a single vector per mode representing each scene. So you use a learnable query which will get mixed differently depending on the keys of each mode. Code is below, let me know if you see anything wrong. valid_mask is dropping out the kinds of agents I mentioned, static types or agents not in the current timestep.

A later recent paper by Dr Zhou (Parallel ModeSeq) actually just goes back to using max pooling, so not sure it's too important to use attentive pooling for good performance.

import torch
import torch.nn as nn
import numpy as np
from torch_geometric.utils import softmax, scatter


class SceneScoringLayer(nn.Module):
    def __init__(self,
                 hidden_dim: int,):
        super().__init__()

        self.hidden_dim = hidden_dim

        self.q = nn.Parameter(torch.randn(self.hidden_dim))
        self.to_kv = nn.Linear(self.hidden_dim, 2 * self.hidden_dim, bias=False)

        self.to_pi = nn.Sequential(
            nn.Linear(self.hidden_dim, self.hidden_dim),
            nn.LayerNorm(self.hidden_dim),
            nn.ReLU(),
            nn.Linear(self.hidden_dim, 1)
        )

    def forward(self, m, valid_mask, batch_mask):
        # Get keys [N, K, D] and values [N, K, D] from mode embeddings [N, K, D] of relevant agents
        k, v = self.to_kv(m[valid_mask]).chunk(2, dim=-1)

        # QK dot product, apply softmax within each sample in batch to get weights [N, K]
        weights = (self.q[None, None] * k).sum(dim=-1) / np.sqrt(self.hidden_dim)
        weights = softmax(weights, batch_mask[valid_mask], dim=0)

        # Weighted sum of values
        out = scatter(weights[..., None] * v, batch_mask[valid_mask], reduce="sum", dim=0)

        # Final MLP
        return self.to_pi(out).squeeze(-1)

[AdamQLMeng]

Then, I think we use the same loss function. It should be better than qcnet on joint metrics after changing loss, max pooling scene score module, row-wise self-attention. By the way, the eval_mask of joint prediction should be data["agent"]["category"]>=2, it is different from marginal prediction.
As for the pooling, it is crucial according to my results. I tried max pooling, average pooling, different versions of attentive pooling. I have read Paralle ModeSeq too, that's why i tried max pooling, but simple max pooling didn't work well.

[caiocj1]

Just to update, I was able to get the joint performance to be better than QCNet (thanks for the help), now I'm on the same boat as you in your original comment, on scenes with a single agent to be evaluated on my MinFDE is worse than QCNet's. Would you mind sharing what kinds of values you get? I'm at 1.28 MinJointFDE for QCNeXt on whole val set (category >=2 & type <= 4), and 1.40 MinFDE on the validation scenes with a single eval agent (vs reported 1.24 of Tab. 2).

[AdamQLMeng]

Congratulations! But you are in the lead now. I am trapped in avgMinFDE6=1.42, avgMinFDE1=2.51, which means 6 guesses metrics are much worse than results in Tab. 1, but closed on 1 guess metrics. Would you tell me if you still use 150 as the radius of row-wise self-attention? Or did you do other modification except row-wise self-attention, scene score, and joint loss? And do you encounter the problem that the Brier doesn't converge?

…

---Original--- From: "Caio ***@***.***&gt; Date: Mon, Oct 20, 2025 18:27 PM To: ***@***.***&gt;; Cc: "Adam QL. ***@***.******@***.***&gt;; Subject: Re: [ZikangZhou/QCNet] the regressive loss of QCNeXt (Issue #52) caiocj1 left a comment (ZikangZhou/QCNet#52) Just to update, I was able to get the joint performance to be better than QCNet (thanks for the help), now I'm on the same boat as you in your original comment, on scenes with a single agent to be evaluated on my MinFDE is worse than QCNet's. Would you mind sharing what kinds of values you get? I'm at 1.28 MinJointFDE for QCNeXt on whole val set (category &gt;=2 &amp; type <= 4), and 1.40 MinFDE on the validation scenes with a single eval agent (vs reported 1.24 of Tab. 2). — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

[caiocj1]

Exactly, still using 150 for radius. I'll share my code for the decoder and the loss, it might help (I'm using a different codebase, so format of init arguments and of outputs change a bit but I think it's easy to adapt back into the format expected in this repo). The loss could be optimized a bit more like you pointed out to avoid useless computation, still have to do it though.

Besides these three modifications you mentioned, there's just (1) weight decay of 0.1 instead of 0.0001 like it was for QCNet and (2) for the classification loss i'm using the whole trajectory to get the NLL, not just the last points.

For the Brier, could you explain a bit more what's happening? Mine converges to a value, although it's a bad one (I get 1.975 avgBrierMinFDE6, which is pretty bad for a avgMinFDE6 of 1.28), so I still have some work to do there, the classification loss is still not good. I'm sharing this version since it was what gave me the better avgMinFDE6.

qcnext_loss.py
qcnext_decoder.py

In the codebase I'm using, the config file has these hyperparameters:

MODEL_CONFIG:
  ENCODER:
    NAME: QCNetEncoder

    INPUT_DIM: 2
    HIDDEN_DIM: 128
    NUM_MAP_LAYERS: 1
    NUM_AGENT_LAYERS: 2
    HEAD_DIM: 16
    NUM_HEADS: 8
    NUM_FREQ_BANDS: 64
    PL2PL_RADIUS: 150
    PL2A_RADIUS: 50
    A2A_RADIUS: 50
    DROPOUT: 0.1
    TIME_SPAN: 10
  
  DECODER:
    NAME: QCNeXtDecoder

    NUM_MODES: 6
    HIDDEN_DIM: 128
    NUM_FREQ_BANDS: 64
    NUM_LAYERS: 2
    DROPOUT: 0.1
    HEAD_DIM: 16
    NUM_HEADS: 8
    PL2M_RADIUS: 150
    A2M_RADIUS: 150
    NUM_T2M_STEPS: 30

    NUM_RECURRENT_STEPS: 3

  
OPT_CONFIG:
  LOSS: QCNeXtLoss
  
  MAX_EPOCHS: 64

  OPTIMIZER: AdamW
  WEIGHT_DECAY: 0.1
  LR: 0.0005

  LR_SCHEDULER: CosineAnnealingLR
  ETA_MIN: 0.0

Hope it helps!

[AdamQLMeng]

Thanks for sharing! I will try the cls loss with all time steps. However, I think the way of using the mask for computing mean is wrong. You use mask for computing mean of time steps before mean of agents. But the ratio of valid agents within each scene is different, then the final loss is influenced by this ratio significantly. If the order of two mean is inverse, the mean of time steps without the mask is correct, because all time steps must have valid agents.

reg loss propose of 4 data by your way
5.715259552001953, 4.802114486694336, 5.054935932159424, 6.188111782073975

reg loss propose of the same 4 data by inverse way
8.129289627075195, 8.047307014465332, 8.04161548614502, 8.250815391540527

[caiocj1]

But I'm applying the loss only on valid agents with the [valid_mask] masking, no?

Also, when you do the average of the agents first, and then average across time, you'd have to do something like this I think:

reg_nll = scatter(reg_nll * future_mask[:, None], batch_mask, reduce="sum") / scatter(future_mask[:, None], batch_mask, reduce="sum") # [B, K, T]
reg_nll = reg_nll.mean(-1) # [B, K]

Because if you don't divide by scatter(future_mask[:, None], batch_mask) the scenes with too many valid agents would have a much higher loss, and if you do reg_nll = scatter(reg_nll * future_mask[:, None], batch_mask, reduce="mean") instead you'd be dividing across the whole number of agents even though just some of them might be present at some timesteps.

But doing it this way seems to give the same results for the loss. Sorry if I'm misunderstanding anything, let me know if that's the case.