请问是否有长序列的比较维度？

[ghostplant]

得益于kvcache，现在的大模型多倾向于 long-kvcache + short-in + long-out，
请问是否有长序列的比较维度？（面向 coding / long thinking 等新需求）

For example, 1K/64K ..

目前的throughput指标被 long-in 污染较为严重，导致漂亮的统计指标与实际output体验有很大gap

[dawenxi-007]

Yes, agreed.

Moreover, it would be great if there is any document to give the updated details of each benchmark and explain the change list. For example, I noticed that the 1K/1K is deprecated but not sure why; Also, I am not sure what "Agentic Traces" benchmark means.

If there is any discussion/forum that is helpful to understand better on these benchmark items, I would be interested in joining.

[ghostplant]

I don't know why they turn to drop a lot of long-out evaluation, maybe it's bad for selling NVIDIA GPUs (since NVIDIA GPUs are the most expensive choice in decoding)?

It is sad that only the numbers look impressive, but customers become frustrated when actually deploying services.