Lacking reference summary in training datasets

[nvduc]

Dear STREAMIND challenge organizers,

Thank you very much for organizing this challenge.
I have the following question regarding the dataset.

Performance evaluation requires having the reference summary of each audio source.
However, the provided rev16 dataset contains only transcribed text, but not the reference summary required for evaluation.
Could you provide the reference summary of the training datasets?

Also, the transcribed text of the ietf dataset is also lacking.

Best,
Duc.

[b3by]

Thank you for joining the challenge and for reaching out!

Our scoring system will leverage LLMs to assess the quality of partial summaries, against the progressive input transcription text. In doing so, we are not using any reference summaries as hypothesis (even tho a reference summary is mentioned in the README file), but rather the transcription text up to the end time of any given chunk. For this reason, we did not include summaries in our dataset.

However, I just pushed on the repository additional data for both datasets:

• rev16 now contains example chunks as they could be generated by a progressive summary pipeline;
• ietf now contains the long form transcriptions

Hope this helps!