Needlebench, llama2-7b-hf : I cannot reproduce the score same as paper!

[ainilian]

dataset : NeedleBench 4K Single-Retrieval; model: hf_llama2_7b
score in the paper: Chinese=32.42, English = 93.84, Overall = 63.13
score I reproduced: Chinese=10.31, English = 4.51, Overall = 7.41

command

CUDA_VISIBLE_DEVICES=2,3,4,5 python run.py configs/eval_needlebench.py --max-num-workers 16

configs/eval_needlebench.py

from mmengine.config import read_base
with read_base():
    # from opencompass.configs.models.hf_internlm.lmdeploy_internlm2_chat_7b import models as internlm2_chat_7b_200k
    # from opencompass.configs.models.hf_internlm.hf_internlm2_chat_7b import models as internlm2_chat_7b
    from opencompass.configs.models.hf_llama.hf_llama2_7b import models as hf_llama2_7b

    # Evaluate needlebench_4k, adjust the configuration to use 8k, 32k, 128k, 200k, or 1000k if necessary.
    # from opencompass.configs.datasets.needlebench.needlebench_4k.needlebench_4k import needlebench_datasets
    # from opencompass.configs.summarizers.needlebench import needlebench_4k_summarizer as summarizer

    # only eval original "needle in a haystack test" in needlebench_4k
    from opencompass.configs.datasets.needlebench.needlebench_4k.needlebench_single_4k import needlebench_zh_datasets, needlebench_en_datasets
    from opencompass.configs.summarizers.needlebench import needlebench_4k_summarizer as summarizer

    # eval Ancestral Tracing Challenge(ATC)
    # from opencompass.configs.datasets.needlebench.atc.atc_choice_50 import needlebench_datasets
    # from opencompass.configs.summarizers.needlebench import atc_summarizer_50 as summarizer

datasets = sum([v for k, v in locals().items() if ('datasets' in k)], [])


for m in hf_llama2_7b:     
    m['path'] = '/base_models/llama-2-7b-hf'
    m['max_seq_len'] = 4096 # Ensure InternLM2-7B model can receive the full length of long texts, adjust for other models based on their supported maximum sequence length.
    m['max_out_len'] = 2000 # Ensure complete responses from the model in multi-needle retrieval tasks.
    m['batch_size'] = 4
    m['stop_words'] = ['</s>']

models = hf_llama2_7b

work_dir = './outputs/needlebench'

result

20240903_095534
tabulate format
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dataset                                                     llama-2-7b-hf
----------------------------------------------------------  ---------------
NeedleBench-Overall-Score                                   -
dataset                                                     llama-2-7b-hf
--------- NeedleBench-4K-Single-Needle-Retrieval ---------  -
dataset                                                     llama-2-7b-hf
Single-Needle-Retrieval(S-RT)                               7.41
Single-Needle-Retrieval-EN                                  4.51
Single-Needle-Retrieval-ZH                                  10.31
--------- NeedleBench-4K-Multi-Needle-Retrieval ---------   -
dataset                                                     llama-2-7b-hf
Multi-Needle-Retrieval(M-RT)                                -
dataset                                                     llama-2-7b-hf
Multi-Needle-Retrieval-EN                                   -
dataset                                                     llama-2-7b-hf
Multi-Needle-Retrieval-ZH                                   -
dataset                                                     llama-2-7b-hf
--------- NeedleBench-4K-Multi-Needle-Reasoning ---------   -
dataset                                                     llama-2-7b-hf
Multi-Needle-Reasoning(M-RS)                                -
dataset                                                     llama-2-7b-hf
Multi-Needle-Reasoning-EN                                   -
dataset                                                     llama-2-7b-hf
Multi-Needle-Reasoning-ZH                                   -
dataset                                                     llama-2-7b-hf
2-Needle-EN-4K                                              -
dataset                                                     llama-2-7b-hf
2-Needle-ZH-4K                                              -
dataset                                                     llama-2-7b-hf
3-Needle-EN-4K                                              -
dataset                                                     llama-2-7b-hf
3-Needle-ZH-4K                                              -
dataset                                                     llama-2-7b-hf
4-Needle-EN-4K                                              -
dataset                                                     llama-2-7b-hf
4-Needle-ZH-4K                                              -
dataset                                                     llama-2-7b-hf
5-Needle-EN-4K                                              -
dataset                                                     llama-2-7b-hf
5-Needle-ZH-4K                                              -
dataset                                                     llama-2-7b-hf
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

[tonysy]

We use a chat model. Can you help confirm this, @DseidLi ?

[ainilian]

all the type of llama2 series in the paper, is chat?

[Mor-Li]

Hi @ainilian , for our tests, we focused on the chat models because base models struggle with long-text instruction following. Therefore, we didn't test base models for NeedleBench, as they have difficulty following complex instructions.

We’ve put significant engineering effort and care into ensuring accuracy. For example, in our default configuration, we optimized by choosing a specific number of repeats to ensure relatively stable results. This measure was implemented to maintain consistency in the outcomes. If you have any further questions, feel free to reach out, and we’d be happy to explore the reasons together.

[ainilian]

OK! Thanks!