About the reproduction.

[jumptoliujj]

We run experiments on PrOntoQA and FOLIO, the results of accuracy are only about 51%~53%. We run logic_program.py, self_refinement.py, and then logic_inference.py, evaluation.py. Any wrong with my steps? Correct me.

[RutaTang]

Hi @jumptoliujj,

Did you figure it out? I have also faced this issue. I am not sure whether and where I was doing wrong.

[jumptoliujj]

Hi @jumptoliujj,

Did you figure it out? I have also faced this issue. I am not sure whether and where I was doing wrong.

Just use text-davinci-003 instead of gpt-3.5-turbo, and you will get the similar results in the paper.

[RutaTang]

Thank you!

[zhuang-li]

I have the same issue.. The logical program only obtained 51.4 on ProntoQA with GPT-3.5-turbo, which is far from the reported result (61) in the paper.

[abhinandan12345678]

I tried to run the models as per the commands in readme but it is giving "Error in generating example".
In the requirements.txt the changes i made are:
changed certifi @ file:///croot/certifi_1671487769961/work/certifi to certifi==2022.12.7, because the url was giving error.
removed sklearn as it got deprecated.
Can anyone please help me regarding this issue.

[wernerolaf]

I have problems with Pyke as it is not always deterministic, also there are problems with cache