According to the code from official VQA api , accuracy is accounted only when agreeing answer is not same as discarded because in line 98, it is computing other GT answers which are not same as discarded one. But it varies from what you have done. you are assuming that it will be matched to all but discarded answer while in their API, they are computing it as if it will be matched to all but discarded and other which are same as discarded answers.
So, seems like it will (10-agreeing)*min(agreeing/3,1).
Please correct me if I am wrong.
@Cyanogenoid @pplantinga @guoyang9
According to the code from official VQA api , accuracy is accounted only when agreeing answer is not same as discarded because in line 98, it is computing other GT answers which are not same as discarded one. But it varies from what you have done. you are assuming that it will be matched to all but discarded answer while in their API, they are computing it as if it will be matched to all but discarded and other which are same as discarded answers.
So, seems like it will (10-agreeing)*min(agreeing/3,1).
Please correct me if I am wrong.
@Cyanogenoid @pplantinga @guoyang9