Several detailed questions about reproduction

[KennardWang]

Hi, I am really interested in this ICSE paper published by you guys. Recently I have planned to reproduce the experiments but I face some difficulties.

1. I have reproduced the standard data set under both attendgru and ast-attendgru method. Due to the limit of my devices and network, I only implement model training for 5 epochs under attendgru (about 11 hours) and 3 epochs under ast-attendgru (about 6 hours). I see the original code provided in GitHub using 100 epochs, which means high accuracy. So I am not sure that my BLEU results is OK or not. Here is my results:

Model	Ba	B1	B2	B3	B4
attendgru, E05	19.14	37.88	21.4	14.66	11.3
attendgru, E03	19.24	38.65	21.77	14.66	11.12
ast-attendgru, E03	19.37	38.74	21.88	14.75	11.27

2. I still plan to do the challenge data set, but I do not find anything on website given by you. I only find a data set named "sbt", which only contains coms.tok and dats.tok. In your paper, you say "The challenge dataset contains two elements for each
   method: 1) the pre-processed comment, and 2) the SBT-AO representation of the Java code". So I guess "sbt" is the challenge data set. If I make a mistake, could you please tell me where to download the challenge data set, thx.

3. I see that you guys provide the final ast-attendgru trained model file (.h5) for both standard and challenge data. But to load the model acquires corresponding history configuration file, which I do not find. So I cannot do the prediction for the next step.

My development environment:

• Google Colab env, GPU with high RAM
• Keras==2.2.5
• tensorflow-gpu==1.14
• h5py==2.10.0

[acleclair]

1. We usually see BLEU scores settling in the 6-10 epoch range on the full FUNCOM dataset. Your scores generally seem to be in line with what we get.
2. The challenge dataset is the sbt data with the dats data set to zeros - so only the SBT AO is used to train the model
3. I am able to load the model without the history files using keras.models.load_model("{path_to_model.h5}"), I am not sure what issue you are running into.

[KennardWang]

Thanks for your reply. I think I need to explain a little bit detailed for my 3rd problem: In the line 175 of predict.py, the code is config = pickle.load(open(outdir+'/histories/'+modeltype+'_conf_'+timestart+'.pkl', 'rb')), which acquires the history configuration .pkl file. As the result, I cannot make the prediction due to lacking this file. I know that after model training, the history file will be generated automatically. But in the worst case, perhaps I can use your results directly if it costs too much on both time and computer resources. That's why I really need the history configuration file.

Another question is still about the 2nd problem. I am not sure how to train the model by using sbt dataset and challenge dataset. I just omit several lines about "sml" in train.py because sbt data set does not contain "sml.tok" like the standard dataset:

tdatvocabsize = tdatstok.vocab_size
comvocabsize = comstok.vocab_size
#smlvocabsize = smltok.vocab_size     // omit
...
print('tdatvocabsize %s' % (tdatvocabsize))
print('comvocabsize %s' % (comvocabsize))
#print('smlvocabsize %s' % (smlvocabsize))    // omit
...
config['tdatvocabsize'] = tdatvocabsize
config['comvocabsize'] = comvocabsize
#config['smlvocabsize'] = smlvocabsize    // omit
...
config['tdatlen'] = len(list(seqdata['dtrain'].values())[0])
config['comlen'] = len(list(seqdata['ctrain'].values())[0])
#config['smllen'] = len(list(seqdata['strain'].values())[0])    // omit

My training command: python train.py --model-type=attendgru --gpu=0

And now I am training attendgru model by using sbt dataset (the root directory changes to the sbt) without modifying any other code. I do not know the result until 6 hours later but I am worried about that my method is incorrect.