method for calculating p-value

[chenypic]

Thank you so much for your ml-stat-util.
I want to know the method you compute the p-value. Is it the same as the DeLong test?

[mateuszbuda]

Hi @chenypic. This repo implements bootstrapping. Based on a quick research, I don't think it's equivalent to the DeLong test.

[Amin-Tajgardoon]

@mateuszbuda Is there any reference or statistical justification behind your implementation? Usually bootstrapping is used on the training set to account for model uncertainty, which is computationally expensive when training large models. Bootstrapping the predicted probabilities is much faster but I haven't found any statistical proof on this method.

[mateuszbuda]

@Amin-Tajgardoon Here are selected results for the phrase "bootstrapping hypothesis testing" from Google:

• https://en.wikipedia.org/wiki/Bootstrapping_(statistics)#Deriving_confidence_intervals_from_the_bootstrap_distribution
• http://www.stat.ucla.edu/~rgould/110as02/bshypothesis.pdf
• http://www.stat.cmu.edu/~cshalizi/402/lectures/08-bootstrap/lecture-08.pdf
• https://www.tau.ac.il/~saharon/StatisticsSeminar_files/Hypothesis.pdf

[Amin-Tajgardoon]

@mateuszbuda Thank you for the reply. I'll take a look at the references.