AMPL loss calculation method.

[MLPharmaTron]

Hi AMPL staff.

I've been running into some training failure as my dataset goes up. One potential reason I want to explore is additive error in metrics calculation.

1. Does AMPL calculate R2/RMSE per batch and average, or calculate the values once over the whole dataset?
2. If it is the former, is there a way to switch it to the latter?

Thanks in advance.

[stewarthe6]

Hello,

I believe we do the later and calculate the metrics over the whole dataset per epoch.

Maybe if you share a stack-trace I can see where the error is coming from.

Thanks

[MLPharmaTron]

Hi Stewar. Thank you for the response.

1. It seems by default there is an early stop built into AMPL? Ie. we don't need to select it. In fact, as a stress test on hardware, we were wondering if there is a way to deactivate this.

2. I am currently using a reverse geometric setup going from 256 → 16 in the hidden layer sizes. My understanding would normally be that the 16-dimensional representation is then passed through an activation function and into a final fully connected/output layer.
   However, I was unsure whether in AMPL the 16-dimensional layer itself is effectively the final layer, or whether it feeds into an additional fully connected output layer afterward.
   Could you please clarify how this is handled internally, and whether there is a way to explicitly configure an additional fully connected output layer if needed?

Thanks,
-M

[stewarthe6]

Hello M,

1. Yes, there is an early_stopping_patience parameter. If you set that to the same value as max_epochs, that will turn off early stopping.

   AMPL/atomsci/ddm/pipeline/parameter_parser.py

   Line 1090 in 7d6254e

   parser.add_argument(

2. I can't say. We don't construct the neural networks ourselves, we wrap a library called DeepChem. https://deepchem.readthedocs.io/en/latest/. It's open source so you can see how they implement each model type. If I had to guess, for single task regression, there would be a final layer that takes in 16 dimensions and produces a 1 dimensional output.

Thanks,
Stewart

[MLPharmaTron]

Hi Stewart.

Thanks for that response. It does seem to be a model complexity problem which we have diagnosed, and have looked through the DeepChem doc. I imagine AMPL's wrapper doesn't allow us from passing multple types of architectures/models through the dictionary at once, as a type of ensemble correct? I can't seem to find this function. If not would your recommendation to be to get closer to the source and use DeepChem or Pytorch itself?

Thanks,
-M

[stewarthe6]

Hi M,

No, we don't support training ensembles as a group. Usually when I work with ensembles, I train each architecture/model individually, with its own hyperparameter search, then combine their predictions.

Thanks,
Stewart

[MLPharmaTron]

Thanks Stewart!

I think we can close this discussion now, as I have a direction to go regarding my failure. And we've moved quite a bit away from the original conversation of loss.

Thanks,
M.