02. PyTorch Neural Network Classification - Excercise 6: Tanh()

[jonorl]

I managed to sucessfully do the excercises for the 2nd module, but under the excercises, number 6 asks to replicate the Tanh (hyperbolic tangent) activation function in pure PyTorch. I did it in a similar way to Sigmoid() and ReLU() but I don't understand if the excercise is referncing Tanh because we were supposed to use it in the previous moons excercise.

I tried using Tanh() replacing nn.ReLU() on my moon module with nn.tanh() as well as replacing torch.sigmoid() with torch.tanh() when transforming logits to predictions but in both cases the model needed more epochs to get to the target 96% accuracy. Am I doing something wrong?

[Haji-Suleman]

🧠 Tanh Activation – Exercise Clarification

I’m not doing anything wrong 🙂

• The exercise asks to replicate tanh using pure PyTorch ops, not just replace ReLU with nn.Tanh().
• tanh(x) outputs values in (-1, 1), so learning is usually slower than ReLU → more epochs is normal.
• Unlike sigmoid, tanh is not a probability function.

Important Notes

• If using BCEWithLogitsLoss, do NOT apply tanh/sigmoid during training.
• Use raw logits for loss.
• For predictions with tanh, use:

  preds = y_hat > 0.0

[sergo8online]

@jonorl i used tanh() as activation function(not Relu) and got accuracy=1 with 180 epoch, but i also got acc=1 with relu and other activation functions, so i think activation function is not matter so you didnt do anything wrong, BUT you can't just easely replace sigmoid with tanh, becouse if you are using tanh your labels will be 1 or -1, but in our dataset there is 0 and 1.

P.S.
Sorry for my english)

[Deetjepateeteke]

The labels of the data will not become -1 or 1. This is not what an activation function does.

Instead, the values that are running through your neural network will be normalized to the interval ]-1, 1[, after going through an nn.Linear layer.

The purpose of an activation function, like Tanh is to add non-linearity to your model, not to transform your labels.

[Deetjepateeteke]

You're not doing anything wrong, it's just that different activation functions serve different purposes.

• ReLU is mostly used in feedforward neural networks.
• Tanh excels when used in RNNs (Recurrent Neural Networks) and some other non-feedforward neural nets.

This does not mean that using Tanh in a feedforward neural network (as you did) will not work.

The Tanh activation function pushes the gradients, that are calculated during backpropagation, close to zero. This is useful in recurrent neural networks, but in combination with a feedforward network (like yours), it will take the model longer to train, because the gradients are so small, thus the changes to the model's weights and biases per epoch will be minimal.

I think the exercise just served the purpose of getting familiar with some other common activation functions that you'll run into when you get deeper into deep learning.