From 4fa0c0a981e751b30949d297ed8cae7c350fde25 Mon Sep 17 00:00:00 2001 From: Luca Favini Date: Tue, 3 Mar 2026 16:39:17 +0100 Subject: [PATCH 1/2] Raw notes for lecture 14 --- chapters/9-deep-learning.typ | 110 +++++++++++++++++++++++++++++++++++ main.typ | 7 +++ 2 files changed, 117 insertions(+) create mode 100644 chapters/9-deep-learning.typ diff --git a/chapters/9-deep-learning.typ b/chapters/9-deep-learning.typ new file mode 100644 index 0000000..b8c11f2 --- /dev/null +++ b/chapters/9-deep-learning.typ @@ -0,0 +1,110 @@ +#import "../template.typ": * + += Deep Learning + +== Neural Network + +Complex aggregation of networks, inspired by the human brain. +A huge number of single cells, which behaviour can be described very simply + +/ Neuron: an input/output system, a computational thing. + It depends on the stimuli that comes from the outside (inputs) but also from some internal state. + Each input is connected, and has a weight, that increases or decreases the perception of that input. + +/ Activation function: ???. Most of the times it is non-linear. + +The computational power is not because of the power of the simple cell, but because of the huge number of them and the strong interconnection. + +We will see a NN that does image classifications of digits. + +For this example, we use for classification, but it can also be used for regression. + +The dataset: +- the digits are not rotated +- the digits are equivalently distributed (10% 0, 10% 1, ...) + +/ Tensor: multi-dimensional vector + +To get the classification of an image is to linarize the image, run throught the network and look at the values of the output neurons. +But this is slow. + +To leverage the power of SIMD instruction of the GPU, we want to process multiple images in parallel. +So we stack multiple digits one over the other to form a matrix of multiple images to process. + +Thanks to this representation, we can multiply it by the matrix representing the weights of the connections, which are compatible for product. +Then we add the bias and we get the result ready to be passed to an activation function: +$ L = X dot W + b $ + +One common activation function is the softmax: +$ "softmax"(L_n) = (e^(L_n))/(|| e^L ||) $ + +To check how good is a network, we can use a loss function. + +How do we "train" the NN? +We initialize the weights of all the connections to some value and then we apply some local gradient descent to optimize the loss function. + +In practice, we can do that using `keras` metalibrary of `tensoflow` that lets use reason directly in terms of layers instead of single neurons. + +We intialize a NN specifying the layers. +In out example, we have $28 times 28$ input neuros for the input layer and a layer of $10$ neurons for the output. +These are strongly interconnected, so it is a dense layer. + +```python +da notebook // TODO +``` + +Instead of computing the derivatives and then using the analytical form for the training, modern ml libraries use a technique called back-propagation. +We will see this better next lecture. + +/ Epoch: unit of measure for time. + Continuing the optimization by processing the train data several times. + Once the whole dataset has been used, an epoch expired. + +With a simple neural model, we get like 90% accuracy, which is not enough. +We can stack more layers to improve accuracy. + +A simple model, with no hidden layers, can separate only linearly separable binary datasets. +Models with hidden layers, can separate any binary function. + +Each hidden layer reduces the number of neurons in that layer. +The idea is that each neuron of a layer represents a feature present in the data of the previous layer. +So they tend to become less and less, by classifying these features. + +How do we know how many layers and how many neurons per layer? Open problem: just try and evaluate the results. + +It is really easy to grow the number of parameters with hidden layers. +Each parameter is an unknown in the loss function that needs to be minimized. + +If we use sigmoid for activation function in the hidden layers, we get a problem: vanishing. // TODO???. +Instead of that, we can use ReLU as activation function. + +We see that there an overfitting trend is starting. + +/ Learning rate: the gradient gives us the direction where to move, but it does not specify how much. + The learning rate is how much we move into that direction, which is also how fast the model learns. + Typically we start with an high learning rate and esponentially lower it with epochs increasing. + +The more complex is the model, the more it is plastic: it can adapt to more complex data. +But this calls for overfitting. +To fight overfitting, historically we used penalization, a more modert technique is drop out. + +/ Drop out: when updating weights in backpropagation, we toss a coin and update only if the coin is ok (the coin is not 50%). + +== Convolutional NN + +Linearization does not work well if we want to identify features that are in a certain area. + +We do not linearize the image. + +We start by fixing a small part of the image and compute the value of the neuron. +Then we start by moving the area and do that for another portion of the image. +But WITHOUT changing the weights of the connections, so that we are calculating the amount of a fixed feature in a certain area. + +We do that for all areas (not forced by moving of 1px), with even padding for borders so that the new layer is as big as the original one. + +Then we can do many sublayers, recognizing different features. +Each of these layers is called a convolutional network. + +Sooner of later, we need to linearize things. // TODO: why? +- we can add a layer with no weight that just moves from a 3d neurons into a linear layer +- each convolutional layer can be converted into a single output, the amount of that feature in the original input diff --git a/main.typ b/main.typ index 17ee1cd..3dab2eb 100644 --- a/main.typ +++ b/main.typ @@ -42,6 +42,13 @@ #include "chapters/1-hdfs-mapreduce.typ" #include "chapters/2-link-analysis.typ" + + + + + +#include "chapters/9-deep-learning.typ" + #show: part.with("Implementation", chapters-numbering: "A.1.", reset-chapters: true) #include "chapters/a-spark.typ" From 1acaa269164fa7f8f494c4268eca0f759ebe5ced Mon Sep 17 00:00:00 2001 From: Luca Favini Date: Thu, 5 Mar 2026 18:11:16 +0100 Subject: [PATCH 2/2] Raw notes for lecture 15 --- chapters/9-deep-learning.typ | 121 +++++++++++++++++++++++++++++++++++ 1 file changed, 121 insertions(+) diff --git a/chapters/9-deep-learning.typ b/chapters/9-deep-learning.typ index b8c11f2..0d52769 100644 --- a/chapters/9-deep-learning.typ +++ b/chapters/9-deep-learning.typ @@ -108,3 +108,124 @@ Each of these layers is called a convolutional network. Sooner of later, we need to linearize things. // TODO: why? - we can add a layer with no weight that just moves from a 3d neurons into a linear layer - each convolutional layer can be converted into a single output, the amount of that feature in the original input + +// TODO: end of convulutional + +== Recurrent Neural Network + +Autoregressive. + +We take $x$ characters as input, and put $x-1$ of these as output. +We are interested in the last one that gets generated. +Then we move all and generate the next character and so forth. + +We add an output layer to have a probability distribution on each possible character and not a crisp output. +So that we get the probability of prediction of each character for the next one. +If we always pick the character with the highest probability, we get a deterministic generation. +Or we can simulate an extraction based on the probabilities. +Usually some mechanism that mix these two approach are used, both probabilistic but not on all the characters. + +Recurrent NN are history. +In the last 10 years, transformers and LLM are the new form of deep learning and are much more efficient. + +== transformer + +A trasnformer is an highly structured Neural network. +It is not complex (uses all things we already saw), but is complicated (there are a lot of them in a complex structure). + +A big advantage is that they can be trained on parallel hardware, GPU. +But they are very very big, the training is very very expensive. + +HuggingFace: repository of pre-trained neural networks (and transformers). + +Take a pre-trained model and fine-tune it: adapt the structure of the model to a specific context and retrain only a small part of the parameters of the model. + +We will use an autorecurrent approach. +We start with the input and an output string as output. + +IN the same way as before, we take the generated token and append to the output. +We don't have a sliding window, the input always stays there and the output icnreases step by step (and never decreases). + +- Tokenization: divide the input in tokens +- Vectorization: transform each token in a number so that it can be processed by the model + +The main problem of RNN is that they have a short memory (because of the sliding window thing). +To overcome this problem, the attention mechanism is introduced. + +Exists multiple architectures of transformers, some with only decoder, some with only encoder and some with both. + +=== Embeddings + +Take a string length $n$ from an alphabet and trasform it into a matrix $n times d$. +Each character is trasformed in a vector, so the string is a matrix. +$d$ is typically 512, this will be important for the whole process. + +Not only encode a word in a vector, but also similarity as proximity. + +=== Positional Encoding + +Because of the next steps are independent of order, we need to inject manually order so that it will be understood. +So we add like a progressive sequence to each embedding. + +=== Encoding + +Multiple encoding blocks. +Learning meaningful representation of the data. + +- MHSA: learning dependencies between the tokens (even dependencies from far tokens) +- Normalization +- Feed Forward Neural Network (FFNN) +- Normalization + +Other than that, some rediduals connections (that skips some blocks) are added. +This is useful to counteract vanishing and exploding gradients. + +=== Self Attention mechanism + +Dealing with long term context and disambiguation. + +The idea is to move the vectors of something, based on the context we are in. + +E.g. the embedding of Apple can be close to organe or bananas, but also close the google or microsoft. +Based on the current context, we move the embedding closer to one or to the other (fruits or tech). + +To do that, we compute the similarity between all pairs of words in the sentence. // TODO: ??? + +Usually, similarity is well caught by inner products, but there are some problems: +- product tend to increase a lot with the dimension of the embedding, so we normalize +- product can be negative, so we apply softmax + +$ X^"att" = "softmax"((X X^T)/sqrt(d)) dot X $ + +=== MUlti-head self attention + +But the same word could have multiple meanings "I was eating an apple at the apple store". + +The idea is to use multiple embeddings and calculate attention for all of these. +That poses multiple problems: +- good embeddings are difficult to find +- big dimensionality + +To fix that problem, we keep only one real embedding and then applying linear transformations. +This "moves" around the points of the space, so that the points are closer in similar ways. + +We apply attention to the embedding with different linear transformations. +The output of all attentions is concatenated. +The size of each one is smaller so that the concatenation gives back to the dimension $d$. + +=== Masked attention + +Because the output is not full, we place empty string. +But that must be encoded into a valid vector, and computing the attention for that vector gives gibberish (the attention is computer for the whole output). + +So, we mask that by placing $-infinity$ where the output token is not yet generated. + +=== Training + +These are trained using back-propagation, like NN. + +All the parameters that can be decided are trained (the linear transformation, the neurons, ...). + +These can be trained using self-supervising techniques. +The web is full of text sequences (like wikipedia articles). +The first part of the sequence is fed and the continuation is the part that needs to be generated.