deploy: 6affc32

Author: msneubauer

_sources/intro.md

@@ -24,7 +24,7 @@ __Note__: *This schedule will evolve throughout the semseter*
 | Mar 17  | __SPRING BREAK - NO CLASSES__ |                                        |                            |
 | Mar 24  | {doc}`_sources/Week_09`       | NO HOMEWORK                            |                            |
 | Mar 31  | {doc}`_sources/Week_10`       | [HW 07](_sources/homework/Homework_07) |                            |
-| Apr 07  | {doc}`_sources/Week_11`       | HW 08                                  |                            |
+| Apr 07  | {doc}`_sources/Week_11`       | [HW 08](_sources/homework/Homework_08) |                            |
 | Apr 14  | {doc}`_sources/Week_12`       | HW 09                                  |                            |
 | Apr 21  | {doc}`_sources/Week_13`       | HW 10                                  | {doc}`_sources/Project_02` |
 | Apr 28  | {doc}`_sources/Week_14`       | HW 11                                  |                            |

_sources/lectures/Attention.html

@@ -1100,8 +1100,8 @@ <h3><span style="color:LightGreen">Sequence Padding and Attention Masking</span>
 <section id="span-style-color-orange-computing-the-reweighted-padded-attention-mask-span">
 <h2><span style="color:Orange">Computing the Reweighted Padded Attention Mask</span><a class="headerlink" href="#span-style-color-orange-computing-the-reweighted-padded-attention-mask-span" title="Permalink to this heading">#</a></h2>
 <p>Lets create some numbers so we can get a better idea of how this works. Let the tokens be <span class="math notranslate nohighlight">\(X = [10, 2, \text{&lt;pad&gt;}]\)</span>, so the third token is a padding token. Lets then also pretend, we pass this to our model, and when we go to compute our attention <span class="math notranslate nohighlight">\(QK^T\)</span>. The raw output before the Softmax is below:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-c5dabc6e-8740-41ec-b9bf-e976882993ce">
-<span class="eqno">(1)<a class="headerlink" href="#equation-c5dabc6e-8740-41ec-b9bf-e976882993ce" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-7df825ad-f729-4e01-a429-0de2dfa7af6f">
+<span class="eqno">(1)<a class="headerlink" href="#equation-7df825ad-f729-4e01-a429-0de2dfa7af6f" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \begin{bmatrix}
   7       &amp; -8   &amp; 6  \\
   -3       &amp; 2   &amp; 4   \\
@@ -1114,8 +1114,8 @@ <h2><span style="color:Orange">Computing the Reweighted Padded Attention Mask</s
 \text{Softmax}(\vec{x}) = \frac{e^{x_i}}{\sum_{j=1}^N{e^{x_j}}}
 \]</div>
 <p>If we ignore padding and everything right now, we can compute softmax for row of the matrix above:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-4338f025-2949-49d0-8afe-11e8c0d9b209">
-<span class="eqno">(2)<a class="headerlink" href="#equation-4338f025-2949-49d0-8afe-11e8c0d9b209" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-3d7fc9d9-5840-49d7-a0bd-c8b204413d60">
+<span class="eqno">(2)<a class="headerlink" href="#equation-3d7fc9d9-5840-49d7-a0bd-c8b204413d60" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \text{Softmax}
 \begin{bmatrix}
   7       &amp; -8   &amp; 6  \\
@@ -1134,17 +1134,17 @@ <h2><span style="color:Orange">Computing the Reweighted Padded Attention Mask</s
 \end{bmatrix}
 \end{equation}\]</div>
 <p>But what we need is to mask out all the tokens in this matrix related to padding. Just like we did in <a class="reference external" href="https://github.com/priyammaz/HAL-DL-From-Scratch/tree/main/PyTorch%20for%20NLP/GPT">GPT</a>, we will fill in the indexes of the that we want to mask with <span class="math notranslate nohighlight">\(-\infty\)</span>. If only the last token was a padding token in our sequence, then the attention before the softmax should be written as:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-2e87ec8a-e434-4c38-80bc-ed2550ad368a">
-<span class="eqno">(3)<a class="headerlink" href="#equation-2e87ec8a-e434-4c38-80bc-ed2550ad368a" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-9a32e649-a89b-4b74-b800-01afda0f53a4">
+<span class="eqno">(3)<a class="headerlink" href="#equation-9a32e649-a89b-4b74-b800-01afda0f53a4" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \begin{bmatrix}
   7       &amp; -8   &amp; -\infty  \\
   -3       &amp; 2   &amp; -\infty   \\
   1       &amp; 6  &amp; -\infty  \\
 \end{bmatrix}
 \end{equation}\]</div>
 <p>Taking the softmax of the rows of this matrix then gives:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-30afbd58-956b-4088-9f37-d260d1e2e2bb">
-<span class="eqno">(4)<a class="headerlink" href="#equation-30afbd58-956b-4088-9f37-d260d1e2e2bb" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-2f873261-4b94-4b67-a967-3fcacb525ea8">
+<span class="eqno">(4)<a class="headerlink" href="#equation-2f873261-4b94-4b67-a967-3fcacb525ea8" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \text{Softmax}
 \begin{bmatrix}
  7       &amp; -8   &amp; -\infty  \\
@@ -1186,8 +1186,8 @@ <h3><span style="color:LightGreen">Repeating to Match Attention Matrix Shape</sp
 <p><code class="docutils literal notranslate"><span class="pre">attn.shape</span></code> - (Batch x seq_len x seq_len)</p>
 <p><code class="docutils literal notranslate"><span class="pre">mask.shape</span></code> - (Batch x seq_len)</p>
 <p>It is clear that our mask is missing a dimension, and we need to repeat it. Lets take sequence_1 for instance that has a mask of [True, True, True, False]. Because the sequence length here is 4, lets repeat this row 4 times:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-0f4663e1-d517-424c-929c-d58bec2a96be">
-<span class="eqno">(5)<a class="headerlink" href="#equation-0f4663e1-d517-424c-929c-d58bec2a96be" title="Permalink to this equation">#</a></span>\[\begin{bmatrix}
+<div class="amsmath math notranslate nohighlight" id="equation-bc1060e2-24cd-4814-9a48-491ec577df1e">
+<span class="eqno">(5)<a class="headerlink" href="#equation-bc1060e2-24cd-4814-9a48-491ec577df1e" title="Permalink to this equation">#</a></span>\[\begin{bmatrix}
 \textrm{True} &amp; \textrm{True} &amp; \textrm{True} &amp; \textrm{False} \\
 \textrm{True} &amp; \textrm{True} &amp; \textrm{True} &amp; \textrm{False} \\
 \textrm{True} &amp; \textrm{True} &amp; \textrm{True} &amp; \textrm{False} \\
@@ -1447,8 +1447,8 @@ <h3><span style="color:LightGreen">Enforcing Causality</span><a class="headerlin
 <section id="span-style-color-lightgreen-computing-the-reweighted-causal-attention-mask-span">
 <h3><span style="color:LightGreen">Computing the Reweighted Causal Attention Mask</span><a class="headerlink" href="#span-style-color-lightgreen-computing-the-reweighted-causal-attention-mask-span" title="Permalink to this heading">#</a></h3>
 <p>Lets pretend the raw outputs of <span class="math notranslate nohighlight">\(QK^T\)</span>, before the softmax, is below:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-55ac036d-1dff-4bc2-9f4e-9b210b99ab2e">
-<span class="eqno">(6)<a class="headerlink" href="#equation-55ac036d-1dff-4bc2-9f4e-9b210b99ab2e" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-0d4eeafd-544e-4690-89fa-344895301282">
+<span class="eqno">(6)<a class="headerlink" href="#equation-0d4eeafd-544e-4690-89fa-344895301282" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \begin{bmatrix}
   7       &amp; -8   &amp; 6  \\
   -3       &amp; 2   &amp; 4   \\
@@ -1459,8 +1459,8 @@ <h3><span style="color:LightGreen">Computing the Reweighted Causal Attention Mas
 <div class="math notranslate nohighlight">
 \[\text{Softmax}(\vec{x}) = \frac{e^{x_i}}{\sum_{j=1}^N{e^{x_j}}}\]</div>
 <p>Then, we can compute softmax for row of the matrix above:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-f7c69116-9ea2-45a6-9347-b9735c86bf72">
-<span class="eqno">(7)<a class="headerlink" href="#equation-f7c69116-9ea2-45a6-9347-b9735c86bf72" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-4be3c332-cf21-433b-974e-898d9180a309">
+<span class="eqno">(7)<a class="headerlink" href="#equation-4be3c332-cf21-433b-974e-898d9180a309" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \text{Softmax}
 \begin{bmatrix}
   7       &amp; -8   &amp; 6  \\
@@ -1499,17 +1499,17 @@ <h3><span style="color:LightGreen">Computing the Reweighted Causal Attention Mas
 \text{Softmax}(x_2) = [\frac{e^{-3}}{e^{-3}+e^{2}+0}, \frac{e^{2}}{e^{-3}+e^{2}+0}, \frac{0}{e^{-3}+e^{2}+0}] = [\frac{e^{-3}}{e^{-3}+e^{2}+0}, \frac{e^{2}}{e^{-3}+e^{2}+0}, \frac{0}{e^{-3}+e^{2}+0}] = [0.0067, 0.9933, 0.0000]
 \]</div>
 <p>So we have exactly what we want! The attention weight of the last value is set to 0, so when we are on the second vector <span class="math notranslate nohighlight">\(x_2\)</span>, we cannot look forward to the future value vectors <span class="math notranslate nohighlight">\(v_3\)</span>, and the remaining parts add up to 1 so its still a probability vector! To do this correctly for the entire matrix, we can just substitute in the top triangle of <span class="math notranslate nohighlight">\(QK^T\)</span> with <span class="math notranslate nohighlight">\(-\infty\)</span>. This would look like:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-52a2ce1c-3b05-4134-9667-6aab57bea206">
-<span class="eqno">(8)<a class="headerlink" href="#equation-52a2ce1c-3b05-4134-9667-6aab57bea206" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-63c27fb0-75f0-4576-beac-bea1f4f13dbe">
+<span class="eqno">(8)<a class="headerlink" href="#equation-63c27fb0-75f0-4576-beac-bea1f4f13dbe" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \begin{bmatrix}
   7       &amp; -\infty   &amp; -\infty  \\
   -3       &amp; 2   &amp; -\infty   \\
   1       &amp; 6  &amp; -2   \\
 \end{bmatrix}
 \end{equation}\]</div>
 <p>Taking the softmax of the rows of this matrix then gives:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-25a0cf38-9596-43ea-af2a-e18ccf1b5c68">
-<span class="eqno">(9)<a class="headerlink" href="#equation-25a0cf38-9596-43ea-af2a-e18ccf1b5c68" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-eb0c75f0-cd5c-41eb-a097-a607d994f596">
+<span class="eqno">(9)<a class="headerlink" href="#equation-eb0c75f0-cd5c-41eb-a097-a607d994f596" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \text{Softmax}
 \begin{bmatrix}
   7       &amp; -\infty   &amp; -\infty  \\

intro.html

@@ -618,7 +618,7 @@ <h2><span style="color:Red">Calendar</span><a class="headerlink" href="#span-sty
 </tr>
 <tr class="row-odd"><td><p>Apr 07</p></td>
 <td><p><a class="reference internal" href="_sources/Week_11.html"><span class="doc">AI Explainablility and Uncertainty Quantification</span></a></p></td>
-<td><p>HW 08</p></td>
+<td><p><a class="reference internal" href="_sources/homework/Homework_08.html"><span class="doc std std-doc">HW 08</span></a></p></td>
 <td><p></p></td>
 </tr>
 <tr class="row-even"><td><p>Apr 14</p></td>