You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<h2><spanstyle="color:Orange">Computing the Reweighted Padded Attention Mask</span><aclass="headerlink" href="#span-style-color-orange-computing-the-reweighted-padded-attention-mask-span" title="Permalink to this heading">#</a></h2>
1101
1101
<p>Lets create some numbers so we can get a better idea of how this works. Let the tokens be <spanclass="math notranslate nohighlight">\(X = [10, 2, \text{<pad>}]\)</span>, so the third token is a padding token. Lets then also pretend, we pass this to our model, and when we go to compute our attention <spanclass="math notranslate nohighlight">\(QK^T\)</span>. The raw output before the Softmax is below:</p>
1102
-
<divclass="amsmath math notranslate nohighlight" id="equation-22d0d05a-babe-4218-84f2-1eaae730f3c9">
1103
-
<spanclass="eqno">(1)<aclass="headerlink" href="#equation-22d0d05a-babe-4218-84f2-1eaae730f3c9" title="Permalink to this equation">#</a></span>\[\begin{equation}
1102
+
<divclass="amsmath math notranslate nohighlight" id="equation-ad706dd6-a52e-4d11-ab77-c830602bba14">
1103
+
<spanclass="eqno">(1)<aclass="headerlink" href="#equation-ad706dd6-a52e-4d11-ab77-c830602bba14" title="Permalink to this equation">#</a></span>\[\begin{equation}
<p>If we ignore padding and everything right now, we can compute softmax for row of the matrix above:</p>
1116
-
<divclass="amsmath math notranslate nohighlight" id="equation-6b7555a5-6b71-4539-865e-23bc4614a3d8">
1117
-
<spanclass="eqno">(2)<aclass="headerlink" href="#equation-6b7555a5-6b71-4539-865e-23bc4614a3d8" title="Permalink to this equation">#</a></span>\[\begin{equation}
1116
+
<divclass="amsmath math notranslate nohighlight" id="equation-82e91dbe-9847-4d4c-ae3e-001922071aa0">
1117
+
<spanclass="eqno">(2)<aclass="headerlink" href="#equation-82e91dbe-9847-4d4c-ae3e-001922071aa0" title="Permalink to this equation">#</a></span>\[\begin{equation}
<p>But what we need is to mask out all the tokens in this matrix related to padding. Just like we did in <aclass="reference external" href="https://github.com/priyammaz/HAL-DL-From-Scratch/tree/main/PyTorch%20for%20NLP/GPT">GPT</a>, we will fill in the indexes of the that we want to mask with <spanclass="math notranslate nohighlight">\(-\infty\)</span>. If only the last token was a padding token in our sequence, then the attention before the softmax should be written as:</p>
1136
-
<divclass="amsmath math notranslate nohighlight" id="equation-947d5f8e-5b2d-4424-9aa8-b5d960360ee5">
1137
-
<spanclass="eqno">(3)<aclass="headerlink" href="#equation-947d5f8e-5b2d-4424-9aa8-b5d960360ee5" title="Permalink to this equation">#</a></span>\[\begin{equation}
1136
+
<divclass="amsmath math notranslate nohighlight" id="equation-4d04027e-7c32-4047-ba97-2231fcf2eafa">
1137
+
<spanclass="eqno">(3)<aclass="headerlink" href="#equation-4d04027e-7c32-4047-ba97-2231fcf2eafa" title="Permalink to this equation">#</a></span>\[\begin{equation}
1138
1138
\begin{bmatrix}
1139
1139
7 & -8 & -\infty \\
1140
1140
-3 & 2 & -\infty \\
1141
1141
1 & 6 & -\infty \\
1142
1142
\end{bmatrix}
1143
1143
\end{equation}\]</div>
1144
1144
<p>Taking the softmax of the rows of this matrix then gives:</p>
1145
-
<divclass="amsmath math notranslate nohighlight" id="equation-ce560d17-0d7e-4acb-a9f3-be4f520bc5dd">
1146
-
<spanclass="eqno">(4)<aclass="headerlink" href="#equation-ce560d17-0d7e-4acb-a9f3-be4f520bc5dd" title="Permalink to this equation">#</a></span>\[\begin{equation}
1145
+
<divclass="amsmath math notranslate nohighlight" id="equation-43d25f1d-801a-4ae0-b49d-4658d866b1fc">
1146
+
<spanclass="eqno">(4)<aclass="headerlink" href="#equation-43d25f1d-801a-4ae0-b49d-4658d866b1fc" title="Permalink to this equation">#</a></span>\[\begin{equation}
1147
1147
\text{Softmax}
1148
1148
\begin{bmatrix}
1149
1149
7 & -8 & -\infty \\
@@ -1185,8 +1185,8 @@ <h3><span style="color:LightGreen">Repeating to Match Attention Matrix Shape</sp
1185
1185
<p><codeclass="docutils literal notranslate"><spanclass="pre">attn.shape</span></code> - (Batch x seq_len x seq_len)</p>
1186
1186
<p><codeclass="docutils literal notranslate"><spanclass="pre">mask.shape</span></code> - (Batch x seq_len)</p>
1187
1187
<p>It is clear that our mask is missing a dimension, and we need to repeat it. Lets take sequence_1 for instance that has a mask of [True, True, True, False]. Because the sequence length here is 4, lets repeat this row 4 times:</p>
1188
-
<divclass="amsmath math notranslate nohighlight" id="equation-b56a0fc0-20d0-4ce0-b47b-12191ff88df9">
1189
-
<spanclass="eqno">(5)<aclass="headerlink" href="#equation-b56a0fc0-20d0-4ce0-b47b-12191ff88df9" title="Permalink to this equation">#</a></span>\[\begin{bmatrix}
1188
+
<divclass="amsmath math notranslate nohighlight" id="equation-75947fc6-ced9-4ca6-bb5d-602c58a7420d">
1189
+
<spanclass="eqno">(5)<aclass="headerlink" href="#equation-75947fc6-ced9-4ca6-bb5d-602c58a7420d" title="Permalink to this equation">#</a></span>\[\begin{bmatrix}
1190
1190
\textrm{True} & \textrm{True} & \textrm{True} & \textrm{False} \\
1191
1191
\textrm{True} & \textrm{True} & \textrm{True} & \textrm{False} \\
1192
1192
\textrm{True} & \textrm{True} & \textrm{True} & \textrm{False} \\
<h3><spanstyle="color:LightGreen">Computing the Reweighted Causal Attention Mask</span><aclass="headerlink" href="#span-style-color-lightgreen-computing-the-reweighted-causal-attention-mask-span" title="Permalink to this heading">#</a></h3>
1448
1448
<p>Lets pretend the raw outputs of <spanclass="math notranslate nohighlight">\(QK^T\)</span>, before the softmax, is below:</p>
1449
-
<divclass="amsmath math notranslate nohighlight" id="equation-92a7365d-03fc-4811-ba7e-240a794e9c1c">
1450
-
<spanclass="eqno">(6)<aclass="headerlink" href="#equation-92a7365d-03fc-4811-ba7e-240a794e9c1c" title="Permalink to this equation">#</a></span>\[\begin{equation}
1449
+
<divclass="amsmath math notranslate nohighlight" id="equation-f2df8944-33d4-419c-8837-3107f1fa9e44">
1450
+
<spanclass="eqno">(6)<aclass="headerlink" href="#equation-f2df8944-33d4-419c-8837-3107f1fa9e44" title="Permalink to this equation">#</a></span>\[\begin{equation}
1451
1451
\begin{bmatrix}
1452
1452
7 & -8 & 6 \\
1453
1453
-3 & 2 & 4 \\
@@ -1458,8 +1458,8 @@ <h3><span style="color:LightGreen">Computing the Reweighted Causal Attention Mas
<p>Then, we can compute softmax for row of the matrix above:</p>
1461
-
<divclass="amsmath math notranslate nohighlight" id="equation-f3baddac-d202-4168-8e1f-b5bae70e5171">
1462
-
<spanclass="eqno">(7)<aclass="headerlink" href="#equation-f3baddac-d202-4168-8e1f-b5bae70e5171" title="Permalink to this equation">#</a></span>\[\begin{equation}
1461
+
<divclass="amsmath math notranslate nohighlight" id="equation-2c719a4b-e8a9-4bee-8946-4ac54ff3388a">
1462
+
<spanclass="eqno">(7)<aclass="headerlink" href="#equation-2c719a4b-e8a9-4bee-8946-4ac54ff3388a" title="Permalink to this equation">#</a></span>\[\begin{equation}
1463
1463
\text{Softmax}
1464
1464
\begin{bmatrix}
1465
1465
7 & -8 & 6 \\
@@ -1498,17 +1498,17 @@ <h3><span style="color:LightGreen">Computing the Reweighted Causal Attention Mas
<p>So we have exactly what we want! The attention weight of the last value is set to 0, so when we are on the second vector <spanclass="math notranslate nohighlight">\(x_2\)</span>, we cannot look forward to the future value vectors <spanclass="math notranslate nohighlight">\(v_3\)</span>, and the remaining parts add up to 1 so its still a probability vector! To do this correctly for the entire matrix, we can just substitute in the top triangle of <spanclass="math notranslate nohighlight">\(QK^T\)</span> with <spanclass="math notranslate nohighlight">\(-\infty\)</span>. This would look like:</p>
1501
-
<divclass="amsmath math notranslate nohighlight" id="equation-349bad15-9a14-4cfa-9921-4f9b9bf0b2f9">
1502
-
<spanclass="eqno">(8)<aclass="headerlink" href="#equation-349bad15-9a14-4cfa-9921-4f9b9bf0b2f9" title="Permalink to this equation">#</a></span>\[\begin{equation}
1501
+
<divclass="amsmath math notranslate nohighlight" id="equation-efa6a921-9021-4f0e-a7d2-518e8e243449">
1502
+
<spanclass="eqno">(8)<aclass="headerlink" href="#equation-efa6a921-9021-4f0e-a7d2-518e8e243449" title="Permalink to this equation">#</a></span>\[\begin{equation}
1503
1503
\begin{bmatrix}
1504
1504
7 & -\infty & -\infty \\
1505
1505
-3 & 2 & -\infty \\
1506
1506
1 & 6 & -2 \\
1507
1507
\end{bmatrix}
1508
1508
\end{equation}\]</div>
1509
1509
<p>Taking the softmax of the rows of this matrix then gives:</p>
1510
-
<divclass="amsmath math notranslate nohighlight" id="equation-c7c3c1b0-3306-42f6-9fbb-3b6d89d557ad">
1511
-
<spanclass="eqno">(9)<aclass="headerlink" href="#equation-c7c3c1b0-3306-42f6-9fbb-3b6d89d557ad" title="Permalink to this equation">#</a></span>\[\begin{equation}
1510
+
<divclass="amsmath math notranslate nohighlight" id="equation-1e84e5b6-1ac3-4bef-b520-dde67d6cca4a">
1511
+
<spanclass="eqno">(9)<aclass="headerlink" href="#equation-1e84e5b6-1ac3-4bef-b520-dde67d6cca4a" title="Permalink to this equation">#</a></span>\[\begin{equation}
<h2><spanstyle="color:Orange">Example: Time Series Anomaly Detection using LSTM Autoencoders</span><aclass="headerlink" href="#span-style-color-orange-example-time-series-anomaly-detection-using-lstm-autoencoders-span" title="Permalink to this heading">#</a></h2>
633
-
<p>In this example,</p>
629
+
<p>In this example, we will learn to:</p>
630
+
<ulclass="simple">
631
+
<li><p>Prepare a dataset for Anomaly Detection from Time Series Data</p></li>
632
+
<li><p>Build an LSTM Autoencoder with PyTorch</p></li>
633
+
<li><p>Train and evaluate your model</p></li>
634
+
<li><p>Choose a threshold for anomaly detection</p></li>
635
+
<li><p>Classify unseen examples as normal or anomaly</p></li>
636
+
</ul>
637
+
<p>While our Time Series data is univariate (we have only 1 feature), the code should work for multivariate datasets (multiple features) with little or no modification. Feel free to try it!</p>
<h3><spanstyle="color:LightGreen">Data</span><aclass="headerlink" href="#span-style-color-lightgreen-data-span" title="Permalink to this heading">#</a></h3>
636
640
<p>The <aclass="reference external" href="http://timeseriesclassification.com/description.php?Dataset=ECG5000">dataset</a> contains 5,000 Time Series examples (obtained with ECG) with 140 timesteps. Each sequence corresponds to a single heartbeat from a single patient with congestive heart failure.</p>
<spanclass="n">class_names</span><spanclass="o">=</span><spanclass="p">[</span><spanclass="s1">'Normal'</span><spanclass="p">,</span><spanclass="s1">'R on T'</span><spanclass="p">,</span><spanclass="s1">'PVC'</span><spanclass="p">,</span><spanclass="s1">'SP'</span><spanclass="p">,</span><spanclass="s1">'UB'</span><spanclass="p">]</span>
885
+
<spanclass="c1">#class_names = ['Normal','R on T','PVC','SP','UB'] # This ordering sometimes produces wrong counts histogram. Need to check if it affects plots that use class_names</span>
886
+
<spanclass="n">class_names</span><spanclass="o">=</span><spanclass="p">[</span><spanclass="s1">'Normal'</span><spanclass="p">,</span><spanclass="s1">'PVC'</span><spanclass="p">,</span><spanclass="s1">'R on T'</span><spanclass="p">,</span><spanclass="s1">'SP'</span><spanclass="p">,</span><spanclass="s1">'UB'</span><spanclass="p">]</span>
882
887
</pre></div>
883
888
</div>
884
889
</div>
@@ -918,14 +923,14 @@ <h3><span style="color:LightGreen">Exploratory Data Analysis</span><a class="hea
0 commit comments