deploy: a311877

Author: msneubauer

_sources/_sources/lectures/UnsupervisedLearningAnomalyDetection.ipynb

@@ -236,7 +236,17 @@
       "source": [
         "The [dataset](http://timeseriesclassification.com/description.php?Dataset=ECG5000) contains 5,000 Time Series examples (obtained with ECG) with 140 timesteps. Each sequence corresponds to a single heartbeat from a single patient with congestive heart failure.\n",
         "\n",
-        "> An electrocardiogram (ECG or EKG) is a test that checks how your heart is functioning by measuring the electrical activity of the heart. With each heart beat, an electrical impulse (or wave) travels through your heart. This wave causes the muscle to squeeze and pump blood from the heart. [Source](https://www.heartandstroke.ca/heart/tests/electrocardiogram)\n",
+        "> An electrocardiogram (ECG or EKG) is a test that checks how your heart is functioning by measuring the electrical activity of the heart. It measures the electrical activity of your heart. It uses small electrodes attached to your skin to detect the tiny electrical signals that control your heartbeat. With each heart beat, an electrical impulse (or wave) travels through your heart. This wave causes the muscle to squeeze and pump blood from the heart. [Source](https://www.heartandstroke.ca/heart/tests/electrocardiogram)\n",
+        "\n",
+        "* ___<span style=\"color:Violet\">What it measures</span>___: An ECG records the electrical impulses that cause your heart to beat, showing how the signals travel through your heart's chambers.\n",
+        "\n",
+        "* ___<span style=\"color:Violet\">How it's done</span>___: During an ECG, electrodes are attached to your chest, arms, and legs using adhesive patches or small suction cups.\n",
+        "\n",
+        "* ___<span style=\"color:Violet\">What it shows</span>___: The ECG produces a graphical representation of your heart's electrical activity, known as an ECG tracing.\n",
+        "\n",
+        "* ___<span style=\"color:Violet\">Why it's used</span>___: ECGs are used to diagnose and monitor various heart conditions, including arrhythmias, coronary artery disease, heart attacks, and heart failure.\n",
+        "\n",
+        "* ___<span style=\"color:Violet\">What it can reveal</span>___: An ECG can help doctors detect changes in heart rate, rhythm, and electrical conduction, which may indicate a heart problem.\n",
         "\n",
         "We have 5 types of hearbeats (classes):\n",
         "\n",

_sources/lectures/Attention.html

@@ -1099,8 +1099,8 @@ <h3><span style="color:LightGreen">Sequence Padding and Attention Masking</span>
 <section id="span-style-color-orange-computing-the-reweighted-padded-attention-mask-span">
 <h2><span style="color:Orange">Computing the Reweighted Padded Attention Mask</span><a class="headerlink" href="#span-style-color-orange-computing-the-reweighted-padded-attention-mask-span" title="Permalink to this heading">#</a></h2>
 <p>Lets create some numbers so we can get a better idea of how this works. Let the tokens be <span class="math notranslate nohighlight">\(X = [10, 2, \text{&lt;pad&gt;}]\)</span>, so the third token is a padding token. Lets then also pretend, we pass this to our model, and when we go to compute our attention <span class="math notranslate nohighlight">\(QK^T\)</span>. The raw output before the Softmax is below:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-c70790f6-1a83-47ed-acde-dc45b63490b5">
-<span class="eqno">(1)<a class="headerlink" href="#equation-c70790f6-1a83-47ed-acde-dc45b63490b5" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-830c9138-1a65-49b8-872b-1acddec3394f">
+<span class="eqno">(1)<a class="headerlink" href="#equation-830c9138-1a65-49b8-872b-1acddec3394f" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \begin{bmatrix}
   7       &amp; -8   &amp; 6  \\
   -3       &amp; 2   &amp; 4   \\
@@ -1113,8 +1113,8 @@ <h2><span style="color:Orange">Computing the Reweighted Padded Attention Mask</s
 \text{Softmax}(\vec{x}) = \frac{e^{x_i}}{\sum_{j=1}^N{e^{x_j}}}
 \]</div>
 <p>If we ignore padding and everything right now, we can compute softmax for row of the matrix above:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-2e2338aa-104a-4755-9040-eba4460729fd">
-<span class="eqno">(2)<a class="headerlink" href="#equation-2e2338aa-104a-4755-9040-eba4460729fd" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-a3fd887d-05d9-4258-be55-51f090e8e047">
+<span class="eqno">(2)<a class="headerlink" href="#equation-a3fd887d-05d9-4258-be55-51f090e8e047" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \text{Softmax}
 \begin{bmatrix}
   7       &amp; -8   &amp; 6  \\
@@ -1133,17 +1133,17 @@ <h2><span style="color:Orange">Computing the Reweighted Padded Attention Mask</s
 \end{bmatrix}
 \end{equation}\]</div>
 <p>But what we need is to mask out all the tokens in this matrix related to padding. Just like we did in <a class="reference external" href="https://github.com/priyammaz/HAL-DL-From-Scratch/tree/main/PyTorch%20for%20NLP/GPT">GPT</a>, we will fill in the indexes of the that we want to mask with <span class="math notranslate nohighlight">\(-\infty\)</span>. If only the last token was a padding token in our sequence, then the attention before the softmax should be written as:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-4c716b8f-af62-4d56-97fe-b00066113dda">
-<span class="eqno">(3)<a class="headerlink" href="#equation-4c716b8f-af62-4d56-97fe-b00066113dda" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-9c3f551c-a052-48b9-965d-1ad4e740f523">
+<span class="eqno">(3)<a class="headerlink" href="#equation-9c3f551c-a052-48b9-965d-1ad4e740f523" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \begin{bmatrix}
   7       &amp; -8   &amp; -\infty  \\
   -3       &amp; 2   &amp; -\infty   \\
   1       &amp; 6  &amp; -\infty  \\
 \end{bmatrix}
 \end{equation}\]</div>
 <p>Taking the softmax of the rows of this matrix then gives:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-ad16f16b-eced-4c4d-9cac-87c514955daf">
-<span class="eqno">(4)<a class="headerlink" href="#equation-ad16f16b-eced-4c4d-9cac-87c514955daf" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-47541027-2870-4ee4-9507-0f0e5a526c4d">
+<span class="eqno">(4)<a class="headerlink" href="#equation-47541027-2870-4ee4-9507-0f0e5a526c4d" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \text{Softmax}
 \begin{bmatrix}
  7       &amp; -8   &amp; -\infty  \\
@@ -1185,8 +1185,8 @@ <h3><span style="color:LightGreen">Repeating to Match Attention Matrix Shape</sp
 <p><code class="docutils literal notranslate"><span class="pre">attn.shape</span></code> - (Batch x seq_len x seq_len)</p>
 <p><code class="docutils literal notranslate"><span class="pre">mask.shape</span></code> - (Batch x seq_len)</p>
 <p>It is clear that our mask is missing a dimension, and we need to repeat it. Lets take sequence_1 for instance that has a mask of [True, True, True, False]. Because the sequence length here is 4, lets repeat this row 4 times:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-9abc9429-fbf9-4e1e-9a38-f772a6250c1f">
-<span class="eqno">(5)<a class="headerlink" href="#equation-9abc9429-fbf9-4e1e-9a38-f772a6250c1f" title="Permalink to this equation">#</a></span>\[\begin{bmatrix}
+<div class="amsmath math notranslate nohighlight" id="equation-8ea9ab45-1c1e-4010-bb7c-cb72e2bc6838">
+<span class="eqno">(5)<a class="headerlink" href="#equation-8ea9ab45-1c1e-4010-bb7c-cb72e2bc6838" title="Permalink to this equation">#</a></span>\[\begin{bmatrix}
 \textrm{True} &amp; \textrm{True} &amp; \textrm{True} &amp; \textrm{False} \\
 \textrm{True} &amp; \textrm{True} &amp; \textrm{True} &amp; \textrm{False} \\
 \textrm{True} &amp; \textrm{True} &amp; \textrm{True} &amp; \textrm{False} \\
@@ -1446,8 +1446,8 @@ <h3><span style="color:LightGreen">Enforcing Causality</span><a class="headerlin
 <section id="span-style-color-lightgreen-computing-the-reweighted-causal-attention-mask-span">
 <h3><span style="color:LightGreen">Computing the Reweighted Causal Attention Mask</span><a class="headerlink" href="#span-style-color-lightgreen-computing-the-reweighted-causal-attention-mask-span" title="Permalink to this heading">#</a></h3>
 <p>Lets pretend the raw outputs of <span class="math notranslate nohighlight">\(QK^T\)</span>, before the softmax, is below:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-a3bcd113-24b6-4be0-8074-bfa67eef5331">
-<span class="eqno">(6)<a class="headerlink" href="#equation-a3bcd113-24b6-4be0-8074-bfa67eef5331" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-8e853f4e-f7df-491d-8aa1-d10aaf537956">
+<span class="eqno">(6)<a class="headerlink" href="#equation-8e853f4e-f7df-491d-8aa1-d10aaf537956" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \begin{bmatrix}
   7       &amp; -8   &amp; 6  \\
   -3       &amp; 2   &amp; 4   \\
@@ -1458,8 +1458,8 @@ <h3><span style="color:LightGreen">Computing the Reweighted Causal Attention Mas
 <div class="math notranslate nohighlight">
 \[\text{Softmax}(\vec{x}) = \frac{e^{x_i}}{\sum_{j=1}^N{e^{x_j}}}\]</div>
 <p>Then, we can compute softmax for row of the matrix above:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-658af593-8f9f-4139-be56-fb460ca2b1eb">
-<span class="eqno">(7)<a class="headerlink" href="#equation-658af593-8f9f-4139-be56-fb460ca2b1eb" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-02a11d57-f965-4eef-aca2-6a672d249997">
+<span class="eqno">(7)<a class="headerlink" href="#equation-02a11d57-f965-4eef-aca2-6a672d249997" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \text{Softmax}
 \begin{bmatrix}
   7       &amp; -8   &amp; 6  \\
@@ -1498,17 +1498,17 @@ <h3><span style="color:LightGreen">Computing the Reweighted Causal Attention Mas
 \text{Softmax}(x_2) = [\frac{e^{-3}}{e^{-3}+e^{2}+0}, \frac{e^{2}}{e^{-3}+e^{2}+0}, \frac{0}{e^{-3}+e^{2}+0}] = [\frac{e^{-3}}{e^{-3}+e^{2}+0}, \frac{e^{2}}{e^{-3}+e^{2}+0}, \frac{0}{e^{-3}+e^{2}+0}] = [0.0067, 0.9933, 0.0000]
 \]</div>
 <p>So we have exactly what we want! The attention weight of the last value is set to 0, so when we are on the second vector <span class="math notranslate nohighlight">\(x_2\)</span>, we cannot look forward to the future value vectors <span class="math notranslate nohighlight">\(v_3\)</span>, and the remaining parts add up to 1 so its still a probability vector! To do this correctly for the entire matrix, we can just substitute in the top triangle of <span class="math notranslate nohighlight">\(QK^T\)</span> with <span class="math notranslate nohighlight">\(-\infty\)</span>. This would look like:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-5c744a47-0d2a-4619-9242-21ec929d3bb8">
-<span class="eqno">(8)<a class="headerlink" href="#equation-5c744a47-0d2a-4619-9242-21ec929d3bb8" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-2165ddf5-f6b4-48e0-b64f-1d4fa56282e1">
+<span class="eqno">(8)<a class="headerlink" href="#equation-2165ddf5-f6b4-48e0-b64f-1d4fa56282e1" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \begin{bmatrix}
   7       &amp; -\infty   &amp; -\infty  \\
   -3       &amp; 2   &amp; -\infty   \\
   1       &amp; 6  &amp; -2   \\
 \end{bmatrix}
 \end{equation}\]</div>
 <p>Taking the softmax of the rows of this matrix then gives:</p>
-<div class="amsmath math notranslate nohighlight" id="equation-9ab9add6-4fdb-430f-8eab-90022d1c4d48">
-<span class="eqno">(9)<a class="headerlink" href="#equation-9ab9add6-4fdb-430f-8eab-90022d1c4d48" title="Permalink to this equation">#</a></span>\[\begin{equation}
+<div class="amsmath math notranslate nohighlight" id="equation-6ea5d2ec-9501-4b7e-b964-36c8d47aa5f7">
+<span class="eqno">(9)<a class="headerlink" href="#equation-6ea5d2ec-9501-4b7e-b964-36c8d47aa5f7" title="Permalink to this equation">#</a></span>\[\begin{equation}
 \text{Softmax}
 \begin{bmatrix}
   7       &amp; -\infty   &amp; -\infty  \\

_sources/lectures/UnsupervisedLearningAnomalyDetection.html

@@ -664,8 +664,15 @@ <h3><span style="color:LightGreen">Brief LSTM Review</span><a class="headerlink"
 <h3><span style="color:LightGreen">Data</span><a class="headerlink" href="#span-style-color-lightgreen-data-span" title="Permalink to this heading">#</a></h3>
 <p>The <a class="reference external" href="http://timeseriesclassification.com/description.php?Dataset=ECG5000">dataset</a> contains 5,000 Time Series examples (obtained with ECG) with 140 timesteps. Each sequence corresponds to a single heartbeat from a single patient with congestive heart failure.</p>
 <blockquote>
-<div><p>An electrocardiogram (ECG or EKG) is a test that checks how your heart is functioning by measuring the electrical activity of the heart. With each heart beat, an electrical impulse (or wave) travels through your heart. This wave causes the muscle to squeeze and pump blood from the heart. <a class="reference external" href="https://www.heartandstroke.ca/heart/tests/electrocardiogram">Source</a></p>
+<div><p>An electrocardiogram (ECG or EKG) is a test that checks how your heart is functioning by measuring the electrical activity of the heart. It measures the electrical activity of your heart. It uses small electrodes attached to your skin to detect the tiny electrical signals that control your heartbeat. With each heart beat, an electrical impulse (or wave) travels through your heart. This wave causes the muscle to squeeze and pump blood from the heart. <a class="reference external" href="https://www.heartandstroke.ca/heart/tests/electrocardiogram">Source</a></p>
 </div></blockquote>
+<ul class="simple">
+<li><p><em><strong><span style="color:Violet">What it measures</span></strong></em>: An ECG records the electrical impulses that cause your heart to beat, showing how the signals travel through your heart’s chambers.</p></li>
+<li><p><em><strong><span style="color:Violet">How it’s done</span></strong></em>: During an ECG, electrodes are attached to your chest, arms, and legs using adhesive patches or small suction cups.</p></li>
+<li><p><em><strong><span style="color:Violet">What it shows</span></strong></em>: The ECG produces a graphical representation of your heart’s electrical activity, known as an ECG tracing.</p></li>
+<li><p><em><strong><span style="color:Violet">Why it’s used</span></strong></em>: ECGs are used to diagnose and monitor various heart conditions, including arrhythmias, coronary artery disease, heart attacks, and heart failure.</p></li>
+<li><p><em><strong><span style="color:Violet">What it can reveal</span></strong></em>: An ECG can help doctors detect changes in heart rate, rhythm, and electrical conduction, which may indicate a heart problem.</p></li>
+</ul>
 <p>We have 5 types of hearbeats (classes):</p>
 <ul class="simple">
 <li><p>Normal (N)</p></li>