Update

Author: edoardob90

.devcontainer/devcontainer.json

@@ -3,13 +3,13 @@
 {
 	"name": "Python Tutorial",
 	// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
-	"image": "ghcr.io/empa-scientific-it/python-tutorial:latest",
+	"image": "ghcr.io/empa-scientific-it/python-tutorial:cpu-arm64",
 
 	// Features to add to the dev container. More info: https://containers.dev/features.
 	// "features": {},
 
 	// Use 'forwardPorts' to make a list of ports inside the container available locally.
-	// "forwardPorts": [],
+	"forwardPorts": [8888],
 
 	// Use 'postCreateCommand' to run commands after the container is created.
 	// "postCreateCommand": "pip3 install --user -r requirements.txt",
@@ -25,5 +25,12 @@
 				"ms-vsliveshare.vsliveshare"
 			]
 		}
+	},
+
+	"remoteUser": "jovyan",
+	"containerUser": "jovyan",
+	"containerEnv": {
+		"UID": "1000",
+		"GID": "100"
 	}
 }

25_library_pytorch_language_modeling.ipynb

@@ -557,7 +557,9 @@
     "        n += 1\n",
     "\n",
     "print(f'{log_likelihood=}')\n",
+    "\n",
     "nll = -log_likelihood\n",
+    "\n",
     "print(f'{nll=}')  # Negative log likelihood\n",
     "print(f'Average NLL: {nll/n:.4f}')  # More descriptive output"
    ]
@@ -694,6 +696,16 @@
    "id": "47",
    "metadata": {},
    "outputs": [],
+   "source": [
+    "import torch"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "48",
+   "metadata": {},
+   "outputs": [],
    "source": [
     "#Create training set of all bigrams\n",
     "xs, ys = [], [] # Input and output character indices\n",
@@ -711,23 +723,188 @@
     "ys = torch.tensor(ys)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "49",
+   "metadata": {},
+   "source": [
+    "We can inspect the first few tokens from the two tensors:\n",
+    "- `xs` will be the input tokens (what the model will see)\n",
+    "- `ys` will be the target tokens (what we want the model to predict)"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "48",
+   "id": "50",
    "metadata": {},
    "outputs": [],
    "source": [
     "for i in range(5):\n",
     "    print(f'For character #{i} \"{itos[xs[i].item()]}\" in xs, we expect the model to predict \"{itos[ys[i].item()]}\"')"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "51",
+   "metadata": {},
+   "source": [
+    "One important detail about PyTorch tensors: there exists `torch.Tensor` (a class) and `torch.tensor()` (a method).\n",
+    "They are related, but different:\n",
+    "- `torch.Tensor` is a class, and every PyTorch tensor is an instance of this class.\n",
+    "- `torch.tensor()` is a method to create a tensor, with `dtype` automatically inferred from the input data.\n",
+    "\n",
+    "Except for when initializing a completely empty tensor, in general there is no reason to choose `torch.Tensor` over `torch.tensor`. \n",
+    "Note that `torch.Tensor` is an alias for `torch.FloatTensor`, with a default `dtype` of `torch.float32`.\n",
+    "\n",
+    "In general, you should use `torch.tensor()` almost always, unless you have a specific reason to use `torch.Tensor`.\n",
+    "\n",
+    "Here you can find more details from the official docs:\n",
+    "- [`torch.tensor()`](https://docs.pytorch.org/docs/stable/generated/torch.tensor.html#torch-tensor)\n",
+    "- [`torch.Tensor`](https://docs.pytorch.org/docs/stable/tensors.html)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "52",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "xs.dtype, xs.type()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "53",
+   "metadata": {},
+   "source": [
+    "`torch.LongTensor` here means that the tensor contains 64-bit (8-byte) integers."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "54",
+   "metadata": {},
+   "source": [
+    "### Feeding the network"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "55",
+   "metadata": {},
+   "source": [
+    "A neural network essentially is made up of **layers**.\n",
+    "At high-level, each layer is typically a linear transformation followed by a non-linear activation function.\n",
+    "Something like: $\\text{output} = \\text{activation}(\\text{weights} \\cdot \\text{input} + \\text{bias})$\n",
+    "\n",
+    "If we were to feed our characters as integer indexes, we would have a sequence of integer indexes as input.\n",
+    "If `a` is 1 and `z` is 25, the weight applied to `z` will have 25 times more impact on the output than `a`. This creates an arbitrary and misleading mathematical relationship.\n",
+    "\n",
+    "Moreover, during the training (optimization) phase, the updates to the weights will be proportional to their input values.\n",
+    "Larger input values will cause larger updates to the weights, which can lead to unstable training.\n",
+    "\n",
+    "And lastly, the network has no reference to the potential value range. It doesn't know that the values are constrained to a specific set (like 0-25 for letters).\n",
+    "\n",
+    "To address all these issues, we can use **one-hot encoding**.\n",
+    "One-hot encoding each letter means creating a vector where only one position has a value of 1 (corresponding to that letter's position in the alphabet) and all other positions are 0.\n",
+    "This gives the neural network a much clearer signal about which letter is present without introducing misleading numerical relationships.\n",
+    "This approach is particularly important in language models where the relationships between symbols (letters, words) are learned from their contexts and co-occurrences, not from arbitrary numeric values assigned to them."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "56",
+   "metadata": {},
+   "source": [
+    "Luckily, PyTorch provides a convenient way to perform one-hot encoding using the `torch.nn.functional.one_hot` function."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "57",
+   "metadata": {},
+   "source": [
+    "Let's encode the first 5 tokens, print and visualize the result:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "58",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import torch.nn.functional as F\n",
+    "\n",
+    "F.one_hot(xs[:5], num_classes=27)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "59",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "xenc = F.one_hot(xs[:5], num_classes=27)\n",
+    "xenc.shape"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "60",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%matplotlib inline\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "plt.imshow(xenc)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "61",
+   "metadata": {},
+   "source": [
+    "One problem is the `dtype` of the one-hot encoded tensor.\n",
+    "It's `torch.int64` by default (inferred from our data), but we need `torch.float32` to have a the input suitable for the mathematical operations the network will perform.\n",
+    "\n",
+    "We can convert it using `.float()`:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "62",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "xenc = F.one_hot(xs, num_classes=27).float()\n",
+    "xenc.dtype"
+   ]
   }
  ],
  "metadata": {
   "kernelspec": {
    "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.10"
   }
  },
  "nbformat": 4,