Remove obsolete Quarto notebooks: visualise.quarto_ipynb_1, workflow-help.quarto_ipynb_1, and workflow-packages-and-environments.quarto_ipynb_1.

Author: brentomagic

visualise.quarto_ipynb_1

@@ -28,7 +28,7 @@
         "\n",
         "The first of the three kinds is *exploratory data visualisation*, and it's the kind that you do when you're looking and data and trying to understand it. Just plotting the data is a really good strategy for getting a feel for any issues there might be. This is perhaps most famously demonstrated by Anscombe's quartet: four different datasets with the same mean, standard deviation, and correlation but very different data distributions."
       ],
-      "id": "2d4a4a09"
+      "id": "f3331573"
     },
     {
       "cell_type": "code",
@@ -96,7 +96,7 @@
         "plt.suptitle(\"Anscombe's Quartet\")\n",
         "plt.show()"
       ],
-      "id": "c38c38fa",
+      "id": "64a0e7f6",
       "execution_count": null,
       "outputs": []
     },
@@ -120,7 +120,7 @@
         "\n",
         "You can find more information on the topic of communicating via data visualisations in the @sec-communicate-plots chapter."
       ],
-      "id": "88d12184"
+      "id": "30b9ff30"
     }
   ],
   "metadata": {

webscraping-and-apis.ipynb

@@ -267,15 +267,19 @@
    "id": "b7bf16d7",
    "metadata": {},
    "source": [
-    "### The OECD API\n",
+    "### The Eurostat SDMX API\n",
     "\n",
-    "Sometimes it's convenient to use APIs directly, and, as an example, the OECD API comes with a LOT of complexity that direct access can take advantage of. The OECD API makes data available in both JSON and XML formats, and we'll use [**pandasdmx**](https://pandasdmx.readthedocs.io/) (aka the Statistical Data and Metadata eXchange (SDMX) package for the Python data ecosystem) to pull down the XML format data and turn it into a regular **pandas** data frame.\n",
+    "Sometimes it’s convenient to use APIs directly. The Eurostat API provides access to a massive repository of European statistical data using the SDMX (Statistical Data and Metadata eXchange) standard. While Eurostat offers multiple formats, using the SDMX-ML (XML) format via the sdmx1 library allows us to pull structured data into the Python ecosystem with high precision.\n",
     "\n",
-    "Now, key to using the OECD API is knowledge of its many codes: for countries, times, resources, and series. You can find some broad guidance on what codes the API uses [here](https://data.oecd.org/api/sdmx-ml-documentation/) but to find exactly what you need can be a bit tricky. Two tips are:\n",
-    "1. If you know what you're looking for is in a particular named dataset, eg \"QNA\" (Quarterly National Accounts), put `https://stats.oecd.org/restsdmx/sdmx.ashx/GetDataStructure/QNA/all?format=SDMX-ML` into your browser and look through the XML file; you can pick out the sub-codes and the countries that are available.\n",
-    "2. Browse around on https://stats.oecd.org/ and use Customise then check all the \"Use Codes\" boxes to see whatever your browsing's code names.\n",
+    "Key to using the Eurostat API is understanding the Data Structure Definition (DSD). Every dataset is essentially a multidimensional \"cube\" where each dimension (like Geography, Unit, or Frequency) has specific codes.\n",
     "\n",
-    "Let's see an example of this in action. We'd like to see the productivity (GDP per hour) data for a range of countries since 2010. We are going to be in the productivity resource (code \"PDB_LV\") and we want the USD current prices (code \"CPC\") measure of GDP per employed worker (code \"T_GDPEMP) from 2010 onwards (code \"startTime=2010\"). We'll grab this for some developed countries where productivity measurements might be slightly more comparable. The comments below explain what's happening in each step."
+    "To find the exact codes you need:\n",
+    "\n",
+    "The Data Browser: Browse the Eurostat Data Navigation Tree. Once you find a table (e.g., \"HICP - monthly data\"), the \"Dataset Code\" (like prc_hicp_manr) is shown in brackets.\n",
+    "\n",
+    "Positional Keys: Eurostat's REST API expects a \"key string\" where codes are placed in a specific order separated by dots (e.g., Freq.Unit.Item.Geo). If you know the order, you can \"slice\" the data cube directly.\n",
+    "\n",
+    "Let’s see an example of this in action. We want to see the Harmonised Index of Consumer Prices (HICP)—specifically the annual rate of change for all items—for Germany and France. We will use the resource prc_hicp_manr, requesting Monthly frequency (M), the Annual Rate of Change unit (RCH_A), and the \"All-items\" classification (CP00)."
    ]
   },
   {
@@ -285,17 +289,32 @@
    "metadata": {},
    "source": [
     "```python\n",
-    "import pandasdmx as pdmx\n",
-    "# Tell pdmx we want OECD data\n",
-    "oecd = pdmx.Request(\"OECD\")\n",
-    "# Set out everything about the request in the format specified by the OECD API\n",
-    "data = oecd.data(\n",
-    "    resource_id=\"PDB_LV\",\n",
-    "    key=\"GBR+FRA+CAN+ITA+DEU+JPN+USA.T_GDPEMP.CPC/all?startTime=2010\",\n",
-    ").to_pandas()\n",
-    "\n",
-    "df = pd.DataFrame(data).reset_index()\n",
-    "df.head()\n",
+    "import sdmx\n",
+    "import polars as pl\n",
+    "\n",
+    "# Tell sdmx we want ESTAT data\n",
+    "client = sdmx.Client('ESTAT')\n",
+    "\n",
+    "# 2. Build the URL-style positional key\n",
+    "# Format: [Freq].[Unit].[Coicop].[Geo]\n",
+    "# We use '+' to join multiple countries (DE and FR)\n",
+    "resource_id = 'prc_hicp_manr'\n",
+    "key_string = 'M.RCH_A.CP00.DE+FR'\n",
+    "\n",
+    "# 3. Fetch the data directly\n",
+    "# 'startPeriod' limits the timeline to recent data\n",
+    "response = client.data(\n",
+    "    resource_id=resource_id,\n",
+    "    key=key_string,\n",
+    "    params={'startPeriod': '2024-01'}\n",
+    ")\n",
+    "\n",
+    "# 4. Convert the SDMX-ML response to a Polars DataFrame\n",
+    "# We bridge through Pandas as sdmx1 is optimized for it\n",
+    "df_pd = sdmx.to_pandas(response).to_frame(name='value').reset_index()\n",
+    "df = pl.from_pandas(df_pd)\n",
+    "\n",
+    "print(df.head())\n",
     "```"
    ]
   },
@@ -305,13 +324,13 @@
    "id": "e5cac233",
    "metadata": {},
    "source": [
-    "|   | LOCATION |  SUBJECT | MEASURE | TIME_PERIOD |        value |\n",
-    "|--:|---------:|---------:|--------:|------------:|-------------:|\n",
-    "| 0 |      CAN | T_GDPEMP |     CPC |        2010 | 78848.604088 |\n",
-    "| 1 |      CAN | T_GDPEMP |     CPC |        2011 | 81422.364748 |\n",
-    "| 2 |      CAN | T_GDPEMP |     CPC |        2012 | 82663.028058 |\n",
-    "| 3 |      CAN | T_GDPEMP |     CPC |        2013 | 86368.582158 |\n",
-    "| 4 |      CAN | T_GDPEMP |     CPC |        2014 | 89617.632446 |"
+    "|   | TIME_PERIOD | geo | unit  | freq | coicop | value |\n",
+    "|--:|------------:|:----|:------|:-----|:-------|------:|\n",
+    "| 0 |     2024-01 | DE  | RCH_A | M    | CP00   |   3.1 |\n",
+    "| 1 |     2024-02 | DE  | RCH_A | M    | CP00   |   2.7 |\n",
+    "| 2 |     2024-03 | DE  | RCH_A | M    | CP00   |   2.3 |\n",
+    "| 3 |     2024-04 | DE  | RCH_A | M    | CP00   |   2.4 |\n",
+    "| 4 |     2024-05 | DE  | RCH_A | M    | CP00   |   2.8 |"
    ]
   },
   {
@@ -489,11 +508,11 @@
    "source": [
     "### Webscraping Tables\n",
     "\n",
-    "Often there are times when you don't actually want to scrape an entire webpage and all you want is the data from a *table* within the page. Fortunately, there is an easy way to scrape individual tables using the **pandas** package.\n",
+    "There are times when you don't need to scrape an entire webpage; you simply want the structured data from a specific table. While Polars is a high-performance data engine, it focuses on strict data formats (like Parquet or CSV) and does not natively include an HTML parser. However, we can easily bridge this gap by using Pandas to fetch the table and then converting it into a Polars DataFrame.\n",
     "\n",
-    "We will read data from a table on 'https://webscraper.io/test-sites/tables' using **pandas**. The function we'll use is `read_html()`, which returns a list of data frames of all the tables it finds when you pass it a URL. If you want to filter the list of tables, use the `match=` keyword argument with text that only appears in the table(s) you're interested in.\n",
+    "We will read data from 'https://webscraper.io/test-sites/tables' using `pd.read_html()`. This function scans the webpage and returns a list of all tables it finds as DataFrames. To target a specific table, we use the match= keyword argument with text that uniquely appears in the table we want—in this case, \"First Name\".\n",
     "\n",
-    "The example below shows how this works; looking at the website, we can see that the table we're interested in, has a 'First Name' column. Therefore we run:"
+    "Once captured, we convert the result to Polars using pl.from_pandas() to take advantage of Polars' superior query performance and expression API."
    ]
   },
   {
@@ -503,10 +522,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "df_list = pd.read_html(\"https://webscraper.io/test-sites/tables\", match=\"First Name\")\n",
+    "import polars as pl\n",
+    "import pandas as pd\n",
+    "\n",
+    "\n",
+    "pd_list = pd.read_html(\"https://webscraper.io/test-sites/tables\", match=\"First Name\")\n",
     "# Retrieve first entry from list of data frames\n",
-    "df = df_list[0]\n",
-    "df.head()"
+    "df = pl.from_pandas(pd_list[0])\n",
+    "\n",
+    "print(df.head())"
    ]
   },
   {
@@ -515,7 +539,7 @@
    "id": "31e49317",
    "metadata": {},
    "source": [
-    "This gives us the table neatly loaded into a **pandas** data frame ready for further use.\n",
+    "This gives us the table neatly loaded into a **polars** data frame ready for further use.\n",
     "\n",
     "If you get a '403' error, it means that the website has blocked **pandas** because it can see that you are engaged in web scraping. This is because some people web scrape irresponsibly, or because websites have provided other, preferred ways for you to obtain the data, eg via a download of the whole thing (think Wikipedia) or through an API. (If you really need to, [you can often get around the 403 error](https://stackoverflow.com/questions/43590153/http-error-403-forbidden-when-reading-html) though.)"
    ]

workflow-help.quarto_ipynb_1

@@ -35,7 +35,7 @@
         "\n",
         "- First, you need to make your code reproducible. This means that you need to capture everything, i.e., include any packages you used and create all necessary objects. The easiest way to make sure you've done this is to use the [**watermark**](https://github.com/rasbt/watermark) package alongside whatever else you are doing:"
       ],
-      "id": "8f49fc2e"
+      "id": "22b3f9e0"
     },
     {
       "cell_type": "code",
@@ -48,7 +48,7 @@
         "print(watermark())\n",
         "print(watermark(iversions=True, globals_=globals()))"
       ],
-      "id": "eca8f7b4",
+      "id": "a119501b",
       "execution_count": null,
       "outputs": []
     },
@@ -72,7 +72,7 @@
         "\n",
         "- **Data**: as others won't be able to easily download the data you're working with, it's often best to create a small amount of data from code that still have the same problem as you're finding with your actual data. Between **numpy** and **pandas**, it's quite easy to generate data from code; here's an example:"
       ],
-      "id": "6ffdbbbe"
+      "id": "c4ac60b4"
     },
     {
       "cell_type": "code",
@@ -87,7 +87,7 @@
         "df[\"random_normal\"] = np.random.normal(size=6)\n",
         "df"
       ],
-      "id": "6b14eb61",
+      "id": "d1e4562c",
       "execution_count": null,
       "outputs": []
     },
@@ -99,7 +99,7 @@
         "\n",
         "Finish by checking that you have actually made a reproducible example by starting a fresh Python session and copying and pasting your reprex in."
       ],
-      "id": "154142ef"
+      "id": "4b75e409"
     }
   ],
   "metadata": {

workflow-packages-and-environments.quarto_ipynb_1

@@ -104,7 +104,7 @@
         "\n",
         "You may have noticed that a file called `pyproject.toml` has been created."
       ],
-      "id": "6b7f7364"
+      "id": "8b889898"
     },
     {
       "cell_type": "code",
@@ -115,7 +115,7 @@
         "\n",
         "print_json(data=toml.load(\"pyproject.toml\"))"
       ],
-      "id": "acc9e141",
+      "id": "688f09f1",
       "execution_count": null,
       "outputs": []
     },
@@ -133,7 +133,7 @@
         "\n",
         "In the screenshot above, you can see the project-environment in two places: on the blue bar at the bottom of the screen, and (in 5), at the top right hand side of the interactive window. A similar top right indicator is present when you have a Jupyter Notebook open too."
       ],
-      "id": "6ac222b7"
+      "id": "148595b3"
     }
   ],
   "metadata": {