Merge pull request #169 from LaunchCodeEducation/databases-python

audit for pandas | python databases

Author: jwoolbright23

content/python-pandas-databases/_index.md

@@ -4,7 +4,7 @@ title = "Databases with Python and pandas"
 date = 2024-04-17T10:00:24-05:00
 draft = false
 weight = 23
-hidden = false
+hidden = true
 +++
 
 ## Learning Objectives

content/python-pandas-databases/reading/pandas-databases/_index.md

@@ -14,7 +14,7 @@ This lesson will also utilize `sqlite3` as the database used to demonstrate how
     - Adding the DataFrame data into the new table
 
 {{% notice red Warning "rocket" %}}
-We have created a new repo for Class 19 and 20 exercises and studios. 
+We have created a new repo for Class 19 and 20 exercises and studios (Class numbers may differ per cohort). 
 Please fork this repo to your Github account, and then clone it to your local device
 
 [Class 19 and 20 Exercise Studio Repo](https://github.com/LaunchCodeEducation/data-analysis-projects-class-19-and-20)
@@ -56,26 +56,22 @@ new_movie = pd.DataFrame([{'title':'Dune', 'genre':'Science Fiction', 'release':
 df = pd.concat([df, new_movie], ignore_index=True)
 ```
 
-It isn't necessary for us to update our DataFrame to add a new table to the database. 
+It isn't necessary for us to update our DataFrame to add a new table to the database.
 But, it will help visually when reading data to show that it was populated into a new table correctly.
 
-```python {linenos=table}
+```python
 # Inject dataframe into database as new table, if the table exists - replace it
-df.to_sql('df', movies_db, if_exists="replace")
-# Execute command to create a new table called new_movie_table with the new_movie dataframe data
-movies_db.execute(
-    """
-    create table new_movie_table as
-    select * from new_movie
-    """
-)
+# Use index=False to avoid storing the DataFrame index as a column
+df.to_sql('updated_movies', movies_db, if_exists="replace", index=False)
+# Commit the changes to the database
+movies_db.commit()
 ```
 
 The pandas `DataFrame.to_sql` function documentation in the above code block can be found here: [pandas.DataFrame.to_sql](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.to_sql.html)
 
 ```python
 # Read data from newly created table, passing in existing movies_db connection as parameter
-new_movies_df = pd.read_sql_query('Select * from new_movie_table;', movies_db)
+new_movies_df = pd.read_sql_query('SELECT * FROM updated_movies;', movies_db)
 # Read first 6 rows
 new_movies_df.head(6)
 ```

content/python-pandas-databases/reading/python-databases/_index.md

@@ -12,6 +12,26 @@ Working within a database command line interface can oftentimes make it cumberso
 
 While you can accomplish more than just the above using python and pandas, like performing joins, it is not always best practice. As it relates to joins, database engines are built and optimized to perform joins extremely well. It is always important to know what you will be doing with your data before acting.
 
+## When to Use sqlite3 vs pandas
+
+Understanding when to use each tool will help you write more efficient code:
+
+**Use sqlite3 directly when:**
+- Performing complex SQL operations (joins, aggregations, subqueries)
+- Working with large datasets that don't fit in memory
+- You need precise control over transactions and commits
+- Creating or modifying database schema (tables, indexes, constraints)
+- The data will remain in the database without further analysis
+
+**Use pandas when:**
+- You need to perform complex data analysis or visualization
+- Applying statistical operations or transformations
+- Cleaning and reshaping data
+- Integrating database data with other data sources
+- The dataset is small to medium-sized and fits in memory
+
+**Best Practice:** Use sqlite3 to filter and join data in the database, then load the results into pandas for analysis. This leverages the strengths of both tools!
+
 {{% notice red Warning "rocket" %}}
 We have created a new repo for Class 19 and 20 exercises and studios. 
 Please fork this repo to your Github account, and then clone it to your local device
@@ -63,6 +83,26 @@ The basic syntax for executing a command with the cursor object is as follows:
 cur.execute("SQL statement")
 ```
 
+## Using Context Managers
+
+A better practice for managing database connections is to use Python's context manager pattern with the `with` statement. This ensures that connections are properly closed even if an error occurs:
+
+{{% notice blue Example "rocket" %}}
+```python
+import sqlite3
+
+# Using a context manager - connection closes automatically
+with sqlite3.connect('Movies.db') as movies_db:
+    cur = movies_db.cursor()
+    cur.execute("SELECT * FROM movies")
+    results = cur.fetchall()
+    print(results)
+# Connection is automatically closed here
+```
+
+This pattern is especially useful when performing multiple operations, as it guarantees proper cleanup.
+{{% /notice %}}
+
 ## Creating a table
 
 ```python
@@ -76,7 +116,9 @@ You can find a list of SQLite data types here: [Data Types in SQLite](https://sq
 ### Insert Table Values
 
 ```python
-cur.execute("INSERT INTO table_name ('value-one', 'value-two', etc..)")
+cur.execute("INSERT INTO table_name (column1, column2) VALUES ('value-one', 'value-two')")
+# Commit the changes to save them to the database
+movies_db.commit()
 ```
 
 ### Reading Data
@@ -87,7 +129,7 @@ There are a couple strategies that you can use to read data from your database.
 
 ```python
 # For loop to iterate over cursor object
-for row in cur.execute("SELECT column FROM table_name")
+for row in cur.execute("SELECT column FROM table_name"):
     print(row)
 ```
 
@@ -113,6 +155,8 @@ update_release_year = 1997 # Value that needs to be updated
 movie_to_update = 'Good Will Hunting'
 # Execute an UPDATE statement using the ? placeholder, passing in the update variables as a list literal
 cur.execute("UPDATE movies SET release = ? WHERE title = ?", [update_release_year, movie_to_update])
+# Commit the changes to save them to the database
+movies_db.commit()
 ```
 {{% /notice %}}
 
@@ -126,9 +170,44 @@ Using parameterized queries to update and delete data is a best practice!
 movie_to_delete = 'Inception' # Too many sci fi movies!
 # Execute a DELETE statement using the ? placeholder, passing in the variable as a list literal
 cur.execute("DELETE FROM movies WHERE title = ?", [movie_to_delete])
+# Commit the changes to save them to the database
+movies_db.commit()
 ```
 {{% /notice %}}
 
+## Error Handling
+
+When working with databases, it's important to handle potential errors gracefully. Common errors include trying to access tables that don't exist, constraint violations, or connection issues.
+
+{{% notice blue Example "rocket" %}}
+```python
+import sqlite3
+
+try:
+    with sqlite3.connect('Movies.db') as movies_db:
+        cur = movies_db.cursor()
+
+        # Attempt to insert a duplicate record or violate a constraint
+        cur.execute("INSERT INTO movies (title, genre, release) VALUES (?, ?, ?)",
+                   ['Inception', 'Sci-Fi', 2010])
+        movies_db.commit()
+
+except sqlite3.IntegrityError as e:
+    print(f"Database integrity error: {e}")
+
+except sqlite3.OperationalError as e:
+    print(f"Database operational error (table may not exist): {e}")
+
+except sqlite3.Error as e:
+    print(f"Database error: {e}")
+
+except Exception as e:
+    print(f"Unexpected error: {e}")
+```
+
+Using try-except blocks helps your program handle errors without crashing and provides useful feedback.
+{{% /notice %}}
+
 ## Check Your Understanding
 
 {{% notice green Question "rocket" %}}