session-08-part-01.md

Session 8 | Part 1

In Part 1, you will run your first PySpark tutorial in Google Colab. The goal is to learn the Spark workflow before worrying about local installation.

1. Goal

You will:

• explain what Apache Spark and PySpark are used for
• open a Google Colab notebook
• install PySpark in Colab
• start a local Spark session inside Colab
• create a Spark DataFrame from Python data
• inspect rows, columns, schema, and counts
• use simple DataFrame transformations
• stop Spark when you are done

2. Prerequisites

Before starting:

1. Open Google Colab and log in using your personal Gmail account.
2. Create a new notebook.
3. Rename it:

session-08-part-01-pyspark-intro

4. Add a text cell at the top with:

# Session 8 Part 1: PySpark in Google Colab

5. Run each code cell in order.

Note

Colab sessions reset. If you reconnect later, run the install and Spark setup cells again.

3. Basics you should know

• Apache Spark: a distributed data processing engine.
• PySpark: the Python API for Apache Spark.
• SparkSession: the main entry point for working with Spark from PySpark.
• DataFrame: a distributed table with rows and columns.
• transformation: a lazy operation that describes work, such as filter.
• action: an operation that triggers work, such as show or count.
• schema: the column names and data types in a DataFrame.
• local mode: Spark running on one machine, useful for learning and small practice jobs.

Checkpoint question:

What is the difference between a DataFrame and a regular Python list?

Show answer

A Python list lives in normal Python memory and is usually processed directly by Python. A Spark DataFrame represents table-like data that Spark can process using its execution engine, including on larger datasets and clusters.

4. Install PySpark in Colab

Some Google Colab runtimes may already have PySpark available. Run this first to test it:

from pyspark.sql import SparkSession

If the import works, continue to the next section.

If PySpark is not installed, run this in a Colab code cell:

!pip install pyspark==4.1.2

After the install finishes, restart the runtime only if Colab asks you to.

Checkpoint question:

Why do we test the import before installing PySpark?

Show answer

Because the Colab environment can change. Testing first avoids installing a package that is already available, but the install command is still there as a fallback.

5. Start Spark

Create a Spark session:

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder
    .appName("Session08Part01")
    .master("local[*]")
    .getOrCreate()
)

spark

What this means:

1. SparkSession.builder starts the setup.
2. appName(...) gives the Spark job a readable name.
3. master("local[*]") tells Spark to use local mode with available CPU cores.
4. getOrCreate() starts Spark or reuses an existing session.

Checkpoint question:

What does local[*] mean in this tutorial?

Show answer

It means Spark runs locally inside the Colab runtime and can use the available local CPU cores. This is enough for learning PySpark without connecting to a real Spark cluster.

Checkpoint question:

Why do we use getOrCreate() instead of always creating a brand-new Spark session?

Show answer

In a notebook, cells can be run more than once. getOrCreate() reuses an existing Spark session if one already exists, which is friendlier for interactive notebook work.

6. Create your first Spark DataFrame

Run:

students = [
    ("Ava", "Data Analytics", 92),
    ("Nikos", "Data Engineering", 85),
    ("Maya", "Data Analytics", 78),
    ("Leo", "Business", 88),
    ("Iris", "Data Engineering", 95),
]

columns = ["name", "track", "score"]

df = spark.createDataFrame(students, columns)

df.show()

Expected idea:

Spark prints a table with name, track, and score.

Checkpoint question:

Why do we pass columns separately instead of leaving Spark to guess every column name?

Show answer

The rows are tuples, so they do not contain column names. Passing columns gives the DataFrame readable names such as name, track, and score.

Task: Add two more students to the students list and run the cell again.

Show solution idea

students = [
    ("Ava", "Data Analytics", 92),
    ("Nikos", "Data Engineering", 85),
    ("Maya", "Data Analytics", 78),
    ("Leo", "Business", 88),
    ("Iris", "Data Engineering", 95),
    ("Elena", "Business", 81),
    ("Omar", "Data Analytics", 90),
]

7. Inspect the DataFrame

Run:

df.printSchema()
print("Rows:", df.count())
print("Columns:", df.columns)

What this shows:

1. printSchema() shows column names and inferred data types.
2. count() counts rows.
3. columns lists the column names.

Checkpoint question:

Which command is an action: printSchema, count, or columns?

Show answer

count() is the clearest action here because it asks Spark to compute a result from the data.

Checkpoint question:

Why is checking the schema useful before writing filters or calculations?

Show answer

The schema tells you which columns exist and what data type Spark thinks each column contains. This helps you avoid mistakes such as doing numeric comparisons on text columns.

8. Select columns

Use select to choose columns:

df.select("name", "score").show()

Checkpoint question:

Does select("name", "score") change the original df?

Show answer

No. Spark DataFrame operations return a new DataFrame. The original df still has all of its columns unless you assign the selected result to a variable.

Task: Show only the name and track columns.

Show solution

df.select("name", "track").show()

9. Select with expressions and conditions

select can do more than choose existing columns. It can also show calculated expressions.

Import col and when:

from pyspark.sql.functions import col, when

Now select the name, the score, and a calculated score:

df.select(
    "name",
    "score",
    (col("score") + 5).alias("score_plus_5")
).show()

Checkpoint question:

Why do we use alias("score_plus_5") here?

Show answer

The expression col("score") + 5 would otherwise have a long generated name. alias gives the calculated column a clear name in the output.

You can also use a condition inside select:

df.select(
    "name",
    "track",
    "score",
    when(col("score") >= 85, "pass").otherwise("review").alias("result")
).show()

What this shows:

1. when(...) checks a condition.
2. otherwise(...) gives the value when the condition is false.
3. alias("result") names the new output column.

Checkpoint question:

Does this conditional select permanently add the result column to df?

Show answer

No. It only shows a selected output with a calculated column. If you want to keep the new column, use withColumn and assign the result to a variable.

Task: Select name, score, and a new column named score_level.

Use:

• "excellent" when the score is at least 90
• "good" when the score is at least 80
• "needs review" for all other scores

Show solution

df.select(
    "name",
    "score",
    when(col("score") >= 90, "excellent")
    .when(col("score") >= 80, "good")
    .otherwise("needs review")
    .alias("score_level")
).show()

10. Filter rows

Use filter to keep rows that match a condition:

df.filter(df["score"] >= 90).show()

Checkpoint question:

What rows should this filter keep?

Show answer

It should keep only students whose score is greater than or equal to 90.

Task: Show students whose score is less than 85.

Show solution

df.filter(df["score"] < 85).show()

You can also filter first and then select only the columns you want to show:

df.filter(col("score") >= 85).select("name", "track", "score").show()

Checkpoint question:

What is the difference between this code and df.select("name", "track", "score")?

Show answer

df.select("name", "track", "score") shows those columns for every row. df.filter(col("score") >= 85).select(...) first keeps only high-score rows, then shows the selected columns.

Task: Show only the name and score columns for students in the Data Analytics track.

Show solution

df.filter(col("track") == "Data Analytics").select("name", "score").show()

11. Create a new column

Import col and when:

from pyspark.sql.functions import col, when

graded_df = df.withColumn(
    "result",
    when(col("score") >= 85, "pass").otherwise("review")
)

graded_df.show()

Checkpoint question:

Why do we save the result as graded_df instead of expecting df to change automatically?

Show answer

Spark DataFrames are immutable. withColumn returns a new DataFrame with the extra column, so assigning it to graded_df keeps the new version.

Task: Add a new column named score_plus_5 that adds 5 points to each score.

Show solution

df.withColumn("score_plus_5", col("score") + 5).show()

12. Group and summarize

Use groupBy to summarize by category:

df.groupBy("track").count().show()

Checkpoint question:

What does each row in this grouped result represent?

Show answer

Each row represents one track, plus the number of students in that track.

Now calculate average score by track:

from pyspark.sql.functions import avg

df.groupBy("track").agg(avg("score").alias("average_score")).show()

Checkpoint question:

Why do we use alias("average_score")?

Show answer

Without an alias, Spark may show a generated column name such as avg(score). The alias gives the summary column a clean readable name.

Task: Find the maximum score in each track.

Show solution

from pyspark.sql.functions import max

df.groupBy("track").agg(max("score").alias("max_score")).show()

13. Exercise 1: PySpark student summary

Create a final Colab section called:

## Exercise 1

Your notebook should:

1. Install PySpark.
2. Start a Spark session.
3. Create a DataFrame with at least 8 students.
4. Include these columns:
   • name
   • track
   • score
   • hours_studied
5. Print the schema.
6. Show the first 5 rows.
7. Filter students with score >= 85.
8. Add a result column with pass or review.
9. Group by track and show:
   • number of students
   • average score
   • average hours studied
10. Stop Spark at the end.

Suggested skeleton:

from pyspark.sql import SparkSession
from pyspark.sql.functions import avg, col, count, when


spark = (
    SparkSession.builder
    .appName("Session08Exercise01")
    .master("local[*]")
    .getOrCreate()
)

students = [
    # TODO: add at least 8 rows
]

columns = ["name", "track", "score", "hours_studied"]

df = spark.createDataFrame(students, columns)

# TODO: print schema
# TODO: show first 5 rows
# TODO: filter high scores
# TODO: add result column
# TODO: group by track

spark.stop()

Minimum completion checklist:

1. The notebook runs from top to bottom in Colab.
2. A Spark session starts successfully.
3. The DataFrame has at least 8 rows.
4. The summary groups by track.
5. Spark is stopped at the end.

Show hint

Use df.show(5) for the first 5 rows. Use groupBy("track").agg(...) for multiple summary calculations.

14. Quiz

quizmd quizzes/python-session-08-part-01-quiz.md