Skip to content

Commit c2e6818

Browse files
mmckyclaude
andcommitted
[polars] Sort joined frame by Date so yearly returns are correct
`read_data_polars` builds the combined price table with a chain of `join(..., how='full', coalesce=True)` calls. Polars does not guarantee join output row order: keys present on only one side are appended to the end of the frame rather than slotted into place. The Exercise 2 solution then takes `drop_nulls().first()/.last()` within `group_by('year')`, which assumes rows are date-ordered. With the index data that assumption breaks, because the Nikkei trades a different holiday calendar from the US indices, so Nikkei-only dates land at the end of the frame. The US columns are null on those appended rows and so `drop_nulls()` skips them, which confines the damage to the Nikkei column and let it pass review unnoticed. Verified against Yahoo Finance data: the fix changes 51 of 204 yearly return cells, all in the Nikkei column. Several were badly wrong and some had the wrong sign (2017 read -6.42% against an actual +16.18%, 2020 read +3.79% against +18.27%). The code ran green throughout and printed plausible numbers, which is the worst failure mode for a published lecture. Also in this commit: - Add a note on join ordering, which reinforces the lecture's own "no index, no automatic alignment" theme and is a genuine gotcha when migrating from pandas. - Drop an orphaned `[^mung]` footnote carried over from pandas.md. The word "munging" appears nowhere in this lecture, so the definition rendered as nothing at all. - Cite the official Polars migration guide for the no-index discussion instead of a personal Medium post, for authority and longevity. - Drop the unused `(pd-series)=` label from pandas.md. Nothing references it, so the PR now touches a single lecture. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 1b7c680 commit c2e6818

2 files changed

Lines changed: 11 additions & 5 deletions

File tree

lectures/pandas.md

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,6 @@ You can think of a `Series` as a "column" of data, such as a collection of obser
7878

7979
A `DataFrame` is a two-dimensional object for storing related columns of data.
8080

81-
(pd-series)=
8281
## Series
8382

8483
```{index} single: Pandas; Series

lectures/polars.md

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ s
8686
Unlike {doc}`pandas <pandas>` Series, Polars Series have no row index.
8787
Polars is column-centric --- data access is managed through column expressions
8888
and boolean masks rather than row labels.
89-
See [this blog post](https://medium.com/@luca.basanisi/understand-polars-lack-of-indexes-526ea75e413) for more detail.
89+
See the [Polars migration guide for pandas users](https://docs.pola.rs/user-guide/migration/pandas/) for more detail.
9090
```
9191

9292
Polars `Series` are built on top of [Apache Arrow](https://arrow.apache.org/) arrays and support many familiar operations
@@ -658,11 +658,20 @@ def read_data_polars(ticker_list,
658658
result = result.join(
659659
df, on='Date', how='full', coalesce=True
660660
)
661-
return result
661+
return result.sort('Date')
662662
663663
ticker = read_data_polars(ticker_list)
664664
```
665665

666+
```{note}
667+
Polars joins do not guarantee the order of the output rows --- keys that match
668+
only one side are appended rather than slotted into place.
669+
This is the same "no index, no automatic alignment" theme from above: with no
670+
row labels to align on, ordering is something we ask for explicitly.
671+
Hence the `sort('Date')` before returning, which any later
672+
`first()`/`last()` calculation relies on.
673+
```
674+
666675
Complete the program to plot the result as a bar graph.
667676

668677
```{exercise-end}
@@ -796,5 +805,3 @@ plt.show()
796805

797806
```{solution-end}
798807
```
799-
800-
[^mung]: Wikipedia defines munging as cleaning data from one raw form into a structured, purged one.

0 commit comments

Comments
 (0)