Skip to content

Docs: Update memory units and update blog with DataFrame optimization Note #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Alokxk merged 1 commit into from
Jul 29, 2025
Merged

Docs: Update memory units and update blog with DataFrame optimization Note #13

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions docs/DataFrame vs Buffer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ Here are the results from running this benchmark on a 8,000,000 row dataset (app
| Metric | DataFrame | Circular Buffer | Improvement |
|--------|-----------|-----------------|-------------|
| **Execution Time** | ~968,804 ms | ~46,258 ms | **21x faster** |
| **Memory Usage** | ~222,730 MB | ~14,148 MB | **16x less memory** |
| **Memory Usage** | ~223 GB | ~14 GB | **~16x smaller growth** |
| **GC Events** | ~14,943 | ~944 | **94% fewer** |
| **GC Time** | ~826,382 ms | ~405 ms | **2,040x less GC overhead** |
| **Results Generated** | 7,999,901 | 7,999,901 | Identical accuracy |
Expand All @@ -207,7 +207,7 @@ Here are the results from running this benchmark on a 8,000,000 row dataset (app

**Key Insights:**
- Circular buffers processed the same data **21x faster**
- Used **10.6x less memory** despite processing the same amount of data
- Used **16x less memory** despite processing the same amount of data
- Had **94% fewer garbage collection events**, leading to smoother performance
- Spent virtually no time on memory cleanup (405ms vs 826+ seconds)
- Produced identical results, proving accuracy isn't compromised
Expand Down Expand Up @@ -237,4 +237,6 @@ DataFrames are, and will remain, an essential, powerful, and correct choice for

However, when dataset is massive & we need only simple manipulations like calculating moving averages which we could do with a sliding window without loading everything into memory, a circular buffer can provide significant performance benefits. The key takeaway is to understand your data processing needs and choose the right tool for the job.

**Note:** It is also fair to mention that other DataFrame libraries vary significantly in optimization and performance. Libraries like [Pandas in Python](https://pandas.pydata.org/) are heavily optimized with specialized type systems, C extensions, and native computational libraries. Pharo's DataFrame implementation, while powerful and growing, has not yet reached that level of optimization. For this reason, if one were to repeat this benchmark with a more optimized DataFrame library, the performance difference would likely be smaller.

Want to explore more? Check out the [Containers-Buffer repository](https://github.com/pharo-containers/Containers-Buffer) to see the complete implementation and examples.
Loading