Skip to content

Commit 24038e8

Browse files
AlokxkAlokxk
authored
Merge pull request #9 from Alokzh/docs/dataframes-vs-buffers-blog-post
Add DataFrames vs Circular Buffers Performance Comparison Blog Post
2 parents 40a61f2 + de5546e commit 24038e8

1 file changed

Lines changed: 243 additions & 0 deletions

File tree

‎docs/DataFrame vs Buffer.md‎

Lines changed: 243 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,243 @@
1+
# DataFrames vs Circular Buffers: Processing Large Datasets Efficiently in Pharo
2+
3+
When processing large datasets in Pharo, developers typically reach for DataFrames and for good reason. They provide a powerful way to manipulate and analyze data, similar to how you would in Python or R. However, there's a hidden problem that many developers overlook: DataFrames can be inefficient and even detrimental to performance when dealing with large files.
4+
5+
In this article, we'll explore why DataFrames can hurt performance, how circular buffers can be a better alternative, and provide practical examples to illustrate the differences. We'll also cover how to generate test data and measure performance effectively.
6+
7+
## The Hidden Problem: Why DataFrames Can Hurt Performance
8+
Let me show you a common scenario that many developers face when working with DataFrames. Imagine you have a CSV file with stock prices, and you want to calculate the average price. Here's how you might do it using DataFrames:
9+
10+
```smalltalk
11+
stockDataFile := 'stock_data.csv'.
12+
13+
"Load entire CSV file into DataFrame"
14+
stockData := DataFrame readFromCsv: stockDataFile asFileReference.
15+
priceColumn := stockData column: 'Price'.
16+
averagePrice := priceColumn average.
17+
Transcript show: 'Total Average Price: ', averagePrice asString; cr.
18+
```
19+
20+
This works perfectly for small files. But here's what happens when your CSV file grows from 1MB to 100MB to 1GB:
21+
22+
**The Problems:**
23+
- **Memory Overhead**: DataFrames load the entire file into memory, which can be 10x larger than the file size itself. For a 1GB file, you might end up using 10GB of RAM.
24+
- **Garbage Collection**: As the DataFrame grows, garbage collection kicks in frequently, slowing down your program. This is because DataFrames create many temporary objects that need to be cleaned up.
25+
- **Performance**: Calculating the average involves iterating through potentially millions of rows, which can take a long time. The more data you have, the slower it gets.
26+
- **Crashes**: If the file is larger than your available RAM, you get "Out of Memory" errors, causing your program to crash.
27+
28+
29+
Think of it like this: you want to know the average height of people in a room, so you ask everyone to stand in line and write down everyone's details in a notebook. That's what DataFrames do - they store everything, even when you only need one number.
30+
31+
32+
## Moving Averages: A Practical Example
33+
Let's take this a step further and calculate moving averages, which is a common task in financial applications. We'll compare the DataFrame approach with the circular buffer approach.
34+
35+
### Generating Test Data
36+
Let's generate some realistic test data to see how these two approaches perform in practice. We'll create a CSV file with multiple columns of stock market data:
37+
```smalltalk
38+
| stockDataFile rowCount windowSize |
39+
40+
stockDataFile := 'moving_avg_test.csv'.
41+
rowCount := 500000.
42+
windowSize := 100.
43+
44+
"Clean up any old files"
45+
stockDataFile asFileReference exists ifTrue: [
46+
stockDataFile asFileReference delete
47+
].
48+
49+
Transcript show: 'BENCHMARK: Moving Average - DataFrame vs Streaming Buffer'; cr.
50+
51+
"Generate simple test data"
52+
Transcript show: 'Generating test data...'; cr.
53+
stockDataFile asFileReference writeStreamDo: [ :stream |
54+
| currentPrice |
55+
currentPrice := 100.0.
56+
stream nextPutAll: 'S.No.,Price,Low,High'; cr.
57+
58+
1 to: rowCount do: [ :day |
59+
| priceChange newPrice lowPrice highPrice |
60+
priceChange := (Random new next - 0.5) * 2.0. "±$1 change"
61+
newPrice := currentPrice + priceChange.
62+
63+
"Generate realistic Low and High values around the price"
64+
lowPrice := newPrice - (Random new next * 2.0). "Low is below price"
65+
highPrice := newPrice + (Random new next * 2.0). "High is above price"
66+
67+
stream nextPutAll: day asString, ',',
68+
(newPrice roundTo: 0.01) asString, ',',
69+
(lowPrice roundTo: 0.01) asString, ',',
70+
(highPrice roundTo: 0.01) asString; cr.
71+
72+
currentPrice := newPrice.
73+
].
74+
].
75+
Transcript show: 'Generated file: ', (stockDataFile asFileReference size / 1024 / 1024) rounded asString, ' MB'; cr; cr.
76+
```
77+
This code generates a CSV file with 500,000 rows of realistic stock market data, including serial numbers, prices, daily lows, and highs. Each price is a random fluctuation around the previous day's price.
78+
## Performance Testing: The Numbers Tell the Story
79+
Now let's compare the performance of the DataFrame approach with the circular buffer approach for calculating moving averages.
80+
81+
### Test Setup
82+
```smalltalk
83+
"Test DataFrame approach"
84+
Transcript show: 'Testing DataFrame approach...'; cr.
85+
3 timesRepeat: [ Smalltalk garbageCollect ].
86+
[
87+
| startTime endTime memoryBefore memoryAfter gcBefore gcAfter gcTimeBefore gcTimeAfter
88+
stockData priceColumn movingAverages |
89+
90+
memoryBefore := Smalltalk vm parameterAt: 3.
91+
gcBefore := (Smalltalk vm parameterAt: 7) + (Smalltalk vm parameterAt: 9).
92+
gcTimeBefore := (Smalltalk vm parameterAt: 8) + (Smalltalk vm parameterAt: 10).
93+
startTime := Time millisecondClockValue.
94+
95+
"Load entire dataset"
96+
stockData := DataFrame readFromCsv: stockDataFile asFileReference.
97+
priceColumn := stockData column: 'Price'.
98+
99+
movingAverages := OrderedCollection new.
100+
101+
windowSize to: priceColumn size do: [ :currentDay |
102+
| windowSum |
103+
windowSum := 0.
104+
105+
(currentDay - windowSize + 1) to: currentDay do: [ :j |
106+
windowSum := windowSum + (priceColumn at: j).
107+
].
108+
109+
movingAverages add: (windowSum / windowSize).
110+
].
111+
112+
endTime := Time millisecondClockValue.
113+
memoryAfter := Smalltalk vm parameterAt: 3.
114+
gcAfter := (Smalltalk vm parameterAt: 7) + (Smalltalk vm parameterAt: 9).
115+
gcTimeAfter := (Smalltalk vm parameterAt: 8) + (Smalltalk vm parameterAt: 10).
116+
Transcript show: 'DataFrame Test Results:'; cr.
117+
Transcript show: ' Time: ', (endTime - startTime) asString, ' ms'; cr.
118+
Transcript show: ' Memory: ', ((memoryAfter - memoryBefore) / 1024 / 1024) rounded asString, ' MB'; cr.
119+
Transcript show: ' GC Events: ', (gcAfter - gcBefore) asString; cr.
120+
Transcript show: ' GC Time: ', (gcTimeAfter - gcTimeBefore) asString, ' ms'; cr.
121+
Transcript show: ' Moving Averages: ', movingAverages size asString; cr.
122+
Transcript show: ' Final MA: $', (movingAverages last roundTo: 0.01) asString; cr; cr.
123+
124+
"Cleanup to be fair"
125+
stockData := nil.
126+
priceColumn := nil.
127+
movingAverages := nil.
128+
] value.
129+
130+
"Test Circular Buffer approach"
131+
Transcript show: 'Testing Buffer approach...'; cr.
132+
133+
3 timesRepeat: [ Smalltalk garbageCollect ].
134+
[
135+
| startTime endTime memoryBefore memoryAfter gcBefore gcAfter gcTimeBefore gcTimeAfter
136+
priceBuffer movingAverages processedCount |
137+
138+
memoryBefore := Smalltalk vm parameterAt: 3.
139+
gcBefore := (Smalltalk vm parameterAt: 7) + (Smalltalk vm parameterAt: 9).
140+
gcTimeBefore := (Smalltalk vm parameterAt: 8) + (Smalltalk vm parameterAt: 10).
141+
startTime := Time millisecondClockValue.
142+
143+
priceBuffer := CTFIFOBuffer new: windowSize.
144+
movingAverages := OrderedCollection new.
145+
processedCount := 0.
146+
147+
stockDataFile asFileReference readStreamDo: [ :fileStream |
148+
| line |
149+
fileStream atEnd ifFalse: [ fileStream nextLine ].
150+
151+
[ fileStream atEnd ] whileFalse: [
152+
line := fileStream nextLine.
153+
154+
line ifNotEmpty: [
155+
| csvParts price |
156+
csvParts := line splitOn: ','.
157+
price := (csvParts at: 2) asNumber. "Price is 2nd column"
158+
159+
priceBuffer push: price.
160+
processedCount := processedCount + 1.
161+
162+
"Calculate moving average when buffer is full"
163+
priceBuffer isFull ifTrue: [
164+
| bufferSum movingAvg |
165+
bufferSum := 0.
166+
priceBuffer do: [ :p | bufferSum := bufferSum + p ].
167+
movingAvg := bufferSum / priceBuffer size.
168+
movingAverages add: movingAvg.
169+
].
170+
].
171+
].
172+
].
173+
174+
endTime := Time millisecondClockValue.
175+
memoryAfter := Smalltalk vm parameterAt: 3.
176+
gcAfter := (Smalltalk vm parameterAt: 7) + (Smalltalk vm parameterAt: 9).
177+
gcTimeAfter := (Smalltalk vm parameterAt: 8) + (Smalltalk vm parameterAt: 10).
178+
179+
Transcript show: 'Buffer Results:'; cr.
180+
Transcript show: ' Time: ', (endTime - startTime) asString, ' ms'; cr.
181+
Transcript show: ' Memory: ', ((memoryAfter - memoryBefore) / 1024 / 1024) rounded asString, ' MB'; cr.
182+
Transcript show: ' GC Events: ', (gcAfter - gcBefore) asString; cr.
183+
Transcript show: ' GC Time: ', (gcTimeAfter - gcTimeBefore) asString, ' ms'; cr.
184+
Transcript show: ' Moving Averages: ', movingAverages size asString; cr.
185+
Transcript show: ' Final MA: $', (movingAverages last roundTo: 0.01) asString; cr; cr.
186+
] value.
187+
188+
189+
"Cleanup..."
190+
stockDataFile asFileReference delete.
191+
Transcript show: 'Tests Done!'; cr.
192+
```
193+
194+
### Benchmark Results
195+
Here are the results from running this benchmark on a 500,000-row dataset (approximately 15MB file):
196+
| Metric | DataFrame | Circular Buffer | Improvement |
197+
|--------|-----------|-----------------|-------------|
198+
| **Execution Time** | ~15,100 ms | ~2,100 ms | **7.2x faster** |
199+
| **Memory Usage** | ~128 MB | ~16 MB | **8x less memory** |
200+
| **GC Events** | ~870 | ~52 | **94% fewer** |
201+
| **GC Time** | ~3,500 ms | ~3 ms | **1200x less GC overhead** |
202+
| **Results Generated** | 499,901 | 499,901 | Identical accuracy |
203+
204+
*Note: Results may vary based on your hardware, Pharo version, and system load*
205+
206+
**Key Insights:**
207+
- Circular buffers processed the same data **7.2x faster**
208+
- Used **8x less memory** despite processing the same amount of data
209+
- Had **94% fewer garbage collection events**, leading to smoother performance
210+
- Spent virtually no time on memory cleanup (3ms vs 3.5+ seconds)
211+
- Produced identical results, proving accuracy isn't compromised
212+
213+
## When to Use Each Approach
214+
215+
### Use DataFrames When:
216+
- Your entire dataset comfortably fits in memory (under 100MB typically)
217+
- You need complex operations like joins, group-by, or statistical functions
218+
- You're doing one-time analysis where you explore data interactively
219+
- You need to sort, filter, or query data in complex ways
220+
221+
### Use Circular Buffers When:
222+
- Processing large files (over 100MB)
223+
- Computing simple statistics (averages, sums, counts)
224+
- Building real-time systems that process continuous data streams
225+
- Working with memory-limited environments
226+
- Processing data bigger than your available RAM
227+
228+
## Try It Yourself
229+
230+
1. Generate some test data using the code above
231+
2. Run both approaches on files of different sizes
232+
3. Watch the memory usage and timing differences
233+
4. See how circular buffers handle files bigger than your RAM
234+
235+
You'll be surprised at how much faster your programs can run when you choose the right data structure for the job.
236+
237+
## Summary
238+
In this article, we explored the limitations of DataFrames when processing large datasets in Pharo and introduced circular buffers as a more efficient alternative. We demonstrated how circular buffers can handle large files without running out of memory, while also being significantly faster for simple computations like averages and moving averages.
239+
240+
We also provided practical examples of generating test data and measuring performance for both approaches. The key takeaway is that sometimes the simplest solution is the fastest solution, especially when dealing with large datasets.
241+
242+
243+
Want to explore more? Check out the [Containers-Buffer repository](https://github.com/pharo-containers/Containers-Buffer) to see the complete implementation and examples.

0 commit comments

Comments
 (0)