Skip to content

Commit a26e407

Browse files
eisuke0912eisuke0912
committed
add app instruction
1 parent b1fce33 commit a26e407

6 files changed

Lines changed: 69 additions & 246 deletions

File tree

‎_toc.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
format: jb-book
22
root: index.md
33
chapters:
4+
- file: personal_writeup.md
45
- file: lecture_16_example.ipynb
56
- file: project 1.ipynb
67
- file: project 2.ipynb

‎images/comparison_chart.png‎

208 KB
Loading

‎images/scatter_plot.png‎

161 KB
Loading

‎images/side_by_side_map.png‎

463 KB
Loading

‎personal_writeup.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Energy Market Dashboard: Visualizing How Electricity Prices Move in New York
2+
3+
**App:** https://advanced-computing-dancing-cactus-streamlit-app-birdi8.streamlit.app/market_analysis
4+
**Github repository:** https://github.com/advanced-computing/dancing-cactus
5+
**Team:** Eisuke, Chenyi
6+
7+
# This page explain the dashboard what we created in the class.
8+
9+
## What we built
10+
11+
Our team built an interactive Streamlit dashboard that explores how electricity prices in New York State respond to demand and fuel costs. The app pulls real-time Locational Marginal Prices (LBMP) and actual load data from NYISO, combines them with Henry Hub natural gas spot prices, and lets users explore the relationships across NYISO's eleven zones and across time.
12+
13+
The question we originally set out to answer was simple: *How does the LBMP in New York City fluctuate with changes in demand?*
14+
15+
![Dashboard overview](screenshots/overview.png)
16+
17+
## The story we ended up telling
18+
19+
Early on we assumed the dashboard would mostly show a clean correlation between demand and price — true on average, but not very revealing. The dashboard evolved into a three-step narrative built around moments where intuition fails:
20+
21+
1. **Two side-by-side maps of NYISO zones**, one colored by load and one by LBMP, surface an asymmetry: the zones with the highest demand are not always the ones with the highest prices. Demand alone does not explain where electricity is expensive.
22+
![Side-by-side load and price maps](images\side_by_side_map.png)
23+
2. **A scatter plot of load vs. LBMP for a chosen zone**, faceted by month, zooms into one zone (typically NYC) and shows the relationship is non-linear. Prices stay low and stable across most of the load range but become sharply more volatile near system peaks.
24+
![A scatter plot of load vs. LBMP](images\scatter_plot.png)
25+
3. **A dual-axis time series of LBMP and Henry Hub gas prices** shows that fuel costs — the other obvious explanatory variable — also fail to account for the spikes, pointing toward structural factors like transmission congestion and the local generation mix.
26+
![A dual-axis time series of LBMP and Henry Hub gas prices](images\comparison_chart.png)
27+
The dashboard is not meant to give a single answer. It walks a policymaker from "demand drives price" to "price is shaped by *where* demand occurs within a constrained grid."
28+
29+
30+
## How it's built
31+
32+
The system is a thin Streamlit front-end on top of an automated BigQuery pipeline:
33+
34+
- **Ingestion.** Two Python scripts per data source — one for the initial bulk backfill, one for daily incremental updates. Splitting them keeps the production path lean and the historical load reproducible.
35+
- **Storage.** Three tables in BigQuery (LBMP, actual load, Henry Hub). Choosing BigQuery over local CSVs let us share data across the team and scale to roughly ten years of hourly data without rethinking the pipeline.
36+
- **Automation.** A GitHub Actions cron job runs the update scripts daily, satisfying the project's "data is automatically updated" requirement.
37+
- **Front-end.** Streamlit, with `@st.cache_data(ttl=3600)` on the BigQuery loaders and URL query parameters for shareable state. Maps use Plotly choropleth on a NYISO zone GeoJSON.
38+
The repo is organized so anyone with a service account can clone it and run the dashboard locally.
39+
40+
## What I learned
41+
42+
Two lessons stuck with me, and neither is about a specific tool:
43+
44+
**Performance forces design choices.** Loading a year of hourly data on every interaction was unworkable, so we exposed a month selector instead of a year-range slider. That UI decision is really a backend constraint in disguise. Splitting the BigQuery loaders into separate cached functions cut typical interactions from ~20 seconds to under one. `@st.cache_data` is powerful, but it also hides bugs — a stale cache once made me think a path fix wasn't working when it actually was.
45+
46+
**Visualization choices are arguments.** Putting the two maps side by side is a different argument than overlaying them; faceting the scatter plot by month is a different argument than aggregating across a year. Every chart in the final dashboard exists because we decided what we wanted the viewer to *notice*, then worked backward to the encoding. The biggest improvements late in the project came not from adding features, but from cutting interpretive text and letting the visuals carry more of the argument.
47+
48+
The presentation went well, and this is now a project I'd be happy to walk through in an interview.

‎project 3.ipynb‎

Lines changed: 20 additions & 246 deletions
Original file line numberDiff line numberDiff line change
@@ -1941,128 +1941,20 @@
19411941
},
19421942
{
19431943
"cell_type": "code",
1944-
"execution_count": 29,
1944+
"execution_count": 3,
19451945
"id": "2c9cc415",
19461946
"metadata": {},
19471947
"outputs": [
19481948
{
1949-
"data": {
1950-
"text/html": [
1951-
"<div>\n",
1952-
"<style scoped>\n",
1953-
" .dataframe tbody tr th:only-of-type {\n",
1954-
" vertical-align: middle;\n",
1955-
" }\n",
1956-
"\n",
1957-
" .dataframe tbody tr th {\n",
1958-
" vertical-align: top;\n",
1959-
" }\n",
1960-
"\n",
1961-
" .dataframe thead th {\n",
1962-
" text-align: right;\n",
1963-
" }\n",
1964-
"</style>\n",
1965-
"<table border=\"1\" class=\"dataframe\">\n",
1966-
" <thead>\n",
1967-
" <tr style=\"text-align: right;\">\n",
1968-
" <th></th>\n",
1969-
" <th>Country Name</th>\n",
1970-
" <th>2020</th>\n",
1971-
" <th>Growth</th>\n",
1972-
" </tr>\n",
1973-
" </thead>\n",
1974-
" <tbody>\n",
1975-
" <tr>\n",
1976-
" <th>0</th>\n",
1977-
" <td>Angola</td>\n",
1978-
" <td>2.667447</td>\n",
1979-
" <td>-0.008447</td>\n",
1980-
" </tr>\n",
1981-
" <tr>\n",
1982-
" <th>1</th>\n",
1983-
" <td>Albania</td>\n",
1984-
" <td>3.325040</td>\n",
1985-
" <td>0.026534</td>\n",
1986-
" </tr>\n",
1987-
" <tr>\n",
1988-
" <th>2</th>\n",
1989-
" <td>Argentina</td>\n",
1990-
" <td>5.276900</td>\n",
1991-
" <td>-0.069984</td>\n",
1992-
" </tr>\n",
1993-
" <tr>\n",
1994-
" <th>3</th>\n",
1995-
" <td>Armenia</td>\n",
1996-
" <td>2.705560</td>\n",
1997-
" <td>0.035840</td>\n",
1998-
" </tr>\n",
1999-
" <tr>\n",
2000-
" <th>4</th>\n",
2001-
" <td>Australia</td>\n",
2002-
" <td>5.386250</td>\n",
2003-
" <td>-0.004416</td>\n",
2004-
" </tr>\n",
2005-
" <tr>\n",
2006-
" <th>...</th>\n",
2007-
" <td>...</td>\n",
2008-
" <td>...</td>\n",
2009-
" <td>...</td>\n",
2010-
" </tr>\n",
2011-
" <tr>\n",
2012-
" <th>103</th>\n",
2013-
" <td>Uruguay</td>\n",
2014-
" <td>4.542920</td>\n",
2015-
" <td>0.000059</td>\n",
2016-
" </tr>\n",
2017-
" <tr>\n",
2018-
" <th>104</th>\n",
2019-
" <td>United States</td>\n",
2020-
" <td>5.395320</td>\n",
2021-
" <td>0.009106</td>\n",
2022-
" </tr>\n",
2023-
" <tr>\n",
2024-
" <th>105</th>\n",
2025-
" <td>South Africa</td>\n",
2026-
" <td>6.170670</td>\n",
2027-
" <td>-0.041478</td>\n",
2028-
" </tr>\n",
2029-
" <tr>\n",
2030-
" <th>106</th>\n",
2031-
" <td>Zambia</td>\n",
2032-
" <td>3.943741</td>\n",
2033-
" <td>0.022487</td>\n",
2034-
" </tr>\n",
2035-
" <tr>\n",
2036-
" <th>107</th>\n",
2037-
" <td>Zimbabwe</td>\n",
2038-
" <td>NaN</td>\n",
2039-
" <td>-0.023958</td>\n",
2040-
" </tr>\n",
2041-
" </tbody>\n",
2042-
"</table>\n",
2043-
"<p>108 rows × 3 columns</p>\n",
2044-
"</div>"
2045-
],
2046-
"text/plain": [
2047-
" Country Name 2020 Growth\n",
2048-
"0 Angola 2.667447 -0.008447\n",
2049-
"1 Albania 3.325040 0.026534\n",
2050-
"2 Argentina 5.276900 -0.069984\n",
2051-
"3 Armenia 2.705560 0.035840\n",
2052-
"4 Australia 5.386250 -0.004416\n",
2053-
".. ... ... ...\n",
2054-
"103 Uruguay 4.542920 0.000059\n",
2055-
"104 United States 5.395320 0.009106\n",
2056-
"105 South Africa 6.170670 -0.041478\n",
2057-
"106 Zambia 3.943741 0.022487\n",
2058-
"107 Zimbabwe NaN -0.023958\n",
2059-
"\n",
2060-
"[108 rows x 3 columns]"
2061-
]
2062-
},
2063-
"execution_count": 29,
2064-
"metadata": {},
2065-
"output_type": "execute_result"
1949+
"ename": "NameError",
1950+
"evalue": "name 'education_2020' is not defined",
1951+
"output_type": "error",
1952+
"traceback": [
1953+
"\u001b[31m---------------------------------------------------------------------------\u001b[39m",
1954+
"\u001b[31mNameError\u001b[39m Traceback (most recent call last)",
1955+
"\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[3]\u001b[39m\u001b[32m, line 1\u001b[39m\n\u001b[32m----> \u001b[39m\u001b[32m1\u001b[39m df_merge = pd.merge(left=\u001b[43meducation_2020\u001b[49m, right=tfp_growth,\n\u001b[32m 2\u001b[39m left_on=\u001b[33m\"\u001b[39m\u001b[33mCountry Name\u001b[39m\u001b[33m\"\u001b[39m, right_on=\u001b[33m\"\u001b[39m\u001b[33mEntity\u001b[39m\u001b[33m\"\u001b[39m)\n\u001b[32m 3\u001b[39m df_merge\n",
1956+
"\u001b[31mNameError\u001b[39m: name 'education_2020' is not defined"
1957+
]
20661958
}
20671959
],
20681960
"source": [
@@ -6018,138 +5910,20 @@
60185910
},
60195911
{
60205912
"cell_type": "code",
6021-
"execution_count": 31,
5913+
"execution_count": 2,
60225914
"id": "093e2dc5",
60235915
"metadata": {},
60245916
"outputs": [
60255917
{
6026-
"data": {
6027-
"text/html": [
6028-
"<table class=\"simpletable\">\n",
6029-
"<caption>OLS Regression Results</caption>\n",
6030-
"<tr>\n",
6031-
" <th>Dep. Variable:</th> <td>Growth</td> <th> R-squared: </th> <td> 0.001</td>\n",
6032-
"</tr>\n",
6033-
"<tr>\n",
6034-
" <th>Model:</th> <td>OLS</td> <th> Adj. R-squared: </th> <td> -0.010</td>\n",
6035-
"</tr>\n",
6036-
"<tr>\n",
6037-
" <th>Method:</th> <td>Least Squares</td> <th> F-statistic: </th> <td> 0.06353</td>\n",
6038-
"</tr>\n",
6039-
"<tr>\n",
6040-
" <th>Date:</th> <td>Sat, 06 Dec 2025</td> <th> Prob (F-statistic):</th> <td> 0.802</td> \n",
6041-
"</tr>\n",
6042-
"<tr>\n",
6043-
" <th>Time:</th> <td>22:47:39</td> <th> Log-Likelihood: </th> <td> 221.71</td>\n",
6044-
"</tr>\n",
6045-
"<tr>\n",
6046-
" <th>No. Observations:</th> <td> 98</td> <th> AIC: </th> <td> -439.4</td>\n",
6047-
"</tr>\n",
6048-
"<tr>\n",
6049-
" <th>Df Residuals:</th> <td> 96</td> <th> BIC: </th> <td> -434.2</td>\n",
6050-
"</tr>\n",
6051-
"<tr>\n",
6052-
" <th>Df Model:</th> <td> 1</td> <th> </th> <td> </td> \n",
6053-
"</tr>\n",
6054-
"<tr>\n",
6055-
" <th>Covariance Type:</th> <td>nonrobust</td> <th> </th> <td> </td> \n",
6056-
"</tr>\n",
6057-
"</table>\n",
6058-
"<table class=\"simpletable\">\n",
6059-
"<tr>\n",
6060-
" <td></td> <th>coef</th> <th>std err</th> <th>t</th> <th>P>|t|</th> <th>[0.025</th> <th>0.975]</th> \n",
6061-
"</tr>\n",
6062-
"<tr>\n",
6063-
" <th>Intercept</th> <td> -0.0075</td> <td> 0.008</td> <td> -0.927</td> <td> 0.356</td> <td> -0.023</td> <td> 0.008</td>\n",
6064-
"</tr>\n",
6065-
"<tr>\n",
6066-
" <th>Q('2020')</th> <td> 0.0004</td> <td> 0.002</td> <td> 0.252</td> <td> 0.802</td> <td> -0.003</td> <td> 0.004</td>\n",
6067-
"</tr>\n",
6068-
"</table>\n",
6069-
"<table class=\"simpletable\">\n",
6070-
"<tr>\n",
6071-
" <th>Omnibus:</th> <td> 4.766</td> <th> Durbin-Watson: </th> <td> 2.138</td>\n",
6072-
"</tr>\n",
6073-
"<tr>\n",
6074-
" <th>Prob(Omnibus):</th> <td> 0.092</td> <th> Jarque-Bera (JB): </th> <td> 5.269</td>\n",
6075-
"</tr>\n",
6076-
"<tr>\n",
6077-
" <th>Skew:</th> <td>-0.237</td> <th> Prob(JB): </th> <td> 0.0718</td>\n",
6078-
"</tr>\n",
6079-
"<tr>\n",
6080-
" <th>Kurtosis:</th> <td> 4.032</td> <th> Cond. No. </th> <td> 16.3</td>\n",
6081-
"</tr>\n",
6082-
"</table><br/><br/>Notes:<br/>[1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
6083-
],
6084-
"text/latex": [
6085-
"\\begin{center}\n",
6086-
"\\begin{tabular}{lclc}\n",
6087-
"\\toprule\n",
6088-
"\\textbf{Dep. Variable:} & Growth & \\textbf{ R-squared: } & 0.001 \\\\\n",
6089-
"\\textbf{Model:} & OLS & \\textbf{ Adj. R-squared: } & -0.010 \\\\\n",
6090-
"\\textbf{Method:} & Least Squares & \\textbf{ F-statistic: } & 0.06353 \\\\\n",
6091-
"\\textbf{Date:} & Sat, 06 Dec 2025 & \\textbf{ Prob (F-statistic):} & 0.802 \\\\\n",
6092-
"\\textbf{Time:} & 22:47:39 & \\textbf{ Log-Likelihood: } & 221.71 \\\\\n",
6093-
"\\textbf{No. Observations:} & 98 & \\textbf{ AIC: } & -439.4 \\\\\n",
6094-
"\\textbf{Df Residuals:} & 96 & \\textbf{ BIC: } & -434.2 \\\\\n",
6095-
"\\textbf{Df Model:} & 1 & \\textbf{ } & \\\\\n",
6096-
"\\textbf{Covariance Type:} & nonrobust & \\textbf{ } & \\\\\n",
6097-
"\\bottomrule\n",
6098-
"\\end{tabular}\n",
6099-
"\\begin{tabular}{lcccccc}\n",
6100-
" & \\textbf{coef} & \\textbf{std err} & \\textbf{t} & \\textbf{P$> |$t$|$} & \\textbf{[0.025} & \\textbf{0.975]} \\\\\n",
6101-
"\\midrule\n",
6102-
"\\textbf{Intercept} & -0.0075 & 0.008 & -0.927 & 0.356 & -0.023 & 0.008 \\\\\n",
6103-
"\\textbf{Q('2020')} & 0.0004 & 0.002 & 0.252 & 0.802 & -0.003 & 0.004 \\\\\n",
6104-
"\\bottomrule\n",
6105-
"\\end{tabular}\n",
6106-
"\\begin{tabular}{lclc}\n",
6107-
"\\textbf{Omnibus:} & 4.766 & \\textbf{ Durbin-Watson: } & 2.138 \\\\\n",
6108-
"\\textbf{Prob(Omnibus):} & 0.092 & \\textbf{ Jarque-Bera (JB): } & 5.269 \\\\\n",
6109-
"\\textbf{Skew:} & -0.237 & \\textbf{ Prob(JB): } & 0.0718 \\\\\n",
6110-
"\\textbf{Kurtosis:} & 4.032 & \\textbf{ Cond. No. } & 16.3 \\\\\n",
6111-
"\\bottomrule\n",
6112-
"\\end{tabular}\n",
6113-
"%\\caption{OLS Regression Results}\n",
6114-
"\\end{center}\n",
6115-
"\n",
6116-
"Notes: \\newline\n",
6117-
" [1] Standard Errors assume that the covariance matrix of the errors is correctly specified."
6118-
],
6119-
"text/plain": [
6120-
"<class 'statsmodels.iolib.summary.Summary'>\n",
6121-
"\"\"\"\n",
6122-
" OLS Regression Results \n",
6123-
"==============================================================================\n",
6124-
"Dep. Variable: Growth R-squared: 0.001\n",
6125-
"Model: OLS Adj. R-squared: -0.010\n",
6126-
"Method: Least Squares F-statistic: 0.06353\n",
6127-
"Date: Sat, 06 Dec 2025 Prob (F-statistic): 0.802\n",
6128-
"Time: 22:47:39 Log-Likelihood: 221.71\n",
6129-
"No. Observations: 98 AIC: -439.4\n",
6130-
"Df Residuals: 96 BIC: -434.2\n",
6131-
"Df Model: 1 \n",
6132-
"Covariance Type: nonrobust \n",
6133-
"==============================================================================\n",
6134-
" coef std err t P>|t| [0.025 0.975]\n",
6135-
"------------------------------------------------------------------------------\n",
6136-
"Intercept -0.0075 0.008 -0.927 0.356 -0.023 0.008\n",
6137-
"Q('2020') 0.0004 0.002 0.252 0.802 -0.003 0.004\n",
6138-
"==============================================================================\n",
6139-
"Omnibus: 4.766 Durbin-Watson: 2.138\n",
6140-
"Prob(Omnibus): 0.092 Jarque-Bera (JB): 5.269\n",
6141-
"Skew: -0.237 Prob(JB): 0.0718\n",
6142-
"Kurtosis: 4.032 Cond. No. 16.3\n",
6143-
"==============================================================================\n",
6144-
"\n",
6145-
"Notes:\n",
6146-
"[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
6147-
"\"\"\""
6148-
]
6149-
},
6150-
"execution_count": 31,
6151-
"metadata": {},
6152-
"output_type": "execute_result"
5918+
"ename": "NameError",
5919+
"evalue": "name 'df_merge' is not defined",
5920+
"output_type": "error",
5921+
"traceback": [
5922+
"\u001b[31m---------------------------------------------------------------------------\u001b[39m",
5923+
"\u001b[31mNameError\u001b[39m Traceback (most recent call last)",
5924+
"\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[2]\u001b[39m\u001b[32m, line 2\u001b[39m\n\u001b[32m 1\u001b[39m \u001b[38;5;28;01mimport\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34;01mstatsmodels\u001b[39;00m\u001b[34;01m.\u001b[39;00m\u001b[34;01mformula\u001b[39;00m\u001b[34;01m.\u001b[39;00m\u001b[34;01mapi\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mas\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34;01msmf\u001b[39;00m\n\u001b[32m----> \u001b[39m\u001b[32m2\u001b[39m model = smf.ols(\u001b[33m\"\u001b[39m\u001b[33mGrowth ~ Q(\u001b[39m\u001b[33m'\u001b[39m\u001b[33m2020\u001b[39m\u001b[33m'\u001b[39m\u001b[33m)\u001b[39m\u001b[33m\"\u001b[39m, data=\u001b[43mdf_merge\u001b[49m)\n\u001b[32m 3\u001b[39m results = model.fit()\n\u001b[32m 4\u001b[39m results.summary()\n",
5925+
"\u001b[31mNameError\u001b[39m: name 'df_merge' is not defined"
5926+
]
61535927
}
61545928
],
61555929
"source": [

0 commit comments

Comments
 (0)