[Data] Refactor br_tse_eleicoes#1476
Conversation
…tions (Phases 0-1) Phase 0 -- Infrastructure: config.py (paths, year ranges, state lists, null sentinels), utils/helpers.py (read_raw_csv, clean_nulls, parse_date_br, pad_cpf, pad_titulo, merge_municipio, save_partitioned), and validate.py for comparing Python parquet outputs against Stata .dta references. Phase 1 -- Cleaning functions ported from Stata fnc/ to Python utils/: clean_string, clean_election_type, clean_education, clean_marital_status, clean_result, clean_party, fix_candidate. Data values remain in Portuguese to match the source data; file and function names are in English.
…les) Implements 13 sub/ modules translating each Stata .do table builder to Python: - candidates, parties, vacancies - voter_profile_mun_zone, voter_profile_section, voter_profile_polling_place - voting_details_mun_zone, voting_details_section, voting_details_state (1945-1990) - results_mun_zone, results_section, results_state (1945-1990) - campaign_finance (bens 2006+, receitas 2002+, despesas 2002+)
…and build runner (Phases 3-4)
- clean_election_type: map 'eleicao YYYY' → 'eleicao ordinaria' for all years (previously only applied for ano > 1982) - voting_details_state: guard proportion calculations against div-by-zero - results_mun_zone: fix id_municipio column order for candidato 1996-2016 (1994 uses natural merge order; partido fix already applied) - results_section: fix partido merge key regression that produced _x/_y suffix columns; enforce column order to match Stata output - results_state: fix apostrophe/accent casing after str.title(); add column order; fix file-path patterns and encoding (latin-1) - candidates/parties/voter_profile_section: fix title-case accent handling, column ordering, and encoding issues - campaign_finance: fix multiline truncation, quote stripping, encoding - helpers: default encoding latin-1 for TSE raw CSV files - validate: improve float normalization and column comparison logic - remove stale [dbt] notebook
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds a modular Python ETL for br_tse_eleicoes: centralized config, per-table/year builders (candidates, parties, results, voting details, voter profiles, campaign finance), normalization/partitioning, aggregation, and a validator; includes a PLAN and removal of an older Jupyter notebook. ChangesDocumentation & Planning
Removed
Core Pipeline Orchestration
Table-Specific Builders
Voter Profile Builders
Voting Details Builders
Campaign Finance Builder
Utility Modules
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
Note
Due to the large number of review comments, Critical severity comments were prioritized as inline comments.
🟠 Major comments (15)
models/br_tse_eleicoes/code/python/utils/clean_party.py-8-13 (1)
8-13:⚠️ Potential issue | 🟠 MajorAdd docstring to
clean_party()and fix mapping order for party variants.The
clean_party()function lacks a docstring. Additionally, year-specific mapping is applied before always-mapping, which causes inconsistency: variants like"PTdoB"and"PT DO B"are normalized to"PT do B"via_ALWAYS_MAP, but this happens after year-specific mapping, so they won't be converted to"AVANTE"in 2014/2016 even though"PT do B"is. To ensure all variants map consistently, apply_ALWAYS_MAPbefore_YEAR_MAP.Also,
clean_party_series()is missing type hints: add type annotation for parametersand return type (should bepandas.Series).🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/utils/clean_party.py` around lines 8 - 13, Add a docstring to clean_party() describing its purpose, inputs and outputs; change the normalization order so that _ALWAYS_MAP is applied before _YEAR_MAP (i.e., normalize variants via _ALWAYS_MAP first, then apply year-specific remapping using _YEAR_MAP) to ensure variants like "PTdoB"/"PT DO B" resolve to "AVANTE" for 2014/2016; and add type hints to clean_party_series(s: pandas.Series) -> pandas.Series for the function signature.models/br_tse_eleicoes/code/python/utils/fix_candidate.py-14-18 (1)
14-18:⚠️ Potential issue | 🟠 MajorFix type mismatch:
anocolumn is numeric, not string.Both masks compare
df["ano"]to string literals ("2000"and"2006"), but upstream code confirmsanois stored as numeric (int). Comparisons likedf["ano"] % 2anddf["ano"] = 201407in voting processing modules prove the column type. String comparisons to numeric values will silently yield no matches, preventing the fixes from being applied.Update both comparisons to convert the column to string:
Suggested fix
mask = ( - (df["ano"] == "2000") + (df["ano"].astype(str) == "2000") & (df["id_municipio"] == "4202909") & (df["nome"] == "Paulo Peixer") )Apply the same fix to the second mask at lines 26-29 for
"2006".🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/utils/fix_candidate.py` around lines 14 - 18, The mask currently compares df["ano"] to the string "2000" (and another mask compares to "2006"), but ano is numeric; update both masks so they compare the string form of the year by calling df["ano"].astype(str) (e.g., replace df["ano"] == "2000" with df["ano"].astype(str) == "2000") to ensure matches for the mask variables that assign to mask (and the second mask that checks "2006").models/br_tse_eleicoes/code/python/validate.py-19-32 (1)
19-32:⚠️ Potential issue | 🟠 MajorThe validator currently skips outputs that are written once for all years.
Both loaders and
discover_tables()only understand{table}_{year}artifacts. That means datasets likeperfil_eleitorado_local_votacao.parquetfrommodels/br_tse_eleicoes/code/python/sub/voter_profile_polling_place.pynever enter validation, so the parity check is incomplete for at least one generated table.Also applies to: 204-212, 259-267
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/validate.py` around lines 19 - 32, The loaders (load_stata_dta, load_python_parquet) and discover_tables() only look for files named with the "{table}_{year}" pattern, so year-agnostic artifacts (e.g., perfil_eleitorado_local_votacao.parquet) are skipped; update load_stata_dta and load_python_parquet to fallback to a yearless filename when the yeared path doesn't exist (check OUTPUT_STATA / f"{table}.dta" and OUTPUT_PYTHON / f"{table}.parquet"), and update discover_tables() to include files that match either the "{table}_{year}" pattern or the standalone "{table}" pattern so those generated tables are discovered and validated.models/br_tse_eleicoes/code/python/validate.py-179-195 (1)
179-195:⚠️ Potential issue | 🟠 MajorReport numeric nullability mismatches even when one side is all-null.
The
if ref_nn > 0 and test_nn > 0:guard suppresses cases likeref_nn == 0andtest_nn > 0, so a materially different numeric column can still produce no numeric issue. Compare non-null counts unconditionally, then skip only the sum check when both sides are fully null.🐛 Proposed fix
ref_nn = ref_num.notna().sum() test_nn = test_num.notna().sum() - if ref_nn > 0 and test_nn > 0: - if ref_nn != test_nn: - issues.append( - f"[{label}] Column '{col}' numeric count: ref={ref_nn}, test={test_nn}" - ) - ref_sum = ref_num.sum() - test_sum = test_num.sum() - if not np.isclose(ref_sum, test_sum, rtol=1e-6, equal_nan=True): - issues.append( - f"[{label}] Column '{col}' sum: ref={ref_sum}, test={test_sum}" - ) + if ref_nn != test_nn: + issues.append( + f"[{label}] Column '{col}' numeric count: ref={ref_nn}, test={test_nn}" + ) + if ref_nn == 0 and test_nn == 0: + continue + ref_sum = ref_num.sum() + test_sum = test_num.sum() + if not np.isclose(ref_sum, test_sum, rtol=1e-6, equal_nan=True): + issues.append( + f"[{label}] Column '{col}' sum: ref={ref_sum}, test={test_sum}" + )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/validate.py` around lines 179 - 195, In the numeric-column loop that iterates over float_cols + numeric_cols, always compare and report non-null count mismatches by computing ref_nn and test_nn and appending the issues entry when ref_nn != test_nn; only guard the sum comparison (ref_sum/test_sum using ref_num/test_num) behind a conditional that both ref_nn > 0 and test_nn > 0 so you skip sum checks when either side is all-null. Update the block around ref_num/test_num, ref_nn/test_nn, and the issues.append calls to enforce unconditional count checks but conditional sum checks.models/br_tse_eleicoes/code/python/sub/voter_profile_section.py-120-157 (1)
120-157:⚠️ Potential issue | 🟠 MajorNormalize
anobefore grouping, or 2014 section data will keep the bad201407key.This builder never destrings
ano, so it also never applies the201407 -> 2014correction that the municipality-zone builder already carries. Grouping on the raw value here will emit 2014 rows under the wrong year and break year-keyed joins/validation for the section output.🐛 Proposed fix
for col in [ + "ano", "id_municipio_tse", "zona", "secao", "eleitores", "eleitores_biometria", "eleitores_deficiencia", "eleitores_inclusao_nome_social", ]: if col in df.columns: df[col] = pd.to_numeric(df[col], errors="coerce") + + df.loc[df["ano"] == 201407, "ano"] = 2014🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/sub/voter_profile_section.py` around lines 120 - 157, The grouping uses the raw 'ano' column so rows like 201407 remain and break year-keyed joins; before building group_cols and grouping on df, coerce/destring the 'ano' column (e.g., pd.to_numeric(df['ano'], errors='coerce')) and normalize the 201407 case to 2014 (apply the same 201407 -> 2014 correction used in the municipality-zone builder) so group_cols/groupby uses the corrected year; update references around df, group_cols, and sum_cols to ensure 'ano' is in numeric/normalized form prior to df.groupby.models/br_tse_eleicoes/code/python/config.py-12-17 (1)
12-17:⚠️ Potential issue | 🟠 MajorDon't bake one developer's filesystem into the checked-in config.
RAW_DATApoints at/Users/rdahis/Downloads/dados_TSE, so every builder and validator imported from this module will fail anywhere except that machine. Please make the data root configurable via env var / CLI / local config instead of a repo-committed absolute path.💡 Portable configuration sketch
+import os from pathlib import Path @@ -RAW_DATA = Path("/Users/rdahis/Downloads/dados_TSE") +RAW_DATA = Path( + os.environ.get("BR_TSE_ELEICOES_RAW_DATA", "dados_TSE") +).expanduser()🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/config.py` around lines 12 - 17, The config currently hardcodes RAW_DATA = Path("/Users/rdahis/Downloads/dados_TSE") which breaks on other machines; change RAW_DATA to be configurable (e.g., read from an environment variable like os.getenv("BR_TSE_RAW_DATA") with a sensible default) and update dependent symbols INPUT_DIR, OUTPUT_STATA, OUTPUT_PYTHON, and MUNICIPIO_DIR_CSV to derive from that value; ensure the module falls back to a relative repo path if the env var is not set and document the env var name in a comment so local devs can override without editing the file.models/br_tse_eleicoes/code/python/sub/parties.py-27-36 (1)
27-36:⚠️ Potential issue | 🟠 MajorDon't let malformed party rows get silently dropped.
The
on_bad_lines="warn"parameter skips bad records without including them in the DataFrame, creating silent data loss. Since this file is explicitly designed to match Stata output exactly (as noted in the module docstring: "Equivalent of sub/partidos.do"), dropping records without failing breaks that equivalence goal. Either fail fast on bad input or explicitly quarantine problematic files instead.🔧 Safer default
return pd.read_csv( path, sep=";", header=None, dtype=str, encoding="latin-1", keep_default_na=False, quotechar='"', - on_bad_lines="warn", )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/sub/parties.py` around lines 27 - 36, The current pd.read_csv call in models/br_tse_eleicoes/code/python/sub/parties.py uses on_bad_lines="warn" which silently drops malformed rows; change this to fail fast by using on_bad_lines="error" so the exception propagates, or implement an on_bad_lines callable that writes each malformed line to a quarantine file (e.g., via a handle_bad_line(line) callback) and then raises an exception to stop processing—update the pd.read_csv invocation accordingly and ensure any raised error is not swallowed so malformed party rows are not silently lost.models/br_tse_eleicoes/code/python/sub/results_state.py-60-61 (1)
60-61:⚠️ Potential issue | 🟠 MajorMove historical sentinel cleanup before
pd.to_numeric().
_HIST_NULLSincludes"-1"and"-3", but both builders coerce numeric columns first. After that, those sentinels are numeric-1/-3and_clean_hist_nulls()no longer removes them, so negative vote counts can leak through.🔧 Suggested fix
- # destring - for col in ["ano", "turno", "votos"]: - df[col] = pd.to_numeric(df[col], errors="coerce") - - # clean nulls - df = _clean_hist_nulls(df) + # clean nulls before destring so "-1"/"-3" are removed + df = _clean_hist_nulls(df) + for col in ["ano", "turno", "votos"]: + df[col] = pd.to_numeric(df[col], errors="coerce")Apply the same reorder in
build_partido()forvotos_nominaisandvotos_nao_nominais.Also applies to: 207-212, 319-324
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/sub/results_state.py` around lines 60 - 61, The historical sentinel set _HIST_NULLS contains string tokens like "-1" and "-3" which get coerced to numeric -1/-3 by pd.to_numeric, so update the data cleaning order: call _clean_hist_nulls(...) to remove those string sentinels before any pd.to_numeric conversions. Specifically modify build_partido() (and the other builders that handle votos_nominais and votos_nao_nominais) to perform _clean_hist_nulls on those columns first, then call pd.to_numeric on the cleaned columns, and ensure _HIST_NULLS and _clean_hist_nulls() are referenced for clarity.models/br_tse_eleicoes/code/python/sub/candidates.py-174-176 (1)
174-176:⚠️ Potential issue | 🟠 Major
fix_candidate()never matches afteranois coerced to numeric.
models/br_tse_eleicoes/code/python/utils/fix_candidate.pycomparesdf["ano"]against the string years"2000"and"2006". By the time Line 230 calls it,anois numeric, so those manual CPF/título fixes never fire.🔧 Suggested fix in
models/br_tse_eleicoes/code/python/utils/fix_candidate.pyano = pd.to_numeric(df["ano"], errors="coerce").astype("Int64") mask = ( (ano == 2000) & (df["id_municipio"] == "4202909") & (df["nome"] == "Paulo Peixer") ) mask2 = ( (ano == 2006) & (df["sigla_uf"] == "SP") & (df["nome"] == "José Carlos Selbach Eymael") )Also applies to: 225-230
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/sub/candidates.py` around lines 174 - 176, The fix_candidate routine is comparing df["ano"] to string years, but earlier code coerces ano to numeric so those masks never match; in fix_candidate.py convert or coerce df["ano"] to a numeric/nullable Int (e.g., via pd.to_numeric(...).astype("Int64")) and then update the masks in fix_candidate (the comparisons inside fix_candidate.py that reference ano, e.g., the masks matching (ano == 2000) and (ano == 2006)) to compare against integer literals rather than strings; ensure you reference the same column names used in the function (df["ano"], df["id_municipio"] or df["id_municipio_tse"] as appropriate, df["sigla_uf"], df["nome"]) so the CPF/título fixes execute.models/br_tse_eleicoes/code/python/sub/voting_details_mun_zone.py-305-317 (1)
305-317:⚠️ Potential issue | 🟠 MajorProtect the percentage calculations from zero turnout/eligibility.
Dividing by raw
aptosandcomparecimentocan produce invalid percentages for empty zones. Reuse the same zero-to-null guard used invoting_details_state.py.🔧 Suggested fix
+ aptos_safe = df["aptos"].replace(0, pd.NA) + comparecimento_safe = df["comparecimento"].replace(0, pd.NA) df["proporcao_comparecimento"] = ( - 100 * df["comparecimento"] / df["aptos"] + 100 * df["comparecimento"] / aptos_safe ) df["proporcao_votos_validos"] = ( - 100 * df["votos_validos"] / df["comparecimento"] + 100 * df["votos_validos"] / comparecimento_safe ) df["proporcao_votos_brancos"] = ( - 100 * df["votos_brancos"] / df["comparecimento"] + 100 * df["votos_brancos"] / comparecimento_safe ) df["proporcao_votos_nulos"] = ( - 100 * df["votos_nulos"] / df["comparecimento"] + 100 * df["votos_nulos"] / comparecimento_safe )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/sub/voting_details_mun_zone.py` around lines 305 - 317, The percentage columns (proporcao_comparecimento, proporcao_votos_validos, proporcao_votos_brancos, proporcao_votos_nulos) currently divide by raw 'aptos' and 'comparecimento' and can produce inf/NaN for empty zones; update these calculations to guard against zero denominators like the implementation in voting_details_state.py by using the same zero-to-null pattern (e.g. replace 0 denominators with NaN or use conditional masking) so that divisions only occur when 'aptos' or 'comparecimento' > 0 and otherwise set the percentage to null.models/br_tse_eleicoes/code/python/sub/voting_details_section.py-145-160 (1)
145-160:⚠️ Potential issue | 🟠 MajorGuard zero denominators in the ratio columns.
Rows with
aptos == 0orcomparecimento == 0will emit invalid percentages here instead of nulls. The state-level builder already protects this path, so this diverges on edge cases.🔧 Suggested fix
+ aptos_safe = df["aptos"].replace(0, pd.NA) + comparecimento_safe = df["comparecimento"].replace(0, pd.NA) df["proporcao_comparecimento"] = ( - 100 * df["comparecimento"] / df["aptos"] + 100 * df["comparecimento"] / aptos_safe ) df["proporcao_votos_nominais"] = ( - 100 * df["votos_nominais"] / df["comparecimento"] + 100 * df["votos_nominais"] / comparecimento_safe ) df["proporcao_votos_legenda"] = ( - 100 * df["votos_legenda"] / df["comparecimento"] + 100 * df["votos_legenda"] / comparecimento_safe ) df["proporcao_votos_brancos"] = ( - 100 * df["votos_brancos"] / df["comparecimento"] + 100 * df["votos_brancos"] / comparecimento_safe ) df["proporcao_votos_nulos"] = ( - 100 * df["votos_nulos"] / df["comparecimento"] + 100 * df["votos_nulos"] / comparecimento_safe )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/sub/voting_details_section.py` around lines 145 - 160, The ratio columns (proporcao_comparecimento, proporcao_votos_nominais, proporcao_votos_legenda, proporcao_votos_brancos, proporcao_votos_nulos) currently divide by df["aptos"] or df["comparecimento"] without guarding zero denominators; update the computation so divisions produce NaN (or None) when the denominator is zero — e.g., compute proporcao_comparecimento only when df["aptos"] > 0 and compute the other proporcoes only when df["comparecimento"] > 0 using a conditional expression (pd.Series.where, np.where, or .mask) on df to set results to NaN for zero denominators; modify the expressions that create these five columns in voting_details_section.py accordingly and keep column names the same.models/br_tse_eleicoes/code/python/utils/helpers.py-63-77 (1)
63-77:⚠️ Potential issue | 🟠 MajorNarrow exception handler to
ParserErroronly.Catching
Exceptionmasks unrelated IO/schema errors and converts them to silent data skipping withon_bad_lines="skip". The fallback should only handle actual parser failures (like the SP 2014 truncated quote case), not all exceptions.🔧 Suggested fix
- except (pd.errors.ParserError, Exception): + except pd.errors.ParserError:🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/utils/helpers.py` around lines 63 - 77, The except block currently catches (pd.errors.ParserError, Exception) which masks unrelated errors; change it to catch only pd.errors.ParserError so only parser failures trigger the fallback read using engine="python" and on_bad_lines="skip", and ensure any other exceptions are not swallowed (i.e., let other exceptions propagate instead of being handled by the fallback). Locate the except line that wraps the fallback pd.read_csv call (the block that uses path, encoding, quotechar='"', engine="python", on_bad_lines="skip") and replace the broad exception tuple with a single pd.errors.ParserError.models/br_tse_eleicoes/code/python/sub/campaign_finance.py-1596-1606 (1)
1596-1606:⚠️ Potential issue | 🟠 MajorNormalize
data_despesain the 2012 despesas builder.This branch patches eight-character values but never performs the final date normalization step, so 2012 becomes the only despesas year that emits
data_despesain a different format than the rest of the table.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/sub/campaign_finance.py` around lines 1596 - 1606, The code prepends "20" for 8-char 2012 dates but never applies the final date normalization, leaving 2012 values in a different format; after the lambda that updates df["data_despesa"], run the same normalization used elsewhere for despesas (e.g., call the shared date-normalizer or apply pd.to_datetime with the project’s format and errors="coerce") on df["data_despesa"] so all branches produce consistent date types; locate this change near the lambda applying to df["data_despesa"] in the 2012 despesas builder and mirror the normalization step used by other years.models/br_tse_eleicoes/code/python/normalization_partition.py-894-905 (1)
894-905:⚠️ Potential issue | 🟠 MajorFilter
perfil_eleitorado_secaoby_UFS_PERFIL_SECAO[ano]before partitioning.The year-specific UF map is currently dead code. This loop partitions every
sigla_ufpresent in the parquet, soDF,ZZ, or other extra partitions can leak into years that should not publish them.Suggested fix
print(" partitioning perfil_eleitorado_secao...") for ano in sorted(_UFS_PERFIL_SECAO.keys()): df = _read_parquet("perfil_eleitorado_secao", ano) if df.empty: continue + allowed_ufs = set(_UFS_PERFIL_SECAO[ano]) + df = df[df["sigla_uf"].isin(allowed_ufs)].copy() save_partitioned( df, "perfil_eleitorado_secao",🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/normalization_partition.py` around lines 894 - 905, The loop currently partitions all sigla_uf present in the parquet for perfil_eleitorado_secao, ignoring the per-year whitelist _UFS_PERFIL_SECAO; before calling save_partitioned you should filter the dataframe to only rows whose "sigla_uf" is in _UFS_PERFIL_SECAO[ano] (e.g., df = df[df["sigla_uf"].isin(_UFS_PERFIL_SECAO[ano])]) and skip/save only when that filtered df is non-empty; update the code around the perfil_eleitorado_secao handling that uses _read_parquet("perfil_eleitorado_secao", ano) and save_partitioned(...) to apply this filter so extraneous UFs (DF, ZZ, etc.) are not written for years that shouldn't publish them.models/br_tse_eleicoes/code/python/aggregation.py-450-475 (1)
450-475:⚠️ Potential issue | 🟠 MajorGuard the recomputed proportions against zero denominators.
aptosandcomparecimentocan be zero after the municipality rollup. Pandas will emitinf/NaNhere, which then gets written to the published CSVs.Zero-safe division sketch
- if "comparecimento" in agg.columns and "aptos" in agg.columns: - agg["proporcao_comparecimento"] = ( - 100 * agg["comparecimento"] / agg["aptos"] - ) + if "comparecimento" in agg.columns: + comparecimento_den = agg["comparecimento"].replace(0, pd.NA) + if "comparecimento" in agg.columns and "aptos" in agg.columns: + aptos_den = agg["aptos"].replace(0, pd.NA) + agg["proporcao_comparecimento"] = ( + 100 * agg["comparecimento"] / aptos_den + ) if ( "votos_validos" in agg.columns and "comparecimento" in agg.columns ): agg["proporcao_votos_validos"] = ( - 100 * agg["votos_validos"] / agg["comparecimento"] + 100 * agg["votos_validos"] / comparecimento_den ) if ( "votos_brancos" in agg.columns and "comparecimento" in agg.columns ): agg["proporcao_votos_brancos"] = ( - 100 * agg["votos_brancos"] / agg["comparecimento"] + 100 * agg["votos_brancos"] / comparecimento_den ) if ( "votos_nulos" in agg.columns and "comparecimento" in agg.columns ): agg["proporcao_votos_nulos"] = ( - 100 * agg["votos_nulos"] / agg["comparecimento"] + 100 * agg["votos_nulos"] / comparecimento_den )🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/aggregation.py` around lines 450 - 475, When recomputing proportion columns on DataFrame agg (proporcao_comparecimento, proporcao_votos_validos, proporcao_votos_brancos, proporcao_votos_nulos), guard against zero denominators (aptos and comparecimento) by performing zero-safe division: compute each proportion only where the denominator is > 0 and otherwise set a safe value (e.g., 0 or NaN) using a conditional mask or numpy.where / pandas.Series.where; update the calculations for proporcao_comparecimento to check agg["aptos"] > 0 and for the other three to check agg["comparecimento"] > 0 so no inf/NaN values are produced and written to CSV.
🧹 Nitpick comments (17)
models/br_tse_eleicoes/code/python/utils/clean_education.py (1)
27-33: Add missing type hints and docstring per coding guidelines.
clean_educationlacks a docstring, andclean_education_serieslacks type hints.Suggested fix
def clean_education(val: str) -> str: + """Return the canonical education level for known variants, else unchanged.""" return _MAP.get(val, val) -def clean_education_series(s): +def clean_education_series(s: "pd.Series") -> "pd.Series": """Apply clean_education to a pandas Series."""🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/utils/clean_education.py` around lines 27 - 33, Add a docstring to clean_education explaining what mappings _MAP contains and that the function returns the cleaned/normalized education string (input: val: str -> output: str), and add explicit type hints to clean_education_series so its signature reads something like clean_education_series(s: pd.Series) -> pd.Series; update the docstring on clean_education_series to describe it applies clean_education element-wise to a pandas Series (preserving non-str values), and ensure pandas is imported as pd if not already.models/br_tse_eleicoes/code/python/utils/clean_string.py (2)
33-36: MissingûandÛin accent map.The map handles
ùandüfor lowercase (andÙ,Üfor uppercase), butû(u with circumflex) andÛare absent. While rare in Portuguese, they can appear in names of foreign origin.Suggested fix
"ú": "u", "ù": "u", + "û": "u", "ü": "u", "ç": "c",And for uppercase:
"Ú": "U", "Ù": "U", + "Û": "U", "Ü": "U", "Ç": "C",🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/utils/clean_string.py` around lines 33 - 36, The accent replacement map in clean_string.py is missing the lowercase 'û' and uppercase 'Û'; update the mapping (the accent map used by the clean_string utilities, e.g., the variable that currently contains "ú","ù","ü","ç", etc.) to include "û": "u" and the corresponding uppercase entry "Û": "U" in the uppercase section so both lowercase and uppercase circumflex-u are normalized correctly.
89-91: Add type hints forclean_string_series.Missing type hints for parameter
sand return type per coding guidelines.Suggested fix
-def clean_string_series(s): +def clean_string_series(s: "pd.Series") -> "pd.Series": """Apply clean_string to a pandas Series."""🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/utils/clean_string.py` around lines 89 - 91, Add type hints to clean_string_series: annotate the parameter as s: pd.Series[Any] and the return type as -> pd.Series[Any] (import Any from typing if not present). Keep the implementation the same (it maps strings and leaves other values), and ensure pandas is referenced as pd so the annotation uses pd.Series[Any]; update imports accordingly if needed.models/br_tse_eleicoes/code/PLAN.md (1)
160-166: Consider adding language specifiers to fenced code blocks.The column schema code blocks lack language specifiers (e.g.,
```textor```csv). Adding them improves rendering in some Markdown viewers and satisfies markdownlint (MD040).This applies to all schema blocks from lines 160-310.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/PLAN.md` around lines 160 - 166, The fenced code blocks that list column schemas (the blocks containing lines like "id_eleicao, tipo_eleicao, data_eleicao, ..." and other schema lists) are missing language specifiers; update each schema fenced code block in PLAN.md to include a language tag (e.g., ```text or ```csv) so Markdown renderers and markdownlint (MD040) recognize them, ensuring you add the specifier to all schema blocks in that section.models/br_tse_eleicoes/code/python/utils/clean_election_type.py (2)
13-14: Remove redundant empty string from the set.The empty string
""at line 14 is unreachable becauseif not val:at line 9 already returns early for empty strings.Suggested fix
ordinaria_variants = { - "", "ordinaria", f"eleicoes ordinarias - {ano_s}",🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/utils/clean_election_type.py` around lines 13 - 14, Remove the redundant empty string entry from the ordinaria_variants set in clean_election_type (the set named ordinaria_variants) since the function already returns early on empty values via the existing if not val: check; update the set literal to exclude "" so the unreachable item is removed and no behavior changes occur.
40-44: Add type hints forclean_election_type_series.Missing type hints for parameter
sand return type per coding guidelines.Suggested fix
-def clean_election_type_series(s, ano: int): +def clean_election_type_series(s: "pd.Series", ano: int) -> "pd.Series": """Apply clean_election_type to a pandas Series."""🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/utils/clean_election_type.py` around lines 40 - 44, Add type hints to clean_election_type_series: annotate the parameter s as a pandas Series and the return type as a pandas Series (e.g., s: pd.Series and -> pd.Series). Ensure the module imports pandas as pd or imports Series from pandas so the annotation resolves; keep the existing use of clean_election_type(…, ano) unchanged. Update the function signature to: def clean_election_type_series(s: pd.Series, ano: int) -> pd.Series:.models/br_tse_eleicoes/code/python/utils/clean_result.py (1)
17-23: Add missing type hints and docstring per coding guidelines.
clean_resultlacks a docstring, andclean_result_serieslacks type hints for both its parameter and return type.Suggested fix
def clean_result(val: str) -> str: + """Return the canonical result label for known variants, else unchanged.""" return _MAP.get(val, val) -def clean_result_series(s): +def clean_result_series(s: "pd.Series") -> "pd.Series": """Apply clean_result to a pandas Series.""" return s.map(lambda v: clean_result(v) if isinstance(v, str) else v)Note: You'll need to add
from __future__ import annotationsat the top or use a string literal for the type hint to avoid importing pandas at module level.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/utils/clean_result.py` around lines 17 - 23, Add a concise docstring to clean_result describing its purpose and parameters/return, and annotate clean_result_series with parameter and return types (e.g., s: "pd.Series" -> -> "pd.Series"); to avoid importing pandas at module scope either add from __future__ import annotations at the top of the module or use string-literal type hints, and ensure the docstring for clean_result_series describes the input Series type and that it applies clean_result to string elements.models/br_tse_eleicoes/code/python/utils/clean_party.py (1)
43-58: Add missing docstring and type hints.
clean_partylacks a docstring, andclean_party_serieslacks type hints forsand return type.Suggested fix
def clean_party(val: str, ano: int) -> str: + """Normalize party abbreviation applying year-specific then universal renames.""" if not val: return val # Year-specific year_map = _YEAR_MAP.get(ano, {}) if val in year_map: val = year_map[val] # Always if val in _ALWAYS_MAP: val = _ALWAYS_MAP[val] return val -def clean_party_series(s, ano: int): +def clean_party_series(s: "pd.Series", ano: int) -> "pd.Series": """Apply clean_party to a pandas Series."""🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/utils/clean_party.py` around lines 43 - 58, Add a short docstring to clean_party describing parameters (val: str, ano: int) and return value (normalized party string), and update clean_party_series to include type hints: change signature to def clean_party_series(s: pandas.Series, ano: int) -> pandas.Series (or use pd.Series with an import alias), ensure pandas is imported as pd if not already, and keep the implementation returning s.map(lambda v: clean_party(v, ano) if isinstance(v, str) else v); mention in the docstring that clean_party uses _YEAR_MAP and _ALWAYS_MAP for normalization.models/br_tse_eleicoes/code/python/utils/clean_marital_status.py (1)
18-26: Add missing type hints and docstring per coding guidelines.
clean_marital_statuslacks a docstring, andclean_marital_status_serieslacks type hints.Suggested fix
def clean_marital_status(val: str) -> str: + """Return the canonical marital status label for known variants, else unchanged.""" return _MAP.get(val, val) -def clean_marital_status_series(s): +def clean_marital_status_series(s: "pd.Series") -> "pd.Series": """Apply clean_marital_status to a pandas Series."""🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/utils/clean_marital_status.py` around lines 18 - 26, Add a concise docstring to clean_marital_status describing its purpose, input, and return value, and add type hints plus a docstring to clean_marital_status_series (e.g., annotate parameter s as pandas.Series and return type as pandas.Series); ensure the module imports pandas as pd if needed and the series function's docstring explains it maps clean_marital_status over string values while leaving non-strings unchanged (refer to clean_marital_status and clean_marital_status_series).models/br_tse_eleicoes/code/python/sub/vacancies.py (1)
34-35: Add Google-style docstrings and an explicit-> Noneon the entrypoint.
build_all()still omits a return annotation, and both function docstrings are the short one-line form. Tightening these signatures now will keep the generated builders consistent.As per coding guidelines,
**/*.py: Add type hints and docstrings for Python functions following Google Style.Also applies to: 123-124
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/sub/vacancies.py` around lines 34 - 35, Update the two functions to use Google-style docstrings and add an explicit return annotation for the entrypoint: change build_all() to build_all() -> None and replace the short one-line docstrings in both build_vagas(ano: int) -> pd.DataFrame and build_all() -> None with full Google-style docstrings that include Args (for ano), Returns (for build_vagas), and a short Description; keep existing type hints (pd.DataFrame and int) and ensure the wording matches the project's docstring conventions.models/br_tse_eleicoes/code/python/sub/voter_profile_polling_place.py (1)
14-15: Add explicit-> Noneand Google-style docstrings to the builders.
build_all()is still unannotated, and the current one-line docstrings do not match the repo’s documented Python style. This module is part of the public ETL surface, so it is worth keeping it aligned.As per coding guidelines,
**/*.py: Add type hints and docstrings for Python functions following Google Style.Also applies to: 125-126
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/sub/voter_profile_polling_place.py` around lines 14 - 15, Update the builder functions to include explicit return type annotations and Google-style docstrings: add "-> None" to build_perfil_local_votacao and to build_all (and any other builder functions in this module), and replace the current one-line triple-quoted strings with full Google-style docstrings describing Args, Returns (use "None" for these builders), and a short description; ensure function signature type hints remain (e.g., ano: int) and that the docstrings follow the repo's Python style guidelines.models/br_tse_eleicoes/code/python/sub/voter_profile_mun_zone.py (1)
12-13: Add explicit-> Noneand Google-style docstrings to the builders.
build_all()is still unannotated, and both docstrings are the short one-line form. This module is likely to become a template for other builders, so it is worth aligning it with the repo’s Python conventions now.As per coding guidelines,
**/*.py: Add type hints and docstrings for Python functions following Google Style.Also applies to: 136-137
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/sub/voter_profile_mun_zone.py` around lines 12 - 13, Update the builder functions to include explicit return type hints and Google-style docstrings: change the signature of build_perfil_mun_zona to include an explicit return type (-> pd.DataFrame if it returns a DataFrame, or -> None if it only performs side effects) and convert its one-line docstring to a full Google-style docstring (Args, Returns, Raises as applicable); do the same for build_all (add explicit -> None if it returns nothing and a Google-style docstring). Locate the functions by name (build_perfil_mun_zona, build_all) and ensure imports/types remain correct when adding annotations and expanding the docstrings.models/br_tse_eleicoes/code/python/build.py (1)
56-56: Type the CLI callables and document the entrypoints.
_run_step()leavesfuncuntyped, and the step runners plusmain()still omit explicit-> Noneand Google-style docstrings. This is the main orchestration surface, so making the signatures self-describing is worth it.As per coding guidelines,
**/*.py: Add type hints and docstrings for Python functions following Google Style.Also applies to: 66-66, 72-72, 78-79, 98-98
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/build.py` at line 56, Add explicit typing and Google-style docstrings for the CLI entrypoints: annotate _run_step(name: str, func) to accept func: Callable[[], None] (or Callable[..., None] if it may take args) and return -> None, and do the same for each step runner function and main() (e.g., add -> None). Import Callable from typing and update signatures for the functions referenced in the review (the step runner functions and main), then add brief Google-style docstrings to each function describing purpose, Args, and Returns. Ensure the annotations match actual usage (adjust Callable[...] if the function takes args) and keep docstrings concise and informative.models/br_tse_eleicoes/code/python/sub/results_section.py (1)
208-226: Collapselegendato one row per merge key before joining.
nominaisis already grouped, butlegendais merged raw. Groupinglegendatoo makes the outer join one-row-per-key and avoids accidental fan-out if the raw data contains duplicate legend rows.♻️ Suggested refactor
legenda = legenda.rename( columns={ "numero_votavel": "numero_partido", "votos": "votos_legenda", } ) - leg_cols = [c for c in group_cols if c in legenda.columns] + [ - "votos_legenda" - ] - legenda = legenda[leg_cols] + leg_group = [c for c in group_cols if c in legenda.columns] + legenda = legenda[leg_group + ["votos_legenda"]] + legenda = legenda.groupby( + leg_group, as_index=False, dropna=False + )["votos_legenda"].sum() # merge nominais + legenda — use all shared group columns as keys merge_keys = [c for c in available_group if c in legenda.columns]🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/sub/results_section.py` around lines 208 - 226, legenda is not aggregated before merging with nom_agg, which can cause fan-out when duplicate legend rows exist; collapse legenda to one row per merge key by grouping on the same keys used for the merge (use merge_keys or the intersection of available_group/group_cols present in legenda) and summing or aggregating "votos_legenda" (and preserving any other needed columns) before performing partido = nom_agg.merge(legenda, on=merge_keys, how="outer"); ensure the grouped dataframe columns match leg_cols so the outer join remains one-row-per-key.models/br_tse_eleicoes/code/python/normalization_partition.py (1)
488-938: Add explicit-> Noneannotations on the partition/orchestration helpers.These functions are pure orchestration and already have short docstrings, but the new
_partition_*helpers andbuild_allare still unannotated. As per coding guidelines,**/*.py: Add type hints and docstrings for Python functions following Google Style.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/normalization_partition.py` around lines 488 - 938, The partition/orchestration helpers lack explicit return type annotations; update each helper (e.g., _partition_candidatos, _partition_partidos, _partition_simple, _partition_results_mun_zone, _partition_results_section, _partition_bens, _partition_finance, _partition_vagas, and build_all) to include a -> None return annotation on their signatures and keep their existing docstrings intact; ensure the function signatures are the only change (no logic modifications) so type checkers recognize these as returning None.models/br_tse_eleicoes/code/python/aggregation.py (1)
16-36: Finish typing the aggregation helpers.The new aggregation entrypoints all return nothing, but they are still missing explicit
-> Noneannotations, and_read_parquetis undocumented. As per coding guidelines,**/*.py: Add type hints and docstrings for Python functions following Google Style.Also applies to: 56-506
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/aggregation.py` around lines 16 - 36, Add explicit return type annotations and docstrings: update all aggregation entrypoint functions that currently return nothing to have an explicit "-> None" return type and add a one-line Google-style docstring describing their purpose and side effects; specifically, add a docstring and a return type annotation "-> pd.DataFrame" for the helper _read_parquet(name: str, ano: int) to match _read_partitioned_csv (which already has a docstring), and ensure all helper functions include concise Google-style docstrings and precise type hints (e.g., _read_parquet: -> pd.DataFrame; aggregation entrypoints: -> None) so they comply with the project's Python typing and documentation guidelines.models/br_tse_eleicoes/code/python/sub/campaign_finance.py (1)
163-1983: Document the year-specific loaders and annotate the entrypoint.The
_build_receitas_*and_build_despesas_*helpers are where the raw layouts diverge, but most of them still have no function docs, andbuild_allhas no explicit-> None. As per coding guidelines,**/*.py: Add type hints and docstrings for Python functions following Google Style.Also applies to: 2005-2033
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/python/sub/campaign_finance.py` around lines 163 - 1983, The review asks to add missing docstrings and explicit type annotations: add Google-style docstrings to each year-specific loader (e.g., _build_receitas_2002, _build_receitas_2004, ..., _build_receitas_2018_plus and _build_despesas_2002, _build_despesas_2004, ..., _build_despesas_2018_plus) describing purpose, parameters (none) and return type, and ensure all functions have explicit return type hints (they already return pd.DataFrame) and any helper signatures include types where missing; also annotate the public entrypoints build_receitas and build_despesas with -> pd.DataFrame and add/annotate build_all (the module entrypoint) with -> None plus a docstring explaining behavior; keep docstrings concise Google-style and update any untyped parameters/returns across these functions to satisfy the guideline.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 7d77f0f4-9174-4cea-b2a9-5c8e56670e45
📒 Files selected for processing (30)
models/br_tse_eleicoes/code/PLAN.mdmodels/br_tse_eleicoes/code/[dbt]br_tse_eleicoes.ipynbmodels/br_tse_eleicoes/code/python/aggregation.pymodels/br_tse_eleicoes/code/python/build.pymodels/br_tse_eleicoes/code/python/config.pymodels/br_tse_eleicoes/code/python/normalization_partition.pymodels/br_tse_eleicoes/code/python/sub/__init__.pymodels/br_tse_eleicoes/code/python/sub/campaign_finance.pymodels/br_tse_eleicoes/code/python/sub/candidates.pymodels/br_tse_eleicoes/code/python/sub/parties.pymodels/br_tse_eleicoes/code/python/sub/results_mun_zone.pymodels/br_tse_eleicoes/code/python/sub/results_section.pymodels/br_tse_eleicoes/code/python/sub/results_state.pymodels/br_tse_eleicoes/code/python/sub/vacancies.pymodels/br_tse_eleicoes/code/python/sub/voter_profile_mun_zone.pymodels/br_tse_eleicoes/code/python/sub/voter_profile_polling_place.pymodels/br_tse_eleicoes/code/python/sub/voter_profile_section.pymodels/br_tse_eleicoes/code/python/sub/voting_details_mun_zone.pymodels/br_tse_eleicoes/code/python/sub/voting_details_section.pymodels/br_tse_eleicoes/code/python/sub/voting_details_state.pymodels/br_tse_eleicoes/code/python/utils/__init__.pymodels/br_tse_eleicoes/code/python/utils/clean_education.pymodels/br_tse_eleicoes/code/python/utils/clean_election_type.pymodels/br_tse_eleicoes/code/python/utils/clean_marital_status.pymodels/br_tse_eleicoes/code/python/utils/clean_party.pymodels/br_tse_eleicoes/code/python/utils/clean_result.pymodels/br_tse_eleicoes/code/python/utils/clean_string.pymodels/br_tse_eleicoes/code/python/utils/fix_candidate.pymodels/br_tse_eleicoes/code/python/utils/helpers.pymodels/br_tse_eleicoes/code/python/validate.py
💤 Files with no reviewable changes (1)
- models/br_tse_eleicoes/code/[dbt]br_tse_eleicoes.ipynb
| 2002: ["AC", "AL", "AM", "AP", "BA", "BR", "CE", "DF", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2004: ["AC", "AL", "AM", "AP", "BA", "CE", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2006: ["AC", "AL", "AM", "AP", "BA", "BR", "CE", "DF", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2008: ["AC", "AL", "AM", "AP", "BA", "CE", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2010: ["AC", "AL", "AM", "AP", "BA", "BR", "CE", "DF", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2012: ["AC", "AL", "AM", "AP", "BA", "CE", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2014: ["AC", "AL", "AM", "AP", "BA", "BR", "CE", "DF", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2016: ["AC", "AL", "AM", "AP", "BA", "CE", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2018: ["AC", "AL", "AM", "AP", "BA", "BR", "CE", "DF", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2020: ["AC", "AL", "AM", "AP", "BA", "CE", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2022: ["AC", "AL", "AM", "AP", "BA", "BR", "CE", "DF", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], |
There was a problem hiding this comment.
Handle BR in the Brazil-only filter.
The mapping switches to "BR" for national files from 2002 onward. Because this branch only checks "BRASIL", those years bypass the presidential/DF restriction and duplicate rows from the per-UF inputs.
🔧 Suggested fix
- if uf == "BRASIL":
+ if uf in {"BR", "BRASIL"}:
df = df[(df["cargo"] == "PRESIDENTE") | (df["sigla_uf"] == "DF")]Also applies to: 248-250
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_tse_eleicoes/code/python/sub/voting_details_mun_zone.py` around
lines 19 - 29, The Brazil-only filter incorrectly only checks for the string
"BRASIL" while the year->UF mapping includes the "BR" code for national files
(e.g., entries in the mapping for years 2002..2022), causing those rows to
bypass the presidential/DF restriction and create duplicates; update the filter
logic used in the relevant function(s) that process per-UF vs national files
(the branch that checks "BRASIL") to also detect the "BR" UF code (and any
equivalent national token) and apply the same presidential/DF restriction for
those records; ensure the same fix is applied to the analogous check around
lines 248-250.
| 2002: ["AC", "AL", "AM", "AP", "BA", "BR", "CE", "DF", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2004: ["AC", "AL", "AM", "AP", "BA", "CE", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2006: ["AC", "AL", "AM", "AP", "BA", "BR", "CE", "DF", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2008: ["AC", "AL", "AM", "AP", "BA", "CE", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2010: ["AC", "AL", "AM", "AP", "BA", "BR", "CE", "DF", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2012: ["AC", "AL", "AM", "AP", "BA", "CE", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2014: ["AC", "AL", "AM", "AP", "BA", "BR", "CE", "DF", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2016: ["AC", "AL", "AM", "AP", "BA", "CE", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2018: ["AC", "AL", "AM", "AP", "BA", "BR", "CE", "DF", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2020: ["AC", "AL", "AM", "AP", "BA", "CE", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], | ||
| 2022: ["AC", "AL", "AM", "AP", "BA", "BR", "CE", "DF", "ES", "GO", "MA", "MG", "MS", "MT", "PA", "PB", "PE", "PI", "PR", "RJ", "RN", "RO", "RR", "RS", "SC", "SE", "SP", "TO"], |
There was a problem hiding this comment.
Treat BR as a national file too.
The year map switches to "BR" from 2002 onward, so this branch never runs for those years. Those national files then keep all cargos instead of only the presidential/DF slice and duplicate rows already loaded from the UF files.
🔧 Suggested fix
- if uf == "BRASIL":
+ if uf in {"BR", "BRASIL"}:
df = df[(df["cargo"] == "PRESIDENTE") | (df["sigla_uf"] == "DF")]Also applies to: 123-125
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_tse_eleicoes/code/python/sub/voting_details_section.py` around
lines 18 - 28, The code that decides whether a file is national currently misses
files labeled "BR" so the national-branch never runs for years where YEAR_UF_MAP
includes "BR"; update the national-file detection to treat uf == "BR" as a
national file (in the function/method that branches on UF, e.g.,
process_voting_details_section / _is_national_file or the code path that slices
to presidential/DF), and ensure the branch for national files applies the
presidential/DF slice and avoids reloading UF rows; also apply the same fix to
the other occurrence referenced around the block at lines 123-125.
| if df is None: | ||
| print( | ||
| f" WARNING: no file found for detalhes_votacao_uf {ano} {uf}" | ||
| ) | ||
| continue |
There was a problem hiding this comment.
Fail the year build when an expected raw file is missing.
continue here writes a partial parquet instead of surfacing incomplete raw storage. For a parity ETL, that is silent data loss.
🔧 Suggested fix
- if df is None:
- print(
- f" WARNING: no file found for detalhes_votacao_uf {ano} {uf}"
- )
- continue
+ if df is None:
+ msg = (
+ f"No file found for detalhes_votacao_uf "
+ f"{ano} {uf}"
+ )
+ raise FileNotFoundError(msg)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if df is None: | |
| print( | |
| f" WARNING: no file found for detalhes_votacao_uf {ano} {uf}" | |
| ) | |
| continue | |
| if df is None: | |
| msg = ( | |
| f"No file found for detalhes_votacao_uf " | |
| f"{ano} {uf}" | |
| ) | |
| raise FileNotFoundError(msg) |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_tse_eleicoes/code/python/sub/voting_details_state.py` around lines
74 - 78, The current block in voting_details_state.py that checks "if df is
None" (for detalhes_votacao_uf with ano and uf) must not silently continue
because that writes partial parquet; instead, stop the build by replacing the
print+continue with a hard failure: raise a descriptive exception (e.g.,
RuntimeError including ano, uf and the missing-file context) or call sys.exit(1)
so the year build fails immediately and surfaces the missing raw file error;
remove the continue and ensure any outer code will not write incomplete output
when df is missing.
| for keys, group in df.groupby(partition_cols, sort=True): | ||
| if isinstance(keys, str): | ||
| keys = (keys,) | ||
|
|
||
| parts = "/".join( | ||
| f"{col}={val}" | ||
| for col, val in zip(partition_cols, keys, strict=True) | ||
| ) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
cat -n models/br_tse_eleicoes/code/python/utils/helpers.py | sed -n '250,280p'Repository: basedosdados/pipelines
Length of output: 1081
🏁 Script executed:
cat -n models/br_tse_eleicoes/code/python/utils/helpers.py | sed -n '240,290p'Repository: basedosdados/pipelines
Length of output: 1360
🏁 Script executed:
python3 << 'EOF'
import pandas as pd
import numpy as np
# Test pandas groupby with single column (non-string type)
df = pd.DataFrame({
'ano': [2020, 2021, 2020, 2021],
'value': [1, 2, 3, 4]
})
print("Test 1: Single numeric column groupby")
for keys, group in df.groupby(['ano'], sort=True):
print(f"Keys type: {type(keys)}, Keys value: {keys}, Is tuple: {isinstance(keys, tuple)}")
print("\nTest 2: Single string column groupby")
df2 = pd.DataFrame({
'estado': ['SP', 'MG', 'SP', 'MG'],
'value': [1, 2, 3, 4]
})
for keys, group in df2.groupby(['estado'], sort=True):
print(f"Keys type: {type(keys)}, Keys value: {keys}, Is tuple: {isinstance(keys, tuple)}")
print("\nTest 3: Multiple column groupby")
df3 = pd.DataFrame({
'ano': [2020, 2021, 2020, 2021],
'estado': ['SP', 'MG', 'SP', 'MG'],
'value': [1, 2, 3, 4]
})
for keys, group in df3.groupby(['ano', 'estado'], sort=True):
print(f"Keys type: {type(keys)}, Keys value: {keys}, Is tuple: {isinstance(keys, tuple)}")
EOFRepository: basedosdados/pipelines
Length of output: 184
🏁 Script executed:
rg -l "save_partitioned" --type pyRepository: basedosdados/pipelines
Length of output: 179
🏁 Script executed:
rg "save_partitioned\(" --type py -B 2 -A 5Repository: basedosdados/pipelines
Length of output: 9469
Single-column numeric partitions break save_partitioned().
For groupby(partition_cols) with one partition column, pandas returns a scalar key. The current normalization only wraps strings, so cases like partition_cols=["ano"] (where "ano" is numeric) reach zip(..., strict=True) with a non-iterable scalar, causing a TypeError.
🔧 Suggested fix
- if isinstance(keys, str):
+ if not isinstance(keys, tuple):
keys = (keys,)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| for keys, group in df.groupby(partition_cols, sort=True): | |
| if isinstance(keys, str): | |
| keys = (keys,) | |
| parts = "/".join( | |
| f"{col}={val}" | |
| for col, val in zip(partition_cols, keys, strict=True) | |
| ) | |
| for keys, group in df.groupby(partition_cols, sort=True): | |
| if not isinstance(keys, tuple): | |
| keys = (keys,) | |
| parts = "/".join( | |
| f"{col}={val}" | |
| for col, val in zip(partition_cols, keys, strict=True) | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_tse_eleicoes/code/python/utils/helpers.py` around lines 262 - 269,
The group key normalization only handles string keys, so numeric or other scalar
keys from df.groupby(partition_cols) (e.g., partition_cols=["ano"]) remain
non-iterable and break zip(..., strict=True); update the normalization in the
save_partitioned helper (around the df.groupby loop handling keys and parts) to
wrap any non-tuple key into a tuple (e.g., if not isinstance(keys, tuple): keys
= (keys,)) so zip receives an iterable of values corresponding to
partition_cols.
|
Tick the box to add this pull request to the merge queue (same as
|
…ão Prefect 3 Track A — corrige o parsing posicional (vN) que dessincronizou quando o TSE reinseriu colunas. Novo utils/layout.py (resolve_columns lê os layouts oficiais do harness); read_raw_csv aceita family/ano. Builders convertidos para seleção por nome oficial TSE: candidates, results_mun_zone, voting_details_mun_zone, parties, vacancies, voter_profile_mun_zone, campaign_finance (despesas 2014). Harness tier3: FAIL 62 -> 0. Track B — 3 rollups municipio reescritos como ref()+GROUP BY in-warehouse (detalhes recomputa as 4 proporcoes via safe_divide). Track D — pipelines/br_tse_eleicoes/ (Prefect 3): 1 flow/deployment por tabela + orquestrador async br_tse_eleicoes__refresh (run_deployment + asyncio.gather nas waves de DEPENDENCIES.md). So orquestracao; invoca os builders de models/code/python. Docs: decisoes_migracao.md, ARQUITETURA.md, DIAGNOSIS.md regenerado.
… harness Abandon the header-based parsing rewrite, dbt rollups and orphan Prefect 3 orchestration. Restore the original positional-column builders and the three municipio dbt models to their aafb007 state so the existing code can be bug-fixed in place, guided by the diagnostics harness (kept intact).
|
Tick the box to add this pull request to the merge queue (same as
|
|
Tick the box to add this pull request to the merge queue (same as
|
|
Posso ajudar com algo nesse PR? |
2 participants
Descrição
PR para refatorar o código de limpeza dos dados de eleições de Stata para Python. O código foi escrito com auxílio do Claude Code.
Substitui o PR #1467.
Detalhes Técnicos:
/inpute tudo já gerado pelo Stata em/output. Fomos escrevendo o código para gerar precisamente o mesmo output a partir do código refatorado em Python. Rodamos diversos testes de validação (ex: ordem das colunas, número de linhas, médias, valores únicos)..parquet, que são depois normalizados e particionados em.csv.Requerimentos
/inputdisponíveis. Hoje eu mantenho tudo no meu Dropbox local.Summary by CodeRabbit