fix: pmtiles generation fails with error: route_id

[davidgamez]

Summary:

Fixes #1358

This PR adds a detect_encoding function to be used by the file loaders. This makes sure the file and columns are properly loaded.
In Python, opening a UTF-8 file with encoding="utf-8" doesn’t remove a BOM if present, so the first string may contain a hidden \ufeff. Using encoding="utf-8-sig" strips the BOM automatically and avoids this issue.

A BOM (Byte Order Mark) is a special hidden character at the start of some text files that indicates the file’s encoding. In UTF-8 it isn’t needed, but some editors add it anyway.

Expected behavior:

CSV files with BOOM characters are properly read.

Testing tips:
Use the verifier with feed id mdb-733 and dataset id mdb-733-202509111637.

Please make sure these boxes are checked before submitting your pull request - thanks!

• Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
• Add or update any needed documentation to the repo
• Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
• Linked all relevant issues
• Include screenshot(s) showing how this pull request works and fixes the issue(s)

[Copilot]

Pull Request Overview

This PR fixes pmtiles generation failures related to route_id errors by implementing dynamic encoding detection for CSV files. The changes address issues where CSV files with different character encodings were causing parsing failures.

• Adds encoding detection functionality using charset_normalizer library
• Updates CSV reading operations to use detected encoding instead of hardcoded UTF-8
• Includes comprehensive test coverage for the new encoding detection feature

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
functions-python/helpers/utils.py	Adds detect_encoding function to dynamically detect file encoding
functions-python/pmtiles_builder/src/main.py	Updates shapes file reading to use detected encoding
functions-python/pmtiles_builder/src/csv_cache.py	Updates CSV reading to use detected encoding
functions-python/helpers/tests/test_helpers.py	Adds unit tests for encoding detection functionality
functions-python/pmtiles_builder/requirements.txt	Adds charset_normalizer dependency
functions-python/helpers/requirements.txt	Adds charset_normalizer dependency
functions-python/pmtiles_builder/src/scripts/pmtiles_builder_verifier.py	Adds new test dataset and updates feed index

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[jcpitre]

LGTM!