Further optimize the pmtiles creation algorithm

[jcpitre]

In #1305 we created the code to generate pmtiles.
It has already been optimized so it can run with 8GB for one of the biggest feed (mdb-1078).
We can further optimize it by using numpy for arrays of coordinates.
This should allow us to not create an index for shapes.txt while still using a similar amount or memory.
And it should make it faster.

[jcpitre]

Optimization of the shapes entries. We use mdb-1078 as an example.

There's a bit over 37 million entries in the shapes files, each with coordinates (long, lat).
There are multiple entries for a given shaped id. There are 23984 distinct shapes id.
The average length of a shape_id is 43 characters.

Currently we have:

• a dict with 23984 entries. Keys are the shape_ids

  • Average 43 bytes per string
  • Python string overhead (approx): ~49 bytes per string (CPython, not including interning or dict overhead)
  • Total for keys: 23,984 × (43 + 49) ≈ 2.2 MB (negligible)

• Each value for an entry of the dict is a python list of integers.

  • On 64-bit CPython, each integer typically uses 28 bytes (not including list overhead).
  • Total int values:
    • 37,000,000 values × 28 bytes ≈ 1,036 MB (about 1 GB)
  • List overhead:
    • Each list object has overhead (~72 bytes per list) plus 8 bytes per reference in the list:
    • 23,984 lists × 72 bytes ≈ 1.7 MB (negligible)
    • 37,000,000 references × 8 bytes ≈ 296 MB

• So a total of 1.036 + 296 MB = 1332 MB

If we keep the dict, but replace the list of integer by 23,984 numpy arrays:

• It has to be an int64 array, since the position in the file can go over an int32 value.

  • 37,000,000 int64 values
  • Each int64: 8 bytes
    • Total for arrays: 37,000,000 × 8 = 296 MB

• So a total of 296 MB (string keys and dict negligible) instead of 1332 MB

[jcpitre]

Added an issue for more optimizations: #1383