We are using osmium renumber to process europe-latest.osm.pbf from Geofabrik.
While it significantly improves processing speed, it also requires substantially more memory than any other step in our pipeline. In our case, renumber is the dominant memory consumer, forcing us to provision significantly more RAM than needed for the rest of the workflow.
Would it be possible to provide an option for a disk-based index? This could mean storing the ID mapping on disk instead of RAM or using a configurable backend (memory vs disk).
We understand that this would likely come with a performance penalty, but it would make it possible to process large datasets on machines with limited memory.
We are using
osmium renumberto process europe-latest.osm.pbf from Geofabrik.While it significantly improves processing speed, it also requires substantially more memory than any other step in our pipeline. In our case,
renumberis the dominant memory consumer, forcing us to provision significantly more RAM than needed for the rest of the workflow.Would it be possible to provide an option for a disk-based index? This could mean storing the ID mapping on disk instead of RAM or using a configurable backend (memory vs disk).
We understand that this would likely come with a performance penalty, but it would make it possible to process large datasets on machines with limited memory.