Skip to content

Update readme for mdbx #110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bovine3dom wants to merge 2 commits into from
Closed

Update readme for mdbx #110

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
24a9266
Update readme to point to mdbx
bovine3dom Feb 18, 2026
9da6bc2
Remove redundant command from readme
bovine3dom Feb 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 12 additions & 13 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ Install `uv` as package management tool.
For example via `pip install uv`.

```sh
uv venv

@skinkie skinkie Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this deleted if it is only a README update.

@bovine3dom bovine3dom Jun 22, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need it, at least not on my machine. (And IIRC uv venv doesn't play nicely with non-POSIX shells, but it's been long enough that that could have changed).

@skinkie skinkie Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a recommended practice, and if you ask Google it becomes highly recommended.

https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/#

@skinkie skinkie Jun 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert this change.

Comment thread
skinkie marked this conversation as resolved.
uv sync
sh scripts/setup.sh
```
Expand All @@ -39,8 +38,8 @@ As such, the processing has an audit trail.

Our (de)serialisation takes place using [xsData](https://xsdata.readthedocs.io/en/latest/), via Python Data Classes.
This guarantees us XML Schema compliance.
Our intermediate presentation is serialised using [cloudpickle](https://github.com/cloudpipe/cloudpickle) and is stored in compressed [lz4](https://lmdb.readthedocs.io/en/release/) form.
During the development of this software we have evaluated various database technologies, for our intermediate computing requirements [lmdb](https://lmdb.readthedocs.io/en/release/) is used.
Our intermediate presentation is serialised using [cloudpickle](https://github.com/cloudpipe/cloudpickle) and is stored in compressed [lz4](https://lz4.github.io/lz4/) form.
During the development of this software we have evaluated various database technologies, for our intermediate computing requirements [mdbx](https://github.com/erthink/libmdbx) is used.
The processing of other CSV-based formats such as GTFS is mediated via [DuckDB](https://duckdb.org/docs/stable/clients/python/overview.html).

### Extract, Transform, Load
Expand All @@ -61,25 +60,25 @@ In the case a different format is required, our intermediate NeTEx presentation

#### NeTEx EPIP example
```sh
uv run python -m conv.netex_to_db path_to_xml.gz path_to_input_netex.lmdb
uv run python -m conv.netex_db_to_generalframe path_to_input_netex.lmdb path_to_intermediate_presentation.xml.gz
uv run python -m conv.epip_db_to_db path_to_input_netex.lmdb path_to_output_epip.lmdb
uv run python -m conv.epip_db_to_xml path_to_output_epip.lmdb path_to_output_epip.xml.gz
uv run python -m conv.netex_to_db path_to_xml.gz path_to_input_netex.mdbx
uv run python -m conv.netex_db_to_generalframe path_to_input_netex.mdbx path_to_intermediate_presentation.xml.gz
uv run python -m conv.epip_db_to_db path_to_input_netex.mdbx path_to_output_epip.mdbx
uv run python -m conv.epip_db_to_xml path_to_output_epip.mdbx path_to_output_epip.xml.gz
```

#### GTFS to NeTEx EPIP example
```sh
uv run python -m conv.gtfs_import_to_db path_to_gtfs.zip path_to_gtfs.duckdb
uv run python -m conv.gtfs_convert_to_db path_to_gtfs.duckdb path_to_intermediate.lmdb
uv run python -m conv.epip_db_to_db path_to_intermediate.lmdb path_to_output_epip.lmdb
uv run python -m conv.epip_db_to_xml path_to_output_epip.lmdb path_to_output_epip.xml.gz
uv run python -m conv.gtfs_convert_to_db path_to_gtfs.duckdb path_to_intermediate.mdbx
uv run python -m conv.epip_db_to_db path_to_intermediate.mdbx path_to_output_epip.mdbx
uv run python -m conv.epip_db_to_xml path_to_output_epip.mdbx path_to_output_epip.xml.gz
```

#### NeTEx EPIP to GTFS example
```sh
uv run python -m conv.netex_to_db path_to_xml.gz path_to_input_netex.lmdb
uv run python -m conv.gtfs_db_to_db path_to_input_netex.lmdb path_to_output_gtfs.lmdb
uv run python -m conv.gtfs_db_to_gtfs path_to_output_gtfs.lmdb path_to_output_gtfs.zip
uv run python -m conv.netex_to_db path_to_xml.gz path_to_input_netex.mdbx
uv run python -m conv.gtfs_db_to_db path_to_input_netex.mdbx path_to_output_gtfs.mdbx
uv run python -m conv.gtfs_db_to_gtfs path_to_output_gtfs.mdbx path_to_output_gtfs.zip
```


Expand Down
14 changes: 7 additions & 7 deletions tools/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,12 +83,12 @@ The following example shows the example of a script file:
"scripts": [
{"step":1,"script": "clean_tmp", "args": "%%dir%%"},
{"step":2,"script": "download_input_file", "args": "%%dir%%"},
{"step":3,"script": "conv.netex_to_db", "args": "[%%inputfilepath%%] %%dir%%/03.lmdb"},
{"step":4,"script": "conv.epip_db_to_db", "args": "%%dir%%/03.lmdb %%dir%%/04.lmdb"},
{"step":5,"script": "conv.epip_db_to_xml", "args": "%%dir%%/04.lmdb %%dir%%/05-epip.xml.gz"},
{"step":6,"script": "conv.netex_to_db", "args": "[%%dir%%/05-epip.xml.gz] %%dir%%/06.lmdb True"},
{"step":7,"script": "conv.gtfs_db_to_db", "args": "%%dir%%/06.lmdb %%dir%%/07.lmdb"},
{"step":8,"script": "conv.gtfs_db_to_gtfs", "args": "%%dir%%/07.lmdb %%dir%%/08-%%block%%-gtfs.zip"},
{"step":3,"script": "conv.netex_to_db", "args": "[%%inputfilepath%%] %%dir%%/03.mdbx"},
{"step":4,"script": "conv.epip_db_to_db", "args": "%%dir%%/03.mdbx %%dir%%/04.mdbx"},
{"step":5,"script": "conv.epip_db_to_xml", "args": "%%dir%%/04.mdbx %%dir%%/05-epip.xml.gz"},
{"step":6,"script": "conv.netex_to_db", "args": "[%%dir%%/05-epip.xml.gz] %%dir%%/06.mdbx True"},
{"step":7,"script": "conv.gtfs_db_to_db", "args": "%%dir%%/06.mdbx %%dir%%/07.mdbx"},
{"step":8,"script": "conv.gtfs_db_to_gtfs", "args": "%%dir%%/07.mdbx %%dir%%/08-%%block%%-gtfs.zip"},
{"step":9,"script": "tools.gtfs_validator", "args": "%%dir%%/08-%%block%%-gtfs.zip %%dir%%"},
{"step":10,"script": "#tools.gtfs_map_visualisation", "args": "%%dir%%/08-%%block%%-gtfs.zip %%dir%%/10-%%block%%-map.html 1"}
]
Expand All @@ -105,7 +105,7 @@ All conversions and tools can be used as `script` with the arguments used as def

### Internal commands of the script runner
Some commands are directly implemented in the script runner
* `clean_tmp`: With the given directory all temporary processing files are deleted (usually duckdb and lmdb)
* `clean_tmp`: With the given directory all temporary processing files are deleted (usually duckdb and mdbx)
* `download_input_file`: The file specified by `download_url` in the block is stored in the directory indicated.
* `set_defaults`: Superseeds defaults set in the configuraiton file. See the configuration file for allowed elements.
* `process_url_list`: see separate section
Expand Down