This project is used to download the CMOL data from transnet.bw https://www.transnetbw.de/en/energy-market/ancillary-services/cmol to load it into a database (SQLite)

The data gets downloaded every 10 minutes and new entries are added into the database while existing data gets ignored

In the case of conflicting IDs in the Market Documents / Market Agreements, the ones with the most recent creation date get saved while the older data is removed

Packages/Modules

• data scraping: contains the web scraper and the xml processing
• database: contains all code used to create and access the database
• input_data: temporary storage for the downloaded import data, gets cleared after the data was processed
• tests: contains test files for each module
• data_viewer.ipynd - Notebook used for the primary data analysis - NOT UP TO DATE
• main.py: main file that creates the database, loads the past data (range defined in config.py) and contains the loop that downloads the data every 10 minutes

Database

• SQLite Database -> easy to use in Python
• In real world application I would probably switch to SQLServer / T-SQL
• Contains 3 tables
• The names of the columns/tables were kept as close as possible to the original xml names
• only "." were replaced with _

• • market_documents: All the information from the MeritOrderList_MarketDocument level

    • Decision: Only one document per mRID, if a second document with the same ID gets loaded, the newer one of them is kept, as the mRID should be unique
    • (according to: https://eepublicdownloads.entsoe.eu/clean-documents/EDI/Library/ERRP/Common_Platform_for_Replacement_Reserves_implementation_guide_v1.3.pdf)

  • timeseries: All the information from the TimeSeries level

    • Same as with market_documents: market_Agreement should be unique (in combination with market document mRID -> foreign key)
    • • Foreign Key: market_document.mRID
      • Primary Key: mRID, marketAgreement_mRID

  • point: All information from the Period/point level

    • In the analysis only one point per TimeSeries was found, but theoretically multiple should be possible -> therefore its own table
    • Foreign Keys: timeseries.mRID and timeeries.marketAgreement_mRID
    • Primary Key: market_document_mRID, marketAgreement_mRID, position

Tests

• Some of the tests might fail in the first run as the SQLite database is too slow to allow for multi-tasking loads and does not allow for operations on a locked database
• In a real world production case a database without these lock-constraints / better constraint handling would be used

Decisions

• Web Scraping every 10 minutes -> requirement is 15 min max, leaving some processing time
• Allow for past data download -> Range can be set in config.py
• SQLite database -> explained in the database section

AI Used for:

• XML File Debugging
• Brainstorming for Web Scraping
• Creating the tests in the test package