📊 Statistics about HTTP response security headers usage mentioned by the OWASP Secure Headers Project (OSHP).
💾 This project gather data, about the usage of HTTP response security headers, into a SQLITE database to allow the generation of statistics in a second time.
💡 See this issue for details.
Tip
💡 MAJESTIC was used instead of the CISCO Top 1 million sites CSV file because it contain less malware domains.
The gather_data.py script will automatically download and parse the Majestic CSV if data/input.csv is not present.
Note
📦 They are all stored in the scripts folder and they are Python 3.x based.
Important
💻 Visual Studio Code is used for the scripts development. A Visual Studio Code workspace file is provided for the project with recommended extensions.
📑 Files:
- gather_data: Script gathering the information about HTTP security headers usage in a SQLITE database based on the "MAJESTIC Top 1 million sites CSV file" data source.
- generate_stats_md_file: Script using the gathered data to generate/update the markdown file stats, with mermaid pie charts with differents statistics about HTTP security headers usage (
⚠️ not used anymore).
Note
📦 During execution, these files are generated and stored locally in the data folder. For distribution, they are published as GitHub Release assets attached to this repository.
📑 Files:
input.csv: MAJESTIC Top 1 million sites list formated as one entryranking,domainby line.data.db: SQLITE database with information about HTTP security headers usage.
Note
💡 Only the first 250000 entries of the CSV datasource are used to fit the processing timeframe allowed for a github action workfows using the free tiers.
💻 The update is scheduled in the following way:
- The first day of every month the database is updated via this workflow.
- Once the data database is updated then the statistic data is updated via this workflow.
- DB Browser for SQLite can be used to access to the raw data of the SQLITE DB.