Skip to content

Commit f99fe46

Browse files
RasmusOrsoeRasmusOrsoe
authored
Merge branch 'main' into parquet_to_sqlite_converter
2 parents 6c3e730 + 9808766 commit f99fe46

73 files changed

Lines changed: 2240 additions & 2238 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/docker.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ env:
1212

1313
jobs:
1414
docker:
15-
if: github.repository == 'icecube/graphnet'
15+
if: github.repository == 'graphnet-team/graphnet'
1616
runs-on: ubuntu-latest
1717
permissions:
1818
packages: write

‎.github/workflows/paper.yml‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# This is a simple action to generate paper draft PDFs.
2+
name: paper
3+
4+
on:
5+
push:
6+
branches: [ joss-paper-action ]
7+
8+
# Allows you to run this workflow manually from the Actions tab
9+
workflow_dispatch:
10+
11+
jobs:
12+
paper-2022-joss:
13+
runs-on: ubuntu-latest
14+
name: paper-draft-pdf
15+
steps:
16+
- name: Checkout
17+
uses: actions/checkout@v2
18+
- name: Build draft PDF
19+
uses: openjournals/openjournals-draft-action@master
20+
with:
21+
journal: joss
22+
# This should be the path to the paper within your repo.
23+
paper-path: paper/paper.md
24+
- name: Upload
25+
uses: actions/upload-artifact@v1
26+
with:
27+
name: paper
28+
# This is the output path where Pandoc will write the compiled
29+
# PDF. Note, this should be the same directory as the input
30+
# paper.md
31+
path: paper/paper.pdf

‎.gitignore‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -134,11 +134,18 @@ docs/**/*.rst
134134

135135
# Badges
136136
assets/badges
137+
138+
# Logs
137139
lightning_logs/
138140
logs/
139141
wandb/
142+
checkpoints/
143+
144+
# Data and outputs
140145
test_data/
141146
**.png
142147
**.pth
148+
**.db
149+
150+
# Notebooks
143151
**.ipynb
144-
**.db

‎README.md‎

Lines changed: 60 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,13 @@
33
| Usage | Development | Quality |
44
| --- | --- | --- |
55
| [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.6720188.svg)](https://doi.org/10.5281/zenodo.6720188) | [![Slack](https://img.shields.io/badge/slack-4A154B.svg?logo=slack)](https://join.slack.com/t/graphnet-team/signup) | [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
6-
| [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) | ![build](https://github.com/icecube/graphnet/actions/workflows/build-matrix.yml/badge.svg) | [![Maintainability](https://api.codeclimate.com/v1/badges/f244df0fc73c77102b47/maintainability)](https://codeclimate.com/github/asogaard/graphnet/maintainability) |
7-
| ![Supported python versions](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9%20%7C%203.10-blue) | ![build](https://github.com/icecube/graphnet/actions/workflows/build-icetray.yml/badge.svg) | [![Test Coverage](https://api.codeclimate.com/v1/badges/f244df0fc73c77102b47/test_coverage)](https://codeclimate.com/github/asogaard/graphnet/test_coverage) |
6+
| [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) | ![build](https://github.com/graphnet-team/graphnet/actions/workflows/build-matrix.yml/badge.svg) | [![Maintainability](https://api.codeclimate.com/v1/badges/b273a774112e32643162/maintainability)](https://codeclimate.com/github/graphnet-team/graphnet/maintainability) |
7+
| ![Supported python versions](https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9%20%7C%203.10-blue) | ![build](https://github.com/graphnet-team/graphnet/actions/workflows/build-icetray.yml/badge.svg) | [![Test Coverage](https://api.codeclimate.com/v1/badges/b273a774112e32643162/test_coverage)](https://codeclimate.com/github/graphnet-team/graphnet/test_coverage) |
8+
| [![Docker image](https://img.shields.io/docker/v/asogaard/graphnet?color=blue&logo=docker&sort=semver)](https://hub.docker.com/repository/docker/asogaard/graphnet) | |
9+
10+
## :rocket: About
11+
12+
**GraphNeT** is an open-source python framework aimed at providing high quality, user friendly, end-to-end functionality to perform reconstruction tasks at neutrino telescopes using graph neural networks (GNNs). GraphNeT makes it fast and easy to train complex models that can provide event reconstruction with state-of-the-art performance, for arbitrary detector configurations, with inference times that are orders of magnitude faster than traditional reconstruction techniques.
813

914
## :gear: Install
1015

@@ -77,11 +82,62 @@ depending on your setup to your `.bashrc` script or similar to make sure that th
7782
</blockquote>
7883
</details>
7984

85+
86+
## :ringed_planet: Use cases
87+
88+
Below is an incomplete list of potential use cases for GNNs in neutrino telescopes.
89+
These are categorised as either "Reconstruction challenges" that are considered common and that may benefit several experiments physics analyses; and those same "Experiments" and "Physics analyses".
90+
91+
<details>
92+
<summary><b>Reconstruction challenges</b></summary>
93+
94+
| Title | Status | People | Materials |
95+
| --- | --- | --- | --- |
96+
| Low-energy neutrino classification and reconstruction | Done | Rasmus Ørsøe | https://arxiv.org/abs/2209.03042 |
97+
| High-energy neutrino classification and reconstruction | Active | Rasmus Ørsøe | |
98+
| Pulse noise cleaning | Paused | Kaare Iversen (past) | |
99+
| (In-)elasticity reconstruction | Paused | Marc Jacquart (past) | |
100+
| Multi-class event classification | | | |
101+
| Data/MC difference mitigation | | | |
102+
| Systematic uncertainty mitigation | | | |
103+
104+
</details>
105+
106+
<details>
107+
<summary><b>Experiments</b></summary>
108+
109+
| Title | Status | People | Materials |
110+
| --- | --- | --- | --- |
111+
| IceCube | Active | (...) | |
112+
| IceCube-Upgrade | Active | (...) | |
113+
| IceCube-Gen2 | Active | (...) | |
114+
| P-ONE | | (...) | |
115+
| KM3NeT-ARCA | | (...) | |
116+
| KM3NeT-ORCA | | (...) | |
117+
118+
</details>
119+
120+
<details>
121+
<summary><b>Physics analyses</b></summary>
122+
123+
| Title | Status | People | Materials |
124+
| --- | --- | --- | --- |
125+
| Neutrino oscillations | | | |
126+
| Point source searches | | | |
127+
| Low-energy cosmic alerts | | | |
128+
| High-energy cosmic alerts | | | |
129+
| Moon pointing | | | |
130+
| Muon decay asymmetry | | | |
131+
| Spectra measurements | | | |
132+
133+
</details>
134+
135+
80136
## :handshake: Contributing
81137

82138
To make sure that the process of contributing is as smooth and effective as possible, we provide a few guidelines in the [contributing guide](CONTRIBUTING.md) that we encourage contributors to follow.
83139

84-
In short, everyone who wants to contribute to this project is more than welcome to do so! Contributions are handled through pull requests, that should be linked to a [GitHub issue](https://github.com/icecube/graphnet/issues) describing the feature to be added or bug to be fixed. Pull requests will be reviewed by the project maintainers and merged into the main branch when accepted.
140+
In short, everyone who wants to contribute to this project is more than welcome to do so! Contributions are handled through pull requests, that should be linked to a [GitHub issue](https://github.com/graphnet-team/graphnet/issues) describing the feature to be added or bug to be fixed. Pull requests will be reviewed by the project maintainers and merged into the main branch when accepted.
85141

86142

87143
## :test_tube: Experiment tracking
@@ -112,4 +168,4 @@ GraphNeT has an Apache 2.0 license, as found in the [LICENSE](LICENSE) file.
112168
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 890778.
113169

114170

115-
[^1]: Examples of this are shown in the [examples/convert_i3_to_sqlite.py](examples/convert_i3_to_sqlite.py) and [examples/convert_i3_to_parquet.py](examples/convert_i3_to_parquet.py) scripts
171+
[^1]: Examples of this are shown in the [examples/convert_i3_files.py](examples/convert_i3_files.py) script

‎examples/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
# Examples
2+
3+
This directory contain several scripts that exemplify various aspects of the `graphnet` package. These are
4+
5+
* [convert_i3_to_sqlite.py](convert_i3_to_sqlite.py): Shows how to convert I3 file(s) to a SQLite database, both for IceCube-86 and IceCube-Upgrade data.
6+
* ...

‎examples/convert_i3_files.py‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,86 @@
1+
"""Minimum working example (MWE) to use ParquetDataConverter."""
2+
3+
import logging
4+
import os
5+
6+
from graphnet.utilities.logging import get_logger
7+
8+
from graphnet.data.extractors import (
9+
I3FeatureExtractorIceCubeUpgrade,
10+
I3RetroExtractor,
11+
I3TruthExtractor,
12+
I3GenericExtractor,
13+
)
14+
from graphnet.data.parquet import ParquetDataConverter
15+
from graphnet.data.sqlite import SQLiteDataConverter
16+
17+
logger = get_logger(level=logging.INFO)
18+
19+
CONVERTER_CLASS = {
20+
"sqlite": SQLiteDataConverter,
21+
"parquet": ParquetDataConverter,
22+
}
23+
24+
25+
def main_icecube86(backend: str):
26+
"""Convert IceCube-86 I3 files to intermediate `backend` format."""
27+
# Check(s)
28+
assert backend in CONVERTER_CLASS
29+
30+
inputs = ["./test_data/"]
31+
outdir = "./temp/test_ic86"
32+
33+
converter = CONVERTER_CLASS[backend](
34+
[
35+
I3GenericExtractor(
36+
keys=[
37+
"SRTInIcePulses",
38+
"I3MCTree",
39+
]
40+
),
41+
I3TruthExtractor(),
42+
],
43+
outdir,
44+
)
45+
converter(inputs)
46+
if backend == "sqlite":
47+
converter.merge_files(os.path.join(outdir, "merged"))
48+
49+
50+
def main_icecube_upgrade(backend: str):
51+
"""Convert IceCube-Upgrade I3 files to intermediate `backend` format."""
52+
# Check(s)
53+
assert backend in CONVERTER_CLASS
54+
55+
inputs = ["test_data_upgrade_2"]
56+
outdir = "./temp/test_upgrade"
57+
workers = 1
58+
59+
converter = CONVERTER_CLASS[backend](
60+
[
61+
I3TruthExtractor(),
62+
I3RetroExtractor(),
63+
I3FeatureExtractorIceCubeUpgrade(
64+
"I3RecoPulseSeriesMapRFCleaned_mDOM"
65+
),
66+
I3FeatureExtractorIceCubeUpgrade(
67+
"I3RecoPulseSeriesMapRFCleaned_DEgg"
68+
),
69+
],
70+
outdir,
71+
workers=workers,
72+
# nb_files_to_batch=10,
73+
# sequential_batch_pattern="temp_{:03d}",
74+
# input_file_batch_pattern="[A-Z]{1}_[0-9]{5}*.i3.zst",
75+
icetray_verbose=1,
76+
)
77+
converter(inputs)
78+
if backend == "sqlite":
79+
converter.merge_files(os.path.join(outdir, "merged"))
80+
81+
82+
if __name__ == "__main__":
83+
backend = "parquet"
84+
# backend = "sqlite"
85+
main_icecube86(backend)
86+
# main_icecube_upgrade(backend)

‎examples/convert_i3_to_parquet.py‎

Lines changed: 0 additions & 74 deletions
This file was deleted.

‎examples/convert_i3_to_sqlite.py‎

Lines changed: 0 additions & 74 deletions
This file was deleted.

examples/weight_fitter_example.py renamed to examples/pisa_weight_fitter_example.py

File renamed without changes.

0 commit comments

Comments
 (0)