bump all product versions to 2

Add support for trigger scaling in compression mode S=0,k=0,M=7 (i4Ds#377)

* Unscale data when compressed with mode S=0,k=0,M=7
* QL - LC, BKG, SPECTRA
* BSD - CPD, SCPD, VIS and SPEC

bump all product versions to 2 (i4Ds#378)

* bump all product versions to 2

* roleout IDB version 2.26.27

* fix test with new version number in mock object

* fix test

* fix for latest IDB as fallback

Fix duplicated hk time bin bug

Ddpdupdates2 (i4Ds#380)

* bump all product versions to 2

* prep for L2 ddpd

* integrate IDB release 2.26.38

* update for common CONF submodule

* fir text for latest IDB to 2.26.38

Config fix (i4Ds#381)

* bump all product versions to 2

* fix module path

Fix bugs found when processing new scaled trigger data. (i4Ds#382)

* Fix bug processing new scaled trigger data.
* Allow for QL file to unscale by packet
* Fix bug that create extra (replica) triggers in Spectrogram

Publish_L2 / RID LUT Manager (i4Ds#385)

* bump all product versions to 2
* RID LUT data manager update and use in processing for trigger descaling
* Update stixcore/io/RidLutManager.py
* fixed API mock via static files on pup99

---------

Co-authored-by: Shane Maloney <maloneys@tcd.ie>

Add warning if trigger scaling changes in request (i4Ds#388)

End2end rid lut manager (i4Ds#390)

* bump all product versions to 2

* add copy and parent option

* fix missing init of singleton

bump all product versions to 2 (i4Ds#389)

Bump conf version (i4Ds#391)

* bump all product versions to 2

* bump common version

Flexible way to fill DATAMAX/MIN BUNIT EXPOSURE in FITS header keywords (i4Ds#393)

* bump all product versions to 2

* add properties for dedicated fits header keywords with class inhe. / override

* add test and minor fixes after review

* fix XPOSURE/XPOMAX header error

* fix tests for dmax...

* relax end2end fitsdiff

* less tolerance in end2end test

Solo anc stix ephemeris (i4Ds#399)

* bump all product versions to 2

* add more FITS header keywords

* fix for ANC also get fits updated headers while publishing

Upgrade_watchdog (i4Ds#401)

* bump all product versions to 2

* bump version

* fix version label

* pin common CONF version for end2end test

* ignore VERS_CFG in end2end testing

* add test for TM folder observing

* bump watchdog==6.0.0

* integrate review fixes

Update docks (i4Ds#409)

* bump all product versions to 2

* update documentation

* add doc for end to end tetsing

* fix format

* add doc for manual (re)processing of fits products

E2e fix sym add (i4Ds#412)

* bump all product versions to 2

* dep np.bool8

* disable test

Get rid reason/scaling lookup error handling (i4Ds#411)

* bump all product versions to 2

* add global error detection in pipeline logger (not just main process)

* add exception handling for get_reason

daily processing pipeline (i4Ds#405)

* bump all product versions to 2

* initial

* daily processing task

* add singleStep

* move ANC-aspect from pipeline to daily_pipeline

* removed requirement for the moment

* fix i4Ds#283 and option to process only latest fits file versions

* update ddpd scripts

* fix circular imports

* test git dep

* skio test on win

* skip test on win/mac

* Update stixcore/io/FlareListManager.py

Co-authored-by: Shane Maloney <maloneys@tcd.ie>

* fix format error

---------

Co-authored-by: Shane Maloney <maloneys@tcd.ie>

Add Zenodo file (i4Ds#413)

* Add Zenodo author information

Update high resolution transmission to match IDL defaults (i4Ds#414)

Pin major and minor versions of some packages (i4Ds#415)

V1.5.1_fixes (i4Ds#420)

* bump all product versions to 2

* fix i4Ds#417

* fix  i4Ds#418

* daily report log files could have different names if the scripts run over multiple days

* fix i4Ds#419

* end2end test: better report if no corresponding file found

* rewrite pipeline to cronjob based approach i4Ds#422

* cleanup before merge

Fix bug in spectrogram time bin calculations (i4Ds#423)

* Only use the closing time offset (NIX00269) for the last science substrure

Update .zenodo.json

Maintenance update project and dependencies (i4Ds#428)

* Update project use project.toml and move config into own files

* Fix most of the tests

* ruff-check, ruff-format and codespell

* Some spelling mistakes left need to check the meaning

* More updates and fixes

* Few more small fixes

* Fix doc build

* Revert unintentional spelling correction in IDB sql script.

* Update tox test dep name

* Add version.py to git

* Skip unclear test add skipped test for later

* Update workflow python version

* Add remote data to github CI

Author: nicHoch

.codespellrc

@@ -0,0 +1,21 @@
+[codespell]
+skip = ,*.fits,*.fts,*.header,*.json,*.xsh,*cache*,*egg*,*extern*,.git,.idea,.tox,_build,*truncated,*.svg,.asv_env,.history,*.sql
+ignore-words-list =
+    alog,
+    nd,
+    nin,
+    observ,
+    ot,
+    te,
+    upto,
+    afile,
+    precessed,
+    process,
+    livetime,
+    technik,
+    thirdparty,
+    commutated,
+    anc,
+    soop,
+    CODIN,
+    allright

.coveragerc

@@ -0,0 +1,32 @@
+[run]
+omit =
+  stixcore/_sunpy_init*
+  stixcore/conftest.py
+  stixcore/*setup_package*
+  stixcore/tests/*
+  stixcore/*/tests/*
+  stixcore/extern/*
+  stixcore/version*
+  */stixcore/_sunpy_init*
+  */stixcore/conftest.py
+  */stixcore/*setup_package*
+  */stixcore/tests/*
+  */stixcore/*/tests/*
+  */stixcore/extern/*
+  */stixcore/version*
+
+[report]
+exclude_lines =
+  # Have to re-enable the standard pragma
+  pragma: no cover
+  # Don't complain about packages we have installed
+  except ImportError
+  # Don't complain if tests don't hit assertions
+  raise AssertionError
+  raise NotImplementedError
+  # Don't complain about script hooks
+  def main\(.*\):
+  # Ignore branches that don't pertain to this version of Python
+  pragma: py{ignore_python_version}
+  # Don't complain about IPython completion helper
+  def _ipython_key_completions_

.github/workflows/end2end_run.yml

@@ -20,7 +20,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.9"]
+        python-version: ["3.11"]
 
     steps:
     - name: Checkout github repo (+ download lfs dependencies)
@@ -41,4 +41,4 @@ jobs:
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
     - name: end2end test with pytest
       run: |
-        pytest -p no:warnings --doctest-rst -m "end2end" --pyargs stixcore
+        pytest -p no:warnings --doctest-rst -m "end2end" --pyargs stixcore --remote-data=any

.gitignore

@@ -153,7 +153,6 @@ docs/_build
 docs/generated
 docs/api
 docs/whatsnew/latest_changelog.txt
-stixcore/version.py
 htmlcov/
 *.prof
 pdpp.html
@@ -163,3 +162,6 @@ stixcore/data/soop/
 stixcore/data/publish/
 stixcore/data/test/products/end2end/
 monitor_status.json
+stixcore/data/test/idb/v2.26.38/idb.sqlite
+.python-version
+sunpy/_version.py

.isort.cfg

@@ -0,0 +1,14 @@
+[settings]
+balanced_wrapping = True
+skip=docs/conf.py
+default_section = THIRDPARTY
+include_trailing_comma = True
+known_astropy = astropy, asdf
+known_first_party = stixcore
+length_sort = False
+length_sort_sections=stdlib
+line_length = 100
+multi_line_output = 3
+no_lines_before = LOCALFOLDER
+sections = STDLIB, THIRDPARTY, ASTROPY, FIRSTPARTY, LOCALFOLDER
+honor_noqa = True

.pre-commit-config.yaml

@@ -1,53 +1,38 @@
+exclude: ".*(.csv|.fits|.fts|.fit|.header|.txt|tca.*|.json|.asdf)$|^CITATION.rst|(^|/)data/test(/|$)"
 repos:
-  # The warnings/errors we check for here are:
-  # E101 - mix of tabs and spaces
-  # W191 - use of tabs
-  # W291 - trailing whitespace
-  # W292 - no newline at end of file
-  # W293 - trailing whitespace
-  # W391 - blank line at end of file
-  # E111 - 4 spaces per indentation level
-  # E112 - 4 spaces per indentation level
-  # E113 - 4 spaces per indentation level
-  # E303 - too many blank lines (3)
-  # E304 - blank lines found after function decorator
-  # E305 - expected 2 blank lines after class or function definition
-  # E306 - expected 1 blank line before a nested definition
-  # E502 - the backslash is redundant between brackets
-  # E722 - do not use bare except
-  # E901 - SyntaxError or IndentationError
-  # E902 - IOError
-  # F822: undefined name in __all__
-  # F823: local variable name referenced before assignment
-  - repo: https://github.com/pycqa/flake8
-    rev: 3.7.9
+    # This should be before any formatting hooks like isort
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: "v0.12.2"
     hooks:
-      - id: flake8
-        args: ['--count', '--select', 'C90,E,W,F']
-        exclude: ".*(.fits|.fts|.fit|.txt|tca.*|extern.*|.rst|.md|__init__.py|docs/conf.py)$"
-  - repo: https://github.com/myint/autoflake
-    rev: v1.4
-    hooks:
-      - id: autoflake
-        args: ['--in-place', '--remove-all-unused-imports', '--remove-unused-variable']
-        exclude: ".*(.fits|.fts|.fit|.txt|tca.*|extern.*|.rst|.md|__init__.py|docs/conf.py)$"
-  - repo: https://github.com/timothycrosley/isort
-    rev: 5.12.0
+      - id: ruff-check
+        args: ["--fix"]
+      # Run the formatter.
+      - id: ruff-format
+  - repo: https://github.com/PyCQA/isort
+    rev: 6.0.1
     hooks:
       - id: isort
-        args: ['--sp','setup.cfg']
-        exclude: ".*(.fits|.fts|.fit|.txt|tca.*|extern.*|.rst|.md|cm/__init__.py|sunpy/extern|docs/conf.py)$"
   - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v2.4.0
+    rev: v5.0.0
     hooks:
       - id: check-ast
       - id: check-case-conflict
       - id: trailing-whitespace
-        exclude: ".*(.fits|.fts|.fit|.txt)$"
       - id: check-yaml
       - id: debug-statements
       - id: check-added-large-files
+        args: ["--enforce-all", "--maxkb=1054"]
+        exclude: ""
       - id: end-of-file-fixer
-        exclude: ".*(.fits|.fts|.fit|.txt|tca.*)$"
       - id: mixed-line-ending
-        exclude: ".*(.fits|.fts|.fit|.txt|tca.*)$"
+  - repo: https://github.com/codespell-project/codespell
+    rev: v2.4.1
+    hooks:
+      - id: codespell
+        additional_dependencies:
+          - tomli
+        args: ["--write-changes"]
+
+ci:
+  autofix_prs: false
+  autoupdate_schedule: "quarterly"

.readthedocs.yml

@@ -19,7 +19,7 @@ formats: []
 build:
   os: ubuntu-22.04
   tools:
-    python: "3.9"
+    python: "3.11"
 
 # Optionally set the version of Python and requirements required to build your docs
 python:

.zenodo.json

@@ -0,0 +1,19 @@
+{
+    "description": "STIXCore is the software pipeline for STIX data, turning raw telemetry into standards compliant FITS files for analysis",
+    "license": "bsd-3-clause",
+    "title": "STIXCore",
+    "upload_type": "software",
+    "creators": [
+        {
+            "name": "Shane Maloney",
+            "orcid": "0000-0002-4715-1805",
+            "affiliation": "Dublin Institute for Advanced Studies"
+        },
+        {
+            "name": "Nicky Hochmuth",
+            "orcid": "0009-0005-3785-7643",
+            "affiliation": "Ateleris GmbH Switzerland"
+        }
+    ],
+    "access_right": "open"
+}

MANIFEST.in

@@ -1,19 +1,21 @@
-include README.rst
-include CHANGES.rst
-include setup.cfg
-include LICENSE.rst
-include pyproject.toml
+# Exclude specific files
+# All files which are tracked by git and not explicitly excluded here are included by setuptools_scm
+exclude .codecov.yaml
+exclude .editorconfig
+exclude .gitattributes
+exclude .gitignore
+exclude .mailmap
+exclude .pre-commit-config.yaml
+exclude .readthedocs.yaml
+exclude .rtd-environment.yml
+exclude .zenodo.json
+exclude tox.ini
 
-recursive-include stixcore *.pyx *.c *.pxd
-recursive-include docs *
-recursive-include licenses *
-recursive-include cextern *
-recursive-include scripts *
-recursive-include stixcore/config/data *
-recursive-include stixcore/data *
-
-prune build
-prune docs/_build
-prune docs/api
+# Prune folders
+prune .circleci
+prune .github
+prune changelog
+prune sunpy/_dev
 
-global-exclude *.pyc *.o
+# Add back in submodule data
+recursive-include stixcore/config/data *

azure-pipelines.yml

@@ -3,7 +3,7 @@ variables:
   CI_NAME: Azure Pipelines
   CI_BUILD_ID: $(Build.BuildId)
   CI_BUILD_URL: "https://dev.azure.com/https://github.com/i4Ds/STIXCore.git/_build/results?buildId=$(Build.BuildId)"
-  CIBW_BUILD: cp39-* cp310-* cp311-*
+  CIBW_BUILD: cp311-* cp312-* cp313-*
   CIBW_SKIP: "*-win32 *-manylinux1_i686"
 
 resources:
@@ -35,17 +35,17 @@ jobs:
     toxdeps: tox-pypi-filter
     submodules: true
     coverage: codecov
-    default_python: '3.9'
+    default_python: '3.11'
     envs:
-      - macos: py39
-        name: py39_mac
+      - macos: py311
+        name: py311_mac
         coverage: false
 
-      - windows: py39
-        name: py39_win
+      - windows: py311
+        name: py311_win
 
-      - linux: py39
-        name: py39_linux
+      - linux: py311
+        name: py311_linux
 
       - linux: codestyle
         name: codestyle
@@ -67,6 +67,6 @@ jobs:
         # - wheels_macos
         - sdist
       dependsOn:
-        - py39_test
-        - py310_test
-        - py311_test
+        - py311_mac
+        - py311_win
+        - py311_linux