Minor tweaks (#469)

* add a comment to explain what the code does

* remove json db

* update consolidation to handle sqlite dbs

* remove unused function

* create data for daily report

* remove unused file

* update to call new version of reportOfLatestMatches

* add test variables

* mildly improve doco

* put test logs in the right CW log group

* handle missing log groups better

* extra value in TrajQualityParams

* bugfixes revealed in testing

* fix issues in reportOfLatestMatches

* put container dbs in a sensible place

* catch logging errors

* tidy up file locations

* simplify calcserver start up

* sorting out errors in container logging

* update container logging info

* avoid unnecessary error

* fix bugs in log capture process

* add MAX_STNS so i can experiement with limits

* add logging.
Add max_stns to constraints and set to 9999
push logs to s3
copy calcserver logs to ukmonhelper

* don't pull or push the databases from S3 during distribution
but do back them up after consolidation

* catch error if trajpickle can't be found

* set rundate more safely
move some processes into nightlyJob to make testing easier

* tidy up backup file names and housekeep backup folder

* move some processes from findAllMatches to nightlyJob to make testing easier

* support for testing of createExecReplot

* pass testmode to createExecReplot

* ensure logs are unique and capture container logs

* realigning createFireballPage with prod. Updating gitignore

* remove verbosity flag from one tar operation

* fixup to handle new style log with datestamps

* get privip after restarting calcserver

* temporarily don't run rerunFailedLambdas in test

* update to work in a test env too

* add 'period' to the match summary api

* add period to match summary api

* re-enabling rerunFailedLambdas

* quieten an S3 transfer

* include the foldername when reporting on container progress

* put daily trajdb in a safe place

* be more specific when finding the daily log

* only tar the current container logs

* save daily obsdb in the right place

* rename ec2.tf to calcserver.tf

* quieten some s3 transfers

* rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is

* update SSM params to reflect new serverid

* bugfix in distributeCandidates - using wrong path

* add newline to getCostMetrics

* weird bug in statsToMqtt

* bugfix in update_wmpl.sh for traj container

* Script to create an AMI for the calcengine

* exclude test data when syncing from s3

* writing databases and logs to the wrong target

* bad test on istest

* typo in serveruserid

* adding to server migration notes

* purge old rawcsvs and fullcsvs older than 180 days

* improved housekeeping of old data

* make sure we scoop up data for 31/12

* stop creation of an unused set of files

* change default locations for container data

* put container logs in the right place

* update dockerfile syntax

* change locations of test container data and output

* remove incorrect comments

* Add script to clean up deleted/duplicate trajs

* simplify the stats gathering and move to python
fix bug in getMatchStats

* comment out unused files

* update reportOfLatestMatches to exclude deleted trajectories

* add scp to requirements

* add SQL database maintenance to dataMaintenance

* incorrect default for datadir

* update call to pymysql.connect

* ensure consolidateOutput rerunnable for prior years

* upgrading WMPL

* update gitignore a bit

* bugfix to catch data from 31/12 each year

* add paramiko to dependencies for matchPickle to support wmpl upgrade

* update getpickle to newest WMPL

* bugfix in api tests

* bugfix - changes to dailydb not saved

* improvements to process to cleanup deleted trajs

* after merging, check and remove any duplicate trajectories from the sqlite db and disk

* Adding version

* bump version 2025.11.1 -> 2026.04.0

* update files containing vesion

* bump version 2026.04.0 -> 2026.04.1

* unmangling side effets of automerge

* correct usage message

* decommission old calcserver and add new batchserver

* add calcserver IP to ssm variables and use it
in scripts

* Update my branch with the latest from master (#436)

* Update version (#433)

* remove disused code

* update livestream to accept normal RMS filenames

* search dialog - fix bug and set default date range

* small change to explain default dates

* improvement in live image search

* remove redundant file

* update deployment scripts

* improvements in routine to convert GMN report to pandas

* more analysis of gmn data

* Update the terraform a little

* remove hardcoded reference to ec2-user

* add SERVERUSERID env /ssm variable for calcserver user id

* update container for new distrib-processing method

* use calcserver SERVERUSERID variable

* support for test ECS containers

* fix paths in deployment scripts

* remove unused file

* missed some templates

* update dev/test env settings

* remove disused file

* make sure paths are expanded

* fixing various config issues

* updates to makeConfig

* updates to distributed processing to align with newest wmpl

* add a comment to explain what the code does

* remove json db

* update consolidation to handle sqlite dbs

* remove unused function

* create data for daily report

* remove unused file

* update to call new version of reportOfLatestMatches

* add test variables

* mildly improve doco

* put test logs in the right CW log group

* handle missing log groups better

* extra value in TrajQualityParams

* bugfixes revealed in testing

* fix issues in reportOfLatestMatches

* put container dbs in a sensible place

* catch logging errors

* tidy up file locations

* simplify calcserver start up

* sorting out errors in container logging

* update container logging info

* avoid unnecessary error

* fix bugs in log capture process

* add MAX_STNS so i can experiement with limits

* add logging.
Add max_stns to constraints and set to 9999
push logs to s3
copy calcserver logs to ukmonhelper

* don't pull or push the databases from S3 during distribution
but do back them up after consolidation

* catch error if trajpickle can't be found

* set rundate more safely
move some processes into nightlyJob to make testing easier

* tidy up backup file names and housekeep backup folder

* move some processes from findAllMatches to nightlyJob to make testing easier

* support for testing of createExecReplot

* pass testmode to createExecReplot

* ensure logs are unique and capture container logs

* realigning createFireballPage with prod. Updating gitignore

* remove verbosity flag from one tar operation

* fixup to handle new style log with datestamps

* get privip after restarting calcserver

* temporarily don't run rerunFailedLambdas in test

* update to work in a test env too

* add 'period' to the match summary api

* add period to match summary api

* re-enabling rerunFailedLambdas

* quieten an S3 transfer

* include the foldername when reporting on container progress

* put daily trajdb in a safe place

* be more specific when finding the daily log

* only tar the current container logs

* save daily obsdb in the right place

* rename ec2.tf to calcserver.tf

* quieten some s3 transfers

* rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is

* update SSM params to reflect new serverid

* bugfix in distributeCandidates - using wrong path

* add newline to getCostMetrics

* weird bug in statsToMqtt

* bugfix in update_wmpl.sh for traj container

* Script to create an AMI for the calcengine

* exclude test data when syncing from s3

* writing databases and logs to the wrong target

* bad test on istest

* typo in serveruserid

* adding to server migration notes

* purge old rawcsvs and fullcsvs older than 180 days

* improved housekeeping of old data

* make sure we scoop up data for 31/12

* stop creation of an unused set of files

* change default locations for container data

* put container logs in the right place

* update dockerfile syntax

* change locations of test container data and output

* remove incorrect comments

* Add script to clean up deleted/duplicate trajs

* simplify the stats gathering and move to python
fix bug in getMatchStats

* comment out unused files

* update reportOfLatestMatches to exclude deleted trajectories

* add scp to requirements

* add SQL database maintenance to dataMaintenance

* incorrect default for datadir

* update call to pymysql.connect

* ensure consolidateOutput rerunnable for prior years

* upgrading WMPL

* update gitignore a bit

* bugfix to catch data from 31/12 each year

* add paramiko to dependencies for matchPickle to support wmpl upgrade

* update getpickle to newest WMPL

* bugfix in api tests

* bugfix - changes to dailydb not saved

* improvements to process to cleanup deleted trajs

* after merging, check and remove any duplicate trajectories from the sqlite db and disk

* Adding version

* bump version 2025.11.1 -> 2026.04.0

* update files containing vesion

* bump version 2026.04.0 -> 2026.04.1

* unmangling side effets of automerge

* correct usage message

* adding install script

* changes to ansible scripts

* remove irrelevant file

* Adding SSM variables for the new batchserver in the ukmda account

* attempt to add logging to consolidateDistTraj

* remove reference to ee account

* temporarily add mariadb deets to ssm in mjmm account

* bugfixes in dataMaintenance and cleanupDeletedTrajs to avoid deleting disk files when trajs share same folder

* updating migration instructions

* more work on deduplication for ukmon

* remove username-specific stuff

* remove the old deployment scripts

* update install script

* updates to improve server migration process

* add bashrc to the installer

* slight tweak to installer

* add a few installer messages

* remove an echo

* be less verbose in the rsyncs

* oops need -a for one of the rsyncs

* rename requirements file for ukmda

* remove dependencies on MeteorTools

* add utils to ukmon_pylib

* more removal of dependencies on MeteorTools

* updating how shower data is obtained

* corercting location of IMO and gmn shower data

* bugfixes

* fixes in imoWorkinShowerlist

* small error in install_or_update

* bugfix when no data found in imo set

* retrieve IMO data after each  code update

* fixing up getShowerDets

* improve installer

* fix createShowerExtracts

* update extractors and showeranalysis

* bug in extractor thats existed since 2023 but nobody noticed...

* trying to fix matplotlib bug

* more working on matplotlib issue

* oops overfixed a bug

* another incorrect bugfix!

* make bars wider

* more removal of MeteorTools

* bugfix in createannualbarchart

* working on bug

* more work on lenght bug

* remove some debugging

* oops forgot to push the fixed version of createReportIndex

* segregating function to get list of images that weren't calibrated by RMS

* reorganize stuff a bit

* add email sending functionality to  ukmon_pylib

* use send email functionality in code

* add database and gmaps keys/passwords as secure variables

* update to use secure keys

* remove dbpw from environment

* remove dbpw from environment

* add script to retrieve db passwd

* add script to retrieve db passwords

* bug in createTables

* updates to migration instructions

* include pip installs in the script

* tweak installer

* remove glob.glob1

* move unused scripts - will delete later

* fix mariadb connections

* typo

* tidy up docs

* fixing mysql calls

* small changes to documentation

* suppress pip messages if already installed

* tweaks to the installer

* add logrotate stuff

* bugfixes in installer

* tidy up installer

* bugfix in key mgmt

* move some functionality into new script to tidy up nightly job and simplify rerunning

* improve install script

* update location of camera count file

* move annual bar charts into a more logical location

* update test script

* add gmaps param to my acct

* relocate coverage maps and front page index

* tidy up useraudit process

* bugfixes in testing

* bugfix in createAnnualBarchart

* bug in testscript

* another small bug in createSummaryTable

* still fighting that bug

* tidy up lastlog data

* remove unused file

* update cam status reporting

* clear down the old brightness data

* simplification of logging and of the nightlyJob script

* bugfixes in process to gather consumed image ifo

* update getUncal to work for historic data

* typo

* improvements in getUncalImages

* bugfix

* add python to get unused images

* move script to right place

* comment out unnecessary echo

* bugfixes in camMetrics while migrating user accts

* fix #448

* make sure window title is sensible

* bugfix in replot and get unused

* bugfix in removeDeletedTraj for search index

* remove deprecation warning issue #450

* whoops naive vs tzaware dates

* remove warning in cameraStatusReport issue #453

* remove warning issue #454

* widen constraints on analysing pickles to get images

* add migration script

* add sudo

* actually do it

* update sftp migration script

* more changes to migration script

* moving sftp migration script

* update messages and docos

* slight tweak to migration script

* further work on sftp account migration

* update ukmon ini files after migration

* typo

* collect old server login records
integration with new

* update migration instructions

* update cam metrics report to include hostname

* bugfix as $USER not available in cron

* removing unused files

* some tidying up

* motre reorg

* remove disused template
rename file to follow convention

* simplifying logging and removing unused aws profile

* removing  profile from python

* adding executable bit

* typo

* bugfix to work around typo

* print message when done

* bugfix to get trajectories from s3 before trying to check for missing trajectories...

* improve to support running for past dates

* errors in bashrc and logrotate config

* prevent email being sent for anything other than current days report

* bugfix: make sure we use the most recent stats

* improvement in reportOfLatestMatches to handle past dates

* bugfix

* small tweak to support reportOfLatestMatches

* tweak getMatchStats to handle historic data better

* add mariadb and ping to the sec grp so that the database can be accessed by apis

* update the db servername

* bugfix in match stats

* prevent bogus renaming of matchJob.log

* bumping version

* bump version 2026.04.1 -> 2026.05.0

* update matchapi to use indexed column

* add api readme

Author: markmac99

api_examples/README.md

@@ -0,0 +1,6 @@
+#Available APIs and Examples
+
+We have a number of APIs which can be used to retrieve our data programatically, either using commandline tools like curl and wget, or from programming languages such as Python. 
+
+Data are returned in JSON format which can readily be converted into a python list, array or Pandas DataFrame. Simple examples are shown on our main website (here)[https://ukmeteornetwork.org/our-data-apis/], and a set of Python examples are available here. A Data Dictionary in Excel format can be downloaded from (here)[https://archive.ukmeteors.co.uk/browse/datadictionary.xlsx].
+

archive/lambdas/matchDataApi/matchDataApi.py

@@ -67,7 +67,7 @@ def getStationData(statid, dtstr, period=None):
         statfrag = f"and s.stations like '%{statid}%' " if statid is not None else ""
         perfrag = periodToSqlFragment(period) if period is not None else ""
         with connection.cursor() as cursor:
-            sql = f"SELECT s.orbname from matches s where s._localtime like '_{dtstr}%' {statfrag} {perfrag}"
+            sql = f"SELECT s.orbname from matches s where s.orbname like '{dtstr}%' {statfrag} {perfrag}"
             cursor.execute(sql)
             result = cursor.fetchall()
     finally:
@@ -85,7 +85,7 @@ def getSummaryData(dtstr, period=None):
     fieldlist = '_localtime,_mjd,_sol,_ID1,_amag,_ra_o,_dc_o,_ra_t,_dc_t,_elng,_elat,_vo,_vi,_vg,_vs,_a,_q,_e,_p,_peri,_node,_incl,'\
         '_stream,_mag,_dur,_lng1,_lat1,_H1,_lng2,_lat2,_H2,_LD21,_az1r,_ev1r,_Nts,_Nos,_leap,_tme,_dt,'\
         'dtstamp,orbname,iau,shwrname as name,mass,pi,Q,true_anom,EA,MA,Tj,T,last_peri,jacchia1,Jacchia2,numstats,stations'
-    expr = f"SELECT {fieldlist} from matches s where s._localtime like '_{dtstr}%' {perfrag}"
+    expr = f"SELECT {fieldlist} from matches s where s.orbname like '{dtstr}%' {perfrag}"
     result=[]
     try:
         with connection.cursor() as cursor:
@@ -171,7 +171,7 @@ def lambda_handler(event, context):
                 res = '{"points": "unavailable"}'
 
     else:
-        res = '{"invalid request type - must be one of \'matches\', \'details\', \'station\',\'summary\'"}'
+        res = '{"invalid request type - must be one of \'matches\', \'detail\', \'station\',\'summary\'"}'
 
     print(res)
     return {