Latest patch release (#473)

* save daily obsdb in the right place

* rename ec2.tf to calcserver.tf

* quieten some s3 transfers

* rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is

* update SSM params to reflect new serverid

* bugfix in distributeCandidates - using wrong path

* add newline to getCostMetrics

* weird bug in statsToMqtt

* bugfix in update_wmpl.sh for traj container

* Script to create an AMI for the calcengine

* exclude test data when syncing from s3

* writing databases and logs to the wrong target

* bad test on istest

* typo in serveruserid

* adding to server migration notes

* purge old rawcsvs and fullcsvs older than 180 days

* improved housekeeping of old data

* make sure we scoop up data for 31/12

* stop creation of an unused set of files

* change default locations for container data

* put container logs in the right place

* update dockerfile syntax

* change locations of test container data and output

* remove incorrect comments

* Add script to clean up deleted/duplicate trajs

* simplify the stats gathering and move to python
fix bug in getMatchStats

* comment out unused files

* update reportOfLatestMatches to exclude deleted trajectories

* add scp to requirements

* add SQL database maintenance to dataMaintenance

* incorrect default for datadir

* update call to pymysql.connect

* ensure consolidateOutput rerunnable for prior years

* upgrading WMPL

* update gitignore a bit

* bugfix to catch data from 31/12 each year

* add paramiko to dependencies for matchPickle to support wmpl upgrade

* update getpickle to newest WMPL

* bugfix in api tests

* bugfix - changes to dailydb not saved

* improvements to process to cleanup deleted trajs

* after merging, check and remove any duplicate trajectories from the sqlite db and disk

* Adding version

* bump version 2025.11.1 -> 2026.04.0

* update files containing vesion

* bump version 2026.04.0 -> 2026.04.1

* unmangling side effets of automerge

* correct usage message

* decommission old calcserver and add new batchserver

* add calcserver IP to ssm variables and use it
in scripts

* Update my branch with the latest from master (#436)

* Update version (#433)

* remove disused code

* update livestream to accept normal RMS filenames

* search dialog - fix bug and set default date range

* small change to explain default dates

* improvement in live image search

* remove redundant file

* update deployment scripts

* improvements in routine to convert GMN report to pandas

* more analysis of gmn data

* Update the terraform a little

* remove hardcoded reference to ec2-user

* add SERVERUSERID env /ssm variable for calcserver user id

* update container for new distrib-processing method

* use calcserver SERVERUSERID variable

* support for test ECS containers

* fix paths in deployment scripts

* remove unused file

* missed some templates

* update dev/test env settings

* remove disused file

* make sure paths are expanded

* fixing various config issues

* updates to makeConfig

* updates to distributed processing to align with newest wmpl

* add a comment to explain what the code does

* remove json db

* update consolidation to handle sqlite dbs

* remove unused function

* create data for daily report

* remove unused file

* update to call new version of reportOfLatestMatches

* add test variables

* mildly improve doco

* put test logs in the right CW log group

* handle missing log groups better

* extra value in TrajQualityParams

* bugfixes revealed in testing

* fix issues in reportOfLatestMatches

* put container dbs in a sensible place

* catch logging errors

* tidy up file locations

* simplify calcserver start up

* sorting out errors in container logging

* update container logging info

* avoid unnecessary error

* fix bugs in log capture process

* add MAX_STNS so i can experiement with limits

* add logging.
Add max_stns to constraints and set to 9999
push logs to s3
copy calcserver logs to ukmonhelper

* don't pull or push the databases from S3 during distribution
but do back them up after consolidation

* catch error if trajpickle can't be found

* set rundate more safely
move some processes into nightlyJob to make testing easier

* tidy up backup file names and housekeep backup folder

* move some processes from findAllMatches to nightlyJob to make testing easier

* support for testing of createExecReplot

* pass testmode to createExecReplot

* ensure logs are unique and capture container logs

* realigning createFireballPage with prod. Updating gitignore

* remove verbosity flag from one tar operation

* fixup to handle new style log with datestamps

* get privip after restarting calcserver

* temporarily don't run rerunFailedLambdas in test

* update to work in a test env too

* add 'period' to the match summary api

* add period to match summary api

* re-enabling rerunFailedLambdas

* quieten an S3 transfer

* include the foldername when reporting on container progress

* put daily trajdb in a safe place

* be more specific when finding the daily log

* only tar the current container logs

* save daily obsdb in the right place

* rename ec2.tf to calcserver.tf

* quieten some s3 transfers

* rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is

* update SSM params to reflect new serverid

* bugfix in distributeCandidates - using wrong path

* add newline to getCostMetrics

* weird bug in statsToMqtt

* bugfix in update_wmpl.sh for traj container

* Script to create an AMI for the calcengine

* exclude test data when syncing from s3

* writing databases and logs to the wrong target

* bad test on istest

* typo in serveruserid

* adding to server migration notes

* purge old rawcsvs and fullcsvs older than 180 days

* improved housekeeping of old data

* make sure we scoop up data for 31/12

* stop creation of an unused set of files

* change default locations for container data

* put container logs in the right place

* update dockerfile syntax

* change locations of test container data and output

* remove incorrect comments

* Add script to clean up deleted/duplicate trajs

* simplify the stats gathering and move to python
fix bug in getMatchStats

* comment out unused files

* update reportOfLatestMatches to exclude deleted trajectories

* add scp to requirements

* add SQL database maintenance to dataMaintenance

* incorrect default for datadir

* update call to pymysql.connect

* ensure consolidateOutput rerunnable for prior years

* upgrading WMPL

* update gitignore a bit

* bugfix to catch data from 31/12 each year

* add paramiko to dependencies for matchPickle to support wmpl upgrade

* update getpickle to newest WMPL

* bugfix in api tests

* bugfix - changes to dailydb not saved

* improvements to process to cleanup deleted trajs

* after merging, check and remove any duplicate trajectories from the sqlite db and disk

* Adding version

* bump version 2025.11.1 -> 2026.04.0

* update files containing vesion

* bump version 2026.04.0 -> 2026.04.1

* unmangling side effets of automerge

* correct usage message

* adding install script

* changes to ansible scripts

* remove irrelevant file

* Adding SSM variables for the new batchserver in the ukmda account

* attempt to add logging to consolidateDistTraj

* remove reference to ee account

* temporarily add mariadb deets to ssm in mjmm account

* bugfixes in dataMaintenance and cleanupDeletedTrajs to avoid deleting disk files when trajs share same folder

* updating migration instructions

* more work on deduplication for ukmon

* remove username-specific stuff

* remove the old deployment scripts

* update install script

* updates to improve server migration process

* add bashrc to the installer

* slight tweak to installer

* add a few installer messages

* remove an echo

* be less verbose in the rsyncs

* oops need -a for one of the rsyncs

* rename requirements file for ukmda

* remove dependencies on MeteorTools

* add utils to ukmon_pylib

* more removal of dependencies on MeteorTools

* updating how shower data is obtained

* corercting location of IMO and gmn shower data

* bugfixes

* fixes in imoWorkinShowerlist

* small error in install_or_update

* bugfix when no data found in imo set

* retrieve IMO data after each  code update

* fixing up getShowerDets

* improve installer

* fix createShowerExtracts

* update extractors and showeranalysis

* bug in extractor thats existed since 2023 but nobody noticed...

* trying to fix matplotlib bug

* more working on matplotlib issue

* oops overfixed a bug

* another incorrect bugfix!

* make bars wider

* more removal of MeteorTools

* bugfix in createannualbarchart

* working on bug

* more work on lenght bug

* remove some debugging

* oops forgot to push the fixed version of createReportIndex

* segregating function to get list of images that weren't calibrated by RMS

* reorganize stuff a bit

* add email sending functionality to  ukmon_pylib

* use send email functionality in code

* add database and gmaps keys/passwords as secure variables

* update to use secure keys

* remove dbpw from environment

* remove dbpw from environment

* add script to retrieve db passwd

* add script to retrieve db passwords

* bug in createTables

* updates to migration instructions

* include pip installs in the script

* tweak installer

* remove glob.glob1

* move unused scripts - will delete later

* fix mariadb connections

* typo

* tidy up docs

* fixing mysql calls

* small changes to documentation

* suppress pip messages if already installed

* tweaks to the installer

* add logrotate stuff

* bugfixes in installer

* tidy up installer

* bugfix in key mgmt

* move some functionality into new script to tidy up nightly job and simplify rerunning

* improve install script

* update location of camera count file

* move annual bar charts into a more logical location

* update test script

* add gmaps param to my acct

* relocate coverage maps and front page index

* tidy up useraudit process

* bugfixes in testing

* bugfix in createAnnualBarchart

* bug in testscript

* another small bug in createSummaryTable

* still fighting that bug

* tidy up lastlog data

* remove unused file

* update cam status reporting

* clear down the old brightness data

* simplification of logging and of the nightlyJob script

* bugfixes in process to gather consumed image ifo

* update getUncal to work for historic data

* typo

* improvements in getUncalImages

* bugfix

* add python to get unused images

* move script to right place

* comment out unnecessary echo

* bugfixes in camMetrics while migrating user accts

* fix #448

* make sure window title is sensible

* bugfix in replot and get unused

* bugfix in removeDeletedTraj for search index

* remove deprecation warning issue #450

* whoops naive vs tzaware dates

* remove warning in cameraStatusReport issue #453

* remove warning issue #454

* widen constraints on analysing pickles to get images

* add migration script

* add sudo

* actually do it

* update sftp migration script

* more changes to migration script

* moving sftp migration script

* update messages and docos

* slight tweak to migration script

* further work on sftp account migration

* update ukmon ini files after migration

* typo

* collect old server login records
integration with new

* update migration instructions

* update cam metrics report to include hostname

* bugfix as $USER not available in cron

* removing unused files

* some tidying up

* motre reorg

* remove disused template
rename file to follow convention

* simplifying logging and removing unused aws profile

* removing  profile from python

* adding executable bit

* typo

* bugfix to work around typo

* print message when done

* bugfix to get trajectories from s3 before trying to check for missing trajectories...

* improve to support running for past dates

* errors in bashrc and logrotate config

* prevent email being sent for anything other than current days report

* bugfix: make sure we use the most recent stats

* improvement in reportOfLatestMatches to handle past dates

* bugfix

* small tweak to support reportOfLatestMatches

* tweak getMatchStats to handle historic data better

* add mariadb and ping to the sec grp so that the database can be accessed by apis

* update the db servername

* bugfix in match stats

* prevent bogus renaming of matchJob.log

* bumping version

* bump version 2026.04.1 -> 2026.05.0

* update matchapi to use indexed column

* add api readme

* tweak to cammetrics to enhance stationlogins report

* slight tweak to formatting

* minor tweak to avoid unexpectedly changing directory

* add support script for during server migrations

* skip 1st row

* add check for todos

* oops strip whitespace

* darn whitespace

* working on migration tasks

* minor tweaks

* daft typo

* bump version 2026.05.0 -> 2026.05.1

* bump version 2026.05.1 -> 2026.05.2

* update CodeQL to v3

* bump codeQL to v4

* tweak to script supporting server migration

* tweak to scripts supporting server migration

* add some stats on migration status

* tweak

* get cursor keys working in python

* set via even if no recent login

* improved reporting on migration

* tweaks to migr scripts

* another small tweak

* make lists unique

* uniq and sort backwards

* testing fix for #460

* more work on status reports

* bugfix in failed matches report

* found the bug

* attempt to fix fails report

* another tweak

* okay, got it

* remove debug

* fix timing metrics data

* small tweak

* looks good

* add EIP for the batchserver

* bugfix to handle bad data in login details

* add function to email problem cameras with bespoke message

* bump version 2026.05.2 -> 2026.05.3

Author: markmac99

README.md

@@ -1,5 +1,5 @@
 # UK Meteor Data Analysis Shared code and libraries
-version: 2026.05.2
+version: 2026.05.3
 
 This repository contains the code behind the UK Meteors data archive and data processing pipeline. 
 

archive/README.md

@@ -1,7 +1,7 @@
 Data Processing and Flows
 ==========================
 
-version: 2026.05.2
+version: 2026.05.3
 
 This diagram shows the overall flow of data from Cameras to websites and out to the public.
 

archive/analysis/getLogData.sh

@@ -62,7 +62,7 @@ echo "the entire night was uncalibrated (indicated by the 'skipping' warning).</
 echo "<p>To interpret these data, see " >> $logfile
 echo "<a href=https://github.com/markmac99/ukmon-pitools/wiki/UKMON-Pi-Toolset-FAQ#Interpreting_the_Uncalibrated_Report>here</a></p>" >> $logfile
 echo "<pre>" >> $logfile
-grep "Skipping" $lastmtch >> $logfile
+grep "Skipping" $lastmtch | egrep -v "directory|pickle" >> $logfile
 echo "</pre>" >> $logfile
 
 echo "<h2 id=fails>Failed Match Report</h2>" >> $logfile
@@ -72,6 +72,7 @@ echo "<a href=https://github.com/markmac99/ukmon-pitools/wiki/Trajectory-Solver-
 echo "<pre>" >> $logfile
 cat $DATADIR/failed/${rundate}_failed.txt >> $logfile
 echo "</pre>" >> $logfile
+
 echo "<h2 id=graph>Chart of Batch Element Runtimes</h2>" >> $logfile
 echo "<p>This section shows the runtime of the different jobs in the batch</p>" >> $logfile
 echo "<p><a href=/reports/batchcharts/$rundate-perfNightly.jpg><img src=/reports/batchcharts/$rundate-perfNightly.jpg width=300\%></a></p>"  >> $logfile
@@ -93,4 +94,7 @@ done
 cat $TEMPLATES/footer.html >> $DATADIR/lastlogs/index.html
 aws s3 cp $DATADIR/lastlogs/index.html  $WEBSITEBUCKET/reports/lastlogs/ --quiet
 
+# housekeep the fails reports
+find $DATADIR/failed -mtime +90 -exec rm -f {} \;
+
 find $DATADIR/lastlogs -name "lastlog*" -mtime +90 -exec rm -f {} \;

archive/server_setup/.bashrc

@@ -55,9 +55,12 @@ fi
 unset __conda_setup
 # <<< conda initialize <<<
 
+# make the function keys work properly in python
+export TERM=putty
+
 # If this is an xterm set the title to user@host:dir
 case "$TERM" in
- xterm*|rxvt*)
+ xterm*|rxvt*|putty*)
     PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
     ;;
  *)

archive/server_setup/acctStatusChecks.sh

@@ -4,22 +4,24 @@ here="$( cd "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
 
 cd $here
 
-sudo ls -1 /var/sftp > moved.txt
-grep batchs /home/ubuntu/prod/data/reports/stationlogins.txt |awk '{print $3}'| sort > switched.txt
-grep ukmonh /home/ubuntu/prod/data/reports/stationlogins.txt |awk '{print $3}'| sort > pending.txt
+sudo ls -1 /var/sftp  | egrep -v "test|logup" > moved.txt
+grep batchs /home/ubuntu/prod/data/reports/stationlogins.txt | cut -c 42-61 | sed 's/ //g' | sort | uniq > switched.txt
+grep ukmonh /home/ubuntu/prod/data/reports/stationlogins.txt | cut -c 42-61 | sed 's/ //g' | sort | uniq > pending.txt
 
 ssh ukmonhelper2 "~/prod/server_setup/get-nbd.sh" | while read i ; do echo $i | awk -F"/" '{print $4}' ; done > not-being-done.txt
 ssh ukmonhelper2 "~/prod/server_setup/get-all.sh" | while read i ; do echo $i | awk -F"/" '{print $4}' ; done > all-accounts.txt
 
 python ~/src/ukmda-dataprocessing/archive/server_setup/checkSftpAccounts.py
 
-echo "Moved:    $(wc -l moved.txt | awk '{print $1}')"
-echo "Switched: $(wc -l switched.txt | awk '{print $1}')"
-echo "Pending:  $(wc -l pending.txt | awk '{print $1}')"
-echo "Total:    $(wc -l still-live.txt | awk '{print $1}')"
-echo ""
-echo "Not Live: $(wc -l not_live.txt | awk '{print $1}')"
-echo "Not Upl:  $(wc -l not_uploading.txt | awk '{print $1}')"
-echo ""
-echo "Dead:     $(wc -l not-being-done.txt | awk '{print $1}')"
-echo ""
+
+echo "Moved:    $(wc -l moved.txt | awk '{print $1}')" > statusreport.txt
+echo "Switched: $(wc -l switched.txt | awk '{print $1}')" >> statusreport.txt
+echo "Pending:  $(wc -l pending.txt | awk '{print $1}')" >> statusreport.txt
+echo "Total:    $(wc -l still-live.txt | awk '{print $1}')" >> statusreport.txt
+echo "" >> statusreport.txt
+echo "Not Live: $(wc -l not_live.txt | awk '{print $1}')" >> statusreport.txt
+echo "Not Upl:  $(wc -l not_uploading.txt | awk '{print $1}')" >> statusreport.txt
+echo "" >> statusreport.txt
+echo "Dead:     $(wc -l not-being-done.txt | awk '{print $1}')" >> statusreport.txt
+echo "" >> statusreport.txt
+cat statusreport.txt

archive/server_setup/checkSftpAccounts.py

@@ -62,4 +62,14 @@
 	for nam in livenames:
 		if nam.strip() not in donelist:
 			print('done list is missing', nam)
-			outf.write(f'{nam}\n')
+			outf.write(f'{nam}\n')
+
+
+moved = open('moved.txt').readlines()
+pending = open('pending.txt').readlines()
+switched = open('switched.txt').readlines()
+
+pendingnotmoved = [x for x in pending if x not in moved]
+
+print('stations marked pending but not moved - check if need done')
+print(pendingnotmoved)

archive/terraform/ukmda/batchserver.tf

@@ -45,3 +45,11 @@ resource "aws_network_interface" "batchserver_if" {
     "billingtag" = "ukmda"
   }
 }
+
+resource "aws_eip" "batchserver" {
+  instance = aws_instance.batchserver.id
+  tags = {
+    billingtag = "ukmda"
+    Name       = "ukmonhelper_eip"
+  }
+}

archive/ukmon_pylib/metrics/camMetrics.py

@@ -216,5 +216,5 @@ def backPopulate(stationid):
             for _,rw in df.iterrows():
                 lastup = '> 1 month ago' if pd.isnull(rw.lastupload) else rw.lastupload.strftime('%Y-%m-%dT%H:%M:%S')
                 lastlo = '> 1 month ago' if pd.isnull(rw.lastseen) else rw.lastseen.strftime('%Y-%m-%dT%H:%M:%S')
-                via = '' if pd.isnull(rw.host) else rw.host
+                via = 'ukmonhelper2' if pd.isnull(rw.host) else rw.host
                 outf.write(f'{lastup:19s}  {lastlo:19s}  {rw.location:20s}  {rw.stationid:10s}  {via}\n')

archive/ukmon_pylib/metrics/timingMetrics.py

@@ -12,6 +12,7 @@
 
 
 def graphOfData(logf, dtstr):
+    
     lis = open(logf,'r').readlines()
     dta = [li for li in lis if li[:8]==dtstr]
 
@@ -55,78 +56,35 @@ def graphOfData(logf, dtstr):
 
 
 def getLogStats(nightlogf, matchlogf, thisdy):
-    with open(nightlogf,'r') as inf:
-        loglines = inf.readlines()
-    
-    #logf = os.path.basename(nightlogf)
-    #spls = logf.split('-')
-    #thisdy= spls[1]
-    matchstr = 'RUNTIME'    
-
-    # <13>Nov  1 11:49:55 nightlyJob: RUNTIME 10554 showerReport ALL 202211
-    startFAM = 0
-    startRD = 0
-    dts = []
-    tss = []
-    tsks = []
-    secs = []
-    msgs = []
-    for li in loglines:
-        if matchstr in li:
-            li = ' '.join(li.split()) # remove double-spaces eg when date is ' 6 Jan' as opposed to '16 Jan'
-            spls = li.split()
-            msg = ''
-            for s in range(6, len(spls)):
-                msg = msg + ' ' + spls[s]
-            if 'start findAllMatches' in msg:
-                startFAM = int(spls[5])
-            dts.append(thisdy)
-            tss.append(spls[2])
-            tsks.append(spls[3].replace(':',''))
-            secs.append(int(spls[5]))
-            msgs.append(msg.strip())
-
-    with open(matchlogf,'r') as inf:
-        loglines = inf.readlines()
+    # logline example
+    # <13>May  8 06:10:02 nightlyJob: start nightlyJob
+    outdir = os.path.split(nightlogf)[0]
 
-    # scan for findAllMatches data first
-    for li in loglines:
-        if matchstr in li:
-            spls = li.split()
-            if 'runDistrib' in spls[3]: 
-                continue
-            msg = ''
-            for s in range(6, len(spls)):
-                msg = msg + ' ' + spls[s]
-            if 'start runDistrib' in msg:
-                startRD = int(spls[5])
-            dts.append(thisdy)
-            tss.append(spls[2])
-            tsks.append(spls[3].replace(':',''))
-            secs.append(int(spls[5])+startFAM)
-            msgs.append(msg.strip())
+    loglines = open(nightlogf,'r').readlines()
+    bsfs = [x for x in loglines if 'start'in x or 'finish' in x or 'end' in x]
+    bsfs = [x for x in bsfs if x[0]=='<']
 
-    # rescan for runDistrib data
-    for li in loglines:
-        if matchstr in li:
-            spls = li.split()
-            if 'runDistrib' not in spls[3]: 
-                continue
-            msg = ''
-            for s in range(6, len(spls)):
-                msg = msg + ' ' + spls[s]
-            dts.append(thisdy)
-            tss.append(spls[2])
-            tsks.append(spls[3].replace(':',''))
-            secs.append(int(spls[5]) + startFAM + startRD)
-            msgs.append(msg.strip())
+    loglines = open(matchlogf,'r').readlines()
+    msfs = [x for x in loglines if 'start'in x or 'finish' in x or 'end' in x]
+    msfs = [x for x in msfs if x[0]=='<']
 
-    df = pd.DataFrame(zip(dts,tss,secs,tsks,msgs), columns=['dts','tss','secs','tsk','msgs'])
-    df = df.sort_values(by=['tss'])
-    #print(df)
-    outdir = os.path.split(nightlogf)[0]
-    df.to_csv(os.path.join(outdir, 'perfNightly.csv'), mode='a', header=False, index=False)
-    
+    alldata = msfs + bsfs
+    alldata.sort()
+    starttime = None
+    yr = datetime.datetime.now().year
+    with open(os.path.join(outdir, 'perfNightly.csv'), 'a+') as outf:
+        for rw in alldata:
+            dtpart = rw[4:19]
+            evtdt = datetime.datetime.strptime(f'{yr} {dtpart}', '%Y %b %d %H:%M:%S')
+            if 'start nightlyJob' in rw:
+                starttime = evtdt
+            elapsed_secs = (evtdt - starttime).seconds
+            txtpart = rw[20:]
+            task = txtpart[:txtpart.find(':')]
+            msg = txtpart[txtpart.find(':')+1:].replace(',', ' ')
+            outstr = f'{evtdt.strftime("%Y%m%d,%H:%M:%S")},{elapsed_secs},{task},{msg}'
+            outf.write(outstr)        
+  
 
 if __name__ == '__main__':
     dtstr = sys.argv[1]

archive/ukmon_pylib/reports/findFailedMatches.py

@@ -16,24 +16,27 @@ def saveData(observation, repfile):
 
 
 def processOneLog(logfile, repfile):
-    with open(logfile, 'r') as inf:
-        lis = inf.readlines()
+    lis = open(logfile, 'r').readlines()
 
     obs = False
     observation = []
     for li in lis:
-        if 'Observations:' in li:
+        if 'Observations ' in li:
             obs = True
         if obs is True:
             observation.append(li)
-        if 'Shower:' in li:
+        if 'Saving trajectory:' in li or 'Updating database' in li:
             obs = False
             observation = []
-        if '-----------------------' in li:
+        if 'Updating database' in li or 'added to fails' in li:
             if obs is True:
+                #print('got fail')
+                observation.append('--------------\n\n')
                 saveData(observation, repfile)
             obs = False
             observation = []
+        if "SOLVING RUN DONE" in li:
+            break
 
     return
 
@@ -48,5 +51,6 @@ def processOneLog(logfile, repfile):
     logdir = os.path.join(srcdir, 'logs','distrib')
     logs = glob.glob(f'{repdt}*.log', root_dir=logdir)
     for logf in logs:
+        #print(logf)
         processOneLog(os.path.join(logdir, logf), reportfile)
     reportfile.close()