Latest updates (#470)

markmac99 · web-flow · commit d13be369c540 · 2026-05-07T01:04:24.000+01:00
* simplify calcserver start up * sorting out errors in container logging * update container logging info * avoid unnecessary error * fix bugs in log capture process * add MAX_STNS so i can experiement with limits * add logging. Add max_stns to constraints and set to 9999 push logs to s3 copy calcserver logs to ukmonhelper * don't pull or push the databases from S3 during distribution but do back them up after consolidation * catch error if trajpickle can't be found * set rundate more safely move some processes into nightlyJob to make testing easier * tidy up backup file names and housekeep backup folder * move some processes from findAllMatches to nightlyJob to make testing easier * support for testing of createExecReplot * pass testmode to createExecReplot * ensure logs are unique and capture container logs * realigning createFireballPage with prod. Updating gitignore * remove verbosity flag from one tar operation * fixup to handle new style log with datestamps * get privip after restarting calcserver * temporarily don't run rerunFailedLambdas in test * update to work in a test env too * add 'period' to the match summary api * add period to match summary api * re-enabling rerunFailedLambdas * quieten an S3 transfer * include the foldername when reporting on container progress * put daily trajdb in a safe place * be more specific when finding the daily log * only tar the current container logs * save daily obsdb in the right place * rename ec2.tf to calcserver.tf * quieten some s3 transfers * rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is * update SSM params to reflect new serverid * bugfix in distributeCandidates - using wrong path * add newline to getCostMetrics * weird bug in statsToMqtt * bugfix in update_wmpl.sh for traj container * Script to create an AMI for the calcengine * exclude test data when syncing from s3 * writing databases and logs to the wrong target * bad test on istest * typo in serveruserid * adding to server migration notes * purge old rawcsvs and fullcsvs older than 180 days * improved housekeeping of old data * make sure we scoop up data for 31/12 * stop creation of an unused set of files * change default locations for container data * put container logs in the right place * update dockerfile syntax * change locations of test container data and output * remove incorrect comments * Add script to clean up deleted/duplicate trajs * simplify the stats gathering and move to python fix bug in getMatchStats * comment out unused files * update reportOfLatestMatches to exclude deleted trajectories * add scp to requirements * add SQL database maintenance to dataMaintenance * incorrect default for datadir * update call to pymysql.connect * ensure consolidateOutput rerunnable for prior years * upgrading WMPL * update gitignore a bit * bugfix to catch data from 31/12 each year * add paramiko to dependencies for matchPickle to support wmpl upgrade * update getpickle to newest WMPL * bugfix in api tests * bugfix - changes to dailydb not saved * improvements to process to cleanup deleted trajs * after merging, check and remove any duplicate trajectories from the sqlite db and disk * Adding version * bump version 2025.11.1 -> 2026.04.0 * update files containing vesion * bump version 2026.04.0 -> 2026.04.1 * unmangling side effets of automerge * correct usage message * decommission old calcserver and add new batchserver * add calcserver IP to ssm variables and use it in scripts * Update my branch with the latest from master (#436) * Update version (#433) * remove disused code * update livestream to accept normal RMS filenames * search dialog - fix bug and set default date range * small change to explain default dates * improvement in live image search * remove redundant file * update deployment scripts * improvements in routine to convert GMN report to pandas * more analysis of gmn data * Update the terraform a little * remove hardcoded reference to ec2-user * add SERVERUSERID env /ssm variable for calcserver user id * update container for new distrib-processing method * use calcserver SERVERUSERID variable * support for test ECS containers * fix paths in deployment scripts * remove unused file * missed some templates * update dev/test env settings * remove disused file * make sure paths are expanded * fixing various config issues * updates to makeConfig * updates to distributed processing to align with newest wmpl * add a comment to explain what the code does * remove json db * update consolidation to handle sqlite dbs * remove unused function * create data for daily report * remove unused file * update to call new version of reportOfLatestMatches * add test variables * mildly improve doco * put test logs in the right CW log group * handle missing log groups better * extra value in TrajQualityParams * bugfixes revealed in testing * fix issues in reportOfLatestMatches * put container dbs in a sensible place * catch logging errors * tidy up file locations * simplify calcserver start up * sorting out errors in container logging * update container logging info * avoid unnecessary error * fix bugs in log capture process * add MAX_STNS so i can experiement with limits * add logging. Add max_stns to constraints and set to 9999 push logs to s3 copy calcserver logs to ukmonhelper * don't pull or push the databases from S3 during distribution but do back them up after consolidation * catch error if trajpickle can't be found * set rundate more safely move some processes into nightlyJob to make testing easier * tidy up backup file names and housekeep backup folder * move some processes from findAllMatches to nightlyJob to make testing easier * support for testing of createExecReplot * pass testmode to createExecReplot * ensure logs are unique and capture container logs * realigning createFireballPage with prod. Updating gitignore * remove verbosity flag from one tar operation * fixup to handle new style log with datestamps * get privip after restarting calcserver * temporarily don't run rerunFailedLambdas in test * update to work in a test env too * add 'period' to the match summary api * add period to match summary api * re-enabling rerunFailedLambdas * quieten an S3 transfer * include the foldername when reporting on container progress * put daily trajdb in a safe place * be more specific when finding the daily log * only tar the current container logs * save daily obsdb in the right place * rename ec2.tf to calcserver.tf * quieten some s3 transfers * rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is * update SSM params to reflect new serverid * bugfix in distributeCandidates - using wrong path * add newline to getCostMetrics * weird bug in statsToMqtt * bugfix in update_wmpl.sh for traj container * Script to create an AMI for the calcengine * exclude test data when syncing from s3 * writing databases and logs to the wrong target * bad test on istest * typo in serveruserid * adding to server migration notes * purge old rawcsvs and fullcsvs older than 180 days * improved housekeeping of old data * make sure we scoop up data for 31/12 * stop creation of an unused set of files * change default locations for container data * put container logs in the right place * update dockerfile syntax * change locations of test container data and output * remove incorrect comments * Add script to clean up deleted/duplicate trajs * simplify the stats gathering and move to python fix bug in getMatchStats * comment out unused files * update reportOfLatestMatches to exclude deleted trajectories * add scp to requirements * add SQL database maintenance to dataMaintenance * incorrect default for datadir * update call to pymysql.connect * ensure consolidateOutput rerunnable for prior years * upgrading WMPL * update gitignore a bit * bugfix to catch data from 31/12 each year * add paramiko to dependencies for matchPickle to support wmpl upgrade * update getpickle to newest WMPL * bugfix in api tests * bugfix - changes to dailydb not saved * improvements to process to cleanup deleted trajs * after merging, check and remove any duplicate trajectories from the sqlite db and disk * Adding version * bump version 2025.11.1 -> 2026.04.0 * update files containing vesion * bump version 2026.04.0 -> 2026.04.1 * unmangling side effets of automerge * correct usage message * adding install script * changes to ansible scripts * remove irrelevant file * Adding SSM variables for the new batchserver in the ukmda account * attempt to add logging to consolidateDistTraj * remove reference to ee account * temporarily add mariadb deets to ssm in mjmm account * bugfixes in dataMaintenance and cleanupDeletedTrajs to avoid deleting disk files when trajs share same folder * updating migration instructions * more work on deduplication for ukmon * remove username-specific stuff * remove the old deployment scripts * update install script * updates to improve server migration process * add bashrc to the installer * slight tweak to installer * add a few installer messages * remove an echo * be less verbose in the rsyncs * oops need -a for one of the rsyncs * rename requirements file for ukmda * remove dependencies on MeteorTools * add utils to ukmon_pylib * more removal of dependencies on MeteorTools * updating how shower data is obtained * corercting location of IMO and gmn shower data * bugfixes * fixes in imoWorkinShowerlist * small error in install_or_update * bugfix when no data found in imo set * retrieve IMO data after each code update * fixing up getShowerDets * improve installer * fix createShowerExtracts * update extractors and showeranalysis * bug in extractor thats existed since 2023 but nobody noticed... * trying to fix matplotlib bug * more working on matplotlib issue * oops overfixed a bug * another incorrect bugfix! * make bars wider * more removal of MeteorTools * bugfix in createannualbarchart * working on bug * more work on lenght bug * remove some debugging * oops forgot to push the fixed version of createReportIndex * segregating function to get list of images that weren't calibrated by RMS * reorganize stuff a bit * add email sending functionality to ukmon_pylib * use send email functionality in code * add database and gmaps keys/passwords as secure variables * update to use secure keys * remove dbpw from environment * remove dbpw from environment * add script to retrieve db passwd * add script to retrieve db passwords * bug in createTables * updates to migration instructions * include pip installs in the script * tweak installer * remove glob.glob1 * move unused scripts - will delete later * fix mariadb connections * typo * tidy up docs * fixing mysql calls * small changes to documentation * suppress pip messages if already installed * tweaks to the installer * add logrotate stuff * bugfixes in installer * tidy up installer * bugfix in key mgmt * move some functionality into new script to tidy up nightly job and simplify rerunning * improve install script * update location of camera count file * move annual bar charts into a more logical location * update test script * add gmaps param to my acct * relocate coverage maps and front page index * tidy up useraudit process * bugfixes in testing * bugfix in createAnnualBarchart * bug in testscript * another small bug in createSummaryTable * still fighting that bug * tidy up lastlog data * remove unused file * update cam status reporting * clear down the old brightness data * simplification of logging and of the nightlyJob script * bugfixes in process to gather consumed image ifo * update getUncal to work for historic data * typo * improvements in getUncalImages * bugfix * add python to get unused images * move script to right place * comment out unnecessary echo * bugfixes in camMetrics while migrating user accts * fix #448 * make sure window title is sensible * bugfix in replot and get unused * bugfix in removeDeletedTraj for search index * remove deprecation warning issue #450 * whoops naive vs tzaware dates * remove warning in cameraStatusReport issue #453 * remove warning issue #454 * widen constraints on analysing pickles to get images * add migration script * add sudo * actually do it * update sftp migration script * more changes to migration script * moving sftp migration script * update messages and docos * slight tweak to migration script * further work on sftp account migration * update ukmon ini files after migration * typo * collect old server login records integration with new * update migration instructions * update cam metrics report to include hostname * bugfix as $USER not available in cron * removing unused files * some tidying up * motre reorg * remove disused template rename file to follow convention * simplifying logging and removing unused aws profile * removing profile from python * adding executable bit * typo * bugfix to work around typo * print message when done * bugfix to get trajectories from s3 before trying to check for missing trajectories... * improve to support running for past dates * errors in bashrc and logrotate config * prevent email being sent for anything other than current days report * bugfix: make sure we use the most recent stats * improvement in reportOfLatestMatches to handle past dates * bugfix * small tweak to support reportOfLatestMatches * tweak getMatchStats to handle historic data better * add mariadb and ping to the sec grp so that the database can be accessed by apis * update the db servername * bugfix in match stats * prevent bogus renaming of matchJob.log * bumping version * bump version 2026.04.1 -> 2026.05.0 * update matchapi to use indexed column * add api readme * tweak to cammetrics to enhance stationlogins report * slight tweak to formatting * minor tweak to avoid unexpectedly changing directory * add support script for during server migrations * skip 1st row * add check for todos * oops strip whitespace * darn whitespace * working on migration tasks * minor tweaks * daft typo * bump version 2026.05.0 -> 2026.05.1 * bump version 2026.05.1 -> 2026.05.2 * update CodeQL to v3 * bump codeQL to v4
diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml
@@ -39,11 +39,11 @@ jobs:
 
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
 
       # Initializes the CodeQL tools for scanning.
       - name: Initialize CodeQL
-        uses: github/codeql-action/init@v2
+        uses: github/codeql-action/init@v4
         with:
           languages: ${{ matrix.language }}
           # If you wish to specify custom queries, you can do so here or in a config file.
@@ -54,7 +54,7 @@ jobs:
       # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
       # If this step fails, then you should remove it and run the build manually (see below)
       - name: Autobuild
-        uses: github/codeql-action/autobuild@v2
+        uses: github/codeql-action/autobuild@v4
 
       # ℹ️ Command-line programs to run using the OS shell.
       # 📚 https://git.io/JvXDl
@@ -68,4 +68,4 @@ jobs:
       #   make release
 
       - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@v2
+        uses: github/codeql-action/analyze@v4
diff --git a/README.md b/README.md
@@ -1,5 +1,5 @@
 # UK Meteor Data Analysis Shared code and libraries
-version: 2026.05.0
+version: 2026.05.2
 
 This repository contains the code behind the UK Meteors data archive and data processing pipeline. 
 
diff --git a/archive/README.md b/archive/README.md
@@ -1,7 +1,7 @@
 Data Processing and Flows
 ==========================
 
-version: 2026.05.0
+version: 2026.05.2
 
 This diagram shows the overall flow of data from Cameras to websites and out to the public.
 
diff --git a/archive/server_setup/acctStatusChecks.sh b/archive/server_setup/acctStatusChecks.sh
@@ -0,0 +1,13 @@
+#!/bin/bash
+
+here="$( cd "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
+
+cd $here
+
+sudo ls -1 /var/sftp > done.txt
+grep ukmonh /home/ubuntu/prod/data/reports/stationlogins.txt |awk '{print $3}'| sort > pending.txt
+
+ssh ukmonhelper2 "~/prod/server_setup/get-nbd.sh" | while read i ; do echo $i | awk -F"/" '{print $4}' ; done > not-being-done.txt
+ssh ukmonhelper2 "~/prod/server_setup/get-all.sh" | while read i ; do echo $i | awk -F"/" '{print $4}' ; done > all-accounts.txt
+
+python ~/src/ukmda-dataprocessing/archive/server_setup/checkSftpAccounts.py
diff --git a/archive/server_setup/backupSftpAccts.sh b/archive/server_setup/backupSftpAccts.sh
@@ -0,0 +1,30 @@
+#!/bin/bash
+
+here="$( cd "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
+cd $here
+
+bkpOneUser() {
+    userid=$1
+    srchost=$2
+    mkdir -p ./backup/$userid
+    sudo rsync -av $srchost:/var/sftp/$userid/ ./backup/$userid 
+    sudo chown -R $USER:$USER ./backup/$userid
+    cat ./backup/$userid/ukmon.ini | sed 's/3.11.55.160/batchserver.ukmeteors.co.uk/g' > /tmp/$userid.ini
+    mv -f /tmp/$userid.ini ./backup/$userid/ukmon.ini
+}
+
+if [ $# -lt 2 ] ; then 
+    echo "Usage: ./backupSftpAccounts.sh oldservername userfile"
+    exit
+fi 
+
+echo "Warning: this must only be run on the new server"
+read -p "press ctrl-c to quit or enter to continue"
+
+oldserver=$1
+srcfile=$2
+
+cat $srcfile | while read stn
+do 
+    bkpOneUser $stn $oldserver
+done
diff --git a/archive/server_setup/checkSftpAccounts.py b/archive/server_setup/checkSftpAccounts.py
@@ -0,0 +1,65 @@
+import pandas as pd
+import datetime
+
+
+data=open('/home/ubuntu/prod/data/reports/stationlogins.txt').readlines()
+camlist = []
+not_live = []
+still_upl = []
+not_upl = []
+livenames = []
+
+livedate = datetime.datetime.now() - datetime.timedelta(days=10)
+for li in data:
+	if 'Last Upload' in li:
+		continue
+	lastup = li[:19]
+	lastlo = li[21:40]
+	loc = li[42:61]
+	gmnid = li[64:73]
+	via = li[76:].strip()
+	camlist.append([lastup, lastlo, loc, gmnid, via])
+
+	if ">" in lastup and ">" in lastlo:
+		not_live.append([loc, gmnid, lastup, lastlo, via])
+
+	elif ">" not in lastup:
+		lastupdt = datetime.datetime.strptime(lastup, '%Y-%m-%dT%H:%M:%S')
+		if lastupdt >= livedate:
+			still_upl.append([loc, gmnid, lastup, lastlo, via])
+			livenames.append(loc.strip())
+		else:
+			if '>' in lastlo:
+				not_live.append([loc, gmnid, lastup, lastlo, via])
+			else:
+				lastlodt = datetime.datetime.strptime(lastlo, '%Y-%m-%dT%H:%M:%S')
+				if lastlodt >= livedate:
+					not_upl.append([loc, gmnid, lastup, lastlo, via])
+				else:
+					not_live.append([loc, gmnid, lastup, lastlo, via])
+	else: # '>' not in lastlo
+		lastlodt = datetime.datetime.strptime(lastlo, '%Y-%m-%dT%H:%M:%S')
+		if lastlodt >= livedate:
+			not_upl.append([loc, gmnid, lastup, lastlo, via])
+		else:
+			not_live.append([loc, gmnid, lastup, lastlo, via])
+
+with open('still-live.txt','w') as outf:
+	for cam in still_upl:
+		outf.write(','.join(cam) + '\n')
+
+with open('not_uploading.txt','w') as outf:
+	for cam in not_upl:
+		outf.write(','.join(cam) + '\n')
+
+with open('not_live.txt','w') as outf:
+	for cam in not_live:
+		outf.write(','.join(cam) + '\n')
+
+donelist = open('done.txt', 'r').readlines()
+donelist = [x.strip() for x in donelist]
+with open('todo.txt', 'w') as outf:
+	for nam in livenames:
+		if nam.strip() not in donelist:
+			print('done list is missing', nam)
+			outf.write(f'{nam}\n')
diff --git a/archive/server_setup/copyTestData.sh b/archive/server_setup/copyTestData.sh
diff --git a/archive/server_setup/migrateSftpAccts.sh b/archive/server_setup/migrateSftpAccts.sh
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+here="$( cd "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
+cd $here
 
 addOneUser() {
     userid=$1
diff --git a/archive/ukmon_pylib/metrics/camMetrics.py b/archive/ukmon_pylib/metrics/camMetrics.py
@@ -206,15 +206,15 @@ def backPopulate(stationid):
         df['uploadtime']=df.uploadtime.astype("str").str.pad(6,fillchar="0")
         df['lastupload']=df.upddate.astype('str') + '_' +df.uploadtime
         df.lastupload = [datetime.datetime.strptime(x, '%Y%m%d_%H%M%S') for x in df.lastupload]
-        df = df.drop(columns=['stationid','manual','rundate', 'upddate','uploadtime'])
+        df = df.drop(columns=['manual','rundate', 'upddate','uploadtime'])
         df = df.sort_values(by=['lastupload'])
 
         outfile=os.path.join(datadir, 'reports', 'stationlogins.txt')
         zerodate = datetime.datetime(1970,1,1,0,0,0)
         with open(outfile,'w') as outf:
-            outf.write('Last Upload,          StationID,            Last Login,          Via\n')
+            outf.write(f'{"Last Upload":19s}  {"Last Connect":19s}  {"StationID":20s}  {"GMN ID":10s}  Via\n')
             for _,rw in df.iterrows():
-                lastup = '> 1 month' if pd.isnull(rw.lastupload) else rw.lastupload.strftime('%Y-%m-%dT%H:%M:%S')
-                lastlo = '> 1 month' if pd.isnull(rw.lastseen) else rw.lastseen.strftime('%Y-%m-%dT%H:%M:%S')
+                lastup = '> 1 month ago' if pd.isnull(rw.lastupload) else rw.lastupload.strftime('%Y-%m-%dT%H:%M:%S')
+                lastlo = '> 1 month ago' if pd.isnull(rw.lastseen) else rw.lastseen.strftime('%Y-%m-%dT%H:%M:%S')
                 via = '' if pd.isnull(rw.host) else rw.host
-                outf.write(f'{lastup} , {rw.location:20s}, {lastlo}, {via}\n')
+                outf.write(f'{lastup:19s}  {lastlo:19s}  {rw.location:20s}  {rw.stationid:10s}  {via}\n')
diff --git a/bumpver.toml b/bumpver.toml
@@ -1,5 +1,5 @@
 [bumpver]
-current_version = "2026.05.0"
+current_version = "2026.05.2"
 version_pattern = "YYYY.0M.PATCH"
 commit_message = "bump version {old_version} -> {new_version}"
 commit = true
diff --git a/install_or_update.sh b/install_or_update.sh
@@ -10,8 +10,8 @@ envname=$(echo $RUNTIME_ENV | tr '[:upper:]' '[:lower:]')
 
 here="$( cd "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"
 
+pushd $here/archive
 echo "Updating codebase..."
-cd $here/archive
 git pull
 
 echo "Updating code..."
@@ -32,6 +32,11 @@ mkdir -p $DATADIR/browse/{annual,monthly,daily,showers}
 mkdir -p ~/$envname/logs
 mkdir -p ~/.logrotate
 mkdir -p ~/.aws
+mkdir -p ~/server_setup
+
+rsync -a server_setup/*.sh ~/server_setup
+rsync -a server_setup/*.py ~/server_setup
+chmod +x ~/server_setup/*.sh
 
 echo "Checking conda environment..."
 if [[ -f ~/.condaon  && -d ~/miniconda3/envs/wmpl ]]
@@ -69,4 +74,5 @@ else
     echo skipping config and bashrc
 fi 
 echo ""
+popd
 echo "$msg complete"
