Conversation
Add max_stns to constraints and set to 9999 push logs to s3 copy calcserver logs to ukmonhelper
but do back them up after consolidation
move some processes into nightlyJob to make testing easier
github-project-automation
Bot
moved this from Backlog
to Done
in Archive Website and Data
markmac99
added a commit
that referenced
this pull request
May 27, 2026
* Minor tweaks (#469) * add a comment to explain what the code does * remove json db * update consolidation to handle sqlite dbs * remove unused function * create data for daily report * remove unused file * update to call new version of reportOfLatestMatches * add test variables * mildly improve doco * put test logs in the right CW log group * handle missing log groups better * extra value in TrajQualityParams * bugfixes revealed in testing * fix issues in reportOfLatestMatches * put container dbs in a sensible place * catch logging errors * tidy up file locations * simplify calcserver start up * sorting out errors in container logging * update container logging info * avoid unnecessary error * fix bugs in log capture process * add MAX_STNS so i can experiement with limits * add logging. Add max_stns to constraints and set to 9999 push logs to s3 copy calcserver logs to ukmonhelper * don't pull or push the databases from S3 during distribution but do back them up after consolidation * catch error if trajpickle can't be found * set rundate more safely move some processes into nightlyJob to make testing easier * tidy up backup file names and housekeep backup folder * move some processes from findAllMatches to nightlyJob to make testing easier * support for testing of createExecReplot * pass testmode to createExecReplot * ensure logs are unique and capture container logs * realigning createFireballPage with prod. Updating gitignore * remove verbosity flag from one tar operation * fixup to handle new style log with datestamps * get privip after restarting calcserver * temporarily don't run rerunFailedLambdas in test * update to work in a test env too * add 'period' to the match summary api * add period to match summary api * re-enabling rerunFailedLambdas * quieten an S3 transfer * include the foldername when reporting on container progress * put daily trajdb in a safe place * be more specific when finding the daily log * only tar the current container logs * save daily obsdb in the right place * rename ec2.tf to calcserver.tf * quieten some s3 transfers * rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is * update SSM params to reflect new serverid * bugfix in distributeCandidates - using wrong path * add newline to getCostMetrics * weird bug in statsToMqtt * bugfix in update_wmpl.sh for traj container * Script to create an AMI for the calcengine * exclude test data when syncing from s3 * writing databases and logs to the wrong target * bad test on istest * typo in serveruserid * adding to server migration notes * purge old rawcsvs and fullcsvs older than 180 days * improved housekeeping of old data * make sure we scoop up data for 31/12 * stop creation of an unused set of files * change default locations for container data * put container logs in the right place * update dockerfile syntax * change locations of test container data and output * remove incorrect comments * Add script to clean up deleted/duplicate trajs * simplify the stats gathering and move to python fix bug in getMatchStats * comment out unused files * update reportOfLatestMatches to exclude deleted trajectories * add scp to requirements * add SQL database maintenance to dataMaintenance * incorrect default for datadir * update call to pymysql.connect * ensure consolidateOutput rerunnable for prior years * upgrading WMPL * update gitignore a bit * bugfix to catch data from 31/12 each year * add paramiko to dependencies for matchPickle to support wmpl upgrade * update getpickle to newest WMPL * bugfix in api tests * bugfix - changes to dailydb not saved * improvements to process to cleanup deleted trajs * after merging, check and remove any duplicate trajectories from the sqlite db and disk * Adding version * bump version 2025.11.1 -> 2026.04.0 * update files containing vesion * bump version 2026.04.0 -> 2026.04.1 * unmangling side effets of automerge * correct usage message * decommission old calcserver and add new batchserver * add calcserver IP to ssm variables and use it in scripts * Update my branch with the latest from master (#436) * Update version (#433) * remove disused code * update livestream to accept normal RMS filenames * search dialog - fix bug and set default date range * small change to explain default dates * improvement in live image search * remove redundant file * update deployment scripts * improvements in routine to convert GMN report to pandas * more analysis of gmn data * Update the terraform a little * remove hardcoded reference to ec2-user * add SERVERUSERID env /ssm variable for calcserver user id * update container for new distrib-processing method * use calcserver SERVERUSERID variable * support for test ECS containers * fix paths in deployment scripts * remove unused file * missed some templates * update dev/test env settings * remove disused file * make sure paths are expanded * fixing various config issues * updates to makeConfig * updates to distributed processing to align with newest wmpl * add a comment to explain what the code does * remove json db * update consolidation to handle sqlite dbs * remove unused function * create data for daily report * remove unused file * update to call new version of reportOfLatestMatches * add test variables * mildly improve doco * put test logs in the right CW log group * handle missing log groups better * extra value in TrajQualityParams * bugfixes revealed in testing * fix issues in reportOfLatestMatches * put container dbs in a sensible place * catch logging errors * tidy up file locations * simplify calcserver start up * sorting out errors in container logging * update container logging info * avoid unnecessary error * fix bugs in log capture process * add MAX_STNS so i can experiement with limits * add logging. Add max_stns to constraints and set to 9999 push logs to s3 copy calcserver logs to ukmonhelper * don't pull or push the databases from S3 during distribution but do back them up after consolidation * catch error if trajpickle can't be found * set rundate more safely move some processes into nightlyJob to make testing easier * tidy up backup file names and housekeep backup folder * move some processes from findAllMatches to nightlyJob to make testing easier * support for testing of createExecReplot * pass testmode to createExecReplot * ensure logs are unique and capture container logs * realigning createFireballPage with prod. Updating gitignore * remove verbosity flag from one tar operation * fixup to handle new style log with datestamps * get privip after restarting calcserver * temporarily don't run rerunFailedLambdas in test * update to work in a test env too * add 'period' to the match summary api * add period to match summary api * re-enabling rerunFailedLambdas * quieten an S3 transfer * include the foldername when reporting on container progress * put daily trajdb in a safe place * be more specific when finding the daily log * only tar the current container logs * save daily obsdb in the right place * rename ec2.tf to calcserver.tf * quieten some s3 transfers * rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is * update SSM params to reflect new serverid * bugfix in distributeCandidates - using wrong path * add newline to getCostMetrics * weird bug in statsToMqtt * bugfix in update_wmpl.sh for traj container * Script to create an AMI for the calcengine * exclude test data when syncing from s3 * writing databases and logs to the wrong target * bad test on istest * typo in serveruserid * adding to server migration notes * purge old rawcsvs and fullcsvs older than 180 days * improved housekeeping of old data * make sure we scoop up data for 31/12 * stop creation of an unused set of files * change default locations for container data * put container logs in the right place * update dockerfile syntax * change locations of test container data and output * remove incorrect comments * Add script to clean up deleted/duplicate trajs * simplify the stats gathering and move to python fix bug in getMatchStats * comment out unused files * update reportOfLatestMatches to exclude deleted trajectories * add scp to requirements * add SQL database maintenance to dataMaintenance * incorrect default for datadir * update call to pymysql.connect * ensure consolidateOutput rerunnable for prior years * upgrading WMPL * update gitignore a bit * bugfix to catch data from 31/12 each year * add paramiko to dependencies for matchPickle to support wmpl upgrade * update getpickle to newest WMPL * bugfix in api tests * bugfix - changes to dailydb not saved * improvements to process to cleanup deleted trajs * after merging, check and remove any duplicate trajectories from the sqlite db and disk * Adding version * bump version 2025.11.1 -> 2026.04.0 * update files containing vesion * bump version 2026.04.0 -> 2026.04.1 * unmangling side effets of automerge * correct usage message * adding install script * changes to ansible scripts * remove irrelevant file * Adding SSM variables for the new batchserver in the ukmda account * attempt to add logging to consolidateDistTraj * remove reference to ee account * temporarily add mariadb deets to ssm in mjmm account * bugfixes in dataMaintenance and cleanupDeletedTrajs to avoid deleting disk files when trajs share same folder * updating migration instructions * more work on deduplication for ukmon * remove username-specific stuff * remove the old deployment scripts * update install script * updates to improve server migration process * add bashrc to the installer * slight tweak to installer * add a few installer messages * remove an echo * be less verbose in the rsyncs * oops need -a for one of the rsyncs * rename requirements file for ukmda * remove dependencies on MeteorTools * add utils to ukmon_pylib * more removal of dependencies on MeteorTools * updating how shower data is obtained * corercting location of IMO and gmn shower data * bugfixes * fixes in imoWorkinShowerlist * small error in install_or_update * bugfix when no data found in imo set * retrieve IMO data after each code update * fixing up getShowerDets * improve installer * fix createShowerExtracts * update extractors and showeranalysis * bug in extractor thats existed since 2023 but nobody noticed... * trying to fix matplotlib bug * more working on matplotlib issue * oops overfixed a bug * another incorrect bugfix! * make bars wider * more removal of MeteorTools * bugfix in createannualbarchart * working on bug * more work on lenght bug * remove some debugging * oops forgot to push the fixed version of createReportIndex * segregating function to get list of images that weren't calibrated by RMS * reorganize stuff a bit * add email sending functionality to ukmon_pylib * use send email functionality in code * add database and gmaps keys/passwords as secure variables * update to use secure keys * remove dbpw from environment * remove dbpw from environment * add script to retrieve db passwd * add script to retrieve db passwords * bug in createTables * updates to migration instructions * include pip installs in the script * tweak installer * remove glob.glob1 * move unused scripts - will delete later * fix mariadb connections * typo * tidy up docs * fixing mysql calls * small changes to documentation * suppress pip messages if already installed * tweaks to the installer * add logrotate stuff * bugfixes in installer * tidy up installer * bugfix in key mgmt * move some functionality into new script to tidy up nightly job and simplify rerunning * improve install script * update location of camera count file * move annual bar charts into a more logical location * update test script * add gmaps param to my acct * relocate coverage maps and front page index * tidy up useraudit process * bugfixes in testing * bugfix in createAnnualBarchart * bug in testscript * another small bug in createSummaryTable * still fighting that bug * tidy up lastlog data * remove unused file * update cam status reporting * clear down the old brightness data * simplification of logging and of the nightlyJob script * bugfixes in process to gather consumed image ifo * update getUncal to work for historic data * typo * improvements in getUncalImages * bugfix * add python to get unused images * move script to right place * comment out unnecessary echo * bugfixes in camMetrics while migrating user accts * fix #448 * make sure window title is sensible * bugfix in replot and get unused * bugfix in removeDeletedTraj for search index * remove deprecation warning issue #450 * whoops naive vs tzaware dates * remove warning in cameraStatusReport issue #453 * remove warning issue #454 * widen constraints on analysing pickles to get images * add migration script * add sudo * actually do it * update sftp migration script * more changes to migration script * moving sftp migration script * update messages and docos * slight tweak to migration script * further work on sftp account migration * update ukmon ini files after migration * typo * collect old server login records integration with new * update migration instructions * update cam metrics report to include hostname * bugfix as $USER not available in cron * removing unused files * some tidying up * motre reorg * remove disused template rename file to follow convention * simplifying logging and removing unused aws profile * removing profile from python * adding executable bit * typo * bugfix to work around typo * print message when done * bugfix to get trajectories from s3 before trying to check for missing trajectories... * improve to support running for past dates * errors in bashrc and logrotate config * prevent email being sent for anything other than current days report * bugfix: make sure we use the most recent stats * improvement in reportOfLatestMatches to handle past dates * bugfix * small tweak to support reportOfLatestMatches * tweak getMatchStats to handle historic data better * add mariadb and ping to the sec grp so that the database can be accessed by apis * update the db servername * bugfix in match stats * prevent bogus renaming of matchJob.log * bumping version * bump version 2026.04.1 -> 2026.05.0 * update matchapi to use indexed column * add api readme * Latest updates (#470) * simplify calcserver start up * sorting out errors in container logging * update container logging info * avoid unnecessary error * fix bugs in log capture process * add MAX_STNS so i can experiement with limits * add logging. Add max_stns to constraints and set to 9999 push logs to s3 copy calcserver logs to ukmonhelper * don't pull or push the databases from S3 during distribution but do back them up after consolidation * catch error if trajpickle can't be found * set rundate more safely move some processes into nightlyJob to make testing easier * tidy up backup file names and housekeep backup folder * move some processes from findAllMatches to nightlyJob to make testing easier * support for testing of createExecReplot * pass testmode to createExecReplot * ensure logs are unique and capture container logs * realigning createFireballPage with prod. Updating gitignore * remove verbosity flag from one tar operation * fixup to handle new style log with datestamps * get privip after restarting calcserver * temporarily don't run rerunFailedLambdas in test * update to work in a test env too * add 'period' to the match summary api * add period to match summary api * re-enabling rerunFailedLambdas * quieten an S3 transfer * include the foldername when reporting on container progress * put daily trajdb in a safe place * be more specific when finding the daily log * only tar the current container logs * save daily obsdb in the right place * rename ec2.tf to calcserver.tf * quieten some s3 transfers * rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is * update SSM params to reflect new serverid * bugfix in distributeCandidates - using wrong path * add newline to getCostMetrics * weird bug in statsToMqtt * bugfix in update_wmpl.sh for traj container * Script to create an AMI for the calcengine * exclude test data when syncing from s3 * writing databases and logs to the wrong target * bad test on istest * typo in serveruserid * adding to server migration notes * purge old rawcsvs and fullcsvs older than 180 days * improved housekeeping of old data * make sure we scoop up data for 31/12 * stop creation of an unused set of files * change default locations for container data * put container logs in the right place * update dockerfile syntax * change locations of test container data and output * remove incorrect comments * Add script to clean up deleted/duplicate trajs * simplify the stats gathering and move to python fix bug in getMatchStats * comment out unused files * update reportOfLatestMatches to exclude deleted trajectories * add scp to requirements * add SQL database maintenance to dataMaintenance * incorrect default for datadir * update call to pymysql.connect * ensure consolidateOutput rerunnable for prior years * upgrading WMPL * update gitignore a bit * bugfix to catch data from 31/12 each year * add paramiko to dependencies for matchPickle to support wmpl upgrade * update getpickle to newest WMPL * bugfix in api tests * bugfix - changes to dailydb not saved * improvements to process to cleanup deleted trajs * after merging, check and remove any duplicate trajectories from the sqlite db and disk * Adding version * bump version 2025.11.1 -> 2026.04.0 * update files containing vesion * bump version 2026.04.0 -> 2026.04.1 * unmangling side effets of automerge * correct usage message * decommission old calcserver and add new batchserver * add calcserver IP to ssm variables and use it in scripts * Update my branch with the latest from master (#436) * Update version (#433) * remove disused code * update livestream to accept normal RMS filenames * search dialog - fix bug and set default date range * small change to explain default dates * improvement in live image search * remove redundant file * update deployment scripts * improvements in routine to convert GMN report to pandas * more analysis of gmn data * Update the terraform a little * remove hardcoded reference to ec2-user * add SERVERUSERID env /ssm variable for calcserver user id * update container for new distrib-processing method * use calcserver SERVERUSERID variable * support for test ECS containers * fix paths in deployment scripts * remove unused file * missed some templates * update dev/test env settings * remove disused file * make sure paths are expanded * fixing various config issues * updates to makeConfig * updates to distributed processing to align with newest wmpl * add a comment to explain what the code does * remove json db * update consolidation to handle sqlite dbs * remove unused function * create data for daily report * remove unused file * update to call new version of reportOfLatestMatches * add test variables * mildly improve doco * put test logs in the right CW log group * handle missing log groups better * extra value in TrajQualityParams * bugfixes revealed in testing * fix issues in reportOfLatestMatches * put container dbs in a sensible place * catch logging errors * tidy up file locations * simplify calcserver start up * sorting out errors in container logging * update container logging info * avoid unnecessary error * fix bugs in log capture process * add MAX_STNS so i can experiement with limits * add logging. Add max_stns to constraints and set to 9999 push logs to s3 copy calcserver logs to ukmonhelper * don't pull or push the databases from S3 during distribution but do back them up after consolidation * catch error if trajpickle can't be found * set rundate more safely move some processes into nightlyJob to make testing easier * tidy up backup file names and housekeep backup folder * move some processes from findAllMatches to nightlyJob to make testing easier * support for testing of createExecReplot * pass testmode to createExecReplot * ensure logs are unique and capture container logs * realigning createFireballPage with prod. Updating gitignore * remove verbosity flag from one tar operation * fixup to handle new style log with datestamps * get privip after restarting calcserver * temporarily don't run rerunFailedLambdas in test * update to work in a test env too * add 'period' to the match summary api * add period to match summary api * re-enabling rerunFailedLambdas * quieten an S3 transfer * include the foldername when reporting on container progress * put daily trajdb in a safe place * be more specific when finding the daily log * only tar the current container logs * save daily obsdb in the right place * rename ec2.tf to calcserver.tf * quieten some s3 transfers * rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is * update SSM params to reflect new serverid * bugfix in distributeCandidates - using wrong path * add newline to getCostMetrics * weird bug in statsToMqtt * bugfix in update_wmpl.sh for traj container * Script to create an AMI for the calcengine * exclude test data when syncing from s3 * writing databases and logs to the wrong target * bad test on istest * typo in serveruserid * adding to server migration notes * purge old rawcsvs and fullcsvs older than 180 days * improved housekeeping of old data * make sure we scoop up data for 31/12 * stop creation of an unused set of files * change default locations for container data * put container logs in the right place * update dockerfile syntax * change locations of test container data and output * remove incorrect comments * Add script to clean up deleted/duplicate trajs * simplify the stats gathering and move to python fix bug in getMatchStats * comment out unused files * update reportOfLatestMatches to exclude deleted trajectories * add scp to requirements * add SQL database maintenance to dataMaintenance * incorrect default for datadir * update call to pymysql.connect * ensure consolidateOutput rerunnable for prior years * upgrading WMPL * update gitignore a bit * bugfix to catch data from 31/12 each year * add paramiko to dependencies for matchPickle to support wmpl upgrade * update getpickle to newest WMPL * bugfix in api tests * bugfix - changes to dailydb not saved * improvements to process to cleanup deleted trajs * after merging, check and remove any duplicate trajectories from the sqlite db and disk * Adding version * bump version 2025.11.1 -> 2026.04.0 * update files containing vesion * bump version 2026.04.0 -> 2026.04.1 * unmangling side effets of automerge * correct usage message * adding install script * changes to ansible scripts * remove irrelevant file * Adding SSM variables for the new batchserver in the ukmda account * attempt to add logging to consolidateDistTraj * remove reference to ee account * temporarily add mariadb deets to ssm in mjmm account * bugfixes in dataMaintenance and cleanupDeletedTrajs to avoid deleting disk files when trajs share same folder * updating migration instructions * more work on deduplication for ukmon * remove username-specific stuff * remove the old deployment scripts * update install script * updates to improve server migration process * add bashrc to the installer * slight tweak to installer * add a few installer messages * remove an echo * be less verbose in the rsyncs * oops need -a for one of the rsyncs * rename requirements file for ukmda * remove dependencies on MeteorTools * add utils to ukmon_pylib * more removal of dependencies on MeteorTools * updating how shower data is obtained * corercting location of IMO and gmn shower data * bugfixes * fixes in imoWorkinShowerlist * small error in install_or_update * bugfix when no data found in imo set * retrieve IMO data after each code update * fixing up getShowerDets * improve installer * fix createShowerExtracts * update extractors and showeranalysis * bug in extractor thats existed since 2023 but nobody noticed... * trying to fix matplotlib bug * more working on matplotlib issue * oops overfixed a bug * another incorrect bugfix! * make bars wider * more removal of MeteorTools * bugfix in createannualbarchart * working on bug * more work on lenght bug * remove some debugging * oops forgot to push the fixed version of createReportIndex * segregating function to get list of images that weren't calibrated by RMS * reorganize stuff a bit * add email sending functionality to ukmon_pylib * use send email functionality in code * add database and gmaps keys/passwords as secure variables * update to use secure keys * remove dbpw from environment * remove dbpw from environment * add script to retrieve db passwd * add script to retrieve db passwords * bug in createTables * updates to migration instructions * include pip installs in the script * tweak installer * remove glob.glob1 * move unused scripts - will delete later * fix mariadb connections * typo * tidy up docs * fixing mysql calls * small changes to documentation * suppress pip messages if already installed * tweaks to the installer * add logrotate stuff * bugfixes in installer * tidy up installer * bugfix in key mgmt * move some functionality into new script to tidy up nightly job and simplify rerunning * improve install script * update location of camera count file * move annual bar charts into a more logical location * update test script * add gmaps param to my acct * relocate coverage maps and front page index * tidy up useraudit process * bugfixes in testing * bugfix in createAnnualBarchart * bug in testscript * another small bug in createSummaryTable * still fighting that bug * tidy up lastlog data * remove unused file * update cam status reporting * clear down the old brightness data * simplification of logging and of the nightlyJob script * bugfixes in process to gather consumed image ifo * update getUncal to work for historic data * typo * improvements in getUncalImages * bugfix * add python to get unused images * move script to right place * comment out unnecessary echo * bugfixes in camMetrics while migrating user accts * fix #448 * make sure window title is sensible * bugfix in replot and get unused * bugfix in removeDeletedTraj for search index * remove deprecation warning issue #450 * whoops naive vs tzaware dates * remove warning in cameraStatusReport issue #453 * remove warning issue #454 * widen constraints on analysing pickles to get images * add migration script * add sudo * actually do it * update sftp migration script * more changes to migration script * moving sftp migration script * update messages and docos * slight tweak to migration script * further work on sftp account migration * update ukmon ini files after migration * typo * collect old server login records integration with new * update migration instructions * update cam metrics report to include hostname * bugfix as $USER not available in cron * removing unused files * some tidying up * motre reorg * remove disused template rename file to follow convention * simplifying logging and removing unused aws profile * removing profile from python * adding executable bit * typo * bugfix to work around typo * print message when done * bugfix to get trajectories from s3 before trying to check for missing trajectories... * improve to support running for past dates * errors in bashrc and logrotate config * prevent email being sent for anything other than current days report * bugfix: make sure we use the most recent stats * improvement in reportOfLatestMatches to handle past dates * bugfix * small tweak to support reportOfLatestMatches * tweak getMatchStats to handle historic data better * add mariadb and ping to the sec grp so that the database can be accessed by apis * update the db servername * bugfix in match stats * prevent bogus renaming of matchJob.log * bumping version * bump version 2026.04.1 -> 2026.05.0 * update matchapi to use indexed column * add api readme * tweak to cammetrics to enhance stationlogins report * slight tweak to formatting * minor tweak to avoid unexpectedly changing directory * add support script for during server migrations * skip 1st row * add check for todos * oops strip whitespace * darn whitespace * working on migration tasks * minor tweaks * daft typo * bump version 2026.05.0 -> 2026.05.1 * bump version 2026.05.1 -> 2026.05.2 * update CodeQL to v3 * bump codeQL to v4 * Trivial changes mainly to automation scripts (#471) * add MAX_STNS so i can experiement with limits * add logging. Add max_stns to constraints and set to 9999 push logs to s3 copy calcserver logs to ukmonhelper * don't pull or push the databases from S3 during distribution but do back them up after consolidation * catch error if trajpickle can't be found * set rundate more safely move some processes into nightlyJob to make testing easier * tidy up backup file names and housekeep backup folder * move some processes from findAllMatches to nightlyJob to make testing easier * support for testing of createExecReplot * pass testmode to createExecReplot * ensure logs are unique and capture container logs * realigning createFireballPage with prod. Updating gitignore * remove verbosity flag from one tar operation * fixup to handle new style log with datestamps * get privip after restarting calcserver * temporarily don't run rerunFailedLambdas in test * update to work in a test env too * add 'period' to the match summary api * add period to match summary api * re-enabling rerunFailedLambdas * quieten an S3 transfer * include the foldername when reporting on container progress * put daily trajdb in a safe place * be more specific when finding the daily log * only tar the current container logs * save daily obsdb in the right place * rename ec2.tf to calcserver.tf * quieten some s3 transfers * rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is * update SSM params to reflect new serverid * bugfix in distributeCandidates - using wrong path * add newline to getCostMetrics * weird bug in statsToMqtt * bugfix in update_wmpl.sh for traj container * Script to create an AMI for the calcengine * exclude test data when syncing from s3 * writing databases and logs to the wrong target * bad test on istest * typo in serveruserid * adding to server migration notes * purge old rawcsvs and fullcsvs older than 180 days * improved housekeeping of old data * make sure we scoop up data for 31/12 * stop creation of an unused set of files * change default locations for container data * put container logs in the right place * update dockerfile syntax * change locations of test container data and output * remove incorrect comments * Add script to clean up deleted/duplicate trajs * simplify the stats gathering and move to python fix bug in getMatchStats * comment out unused files * update reportOfLatestMatches to exclude deleted trajectories * add scp to requirements * add SQL database maintenance to dataMaintenance * incorrect default for datadir * update call to pymysql.connect * ensure consolidateOutput rerunnable for prior years * upgrading WMPL * update gitignore a bit * bugfix to catch data from 31/12 each year * add paramiko to dependencies for matchPickle to support wmpl upgrade * update getpickle to newest WMPL * bugfix in api tests * bugfix - changes to dailydb not saved * improvements to process to cleanup deleted trajs * after merging, check and remove any duplicate trajectories from the sqlite db and disk * Adding version * bump version 2025.11.1 -> 2026.04.0 * update files containing vesion * bump version 2026.04.0 -> 2026.04.1 * unmangling side effets of automerge * correct usage message * decommission old calcserver and add new batchserver * add calcserver IP to ssm variables and use it in scripts * Update my branch with the latest from master (#436) * Update version (#433) * remove disused code * update livestream to accept normal RMS filenames * search dialog - fix bug and set default date range * small change to explain default dates * improvement in live image search * remove redundant file * update deployment scripts * improvements in routine to convert GMN report to pandas * more analysis of gmn data * Update the terraform a little * remove hardcoded reference to ec2-user * add SERVERUSERID env /ssm variable for calcserver user id * update container for new distrib-processing method * use calcserver SERVERUSERID variable * support for test ECS containers * fix paths in deployment scripts * remove unused file * missed some templates * update dev/test env settings * remove disused file * make sure paths are expanded * fixing various config issues * updates to makeConfig * updates to distributed processing to align with newest wmpl * add a comment to explain what the code does * remove json db * update consolidation to handle sqlite dbs * remove unused function * create data for daily report * remove unused file * update to call new version of reportOfLatestMatches * add test variables * mildly improve doco * put test logs in the right CW log group * handle missing log groups better * extra value in TrajQualityParams * bugfixes revealed in testing * fix issues in reportOfLatestMatches * put container dbs in a sensible place * catch logging errors * tidy up file locations * simplify calcserver start up * sorting out errors in container logging * update container logging info * avoid unnecessary error * fix bugs in log capture process * add MAX_STNS so i can experiement with limits * add logging. Add max_stns to constraints and set to 9999 push logs to s3 copy calcserver logs to ukmonhelper * don't pull or push the databases from S3 during distribution but do back them up after consolidation * catch error if trajpickle can't be found * set rundate more safely move some processes into nightlyJob to make testing easier * tidy up backup file names and housekeep backup folder * move some processes from findAllMatches to nightlyJob to make testing easier * support for testing of createExecReplot * pass testmode to createExecReplot * ensure logs are unique and capture container logs * realigning createFireballPage with prod. Updating gitignore * remove verbosity flag from one tar operation * fixup to handle new style log with datestamps * get privip after restarting calcserver * temporarily don't run rerunFailedLambdas in test * update to work in a test env too * add 'period' to the match summary api * add period to match summary api * re-enabling rerunFailedLambdas * quieten an S3 transfer * include the foldername when reporting on container progress * put daily trajdb in a safe place * be more specific when finding the daily log * only tar the current container logs * save daily obsdb in the right place * rename ec2.tf to calcserver.tf * quieten some s3 transfers * rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is * update SSM params to reflect new serverid * bugfix in distributeCandidates - using wrong path * add newline to getCostMetrics * weird bug in statsToMqtt * bugfix in update_wmpl.sh for traj container * Script to create an AMI for the calcengine * exclude test data when syncing from s3 * writing databases and logs to the wrong target * bad test on istest * typo in serveruserid * adding to server migration notes * purge old rawcsvs and fullcsvs older than 180 days * improved housekeeping of old data * make sure we scoop up data for 31/12 * stop creation of an unused set of files * change default locations for container data * put container logs in the right place * update dockerfile syntax * change locations of test container data and output * remove incorrect comments * Add script to clean up deleted/duplicate trajs * simplify the stats gathering and move to python fix bug in getMatchStats * comment out unused files * update reportOfLatestMatches to exclude deleted trajectories * add scp to requirements * add SQL database maintenance to dataMaintenance * incorrect default for datadir * update call to pymysql.connect * ensure consolidateOutput rerunnable for prior years * upgrading WMPL * update gitignore a bit * bugfix to catch data from 31/12 each year * add paramiko to dependencies for matchPickle to support wmpl upgrade * update getpickle to newest WMPL * bugfix in api tests * bugfix - changes to dailydb not saved * improvements to process to cleanup deleted trajs * after merging, check and remove any duplicate trajectories from the sqlite db and disk * Adding version * bump version 2025.11.1 -> 2026.04.0 * update files containing vesion * bump version 2026.04.0 -> 2026.04.1 * unmangling side effets of automerge * correct usage message * adding install script * changes to ansible scripts * remove irrelevant file * Adding SSM variables for the new batchserver in the ukmda account * attempt to add logging to consolidateDistTraj * remove reference to ee account * temporarily add mariadb deets to ssm in mjmm account * bugfixes in dataMaintenance and cleanupDeletedTrajs to avoid deleting disk files when trajs share same folder * updating migration instructions * more work on deduplication for ukmon * remove username-specific stuff * remove the old deployment scripts * update install script * updates to improve server migration process * add bashrc to the installer * slight tweak to installer * add a few installer messages * remove an echo * be less verbose in the rsyncs * oops need -a for one of the rsyncs * rename requirements file for ukmda * remove dependencies on MeteorTools * add utils to ukmon_pylib * more removal of dependencies on MeteorTools * updating how shower data is obtained * corercting location of IMO and gmn shower data * bugfixes * fixes in imoWorkinShowerlist * small error in install_or_update * bugfix when no data found in imo set * retrieve IMO data after each code update * fixing up getShowerDets * improve installer * fix createShowerExtracts * update extractors and showeranalysis * bug in extractor thats existed since 2023 but nobody noticed... * trying to fix matplotlib bug * more working on matplotlib issue * oops overfixed a bug * another incorrect bugfix! * make bars wider * more removal of MeteorTools * bugfix in createannualbarchart * working on bug * more work on lenght bug * remove some debugging * oops forgot to push the fixed version of createReportIndex * segregating function to get list of images that weren't calibrated by RMS * reorganize stuff a bit * add email sending functionality to ukmon_pylib * use send email functionality in code * add database and gmaps keys/passwords as secure variables * update to use secure keys * remove dbpw from environment * remove dbpw from environment * add script to retrieve db passwd * add script to retrieve db passwords * bug in createTables * updates to migration instructions * include pip installs in the script * tweak installer * remove glob.glob1 * move unused scripts - will delete later * fix mariadb connections * typo * tidy up docs * fixing mysql calls * small changes to documentation * suppress pip messages if already installed * tweaks to the installer * add logrotate stuff * bugfixes in installer * tidy up installer * bugfix in key mgmt * move some functionality into new script to tidy up nightly job and simplify rerunning * improve install script * update location of camera count file * move annual bar charts into a more logical location * update test script * add gmaps param to my acct * relocate coverage maps and front page index * tidy up useraudit process * bugfixes in testing * bugfix in createAnnualBarchart * bug in testscript * another small bug in createSummaryTable * still fighting that bug * tidy up lastlog data * remove unused file * update cam status reporting * clear down the old brightness data * simplification of logging and of the nightlyJob script * bugfixes in process to gather consumed image ifo * update getUncal to work for historic data * typo * improvements in getUncalImages * bugfix * add python to get unused images * move script to right place * comment out unnecessary echo * bugfixes in camMetrics while migrating user accts * fix #448 * make sure window title is sensible * bugfix in replot and get unused * bugfix in removeDeletedTraj for search index * remove deprecation warning issue #450 * whoops naive vs tzaware dates * remove warning in cameraStatusReport issue #453 * remove warning issue #454 * widen constraints on analysing pickles to get images * add migration script * add sudo * actually do it * update sftp migration script * more changes to migration script * moving sftp migration script * update messages and docos * slight tweak to migration script * further work on sftp account migration * update ukmon ini files after migration * typo * collect old server login records integration with new * update migration instructions * update cam metrics report to include hostname * bugfix as $USER not available in cron * removing unused files * some tidying up * motre reorg * remove disused template rename file to follow convention * simplifying logging and removing unused aws profile * removing profile from python * adding executable bit * typo * bugfix to work around typo * print message when done * bugfix to get trajectories from s3 before trying to check for missing trajectories... * improve to support running for past dates * errors in bashrc and logrotate config * prevent email being sent for anything other than current days report * bugfix: make sure we use the most recent stats * improvement in reportOfLatestMatches to handle past dates * bugfix * small tweak to support reportOfLatestMatches * tweak getMatchStats to handle historic data better * add mariadb and ping to the sec grp so that the database can be accessed by apis * update the db servername * bugfix in match stats * prevent bogus renaming of matchJob.log * bumping version * bump version 2026.04.1 -> 2026.05.0 * update matchapi to use indexed column * add api readme * tweak to cammetrics to enhance stationlogins report * slight tweak to formatting * minor tweak to avoid unexpectedly changing directory * add support script for during server migrations * skip 1st row * add check for todos * oops strip whitespace * darn whitespace * working on migration tasks * minor tweaks * daft typo * bump version 2026.05.0 -> 2026.05.1 * bump version 2026.05.1 -> 2026.05.2 * update CodeQL to v3 * bump codeQL to v4 * tweak to script supporting server migration * tweak to scripts supporting server migration * add some stats on migration status * tweak * Latest patch release (#473) * save daily obsdb in the right place * rename ec2.tf to calcserver.tf * quieten some s3 transfers * rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is * update SSM params to reflect new serverid * bugfix in distributeCandidates - using wrong path * add newline to getCostMetrics * weird bug in statsToMqtt * bugfix in update_wmpl.sh for traj container * Script to create an AMI for the calcengine * exclude test data when syncing from s3 * writing databases and logs to the wrong target * bad test on istest * typo in serveruserid * adding to server migration notes * purge old rawcsvs and fullcsvs older than 180 days * improved housekeeping of old data * make sure we scoop up data for 31/12 * stop creation of an unused set of files * change default locations for container data * put container logs in the right place * update dockerfile syntax * change locations of test container data and output * remove incorrect comments * Add script to clean up deleted/duplicate trajs * simplify the stats gathering and move to python fix bug in getMatchStats * comment out unused files * update reportOfLatestMatches to exclude deleted trajectories * add scp to requirements * add SQL database maintenance to dataMaintenance * incorrect default for datadir * update call to pymysql.connect * ensure consolidateOutput rerunnable for prior years * upgrading WMPL * update gitignore a bit * bugfix to catch data from 31/12 each year * add paramiko to dependencies for matchPickle to support wmpl upgrade * update getpickle to newest WMPL * bugfix in api tests * bugfix - changes to dailydb not saved * improvements to process to cleanup deleted trajs * after merging, check and remove any duplicate trajectories from the sqlite db and disk * Adding version * bump version 2025.11.1 -> 2026.04.0 * update files containing vesion * bump version 2026.04.0 -> 2026.04.1 * unmangling side effets of automerge * correct usage message * decommission old calcserver and add new batchserver * add calcserver IP to ssm variables and use it in scripts * Update my branch with the latest from master (#436) * Update version (#433) * remove disused code * update livestream to accept normal RMS filenames * search dialog - fix bug and set default date range * small change to explain default dates * improvement in live image search * remove redundant file * update deployment scripts * improvements in routine to convert GMN report to pandas * more analysis of gmn data * Update the terraform a little * remove hardcoded reference to ec2-user * add SERVERUSERID env /ssm variable for calcserver user id * update container for new distrib-processing method * use calcserver SERVERUSERID variable * support for test ECS containers * fix paths in deployment scripts * remove unused file * missed some templates * update dev/test env settings * remove disused file * make sure paths are expanded * fixing various config issues * updates to makeConfig * updates to distributed processing to align with newest wmpl * add a comment to explain what the code does * remove json db * update consolidation to handle sqlite dbs * remove unused function * create data for daily report * remove unused file * update to call new version of reportOfLatestMatches * add test variables * mildly improve doco * put test logs in the right CW log group * handle missing log groups better * extra value in TrajQualityParams * bugfixes revealed in testing * fix issues in reportOfLatestMatches * put container dbs in a sensible place * catch logging errors * tidy up file locations * simplify calcserver start up * sorting out errors in container logging * update container logging info * avoid unnecessary error * fix bugs in log capture process * add MAX_STNS so i can experiement with limits * add logging. Add max_stns to constraints and set to 9999 push logs to s3 copy calcserver logs to ukmonhelper * don't pull or push the databases from S3 during distribution but do back them up after consolidation * catch error if trajpickle can't be found * set rundate more safely move some processes into nightlyJob to make testing easier * tidy up backup file names and housekeep backup folder * move some processes from findAllMatches to nightlyJob to make testing easier * support for testing of createExecReplot * pass testmode to createExecReplot * ensure logs are unique and capture container logs * realigning createFireballPage with prod. Updating gitignore * remove verbosity flag from one tar operation * fixup to handle new style log with datestamps * get privip after restarting calcserver * temporarily don't run rerunFailedLambdas in test * update to work in a test env too * add 'period' to the match summary api * add period to match summary api * re-enabling rerunFailedLambdas * quieten an S3 transfer * include the foldername when reporting on container progress * put daily trajdb in a safe place * be more specific when finding the daily log * only tar the current container logs * save daily obsdb in the right place * rename ec2.tf to calcserver.tf * quieten some s3 transfers * rename ec2.tf in my acct to ukmonhelper_ec2.tf to make it clearer what it is * update SSM params to reflect new serverid * bugfix in distributeCandidates - using wrong path * add newline to getCostMetrics * weird bug in statsToMqtt * bugfix in update_wmpl.sh for traj container * Script to create an AMI for the calcengine * exclude test data when syncing from s3 * writing databases and logs to the wrong target * bad test on istest * typo in serveruserid * adding to server migration notes * purge old rawcsvs and fullcsvs older than 180 days * improved housekeeping of old data * make sure we scoop up data for 31/12 * stop creation of an unused set of files * change default locations for container data * put container logs in the right place * update dockerfile syntax * change locations of test container data and output * remove incorrect comments * Add script to clean up deleted/duplicate trajs * simplify the stats gathering and move to python fix bug in getMatchStats * comment out unused files * update reportOfLatestMatches to exclude deleted trajectories * add scp to requirements * add SQL database maintenance to dataMaintenance * incorrect default for datadir * update call to pymysql.connect * ensure consolidateOutput rerunnable for prior years * upgrading WMPL * update gitignore a bit * bugfix to catch data from 31/12 each year * add paramiko to dependencies for matchPickle to support wmpl upgrade * update getpickle to newest WMPL * bugfix in api tests * bugfix - changes to dailydb not saved * improvements to process to cleanup deleted trajs * after merging, check and remove any duplicate trajectories from the sqlite db and disk * Adding version * bump version 2025.11.1 -> 2026.04.0 * update files containing vesion * bump version 2026.04.0 -> 2026.04.1 * unmangling side effets of automerge * correct usage message * adding install script * changes to ansible scripts * remove irrelevant file * Adding SSM variables for the new batchserver in the ukmda account * attempt to add logging to consolidateDistTraj * remove reference to ee account * temporarily add mariadb deets to ssm in mjmm account * bugfixes in dataMaintenance and cleanupDeletedTrajs to avoid deleting disk files when trajs share same folder * updating migration instructions * more work on deduplication for ukmon * remove username-specific stuff * remove the old deployment scripts * update install script * updates to improve server migration process * add bashrc to the installer * slight tweak to installer * add a few installer messages * remove an echo * be less verbose in the rsyncs * oops need -a for one of the rsyncs * rename requirements file for ukmda * remove dependencies on MeteorTools * add utils to ukmon_pylib * more removal of dependencies on MeteorTools * updating how shower data is obtained * corercting location of IMO and gmn shower data * bugfixes * fixes in imoWorkinShowerlist * small error in install_or_update * bugfix when no data found in imo set * retrieve IMO data after each code update * fixing up getShowerDets * improve installer * fix createShowerExtracts * update extractors and showeranalysis * bug in extractor thats existed since 2023 but nobody noticed... * trying to fix matplotlib bug * more working on matplotlib issue * oops overfixed a bug * another incorrect bugfix! * make bars wider * more removal of MeteorTools * bugfix in createannualbarchart * working on bug * more work on lenght bug * remove some debugging * oops forgot to push the fixed version of createReportIndex * segregating function to get list of images that weren't calibrated by RMS * reorganize stuff a bit * add email sending functionality to ukmon_pylib * use send email functionality in code * add database and gmaps keys/passwords as secure variables * update to use secure keys * remove dbpw from environment * remove dbpw from environment * add script to retrieve db passwd * add script to retrieve db passwords * bug in createTables * updates to migration instructions * include pip installs in the script * tweak installer * remove glob.glob1 * move unused scripts - will delete later * fix mariadb connections * typo * tidy up docs * fixing mysql calls * small changes to documentation * suppress pip messages if already installed * tweaks to the installer * add logrotate stuff * bugfixes in installer * tidy up installer * bugfix in key mgmt * move some functionality into new script to tidy up nightly job and simplify rerunning * improve install script * update location of camera count file * move annual bar charts into a more logical location * update test script * add gmaps param to my acct * relocate coverage maps and front page index * tidy up useraudit process * bugfixes in testing * bugfix in createAnnualBarchart * bug in testscript * another small bug in createSummaryTable * still fighting that bug * tidy up lastlog data * remove unused file * update cam status reporting * clear down the old brightness data * simplification of logging and of the nightlyJob script * bugfixes in process to gather consumed image ifo * update getUncal to work for historic data * typo * improvements in getUncalImages * bugfix * add python to get unused images * move script to right place * comment out unnecessary echo * bugfixes in camMetrics while migrating user accts * fix #448 * make sure window title is sensible * bugfix in replot and get unused * bugfix in removeDeletedTraj for search index * remove deprecation warning issue #450 * whoops naive vs tzaware dates * remove warning in cameraStatusReport issue #453 * remove warning issue #454 * widen constraints on analysing pickles to get images * add migration script * add sudo * actually do it * update sftp migration script * more changes to migration script * moving sftp migration script * update messages and docos * slight tweak to migration script * further work on sftp account migration * update ukmon ini files after migration * typo * collect old server login records integration with new * update migration instructions * update cam metrics report to include hostname * bugfix as $USER not available in cron * removing unused files * some tidying up * motre reorg * remove disused template rename file to follow convention * simplifying logging and removing unused aws profile * removing profile from python * adding executable bit * typo * bugfix to work around typo * print message when done * bugfix to get trajectories from s3 before trying to check for missing trajectories... * improve to support running for past dates * errors in bashrc and logrotate config * prevent email being sent for anything other than current days report * bugfix: make sure we use the most recent stats * improvement in reportOfLatestMatches to handle past dates * bugfix * small tweak to support reportOfLatestMatches * tweak getMatchStats to handle historic data better * add mariadb and ping to the sec grp so that the database can be accessed by apis * update the db servername * bugfix in match stats * prevent bogus renaming of matchJob.log * bumping version * bump version 2026.04.1 -> 2026.05.0 * update matchapi to use indexed column * add api readme * tweak to cammetrics to enhance stationlogins report * slight tweak to formatting * minor tweak to avoid unexpectedly changing directory * add support script for during server migrations * skip 1st row * add check for todos * oops strip whitespace * darn whitespace * working on migration tasks * minor tweaks * daft typo * bump version 2026.05.0 -> 2026.05.1 * bump version 2026.05.1 -> 2026.05.2 * update CodeQL to v3 * bump codeQL to v4 * tweak to script supporting server migration * tweak to scripts supporting server migration * add some stats on migration status * tweak * get cursor keys working in python * set via even if no recent login * improved reporting on migration * tweaks to migr scripts * another small tweak * make lists unique * uniq and sort backwards * testing fix for #460 * more work on status reports * bugfix in failed matches report * found the bug * attempt to fix fails report * another tweak * okay, got it * remove debug * fix timing metrics data * small tweak * looks good * add EIP for the batchserver * bugfix to handle bad data in login details * add function to email problem cameras with bespoke message * bump version 2026.05.2 -> 2026.05.3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
minor tweaks