[DevOps] Discover stability parameters of performance archive directories (PACE)

[TonyB9000]

Summary

Find out how PACE treats/mistreats the performance archive directories

Details

1. Find out how the performance archive is periodically cleaned (cron job?)
2. Investigate PACE to determine if it moves/alters files in the performance archive.  2) is the higher priority item.

Additional Context

1. Chrysalis PACE Script (wrapper to 2): https://github.com/E3SM-Project/E3SM_test_scripts/blob/master/jenkins/chrysalis_pace.sh
2. PACE Archive script: https://github.com/E3SM-Project/E3SM_test_scripts/blob/master/util/pace_archive.sh
3. Main Perl Ingestion Script: https://pace.ornl.gov/static/tools/e3sm_perf_archive.perl

Deliverables

Negotiated stability or standardization of PACE treatment of "performance archive" directories, sufficient to allow predictable ingestion by SimBoard backend.

• Confirm whether the e3sm_perf_archive.perl script is still the active PACE archival mechanism.
  • This is an underlying processing script for a machine specific script at each site.
• Figure out if the e3sm_perf_archive.perl script runs on a scheduled job
  • No, this is an underlying script that is wrapped by a machine specific bash script.
• Figure out the frequency the machine specific bash scripts are ran
  • Seems like daily based on OLD_PERF_ARCHIVE_DIR directory structure.
• Find output directory where timestamped performance archive is stored:
  • SimBoard should scan and ingest OLD_PERF_ARCHIVE_DIR with PERF_ARCHIVE_DIR.

Refer to https://e3sm.atlassian.net/wiki/spaces/EPG/pages/5477335106/PACE+Collection+and+Upload+Reference#Machine-Specific-Processing%2FUploading-Table for latest machine mapping table.

Dependencies & Blockers

Consultation with @youngsung Kim regarding PACE processing.

Due Date

2026-07-03

Additional Context

e3sm_perf_archive.perl script at NERSC with Perlmutter support
Source: /global/cfs/cdirs/e3sm/performance_archive/e3sm_perf_archive.perl.sarat

#!/usr/bin/perl

#  Perl 5 required.
require 5.004;

#  Strict syntax checking.
use strict;

#  Modules.
use Cwd;
use Time::Local;
use Sys::Hostname;
use Net::Domain qw(domainname hostdomain);

=head1 NAME

e3sm_perf_archive

=head1 SYNOPSIS

Move performance archive data for completed or failed
jobs into time-stamped directory for permanent archiving.

=head1 DESCRIPTION

If it does not already exist, create a directory with the name 

  performance_archive_<system>_<project_space_name>_<year>_<month>_<day>_<hour>_<min>_<sec>

Next, move performance data for completed or failed jobs 
into the new directory, preserving the same directory
structure (name/case name/job id). Uses system job monitoring
tools to identify which jobs are still running (so
that associated data are not moved).

Usage:

perl e3sm_perf_archive.perl

within the top level of a performance_archive.

=head1 AUTHOR

Pat Worley E<lt>F<worleyph@ornl.gov>E<gt>
Sarat Sreepathi E<lt>F<sarat@ornl.gov>E<gt>

=cut

###############################################################################
#
#  Miscellaneous global flags and variables.
#
###############################################################################

###############################################################################
#
#  This is the call to main.
#
###############################################################################

    &main();

exit 0;

###############################################################################
#
#  This is main.
#
###############################################################################

sub main {

  # Define variables.
  my $timestamp;
  my $i;
  my $arg_flag;
  my $host;
  my $domain;
  my $system;
  my $center;
  my $performance_archive;
  my @path;
  my $project_space;
  my $sec;
  my $min;
  my $hour;
  my $mday;
  my $mon;
  my $year;
  my $wday;
  my $yday;
  my $isdst;
  my $calyear;
  my $twodigitsec;
  my $twodigitmin;
  my $twodigithour;
  my $twodigitday;
  my $twodigitmonth;
  my $new_archive_name;
  my $new_archive_directory;
  my $mode;
  my @users;
  my $thisuser;
  my $new_user_directory;
  my $thisuser_directory;
  my @cases;
  my $thiscase;
  my $new_case_directory;
  my $thiscase_directory;
  my @jlids;
  my $thisjlid;
  my $thislid;
  my $thisjobid;
  my $thisjobid2;
  my $thisjlid_directory;
  my $thissystem;
  my $job_statsf_string;
  my $job_statsf;
  my $job_statsf_readable;
  my @squeueall_field;
  my $thisfeatures;
  my $thisexechost;
  my $thiscompleted;
  my $CaseStatusf;
  my @jlidfiles;
  my $thisjobstat;
  my $acme_timingf;
  my $e3sm_timingf;

  $timestamp = "unset";
  for (my $i=0; $i <= $#ARGV; $i+=2){
    $arg_flag = $ARGV[$i];
    if (($arg_flag =~ m/^-h$/) || ($arg_flag =~ m/^--help$/)){
      print
"Usage:

perl e3sm_perf_archive.perl

within the top level of a performance_archive at NERSC, 
at ALCF, at OLCF, on Anvil, on Chrysalis, or on Compy

Purpose:
Tool to aid in the permanent archiving of E3SM performance data.

If it does not already exist, create a directory with the name 

  performance_archive_<system>_<project_space_name>_<year>_<month>_<day>_<hour>_<min>_<sec>

Next, move performance data for completed or failed jobs 
into the new directory, preserving the same directory
structure (name/case name/job id). This uses system job monitoring
tools to identify which jobs are still running (so that
associated data are not moved). For centers with multiple systems, 
this tool will need to be run once on each system front end for
the given performance_archive directory if multiple
systems save performance data in the same directory.

Optional arguments:

  -h, --help      show this help message and exit 
  -t, --timestamp specify the timestamp (<year>_<month>_<day>_<hour>_<min>_<sec>)
                  to use in the name of the performance_archive directory. The
                  input is checked that it satisfies the general format, and aborts
                  otherwise. (Default is to use the script execution local time.) 
\n";
      exit;
    }elsif(($arg_flag =~ m/^--timestamp$/) || ($arg_flag =~ m/^-t$/)){
      $timestamp = $ARGV[++$i];
      if ($timestamp !~ m/^(\d+)\_(\d+)\_(\d+)\_(\d+)\_(\d+)\_(\d+)$/){
        print "Error: timestamp input (\"$timestamp\") does not match required format\n";
        exit;
      }
    }
  }

  $host = hostname;
  $domain = hostdomain;
  print $host;
  print $domain;

  if    ($host =~ m/^blueslogin\S*/){
    $system = "anvil";
    $center = "anvil";
  } 
  elsif ($host =~ m/^chrlogin\S*/){
    $system = "chrysalis";
    $center = "chrysalis";
  }
  elsif ($host =~ m/^compy\S*/){
    $system = "compy";
    $center = "compy";
  } 
  elsif ($host =~ m/^frontier\S*/){
    $system = "frontier";
    $center = "frontier";
  }
  elsif ($domain =~ m/\S*perlmutter\S*/){
    $system = "perlmutter";
    $center = "perlmutter";
  } 
  else{
    die "ERROR: Can only be used on a supported system: anvil, chrysalis, compy, perlmutter, frontier";
  }

# print "$system\n";

  $performance_archive = cwd();

# print "$performance_archive\n";

  if ($system =~ m/perlmutter/){
    if ($performance_archive =~ m/^\/global\/cfs\/cdirs\/\S*/){
      @path = split(/\//,$performance_archive);
      $project_space = $path[4];
    }else{
      die "ERROR: On Perlmutter, must be in a /global/cfs/cdirs/ subdirectory for this script to work";
      exit;
    }
  }elsif ($system =~ m/anvil/){
    if ($performance_archive =~ m/^\/lcrc\/group\/\S*/){
      @path = split(/\//,$performance_archive);
      $project_space = $path[3];
    }else{
      die "ERROR: On Anvil, must be in a /lcrc/group/ subdirectory for this script to work";
      exit;
    }
  }elsif ($system =~ m/chrysalis/){
    if ($performance_archive =~ m/^\/lcrc\/group\/\S*/){
      @path = split(/\//,$performance_archive);
      $project_space = $path[3];
    }else{
      die "ERROR: On Chrysalis, must be in a /lcrc/group/ subdirectory for this script to work";
      exit;
    }
  }elsif ($system =~ m/compy/){
    if ($performance_archive =~ m/^\/compyfs\/\S*/){
      $project_space = "all";
    }else{
      die "ERROR: On Compy, must be in a /compyfs/ subdirectory for this script to work";
      exit;
    }
  }elsif ($system =~ m/frontier/){
    if ($performance_archive =~ m/^\/lustre\/orion\/cli115\/world-shared\/\S*/){
      @path = split(/\//,$performance_archive);
      $project_space = $path[3];
    }else{
      die "ERROR: On Frontier, must be in a /lustre/orion/cli115/world-shared/ subdirectory for this script to work";
      exit;
    }
  }
# print "$project_space\n";

  if ($timestamp =~ m/unset/){
    ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();
    $twodigitsec = sprintf("%02d", $sec);
    $twodigitmin = sprintf("%02d", $min);
    $twodigithour = sprintf("%02d", $hour);
    $twodigitday = sprintf("%02d", $mday);
    $twodigitmonth = sprintf("%02d", $mon+1);
    $calyear = $year + 1900;
    $timestamp = "$calyear\_$twodigitmonth\_$twodigitday\_$twodigithour\_$twodigitmin\_$twodigitsec";
  }
  $new_archive_name = "performance\_archive\_$center\_$project_space\_$timestamp";

  if (-e $new_archive_name){
    if (!(-d $new_archive_name)){
      die "ERROR: $new_archive_name must be a directory";
    }
  }else{
    mkdir "$new_archive_name";
  }

  $new_archive_directory = "$performance_archive/$new_archive_name";
  $mode = 0775;
  $mode = $mode | 02000;  # Set group sticky bit.
  chmod $mode, "$new_archive_directory";

  print "Creating/Using $new_archive_directory\n";

  opendir USERS, $performance_archive;
  @users = grep { -d $_ && -r _ && /^(?!\.).*$/ } readdir USERS;

  USERLOOP: for ( @users ){
    $thisuser = $_;
    if (!($thisuser =~ m/^performance\_archive\S*/)){

      print "USER: $thisuser\n";

      $new_user_directory = "$new_archive_directory/$thisuser";

      chdir "$performance_archive/$thisuser" || next USERLOOP;
      $thisuser_directory = cwd();

      opendir CASES, $thisuser_directory || next USERLOOP;
      @cases = grep { -d $_ && -r _ && /^(?!\.).*$/ } readdir CASES;

      CASELOOP: for ( @cases ){
        $thiscase = $_;

        print "  CASE: $thiscase\n";

        $new_case_directory = "$new_user_directory/$thiscase";

        chdir "$thisuser_directory/$thiscase" || next CASELOOP;
        $thiscase_directory = cwd();

        opendir JLIDS, $thiscase_directory || next CASELOOP;
        @jlids = grep { -d $_ && -r _ && /^(?!\.).*$/ } readdir JLIDS;

        JLIDSLOOP: for ( @jlids ){
          $thisjlid = $_;

          $thisjobid = 0;
          $thisjobid2 = 0;
          if ($thisjlid =~ m/^(\d+)\S*\.(\d+\-\d+)$/){
            $thisjobid = $1;
            $thislid   = $2;
          }

          print "    LID: $thislid JOBID: $thisjobid\n";

          chdir "$thiscase_directory/$thisjlid" || next JLIDSLOOP;
          $thisjlid_directory = cwd();
          # Verify job id from job_statsf file.
          if ($system =~ m/anvil|chrysalis/){
            $job_statsf_string = "squeueall_jobid";
            # For LCRC systems, get system name from job_statsf file.
            $thissystem = "unset";
          }elsif ($system =~ m/compy/){
            $job_statsf_string = "squeueall_jobid";
            # For non-NERSC and non-LCRC systems, already know system.
            $thissystem = $system;
          }elsif ($system =~ m/theta/){
            $job_statsf_string = "qstatf_jobid";
            # For non-NERSC and non-LCRC systems, already know system.
            $thissystem = $system;
          }elsif ($system =~ m/perlmutter/){
            $job_statsf_string = "squeues";
            # For NERSC, get system name from job_statsf file.
            $thissystem = "perlmutter";
          }elsif ($system =~ m/summit/){
            $job_statsf_string = "bjobslUF_jobid";
            # For non-NERSC and non-LCRC systems, already know system.
            $thissystem = $system;
          }
          $job_statsf = "$thisjlid_directory/$job_statsf_string.$thisjlid.gz";
          $job_statsf_readable = "false";

          if (-f $job_statsf && -r _ ){
            open(JOBSTATSF, "gunzip -c $job_statsf |") || die "can't open pipe to $job_statsf";
            $job_statsf_readable = "true";
          }else{
            $job_statsf = "$thisjlid_directory/$job_statsf_string.$thisjlid";
            if (-f $job_statsf && -r _ ){
              open(JOBSTATSF, "$job_statsf");
              $job_statsf_readable = "true";
            }
          }

          if ($job_statsf_readable =~ m/true/){

            if ($job_statsf_string =~ m/sqsf_jobid/){
              JOBFLOOP1: while (<JOBSTATSF>){
                if     (m/^JobId=(\d+)\s*JobName=(\S*)\s*$/){
                  $thisjobid2 = $1;
                  if ($thisjobid =~ m/0/){
                    $thisjobid = $thisjobid2;
                  }elsif (!($thisjobid =~ m/($thisjobid2)/)){
                    print "MISMATCHED JOB IDS: $thisjlid_directory\n";
                    print "  USER: $thisuser\n";
                    print "    CASE: $thiscase\n";
                    print "      JLID: $thisjlid\n";
                    print "        JOB IDS: $thisjobid $thisjobid2\n";
                    close JOBSTATSF;
                    next JLIDSLOOP;
                  }
                }elsif (m/^\s*Features=(\S+)\s+/){
                  # In the past the Cori systems and the Edison system used the
                  # same performance_archive directory. To differentiate on which
                  # system a given job ran, we used information in the job_statsf
                  # file. While Edison has been decommissioned, this issue may
                  # reoccur when the Perlmutter system comes on-line. We retain
                  # the following logic in case it becomes useful again.
                  $thisfeatures = $1;
                  if ($thisfeatures =~ m/haswell/){
                    $thissystem = "cori";
                  }
                  elsif ($thisfeatures =~ m/knl/){
                    $thissystem = "cori";
                  }
                  last JOBFLOOP1;
                }
              }
            }elsif ($job_statsf_string =~ m/squeueall_jobid/){
              JOBFLOOP2: while (<JOBSTATSF>){
                @squeueall_field = split(/\|/,$_);
                $thisjobid2 = $squeueall_field[8];
                if ($thisjobid2 =~ m/JOBID/){
                  next JOBFLOOP2;
                }
                if ($thisjobid =~ m/0/){
                  $thisjobid = $thisjobid2;
                }elsif (!($thisjobid =~ m/($thisjobid2)/)){
                  print "MISMATCHED JOB IDS: $thisjlid_directory\n";
                  print "  USER: $thisuser\n";
                  print "    CASE: $thiscase\n";
                  print "      JLID: $thisjlid\n";
                  print "        JOB IDS: $thisjobid $thisjobid2\n";
                  close JOBSTATSF;
                  next JLIDSLOOP;
                }
                # At one point Anvil and Chrysalis used the same performance_archive
                # directory. To differentiate on which system a given job ran, use
                # information in the job_statsf file.
                if ($system =~ m/anvil|chrysalis/){
                  $thisexechost = $squeueall_field[27];
                  if     ($thisexechost =~ m/b\S+/){
                    $thissystem = "anvil";
                  }elsif ($thisexechost =~ m/chr\S+/){
                    $thissystem = "chrysalis";
                  }
                }
                last JOBFLOOP2;
              }
            }elsif ($job_statsf_string =~ m/bjobslUF_jobid/){
              JOBFLOOP3: while (<JOBSTATSF>){
                if (m/^Job\s*<(\S+)?>.*$/){
                  $thisjobid2 = $1;
                  if ($thisjobid =~ m/0/){
                    $thisjobid = $thisjobid2;
                  }elsif (!($thisjobid =~ m/($thisjobid2)/)){
                    print "MISMATCHED JOB IDS: $thisjlid_directory\n";
                    print "  USER: $thisuser\n";
                    print "    CASE: $thiscase\n";
                    print "      JLID: $thisjlid\n";
                    print "        JOB IDS: $thisjobid $thisjobid2\n";
                    close JOBSTATSF;
                    next JLIDSLOOP;
                  }
                  last JOBFLOOP3;
                }
              }
            }else{
              JOBFLOOP4: while (<JOBSTATSF>){
                if ((m/^Job Id:\s*(\d+)\S*\s*$/) || 
                    (m/^JobID:\s*(\d+)\S*\s*$/)){
                  $thisjobid2 = $1;
                  if ($thisjobid =~ m/0/){
                    $thisjobid = $thisjobid2;
                  }elsif (!($thisjobid =~ m/($thisjobid2)/)){
                    print "MISMATCHED JOB IDS: $thisjlid_directory\n";
                    print "  USER: $thisuser\n";
                    print "    CASE: $thiscase\n";
                    print "      JLID: $thisjlid\n";
                    print "        JOB IDS: $thisjobid $thisjobid2\n";
                    close JOBSTATSF;
                    next JLIDSLOOP;
                  }
                  last JOBFLOOP4;
                }
              }
            }

            close JOBSTATSF;

          }

          if ($thissystem =~ m/unset/){
            # If system still not defined, check the acme_timing or e3sm_timing file (if available)
            $acme_timingf = "$thisjlid_directory/acme_timing.$thiscase.$thisjlid.gz";
            $e3sm_timingf = "$thisjlid_directory/e3sm_timing.$thiscase.$thisjlid.gz";
            if ((-f $acme_timingf) || (-f $e3sm_timingf)) {
              if (-f $acme_timingf) {
                open(TIMINGF, "gunzip -c $acme_timingf |") || die "can't open pipe to $acme_timingf";
              }else{
                open(TIMINGF, "gunzip -c $e3sm_timingf |") || die "can't open pipe to $e3sm_timingf";
              }

              TIMINGFLOOP: while (<TIMINGF>){
                if (m/^\s*Machine\s*:\s*(\S+)/){
                  $thissystem = $1;
                  last TIMINGFLOOP;
                }
              }

              close TIMINGF;
            }
          }

          if ($thissystem =~ m/unset/){
            print "POTENTIAL EARLY FAILURE: $thisjlid_directory\n";
            print "  USER: $thisuser\n";
            print "    CASE: $thiscase\n";
            print "      JLID: $thisjlid\n";
            next JLIDSLOOP;
          }

          print "      JOBID: $thisjobid JOBID2: $thisjobid2\n";
          print "      SYSTEM: $thissystem  SCRIPT-SYSTEM: $system\n";

          # If a gzipped version of CaseStatus exists, then job very likely completed.
          $thiscompleted = "false";
          $CaseStatusf = "CaseStatus.$thisjlid.gz";
          if (-f $CaseStatusf){
            $thiscompleted = "true";

            # But also check that all files are gzipped in this directory.
            opendir JLIDFILES, $thisjlid_directory;

            @jlidfiles = grep { -f $_ && -r _ && /^.*(?<!\.gz)$/ } readdir JLIDFILES;
            for ( @jlidfiles ){
              $thiscompleted = "false";
            }

          }

          print "        COMPLETED: $thiscompleted\n";

          # If not obviously completed, next check whether running, but should
          # by queried only on front end of system where the job ran.
          if (($thissystem =~ m/$system/) && ($thiscompleted =~ m/false/)){

            if ($system =~ m/anvil|chrysalis|compy/){
              $thisjobstat = `/usr/bin/squeue --job $thisjobid --noheader 2>&1`;
              if (($thisjobstat =~ m/^.*(slurm_load_jobs error: Invalid job id specified).*$/) ||
                  ($thisjobstat =~ m/^\s*$/)){
                $thiscompleted = "true";
              } else {
                print "RUNNING: $thisjlid_directory\n";
                print "  USER: $thisuser\n";
                print "    CASE: $thiscase\n";
                print "      JLID: $thisjlid\n";
              }
            }elsif ($system =~ m/cori/){
              $thisjobstat = `/usr/bin/squeue --job $thisjobid --noheader 2>&1`;
              if ($thisjobstat =~ m/^.*(slurm_load_jobs error: Invalid job id specified).*$/){
                $thiscompleted = "true";
              } else {
                print "RUNNING: $thisjlid_directory\n";
                print "  USER: $thisuser\n";
                print "    CASE: $thiscase\n";
                print "      JLID: $thisjlid\n";
	      }
            }elsif ($system =~ m/summit/){
              $thisjobstat = `bjobs $thisjobid 2>&1`;
              if ($thisjobstat =~ m/^Job <$thisjobid> is not found\s*$/){
                $thiscompleted = "true";
              } else {
                print "RUNNING: $thisjlid_directory\n";
                print "  USER: $thisuser\n";
                print "    CASE: $thiscase\n";
                print "      JLID: $thisjlid\n";
	      }
            }elsif ($system =~ m/theta/){
              $thisjobstat = `/usr/bin/qstat $thisjobid 2>&1`;
              if ($thisjobstat =~ m/^\s*$/){
                $thiscompleted = "true";
              } else {
                print "RUNNING: $thisjlid_directory\n";
                print "  USER: $thisuser\n";
                print "    CASE: $thiscase\n";
                print "      JLID: $thisjlid\n";
	      }
            }

            print "        COMPLETED2: $thiscompleted\n";

	  }

          # If completed, move data for this job to the new time-stamped performance 
          # archive. While not scrictly necessary, only allow script running
          # on the front end of a system to move data for jobs running on the
          # corresponding system.
          if (($thissystem =~ m/$system/) && ($thiscompleted =~ m/true/)){

             # First, check whether need to create a new user directory.
             if (-e $new_user_directory){
               if (!(-d $new_user_directory)){
                 die "ERROR: $new_user_directory must be a directory";
               }
             }else{
               mkdir "$new_user_directory";
               chmod $mode, "$new_user_directory";
             }

             # Next, check whether need to create a new case directory.
             if (-e $new_case_directory){
               if (!(-d $new_case_directory)){
                 die "ERROR: $new_case_directory must be a directory";
               }
             }else{
               mkdir "$new_case_directory";
               chmod $mode, "$new_case_directory";
             }

             # Finally, move the job directory.
             chdir "$thiscase_directory";
print "$thiscase_directory\n";
print "mv $thisjlid_directory $new_case_directory/$thisjlid`\n";
             `mv $thisjlid $new_case_directory/$thisjlid`

          }         

        }

        # Delete the case directory if it is now empty.
        rmdir "$thiscase_directory";

      }

      # Delete the user directory if it is now empty.
      rmdir "$thisuser_directory";

    }

  }

}

1;

__END__

What this script does

This script is a one-shot maintenance tool for archiving completed or failed E3SM performance jobs.

Step by step:

1. Parses optional arguments:
   • -h / --help prints usage.
   • -t / --timestamp lets the caller provide the timestamp used in the archive directory name.
2. Detects the current system from the hostname/domain:
   • anvil
   • chrysalis
   • compy
   • frontier
   • perlmutter
3. Verifies that the script is being run from an expected filesystem path for that system.
4. Determines the current performance_archive directory from cwd().
5. Creates or reuses a timestamped archive directory named like:

   performance_archive_<system>_<project_space>_<year>_<month>_<day>_<hour>_<min>_<sec>
   

6. Scans the current archive layout:

   <performance_archive>/<user>/<case>/<job_lid_directory>
   

7. For each job/LID directory, it tries to determine:
   • job ID
   • LID
   • system where the job ran
   • whether the job is still running
8. It checks job metadata files such as scheduler output files and timing files to verify the job ID and system.
9. It treats a job as completed if:
   • CaseStatus.<jlid>.gz exists and all files in the job directory are gzipped, or
   • the scheduler no longer reports the job as running.
10. If the job is completed/failed and belongs to the same system where the script is running, it moves the job directory into the timestamped archive while preserving the layout:

    <timestamped_archive>/<user>/<case>/<job_lid_directory>
    

11. After processing a case, it tries to remove the original case directory:

    rmdir "$thiscase_directory";

    This only succeeds if the case directory is empty. If any job directories remain, the case directory is left in place.
12. After processing a user, it also tries to remove the original user directory:

    rmdir "$thisuser_directory";

    This also only succeeds if the user directory is empty.

Summary: the script moves completed/failed job performance data into a timestamped archive directory and cleans up now-empty original case/user directories. It does not delete non-empty case directories, and it does not remove jobs that are still running or cannot be safely classified.

Example log output: /global/cfs/cdirs/e3sm/performance_archive_2026_04_15_15_43_10/process-perf-archive-2026_04_15_15_43_10.log

[TonyB9000]

The following (partially-annotated) code snippets from "https://pace.ornl.gov/static/tools/e3sm_perf_archive.perl" indicate that PACE restructures the layout of files in the user's "performance_archive" directories:

  #### @users = directories in "performance_archive" that are readable
  #### and do not begin with "."

  opendir USERS, $performance_archive;
  @users = grep { -d $_ && -r _ && /^(?!\.).*$/ } readdir USERS;

  USERLOOP: for ( @users ){
    $thisuser = $_;

    #### ONLY process directory if it does NOT begin with "performance_archive"

    if (!($thisuser =~ m/^performance\_archive\S*/)){

      print "USER: $thisuser\n";

      $new_user_directory = "$new_archive_directory/$thisuser";

      #### Enter this USER directory or skip

      chdir "$performance_archive/$thisuser" || next USERLOOP;
      $thisuser_directory = cwd();

      #### Find readble directories that do not begione with "."

      opendir CASES, $thisuser_directory || next USERLOOP;
      @cases = grep { -d $_ && -r _ && /^(?!\.).*$/ } readdir CASES;

      CASELOOP: for ( @cases ){
        $thiscase = $_;

        print "  CASE: $thiscase\n";

        $new_case_directory = "$new_user_directory/$thiscase";

        #### Enter this CASE directory or skip

        chdir "$thisuser_directory/$thiscase" || next CASELOOP;
        $thiscase_directory = cwd();

        opendir JLIDS, $thiscase_directory || next CASELOOP;
        @jlids = grep { -d $_ && -r _ && /^(?!\.).*$/ } readdir JLIDS;

        #### Find readable directories that do not begin with "."

        JLIDSLOOP: for ( @jlids ){
          $thisjlid = $_;

############ eventually

          # If completed, move data for this job to the new time-stamped performance
          # archive. While not scrictly necessary, only allow script running
          # on the front end of a system to move data for jobs running on the
          # corresponding system.
          if (($thissystem =~ m/$system/) && ($thiscompleted =~ m/true/)){

             # First, check whether need to create a new user directory.
             if (-e $new_user_directory){
               if (!(-d $new_user_directory)){
                 die "ERROR: $new_user_directory must be a directory";
               }
             }else{
               mkdir "$new_user_directory";
               chmod $mode, "$new_user_directory";
             }

             # Next, check whether need to create a new case directory.
             if (-e $new_case_directory){
               if (!(-d $new_case_directory)){
                 die "ERROR: $new_case_directory must be a directory";
               }
             }else{
               mkdir "$new_case_directory";
               chmod $mode, "$new_case_directory";
             }

             # Finally, move the job directory.
             chdir "$thiscase_directory";
print "$thiscase_directory\n";
print "mv $thisjlid_directory $new_case_directory/$thisjlid`\n";
             `mv $thisjlid $new_case_directory/$thisjlid`

In short, the last command MOVES the $thisjlid directory to $new_case_directory, potentially rendering uptake by SimBoard unpredictable.

[tomvothecoder]

I took e3sm_perf_archive.perl and gave it to GPT to explain what it does.

Also, maybe there isn't a job that cleans up the performance archive but instead a job the moves things out of them into other time-stamped directories for "permanent archiving" (according to the script's synopsis).

Summary

PACE's e3sm_perf_archive.perl script moves completed JLID/job directories from the active performance_archive tree into a timestamped archive tree.

Effective move:

FROM:
$performance_archive/$user/$case/$jlid

TO:
$performance_archive/performance_archive_${center}_${project_space}_${timestamp}/$user/$case/$jlid

The $user/$case/$jlid layout is preserved, but the root changes.

Supported destination root patterns include:

Platform	Destination archive root
LCRC Anvil	performance_archive_anvil_<project>_<timestamp>
LCRC Chrysalis	performance_archive_chrysalis_<project>_<timestamp>
Compy	performance_archive_compy_all_<timestamp>
NERSC Cori	performance_archive_cori_<project>_<timestamp>
OLCF Summit	performance_archive_summit_<project>_<timestamp>
ALCF Theta	performance_archive_theta_<project>_<timestamp>

Notes:

• The pasted script does not directly support Perlmutter.
• Directories are moved only after the script decides the job is completed.
• The original active path is therefore not durable.
• SimBoard should not treat $performance_archive/$user/$case/$jlid as a stable ingestion path.

Suggested Action Items

Moved to issue descripton.

[TonyB9000]

@tomvothecoder My recent work assessing the various "root_PA" directories has turned up surprises.

The (TWO) root_PA directories on chrysalis have the following statistics (as of ~May 25):

  2976 Raw_Exec_ID_Paths-lcrc.group.e3sm.PERF_Chrysalis.performance_archive

       134   (4.5%) in USER dirs                (date-range 210201-123024 to 260527-173200)
      2842  (95.5%) in dated PA subdirectories  (date_range 241224-104433 to 260326-230054)

   108 Raw_Exec_ID_Paths-lcrc.group.e3sm.performance_archive

        40  (37%) in USER dirs                  (date-range 210610-153146 to 260201-175038)
        68  (63%) in dated PA subdirectories    (date_range 250120-195205 to 250920-102808)

(In each case, NO TWO directories anywhere contain the same "exec_id" subdirectories). Note, however, that these date-ranges tend to destroy my plan to seek the "current earliest exec_id date" and seek SimBoard state only for dates on-or-after, as it seems that user directories have materials dating back to 2021.

The situation at NERSC (Root_PA directory /global/cfs/projectdirs/e3sm/performance_archive/) was very different.

        Raw USER count: 524 exec_ids (387 unique)
        Raw PA count:   0 exec_ids.
        USER exec_id date range: 250904-163426 to 260529-10095

First, the subordinate (PACE-produced)? PA directories contained NO exec_id subdirectories - only a few text files.
Far more disturbing, the occasions of "exec_id" subdirectories were NOT unique. For instance, the exec_id "53501875.260529-072257" occurred 25 times, under different intermediate user directories: (Example):

tonyb@perlmutter:login29:~/test/simboard> grep 53501875.260529-072257 NERSC_PA_Listing

zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1984050100.EN00/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1984050100.EN01/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1984050100.EN03/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1984050100.EN04/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1984050100.EN07/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1984050100.EN08/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1984110100.EN00/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1984110100.EN02/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1984110100.EN04/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1984110100.EN06/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1984110100.EN07/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1984110100.EN08/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1984110100.EN09/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1985050100.EN00/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1985050100.EN01/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1985050100.EN02/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1985050100.EN03/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1985050100.EN07/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1985110100.EN02/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1985110100.EN03/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1985110100.EN04/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1985110100.EN06/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1985110100.EN07/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1985110100.EN08/53501875.260529-072257
zhan391/WCYCL20TR_ne30pg2_r05_IcoswISC30E3r5_BruteForce_1985110100.EN09/53501875.260529-072257

How does SimBoard (NERSC) “Path-based ingestion” process such directories? What “unique key” identifies each entry, for selection in the front-end UI?

Clearly, I cannot “flatten” these directories down to mere “exec_id” directories in a tarfile. I will have to include the full (PA-root-relative) path to each execution_id.

[tomvothecoder]

First, the subordinate (PACE-produced)? PA directories contained NO exec_id subdirectories - only a few text files.

SimBoard and PACE only ingest cases with at least one fully complete execution (an execution directory that contains all the required files with required metadata).

As far as I understand, sometimes case dirs and their execution dirs may not contain many files because the run is incomplete or killed. These directories are ignored during the ingestion process with state management.

How does SimBoard (NERSC) “Path-based ingestion” process such directories? What “unique key” identifies each entry, for selection in the front-end UI?

SimBoard ingests each execution (as a simulation object) and assigns it a UUID (id field) in the database. It should be fine for different cases to share the same execution_id, as that is not the canonical identifier of uniqueness (although not sure why this would happened unless a job was executed in parallel at the exact same time).

Refer to the simulations API for more info on the simulation schema: https://simboard-dev-api.e3sm.org/api/v1/simulations

Clearly, I cannot “flatten” these directories down to mere “exec_id” directories in a tarfile. I will have to include the full (PA-root-relative) path to each execution_id.

To clarify, are you writing your own bash scripts to understand the directory layout of the performance archive and how to tar them up? This is a good learning exercise to understand the structure and find any nuances in the process.

As a reminder, #207 already implements remote upload ingestion with state management. #169 will be updated to adapt this process for Chrysalis as needed. Once #169 is done, we can try deploying Chrysalis.

[TonyB9000]

@tomvothecoder No bash scripts. I am experimenting with a python "simboard_backend_remote_collection.py" app.

It sets up logging, acquires a "PID_lock" file (to prevent a scheduler from launching it if a previous invocation is still running. (Would be nice if PACE did the same - then we wouldn't clobber one another). It finds ALL paths to "exec_id" directories, and is then prepared to pass that list to a "qualified_candidates" function that tests for "completeness". THEN prepares to create a TGZ file for upload.

This python app can be seen in "/home/ac.bartoletti1/gitrepo/simboard/backend/app/scripts/ingestion". As I wrote last week, I was going to push it somewhere for safe-keeping but ran into permission issues.

The list I gave above repeats the execution_id "53501875.260529-072257" 25 times, within different subdirectories. I had not yet investigated their "completeness". But "bruteForce_1984050100.EN0[0-9], BruteForce_1985050100.EN0[0-9] seem quite deliberate.

Here is a run on the "small" AP_directory at chrysalis (stubbed for qualified candidates and tarfile creation):

20260601_202232_218:INFO:__main__:Created new lock file: PID_lock_3483781
20260601_202232_219:INFO:__main__:Processing can proceed
20260601_202232_219:INFO:__main__:Begin Simboard Remote Ingestion (NO ARGS)
20260601_202232_219:INFO:__main__:Source PA directory:/lcrc/group/e3sm/performance_archive
20260601_202253_490:INFO:__main__:Obtained cadidate executions: (108 found)
20260601_202253_490:INFO:__main__:PA_Assessment: Raw USER count: 68 exec_ids.
20260601_202253_490:INFO:__main__:PA_Assessment: Raw PA count:   40 exec_ids.
20260601_202253_490:INFO:__main__:PA_Assessment: Uniq USER count: 68 exec_ids.
20260601_202253_491:INFO:__main__:PA_Assessment: Uniq PA count:   40 exec_ids.
20260601_202253_491:INFO:__main__:PA_Assessment: USER exec_id date range: 210610-153146 to 260201-175038
20260601_202253_491:INFO:__main__:PA_Assessment:   PA exec_id date range: 250120-195205 to 250920-103642
20260601_202253_491:INFO:__main__:Completed initial candidate assessment
20260601_202253_491:INFO:__main__:Obtained qualified cadidates: (108 found)
20260601_202253_491:INFO:__main__:Begin transfer package creation (name = some_unique_name.tar.gz)

I will examine #207 to see what I can glean from it.

[TonyB9000]

@tomvothecoder wrote:

SimBoard ingests each execution (as a simulation object) and assigns it a UUID (id field) in the database. It should be fine for different cases to share the same execution_id, as that is not the canonical identifier of uniqueness (although not sure why this would happened unless a job was executed in parallel at the exact same time)

According to Rob Jacob, zhan391 wrote a single "job" that created all ensembles, so they all ran with the same "exec_id":

Rob wrote:

That zhan391 situation is interesting. He is doing ensemble simulations (the "ENXY"). What he must be doing is submitting one job which executes multiple ensembles at once. That would give you one execution id with many cases.

P.S. This "database UUID" - is it derived in any way from the case/exec numbers, or is it entirely synthetic? I need to know what I must extract from a given "case_id"/"exec_id" directory to compare with "state" delivered by the database.

[tomvothecoder]

P.S. This "database UUID" - is it derived in any way from the case/exec numbers, or is it entirely synthetic?

UUID is an auto-generated primary key on the PostgreSQL database. It is synthetic and not based on any identifier from the case and execution directories. This ensures uniqueness on the database level.

I need to know what I must extract from a given "case_id"/"exec_id" directory to compare with "state" delivered by the database.

Tony, SimBoard already employs ingestion scripts that does all of this work.

This is the NERSC one for local path ingestion at NERSC: https://github.com/E3SM-Project/simboard/blob/main/backend/app/scripts/ingestion/nersc_archive_ingestor.py

1. Look through a mounted NERSC performance archive
2. Find valid E3SM execution directories.
3. Compare what it found against what SimBoard says was already ingested (via ingestion state API).
4. Submit ingestion requests only for cases with new executions.
5. Log a summary and exit with success or failure.

This is the remote one for remote upload at other DOE sites like LCRC: https://github.com/E3SM-Project/simboard/blob/main/backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py. It is mostly the same as the NERSC script above.

1. Look through an HPC performance archive available on the machine running the script.
2. Find valid E3SM execution directories.
3. Compare what it found against what SimBoard says was already ingested (via ingestion state API).
4. Package only cases with new executions into .tar.gz archives.
5. Upload those archives to SimBoard for ingestion.
6. Log a summary and exit with success or failure.

We just need to modify these scripts to work on the correct performance archive directory based on our PACE findings. Any edge cases you find are good to know (e.g., duplicate execution ids for different cases), but the actual design and implementation of the ingestion process is mostly done already.

Please review these scripts if you want a better understanding of the process. There is no need to duplicate work here.

[TonyB9000]

@tomvothecoder OK, so I will conduct a careful walkthrough of "nersc_archive_ingestor.py", with respect to the present root PA directory (on NERSC), fully determine how "state" is compared and applied, take note of the logs indicating "submission of (path-based) ingestion requests" and their outcomes.

I must assume, if "PARTIAL" is a state, that SimBoard lets users know when "runs" (UUIDs) are incomplete and why. Otherwise, if SimBoard ingestion only accepts "completed" runs, there should not be a "PARTIAL" state to report.

I will then transition to Chrysalis, determine the Chrysalis-specific parameters that must be changed, and attempt some "dry_run" generation of logs to verify my understanding.