Skip to content

Commit 241b6ef

Browse files
authored
QC-1142 fix object preservation for 1_per_run policy (#2228)
In some cases (e.g. a postprocessing task trending moving windows), the "validFrom" of the created objects might evolve slightly to the past. This is what we may see at the beginning of run: I 11 09:25:34.936 check/sink-PTOF_TOFTrendingO1e7a TOF 2mZYw7qfc8t 549286 Validity of MO 'TOFTrendingOrbits' is (1711956219957, 1711956273223) I 11 09:26:34.055 check/sink-PTOF_TOFTrendingO1e7a TOF 2mZYw7qfc8t 549286 Validity of MO 'TOFTrendingOrbits' is (1711956219854, 1711956333793) I 11 09:27:34.090 check/sink-PTOF_TOFTrendingO1e7a TOF 2mZYw7qfc8t 549286 Validity of MO 'TOFTrendingOrbits' is (1711956219854, 1711956392709) On the other hand, the 1_per_run policy tries to sort the objects by "validFrom" and only if they are equal, it sorts them by "createdAt". Thus, in the discussed case it will preserve the object which is the latest in terms of "validForm", thus the object from the beginning of the run. This commit fixes 1_per_run policy by relying solely on "createdAt" to find the latest object for a run. I could not think of any cases when this could not be applied.
1 parent a793c24 commit 241b6ef

1 file changed

Lines changed: 9 additions & 8 deletions

File tree

  • Framework/script/RepoCleaner/qcrepocleaner/rules

Framework/script/RepoCleaner/qcrepocleaner/rules/1_per_run.py

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ def process(ccdb: Ccdb, object_path: str, delay: int, from_timestamp: int, to_t
1515
'''
1616
Process this deletion rule on the object. We use the CCDB passed by argument.
1717
Objects which have been created recently are spared (delay is expressed in minutes).
18-
This specific policy, 1_per_run, keeps only the most recent version for a given run based on the validity_from.
18+
This specific policy, 1_per_run, keeps only the most recent version for a given run based on the createdAt.
1919
2020
Config Parameters:
2121
- period_pass: Keep 1 version for a combination of run+pass+period if true.
@@ -68,16 +68,17 @@ def process(ccdb: Ccdb, object_path: str, delay: int, from_timestamp: int, to_t
6868
# Dispatch the versions to deletion and preservation lists
6969
for bucket, run_versions in versions_buckets_dict.items():
7070
# logger.debug(f"- bucket {bucket}")
71-
sorted_run_versions = sorted(run_versions, key=lambda x: (x.validFrom, -x.createdAt))
71+
sorted_run_versions = sorted(run_versions, key=lambda x: (x.createdAt))
7272

7373
freshest: ObjectVersion = None
7474
for v in sorted_run_versions:
75-
if freshest is None or freshest.validFromAsDt < v.validFromAsDt:
76-
if freshest is not None:
77-
if policies_utils.in_grace_period(freshest, delay):
78-
preservation_list.append(freshest)
79-
else:
80-
deletion_list.append(freshest)
75+
if freshest is None:
76+
freshest = v
77+
elif freshest.createdAtDt < v.createdAtDt:
78+
if policies_utils.in_grace_period(freshest, delay):
79+
preservation_list.append(freshest)
80+
else:
81+
deletion_list.append(freshest)
8182
freshest = v
8283
else:
8384
if policies_utils.in_grace_period(freshest, delay):

0 commit comments

Comments
 (0)