[int] Add cleanDirectory utility function for general directory clean-up

[sfayer]

Hi,

Here is the reworked version of this to create a new cleanDirectory utility function and use it in a few places where it makes sense (including the HTCondorCE cleanup).

Regards,
Simon

BEGINRELEASENOTES
*Core
CHANGE: Add cleanDirectory utility function for general directory clean-up
ENDRELEASENOTES

[fstagni]

I asked copilot if it could identify similar patterns. Answer in https://github.com/copilot/share/081500a8-01a0-88f7-b801-0801e4c80916

Snippet from the above:

There are multiple locations that could benefit from a generic file cleanup utility like __cleanDir:

PilotLoggingAgent.clearOldPilotLogs() – Lines 200-221

Currently manually iterates files and checks st_mtime
Could use __cleanDir with appropriate pattern matching

SecurityFileLog.__walkOldLogs() – Lines 86-104

Recursively walks directories and deletes files older than a threshold
Uses regex patterns for file matching
Very similar logic to __cleanDir – could be refactored

TransformationAgent.cleanOldTransformationCache() – Lines 172-186

Deletes cache files older than 60 days based on st_mtime
Could use __cleanDir with glob pattern *.pkl

[fstagni]

Given you are on it, could you propose the following changes?

[sfayer]

Hi,

Just to clarify this before I go off in the wrong direction:: I think you're asking me to make a general utility function for this and then use it in these places (wherever it makes sense); is that correct?

Regards,
Simon

[fstagni]

Yes, correct.

[aldbr]

For my education, what are you trying to solve here exactly?

[sfayer]

Here is my draft for this: We're testing it on one of our test servers before I'll mark it for review. Please feel free to have a quick look and let me know if it needs any major changes before we finish testing it!

It didn't make sense for the TransformationAgent.cleanOldTransformationCache case: This would need an extra filter (to check for files that are still being used) and the whole thing would be much more complicated that the existing code.

Regards,
Simon