Using cgroups to limit job resource usage

[chrisburr]

We recently failed to add support for cgroups in #7723 due to there being no session dbus instance started for the non-interactive jobs started by the batch system.

@fstagni pointed out to me that ALICE has made progress with this in:

https://indico.cern.ch/event/1338689/contributions/6010982/attachments/2952948/5191366/CHEP2024_Subdivision_GM.pdf

We should see if we can profit from it also.

cc @sfayer as you helped me understand this in the first place.

[fstagni]

Few additional minor bullets after talking with the author from ALICE:

• they patched HtCondor and SLURM
• the WN needs to be Rhel9
• they create a "slot cgroup" and inside it sub-cgroups
• they create first the cgroup and inside it they launch apptainer (IIUC we were try to do the opposite) and inside it the job
• check the backup slides for some implementation (and caveat) details

[sfayer]

Hi,

I had a quick look on our production HTCondor at IC and we do seem to have the patch already: A few of our cgroup control files for the per-slot cgroup tree are owned by the job user (rather than root), so it should be possible to create custom groups below that.

We're running the feature release of condor (to fix an unrelated cgroup bug!), so I'm not sure if it'll be available at most condor sites yet, but it certainly looks promising... I can't comment about the other batch systems, I felt they seemed less optimistic about getting the patches generally available for those in the talk.

Regards,
Simon

[sfayer]

Hi,

I wasn't really sure what you'd like me to do about this, but I made a guess:

I did a quick survey of UK sites and around 60% of them support the required cgroup2 delegation now (it's all the sites that use HTCondor batch, regardless of CE type, minus a few that do special things).

I've written a demo class/test program for doing the cgroup stuff, which I've tested on the UK HTCondor CEs successfully:
https://gist.github.com/sfayer/304b053c38d2f76a4ce4d2547160fd05
There are a number of caveats listed at the start.

The next steps (probably for someone who has worked on the pilot/CE code more recently than me), would likely be to:

• Integrate the cgroup class into the pilot and have it set-up the cgroups on start-up if they're available.
• Write a version of (or modify) systemCall / shellCall to be cgroup aware (avoiding any subprocess race conditions, which is the main challenge as described in the demo).
• Update the execution CEs (SingularityCE/InProcessCE/PoolCE) to create a slot cgroups and do the modified systemCall/shellCall with the cgroup at the correct point (and optionally use this for more precise accounting information at the end, keeping in mind that it's not always going to be available).

Regards,
Simon

[fstagni]

Fixed in #8110

[fstagni]

Hi, I am looking into the development done for this issue, and I have a few questions:

• IIUC as implemented the functionality will work per CE (and if the WN allows it). Suppose that jobs of different "size" are submitted (e.g. Simulation jobs requesting 1 processor and 2 GBs, and user jobs requesting 8 processors and 8 GBs). At the moment it does not seem like there's a way to apply different limits for different jobs, am I right?
• A use case for using cGroups limits are partitioning of (possibly very) large WN slots (for DIRAC it means using Pool Computing Element, with InProcess or Singularity). Do I understand correctly that we are missing a connection with the knowledge of the slot limits?

[sfayer]

Hi,

Yes, as it stands the source of the limits is fairly restrictive as they're only read from the ceParameters, based on the design from the original pull request (#7723).

I think the answers to both questions are fairly similar: As far as I'm aware, the dynamic "slot size" information isn't readily available at the levels it would need to be:

• At the pilot/JobAgent level, the overall size of the slot should be passed down to the execution CE (based on what was requested at the submission CE? or the ceParameters value?).
• The Pool Computing Element should then take the total slot size, subdivide it as it splits off the sub-slots and then pass that value down to the innerCE.

If that information was available, it'd be trivial to use it to set the cgroup limits.

I'm not sure the user job sizes specifically matter for this: I generally think of it from the slot perspective where the limits are set to the "slot size" (which should always be the same or bigger than the "job size" as long as the matching has worked correctly).

Regards,
Simon

[fstagni]

In general, DIRAC has a missing feature when it comes down to the size of the slot: the notion of RAM needed (from the job) and available (from the slot) is not fully there. Specifically, I think (I'd need to re-check the details) it's not taken into consideration at matching time.
The pilot at the same time knows the slot parameters, and so does the JobAgent. I will take note to look into this.

[fstagni]

Contrary to what I wrote previously, DIRAC has the notion of RAM needed (as requirement from the job) and available (from the WN slot). As requirement, it is as a "Tag", e.g. "8GB". The RAM available in the slot is passed to the Matcher as "MaxRAM", and a matching is performed considering it. What was missing is the case of PoolComputingElement, for which I just made PR #8232