Skip to content

feat: remove PilotManager dependency from HTCondorCE#8211

Merged
fstagni merged 1 commit into
DIRACGrid:integrationfrom
aldbr:v9.0_FEAT_remove-deps-from-htcondorce-to-pilotmanager
Jun 2, 2025
Merged

feat: remove PilotManager dependency from HTCondorCE#8211
fstagni merged 1 commit into
DIRACGrid:integrationfrom
aldbr:v9.0_FEAT_remove-deps-from-htcondorce-to-pilotmanager

Conversation

@aldbr

@aldbr aldbr commented Jun 2, 2025

Copy link
Copy Markdown
Contributor

The following change was not accepted in #5406, but I still think it makes sense because:

  • we have the following relationship: WMS agents/services use Resources/Computing (Resources/Computing should work as a standalone, so without having to interact with a WMS service).
  • there is no reliable way to get the CE status from HTCondor itself
  • the caller (Site Director) now tries to get the CE status from the CE itself, and if it does not work, tries from the PilotAgentsDB, so we duplicate the operation:
    result = ce.available()
    if result["OK"]:
    ceInfoDict = result["CEInfoDict"]
    self.log.info(
    "CE queue report",
    f"({ceName}_{queueName}): Wait={ceInfoDict['WaitingJobs']}, Run={ceInfoDict['RunningJobs']}, Max={ceInfoDict['MaxTotalJobs']}",
    )
    return (result["Value"], ceInfoDict["WaitingJobs"])
    # If we cannot get available slots from the CE, then we get them from the pilotAgentsDB
    maxWaitingJobs = int(self.queueDict[queue]["ParametersDict"].get("MaxWaitingJobs", 10))
    maxTotalJobs = int(self.queueDict[queue]["ParametersDict"].get("MaxTotalJobs", 10))
    # Get the number of transient pilots
    result = self.pilotAgentsDB.countPilots(
    {"DestinationSite": ceName, "Queue": queueName, "Status": PilotStatus.PILOT_TRANSIENT_STATES}
    )
    if not result["OK"]:
    self.log.warn("Failed to check PilotAgentsDB", f"for queue {queue}: \n{result['Message']}")
    self.failedQueues[queue] += 1
    return (0, 0)
    totalJobs = result["Value"]
    # Get the number of waiting pilots
    result = self.pilotAgentsDB.countPilots(
    {"DestinationSite": ceName, "Queue": queueName, "Status": PilotStatus.PILOT_WAITING_STATES}
    )
    if not result["OK"]:
    self.log.warn("Failed to check PilotAgentsDB", f"for queue {queue}: \n{result['Message']}")
    self.failedQueues[queue] += 1
    return (0, 0)
    waitingJobs = result["Value"]
    runningJobs = totalJobs - waitingJobs
    self.log.info(
    "PilotAgentsDB report",
    f"({ceName}_{queueName}): Wait={waitingJobs}, Run={runningJobs}, Max={maxTotalJobs}",
    )
    totalSlots = min((maxTotalJobs - totalJobs), (maxWaitingJobs - waitingJobs))
  • if we would try to call Resources/Computing from diracx, we would have troubles because HTCondorComputingElement relies on a DIRAC service.

Any reason to keep this dependency?

BEGINRELEASENOTES
*Resources
CHANGE: remove dependency from HTCondorComputingElement to PilotManager
ENDRELEASENOTES

@aldbr aldbr requested review from atsareg and fstagni as code owners June 2, 2025 13:39
Comment thread src/DIRAC/Resources/Computing/HTCondorCEComputingElement.py
@aldbr aldbr force-pushed the v9.0_FEAT_remove-deps-from-htcondorce-to-pilotmanager branch from 6dd34b5 to b237b3d Compare June 2, 2025 14:10

return result
"""Method to return information on running and pending jobs."""
return S_ERROR("getCEStatus not implemented for HTCondorCEComputingElement")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is slightly misleading, as one can read it like "not yet implemented, maybe it will". So better specify here why it's better not implemented.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed the message, please let me know if you want further details.

@aldbr aldbr force-pushed the v9.0_FEAT_remove-deps-from-htcondorce-to-pilotmanager branch from b237b3d to cdca59b Compare June 2, 2025 14:24
@fstagni fstagni merged commit 12c6a51 into DIRACGrid:integration Jun 2, 2025
24 checks passed
@DIRACGridBot DIRACGridBot added the sweep:ignore Prevent sweeping from being ran for this PR label Jun 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sweep:ignore Prevent sweeping from being ran for this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants