feat: add pilot database and router

[Robin-Van-de-Merghel]

Split of #421 , first part : pilot management

[fstagni]

First review.

[fstagni]

All exceptions apart from the 2 below inherit from DiracError, and this one is further customization for only a few related to pilot. What if instead you modify DiracError?

[Robin-Van-de-Merghel]

I don't know really if I should keep as before classes with code duplication, or if a generic thingy will help... I had that in mind, but seems overkill:

class DiracFormattedError(Exception):
     http_status_code = HTTPStatus.BAD_REQUEST  # 400
    pattern = "Error [args]"

    def __init__(self, data: dict[str, str], detail: str = ""):
        self.data = data
        self.detail = detail
        super().__init__(self._render_message())

    def _render_message(self) -> str:
        context = {
            **self.data,
            "args": self._format_args(),
            "detail": self._format_detail(),
        }

        return re.sub(r'\[([^\]]+)\]', lambda m: context.get(m.group(1), ""), self.pattern)

    def _format_args(self) -> str:
        if not self.data:
            return ""
        return " ".join(f"({k}: {v})" for k, v in self.data.items())

    def _format_detail(self) -> str:
        return f": {self.detail}" if self.detail else ""

Then:

class PilotAlreadyExistsError(DiracFormattedError):
    http_status_code = HTTPStatus.CONFLICT 
    pattern = "Pilot [args] already exists[detail]"

[Robin-Van-de-Merghel]

Where an issue?

Whether we have a base router with require_auth or not, we won't be able to override it in its children (cf #417 ).
So for pilots we must have require_auth=False at the base router because of secret-exchange. But now every pilot endpoint is opened.

Possibilities

Splitting

Let's start with the routers themselves. We can separate pilots and users endpoints : /api/pilots (only for pilots) and ~/api/pilot_management (only for users).

That brings us:

1. Cleaner: each their own territory, no overlapping, easier to read (you know that on /pilots its only pilot resources)
2. More secure: we can add a dependency to the pilot router (same as verify_dirac_pilot_token but for pilots)
3. We can have require_auth for /pilot_management, and enforce tokens

Using another approach

We could have a bare base router without dependency injection, with sub routers with verify_dirac_access_token:

# Authenticated parent router
protected_router = APIRouter(dependencies=[Depends(verify_dirac_access_token)])

@protected_router.get("/secure")
def secure_route():
    return {"msg": "secure"}

# Public sub-router (no auth)
public_router = APIRouter()

@public_router.get("/public")
def public_route():
    return {"msg": "public"}

Its goal would be to replacer DiracxRouter by fixing the issue, and keeping an explicit depedency injection.
Or have @router.authentificated.get(...) instead of @router.get, to explicitely say if we want auth or not

[Robin-Van-de-Merghel]

No description provided.

[Robin-Van-de-Merghel]

(For deletePilots: delete mapping also, same later for logs #511 + PilotOutput)

[Robin-Van-de-Merghel]

Left open as a reminder (already did mapping and output)

[read-the-docs-community]

Documentation build overview

📚 diracx | 🛠️ Build #32257809 | 📁 Comparing 6e47a7a against latest (ea405cc)

  🔍 Preview build  

2 files changed

+ admin/explanations/pilots/index.html
+ dev/explanations/pilots/index.html

[fstagni]

This PR is I think ready for being tested in the certification setup. In the meantime, please review.

[aldbr]

It's ready to be reviewed again @fstagni

diracx-core

• PilotFieldsMapping → PilotMetadata. Renamed for clarity ("fields mapping" didn't say what it was), and every field now has a Pydantic Field(description=...) so the OpenAPI schema is self-documenting.
• ScalarSearchSpec.value: removed datetime. Wasn't needed.

diracx-db

• Method rename pass: add_pilots → register_pilots, add_jobs_to_pilot → assign_jobs_to_pilot, update_pilot_fields → update_pilot_metadata. Names now match what they do.
• Heterogeneous bulk metadata update. update_pilot_metadata accepts a list of PilotMetadata objects where each row may set a different subset of fields. Mirrors the pattern used in JobDB.set_job_attributes.
• Removed unused search_pilot_to_job_mapping (no callers).
• Schema typing cleanup. Optional[datetime] → datetime | None, dropped the typing.Optional import to match the rest of diracx-db/sql.

diracx-logic

• logic/pilots/management.py. register_new_pilots, delete_pilots, update_pilots_metadata, assign_jobs_to_pilot. The delete_pilots entry point handles both stamp-based and age-based branches; the age-based branch is reserved for the maintenance task worker, am I understanding correctly?
• logic/pilots/query.py. search, summary, get_pilots_by_stamp, plus the JobID pseudo-parameter rewriter (see below). Adds a _add_vo_constraint helper that mirrors the intra-VO filter pattern from logic/jobs/query.py.
• JobID ↔ PilotStamp pseudo-parameters. Both POST /api/jobs/search and POST /api/pilots/search now accept a pseudo-parameter that is transparently resolved through JobToPilotMapping:
  • POST /api/pilots/search accepts JobID (eq/in only), rewritten into a PilotID IN (...) filter.
  • POST /api/jobs/search accepts PilotStamp (eq/in only), rewritten into a JobID IN (...) filter.
  • Mixing a pseudo-parameter with its real counterpart in the same request body is refused with InvalidQueryError. Unsupported operators are rejected with the same error (the join semantics would be ambiguous). If the resolution produces an empty set, the search short-circuits to an empty result.

diracx-routers

• POST /api/pilots/ — bulk register. Rejects any caller with GENERIC_PILOT property sending more than one stamp per call (limits the blast radius of a stolen pilot credential). Non-admins can only register pilots for their own VO.
• DELETE /api/pilots/ — stamp-based delete only. Deletes the pilot rows, their logs, and their job associations. Age-based retention is not exposed on the HTTP API (handled by a diracx-task?).

Docs

Updated docs/admin/explanations/pilots.md and docs/dev/explanations/pilots.md.