Replace per-iter conan info subprocess with cached /search API (#2098)

## Summary

Library-builder hot path was shelling out to `conan info -r ceserver .` once per build combination to derive the conan package_id. Replaced with a single `GET /v1/conans/{lib}/{ver}/{lib}/{ver}/search` cached on the builder instance, with client-side settings matching.

Pattern ported from `compiler-explorer/lib/buildenvsetup/ceconan.ts` (@partouf's existing rewrite on the Node.js side). Match rules - headeronly / cshared compiler wildcards, libcxx and arch wildcards under those, stdver always-match - extracted as module-level helpers (`match_conan_settings`, `conan_search_url`, `build_target_settings` in `library_builder.py`) and shared across all four builder classes (c++, rust, fortran, go) to avoid drift.

Refs #1342.

## Performance

Measured locally with conan 1.66 against the live conan proxy:

| | per-iteration cost |
|---|---|
| old (`conan info` subprocess) | ~335ms |
| new (cached `/search` + dict scan) | ~0.5ms after a one-off ~1s search |

| iterations | old | new | speedup |
|--:|--:|--:|--:|
| 20 | 6.7s | 1.0s | 7x |
| 40 | 13.4s | 1.0s | 13x |
| 200 | 67s | 1.1s | 60x |

End-to-end timing on a real lin-builder is the natural follow-up; couldn't run that locally because CE customises conan's `settings.yml` with non-standard values (e.g. `compiler.version=g141`, the `flagcollection` setting) that only exist on the builder bootstrap.

## Subtle bits

- **`set_as_uploaded` ordering.** The new search endpoint can only see packages already on the remote, so the original ordering (`get_conan_hash` -> conditional `upload_builds`) would have crashed every first-time upload with `RuntimeError: Error determining conan hash`. Reordered to `get_build_annotations` -> conditional `upload_builds` -> `get_conan_hash`. `upload_builds` invalidates `_possible_builds`, so the post-upload `get_conan_hash` sees the fresh search response. `test_set_as_uploaded_first_time_does_not_raise` is the regression test for this path; it fails on the un-reordered version.
- **TS port faithfulness.** Match rules check the candidate's specific key (e.g. `candidate.get("compiler.version") in ("headeronly", "cshared")`) rather than a single up-front predicate, mirroring the TS implementation. Worth comparing to `ceconan.ts:249-275` if reviewing the matcher.
- **URL encoding** uses `urllib.parse.quote(s, safe="")` to match `encodeURIComponent` semantics for lib/version path segments.
- **Cache-invalidation point.** All four `upload_builds` set `self._possible_builds = None` after a successful upload, so the next `get_conan_hash` in the same builder instance refetches.

## Out of scope

- TTL on `hasFailedBefore` (the trunk-poisoning aspect of #1342). Separate, server-side concern.
- Consolidating `resil_get` / HTTP-retry into a shared helper across the four builders. Pre-existing drift, not made worse by this PR.

## Test plan

- [x] `make test` (636 passed, 1 skipped)
- [x] `make static-checks` clean
- [x] Two passes of code review by an LLM agent
- [x] Live validation against `conan.compiler-explorer.com`: real boost_bin/clang_barry hash returned by the matcher resolves to a real annotation; libcxx mismatch returns None; eigen/3.4.0 headeronly matches across compiler/libcxx/arch but rejects on os mismatch
- [x] Regression test for fresh-upload path verified to fail when the ordering fix is reverted
- [ ] End-to-end timing on a real lin-builder (follow-up; needs CE's customised conan settings)

Author: mattgodbolt

bin/lib/fortran_library_builder.py

@@ -25,6 +25,7 @@
 from lib.amazon_properties import get_properties_compilers_and_libraries, get_specific_library_version_details
 from lib.installation_context import FetchFailure, PostFailure
 from lib.library_build_config import LibraryBuildConfig
+from lib.library_builder import PossibleBuilds, build_target_settings, fetch_possible_builds, find_matching_package_id
 from lib.library_platform import LibraryPlatform
 from lib.staging import StagingDir
 
@@ -46,7 +47,6 @@
 _supports_x86: dict[str, Any] = defaultdict(lambda: [])
 
 GITCOMMITHASH_RE = re.compile(r"^(\w*)\s.*")
-CONANINFOHASH_RE = re.compile(r"\s+ID:\s(\w*)")
 
 
 def _quote(string: str) -> str:
@@ -98,7 +98,7 @@ def __init__(
         self.needs_uploading = 0
         self.libid = self.libname  # TODO: CE libid might be different from yaml libname
         self.conanserverproxy_token = None
-        self._conan_hash_cache: dict[str, str | None] = {}
+        self._possible_builds: PossibleBuilds | None = None
         self._annotations_cache: dict[str, dict] = {}
         self.http_session = requests.Session()
 
@@ -450,25 +450,21 @@ def makebuildhash(self, compiler, options, toolchain, buildos, buildtype, arch,
 
         return compiler + "_" + hasher.hexdigest()
 
-    def get_conan_hash(self, buildfolder: str) -> str | None:
-        if buildfolder in self._conan_hash_cache:
-            self.logger.debug(f"Using cached conan hash for {buildfolder}")
-            return self._conan_hash_cache[buildfolder]
-
-        if not self.install_context.dry_run:
-            self.logger.debug(["conan", "info", "."] + self.current_buildparameters)
-            conaninfo = subprocess.check_output(
-                ["conan", "info", "-r", "ceserver", "."] + self.current_buildparameters, cwd=buildfolder
-            ).decode("utf-8", "ignore")
-            self.logger.debug(conaninfo)
-            match = CONANINFOHASH_RE.search(conaninfo, re.MULTILINE)
-            if match:
-                result = match[1]
-                self._conan_hash_cache[buildfolder] = result
-                return result
+    def _get_possible_builds(self) -> PossibleBuilds:
+        if self._possible_builds is None:
+            self._possible_builds = fetch_possible_builds(
+                self.libname,
+                self.target_name,
+                lambda url: self.http_session.get(url, timeout=_TIMEOUT),
+                self.logger,
+            )
+        return self._possible_builds
 
-        self._conan_hash_cache[buildfolder] = None
-        return None
+    def get_conan_hash(self, buildfolder: str) -> str | None:
+        if self.install_context.dry_run:
+            return None
+        target = build_target_settings(self.current_buildparameters_obj)
+        return find_matching_package_id(self._get_possible_builds(), target)
 
     def conanproxy_login(self):
         url = f"{conanserver_url}/login"
@@ -581,15 +577,19 @@ def is_already_uploaded(self, buildfolder):
             return False
 
     def set_as_uploaded(self, buildfolder):
+        # We need the conan package_id (a deterministic SHA from compiler+version+libcxx+arch+...)
+        # to PUT annotations against /annotations/{lib}/{ver}/{package_id}. get_conan_hash now
+        # derives the id by querying the server's /search index, which only lists packages
+        # already on the server -- so for a freshly built package, we must upload first.
+        annotations = self.get_build_annotations(buildfolder)
+        if "commithash" not in annotations:
+            self.upload_builds()
+
         conanhash = self.get_conan_hash(buildfolder)
         if conanhash is None:
             raise RuntimeError(f"Error determining conan hash in {buildfolder}")
 
-        self.logger.info(f"commithash: {conanhash}")
-
-        annotations = self.get_build_annotations(buildfolder)
-        if "commithash" not in annotations:
-            self.upload_builds()
+        self.logger.info(f"conanhash: {conanhash}")
         annotations["commithash"] = self.get_commit_hash()
 
         self.logger.info(annotations)
@@ -705,6 +705,7 @@ def upload_builds(self):
                 self.logger.debug("Clearing cache to speed up next upload")
                 subprocess.check_call(["conan", "remove", "-f", f"{self.libname}/{self.target_name}"])
             self.needs_uploading = 0
+            self._possible_builds = None
 
     def get_compiler_type(self, compiler):
         compilerType = ""

bin/lib/go_library_builder.py

@@ -11,7 +11,6 @@
 import json
 import logging
 import os
-import re
 import shutil
 import subprocess
 from collections import defaultdict
@@ -29,6 +28,7 @@
 from lib.golang_stdlib import go_supports_trimpath
 from lib.installation_context import InstallationContext, PostFailure
 from lib.library_build_config import LibraryBuildConfig
+from lib.library_builder import PossibleBuilds, build_target_settings, fetch_possible_builds, find_matching_package_id
 from lib.library_platform import LibraryPlatform
 from lib.staging import StagingDir
 
@@ -77,8 +77,6 @@ def get_build_method(compiler_id: str) -> str:
 # Cache for compiler properties
 _propsandlibs: dict[str, Any] = defaultdict(lambda: [])
 
-CONANINFOHASH_RE = re.compile(r"\s+ID:\s(\w*)")
-
 
 def clear_properties_cache() -> None:
     """Clear the compiler properties cache. Used for testing."""
@@ -133,7 +131,7 @@ def __init__(
         # Prefix with 'go_' to avoid Conan namespace collisions with other languages
         self.libid = f"go_{self.libname}"
         self.conanserverproxy_token: str | None = None
-        self._conan_hash_cache: dict[str, str | None] = {}
+        self._possible_builds: PossibleBuilds | None = None
         self._annotations_cache: dict[str, dict] = {}
         self.http_session = requests.Session()
 
@@ -412,28 +410,22 @@ def writeconanfile(self, buildfolder: Path) -> None:
             f.write('        self.copy("module_sources/*", dst=".", keep_path=True)\n')
             f.write('        self.copy("metadata.json", dst=".", keep_path=False)\n')
 
-    def get_conan_hash(self, buildfolder: Path) -> str | None:
-        """Query Conan for package hash."""
-        if str(buildfolder) in self._conan_hash_cache:
-            return self._conan_hash_cache[str(buildfolder)]
+    def _get_possible_builds(self) -> PossibleBuilds:
+        if self._possible_builds is None:
+            self._possible_builds = fetch_possible_builds(
+                self.libid,
+                self.target_name,
+                lambda url: self.http_session.get(url, timeout=_TIMEOUT),
+                self.logger,
+            )
+        return self._possible_builds
 
-        if not self.install_context.dry_run:
-            try:
-                conaninfo = subprocess.check_output(
-                    ["conan", "info", "-r", "ceserver", "."] + self.current_buildparameters,
-                    cwd=buildfolder,
-                    timeout=_TIMEOUT,
-                ).decode("utf-8", "ignore")
-                match = CONANINFOHASH_RE.search(conaninfo)
-                if match:
-                    result = match[1]
-                    self._conan_hash_cache[str(buildfolder)] = result
-                    return result
-            except (subprocess.CalledProcessError, subprocess.TimeoutExpired) as e:
-                self.logger.debug("Conan info failed: %s", e)
-
-        self._conan_hash_cache[str(buildfolder)] = None
-        return None
+    def get_conan_hash(self, buildfolder: Path) -> str | None:
+        """Find the conan package_id matching the current build parameters, via the proxy search API."""
+        if self.install_context.dry_run:
+            return None
+        target = build_target_settings(self.current_buildparameters_obj)
+        return find_matching_package_id(self._get_possible_builds(), target)
 
     def resil_post(self, url: str, json_data: str, headers: dict | None = None) -> requests.Response | dict:
         """Resilient POST request with retries."""
@@ -557,15 +549,19 @@ def is_already_uploaded(self, buildfolder: Path) -> bool:
 
     def set_as_uploaded(self, buildfolder: Path, build_method: str) -> None:
         """Mark build as uploaded in Conan server."""
+        # We need the conan package_id (a deterministic SHA from compiler+version+libcxx+arch+...)
+        # to PUT annotations against /annotations/{lib}/{ver}/{package_id}. get_conan_hash now
+        # derives the id by querying the server's /search index, which only lists packages
+        # already on the server -- so for a freshly built package, we must upload first.
+        annotations = self.get_build_annotations(buildfolder)
+        if "commithash" not in annotations:
+            self.upload_builds()
+
         conanhash = self.get_conan_hash(buildfolder)
         if conanhash is None:
             raise RuntimeError(f"Error determining conan hash in {buildfolder}")
 
         self.logger.info("conanhash: %s", conanhash)
-
-        annotations = self.get_build_annotations(buildfolder)
-        if "commithash" not in annotations:
-            self.upload_builds()
         annotations["commithash"] = self.target_name
         annotations["build_method"] = build_method
 
@@ -606,6 +602,7 @@ def upload_builds(self) -> None:
                 self.logger.debug("Clearing cache to speed up next upload")
                 subprocess.check_call(["conan", "remove", "-f", f"{self.libid}/{self.target_name}"])
             self.needs_uploading = 0
+            self._possible_builds = None
 
     def build_cleanup(self, buildfolder: Path) -> None:
         """Clean up build folder."""