feat: build from private repositories (#1250)

* feat: add private image prefix to config

* feat(repositories/db): add token getter

This will be used to create the secret allowing
Shipwright to create images out of private repositories.

* feat(session/models): add authentication secret

This will allow to pass authentication information to
the Shipwright client for private repositories.

* feat(session/k8s_client): implement support for clone secret

When getting an authentication secret in the parameter, the
client will now:

- create a secret for the BuildRun
- configure the BuildRun to use it
- patch the secret to be owned by the BuildRun object

The last one ensures that the secret is delete alongside
the BuildRun object.

Behaviour change: the BuildRun object is refreshed so that
the cache will contain its UID which is mandatory for the
ownership setup.

* feat(session/db): implement repo visibility check and token retrieval

Now retrieving the build parameter will include the token
for private repositories.

* feat(session/constants): update default buildstragy to v3

It's the name of the current strategy in components/renku_pack_builder.

* refactor(test): allow reuse of existing cluster in tests

This allows to reduce testing time with regard to cluster
creation and setup.

* feat(test): add optional Shipwright setup

This allows to run tests that will create BuildRun objects.

* feat(test): implement support for real Shipwright client

This will allow to run tests as they are currently as well
as with Shipwright deployed.

* refactor(test_sessions): update tests for builds

They now can run with and without builds activated.

Note: nothing will get pushed nor will BuildRun succeed
to finish. The goal is to ensure that thing are properly
working from an object creation point of view.

* refactor(k8s/client): cache object once created

Currently the cache is populated before the object is
created and the entry is deleted if creation failed.

This has two drawbacks:

- requires two operations to the cache that might fail
- does not store the latest version of the object if
  refresh is true

For the latter, a third call could be done to update the
cache content but it's sub-optimal.

An up to date cache content is required for the support of
setting the owner references of the clone secret for BuildRuns
that must manage private repositories.

* feat(session): push images from private repo to a different registry

This allows to cleanly separate were both set of images are
stored.

* refactor(test/session): add BuildRun validation

These checks ensure that the BuildRun objects are:
- Using the proper registry to push to
- Get the secret for private builds

* refactor(session/db): better handling of union return types

* fix(test/utils): improve fake get_repository return

- "/some/repo" now returns a GitUrlError
- Not managed cases calls base class implementation

* refactor(test/utils): only use FakeGitRepositoriesRepository when testing builds

* refactor(k8s/client): change create back to almost original implementation

The optimization done somehow breaks information retrieval
during test with getting duplicated entries in some tests.

Revert back to the original implementation but with added
cache update when object refresh is requested.

* fix(test): move cluster-creation option handling in main conftest

* refactor(test/sessions): move BuildRun content check behind feature check

* fix: add "sessions" marker to pyproject.toml

This addresses a warning from pytest

* feat: ensure kind configuration file is test run specific

* fix(session/db): add missing parameter to make_session_environment_repo

* refactor(test): rename class starting with Test

They are detected by pytest as potential tests cases
although they are only helpers.

Renaming fixes the associated warning.

* fix(builds): add missing secret annotation for Shipwright

* refactor: port deprecated Config use to model_config

This fixes all the pydantic warnings and makes the
code base consistant.

* feat: add configuration for push secret for private registry

* fix(test): make DummyAuthenticator return a AuthenticatedAPIUser

This matches the behaviour of real authenticator

* feat(notebooks): add check based on build parameters

Ensure, when a session start, that if an image is from
a built environment, the image source repository is
contained in the list of repositories from the project.

* feat(test): fix and implement session start tests

* feat(builds): add repo accessibility check when starting a session

Ensure that the user has pull access from the repository when
starting a session that was built.

Visibility is not enough as for example in the case of GitLab,
a user can have access to the code in GitLab but is not allowed
to clone the repository.

See https://docs.gitlab.com/user/permissions/ and more specifically
the guest role.

* chore(tests): fix tuple deprecation warnings from SQLAlchemy

* refactor(DummyAuthenticator): return an empty APIUser in case of missing id

This distinguishes between a successful and a failed authentication
attempt vs an anonymous user.

* chore(test): fix handling of parametrize arguments

* feat(crc): make the config file pattern more flexible

While containing yaml, k8s config files do not end with
.yaml by default so if testing with a local cluster, the
tests will fail.

* fix(tests): fix authorisation headers for resource pool creation

Author: sgaist

bases/renku_data_services/data_api/app.py

@@ -221,6 +221,7 @@ def register_all_handlers(app: Sanic, dm: DependencyManager) -> Sanic:
         session_repo=dm.session_repo,
         storage_repo=dm.storage_repo,
         user_repo=dm.kc_user_repo,
+        git_repositories_repo=dm.git_repositories_repo,
     )
     platform_config = PlatformConfigBP(
         name="platform_config",

bases/renku_data_services/data_api/dependencies.py

@@ -291,7 +291,6 @@ def from_env(cls) -> DependencyManager:
             )
             if config.builds.enabled:
                 k8s_db_cache = K8sDbCache(config.db.async_session_maker)
-                default_kubeconfig = KubeConfigEnv()
                 shipwright_client = ShipwrightClient(
                     client=K8sClusterClientsPool(
                         lambda: get_clusters(
@@ -342,12 +341,21 @@ def from_env(cls) -> DependencyManager:
             group_repo=group_repo,
             search_updates_repo=search_updates_repo,
         )
+
+        git_repositories_repo = GitRepositoriesRepository(
+            session_maker=config.db.async_session_maker,
+            oauth_client_factory=oauth_http_client_factory,
+            internal_gitlab_url=config.gitlab_url,
+            enable_internal_gitlab=config.enable_internal_gitlab,
+        )
+
         session_repo = SessionRepository(
             session_maker=config.db.async_session_maker,
             project_authz=authz,
             resource_pools=rp_repo,
             shipwright_client=shipwright_client,
             builds_config=config.builds,
+            git_repositories_repo=git_repositories_repo,
         )
         project_migration_repo = ProjectMigrationRepository(
             session_maker=config.db.async_session_maker,
@@ -378,12 +386,6 @@ def from_env(cls) -> DependencyManager:
             user_repo=kc_user_repo,
             secret_service_public_key=config.secrets.public_key,
         )
-        git_repositories_repo = GitRepositoriesRepository(
-            session_maker=config.db.async_session_maker,
-            oauth_client_factory=oauth_http_client_factory,
-            internal_gitlab_url=config.gitlab_url,
-            enable_internal_gitlab=config.enable_internal_gitlab,
-        )
         platform_repo = PlatformRepository(
             session_maker=config.db.async_session_maker,
         )

components/renku_data_services/authn/dummy.py

@@ -46,6 +46,7 @@ class DummyAuthenticator:
     async def authenticate(self, access_token: str, request: Request) -> base_models.APIUser:
         """Indicates whether the user has successfully logged in."""
         access_token = request.headers.get(self.token_field) or ""
+
         if not access_token or len(access_token) == 0:
             # Try to get an anonymous user ID if the validation of keycloak credentials failed
             anon_id = request.headers.get(self.anon_id_header_key)
@@ -60,23 +61,17 @@ async def authenticate(self, access_token: str, request: Request) -> base_models
         with contextlib.suppress(Exception):
             user_props = json.loads(access_token)
 
-        is_set = bool(
-            user_props.get("id")
-            or user_props.get("full_name")
-            or user_props.get("is_admin") is not None
-            or user_props.get("first_name")
-            or user_props.get("last_name")
-            or user_props.get("email")
-        )
+        if user_props.get("id") is None:
+            return base_models.APIUser()
 
-        return base_models.APIUser(
+        return base_models.AuthenticatedAPIUser(
             is_admin=user_props.get("is_admin", False),
-            id=user_props.get("id", "some-id") if is_set else None,
+            id=user_props.get("id", ""),
             access_token=access_token,
-            first_name=user_props.get("first_name", "John") if is_set else None,
-            last_name=user_props.get("last_name", "Doe") if is_set else None,
-            email=user_props.get("email", "john.doe@gmail.com") if is_set else None,
-            full_name=user_props.get("full_name", "John Doe") if is_set else None,
+            first_name=user_props.get("first_name"),
+            last_name=user_props.get("last_name"),
+            email=user_props.get("email", ""),
+            full_name=user_props.get("full_name"),
             refresh_token=request.headers.get("Renku-Auth-Refresh-Token"),
             roles=user_props.get("roles", []),
         )

components/renku_data_services/capacity_reservation/apispec_base.py

@@ -2,17 +2,17 @@
 
 from typing import Any
 
-from pydantic import BaseModel, field_validator
+from pydantic import BaseModel, ConfigDict, field_validator
 from ulid import ULID
 
 
 class BaseAPISpec(BaseModel):
     """Base API specification."""
 
-    class Config:
-        """Enables orm mode for pydantic."""
-
-        from_attributes = True
+    # Enables orm mode for pydantic.
+    model_config = ConfigDict(
+        from_attributes=True,
+    )
 
     @field_validator("*", mode="before", check_fields=False)
     @classmethod

components/renku_data_services/connected_services/apispec_base.py

@@ -1,19 +1,19 @@
 """Base models for API specifications."""
 
-from pydantic import BaseModel, Field, field_validator
+from pydantic import BaseModel, ConfigDict, Field, field_validator
 from ulid import ULID
 
 
 class BaseAPISpec(BaseModel):
     """Base API specification."""
 
-    class Config:
-        """Enables orm mode for pydantic."""
-
-        from_attributes = True
+    model_config = ConfigDict(
+        # Enables orm mode for pydantic."""
+        from_attributes=True,
         # NOTE: By default the pydantic library does not use python for regex but a rust crate
         # this rust crate does not support lookahead regex syntax but we need it in this component
-        regex_engine = "python-re"
+        regex_engine="python-re",
+    )
 
     @field_validator("id", mode="before", check_fields=False)
     @classmethod
@@ -25,20 +25,14 @@ def serialize_id(cls, id: str | ULID) -> str:
 class AuthorizeParams(BaseAPISpec):
     """The schema for the query parameters used in the authorize request."""
 
-    class Config:
-        """Configuration."""
-
-        extra = "ignore"
+    model_config = ConfigDict(extra="ignore")
 
     next_url: str = Field(default="")
 
 
 class CallbackParams(BaseAPISpec):
     """The schema for the query parameters used in the authorize callback request."""
 
-    class Config:
-        """Configuration."""
-
-        extra = "ignore"
+    model_config = ConfigDict(extra="ignore")
 
     state: str = Field(default="")

components/renku_data_services/crc/apispec.py

@@ -250,6 +250,8 @@ class UsersUserIdResourcePoolsGetParametersQuery(BaseAPISpec):
     user_resource_params: Optional[UserResourceParams] = None
 
 
+config_name_pattern = r"^[a-zA-Z0-9._-]+(\.yaml)?$"
+
 class Cluster(BaseAPISpec):
     model_config = ConfigDict(
         extra="forbid",
@@ -264,7 +266,7 @@ class Cluster(BaseAPISpec):
         ...,
         description="The name of the Kubernetes configuration to use to connect to the remote cluster. This is currently used to find a file named `<KubeConfigRoot>/<config_name>`.\n\nThis configuration is expected to have a default namespace defined. It will be used for all remote operations requiring a namespace, as well for namespaced objects.\n",
         examples=["a-remote-cluster.yaml"],
-        pattern="^[a-zA-Z0-9._-]+[.]yaml$",
+        pattern=config_name_pattern,
     )
     session_protocol: Protocol
     session_host: str = Field(
@@ -301,7 +303,7 @@ class ClusterPatch(BaseAPISpec):
         None,
         description="The name of the Kubernetes configuration to use to connect to the remote cluster. This is currently used to find a file named `<KubeConfigRoot>/<config_name>`.\n\nThis configuration is expected to have a default namespace defined. It will be used for all remote operations requiring a namespace, as well for namespaced objects.\n",
         examples=["a-remote-cluster.yaml"],
-        pattern="^[a-zA-Z0-9._-]+[.]yaml$",
+        pattern=config_name_pattern,
     )
     session_protocol: Optional[Protocol] = None
     session_host: Optional[str] = Field(
@@ -338,7 +340,7 @@ class ClusterWithId(BaseAPISpec):
         ...,
         description="The name of the Kubernetes configuration to use to connect to the remote cluster. This is currently used to find a file named `<KubeConfigRoot>/<config_name>`.\n\nThis configuration is expected to have a default namespace defined. It will be used for all remote operations requiring a namespace, as well for namespaced objects.\n",
         examples=["a-remote-cluster.yaml"],
-        pattern="^[a-zA-Z0-9._-]+[.]yaml$",
+        pattern=config_name_pattern,
     )
     id: str = Field(
         ...,

components/renku_data_services/crc/apispec_base.py

@@ -3,7 +3,7 @@
 from pathlib import PurePosixPath
 from typing import Any
 
-from pydantic import BaseModel, field_validator
+from pydantic import BaseModel, ConfigDict, field_validator
 from ulid import ULID
 
 from renku_data_services.session import models
@@ -12,10 +12,10 @@
 class BaseAPISpec(BaseModel):
     """Base API specification."""
 
-    class Config:
-        """Enables orm mode for pydantic."""
-
-        from_attributes = True
+    # Enables orm mode for pydantic.
+    model_config = ConfigDict(
+        from_attributes=True,
+    )
 
     @field_validator("*", mode="before", check_fields=False)
     @classmethod

components/renku_data_services/crc/server_options.py

@@ -8,7 +8,7 @@
 from collections.abc import Generator
 from typing import Any, Union
 
-from pydantic import BaseModel, ByteSize, Field, validator
+from pydantic import BaseModel, ByteSize, ConfigDict, Field, validator
 
 from renku_data_services.crc import models
 from renku_data_services.crc.constants import DEFAULT_RUNTIME_PLATFORM
@@ -28,17 +28,13 @@ class ServerOptionsDefaults(BaseModel):
     disk_request: ByteSize
     gpu_request: int = Field(ge=0, default=0)
 
-    class Config:
-        """Configuration."""
-
-        extra = "ignore"
+    model_config = ConfigDict(extra="ignore")
 
 
 class _ServerOptionsCpu(BaseModel):
     options: list[float] = Field(min_length=1)
 
-    class Config:
-        extra = "ignore"
+    model_config = ConfigDict(extra="ignore")
 
     @validator("options", pre=False, each_item=True)
     def greater_than_zero(cls, val: Union[float, int]) -> Union[float, int]:
@@ -61,8 +57,7 @@ def greater_than_or_equal_to_zero(cls, v: int) -> int:
 class _ServerOptionsBytes(BaseModel):
     options: list[ByteSize] = Field(min_length=1)
 
-    class Config:
-        extra = "ignore"
+    model_config = ConfigDict(extra="ignore")
 
     @validator("options", pre=True)
     def convert_units(cls, vals: list[str]) -> list[str]:
@@ -84,10 +79,7 @@ class ServerOptions(BaseModel):
     disk_request: _ServerOptionsBytes
     gpu_request: _ServerOptionsGpu = Field(default_factory=lambda: _ServerOptionsGpu(options=[0]))
 
-    class Config:
-        """Configuration."""
-
-        extra = "ignore"
+    model_config = ConfigDict(extra="ignore")
 
     def find_largest_attribute(self) -> str:
         """Find the attribute with the largest number of choices."""

components/renku_data_services/data_connectors/apispec_base.py

@@ -1,19 +1,19 @@
 """Base models for API specifications."""
 
-from pydantic import BaseModel, field_validator
+from pydantic import BaseModel, ConfigDict, field_validator
 from ulid import ULID
 
 
 class BaseAPISpec(BaseModel):
     """Base API specification."""
 
-    class Config:
-        """Enables orm mode for pydantic."""
-
-        from_attributes = True
+    model_config = ConfigDict(
+        # Enables orm mode for pydantic."""
+        from_attributes=True,
         # NOTE: By default the pydantic library does not use python for regex but a rust crate
         # this rust crate does not support lookahead regex syntax but we need it in this component
-        regex_engine = "python-re"
+        regex_engine="python-re",
+    )
 
     @field_validator("id", mode="before", check_fields=False)
     @classmethod

components/renku_data_services/k8s/clients.py

@@ -346,13 +346,18 @@ async def create(self, obj: K8sObject, refresh: bool) -> K8sObject:
             if not obj.namespaced():
                 raise NotImplementedError("Caching of cluster scoped K8s resources is not supported")
             await self.__cache.upsert(obj)
+
         try:
             obj = await super().create(obj, refresh)
         except:
             # if there was an error creating the k8s object, we delete it from the db again to not have ghost entries
             if obj.gvk in self.__kinds_to_cache:
                 await self.__cache.delete(obj)
             raise
+
+        if refresh and obj.gvk in self.__kinds_to_cache:
+            await self.__cache.upsert(obj)
+
         return obj
 
     async def patch(self, meta: K8sObjectMeta, patch: K8sPatches) -> K8sObject: