Implement Sparse Checkout for GitRepository

[dipti-pai]

Sparse Checkout Directories in GitRepositories.

- Add `.spec.sparseCheckout` and `.status.observedSparseCheckout` fields to `GitRepository`.
- Add controller support to send the sparse checkout directories to go-git via pkg methods.
- Use `.status/observedSparseCheckout` to detect drift in configuration.
- Trim leading "./" in directory paths.
- Validate spec configuration by checking directories specified in spec exist in the cloned repository after successful checkout
- Add tests for testing the observed sparse checkout behavior.
- Add docs describing the new fields.

Fixes: #1707

Current behavior worth calling out.

1. If .spec.sparseCheckout directory list includes a directory that does not exist, the underlying go-git checkout method does not return an error and silently continues. See SparseCheckoutDirectories behavior when directory path does not exist go-git/go-git#1500 . This is being handled in the controller.
2. If .spec.sparseCheckout directory list includes directory with more than one level of nesting, sparse checkout is not honored and all directories at top-level are checked out. Will be fixed by SparseCheckoutDirectories works only for 1st level directory. go-git/go-git#1455
3. If .spec.sparseCheckout uses relative path beginning with a ./, the path is ignored and nothing is checked out and no errors are thrown. See SparseCheckoutDirectories ignores relative paths beginning with (./) while checking out git repository go-git/go-git#1506 . This is being handled in source-controller.
   4.. If .spec.sparseCheckout includes an empty directory, the entire repository is checked out (Expected)

[stefanprodan]

If .spec.sparseCheckout uses relative path beginning with a ./, the path is ignored and nothing is checked out and no errors are thrown.

@dipti-pai I think we should handle this on our own. We could trim ./ by prefix, before passing the array to the Git client.

[pjbgf]

From the abnormal behaviours called out, #2 was the only one which there would not be a way to work around straight from the controller. That is now fixed and merged into main.

I'd agree with @stefanprodan on #3, and the fix LGTM.

For #1, we could workaround by adding a os.Lstat after the clone operation to confirm the dir exists - only when sparse checkout is used. @stefanprodan WDYT?

[stefanprodan]

we could workaround by adding a os.Lstat after the clone operation to confirm the dir exists - only when sparse checkout is used

Yes we could verify the dirs and error out in the controller 👍

[dipti-pai]

Updated the code to handle incorrect user configuration where .spec.sparseCheckout points to a directory that does not exist. Tested scenarios where there is an error in configuration during the initial setup of gitrepository and scenarios where initially the repository is cloned successfully, followed by an updated configuration that has an error.

Spec:
  Interval:  5m
  Ref:
    Branch:  master
  Sparse Checkout:
    charts
    **./kustomizeabc/**
  Timeout:  60s
  URL:      https://github.com/stefanprodan/podinfo
Status:
  Artifact:
    Digest:            sha256:b78631f0a004c1b31891d1a481e9962312bcb7cef306c3cebbdfc3fca484a551
    Last Update Time:  2025-04-15T19:35:36Z
    Path:              gitrepository/default/podinfo/b3396adb98a6a0f5eeedd1a600beaf5e954a1f28.tar.gz
    Revision:          master@sha1:b3396adb98a6a0f5eeedd1a600beaf5e954a1f28
    Size:              14671
    URL:               http://source-controller.kustomize-system.svc.cluster.local./gitrepository/default/podinfo/b3396adb98a6a0f5eeedd1a600beaf5e954a1f28.tar.gz
  Conditions:
    Last Transition Time:  2025-04-15T19:40:08Z
    Message:               **failed to sparse checkout directories : sparse checkout dir './kustomizeabc/' does not exist in repository: lstat /tmp/gitrepository-default-podinfo-1053626913/kustomizeabc: no such file or directory**
    Observed Generation:   7
    Reason:                GitOperationFailed
    Status:                True
    Type:                  Stalled
    Last Transition Time:  2025-04-15T19:40:08Z
    Message:               failed to sparse checkout directories : sparse checkout dir './kustomizeabc/' does not exist in repository: lstat /tmp/gitrepository-default-podinfo-1053626913/kustomizeabc: no such file or directory
    Observed Generation:   7
    Reason:                GitOperationFailed
    Status:                False
    Type:                  Ready
    Last Transition Time:  2025-04-15T19:40:08Z
    Message:               failed to sparse checkout directories : sparse checkout dir './kustomizeabc/' does not exist in repository: lstat /tmp/gitrepository-default-podinfo-1053626913/kustomizeabc: no such file or directory
    Observed Generation:   7
    Reason:                GitOperationFailed
    Status:                True
    Type:                  FetchFailed
    Last Transition Time:  2025-04-15T19:35:36Z
    Message:               stored artifact for revision 'master@sha1:b3396adb98a6a0f5eeedd1a600beaf5e954a1f28'
    Observed Generation:   6
    Reason:                Succeeded
    Status:                True
    Type:                  ArtifactInStorage
  Observed Generation:     7
  Observed Sparse Checkout:
    charts
    **./kustomize**

Once the error in configuration is fixed, the reconciliation succeeds -

Spec:
  Interval:  5m
  Ref:
    Branch:  master
  Sparse Checkout:
    charts
    **./kustomize/**
  Timeout:  60s
  URL:      https://github.com/stefanprodan/podinfo
Status:
  Artifact:
    Digest:            sha256:b78631f0a004c1b31891d1a481e9962312bcb7cef306c3cebbdfc3fca484a551
    Last Update Time:  2025-04-15T19:43:01Z
    Path:              gitrepository/default/podinfo/b3396adb98a6a0f5eeedd1a600beaf5e954a1f28.tar.gz
    Revision:          master@sha1:b3396adb98a6a0f5eeedd1a600beaf5e954a1f28
    Size:              14671
    URL:               http://source-controller.kustomize-system.svc.cluster.local./gitrepository/default/podinfo/b3396adb98a6a0f5eeedd1a600beaf5e954a1f28.tar.gz
  Conditions:
    Last Transition Time:  2025-04-15T19:43:01Z
    Message:               **stored artifact for revision 'master@sha1:b3396adb98a6a0f5eeedd1a600beaf5e954a1f28'**
    Observed Generation:   8
    Reason:                Succeeded
    Status:                True
    Type:                  Ready
    Last Transition Time:  2025-04-15T19:35:36Z
    Message:               stored artifact for revision 'master@sha1:b3396adb98a6a0f5eeedd1a600beaf5e954a1f28'
    Observed Generation:   8
    Reason:                Succeeded
    Status:                True
    Type:                  ArtifactInStorage
  Observed Generation:     8
  Observed Sparse Checkout:
    charts
    **./kustomize/**

[stefanprodan]

LGTM

Thanks @dipti-pai 🏅