feat(hosting): add better k3s support (#1173)

- Introduced generic `RunWithKubernetes<TResource>` overload for
resources exposing kubeconfig as connection string.
- Enhanced CRD management to work seamlessly with
non-`KubernetesEnvironmentResource` clusters.
- Updated tests and documentation to reflect new functionality.

fixes #1170

Author: kimpenhaus

docs/docs/operator/aspire-kubernetes-model.mdx

@@ -223,16 +223,17 @@ builder.AddKubeOps<Projects.AspireOperator>("operator")
 
 KubeOps runtime watchers require a Kubernetes API server. Unit tests can replace `IKubernetesClient`, but the normal operator runtime is not an offline simulator.
 
-Additional targets can be modeled as Kubernetes environments if they provide Kubernetes API semantics:
+Cluster resources that are not `KubernetesEnvironmentResource`s but expose their kubeconfig path as a connection string — such as the `K3sClusterResource` from the [CommunityToolkit Aspire k3s integration](https://github.com/CommunityToolkit/Aspire) — are supported by the generic `RunWithKubernetes<TResource>` overload (constrained to `IResourceWithConnectionString`):
 
 ```csharp
 var k3s = builder.AddK3sCluster("k3s");
-var mock = builder.AddKubeOpsMockKubernetes("mock-kube");
 
 builder.AddKubeOps<Projects.AspireOperator>("operator")
     .RunWithKubernetes(k3s);
 ```
 
+This overload injects the cluster's `KUBECONFIG` into the operator process (`WithReference`), waits for the cluster (`WaitFor`), starts the operator automatically, and manages CRDs through the same lifecycle as the environment overload — no `KubernetesEnvironmentResource` inheritance required. Publishing still targets a `KubernetesEnvironmentResource`.
+
 A mock target would require operator-side runtime support to replace the Kubernetes client and watcher behavior. It should not be represented as a normal Kubernetes environment unless it provides Kubernetes API semantics.
 
 ## Non-Goals

docs/docs/operator/aspire.mdx

@@ -209,6 +209,17 @@ builder.AddKubeOps<Projects.AspireOperator>("operator")
     .RunWithKubernetes(dev, run => run.WithPersistentCrds());
 ```
 
+Local development against a k3s cluster from the [CommunityToolkit Aspire k3s integration](https://github.com/CommunityToolkit/Aspire) (or any resource whose connection string is a kubeconfig path):
+
+```csharp
+var k3s = builder.AddK3sCluster("k3s");
+
+builder.AddKubeOps<Projects.AspireOperator>("operator")
+    .RunWithKubernetes(k3s, run => run.WithPersistentCrds());
+```
+
+The generic `RunWithKubernetes<TResource>` overload accepts any `IResourceWithConnectionString` whose connection string is a kubeconfig path. It injects the cluster's `KUBECONFIG` into the operator process, waits for the cluster to become ready, starts the operator automatically, and manages CRDs exactly like the `KubernetesEnvironmentResource` overload. Publishing (`PublishAsKubernetesOperator`) still requires a `KubernetesEnvironmentResource`.
+
 Azure deployment into AKS:
 
 ```csharp

src/KubeOps.Aspire.Hosting/KubeOpsHostingExtensions.cs

@@ -159,6 +159,53 @@ public static IResourceBuilder<ProjectResource> RunWithKubernetes(
         return builder;
     }
 
+    /// <summary>
+    /// Configures a KubeOps operator project to run locally against a cluster resource that exposes its kubeconfig
+    /// path as a connection string.
+    /// </summary>
+    /// <remarks>
+    /// This overload targets cluster resources that are not <see cref="KubernetesEnvironmentResource"/>s but expose a
+    /// kubeconfig path through <see cref="IResourceWithConnectionString"/> — for example a
+    /// <c>K3sClusterResource</c> from the CommunityToolkit Aspire k3s integration, whose connection string is the
+    /// host-accessible kubeconfig path. The operator process receives the cluster's <c>KUBECONFIG</c> (via
+    /// <c>WithReference</c>), waits for the cluster to become ready (via <c>WaitFor</c>), starts automatically, and has
+    /// its CRDs managed according to the configured <see cref="KubeOpsRunCrdMode"/>.
+    /// </remarks>
+    /// <typeparam name="TResource">The cluster resource type exposing a kubeconfig path as its connection string.</typeparam>
+    /// <param name="builder">The operator project resource builder.</param>
+    /// <param name="target">The cluster resource used by the local operator process.</param>
+    /// <param name="configure">Configures local run behavior.</param>
+    /// <returns>The same project builder after local run configuration is attached.</returns>
+    public static IResourceBuilder<ProjectResource> RunWithKubernetes<TResource>(
+        this IResourceBuilder<ProjectResource> builder,
+        IResourceBuilder<TResource> target,
+        Action<KubeOpsRunOptions>? configure = null)
+        where TResource : class, IResource, IResourceWithConnectionString
+    {
+        ArgumentNullException.ThrowIfNull(builder);
+        ArgumentNullException.ThrowIfNull(target);
+
+        RemoveExplicitStart(builder.Resource);
+
+        var publishAnnotation = builder.Resource.Annotations.OfType<KubeOpsPublishAnnotation>().SingleOrDefault()
+            ?? throw new InvalidOperationException(
+                $"{nameof(RunWithKubernetes)} can only be used with project resources added via " +
+                $"{nameof(AddKubeOps)}.");
+
+        // Inject the cluster's KUBECONFIG into the operator process and defer start/CRD prep until it is ready.
+        builder.WithReference(target);
+        builder.WaitFor(target);
+
+        var options = new KubeOpsRunOptions(
+            target.Resource,
+            cancellationToken => target.Resource.GetConnectionStringAsync(cancellationToken));
+        configure?.Invoke(options);
+        var runAnnotation = new KubeOpsRunAnnotation(options);
+        builder.Resource.Annotations.Add(runAnnotation);
+        ConfigureKubeOpsRun(builder, publishAnnotation, runAnnotation);
+        return builder;
+    }
+
     private static KubeOpsKubernetesManifestOptions CreateDefaultManifestOptions(string name)
         => new()
         {
@@ -243,6 +290,8 @@ private static async Task PrepareRunCrdsAsync(
             return;
         }
 
+        var kubeConfigPath = await run.Options.ResolveKubeConfigPathAsync(cancellationToken);
+
         foreach (var crd in GenerateKubeOpsJsonResources(publish.ProjectPath, publish.Options)
                      .Where(resource => IsKind(resource, "CustomResourceDefinition")))
         {
@@ -252,7 +301,7 @@ private static async Task PrepareRunCrdsAsync(
                 continue;
             }
 
-            var exists = await KubernetesResourceExistsAsync(run.Options.Target, "crd", name, cancellationToken);
+            var exists = await KubernetesResourceExistsAsync(kubeConfigPath, "crd", name, cancellationToken);
             if (run.Options.CrdMode is KubeOpsRunCrdMode.RequireExisting)
             {
                 if (!exists)
@@ -267,7 +316,7 @@ private static async Task PrepareRunCrdsAsync(
 
             if (!exists || run.Options.CrdMode is KubeOpsRunCrdMode.Persistent)
             {
-                await ApplyKubernetesJsonAsync(run.Options.Target, crd, cancellationToken);
+                await ApplyKubernetesJsonAsync(kubeConfigPath, crd, cancellationToken);
             }
 
             if (!exists && run.Options.CrdMode is KubeOpsRunCrdMode.Ephemeral)
@@ -286,24 +335,26 @@ private static async Task CleanupRunCrdsAsync(
             return;
         }
 
+        var kubeConfigPath = await run.Options.ResolveKubeConfigPathAsync(cancellationToken);
+
         foreach (var crd in run.CreatedCrds)
         {
             await RunKubectlAsync(
-                run.Options.Target,
+                kubeConfigPath,
                 ["delete", "crd", crd, "--ignore-not-found"],
                 cancellationToken,
                 throwOnError: false);
         }
     }
 
     private static async Task<bool> KubernetesResourceExistsAsync(
-        KubernetesEnvironmentResource target,
+        string? kubeConfigPath,
         string kind,
         string name,
         CancellationToken cancellationToken)
     {
         var exitCode = await RunKubectlAsync(
-            target,
+            kubeConfigPath,
             ["get", kind, name, "-o", "name"],
             cancellationToken,
             throwOnError: false);
@@ -312,7 +363,7 @@ private static async Task<bool> KubernetesResourceExistsAsync(
     }
 
     private static async Task ApplyKubernetesJsonAsync(
-        KubernetesEnvironmentResource target,
+        string? kubeConfigPath,
         JsonObject resource,
         CancellationToken cancellationToken)
     {
@@ -322,7 +373,7 @@ private static async Task ApplyKubernetesJsonAsync(
 
         try
         {
-            await RunKubectlAsync(target, ["apply", "-f", file], cancellationToken);
+            await RunKubectlAsync(kubeConfigPath, ["apply", "-f", file], cancellationToken);
         }
         finally
         {
@@ -335,7 +386,7 @@ private static async Task ApplyKubernetesJsonAsync(
         "S4036:Searching OS commands in PATH is security-sensitive",
         Justification = "kubectl is intentionally resolved from PATH; its location is environment-specific and cannot be hardcoded.")]
     private static async Task<int> RunKubectlAsync(
-        KubernetesEnvironmentResource target,
+        string? kubeConfigPath,
         IReadOnlyList<string> arguments,
         CancellationToken cancellationToken,
         bool throwOnError = true)
@@ -348,10 +399,10 @@ private static async Task<int> RunKubectlAsync(
             UseShellExecute = false,
         };
 
-        if (!string.IsNullOrWhiteSpace(target.KubeConfigPath))
+        if (!string.IsNullOrWhiteSpace(kubeConfigPath))
         {
             startInfo.ArgumentList.Add("--kubeconfig");
-            startInfo.ArgumentList.Add(target.KubeConfigPath);
+            startInfo.ArgumentList.Add(kubeConfigPath);
         }
 
         foreach (var argument in arguments)

src/KubeOps.Aspire.Hosting/KubeOpsRunOptions.cs

@@ -2,6 +2,7 @@
 // The .NET Foundation licenses this file to you under the Apache 2.0 License.
 // See the LICENSE file in the project root for more information.
 
+using Aspire.Hosting.ApplicationModel;
 using Aspire.Hosting.Kubernetes;
 
 namespace Aspire.Hosting;
@@ -12,20 +13,40 @@ namespace Aspire.Hosting;
 public sealed class KubeOpsRunOptions
 {
     internal KubeOpsRunOptions(KubernetesEnvironmentResource target)
+        : this(target, _ => new ValueTask<string?>(target.KubeConfigPath))
     {
-        Target = target;
+    }
+
+    internal KubeOpsRunOptions(
+        IResource targetResource,
+        Func<CancellationToken, ValueTask<string?>> resolveKubeConfigPathAsync)
+    {
+        TargetResource = targetResource;
+        Target = targetResource as KubernetesEnvironmentResource;
+        ResolveKubeConfigPathAsync = resolveKubeConfigPathAsync;
     }
 
     /// <summary>
-    /// Gets the Kubernetes environment used by the local operator process.
+    /// Gets the Kubernetes environment used by the local operator process, or <see langword="null"/> when the target
+    /// is a connection-string resource (e.g. a k3s cluster) rather than a <see cref="KubernetesEnvironmentResource"/>.
     /// </summary>
-    public KubernetesEnvironmentResource Target { get; }
+    public KubernetesEnvironmentResource? Target { get; }
+
+    /// <summary>
+    /// Gets the underlying target resource used by the local operator process.
+    /// </summary>
+    public IResource TargetResource { get; }
 
     /// <summary>
     /// Gets the CRD lifecycle mode used for local run.
     /// </summary>
     public KubeOpsRunCrdMode CrdMode { get; private set; } = KubeOpsRunCrdMode.Ephemeral;
 
+    /// <summary>
+    /// Resolves the kubeconfig path used to communicate with the target cluster.
+    /// </summary>
+    internal Func<CancellationToken, ValueTask<string?>> ResolveKubeConfigPathAsync { get; }
+
     /// <summary>
     /// Creates missing CRDs before run and removes only CRDs created by this run on shutdown.
     /// </summary>

src/KubeOps.Aspire.Hosting/README.md

@@ -42,6 +42,15 @@ builder.AddKubeOps<Projects.AspireOperator>("operator")
     .RunWithKubernetes(dev, run => run.WithPersistentCrds());
 ```
 
+`RunWithKubernetes` also has a generic overload that accepts any `IResourceWithConnectionString` whose connection string is a kubeconfig path — for example a `K3sClusterResource` from the CommunityToolkit Aspire k3s integration. It injects the cluster's `KUBECONFIG` into the operator process, waits for the cluster, starts the operator automatically, and manages CRDs identically:
+
+```csharp
+var k3s = builder.AddK3sCluster("k3s");
+
+builder.AddKubeOps<Projects.AspireOperator>("operator")
+    .RunWithKubernetes(k3s);
+```
+
 When the AppHost uses Aspire's Kubernetes publishing support, `PublishAsKubernetesOperator(k8s)` invokes `kubeops generate operator` and appends the generated CRDs, RBAC resources, and service account to the Aspire-generated Helm chart. The operator deployment itself remains Aspire-owned, but KubeOps' generated deployment settings are merged into that workload so the chart deploys one correctly wired operator deployment.
 
 ```csharp

test/KubeOps.Aspire.Hosting.Test/KubeOpsHosting.Test.cs

@@ -86,6 +86,52 @@ public void RunWithKubernetes_Can_Use_Persistent_Crds()
             .Which.Options.CrdMode.Should().Be(KubeOpsRunCrdMode.Persistent);
     }
 
+    [Fact]
+    public void RunWithKubernetes_With_ConnectionString_Resource_Enables_Local_Run_With_Ephemeral_Crds_By_Default()
+    {
+        var builder = CreateBuilder();
+        var cluster = builder.AddResource(new FakeClusterResource("fake-k3s"));
+
+        var resourceBuilder = builder.AddKubeOps<TestProjectMetadata>("operator-test")
+            .RunWithKubernetes(cluster);
+
+        resourceBuilder.Resource.Annotations
+            .Should().NotContain(annotation => annotation is ExplicitStartupAnnotation);
+        var runOptions = resourceBuilder.Resource.Annotations.OfType<KubeOpsRunAnnotation>()
+            .Should().ContainSingle().Which.Options;
+        runOptions.CrdMode.Should().Be(KubeOpsRunCrdMode.Ephemeral);
+        runOptions.Target.Should().BeNull();
+        runOptions.TargetResource.Should().BeSameAs(cluster.Resource);
+    }
+
+    [Fact]
+    public void RunWithKubernetes_With_ConnectionString_Resource_Adds_Reference_And_WaitFor()
+    {
+        var builder = CreateBuilder();
+        var cluster = builder.AddResource(new FakeClusterResource("fake-k3s"));
+
+        var resourceBuilder = builder.AddKubeOps<TestProjectMetadata>("operator-test")
+            .RunWithKubernetes(cluster);
+
+        resourceBuilder.Resource.Annotations
+            .Should().Contain(annotation => annotation is EnvironmentCallbackAnnotation)
+            .And.Contain(annotation => annotation is WaitAnnotation);
+    }
+
+    [Fact]
+    public void RunWithKubernetes_With_ConnectionString_Resource_Can_Use_Persistent_Crds()
+    {
+        var builder = CreateBuilder();
+        var cluster = builder.AddResource(new FakeClusterResource("fake-k3s"));
+
+        var resourceBuilder = builder.AddKubeOps<TestProjectMetadata>("operator-test")
+            .RunWithKubernetes(cluster, run => run.WithPersistentCrds());
+
+        resourceBuilder.Resource.Annotations.OfType<KubeOpsRunAnnotation>()
+            .Should().ContainSingle()
+            .Which.Options.CrdMode.Should().Be(KubeOpsRunCrdMode.Persistent);
+    }
+
     [Fact]
     public void PublishAsKubernetesOperator_Configures_Service_Account()
     {
@@ -134,4 +180,14 @@ private sealed class TestProjectMetadata : IProjectMetadata
     {
         public string ProjectPath => typeof(TestProjectMetadata).Assembly.Location;
     }
+
+    // Mimics CommunityToolkit's K3sClusterResource: a non-KubernetesEnvironmentResource cluster that exposes its
+    // kubeconfig path as a connection string injected as KUBECONFIG.
+    private sealed class FakeClusterResource(string name) : Resource(name), IResourceWithConnectionString
+    {
+        public string ConnectionStringEnvironmentVariable => "KUBECONFIG";
+
+        public ReferenceExpression ConnectionStringExpression =>
+            ReferenceExpression.Create($"/tmp/{Name}/kubeconfig.yaml");
+    }
 }