Skip to content

embedded-using.mdx

Latest commit

 

History

History
234 lines (159 loc) · 15.1 KB

embedded-using.mdx

File metadata and controls

234 lines (159 loc) · 15.1 KB

import UpdateOverview from "../docs/partials/embedded-cluster/v3/_update-overview.mdx" import EcConfig from "../docs/partials/embedded-cluster/v3/_ec-config.mdx" import ShellCommand from "../docs/partials/embedded-cluster/_shell-command.mdx" import RewriteHelmValues from "../docs/partials/proxy-service/_step-rewrite-helm-values.mdx" import DependencyYaml from "../docs//partials/replicated-sdk/_dependency-yaml.mdx" import SupportAirGapV3 from "../docs/partials/embedded-cluster/v3/_support-air-gap-releases.mdx"

Configure Embedded Cluster (Beta)

This topic describes how to configure and use Replicated Embedded Cluster with your application. For more information about Embedded Cluster, see Embedded Cluster Overview. For information about updating an existing release from Embedded Cluster v2 to v3, see Migrate from Embedded Cluster v2.

Create a release with Embedded Cluster v3

To create an application release that supports installation with Embedded Cluster v3:

  1. If you use the Replicated proxy registry, update all references to private or third-party images to use the Replicated proxy registry domain. See the Embedded Cluster v3 steps in Configure your application to use the proxy registry.

  2. In your application Helm chart Chart.yaml file, add the SDK as a dependency. If your application uses multiple charts, declare the SDK as a dependency of the chart that customers install first. Do not declare the SDK in more than one chart.

  3. Package each chart into a .tgz chart archive. See Package a Helm chart for a release.

  4. For each chart archive, add a unique HelmChart v2 custom resource (version kots.io/v1beta2).

    # HelmChart custom resource
apiVersion: kots.io/v1beta2
kind: HelmChart
metadata:
  name: samplechart
spec:
  # chart identifies a matching chart from a .tgz
  chart:
    name: samplechart
    chartVersion: 3.1.7

  5. If you support air gap installations, update all image references so they resolve correctly in both online and air gap installations. See Add support for air gap installations on this page.

  6. Add an Embedded Cluster Config manifest to the release. At minimum, the Config must specify the Embedded Cluster version to use.

  7. If you use custom domains for the Replicated proxy registry or Replicated app service, add them to the Embedded Cluster Config domains key. See Configure Embedded Cluster to use custom domains in Use custom domains.

  8. If you need Embedded Cluster to deploy certain components to the cluster before it deploys your application, add the Helm charts for those components to the Embedded Cluster Config extensions key. See (Optional) Add Helm chart extensions on this page.

  9. Save the release and promote it to the channel that you use for testing internally.

  10. Install with Embedded Cluster in a development environment to test. See Online installation with Embedded Cluster or Air gap installation with Embedded Cluster.

Set up the wizard's Configure screen {#configure-screen}

During installation and upgrades, the Embedded Cluster wizard includes a Configure step where end customers provide configuration values for your application. This screen is defined by the Config custom resource in your release.

The Config custom resource lets you define groups of configuration fields that appear in the wizard. Each group becomes a section in the sidebar, and customers navigate between groups to complete the configuration.

How it works

The Config custom resource defines one or more groups, each containing one or more items. During installation, the wizard renders these as form fields. The values that customers provide are available to your Helm chart through Replicated template functions.

For example, the following Config custom resource defines two groups:

apiVersion: kots.io/v1beta1
kind: Config
metadata:
  name: config-sample
spec:
  groups:
    - name: database
      title: Database Configuration
      description: Choose 'Embedded' to deploy PostgreSQL within the cluster, or 'External' to connect to your own PostgreSQL instance.
      items:
        - name: database_type
          title: PostgreSQL Type
          type: select_one
          default: embedded
          items:
            - name: embedded
              title: Embedded PostgreSQL
            - name: external
              title: External PostgreSQL
        - name: external_host
          title: PostgreSQL Host
          type: text
          when: '{{repl ConfigOptionEquals "database_type" "external"}}'
    - name: features
      title: Features
      items:
        - name: enable_feature_x
          title: Enable Feature X
          type: bool
          default: "0"

In the wizard, this renders as two groups in the sidebar ("Database Configuration" and "Features"). Customers select their database type, and if they choose "External PostgreSQL", additional fields appear for the database connection details.

Key concepts

  • Groups define the sidebar sections. Each group has a title that appears in the sidebar and a list of items.
  • Items define individual form fields. Supported types include text, password, bool, select_one, textarea, and more. See the Config custom resource reference for all available types.
  • Conditional fields use the when property with template functions to show or hide fields based on other values. This lets you create dynamic forms that adapt to customer choices.
  • Default values and generated values (such as auto-generated passwords) reduce the number of fields customers need to fill in manually.
  • Validation can be added using the validation property to check field values against regex patterns before the customer proceeds.

During upgrades, the wizard pre-populates the Configure screen with the customer's existing values, allowing them to review and update their configuration before proceeding.

For the complete field reference, see Config custom resource.

Customize the wizard branding {#branding}

You can customize the appearance of the Embedded Cluster install and upgrade wizard using the Application custom resource in your release. The Application custom resource lets you set a custom title and icon that appear throughout the wizard.

Set the title and icon

Add an Application custom resource to your release with the title and icon fields:

apiVersion: kots.io/v1beta1
kind: Application
metadata:
  name: my-app
spec:
  title: My Application
  icon: https://example.com/icon.png

The title appears in the wizard header on every screen, along with "Installation Wizard" or "Upgrade Wizard" as the subtitle. The icon appears next to the title. If no icon is provided, a default letter avatar is generated from the first character of the title.

:::note For air gap installations, use a Base64 encoded image for the icon field because remote URLs are not accessible in air gap environments. :::

For more information about the Application custom resource fields, see Application custom resource.

Add support for air gap installations {#local-image-registry}

This section describes how to support air gap installations with Embedded Cluster v3. It includes information about how to configure your release and lists the limitations and known issues of air gap installations with Embedded Cluster v3.

Configure the release

Limitations and known issues

Embedded Cluster installations in air gap environments have the following limitations and known issues:

  • If you pass ?airgap=true to the replicated.app endpoint but an air gap bundle is not built for the latest release, the API will not return a 404. Instead it will return the tarball without the air gap bundle (as in, with the installer and the license in it, like for online installations).

  • Images used by Helm extensions must not refer to a multi-architecture image by digest. Air gap bundles include only x64 images, and the digest for the x64 image will be different from the digest for the multi-architecture image, preventing Kubernetes from locating the image in the bundle. An example of a chart that does this is ingress-nginx/ingress-nginx chart. For an example of how to set digests to empty strings and pull by tag only, see extensions in Embedded Cluster Config.

  • Embedded Cluster loads images for Helm extensions directly into containerd so that they are available without internet access. But if an image used by a Helm extension has Always set as the image pull policy, Kubernetes will try to pull the image from the internet. If necessary, use the Helm values to configure IfNotPresent as the image pull policy to ensure the extension works in air gap environments.

  • On the channel release history page, the links for Download air gap bundle, Copy download URL, and View bundle contents pertain to the application air gap bundle only, not the Embedded Cluster bundle.

Add Helm chart extensions {#add-extensions}

If your application requires certain components deployed before the application and as part of the cluster itself, add them as extensions in the Embedded Cluster Config. For example, you can add a Helm extension to deploy an ingress controller. You can add extensions for Helm charts that you own or for third-party charts.

To add Helm extensions:

  1. In the Embedded Cluster Config, add the Helm chart to the extensions key.

  2. If you support air gap installations, configure each of your extensions so that they resolve correctly for both online and air gap installations. See Add support for air gap installations on this page.

  3. Save the release and promote it to the channel that you use for testing internally.

  4. Install with Embedded Cluster in a development environment to test. See Online installation with Embedded Cluster or Air gap installation with Embedded Cluster.

Serve installation assets using the Vendor API

To install with Embedded Cluster, your end customers need to download the Embedded Cluster installer binary and their license. Air gap installations also require an air gap bundle. End customers can download all these installation assets using a curl command by following the installation steps available in the Replicated Enterprise Portal.

However, some vendors already have a portal where their customers can log in to access documentation or download artifacts. In cases like this, you can serve the Embedded Cluster installation assets yourself using the Replicated Vendor API. This removes the need for customers to download assets from the Replicated app service using a curl command during installation.

To serve Embedded Cluster installation assets with the Vendor API:

  1. If you have not done so already, create an API token for the Vendor API. See Use the Vendor API v3.

  2. Call the Get an Embedded Cluster release endpoint to download the assets needed to install your application with Embedded Cluster. Your customers must take this binary and their license and copy them to the machine where they will install your application.

    Note the following:

    • (Recommended) Provide the customerId query parameter so that the downloaded tarball includes the customer’s license. This mirrors what the Replicated app service returns when a customer downloads the binary directly and is the most useful option. Excluding the customerId is useful if you plan to distribute the license separately.

    • If you do not provide any query parameters, this endpoint downloads the Embedded Cluster binary for the latest release on the specified channel. You can provide the channelSequence query parameter to download the binary for a particular release.

Distribute the NVIDIA gpu operator with Embedded Cluster {#nvidia-gpu-operator}

:::note Distributing the NVIDIA GPU Operator with Embedded Cluster is not an officially supported feature from Replicated. However, it is a common use case. :::

The NVIDIA GPU Operator uses the operator framework within Kubernetes to automate the management of all NVIDIA software components needed to provision GPUs. For more information about this operator, see the NVIDIA GPU Operator documentation.

Include the NVIDIA gpu operator and configure containerd options

You can include the NVIDIA GPU Operator in your release as an additional Helm chart, or using Embedded Cluster Helm extensions. For information about adding Helm extensions, see extensions in Embedded Cluster Config.

Using the NVIDIA GPU Operator with Embedded Cluster requires configuring the containerd options in the operator as follows:

# Embedded Cluster Config

  extensions:
    helm:
      repositories:
        - name: nvidia
          url: https://nvidia.github.io/gpu-operator
      charts:
        - name: gpu-operator
          chartname: nvidia/gpu-operator
          namespace: gpu-operator
          version: "v24.9.1"
          values: |
            # configure the containerd options
            toolkit:
             env:
             - name: CONTAINERD_CONFIG
               value: /etc/k0s/containerd.d/nvidia.toml
             - name: CONTAINERD_SOCKET
               value: /run/k0s/containerd.sock

containerd known issue

When you configure the containerd options as shown earlier on this page, the NVIDIA GPU Operator automatically creates the required configurations in the /etc/k0s/containerd.d/nvidia.toml file. It is not necessary to create this file manually, or modify any other configuration on the hosts.

If you include the NVIDIA GPU Operator as a Helm extension, remove any existing containerd services from the host before installing with Embedded Cluster. This includes services deployed by Docker. If any containerd services are present on the host, the NVIDIA GPU Operator will generate an invalid containerd config, causing the installation to fail. For more information, see Installation failure when NVIDIA GPU Operator is included as Helm extension in Troubleshooting Embedded Cluster.

This is the result of a known issue with v24.9.x of the NVIDIA GPU Operator. For more information about the known issue, see container-toolkit does not modify the containerd config correctly when there are multiple instances of the containerd binary in the nvidia-container-toolkit repository in GitHub.