Skip to content

remote-elasticsearch-output.md

Latest commit

 

History

History
234 lines (157 loc) · 12.2 KB

remote-elasticsearch-output.md

File metadata and controls

234 lines (157 loc) · 12.2 KB
Raw
navigation_title Remote Elasticsearch output
mapped_pages
description Remote ES output allows you to send agent data to a remote cluster, keeping data separate and independent from the deployment where you use Fleet.
applies_to
stack serverless
ga
ga
products
id
fleet
id
elastic-agent

Remote {{es}} output [remote-elasticsearch-output]

Remote {{es}} outputs allow you to send {{agent}} data to a remote {{es}} cluster. This is especially useful for data that you want to keep separate and independent from the deployment where you use {{fleet}} to manage the {{agent}}s.

Both the management cluster (where you configure {{fleet}}) and the remote cluster (where {{agents}} send data) can be either an {{stack}} deployment or a {{serverless-short}} project. When the remote cluster is a {{serverless-short}} project, some features are not available. Refer to Limitations.

A remote {{es}} cluster supports the same output settings as your management {{es}} cluster.

:::{note} :applies_to: {serverless: preview, stack: preview 9.5+}

The remote {{es}} output can also serve as the exporter target for OpenTelemetry input packages managed by {{fleet}}. :::

Limitations [remote-output-limitations]

These limitations apply to remote {{es}} output:

  • All {{fleet-server}} hosts that are configured for the remote output must be able to reach the remote {{es}} cluster with a service token to generate API keys for the {{agents}} that use the remote output for data ingestion.
  • Using a remote {{es}} output with a target cluster that has network security enabled is not currently supported.
  • Using {{elastic-defend}} when a remote {{es}} output is configured for an {{agent}} is not currently supported.
  • When the remote {{es}} cluster is a {{serverless-short}} project, automatic integrations synchronization is not available.

Configure the remote output [remote-output-config]

To configure a remote {{es}} cluster for your {{agent}} data:

::::::{stepper}

:::::{step} Create a new remote output

  1. In your management {{es}} cluster, open {{kib}}, and search for Fleet settings in the search bar. Select Fleet/Settings in the results.
  2. In the Outputs section, select Add output.
  3. In the Add new output flyout, provide a name for the output.
  4. Select Remote Elasticsearch as the output type. :::::

:::::{step} Add remote cluster host URL

In the Hosts field, add the URL that {{agents}} should use to access the remote {{es}} cluster.

:::{dropdown} Find the remote host address of the remote cluster :open:

  1. In the remote cluster, open {{kib}}, and search for Fleet settings in the search bar. Select Fleet/Settings in the results.
  2. In the Outputs section, copy the Hosts value of the default {{es}} output. If the value isn't fully visible, edit the default {{es}} output to display the full value.
  3. In your management cluster, paste the value you copied into the Hosts field of the remote output configuration. ::: :::::

:::::{step} Configure service token authentication

In the Service Token field, add a service token to access the remote cluster.

:::{dropdown} Create a service token to access the remote cluster :open:

  1. Copy the API request located below the Service Token field.
  2. In the remote cluster, open {{kib}} and go to Developer tools using the navigation menu or the global search field.
  3. Paste the API request in the console, then run it.
  4. Copy the value for the generated service token.
  5. In the management cluster, paste the value you copied into the Service Token field of the remote output configuration. :::

:::{note} To prevent unauthorized access, the {{es}} service token is stored as a secret value. While secret storage is recommended, you can override this setting and store the password as plain text in the agent policy definition. Secret storage requires {{fleet-server}} version 8.12 or later. This setting can also be stored as a secret value or as plain text for preconfigured outputs. To learn more about this option, check Preconfiguration settings. ::: :::::

:::::{step} Configure SSL certificate authorities (optional)

Configure SSL certificate authorities if the remote {{es}} cluster uses certificates that are not publicly trusted. The certificate authority (CA) is used to sign the remote {{es}} cluster's SSL certificate. This allows {{fleet-server}} to validate the remote cluster's certificate.

::::{applies-switch}

:::{applies-item} { stack: ga 9.1+, serverless: ga }

Expand the Authentication section, and in the Server SSL certificate authorities field, enter the path to the CA certificate or paste the certificate content directly.

:::

:::{applies-item} stack: ga =9.0

Add the SSL certificate authorities in the Advanced YAML configuration section. For example:

ssl:
  certificate_authorities: ["/path/to/ca.pem"]

Alternatively, you can embed the CA certificate directly in the YAML configuration:

ssl:
  certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
    ...
    -----END CERTIFICATE-----

:::

::::

:::::

:::::{step} Configure mutual TLS (optional)

If your remote {{es}} cluster requires mutual TLS (mTLS) authentication, configure the client certificate and key.

::::{applies-switch}

:::{applies-item} { stack: ga 9.1+, serverless: ga }

Expand the Authentication section to configure mTLS settings:

  • Client SSL certificate: Enter the path to the client certificate that the {{agents}} will use to authenticate with the remote cluster, or paste the certificate content directly.
  • Client SSL certificate key: Enter the path to the private key associated with the client certificate, or paste the private key content directly.

:::

:::{applies-item} stack: ga =9.0

Add the client certificate settings in the Advanced YAML configuration section. For example:

ssl:
  certificate: "/path/to/client-cert.pem"
  key: "/path/to/client-cert.key"

Alternatively, you can embed the certificate and key directly in the YAML configuration:

ssl:
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ...
    -----END CERTIFICATE-----
  key: |
    -----BEGIN PRIVATE KEY-----
    MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDXHufGPycpCOfI
    ...
    -----END PRIVATE KEY-----

:::

::::

For more information about TLS configuration options, refer to One-way and mutual TLS certifications flow → Output SSL options. :::::

:::::{step} Configure output preferences

  1. {applies_to}stack: ga 9.1 Choose whether integrations should be automatically synchronized on the remote {{es}} cluster. To configure this feature, refer to Automatic integrations synchronization.

    :::{note} Automatic integrations synchronization is available only for certain subscription levels. For more information, check Fleet Multi-Cluster support on the Elastic subscriptions page. This feature is not available when the remote {{es}} cluster is a {{serverless-short}} project. Refer to Limitations. :::

  2. Choose whether the remote output should be the default for agent integrations or for agent monitoring data. When set as the default, {{agents}} use this output to send data if no other output is set in the agent policy.

  3. Select the performance tuning settings to optimize {{agent}}s for throughput, scale, or latency, or leave the default balanced setting.

  4. {applies_to}stack: preview 9.2 Choose whether {{agents}} using this output should send data to wired streams. Using this feature requires additional steps. For more details, refer to Ship data to streams → {{fleet}}. :::::

:::::{step} Configure advanced settings (optional)

Add any advanced YAML configuration settings that you'd like for the remote output. :::::

:::::{step} Save the output configuration

Click Save and apply settings. :::::

::::::

Use the remote {{es}} output [remote-output-usage]

After creating the output, update an {{agent}} policy to use it and send data to the remote {{es}} cluster:

  1. In the management cluster, go to {{fleet}}, then open the Agent policies tab.
  2. Click the agent policy you want to update, then click Settings.
  3. To send integrations data, set the Output for integrations option to use the output that you configured in the previous steps.
  4. To send {{agent}} monitoring data, set the Output for agent monitoring option to use the output that you configured in the previous steps.
  5. Click Save changes.

The remote {{es}} output is now configured for the remote cluster.

If you choose not to synchronize integrations automatically, you need to ensure that for any integrations added to your {{agent}} policy, you also install the integration assets on the remote {{es}} cluster. For detailed steps on this process, refer to Install and uninstall {{agent}} integration assets.

Remote output health status [remote-output-health-status]

{{fleet-server}} requires connectivity to the remote cluster to generate API keys for {{agents}}. When you use a remote {{es}} output, {{fleet-server}} tests whether it can reach the remote cluster. The result determines whether the remote output is reported as healthy or unhealthy in the Status column for the output on the {{fleet}} → Settings page.

If you have multiple {{fleet-server}} instances, each {{fleet-server}} tests connectivity to all remote {{es}} outputs. If the {{fleet-server}} instances have different network scopes, some of them might not be able to reach a certain remote output. In this case, the remote output will display as unhealthy in the UI with an Unable to connect error. You can ignore this status as long as the {{fleet-server}} instance that manages the {{agents}} using that remote output has connectivity to the remote cluster.

Set up {{ccs}} to query remote data [set-up-ccs]

In some cases, {{ccs}} (CCS) is required to query data on the remote cluster from the management cluster, such as Osquery queries on {{agents}} that use a remote {{es}} output.

:::{note} You don't need to set up {{ccs-init}} for {{agents}} to send data to a remote {{es}} output or for automatic integrations synchronization. Add {{ccs}} only when the management cluster must search data stored on the remote cluster. :::

To configure {{ccs-init}} in the management cluster:

  1. Open the {{kib}} menu, and go to {{manage-app}} → {{stack-manage-app}} → Remote Clusters.

  2. Select Add a remote cluster, then follow the steps to add your remote cluster.

    When prompted to add the remote address, enter your remote cluster's proxy address:

    1. In your remote cluster, go to Deployment → Manage this deployment → Security (or go to deployments/<deployment_id>/security).
    2. Scroll to the Remote cluster parameters section, then copy the Proxy Address.
    3. In your management cluster, enter the copied value in the Remote address field of the remote cluster setup.

    Refer to Remote clusters for more details on adding a remote cluster.