Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion local-antora-playbook.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ content:
- url: https://github.com/redpanda-data/docs
branches: [v/*, api, shared, site-search,'!v-end-of-life/*']
- url: https://github.com/redpanda-data/cloud-docs
branches: 'main'
branches: 'DOC-1236-remove-wasm-quickstart-from-cloud'
- url: https://github.com/redpanda-data/redpanda-labs
branches: main
start_paths: [docs,'*/docs']
Expand Down
32 changes: 32 additions & 0 deletions modules/develop/pages/data-transforms/build.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,38 @@ endif::[]
* For Rust projects, you must have the latest stable version of https://rustup.rs/[Rust^].
* For JavaScript and TypeScript projects, you must have the https://nodejs.org/en/download/package-manager[latest long-term-support release of Node.js^].

ifdef::env-cloud[]
== Enable data transforms

Data transforms is disabled on all clusters by default. Before you can deploy data transforms to a cluster, you must first enable the feature.

Use the Cloud API to set xref:ROOT:reference:properties/cluster-properties.adoc#data_transforms_enabled[`data_transforms_enabled`] to `true`.

[,bash]
----
# Store your cluster ID in a variable
export RP_CLUSTER_ID=<cluster-id>

# Retrieve a Redpanda Cloud access token
export RP_CLOUD_TOKEN=`curl -X POST "https://auth.prd.cloud.redpanda.com/oauth/token" \
-H "content-type: application/x-www-form-urlencoded" \
-d "grant_type=client_credentials" \
-d "client_id=<client-id>" \
-d "client_secret=<client-secret>"`

# Update cluster configuration to enable data transforms
curl -H "Authorization: Bearer ${RP_CLOUD_TOKEN}" -X PATCH \
"https://api.cloud.redpanda.com/v1/clusters/${RP_CLUSTER_ID}" \
-H 'accept: application/json'\
-H 'content-type: application/json' \
-d '{"cluster_configuration":{"custom_properties": {"data_transforms_enabled":true}}}'
----

The xref:api:ROOT:cloud-controlplane-api.adoc#patch-/v1/clusters/-cluster.id-[`PATCH /clusters/{cluster.id}`] request returns the ID of a long-running operation. The operation may take up to ten minutes to complete. You can check the status of the operation by polling the xref:api:ROOT:cloud-controlplane-api.adoc#get-/v1/operations/-id-[`GET /operations/\{id}`] endpoint.

NOTE: You must restart your cluster if you change this configuration for a running cluster.
endif::[]

[[init]]
== Initialize a data transforms project

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ This section outlines the limitations of data transforms. These constraints are
=== General

- *No external access*: Transform functions have no external access to disk or network resources.
- *Single record transforms*: Only single record transforms are supported, but multiple output records from a single input record are supported. For aggregations, joins, or complex transformations, consider xref:redpanda-connect:ROOT:about.adoc[Redpanda Connect] or https://flink.apache.org/[Apache Flink^].
- *Single message transforms*: Only single record transforms are supported, but multiple output records from a single input record are supported. For aggregations, joins, or complex transformations, consider using xref:redpanda-connect:ROOT:about.adoc[Redpanda Connect] or https://flink.apache.org/[Apache Flink^].
- *Output topic limit*: Up to eight output topics are supported.
- *Delivery semantics*: Transform functions have at-least-once delivery.
- *Transactions API*: When clients use the Kafka Transactions API on partitions of an input topic, transform functions process only committed records.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,4 @@
:description: Choose your deployment environment to get started with building and deploying your first transform function in Redpanda.
:page-aliases: reference:rpk/rpk-wasm/rpk-wasm.adoc, reference:rpk/rpk-wasm.adoc, reference:rpk/rpk-wasm/rpk-wasm-deploy.adoc, reference:rpk/rpk-wasm/rpk-wasm-generate.adoc, reference:rpk/rpk-wasm/rpk-wasm-remove.adoc, data-management:data-transform.adoc, labs:data-transform/index.adoc
:page-layout: index
:page-categories: Development, Stream Processing, Data Transforms
// tag::single-source[]

// end::single-source[]
:page-categories: Development, Stream Processing, Data Transforms
3 changes: 0 additions & 3 deletions modules/develop/pages/data-transforms/run-transforms.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,5 @@
:description: Learn how to build and deploy your first transform function in Linux deployments.
:page-context-links: [{"name": "Linux", "to": "develop:data-transforms/run-transforms.adoc" },{"name": "Kubernetes", "to": "develop:data-transforms/k-run-transforms.adoc" } ]
:page-categories: Development, Stream Processing, Data Transforms
// tag::single-source[]

include::develop:partial$run-transforms.adoc[]

// end::single-source[]
4 changes: 0 additions & 4 deletions modules/develop/partials/run-transforms.adoc
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
// tag::single-source[]

Data transforms let you run common data streaming tasks, like filtering, scrubbing, and transcoding, within Redpanda. For example, you may have consumers that require you to redact credit card numbers or convert JSON to Avro. Data transforms can also interact with the Redpanda Schema Registry to work with encoded data types.

Data transforms use a WebAssembly (Wasm) engine inside a Redpanda broker. A Wasm function acts on a single record in an input topic. You can develop and manage data transforms with xref:reference:rpk/rpk-transform/rpk-transform.adoc[`rpk transform`] commands.
Expand Down Expand Up @@ -700,5 +698,3 @@ rpk transform delete data-transforms-tutorial --no-confirm
- xref:reference:data-transforms/golang-sdk.adoc[]
- xref:reference:data-transforms/rust-sdk.adoc[]
- xref:reference:rpk/rpk-transform/rpk-transform.adoc[`rpk transform` commands]

// end::single-source[]
Loading