Skip to content

Commit bb01ce1

Browse files
author
Ignacio Van Droogenbroeck
committed
docs: add Redpanda Connect integration guides for Arc, Arc Enterprise, and Arc Cloud
- New full tutorial at docs/integrations/redpanda-connect.md covering the Arc output plugin (MessagePack columnar/row formats, zstd/gzip compression, Bloblang transformations, dynamic measurement routing, performance tuning, troubleshooting) with Kafka, webhook, MQTT, and multi-destination examples - Same guide added to docs-arc-enterprise/integrations/redpanda-connect.md - Condensed Redpanda Connect section added to the Arc Cloud integrations guide with an instance-URL example and link to the full Arc guide - Normalize Arc Enterprise telegraf.md to use \$ARC_TOKEN consistently (was a mix of YOUR_TOKEN, YOUR_ARC_TOKEN, PROD_TOKEN, STAGING_TOKEN)
1 parent c0e947d commit bb01ce1

4 files changed

Lines changed: 1062 additions & 10 deletions

File tree

docs-arc-cloud/guides/integrations.md

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,76 @@ Telegraf has 300+ input plugins. Some commonly used with Arc Cloud:
8888

8989
See the [full Telegraf integration guide](/integrations/telegraf) for query examples, Docker monitoring, and troubleshooting.
9090

91+
## Redpanda Connect
92+
93+
[Redpanda Connect](https://github.com/redpanda-data/connect) (formerly Benthos) is a stream processor with 200+ source connectors. Arc Cloud accepts data directly from the native Arc output plugin, so you can pipe Kafka events, HTTP webhooks, CDC streams, or MQTT data into Arc Cloud with a single YAML config.
94+
95+
### Arc Output Plugin
96+
97+
The native Arc output uses MessagePack columnar format with zstd compression for best throughput. Requires **Redpanda Connect 4.88+**.
98+
99+
```yaml
100+
input:
101+
kafka:
102+
addresses: ["kafka:9092"]
103+
topics: ["app-events"]
104+
consumer_group: "arc-cloud-ingest"
105+
106+
output:
107+
arc:
108+
base_url: https://<instance-id>.arc.<region>.basekick.net
109+
token: "${ARC_TOKEN}"
110+
database: events
111+
measurement: user_actions
112+
format: columnar
113+
compression: zstd
114+
batching:
115+
count: 5000
116+
period: 1s
117+
```
118+
119+
Install Redpanda Connect:
120+
121+
```bash
122+
brew install redpanda-data/tap/redpanda-connect
123+
export ARC_TOKEN="<your-instance-token>"
124+
redpanda-connect run arc-pipeline.yaml
125+
```
126+
127+
### Dynamic Measurement Routing
128+
129+
Route messages to different tables based on their content:
130+
131+
```yaml
132+
output:
133+
arc:
134+
base_url: https://<instance-id>.arc.<region>.basekick.net
135+
token: "${ARC_TOKEN}"
136+
database: telemetry
137+
# Messages with {"asset_type": "truck", ...} go to the "truck" table
138+
measurement: ${!json("asset_type")}
139+
format: columnar
140+
compression: zstd
141+
```
142+
143+
### Transformations with Bloblang
144+
145+
Filter, reshape, or enrich data before it reaches Arc Cloud:
146+
147+
```yaml
148+
pipeline:
149+
processors:
150+
- mapping: |
151+
# Drop bot traffic
152+
root = if this.user_id.has_prefix("bot-") { deleted() }
153+
# Reshape fields
154+
root.user_id = this.user_id
155+
root.page = this.page
156+
root.duration_ms = this.duration_ms
157+
```
158+
159+
See the [full Redpanda Connect integration guide](/integrations/redpanda-connect) for more examples, configuration reference, and troubleshooting.
160+
91161
## Grafana
92162
93163
Connect Arc Cloud to [Grafana](https://grafana.com/) for real-time dashboards, alerting, and time-series visualization using the Arc datasource plugin.

0 commit comments

Comments
 (0)