|
| 1 | +--- |
| 2 | +title: Run the Processing Engine in a cluster |
| 3 | +description: > |
| 4 | + Configure, start, and troubleshoot Processing Engine plugins in a multi-node |
| 5 | + InfluxDB 3 Enterprise cluster — including how triggers fan out across nodes, |
| 6 | + how to pin triggers, and how to recognize common misconfiguration errors. |
| 7 | +menu: |
| 8 | + influxdb3_enterprise: |
| 9 | + name: Run the Processing Engine in a cluster |
| 10 | + parent: Administer InfluxDB |
| 11 | +weight: 102 |
| 12 | +related: |
| 13 | + - /influxdb3/enterprise/admin/clustering/ |
| 14 | + - /influxdb3/enterprise/plugins/ |
| 15 | + - /influxdb3/enterprise/reference/processing-engine/ |
| 16 | + - /influxdb3/enterprise/reference/cli/influxdb3/create/trigger/ |
| 17 | + - /influxdb3/enterprise/reference/cli/influxdb3/serve/ |
| 18 | +influxdb3/enterprise/tags: [processing engine, plugins, clustering, triggers, troubleshooting] |
| 19 | +--- |
| 20 | + |
| 21 | +This guide covers how the Processing Engine behaves in a multi-node {{% product-name %}} cluster and how to troubleshoot common misconfigurations. |
| 22 | + |
| 23 | +For single-node deployments, defaults work as documented in [Set up the Processing Engine](/influxdb3/enterprise/plugins/#set-up-the-processing-engine). |
| 24 | +The cluster-specific behavior described here applies when you run more than one `influxdb3 serve` process against a shared catalog and object store. |
| 25 | + |
| 26 | +- [How trigger execution works in a cluster](#how-trigger-execution-works-in-a-cluster) |
| 27 | +- [Start the cluster](#start-the-cluster) |
| 28 | +- [Worked example: 5-node reference architecture](#worked-example-5-node-reference-architecture) |
| 29 | +- [Troubleshoot misconfigurations](#troubleshoot-misconfigurations) |
| 30 | + |
| 31 | +## How trigger execution works in a cluster |
| 32 | + |
| 33 | +Three independent factors determine whether a Processing Engine trigger runs on a given node: |
| 34 | + |
| 35 | +1. The node has [`--plugin-dir`](/influxdb3/enterprise/reference/config-options/#plugin-dir) configured. |
| 36 | +2. The trigger's [`--node-spec`](/influxdb3/enterprise/reference/cli/influxdb3/create/trigger/#options) includes the node — by default, `all` (every node with `--plugin-dir`). |
| 37 | +3. The trigger's plugin imports modules that are available in that node's per-node Python virtual environment. |
| 38 | + |
| 39 | +`--mode` controls which APIs the node serves (writes, queries, compaction). |
| 40 | +**`--mode` does not gate trigger execution.** |
| 41 | +A trigger pinned to a `compact`-only node still fires on that node — it just fails per tick if the plugin needs APIs the node doesn't serve. |
| 42 | + |
| 43 | +| Trigger type | Pin to | Why | |
| 44 | +|----------------|--------------------------------------------------------------------|--------------------------------------------------------------------------------------------------| |
| 45 | +| WAL (`table:`) | An ingest-capable node | Each ingester owns its own WAL; pinning to one ingester sees only that node's writes. | |
| 46 | +| Schedule (`every:` or `cron:`) | A node with `process,query` mode | The plugin reads via `influxdb3_local.query()` locally; results write back to an ingester via HTTP. | |
| 47 | +| Request (`request:`) | A node with `query` mode (the host-exposed port) | The HTTP route exists only on pinned nodes; unpinned nodes return `404 not found`. | |
| 48 | + |
| 49 | +## Start the cluster |
| 50 | + |
| 51 | +Each cluster node runs `influxdb3 serve` with a unique `--node-id`, the same `--cluster-id`, and a shared object store and catalog. |
| 52 | +Configure `--plugin-dir` on **every** node, even nodes that don't execute plugins — see [Configure `--plugin-dir` on every cluster node](/influxdb3/enterprise/admin/clustering/#configure-process-capable-nodes). |
| 53 | + |
| 54 | +```bash { placeholders="CLUSTER_ID|DATA_DIR|PLUGINS_DIR|NODE_ID" } |
| 55 | +# Ingest node |
| 56 | +influxdb3 serve \ |
| 57 | + --cluster-id CLUSTER_ID \ |
| 58 | + --node-id NODE_ID \ |
| 59 | + --mode ingest \ |
| 60 | + --object-store file \ |
| 61 | + --data-dir DATA_DIR \ |
| 62 | + --plugin-dir PLUGINS_DIR |
| 63 | + |
| 64 | +# Query node (host-exposed) |
| 65 | +influxdb3 serve \ |
| 66 | + --cluster-id CLUSTER_ID \ |
| 67 | + --node-id NODE_ID \ |
| 68 | + --mode query \ |
| 69 | + --object-store file \ |
| 70 | + --data-dir DATA_DIR \ |
| 71 | + --plugin-dir PLUGINS_DIR |
| 72 | + |
| 73 | +# Compact node (one per cluster) |
| 74 | +influxdb3 serve \ |
| 75 | + --cluster-id CLUSTER_ID \ |
| 76 | + --node-id NODE_ID \ |
| 77 | + --mode compact \ |
| 78 | + --object-store file \ |
| 79 | + --data-dir DATA_DIR \ |
| 80 | + --plugin-dir PLUGINS_DIR |
| 81 | + |
| 82 | +# Process,query node (hosts schedule plugins) |
| 83 | +influxdb3 serve \ |
| 84 | + --cluster-id CLUSTER_ID \ |
| 85 | + --node-id NODE_ID \ |
| 86 | + --mode process,query \ |
| 87 | + --object-store file \ |
| 88 | + --data-dir DATA_DIR \ |
| 89 | + --plugin-dir PLUGINS_DIR |
| 90 | +``` |
| 91 | + |
| 92 | +After all nodes are up, register triggers from any node and pin them with `--node-spec`: |
| 93 | + |
| 94 | +```bash { placeholders="AUTH_TOKEN|DATABASE_NAME|NODE_ID" } |
| 95 | +# Schedule trigger pinned to the process,query node |
| 96 | +influxdb3 create trigger \ |
| 97 | + --database DATABASE_NAME \ |
| 98 | + --token AUTH_TOKEN \ |
| 99 | + --path schedule_rollup.py \ |
| 100 | + --trigger-spec "every:5s" \ |
| 101 | + --node-spec "nodes:NODE_ID" \ |
| 102 | + hourly_rollup |
| 103 | +``` |
| 104 | + |
| 105 | +## Worked example: 5-node reference architecture |
| 106 | + |
| 107 | +The [`influxdata/influxdb3-ref-network-telemetry`](https://github.com/influxdata/influxdb3-ref-network-telemetry) repo provides a complete 5-node {{% product-name %}} cluster you can run locally with `docker compose`: |
| 108 | + |
| 109 | +- 2 ingest nodes (`--mode=ingest`) |
| 110 | +- 1 query node (`--mode=query`, host-exposed on port 8181) |
| 111 | +- 1 compact node (`--mode=compact`) |
| 112 | +- 1 process,query node (`--mode=process,query`, hosts schedule plugins) |
| 113 | + |
| 114 | +The repo demonstrates: |
| 115 | + |
| 116 | +- Pinning schedule triggers to the process node and request triggers to the query node with `--node-spec nodes:<id>`. |
| 117 | +- Cross-node write-back from schedule plugins via HTTP — see [`plugins/_writeback.py`](https://github.com/influxdata/influxdb3-ref-network-telemetry/blob/main/plugins/_writeback.py). |
| 118 | +- Mounting the same plugin directory on every node (including ingest and compact) for catalog validation at startup. |
| 119 | + |
| 120 | +Use this repo as a template when designing your own cluster. |
| 121 | + |
| 122 | +## Troubleshoot misconfigurations |
| 123 | + |
| 124 | +### `invalid node name (<id>)` when creating a trigger |
| 125 | + |
| 126 | +The cluster validates `--node-spec nodes:<id>` against current cluster membership at create time. |
| 127 | +A typo or unknown node ID returns `HTTP 500: invalid node name (<id>)`. |
| 128 | + |
| 129 | +To fix: |
| 130 | + |
| 131 | +1. List current cluster members and their node IDs: |
| 132 | + |
| 133 | + ```bash { placeholders="AUTH_TOKEN" } |
| 134 | + influxdb3 show nodes --token AUTH_TOKEN |
| 135 | + ``` |
| 136 | + |
| 137 | + The `mode` column shows the node's runtime modes — `process` is included automatically on any node that has `--plugin-dir` configured. |
| 138 | + |
| 139 | +2. Reissue `influxdb3 create trigger` with the correct `--node-spec`. |
| 140 | + |
| 141 | +### `HTTP 404 {error: "not found"}` when calling a request trigger |
| 142 | + |
| 143 | +The `/api/v3/engine/<trigger_name>` route exists only on the node(s) the trigger is pinned to. |
| 144 | +There is no internal cross-node routing for request triggers. |
| 145 | + |
| 146 | +To fix: |
| 147 | + |
| 148 | +- Verify the node-spec on the trigger: |
| 149 | + |
| 150 | + ```bash { placeholders="AUTH_TOKEN|DATABASE_NAME" } |
| 151 | + influxdb3 query \ |
| 152 | + --database DATABASE_NAME \ |
| 153 | + --token AUTH_TOKEN \ |
| 154 | + "SELECT trigger_name, trigger_specification FROM system.processing_engine_triggers" |
| 155 | + ``` |
| 156 | + |
| 157 | +- Either pin the trigger to the node receiving the HTTP request (typically a `query`-mode node), or route the request to a node where the trigger is already pinned. |
| 158 | + |
| 159 | +### Schedule trigger logs `ModuleNotFoundError` per tick |
| 160 | + |
| 161 | +The trigger fired on its pinned node, but the plugin imports a module that's not in that node's per-node Python virtual environment. |
| 162 | + |
| 163 | +To fix: |
| 164 | + |
| 165 | +- Install the missing package on the pinned node: |
| 166 | + |
| 167 | + ```bash { placeholders="PACKAGE_NAME" } |
| 168 | + influxdb3 install package PACKAGE_NAME |
| 169 | + ``` |
| 170 | + |
| 171 | +- Or pin the trigger to a node that has the required module already installed. |
| 172 | + |
| 173 | +### Engine panics on startup with a missing plugin file |
| 174 | + |
| 175 | +The Enterprise catalog registers triggers cluster-wide. |
| 176 | +Every node validates the registered triggers at startup, including nodes that don't execute them. |
| 177 | +If a plugin file referenced by a registered trigger is missing on a node, the engine panics on startup. |
| 178 | + |
| 179 | +To fix: |
| 180 | + |
| 181 | +- Configure `--plugin-dir` on every cluster node and copy or mount the same plugin files to each one. |
| 182 | +- If a plugin was deleted but the trigger still references it, drop the orphaned trigger: |
| 183 | + |
| 184 | + ```bash { placeholders="AUTH_TOKEN|DATABASE_NAME|TRIGGER_NAME" } |
| 185 | + influxdb3 delete trigger \ |
| 186 | + --database DATABASE_NAME \ |
| 187 | + --token AUTH_TOKEN \ |
| 188 | + --force TRIGGER_NAME |
| 189 | + ``` |
| 190 | + |
| 191 | +### Plugin operations fail in administrative tools |
| 192 | + |
| 193 | +If an administrative tool reports a generic plugin error against your cluster, check whether any node satisfies the request: |
| 194 | + |
| 195 | +1. Confirm at least one node has `--plugin-dir` configured and runs the plugin's required mode (typically `process,query` for schedule plugins, `query` for request plugins). |
| 196 | +2. Confirm the trigger's `--node-spec` includes a running, healthy node. |
| 197 | +3. Inspect the `system.processing_engine_logs` table on the pinned node for execution errors: |
| 198 | + |
| 199 | + ```bash { placeholders="AUTH_TOKEN|DATABASE_NAME" } |
| 200 | + influxdb3 query \ |
| 201 | + --database DATABASE_NAME \ |
| 202 | + --token AUTH_TOKEN \ |
| 203 | + "SELECT event_time, trigger_name, log_level, log_text \ |
| 204 | + FROM system.processing_engine_logs \ |
| 205 | + ORDER BY event_time DESC LIMIT 20" |
| 206 | + ``` |
0 commit comments