You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/cloud/features/scheduler/scheduler.md
+45-2Lines changed: 45 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -163,8 +163,51 @@ This gives you complete control over data security and network access while stil
163
163
164
164
### How it works
165
165
166
-
Coming soon!
166
+
Self-hosted executors as the name indicates are self-hosted workers that take on the responsibility of "executing" changes to the data warehouse. While Tobiko Cloud schedules and plans changes, the executors are responsible for executing those changes. Executors are docker containers that are configured to connect to Tobiko Cloud as well as your data warehouse. Connected to both, the executor pulls work from the cloud, whether it's a plan or scheduled background work from a run, and execute it on your data warehouse.
167
167
168
168
### Configuration
169
169
170
-
Coming soon!
170
+
Exact configuration is left to the user and will vary based on the infrastructure and setup of the user, for example the executors could be run on a Kubernetes cluster or as a standalone pair of Docker containers depending on the user's infrastructure. The following details are an example of how this is done for a Postgres data warehouse and a pair of local containers running in docker.
171
+
172
+
Tobiko Cloud requires 2 docker instances to be running, one to pick up runs and one for plans. The entrypoint for both is `executor run` and `executor plan` respectively. The executor container can be found on [Docker Hub](https://hub.docker.com/r/tobikodata/tcloud). In addition to running the container, the user will need to configure the executor with environment variables that point to Tobiko Cloud as well as the data warehouse.
173
+
174
+
To connect to the Tobiko Cloud, the user will need to provide the following environment variables, replaced with the user's own values.
In addition to the above variables, a gateway is needed to provide a connection to a data warehouse. The following details are an example of how this is done for a Postgres data warehouse where the gateway is named `GATEWAY_A`. For more details on how to configure a gateway, see the details on other [engines](../../../integrations/overview.md#execution-engines) and how to [overide variables](../../../guides/configuration.md#overrides) as done below.
**Note**: If there are multiple gateways, each gateway will need to have its own set of environment variables. For example, if there are two gateways, `GATEWAY_A` and `GATEWAY_B`, the environment variables will need to be set for both gateways.
Once you have set up both sets of environment variables in a file named `local.env`, you can run the following command to start the executor:
203
+
204
+
```shell
205
+
docker run -d --env-file local.env tobikodata/tcloud:latest -- executor run
206
+
docker run -d --env-file local.env tobikodata/tcloud:latest -- executor plan
207
+
```
208
+
209
+
After the executors are properly configured, they will appear in the cloud UI where they can be used to execute plans and scheduled tasks.
210
+
211
+

212
+
213
+
We recommend setting up monitoring for the executors to ensure they run smoothly and to help troubleshoot issues. This monitoring should include logs and system metrics like memory and CPU usage.
0 commit comments