Skip to content

Commit 59f062b

Browse files
dervoetidervoeti
committed
docs: fault-tolerant execution documentation
1 parent b31ec78 commit 59f062b

4 files changed

Lines changed: 398 additions & 0 deletions

File tree

‎docs/modules/trino/pages/usage-guide/configuration.adoc‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,9 @@ For a role or role group, at the same level of `config`, you can specify `config
1818

1919
For a list of possible configuration properties consult the https://trino.io/docs/current/admin/properties.html[Trino Properties Reference].
2020

21+
TIP: For fault-tolerant execution configuration, use the dedicated `faultTolerantExecution` section in the cluster configuration instead of `configOverrides`.
22+
See xref:usage-guide/fault-tolerant-execution.adoc[] for detailed instructions.
23+
2124
[source,yaml]
2225
----
2326
workers:

‎docs/modules/trino/pages/usage-guide/fault-tolerant-execution.adoc‎

Lines changed: 285 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,285 @@
1+
= Fault-tolerant execution
2+
:description: Configure fault-tolerant execution in Trino clusters for improved query resilience and automatic retry capabilities.
3+
:keywords: fault-tolerant execution, retry policy, exchange manager, spooling, query resilience
4+
5+
Fault-tolerant execution is a mechanism in Trino that enables a cluster to mitigate query failures by retrying queries or their component tasks in the event of failure.
6+
With fault-tolerant execution enabled, intermediate exchange data is spooled and can be re-used by another worker in the event of a worker outage or other fault during query execution.
7+
8+
By default, if a Trino node lacks the resources to execute a task or otherwise fails during query execution, the query fails and must be run again manually.
9+
The longer the runtime of a query, the more likely it is to be susceptible to such failures.
10+
11+
NOTE: Fault tolerance does not apply to broken queries or other user error.
12+
For example, Trino does not spend resources retrying a query that fails because its SQL cannot be parsed.
13+
14+
Take a look at the link:https://trino.io/docs/current/admin/fault-tolerant-execution.html[Trino documentation for fault-tolerant execution {external-link-icon}^] to learn more.
15+
16+
== Configuration
17+
18+
Fault-tolerant execution is turned off by default.
19+
To enable the feature, you need to configure it in your `TrinoCluster` resource by adding a `faultTolerantExecution` section to the cluster configuration:
20+
21+
[source,yaml]
22+
----
23+
spec:
24+
clusterConfig:
25+
faultTolerantExecution:
26+
retryPolicy: QUERY # <1>
27+
queryRetryAttempts: 3 # <2>
28+
----
29+
<1> The retry policy - either `QUERY` or `TASK`
30+
<2> Maximum number of times to retry a query (QUERY policy only)
31+
32+
== Retry policies
33+
34+
The `retryPolicy` configuration property designates whether Trino retries entire queries or a query's individual tasks in the event of failure.
35+
36+
=== QUERY retry policy
37+
38+
A `QUERY` retry policy instructs Trino to automatically retry a query in the event of an error occurring on a worker node.
39+
A `QUERY` retry policy is recommended when the majority of the Trino cluster's workload consists of many small queries.
40+
41+
By default, Trino does not implement fault tolerance for queries whose result set exceeds 32MB in size.
42+
This limit can be increased by modifying the `exchangeDeduplicationBufferSize` configuration property to be greater than the default value of `32MB`, but this results in higher memory usage on the coordinator.
43+
44+
[source,yaml]
45+
----
46+
...
47+
spec:
48+
clusterConfig:
49+
faultTolerantExecution:
50+
retryPolicy: QUERY
51+
queryRetryAttempts: 3
52+
exchangeDeduplicationBufferSize: 64MB # Increased from default 32MB
53+
...
54+
----
55+
56+
=== TASK retry policy
57+
58+
A `TASK` retry policy instructs Trino to retry individual query tasks in the event of failure.
59+
You **must** configure an exchange manager to use the task retry policy.
60+
This policy is recommended when executing large batch queries, as the cluster can more efficiently retry smaller tasks within the query rather than retry the whole query.
61+
62+
IMPORTANT: A `TASK` retry policy is best suited for long-running queries, but this policy can result in higher latency for short-running queries executed in high volume.
63+
As a best practice, it is recommended to run a dedicated cluster with a `TASK` retry policy for large batch queries, separate from another cluster that handles short queries.
64+
65+
[source,yaml]
66+
----
67+
spec:
68+
clusterConfig:
69+
faultTolerantExecution:
70+
retryPolicy: TASK
71+
taskRetryAttemptsPerTask: 4
72+
exchangeManager:
73+
s3:
74+
baseDirectories:
75+
- "s3://trino-exchange-bucket/spooling"
76+
connection:
77+
reference: my-s3-connection # <1>
78+
----
79+
<1> Reference to an xref:concepts:s3.adoc[S3Connection] resource
80+
81+
== Exchange manager
82+
83+
Exchange spooling is responsible for storing and managing spooled data for fault-tolerant execution.
84+
You can configure a filesystem-based exchange manager that stores spooled data in a specified location, such as AWS S3 and S3-compatible systems, Azure Blob Storage or HDFS.
85+
86+
NOTE: An exchange manager is required when using the `TASK` retry policy and optional for the `QUERY` retry policy.
87+
88+
=== S3-compatible storage
89+
90+
You can use S3-compatible storage systems for exchange spooling, including AWS S3, MinIO, and Google Cloud Storage.
91+
92+
[source,yaml]
93+
----
94+
spec:
95+
clusterConfig:
96+
faultTolerantExecution:
97+
retryPolicy: TASK
98+
exchangeManager:
99+
s3:
100+
baseDirectories: # <1>
101+
- "s3://exchange-bucket-1/trino-spooling"
102+
connection:
103+
reference: minio-s3-connection # <2>
104+
---
105+
apiVersion: s3.stackable.tech/v1alpha1
106+
kind: S3Connection
107+
metadata:
108+
name: minio-s3-connection
109+
spec:
110+
host: minio.default.svc.cluster.local
111+
port: 9000
112+
accessStyle: Path
113+
credentials:
114+
secretClass: minio-secret-class
115+
tls:
116+
verification:
117+
server:
118+
caCert:
119+
secretClass: tls
120+
----
121+
<1> Multiple S3 buckets can be specified to distribute I/O load
122+
<2> S3 connection defined as a reference to an xref:concepts:s3.adoc[S3Connection] resource
123+
124+
For Google Cloud Storage, you can use GCS buckets with S3 compatibility:
125+
126+
[source,yaml]
127+
----
128+
spec:
129+
clusterConfig:
130+
faultTolerantExecution:
131+
exchangeManager:
132+
s3:
133+
baseDirectories:
134+
- "gs://my-gcs-bucket/trino-spooling"
135+
connection:
136+
inline:
137+
host: storage.googleapis.com
138+
port: 443
139+
accessStyle: Path
140+
credentials:
141+
secretClass: gcs-hmac-credentials
142+
tls:
143+
verification:
144+
server:
145+
caCert:
146+
webPki: {}
147+
gcsServiceAccountKey:
148+
secretClass: "gcs-service-account-secret-class"
149+
key: "service-account.json"
150+
----
151+
152+
=== Azure Blob Storage
153+
154+
You can configure Azure Blob Storage as the exchange spooling destination:
155+
156+
[source,yaml]
157+
----
158+
spec:
159+
clusterConfig:
160+
faultTolerantExecution:
161+
retryPolicy: TASK
162+
exchangeManager:
163+
azure:
164+
baseDirectories:
165+
- "abfs://exchange-container@mystorageaccount.dfs.core.windows.net/exchange-spooling"
166+
secretClass: azure-credentials # <1>
167+
----
168+
169+
The Azure connection string should be provided via a SecretClass that refers to a Kubernetes Secret containing the Azure storage account connection string, like this:
170+
171+
[source,yaml]
172+
----
173+
apiVersion: secrets.stackable.tech/v1alpha1
174+
kind: SecretClass
175+
metadata:
176+
name: azure-credentials
177+
spec:
178+
backend:
179+
k8sSearch:
180+
searchNamespace:
181+
pod: {}
182+
----
183+
184+
[source,yaml]
185+
----
186+
apiVersion: v1
187+
kind: Secret
188+
metadata:
189+
name: azure-secret
190+
labels:
191+
secrets.stackable.tech/class: azure-credentials
192+
type: Opaque
193+
stringData:
194+
connectionString: "DefaultEndpointsProtocol=https;AccountName=mystorageaccount;AccountKey=your_account_key;EndpointSuffix=core.windows.net"
195+
----
196+
197+
=== HDFS storage
198+
199+
You can configure HDFS as the exchange spooling destination:
200+
201+
[source,yaml]
202+
----
203+
spec:
204+
clusterConfig:
205+
faultTolerantExecution:
206+
retryPolicy: TASK
207+
exchangeManager:
208+
hdfs:
209+
baseDirectories:
210+
- "hdfs://simple-hdfs/exchange-spooling"
211+
hdfs:
212+
configMap: simple-hdfs # <1>
213+
----
214+
<1> ConfigMap containing HDFS configuration files (created by the HDFS operator)
215+
216+
=== Local filesystem storage
217+
218+
Local filesystem storage is supported but only recommended for development or single-node deployments:
219+
220+
WARNING: It is only recommended to use a local filesystem for exchange in standalone, non-production clusters.
221+
A local directory can only be used for exchange in a distributed cluster if the exchange directory is shared and accessible from all nodes.
222+
223+
[source,yaml]
224+
----
225+
spec:
226+
clusterConfig:
227+
faultTolerantExecution:
228+
retryPolicy: TASK
229+
exchangeManager:
230+
local:
231+
baseDirectories:
232+
- "/trino-exchange"
233+
coordinators:
234+
roleGroups:
235+
default:
236+
replicas: 1
237+
podOverrides:
238+
spec:
239+
volumes:
240+
- name: trino-exchange
241+
persistentVolumeClaim:
242+
claimName: trino-exchange-pvc
243+
containers:
244+
- name: trino
245+
volumeMounts:
246+
- name: trino-exchange
247+
mountPath: /trino-exchange
248+
workers:
249+
roleGroups:
250+
default:
251+
replicas: 1
252+
podOverrides:
253+
spec:
254+
volumes:
255+
- name: trino-exchange
256+
persistentVolumeClaim:
257+
claimName: trino-exchange-pvc
258+
containers:
259+
- name: trino
260+
volumeMounts:
261+
- name: trino-exchange
262+
mountPath: /trino-exchange
263+
---
264+
kind: PersistentVolumeClaim
265+
apiVersion: v1
266+
metadata:
267+
name: trino-exchange-pvc
268+
spec:
269+
accessModes:
270+
- ReadWriteOnce
271+
resources:
272+
requests:
273+
storage: 10Gi
274+
----
275+
276+
== Connector support
277+
278+
Support for fault-tolerant execution of SQL statements varies on a per-connector basis.
279+
Take a look at the link:https://trino.io/docs/current/admin/fault-tolerant-execution.html#configuration[Trino documentation {external-link-icon}^] to see which connectors support fault-tolerant execution.
280+
281+
When using connectors that do not explicitly support fault-tolerant execution, you may encounter a "This connector does not support query retries" error message.
282+
283+
== Examples
284+
285+
* link:https://github.com/stackabletech/trino-operator/blob/main/examples/simple-trino-cluster-fault-tolerant-execution.yaml[TrinoCluster with TASK retry policy and S3 exchange manager {external-link-icon}^]

‎docs/modules/trino/partials/nav.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
** xref:trino:usage-guide/connect_to_trino.adoc[]
77
** xref:trino:usage-guide/listenerclass.adoc[]
88
** xref:trino:usage-guide/configuration.adoc[]
9+
** xref:trino:usage-guide/fault-tolerant-execution.adoc[]
910
** xref:trino:usage-guide/s3.adoc[]
1011
** xref:trino:usage-guide/security.adoc[]
1112
** xref:trino:usage-guide/monitoring.adoc[]

0 commit comments

Comments
 (0)