You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -50,8 +50,21 @@ The operator introduces two Custom Resource Definitions (CRDs): `RestateCluster`
50
50
51
51
The `RestateCluster` CRD defines a Restate cluster. The operator watches for these objects and creates the necessary Kubernetes resources, such as `StatefulSet`, `Service`, and `NetworkPolicy` objects in a new namespace that matches the `RestateCluster` name.
52
52
53
+
**By default, the operator enforces network isolation on the cluster, allowing only the following**:
54
+
1. Peer to peer traffic between Restate pods
55
+
2. Traffic from the operator to Restate pods
56
+
3. Egress traffic to the public internet
57
+
4. Egress traffic to coredns
58
+
4. Egress traffic to pods in any namespace labelled with `allow.restate.dev/<cluster-name>`
59
+
60
+
**All other traffic is denied by default.**
61
+
62
+
The default behaviour can be disabled with `spec.security.disableNetworkPolicies: true`.
63
+
Alternatively, you can add new allowed inbound callers of the Restate ports with `spec.security.networkPeers.{ingress,admin,metrics}`, which are arrays of [`NetworkPolicyPeer`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#networkpolicypeer-v1-networking-k8s-io).
64
+
You can allow new outbound destinations by adding to the `spec.security.networkEgressRules` list, which is an array of [`NetworkPolicyEgressRule`](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.28/#networkpolicyegressrule-v1-networking-k8s-io).
65
+
53
66
**NOTE**: Each cluster is created in its own namespace (enforced by the operator). Do not create the namespace manually,
54
-
and do not use the same namespace as the operator is in.
67
+
and do not use the same namespace that the operator is in.
55
68
56
69
#### Minimal Example
57
70
@@ -69,6 +82,8 @@ spec:
69
82
storageRequestBytes: 2147483648# 2 GiB
70
83
```
71
84
85
+
For the full schema as a [Pkl](https://pkl-lang.org/) template see [`crd/RestateCluster.pkl`](./crd/RestateCluster.pkl).
86
+
72
87
More examples are available just below the spec that follows.
73
88
74
89
#### Spec Fields
@@ -269,146 +284,10 @@ spec:
269
284
# aws-allow-http = true
270
285
```
271
286
272
-
#### MinIO Configuration Example
273
-
274
-
To configure a `RestateCluster` with a self-hosted S3-compatible object store like [MinIO](https://min.io/), you can point the server to your MinIO instance. For security, it's best to create a dedicated service account with credentials scoped only to the buckets Restate needs.
275
-
276
-
##### 1. Create a Scoped Access Key & Buckets
277
-
278
-
First, we'll define a policy, create the buckets, and then create a service account with a new access key that is restricted by that policy.
279
-
280
-
The following commands use the `mc` client to set up your MinIO instance. They assume you have port-forwarded your MinIO service.
# Use https:// and --insecure if connecting to a port-forwarded TLS port (like 443)
288
-
mc alias set local-minio https://localhost:9000 YOUR_ADMIN_ACCESS_KEY YOUR_ADMIN_SECRET_KEY --insecure
289
-
290
-
# Create the buckets
291
-
mc mb --insecure local-minio/restate-metadata
292
-
mc mb --insecure local-minio/restate-snapshots
293
-
294
-
# Add the new policy to MinIO
295
-
296
-
cat <<EOF | mc admin policy add local-minio restate-s3-policy
297
-
{
298
-
"Version": "2012-10-17",
299
-
"Statement": [
300
-
{
301
-
"Effect": "Allow",
302
-
"Action": [
303
-
"s3:GetBucketLocation",
304
-
"s3:ListBucket"
305
-
],
306
-
"Resource": [
307
-
"arn:aws:s3:::restate-metadata",
308
-
"arn:aws:s3:::restate-snapshots"
309
-
]
310
-
},
311
-
{
312
-
"Effect": "Allow",
313
-
"Action": [
314
-
"s3:PutObject",
315
-
"s3:GetObject",
316
-
"s3:DeleteObject",
317
-
"s3:ListMultipartUploadParts",
318
-
"s3:AbortMultipartUpload"
319
-
],
320
-
"Resource": [
321
-
"arn:aws:s3:::restate-metadata/*",
322
-
"arn:aws:s3:::restate-snapshots/*"
323
-
]
324
-
}
325
-
]
326
-
}
327
-
EOF
328
-
329
-
# Create a new service account for the Restate application
330
-
# This command will output a new AccessKey and SecretKey.
331
-
mc admin service-account add local-minio restate
332
-
333
-
# Attach the policy to the new service account
334
-
mc admin policy set local-minio restate-s3-policy service-account=restate
335
-
```
336
-
337
-
When you run `mc admin service-account add`, it will output a new `AccessKey` and `SecretKey`. **Save these securely**, as you will use them in the next step.
338
-
339
-
##### 2. Create a Kubernetes Secret
340
-
341
-
Next, create a Kubernetes `Secret` containing the new, **scoped credentials** you just generated for the `restate` service account.
342
-
343
-
**Important**: This secret must be created in the namespace where the cluster will run, which is the same as the `metadata.name` of your `RestateCluster`. For this example, the namespace is `restate-with-minio`.
344
-
345
-
```yaml
346
-
apiVersion: v1
347
-
kind: Secret
348
-
metadata:
349
-
name: minio-credentials
350
-
namespace: restate-with-minio
351
-
type: Opaque
352
-
stringData:
353
-
# Use the keys generated from the 'mc admin service-account add' command
Finally, define your `RestateCluster` resource. This manifest is the same as before, but it will now use the Kubernetes secret containing the limited-permission keys.
> **A Note on AWS Credentials**: You might notice we are using standard AWS environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) directly, rather than the Restate-specific format like `RESTATE_WORKER__SNAPSHOTS_AWS_ACCESS_KEY_ID`. This is because Restate uses the underlying AWS SDK, which automatically and conventionally discovers credentials from these standard environment variables. However, if you are using Minio for snapshots and also need to use AWS Lambda services from your Restate cluster, then you may need different AWS credentials for different components.
409
-
410
-
For the full schema as a [Pkl](https://pkl-lang.org/) template see [`crd/RestateCluster.pkl`](./crd/RestateCluster.pkl).
287
+
Note that Restate needs `s3:ListBucket` on the bucket, and `s3:GetObject`/`s3:PutObject` on the bucket contents.
To configure a `RestateCluster` with a self-hosted S3-compatible object store like [MinIO](https://min.io/), you can point the server to your MinIO instance. For security, it's best to create a dedicated service account with credentials scoped only to the buckets Restate needs.
4
+
5
+
## 1. Create a Scoped Access Key & Buckets
6
+
7
+
First, we'll define a policy, create the buckets, and then create a service account with a new access key that is restricted by that policy.
8
+
9
+
The following commands use the `mc` client to set up your MinIO instance. They assume you have port-forwarded your MinIO service.
# Use https:// and --insecure if connecting to a port-forwarded TLS port (like 443)
17
+
mc aliasset local-minio https://localhost:9000 YOUR_ADMIN_ACCESS_KEY YOUR_ADMIN_SECRET_KEY --insecure
18
+
19
+
# Create the buckets
20
+
mc mb --insecure local-minio/restate-metadata
21
+
mc mb --insecure local-minio/restate-snapshots
22
+
23
+
# Add the new policy to MinIO
24
+
25
+
cat <<EOF | mc admin policy add local-minio restate-s3-policy
26
+
{
27
+
"Version": "2012-10-17",
28
+
"Statement": [
29
+
{
30
+
"Effect": "Allow",
31
+
"Action": [
32
+
"s3:ListBucket"
33
+
],
34
+
"Resource": [
35
+
"arn:aws:s3:::restate-metadata",
36
+
"arn:aws:s3:::restate-snapshots"
37
+
]
38
+
},
39
+
{
40
+
"Effect": "Allow",
41
+
"Action": [
42
+
"s3:PutObject",
43
+
"s3:GetObject",
44
+
],
45
+
"Resource": [
46
+
"arn:aws:s3:::restate-metadata/*",
47
+
"arn:aws:s3:::restate-snapshots/*"
48
+
]
49
+
}
50
+
]
51
+
}
52
+
EOF
53
+
54
+
# Create a new service account for the Restate application
55
+
# This command will output a new AccessKey and SecretKey.
56
+
mc admin service-account add local-minio restate
57
+
58
+
# Attach the policy to the new service account
59
+
mc admin policy set local-minio restate-s3-policy service-account=restate
60
+
```
61
+
62
+
When you run `mc admin service-account add`, it will output a new `AccessKey` and `SecretKey`. **Save these securely**, as you will use them in the next step.
63
+
64
+
### 2. Create a Kubernetes Secret
65
+
66
+
Next, create a Kubernetes `Secret` containing the new, **scoped credentials** you just generated for the `restate` service account.
67
+
68
+
**Important**: This secret must be created in the namespace where the cluster will run, which is the same as the `metadata.name` of your `RestateCluster`. For this example, the namespace is `restate-with-minio`.
69
+
70
+
```yaml
71
+
apiVersion: v1
72
+
kind: Secret
73
+
metadata:
74
+
name: minio-credentials
75
+
namespace: restate-with-minio
76
+
type: Opaque
77
+
stringData:
78
+
# Use the keys generated from the 'mc admin service-account add' command
79
+
access-key: YOUR_NEW_RESTATE_ACCESS_KEY
80
+
secret-key: YOUR_NEW_RESTATE_SECRET_KEY
81
+
```
82
+
83
+
##### 3. Configure the `RestateCluster`
84
+
85
+
Finally, define your `RestateCluster` resource. This manifest is the same as before, but it will now use the Kubernetes secret containing the limited-permission keys.
86
+
Additionally, we add the MinIO namespace to the allowed list of egress peers.
0 commit comments