You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/_sources/source/compute_config/aws_ec2.md
+50-8Lines changed: 50 additions & 8 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,9 +6,9 @@ The AWS EC2 client of Lithops can provide a truely serverless user experience on
6
6
The assumption that you already familiar with AWS, and you have AUTH credentials to your account (HMAC Credentials).
7
7
8
8
### Choose an operating system image for the VM
9
-
Any Virtual Machine (VM) need to define the instance’s operating system and version. Lithops support both standard operating system choices provided by the VPC or using pre-defined custom images that already contains all dependencies required by Lithops.
9
+
Any Virtual Machine (VM) needs to define the instance’s operating system and version. Lithops supports both standard operating system choices provided by the VPC or using pre-defined custom images that already contains all dependencies required by Lithops.
10
10
11
-
- Option 1: By default, Lithops uses an Ubuntu 22.04 image. In this case, no further action is required and you can continue to the next step. Lithops will install all required dependencies in the VM by itself. Notice this can consume about 3 min to complete all installations.
11
+
- Option 1: By default, Lithops uses an Ubuntu 22.04 image. In this case, no further action is required and you can continue to the next step. Lithops will install all required dependencies in the VM by itself. Note this can consume about 3 min to complete all installations.
12
12
13
13
- Option 2: Alternatively, you can use a pre-built custom image that will greatly improve VM creation time for Lithops jobs. To benefit from this approach, navigate to [runtime/aws_ec2](https://github.com/lithops-cloud/lithops/tree/master/runtime/aws_ec2), and follow the instructions.
14
14
@@ -109,13 +109,14 @@ In summary, you can use one of the following settings:
109
109
|---|---|---|---|---|
110
110
|aws_ec2 | region | |no | Region name, for example: `eu-west-1`. Lithops will use the `region` set under the `aws` section if it is not set here |
111
111
|aws_ec2 | instance_role | | yes | EC2 Instance role name created in the configuration section above. Do not use the full ARN here; only the role name is required. For example: `ec2LithopsInstanceRole`|
112
-
|aws_ec2 | vpc_id | | no | VPC id. You can find all the available VPCs in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#vpcs:) |
113
-
|aws_ec2 | subnet_id | | no | Subnet id. You can find all the available Subnets in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#subnets:) |
114
-
|aws_ec2 | security_group_id | | no | Security group ID. You can find the available security groups in the [VPC console page](https://console.aws.amazon.com/vpc/v2/home#SecurityGroups:). The security group must have ports 22 and 8080 open |
112
+
|aws_ec2 | vpc_id | | no | VPC id. You can find all the available VPCs in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#vpcs:). If not provided, Lithops will create a new VPC |
113
+
|aws_ec2 | public_subnet_id | | no | Public subnet id. You can find all the available Subnets in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#subnets:). If not provided, Lithops will create a new public subnet |
114
+
|aws_ec2 | public_subnet_cidr_block | 10.0.1.0/24 | no | In case a `public_subnet_id` is not provided, Lithops will create a new subnet with this CIDR block |
115
+
|aws_ec2 | security_group_id | | no | Security group ID. You can find the available security groups in the [VPC console page](https://console.aws.amazon.com/vpc/v2/home#SecurityGroups:). The security group must have ports 22, 6379, 8080 and 8081 open. If not provided, Lithops will create a new security group |
115
116
|aws_ec2 | ssh_key_name | | no | SSH Key name. You can find the available keys in the [EC2 console page](https://console.aws.amazon.com/ec2/v2/home#KeyPairs:). Create a new one or upload your own key if it does not exist|
116
117
|aws_ec2 | ssh_username | ubuntu |no | Username to access the VM |
117
118
|aws_ec2 | ssh_password | |no | Password for accessing the worker VMs. If not provided, it is created randomly|
118
-
|aws_ec2 | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to access the VPC. It will use the default path if not provided |
119
+
|aws_ec2 | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to access the VPC. If not provided, Lithops will use the default path and create a new ssh key for the VPC |
119
120
|aws_ec2 | request_spot_instances | True | no | Request spot instance for worker VMs|
120
121
|aws_ec2 | target_ami | | no | Virtual machine image id. Default is Ubuntu Server 22.04 |
121
122
|aws_ec2 | master_instance_type | t2.micro | no | Profile name for the master VM |
@@ -130,6 +131,47 @@ In summary, you can use one of the following settings:
130
131
|aws_ec2 | exec_mode | reuse | no | One of: **consume**, **create** or **reuse**. If set to **create**, Lithops will automatically create new VMs for each map() call based on the number of elements in iterdata. If set to **reuse** will try to reuse running workers if exist |
131
132
132
133
134
+
## Additional configuration
135
+
136
+
# Elastic Block Store (EBS)
137
+
138
+
To attach EBS volumes to an EC2 instance in Lithops, you can configure the `aws_ec2` section as follows.
139
+
140
+
```yaml
141
+
aws_ec2:
142
+
execution_role: <EXECUTION_ROLE_ARN>
143
+
region: <REGION_NAME>
144
+
...
145
+
ebs_volumes:
146
+
- device_name: /dev/xvda
147
+
ebs:
148
+
volume_size: 100
149
+
volume_type: gp2
150
+
delete_on_termination: true
151
+
encrypted: false
152
+
kms_key_id: <KMS_KEY_ARN>
153
+
- device_name: /dev/xvdf
154
+
ebs:
155
+
volume_size: 50
156
+
volume_type: gp3
157
+
delete_on_termination: true
158
+
encrypted: false
159
+
iops: 3000
160
+
throughput: 125
161
+
...
162
+
```
163
+
164
+
|Group|Key|Default|Mandatory|Additional info|
165
+
|---|---|---|---|---|
166
+
| ebs | volume_size | 8 | No | Size of the volume in GiB |
167
+
| ebs | volume_type | gp2 | No | Type of volume. Options: `gp2`, `gp3`, `io1`, `io2`, `sc1`, `st1`, `standard`|
168
+
| ebs | delete_on_termination| True | No | Whether the volume is deleted automatically when the instance is terminated |
169
+
| ebs | encrypted | False | No | Whether the volume is encrypted |
170
+
| ebs | kms_key_i | | No | ARN of the KMS key used for encryption. If not provided, the default AWS-managed key is used |
171
+
| ebs | iops | | No | Provisioned IOPS for `io1`, `io2`, or `gp3` volumes |
172
+
| ebs | throughput | | No | Throughput in MiB/s for `gp3` volumes |
173
+
174
+
133
175
## Consume mode
134
176
135
177
In this mode, Lithops can start and stop an existing VM, and deploy an entire job to that VM. The partition logic in this scenario is different from the `create/reuse` modes, since the entire job is executed in the same VM.
@@ -188,7 +230,7 @@ In summary, you can use one of the following settings:
188
230
|aws_ec2 | ssh_username | ubuntu |no | Username to access the VM |
189
231
|aws_ec2 | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to create the VM. It will use the default path if not provided |
190
232
|aws_ec2 | worker_processes | AUTO | no | Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the VM|
191
-
|aws_ec2 | runtime | python3 | no | Runtime name to run the functions. Can be a container image name. If not set Lithops will use the defeuv python3 interpreter of the VM |
233
+
|aws_ec2 | runtime | python3 | no | Runtime name to run the functions. Can be a container image name. If not set Lithops will use the default python3 interpreter of the VM |
192
234
|aws_ec2 | auto_dismantle | True |no | If False then the VM is not stopped automatically.|
193
235
|aws_ec2 | soft_dismantle_timeout | 300 |no| Time in seconds to stop the VM instance after a job **completed** its execution |
194
236
|aws_ec2 | hard_dismantle_timeout | 3600 | no | Time in seconds to stop the VM instance after a job **started** its execution |
@@ -211,7 +253,7 @@ lithops logs poll
211
253
212
254
## VM Management
213
255
214
-
Lithops for AWS EC2 follows a Mater-Worker architecture (1:N).
256
+
Lithops for AWS EC2 follows a Master-Worker architecture (1:N).
215
257
216
258
All the VMs, including the master VM, are automatically stopped after a configurable timeout (see hard/soft dismantle timeouts).
Copy file name to clipboardExpand all lines: docs/_sources/source/design.rst
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -43,7 +43,7 @@ In Lithops, each map or reduce computation is executed as a separate compute *jo
43
43
44
44
As mentioned above, the ``FunctionExecutor`` class is responsible for orchestrating the computation in Lithops. One ``FunctionExecutor`` object is instantiated prior to any use of Lithops. Its initialization includes these important steps: 1. It sets up the workers (depending on the specific compute backend), such as constructing docker images, defining IBM Cloud Functions, etc. This step may not include actually creating the workers, as this may be done automatically by the backend on-demand. 2. It defines a bucket in object storage (depending on the storage backend) in which each job will store job and call data (prior to computation) and results (when computation is complete). 3. It creates a ``FunctionInvoker`` object, which is responsible for executing a job as a set of independent per-worker calls.
45
45
46
-
Compute jobs are created in the functions of the ``job`` module (see chart above), invoked from the respective API method of ``FunctionExecutor``. Map jobs are created in ``create_map_job()`` and reduce jobs in ``create_reduce_job()``. The flow in both functions is quite similar. First, data is partitioned, with the intention of each partition be processed by one worker. For map jobs, this is done by invoking the ``create_partitions()`` function of the ``partitioner`` module, yielding a partition map.
46
+
Compute jobs are created in the functions of the ``job`` module (see chart above), invoked from the respective API method of ``FunctionExecutor``. Map jobs are created in ``create_map_job()`` and reduce jobs in ``create_reduce_job()``. The flow in both functions is quite similar. First, data is partitioned, with the intention that each partition be processed by one worker. For map jobs, this is done by invoking the ``create_partitions()`` function of the ``partitioner`` module, yielding a partition map.
47
47
48
48
For reduce jobs, Lithops currently supports two modes: reduce per object, where each object is processed by a reduce function, and global (default) reduce, where all data is processed by a single reduce function. Respectively, data is partitioned as either one partition per storage object, or one global partition with all data. This process yields a partition map similar to map jobs. Additionally, ``create_reduce_job()`` wraps the reduce function in a special wrapper function that forces waiting for data before the actual reduce function is invoked. This is because reduce jobs follow map jobs, so the output of the map jobs needs to finish before reduce can run.
49
49
@@ -58,4 +58,4 @@ Completion of a computation job in Lithops is detected in one of two techniques:
58
58
59
59
**RabbitMQ**: A unique RabbitMQ topic is defined for each job. combining the executor id and job id. Each worker, once completes a call, posts a notification message on that topic (code in ``function_handler()`` in ``handler`` module, called from ``entry_point`` module of the worker). The ``wait_rabbitmq()`` function from ``wait_rabbitmq`` module, which is called from ``FunctionExecutor.wait()``, consumes a number of messages on that topic equal to ``total_calls`` and determines completion.
60
60
61
-
**Object Storage**: As explained above, each call persists its computation results in a specific object. Determining completion of a job is by the ``FunctionExecutor.wait()`` invoking the ``wait_storage()`` function from the ``wait_storage`` module. This function repeatedly, once per fixed period (controllable), polls the executor’s bucket for status objects of a subset of calls that have still not completed. This allows control of resource usage and eventual detection of all calls.
61
+
**Object Storage**: As explained above, each call persists its computation results in a specific object. Determining completion of a job is by the ``FunctionExecutor.wait()`` invoking the ``wait_storage()`` function from the ``wait_storage`` module. This function repeatedly, once per fixed period (controllable), polls the executor’s bucket for status objects of a subset of calls that have still not completed. This allows control of resource usage and eventual detection of all calls.
0 commit comments