Skip to content

Commit f2f5a62

Browse files
JosepSampeJosepSampe
committed
Update docs to 3.4.0
1 parent 1c17fea commit f2f5a62

97 files changed

Lines changed: 812 additions & 400 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/_sources/source/api_multiprocessing.rst‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,13 @@ Multiprocessing API
33

44
Lithops implements Python's `multiprocessing API <https://docs.python.org/3/library/multiprocessing.html>`_ to transparently run local-parallel applications but using serverless functions for Processes and a Redis instance for shared state and Inter-Process Communication (IPC).
55

6+
Before utilizing this API, you will need to install its dependencies:
7+
8+
.. code-block:: bash
9+
10+
python3 -m pip install lithops[multiprocessing]
11+
12+
613
Process and Pool
714
----------------
815

@@ -50,9 +57,9 @@ The Redis credentials (host, password...) is loaded from the ``redis`` section o
5057
5158
The fastest way to deploy a Redis instance is using Docker in a VM located in the cloud of your choice:
5259
53-
.. code::
60+
.. code:: bash
5461
55-
$ docker run --rm -d --network host --name redis redis:6.2.1 --requirepass redispassword
62+
docker run --rm -d --network host --name redis redis:6.2.1 --requirepass redispassword
5663
5764
To have lower latency, you can deploy the functions and the VM in the same VPC and use route through internal traffic instead of the internet.
5865
For example, in AWS, the functions and VM can be deployed in the same VPC: Lambdas go to a private subnet and the VM in a public subnet. This way, the VM has access to the internet and the local Lithops process can also access it.
@@ -79,7 +86,7 @@ For this reason, to set specific configuration in runtime, the ``Lithops.multipr
7986
8087
# To set a config parameter, use the set_parameter
8188
# function and specify the parameter and the desired value
82-
mp_config.set_parameter(mp_config.LITHOPS_CONFIG, {'lithops': {'mode': 'localhost'}})
89+
mp_config.set_parameter(mp_config.LITHOPS_CONFIG, {'lithops': {'backend': 'localhost'}})
8390
mp_config.set_parameter(mp_config.STREAM_STDOUT, True)
8491
mp_config.set_parameter(mp_config.REDIS_EXPIRY_TIME, 1800)
8592
mp_config.set_parameter(mp_config.PIPE_CONNECTION_TYPE, 'redislist')
@@ -120,5 +127,4 @@ Multiprocessing configuration keys
120127
- ``None``
121128
122129
123-
124130
* To use nanomsg for Pipes, you must still deploy a Redis instance (used for pipe directory). Note that this feature only works in environments where functions can open a port and communicate with each other.

‎docs/_sources/source/api_stats.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,12 @@ Execution summary plots
1010

1111
The :code:`plot()` method from :code:`FunctionExecutor` creates a scatter plot and a histogram plot showing a summary of the tasks executed by a :code:`FunctionExecutor`. By default, lithops creates a :code:`plots/` directory in the working directory path containing both plots in PNG format. For more details refer to the `FunctionExecutor API reference <https://lithops-cloud.github.io/docs/source/api_futures.html>`_.
1212

13+
To get started, first install Lithops and the plotting dependencies with:
14+
15+
.. code-block:: bash
16+
17+
python3 -m pip install lithops[plotting]
18+
1319
* **Scatter Plot**: the scatter plot shows a timeline on the horizontal axis where the stages of all invocations are arranged on the vertical axis.
1420
- :code:`host submit` indicates the time that the orchestrator process has invoked the function.
1521
- :code:`call start` indicates the timestamp at which the function starts its execution.

‎docs/_sources/source/compute_config/aliyun_functions.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Lithops with *Aliyun Function Compute* as serverless compute backend.
66

77
1. Install Alibaba Cloud backend dependencies:
88

9-
```
9+
```bash
1010
python3 -m pip install lithops[aliyun]
1111
```
1212

‎docs/_sources/source/compute_config/aws_batch.md‎

Lines changed: 55 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -2,19 +2,58 @@
22

33
Lithops with *AWS Batch* as serverless batch compute backend.
44

5+
## Installation
6+
7+
1. Install AWS backend dependencies:
8+
9+
```bash
10+
python3 -m pip install lithops[aws]
11+
```
12+
513
## Configuration
614

715
1. [Login](https://console.aws.amazon.com/?nc2=h_m_mc) to Amazon Web Services Console (or signup if you don't have an account)
816

9-
2. Navigate to **IAM > Roles** to create the ECS Task Execution Role. AWS provides a defualt role named `ecsTaskExecutionRole`, which can be used instead. If you want to create another role or it is missing, create a new role attached to `Elastic Container Service Task`, and add the following policies:
17+
2. Navigate to **IAM > Roles** to create the ECS Task Execution Role. AWS provides a default role named `ecsTaskExecutionRole`, which can be used instead. If you want to create another role or it is missing, create a new role attached to `Elastic Container Service Task`, and add the following policies:
1018
- `SecretsManagerReadWrite`
1119
- `AmazonEC2ContainerRegistryFullAccess`
1220
- `CloudWatchFullAccess`
1321
- `AmazonECSTaskExecutionRolePolicy`
1422

15-
3. Navigate to **IAM > Roles** to create the ECS Instance Role. AWS provides a defualt role named `ecsInstanceRole`, which can be used instead. If you want to create another role or it is missing, create a new role attached to `EC2`, and add the following policy:
23+
3. Navigate to **IAM > Roles** to create the ECS Instance Role. AWS provides a default role named `ecsInstanceRole`, which can be used instead. If you want to create another role or it is missing, create a new role attached to `EC2`, and add the following policy:
1624
- `AmazonEC2ContainerServiceforEC2Role`
1725

26+
4. Navigate to **IAM > Policies**. Click on **Create policy**. If you already created this policy for the AWS Lambda or AWS EC2 backend, jump to step 7.
27+
28+
5. Select **JSON** tab and paste the following JSON policy:
29+
```json
30+
{
31+
"Version": "2012-10-17",
32+
"Statement": [
33+
{
34+
"Effect": "Allow",
35+
"Action": [
36+
"s3:*",
37+
"lambda:*",
38+
"ec2:*",
39+
"ecr:*",
40+
"sts:GetCallerIdentity",
41+
"logs:CreateLogGroup",
42+
"logs:CreateLogStream",
43+
"logs:PutLogEvents"
44+
],
45+
"Resource": "*"
46+
}
47+
]
48+
}
49+
```
50+
51+
6. Click **Next: Tags** and **Next: Review**. Fill the policy name field (you can name it `lithops-policy` or similar) and create the policy.
52+
53+
7. Go back to **IAM** and navigate to **Roles** tab. Click **Create role**.
54+
55+
8. Choose **Elastic Container Service** on the use case list and then click on **Elastic Container Service Task**. Click **Next: Permissions**. Select the policy created before (`lithops-policy`). Click **Next: Tags** and **Next: Review**. Type a role name, for example `ecsTaskJobRole`. Click on **Create Role**.
56+
1857
## AWS Credential setup
1958

2059
Lithops loads AWS credentials as specified in the [boto3 configuration guide](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html).
@@ -31,18 +70,17 @@ In summary, you can use one of the following settings:
3170
aws_batch:
3271
region : <REGION_NAME>
3372
execution_role: <EXECUTION_ROLE_ARN>
34-
instance_role: <INSTANCE_ROLE_ARN>
73+
job_role: <JOB_ROLE_ARN>
3574
subnets:
3675
- <SUBNET_ID_1>
3776
- <SUBNET_ID_2>
3877
- ...
3978
security_groups:
4079
- <SECURITY_GROUP_1>
41-
- <SECURITY_GROUP_2>
4280
- ...
4381
```
4482
45-
2. Provide the credentials in the `aws` section of the Lithops config file:
83+
2. Provide the credentials in the `aws` section of the Lithops config file. In this case you can omit setting the `job_role`:
4684
```yaml
4785
lithops:
4886
backend: aws_batch
@@ -54,14 +92,13 @@ In summary, you can use one of the following settings:
5492
5593
aws_batch:
5694
execution_role: <EXECUTION_ROLE_ARN>
57-
instance_role: <INSTANCE_ROLE_ARN>
95+
job_role: <JOB_ROLE_ARN> # Not mandatory if the credentials are in the aws section
5896
subnets:
5997
- <SUBNET_ID_1>
6098
- <SUBNET_ID_2>
6199
- ...
62100
security_groups:
63101
- <SECURITY_GROUP_1>
64-
- <SECURITY_GROUP_2>
65102
- ...
66103
```
67104

@@ -82,17 +119,18 @@ In summary, you can use one of the following settings:
82119
|Group|Key|Default|Mandatory|Additional info|
83120
|---|---|---|---|---|
84121
| aws_batch | execution_role | | yes | ARN of the execution role used to execute AWS Batch tasks on ECS for Fargate environments |
85-
| aws_batch | instance_role | | yes | ARN of the execution role used to execute AWS Batch tasks on ECS for EC2 environments |
122+
| aws_batch | job_role | | yes | ARN of the job role used to execute AWS Batch tasks on ECS for Fargate environments. Not mandatory if the credentials are in the `aws` section of the configuration|
86123
| aws_batch | security_groups | | yes | List of Security groups to attach for ECS task containers. By default, you can use a security group that accepts all outbound traffic but blocks all inbound traffic. |
87-
| aws_batch | subnets | | yes | List of subnets from a VPC where to deploy the ECS task containers. Note that if you are using a **private subnet**, you can set `assing_public_ip` to `false` but make sure containers can reach other AWS services like ECR, Secrets service, etc., by, for example, using a NAT gateway. If you are using a **public subnet** you must set `assing_public_ip` to `true` |
124+
| aws_batch | subnets | | yes | List of subnets from a VPC where to deploy the ECS task containers. Note that if you are using a **private subnet**, you can set `assign_public_ip` to `false` but make sure containers can reach other AWS services like ECR, Secrets service, etc., by, for example, using a NAT gateway. If you are using a **public subnet** you must set `assign_public_up` to `true` |
125+
| aws_batch | instance_role | | no | ARN of the execution role used to execute AWS Batch tasks on ECS for EC2 environments. Mandatory if using the **EC2** or **SPOT** `env_type` |
88126
| aws_batch | region | | no | Region name (like `us-east-1`) where to deploy the ECS cluster. Lithops will use the region set under the `aws` section if it is not set here |
89-
| aws_batch | assign_public_ip | `true` | no | Assing public IPs to ECS task containers. Set to `true` if the tasks are being deployed in a public subnet. Set to `false` when deploying on a private subnet. |
90-
| aws_batch | runtime | `default_runtime-v3X` | no | Runtime name |
91-
| aws_batch | runtime_timeout | 180 | no | Runtime timeout |
92-
| aws_batch | runtime_memory | 1024 | no | Runtime memory |
93-
| aws_batch | worker_processes | 1 | no | Worker processes |
94-
| aws_batch | container_vcpus | 0.5 | no | Number of vCPUs assigned to each task container. It can be different from `worker_processes`. Use it to run a task that uses multiple processes within a container. |
95-
| aws_batch | service_role | `None` | no | Service role for AWS Batch. Leave empty for use a service-linked execution role. More info [here](https://docs.aws.amazon.com/batch/latest/userguide/using-service-linked-roles.html) |
127+
| aws_batch | assign_public_ip | `true` | no | Assign public IPs to ECS task containers. Set to `true` if the tasks are being deployed in a public subnet. Set to `false` when deploying on a private subnet. |
128+
| aws_batch | runtime | | no | Container runtime name in ECR. If not provided Lithops will automatically build a default runtime |
129+
| aws_batch | runtime_timeout | 180 | no | Runtime timeout managed by the cloud provider. |
130+
| aws_batch | runtime_memory | 1024 | no | Runtime memory assigned to each task container. |
131+
| aws_batch | runtime_cpu | 0.5 | no | Number of vCPUs assigned to each task container. It can be different from `worker_processes`. |
132+
| aws_batch | worker_processes | 1 | no | Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. |
133+
| aws_batch | service_role | | no | Service role for AWS Batch. Leave empty to use a service-linked execution role. More info [here](https://docs.aws.amazon.com/batch/latest/userguide/using-service-linked-roles.html) |
96134
| aws_batch | env_max_cpus | 10 | no | Maximum total CPUs of the compute environment |
97135
| aws_batch | env_type | FARGATE_SPOT | no | Compute environment type, one of: `["EC2", "SPOT", "FARGATE", "FARGATE_SPOT"]` |
98136

@@ -110,4 +148,4 @@ You can view the function executions logs in your local machine using the *litho
110148

111149
```bash
112150
lithops logs poll
113-
```
151+
```

‎docs/_sources/source/compute_config/aws_ec2.md‎

Lines changed: 49 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,14 @@ Any Virtual Machine (VM) need to define the instance’s operating system and ve
1212

1313
- Option 2: Alternatively, you can use a pre-built custom image that will greatly improve VM creation time for Lithops jobs. To benefit from this approach, navigate to [runtime/aws_ec2](https://github.com/lithops-cloud/lithops/tree/master/runtime/aws_ec2), and follow the instructions.
1414

15+
## Installation
16+
17+
1. Install AWS backend dependencies:
18+
19+
```bash
20+
python3 -m pip install lithops[aws]
21+
```
22+
1523
## Lithops Consume mode
1624

1725
In this mode, Lithops can start and stop an existing VM, and deploy an entire job to that VM. The partition logic in this scenario is different from the `create/reuse` modes, since the entire job is executed in the same VM.
@@ -69,7 +77,7 @@ In summary, you can use one of the following settings:
6977
|aws_ec2 | region | |yes | Region name of the VPC. For example `us-east-1`. Lithops will use the region set under the `aws` section if it is not set here |
7078
|aws_ec2 | ssh_username | ubuntu |no | Username to access the VM |
7179
|aws_ec2 | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to create the VM. It will use the default path if not provided |
72-
|aws_ec2 | worker_processes | AUTO | no | Number of Lithops processes within a given worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the VM|
80+
|aws_ec2 | worker_processes | AUTO | no | Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the VM|
7381
|aws_ec2 | runtime | python3 | no | Runtime name to run the functions. Can be a container image name. If not set Lithops will use the defeuv python3 interpreter of the VM |
7482
|aws_ec2 | auto_dismantle | True |no | If False then the VM is not stopped automatically.|
7583
|aws_ec2 | soft_dismantle_timeout | 300 |no| Time in seconds to stop the VM instance after a job **completed** its execution |
@@ -80,6 +88,41 @@ In summary, you can use one of the following settings:
8088
In the `create` mode, Lithops will automatically create new worker VM instances in runtime, scale Lithops job against generated VMs, and automatically delete the VMs when the job is completed.
8189
Alternatively, you can set the `reuse` mode to keep running the started worker VMs, and reuse them for further executions. In the `reuse` mode, Lithops checks all the available worker VMs and start new workers only if necessary.
8290

91+
92+
### Configuration
93+
94+
1. Navigate to **IAM > Policies**. Click on **Create policy**. If you already created this policy for the AWS Lambda or AWS Batch backend, jump to step 4.
95+
96+
2. Select **JSON** tab and paste the following JSON policy:
97+
```json
98+
{
99+
"Version": "2012-10-17",
100+
"Statement": [
101+
{
102+
"Effect": "Allow",
103+
"Action": [
104+
"s3:*",
105+
"lambda:*",
106+
"ec2:*",
107+
"ecr:*",
108+
"sts:GetCallerIdentity",
109+
"logs:CreateLogGroup",
110+
"logs:CreateLogStream",
111+
"logs:PutLogEvents"
112+
],
113+
"Resource": "*"
114+
}
115+
]
116+
}
117+
```
118+
119+
3. Click **Next: Tags** and **Next: Review**. Fill the policy name field (you can name it `lithops-policy` or similar) and create the policy.
120+
121+
4. Go back to **IAM** and navigate to **Roles** tab. Click **Create role**.
122+
123+
5. Choose **EC2** on the use case list. Click **Next: Permissions**. Select the policy created before (`lithops-policy`). Click **Next: Tags** and **Next: Review**. Type a role name, for example `ec2LithopsInstanceRole`. Click on **Create Role**.
124+
125+
83126
### AWS Credential setup
84127

85128
Lithops loads AWS credentials as specified in the [boto3 configuration guide](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html).
@@ -95,7 +138,7 @@ In summary, you can use one of the following settings:
95138
96139
aws_ec2:
97140
region : <REGION_NAME>
98-
iam_role: <IAM_ROLE_NAME>
141+
instance_role: <IAM_INSTANCE_ROLE_NAME>
99142
exec_mode: reuse
100143
```
101144

@@ -110,7 +153,7 @@ In summary, you can use one of the following settings:
110153
region: <REGION_NAME>
111154
112155
aws_ec2:
113-
iam_role: <IAM_ROLE_NAME>
156+
instance_role: <IAM_INSTANCE_ROLE_NAME>
114157
exec_mode: reuse
115158
```
116159

@@ -129,7 +172,7 @@ In summary, you can use one of the following settings:
129172
|Group|Key|Default|Mandatory|Additional info|
130173
|---|---|---|---|---|
131174
|aws_ec2 | region | |no | Region name, for example: `eu-west-1`. Lithops will use the `region` set under the `aws` section if it is not set here |
132-
|aws_ec2 | iam_role | | yes | IAM EC2 role name. You can find it in the [IAM Console page](https://console.aws.amazon.com/iamv2/home#/roles). Create a new EC2 role if it does not exist|
175+
|aws_ec2 | instance_role | | yes | EC2 Instance role name created in the configuration section above. Do not use the full ARN here; only the role name is required. For example: `ec2LithopsInstanceRole`|
133176
|aws_ec2 | vpc_id | | no | VPC id. You can find all the available VPCs in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#vpcs:) |
134177
|aws_ec2 | subnet_id | | no | Subnet id. You can find all the available Subnets in the [VPC Console page](https://console.aws.amazon.com/vpc/v2/home#subnets:) |
135178
|aws_ec2 | security_group_id | | no | Security group ID. You can find the available security groups in the [VPC console page](https://console.aws.amazon.com/vpc/v2/home#SecurityGroups:). The security group must have ports 22 and 8080 open |
@@ -143,7 +186,7 @@ In summary, you can use one of the following settings:
143186
|aws_ec2 | worker_instance_type | t2.medium | no | Profile name for the worker VMs |
144187
|aws_ec2 | delete_on_dismantle | True | no | Delete the worker VMs when they are stopped. Master VM is never deleted when stopped |
145188
|aws_ec2 | max_workers | 100 | no | Max number of workers per `FunctionExecutor()`|
146-
|aws_ec2 | worker_processes | 2 | no | Number of Lithops processes within a given worker. This can be used to parallelize function activations within a worker. It is recommendable to set this value to the same number of CPUs of a worker VM. |
189+
|aws_ec2 | worker_processes | AUTO | no | Number of parallel Lithops processes in a worker. This is used to parallelize function activations within the worker. By default it detects the amount of CPUs in the `worker_instance_type` VM|
147190
|aws_ec2 | runtime | python3 | no | Runtime name to run the functions. Can be a container image name. If not set Lithops will use the default python3 interpreter of the VM |
148191
|aws_ec2 | auto_dismantle | True |no | If False then the VM is not stopped automatically.|
149192
|aws_ec2 | soft_dismantle_timeout | 300 |no| Time in seconds to stop the VM instance after a job **completed** its execution |
@@ -168,7 +211,7 @@ lithops logs poll
168211

169212
## VM Management
170213

171-
Lithops for AWS EC2 follows a Mater-Worker architecrue (1:N).
214+
Lithops for AWS EC2 follows a Mater-Worker architecture (1:N).
172215

173216
All the VMs, including the master VM, are automatically stopped after a configurable timeout (see hard/soft dismantle timeouts).
174217

0 commit comments

Comments
 (0)