Skip to content
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,18 @@
- [K8s] Added configuration for pod and container `securityContext`.
- [Docs] Corrected MinIO/Ceph config template keys and removed obsolete Kubernetes image references.
- [GCP Functions] Updated `gcp_functions` backend to Google Cloud Run functions (Cloud Functions v2 API).
- [Python] Updated all backend python versions (deprecated Python 3.9, added Python 3.14)
- [Azure Funcions] Updated default funcions plan to Flex Consumption
- [AWS EC2] Updated default Ubuntu Image to Ubuntu 24
- [Azure VMS] Updated default Ubuntu Image to Ubuntu 24

### Fixed
- [K8s] Fixed default runtime builds impacted by Debian Buster end-of-life.
- [GCP Cloud Run] Added Artifact Registry (`pkg.dev`) runtime deployment support
- [K8s] Run default runtime image as non-root user (uid 1000) (#1469)
- [AWS] Fixed AWS backends execution
- [GCP] Fixed GCP backends execution
- [Azure] Fixed Azure backends execution


## [v3.6.4]
Expand Down
5 changes: 3 additions & 2 deletions config/config_template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@
#ssh_password: <RANDOM_IF_NOT_PROVIDED>
#ssh_key_filename: ~/.ssh/id_rsa
#request_spot_instances: True
#target_ami: <AMI_ID> # Default: Ubuntu Server 22.04
#target_ami: <AMI_ID> # Default: Ubuntu Server 24.04
#master_instance_type: t3.micro
#worker_instance_type: t3.medium
#delete_on_dismantle: True
Expand Down Expand Up @@ -220,6 +220,7 @@
#resource_group: <RESOURCE_GROUP_NAME># Falls back to azure.resource_group
#region: <LOCATION> # Falls back to azure.region
#environment: lithops # Container Apps environment name
#environment_id: <ENVIRONMENT_ID> # Optional. Full resource ID of the Container Apps environment
#docker_server: index.docker.io
#docker_user: <REGISTRY_USERNAME>
#docker_password: <REGISTRY_TOKEN>
Expand All @@ -236,7 +237,7 @@
#azure_vms:
#region: <LOCATION> # Falls back to azure.region
#instance_name: <VM_NAME> # Mandatory in consume mode
#image_id: Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest
#image_id: Canonical:ubuntu-24_04-lts:server:latest
#master_instance_type: Standard_B1s
#worker_instance_type: Standard_B2s
#ssh_username: ubuntu
Expand Down
4 changes: 2 additions & 2 deletions docs/source/compute_config/aws_ec2.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ This guide assumes that you are already familiar with AWS, and that you have aut
### Choose an operating system image for the VM
Any Virtual Machine (VM) needs to define the instance's operating system and version. Lithops supports both standard operating system choices provided by AWS and pre-defined custom images that already contain all dependencies required by Lithops.

- Option 1: By default, Lithops uses an Ubuntu 22.04 image. In this case, no further action is required and you can continue to the next step. Lithops will install all required dependencies in the VM by itself. Note that this can take about 3 minutes to complete all installations.
- Option 1: By default, Lithops uses an Ubuntu 24.04 image. In this case, no further action is required and you can continue to the next step. Lithops will install all required dependencies in the VM by itself. Note that this can take about 3 minutes to complete all installations.

- Option 2: Alternatively, you can use a pre-built custom image that will greatly improve VM creation time for Lithops jobs. To benefit from this approach, navigate to [runtime/aws_ec2](https://github.com/lithops-cloud/lithops/tree/master/runtime/aws_ec2), and follow the instructions.

Expand Down Expand Up @@ -119,7 +119,7 @@ In summary, you can use one of the following settings:
|aws_ec2 | ssh_password | |no | Password for accessing the worker VMs. If not provided, it is created randomly|
|aws_ec2 | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to access the VPC. If not provided, Lithops will use the default path and create a new ssh key for the VPC |
|aws_ec2 | request_spot_instances | True | no | Request spot instance for worker VMs|
|aws_ec2 | target_ami | | no | Virtual machine image id. Default is Ubuntu Server 22.04 |
|aws_ec2 | target_ami | | no | Virtual machine image id. Default is Ubuntu Server 24.04 |
|aws_ec2 | master_instance_type | t2.micro | no | Profile name for the master VM |
|aws_ec2 | worker_instance_type | t2.medium | no | Profile name for the worker VMs |
|aws_ec2 | delete_on_dismantle | True | no | Delete the worker VMs when they are stopped. Master VM is never deleted when stopped |
Expand Down
5 changes: 3 additions & 2 deletions docs/source/compute_config/azure_containers.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,14 +95,15 @@ az containerapp env create --name lithops --resource-group LithopsResourceGroup
|---|---|---|---|---|
|azure_containers| resource_group | |no | Name of a resource group, for example: `LithopsResourceGroup`. Lithops will use the `resource_group` set under the `azure` section if it is not set here |
|azure_containers| region | |no | The location where you created the `lithops` Container APP environment. For example: `westeurope`, `westus2`, etc. Lithops will use the `region` set under the `azure` section if it is not set here|
|azure_containers| environment | lithops |no | The environment name you created in the step 5 of the installation |
|azure_containers| environment | lithops |no | Container Apps environment name. Used to resolve the environment resource ID at deploy time when `environment_id` is not set |
|azure_containers| environment_id | |no | Full Azure resource ID of the Container Apps environment (for example: `/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.App/managedEnvironments/lithops`). If set, Lithops uses it directly and skips the `az containerapp env show` lookup |
|azure_containers | docker_server | index.docker.io |no | Container registry URL |
|azure_containers | docker_user | |no | Container registry user name |
|azure_containers | docker_password | |no | Container registry password/token. In case of Docker hub, login to your docker hub account and generate a new access token [here](https://hub.docker.com/settings/security)|
|azure_containers | max_workers | 1000 | no | Max number of parallel workers. Although Azure limits the number of parallel workers to 30, it is convenient to keep this value high|
|azure_containers | worker_processes | 1 | no | Number of Lithops processes within a given worker. This can be used to parallelize function activations within a worker |
|azure_containers| runtime | |no | Docker image name|
|azure_containers | runtime_memory | 512 |no | Memory limit in MB. Default 512Mi |
|azure_containers | runtime_memory | 512 |no | Memory limit in MB. Allowed values (MB): 512, 1024, 1536, 2048, 2560, 3072, 3584, 4096, 4608, 5120, 5632, 6144, 6656, 7168, 7680, 8192. Each maps to a fixed CPU/memory pair on the Consumption plan. Consumption-only environments are limited to 4096 MB (2 vCPU / 4 Gi) |
|azure_containers | runtime_timeout | 600 |no | Runtime timeout in seconds. Default 10 minutes |
|azure_containers| trigger | pub/sub | no | Currently it supports pub/sub invocation|
|azure_containers | invoke_pool_threads | 32 |no | Number of concurrent threads used for invocation |
Expand Down
13 changes: 6 additions & 7 deletions docs/source/compute_config/azure_functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,13 @@ python3 -m pip install lithops[azure]

2. Install [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest)

3. Install the [Azure Functions core tools](https://github.com/Azure/azure-functions-core-tools)

4. Sign in with the Azure CLI:
3. Sign in with the Azure CLI:

```bash
az login
```

5. Create a Resource Group and a Storage Account:
4. Create a Resource Group and a Storage Account:

Option 1:

Expand Down Expand Up @@ -80,17 +78,18 @@ az login

|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|azure_storage| storage_account_name | |yes | Storage account name. The name generated in step 5 of the installation if you followed these instructions |
|azure_storage| storage_account_name | |yes | Storage account name. The name generated in step 4 of the installation if you followed these instructions |
|azure_storage| storage_account_key | | yes | An Account Key, found in *Storage Accounts* > `account_name` > *Security + networking* > *Access Keys*|

### Azure Functions

|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|azure_functions| resource_group | |no | Name of a resource group, for example: `LithopsResourceGroup`. Lithops will use the `resource_group` set under the `azure` section if it is not set here |
|azure_functions| region | |no | The location of the consumption plan for the runtime. Use `az functionapp list-consumption-locations` to view the available locations. For example: `westeurope`, `westus2`, etc. Lithops will use the `region` set under the `azure` section if it is not set here|
|azure_functions| region | |no | The Flex Consumption plan location for the runtime. Use `az functionapp list-flexconsumption-locations` to view the available locations. For example: `westeurope`, `westus2`, etc. Lithops will use the `region` set under the `azure` section if it is not set here|
|azure_functions | runtime_memory | 2048 | no | Flex Consumption instance memory in MB. Supported values: `512`, `2048`, `4096`. Other values are mapped to the nearest supported size |
|azure_functions | max_workers | 1000 | no | Max number of parallel workers. Although Azure limits the number of workers to 200, it is convenient to keep this value high|
|azure_functions | worker_processes | 1 | no | Number of Lithops processes within a given worker. This can be used to parallelize function activations within a worker |
|azure_functions | worker_processes | 1 | no | Lithops-side parallelism setting. Not applied as an Azure Functions app setting on Flex Consumption |
|azure_functions| runtime | |no | Runtime name already deployed in the service|
|azure_functions | runtime_timeout | 300 |no | Runtime timeout in seconds. Default 5 minutes |
|azure_functions| trigger | pub/sub | no | One of 'https' or 'pub/sub'|
Expand Down
2 changes: 1 addition & 1 deletion docs/source/compute_config/azure_vms.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ Edit your lithops config and add the relevant keys:
|Group|Key|Default|Mandatory|Additional info|
|---|---|---|---|---|
|azure_vms| region | |no | Azure location for deploying the VMs. For example: `westeurope`, `westus2`, etc. Lithops will use the `region` set under the `azure` section if it is not set here|
|azure_vms | image_id | Canonical:0001-com-ubuntu-server-jammy:22_04-lts-gen2:latest |no | Image ID. ARM resource identifier |
|azure_vms | image_id | Canonical:ubuntu-24_04-lts:server:latest |no | Image ID. ARM resource identifier |
|azure_vms | ssh_username | ubuntu |no | Username to access the VM |
|azure_vms | ssh_password | |no | Password for accessing the worker VMs. If not provided, it is created randomly|
|azure_vms | ssh_key_filename | ~/.ssh/id_rsa | no | Path to the ssh key file provided to access the VPC. It will use the default path if not provided |
Expand Down
4 changes: 2 additions & 2 deletions docs/source/compute_config/gcp_compute_engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ Lithops attaches the service account from `credentials_path` to master and worke
|gcp_compute_engine | worker_instance_type | e2-standard-2 |no | Worker VM machine type |
|gcp_compute_engine | ssh_username | ubuntu |no | Username to access the VM |
|gcp_compute_engine | ssh_password | |no | Password for worker VMs. If not provided, it is created randomly |
|gcp_compute_engine | ssh_key_filename | ~/.ssh/id_rsa |no | SSH private key for the master VM. If not provided, Lithops creates one |
|gcp_compute_engine | ssh_key_filename | |no | SSH private key for the master VM. If not provided, Lithops creates `~/.ssh/lithops-key-<id>.gcp_ce.id_rsa` |
|gcp_compute_engine | request_spot_instances | False |no | Use Spot VMs for workers |
|gcp_compute_engine | delete_on_dismantle | True |no | Delete worker VMs when stopped. Master VM is never deleted when stopped |
|gcp_compute_engine | max_workers | 100 |no | Max number of workers per `FunctionExecutor()` |
Expand Down Expand Up @@ -138,7 +138,7 @@ gcp_compute_engine:
|gcp_compute_engine | project_name | |yes | GCP project ID |
|gcp_compute_engine | zone | |yes | Compute Engine zone |
|gcp_compute_engine | ssh_username | ubuntu |no | Username to access the VM |
|gcp_compute_engine | ssh_key_filename | ~/.ssh/id_rsa |no | Path to the SSH private key |
|gcp_compute_engine | ssh_key_filename | |no | Path to the SSH private key. If not provided, Lithops creates `~/.ssh/lithops-key-<id>.gcp_ce.id_rsa` |
|gcp_compute_engine | worker_processes | AUTO |no | Parallel Lithops processes per worker |

## Test Lithops
Expand Down
21 changes: 21 additions & 0 deletions lithops/_grpc_env.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

import os

# Google Cloud clients (Pub/Sub, etc.) use gRPC. Forking while gRPC is active
# (tqdm/multiprocessing, cleaner subprocess) logs noisy C-core messages on macOS.
# Must be set before gRPC initializes.
os.environ.setdefault('GRPC_ENABLE_FORK_SUPPORT', '0')
os.environ.setdefault('GRPC_VERBOSITY', 'NONE')
4 changes: 0 additions & 4 deletions lithops/executors.py
Original file line number Diff line number Diff line change
Expand Up @@ -628,10 +628,6 @@ def save_data_to_clean(data):
if (jobs_to_clean or cs) and spawn_cleaner:
cmd = [sys.executable, '-m', 'lithops.scripts.cleaner']
env = os.environ.copy()
# Cleaner is forked while gRPC clients may already be active in the parent process.
# Reduce noisy gRPC fork logs in the detached cleaner subprocess.
env.setdefault('GRPC_ENABLE_FORK_SUPPORT', '0')
env.setdefault('GRPC_VERBOSITY', 'ERROR')
CLEANER_PROCESS = sp.Popen(
cmd,
start_new_session=True,
Expand Down
2 changes: 1 addition & 1 deletion lithops/libs/imp/imp.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
from _imp import is_builtin, is_frozen

from importlib._bootstrap import _ERR_MSG
from importlib import machinery
import os
import sys
import tokenize

_ERR_MSG = 'No module named {!r}'

SEARCH_ERROR = 0
PY_SOURCE = 1
Expand Down
9 changes: 6 additions & 3 deletions lithops/localhost/v1/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,9 +40,12 @@
logger = logging.getLogger('lithops.localhost.runner')


# Change spawn method for MacOS
if platform.system() == 'Darwin':
mp.set_start_method("fork")
# Python 3.14 defaults to forkserver on Linux; Lithops requires fork.
if platform.system() != 'Windows':
try:
mp.set_start_method('fork')
except RuntimeError:
pass


def run_job():
Expand Down
9 changes: 6 additions & 3 deletions lithops/localhost/v2/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,12 @@
logger = logging.getLogger('lithops.localhost.runner')


# Change spawn method for MacOS
if platform.system() == 'Darwin':
mp.set_start_method("fork")
# Python 3.14 defaults to forkserver on Linux; Lithops requires fork.
if platform.system() != 'Windows':
try:
mp.set_start_method('fork')
except RuntimeError:
pass


def run_job():
Expand Down
4 changes: 1 addition & 3 deletions lithops/serverless/backends/aws_batch/aws_batch.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@
import os
import re
import logging
import subprocess
import sys
import botocore
import time
Expand Down Expand Up @@ -400,8 +399,7 @@ def build_runtime(self, runtime_name, runtime_file, extra_args=[]):
finally:
os.remove(batch_config.RUNTIME_ZIP)

cmd = f'{docker_path} login --username AWS --password-stdin {registry}'
subprocess.check_output(cmd.split(), input=ecr_token)
utils.docker_login('AWS', ecr_token.decode('utf-8'), registry)

try:
self.ecr_client.create_repository(repositoryName=repo_name,
Expand Down
4 changes: 1 addition & 3 deletions lithops/serverless/backends/aws_lambda/aws_lambda.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
import time
import json
import zipfile
import subprocess
import botocore.exceptions
import base64

Expand Down Expand Up @@ -357,8 +356,7 @@ def build_runtime(self, runtime_name, runtime_file, extra_args=[]):
auth_data = res['authorizationData'].pop()
ecr_token = base64.b64decode(auth_data['authorizationToken']).split(b':')[1]

cmd = f'{docker_path} login --username AWS --password-stdin {registry}'
subprocess.check_output(cmd.split(), input=ecr_token)
utils.docker_login('AWS', ecr_token.decode('utf-8'), registry)

repo_name = self._format_repo_name(runtime_name)

Expand Down
Loading
Loading