Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Reporting bugs and asking questions

You can post questions or issues or feedback through the following channels:

1. [Github Discussions](https://github.com/lithops-cloud/lithops/discussions): For discussions about development, questions about usage, and feature requests.
1. [GitHub Discussions](https://github.com/lithops-cloud/lithops/discussions): For discussions about development, questions about usage, and feature requests.
2. [GitHub Issues](https://github.com/lithops-cloud/lithops/issues): For bug reports and feature requests.


Expand All @@ -17,7 +17,7 @@ To contribute a patch:
harder to merge in a large change with a lot of disjoint features.
2. Submit the patch as a GitHub pull request against the master branch.
3. Make sure that your code passes the tests.
4. Make sure that your code passes the linter. Install `flake8` with `pip3 install flake8` and run the following command until you don't see any linting error:
4. Make sure that your code passes the linter. Install `flake8` with `pip3 install flake8` and run the following command until you see no linting errors:
```bash
flake8 lithops --count --max-line-length=180 --statistics --ignore W605,W503
```
Expand All @@ -27,7 +27,7 @@ To contribute a patch:
Testing
-------

To test that all is working as expected, you must install `pytest`, navigate to the tests folder `lithops/tests/`, and execute:
To test that everything is working as expected, you must install `pytest`, navigate to the tests folder `lithops/tests/`, and execute:
```bash
pytest -v
```
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
</p>

<p align="center">
<a aria-label="License" href="https://github.com/lithops-cloud/lithops/blob/master/LICENSE"><img alt="" src="https://img.shields.io/github/license/lithops-cloud/lithops?style=for-the-badge&labelColor=000000"></a>&nbsp<a aria-label="PyPi" href="https://pypi.org/project/lithops/"><img alt="" src="https://img.shields.io/pypi/v/lithops?style=for-the-badge&labelColor=000000"></a>&nbsp<a aria-label="Python" href="#lithops"><img alt="" src="https://img.shields.io/pypi/pyversions/lithops?style=for-the-badge&labelColor=000000"></a>&nbsp<a href="https://pypistats.org/packages/lithops"><img alt="PyPI - Downloads" src="https://img.shields.io/pypi/dm/lithops?label=pypi%7Cdownloads&style=for-the-badge&labelColor=000000"></a>&nbsp<a href="https://deepwiki.com/lithops-cloud/lithops" target="_blank" rel="noopener"><img alt="Ask DeepWiki" src="https://img.shields.io/badge/DeepWiki-Ask%20DeepWiki-blue.svg?style=for-the-badge&labelColor=000000&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACwAAAAyCAYAAAAnWDnqAAAAAXNSR0IArs4c6QAAA05JREFUaEPtmUtyEzEQhtWTQyQLHNak2AB7ZnyXZMEjXMGeK/AIi+QuHrMnbChYY7MIh8g01fJoopFb0uhhEqqcbWTp06/uv1saEDv4O3n3dV60RfP947Mm9/SQc0ICFQgzfc4CYZoTPAswgSJCCUJUnAAoRHOAUOcATwbmVLWdGoH//PB8mnKqScAhsD0kYP3j/Yt5LPQe2KvcXmGvRHcDnpxfL2zOYJ1mFwrryWTz0advv1Ut4CJgf5uhDuDj5eUcAUoahrdY/56ebRWeraTjMt/00Sh3UDtjgHtQNHwcRGOC98BJEAEymycmYcWwOprTgcB6VZ5JK5TAJ+fXGLBm3FDAmn6oPPjR4rKCAoJCal2eAiQp2x0vxTPB3ALO2CRkwmDy5WohzBDwSEFKRwPbknEggCPB/imwrycgxX2NzoMCHhPkDwqYMr9tRcP5qNrMZHkVnOjRMWwLCcr8ohBVb1OMjxLwGCvjTikrsBOiA6fNyCrm8V1rP93iVPpwaE+gO0SsWmPiXB+jikdf6SizrT5qKasx5j8ABbHpFTx+vFXp9EnYQmLx02h1QTTrl6eDqxLnGjporxl3NL3agEvXdT0WmEost648sQOYAeJS9Q7bfUVoMGnjo4AZdUMQku50McDcMWcBPvr0SzbTAFDfvJqwLzgxwATnCgnp4wDl6Aa+Ax283gghmj+vj7feE2KBBRMW3FzOpLOADl0Isb5587h/U4gGvkt5v60Z1VLG8BhYjbzRwyQZemwAd6cCR5/XFWLYZRIMpX39AR0tjaGGiGzLVyhse5C9RKC6ai42ppWPKiBagOvaYk8lO7DajerabOZP46Lby5wKjw1HCRx7p9sVMOWGzb/vA1hwiWc6jm3MvQDTogQkiqIhJV0nBQBTU+3okKCFDy9WwferkHjtxib7t3xIUQtHxnIwtx4mpg26/HfwVNVDb4oI9RHmx5WGelRVlrtiw43zboCLaxv46AZeB3IlTkwouebTr1y2NjSpHz68WNFjHvupy3q8TFn3Hos2IAk4Ju5dCo8B3wP7VPr/FGaKiG+T+v+TQqIrOqMTL1VdWV1DdmcbO8KXBz6esmYWYKPwDL5b5FA1a0hwapHiom0r/cKaoqr+27/XcrS5UwSMbQAAAABJRU5ErkJggg==" style="vertical-align:middle;"></a>
<a aria-label="License" href="https://github.com/lithops-cloud/lithops/blob/master/LICENSE"><img alt="" src="https://img.shields.io/github/license/lithops-cloud/lithops?style=for-the-badge&labelColor=000000"></a>&nbsp<a aria-label="PyPI" href="https://pypi.org/project/lithops/"><img alt="" src="https://img.shields.io/pypi/v/lithops?style=for-the-badge&labelColor=000000"></a>&nbsp<a aria-label="Python" href="#lithops"><img alt="" src="https://img.shields.io/pypi/pyversions/lithops?style=for-the-badge&labelColor=000000"></a>&nbsp<a href="https://pypistats.org/packages/lithops"><img alt="PyPI - Downloads" src="https://img.shields.io/pypi/dm/lithops?label=pypi%7Cdownloads&style=for-the-badge&labelColor=000000"></a>&nbsp<a href="https://deepwiki.com/lithops-cloud/lithops" target="_blank" rel="noopener"><img alt="Ask DeepWiki" src="https://img.shields.io/badge/DeepWiki-Ask%20DeepWiki-blue.svg?style=for-the-badge&labelColor=000000&logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACwAAAAyCAYAAAAnWDnqAAAAAXNSR0IArs4c6QAAA05JREFUaEPtmUtyEzEQhtWTQyQLHNak2AB7ZnyXZMEjXMGeK/AIi+QuHrMnbChYY7MIh8g01fJoopFb0uhhEqqcbWTp06/uv1saEDv4O3n3dV60RfP947Mm9/SQc0ICFQgzfc4CYZoTPAswgSJCCUJUnAAoRHOAUOcATwbmVLWdGoH//PB8mnKqScAhsD0kYP3j/Yt5LPQe2KvcXmGvRHcDnpxfL2zOYJ1mFwrryWTz0advv1Ut4CJgf5uhDuDj5eUcAUoahrdY/56ebRWeraTjMt/00Sh3UDtjgHtQNHwcRGOC98BJEAEymycmYcWwOprTgcB6VZ5JK5TAJ+fXGLBm3FDAmn6oPPjR4rKCAoJCal2eAiQp2x0vxTPB3ALO2CRkwmDy5WohzBDwSEFKRwPbknEggCPB/imwrycgxX2NzoMCHhPkDwqYMr9tRcP5qNrMZHkVnOjRMWwLCcr8ohBVb1OMjxLwGCvjTikrsBOiA6fNyCrm8V1rP93iVPpwaE+gO0SsWmPiXB+jikdf6SizrT5qKasx5j8ABbHpFTx+vFXp9EnYQmLx02h1QTTrl6eDqxLnGjporxl3NL3agEvXdT0WmEost648sQOYAeJS9Q7bfUVoMGnjo4AZdUMQku50McDcMWcBPvr0SzbTAFDfvJqwLzgxwATnCgnp4wDl6Aa+Ax283gghmj+vj7feE2KBBRMW3FzOpLOADl0Isb5587h/U4gGvkt5v60Z1VLG8BhYjbzRwyQZemwAd6cCR5/XFWLYZRIMpX39AR0tjaGGiGzLVyhse5C9RKC6ai42ppWPKiBagOvaYk8lO7DajerabOZP46Lby5wKjw1HCRx7p9sVMOWGzb/vA1hwiWc6jm3MvQDTogQkiqIhJV0nBQBTU+3okKCFDy9WwferkHjtxib7t3xIUQtHxnIwtx4mpg26/HfwVNVDb4oI9RHmx5WGelRVlrtiw43zboCLaxv46AZeB3IlTkwouebTr1y2NjSpHz68WNFjHvupy3q8TFn3Hos2IAk4Ju5dCo8B3wP7VPr/FGaKiG+T+v+TQqIrOqMTL1VdWV1DdmcbO8KXBz6esmYWYKPwDL5b5FA1a0hwapHiom0r/cKaoqr+27/XcrS5UwSMbQAAAABJRU5ErkJggg==" style="vertical-align:middle;"></a>
</p>

Lithops is a Python multi-cloud distributed computing framework that lets you run unmodified Python code at massive scale across cloud, HPC, and on-premise platforms. It supports major cloud providers and Kubernetes platforms, running your code transparently without requiring you to manage deployment or infrastructure.
Expand All @@ -14,7 +14,7 @@ Lithops is ideal for highly parallel workloads—such as Monte Carlo simulations

## Installation

1. Install Lithops from the PyPi repository:
1. Install Lithops from the PyPI repository:

```bash
pip install lithops
Expand All @@ -40,7 +40,7 @@ Lithops provides an extensible backend architecture (compute, storage) designed

## High-level API

Lithops is shipped with 2 different high-level Compute APIs, and 2 high-level Storage APIs
Lithops ships with 2 different high-level Compute APIs, and 2 high-level Storage APIs.

<div align="center">
<table>
Expand Down
14 changes: 7 additions & 7 deletions config/README.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,27 @@
# Lithops configuration

By default Lithops works on Localhost if no configuration is provided. To run workloads on the Cloud, you must configure both a compute and a storage backend. Failing to configure them properly will prevent Lithops to submit workloads. Lithops configuration can be provided either in a configuration file or in runtime via a Python dictionary.
By default Lithops works on Localhost if no configuration is provided. To run workloads on the Cloud, you must configure both a compute and a storage backend. Failing to configure them properly will prevent Lithops from submitting workloads. Lithops configuration can be provided either in a configuration file or at runtime via a Python dictionary.

### Configuration file

To configure Lithops through a [configuration file](config_template.yaml) you have multiple options:

1. Create a new file called `config` in the `~/.lithops` folder (i.e: `~/.lithops/config`).
1. Create a new file called `config` in the `~/.lithops` folder (i.e., `~/.lithops/config`).

2. Create a new file called `.lithops_config` in the root directory of your project from where you will execute your Lithops scripts.

3. Create a new file called `config` in the `/etc/lithops/` folder (i.e: `/etc/lithops/config`). Useful for sharing the config file on multi-user machines.
3. Create a new file called `config` in the `/etc/lithops/` folder (i.e., `/etc/lithops/config`). Useful for sharing the config file on multi-user machines.

4. Create the config file in any other location and configure the `LITHOPS_CONFIG_FILE` system environment variable:

LITHOPS_CONFIG_FILE=<CONFIG_FILE_LOCATION>

### Configuration keys in runtime

An alternative mode of configuration is to use a python dictionary. This option allows to pass all the configuration details as part of the Lithops invocation in runtime. An entire list of sections and keys is [here](config_template.yaml)
An alternative mode of configuration is to use a Python dictionary. This option allows you to pass all the configuration details as part of the Lithops invocation at runtime. The full list of sections and keys is [here](config_template.yaml)

## Compute and Storage backends
Choose your compute and storage engines from the table below
Choose your compute and storage backends from the table below

<table>
<tr>
Expand Down Expand Up @@ -49,7 +49,7 @@ Storage Backends

<b>Serverless (FaaS) Backends:</b>
- [AWS Lambda](../docs/source/compute_config/aws_lambda.md)
- [Google Cloud Functions](../docs/source/compute_config/gcp_functions.md)
- [Google Cloud Run functions](../docs/source/compute_config/gcp_functions.md)
- [Azure Functions](../docs/source/compute_config/azure_functions.md)
- [Aliyun Functions](../docs/source/compute_config/aliyun_functions.md)
- [Oracle Cloud Functions](../docs/source/compute_config/oracle_functions.md)
Expand All @@ -59,7 +59,7 @@ Storage Backends
- [IBM Code Engine](../docs/source/compute_config/code_engine.md)
- [AWS Batch](../docs/source/compute_config/aws_batch.md)
- [Google Cloud Run](../docs/source/compute_config/gcp_cloudrun.md)
- [Azure Container APPs](../docs/source/compute_config/azure_containers.md)
- [Azure Container Apps](../docs/source/compute_config/azure_containers.md)
- [Kubernetes](../docs/source/compute_config/kubernetes.md)
- [Knative](../docs/source/compute_config/knative.md)
- [Singularity](../docs/source/compute_config/singularity.md)
Expand Down
2 changes: 1 addition & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Object Storage Made Easy
Lithops simplifies working with data lakes and object storage by providing:

- 🔍 **Automatic data discovery**: Detects and lists files across nested directories.
- 📂 **Transparent data partitioning**: Splits large files (e.g., CSV, Parquet, JSON) into chunks for efficient parallel processing.
- 📂 **Transparent data partitioning**: Splits large text-based files (for example, CSV) into chunks for efficient parallel processing.
- 🧰 **Unified, Pythonic API**: Interact with your data using a single interface, regardless of where it's stored.

You write simple Python code — Lithops handles the complexity of parallel I/O, data distribution, and storage backends under the hood.
Expand Down
6 changes: 3 additions & 3 deletions docs/source/airflow.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ The Airflow/Lithops integration allows Airflow users to keep all of their Lithop
define task dependencies by moving data through Python functions.


Refer to the `integration repository <https://github.com/lithops-cloud/airflow-plugin>`_ .
Refer to the `integration repository <https://github.com/lithops-cloud/airflow-plugin>`_.

Examples
--------
Expand All @@ -18,7 +18,7 @@ Define a function in a separate file (``my_functions.py``):
.. code:: python

def add(x, y):
return x + y
return x + y


Import the Lithops operator and the function, and create the DAG to execute:
Expand Down Expand Up @@ -48,7 +48,7 @@ Import the Lithops operator and the function, and create the DAG to execute:

mult_num_map = LithopsMapOperator(
task_id='mult_num_map',
map_function=example_functions.add_num,
map_function=add,
iterdata_from_task={'a': 'gen_list'},
extra_args={'b': 10},
dag=dag
Expand Down
8 changes: 4 additions & 4 deletions docs/source/api_stats.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Lithops provides built-in internal tools to supply the user with technical stati
Execution summary plots
-----------------------

The :code:`plot()` method from :code:`FunctionExecutor` creates a scatter plot and a histogram plot showing a summary of the tasks executed by a :code:`FunctionExecutor`. By default, lithops creates a :code:`plots/` directory in the working directory path containing both plots in PNG format. For more details refer to the `FunctionExecutor API reference <https://lithops-cloud.github.io/docs/source/api_futures.html>`_.
The :code:`plot()` method from :code:`FunctionExecutor` creates a scatter plot and a histogram plot showing a summary of the tasks executed by a :code:`FunctionExecutor`. By default, Lithops creates a :code:`plots/` directory in the working directory path containing both plots in PNG format. For more details, refer to the `FunctionExecutor API reference <https://lithops-cloud.github.io/docs/source/api_futures.html>`_.

To get started, first install Lithops and the plotting dependencies with:

Expand Down Expand Up @@ -115,7 +115,7 @@ The user can obtain these statistics through the future object:
* - Stat
- Description
* - :code:`func_data_size_bytes`
- Size in bytes of the input data processed by this function. That is, the object size of the input list item processed by this function. Note that if the function processes data obtained from object storage, this value does not include the size of that data, only those that have been serialized and sent from the host process to the function.
- Size in bytes of the input data processed by this function. That is, the object size of the input list item processed by this function. Note that if the function processes data obtained from object storage, this value does not include the size of that dataonly the data serialized and sent from the host process to the function.
* - :code:`func_module_size_bytes`
- Size in bytes of the dependencies (function and modules) serialized and uploaded by the host process.
* - :code:`func_result_size`
Expand All @@ -131,7 +131,7 @@ The user can obtain these statistics through the future object:
* - :code:`host_job_serialize_time`
- Total time taken by the host process to serialize the input data and dependencies (functions and modules).
* - :code:`host_result_done_tstamp`
- Timestamp of when host received the function result from cloud object storage.
- Timestamp of when the host received the function result from cloud object storage.
* - :code:`host_result_query_count`
- Number of queries to the object storage to get the result object.
* - :code:`host_status_done_tstamp`
Expand All @@ -143,7 +143,7 @@ The user can obtain these statistics through the future object:
* - :code:`worker_cold_start`
- Indicates whether it was a "warm" invocation (the container was already running) or "cold" (the container had to be deployed).
* - :code:`worker_end_tstamp`
- Timestamp in which the worker function had finished its execution.
- Timestamp when the worker function finished its execution.
* - :code:`worker_exec_time`
- Total execution time of the worker function (Lithops wrapper + user-defined function execution time).
* - :code:`worker_func_cpu_usage`
Expand Down
2 changes: 1 addition & 1 deletion docs/source/api_storage_os.rst
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Cloud Proxy Storage API

Similar to Python's built-in function `open() <https://docs.python.org/3/library/functions.html#open>`__.

Manipulate an object stored in Cloud Object Storage.
Manipulate an object storaged in Cloud Object Storage.

+-------------+-------------------------------------------------------------------------------------------------------------------+
| Parameter | Description |
Expand Down
Loading
Loading