Error when using Minio and Openwhisk as storage and compute backends.

[faraz-ahmd]

Hi,

I am trying to use Lithops with Minio as the storage backend and Openwhisk as the compute backend. I can create/delete/list aliases and buckets using mcli. I can also invoke actions using openwhisk-cli wsk. However, when running a lithops helloworld example I get an error. Is there something I am missing in the configuration or something else?

This is my example script:

import lithops
config = {'lithops': {'backend': 'openwhisk', 'storage': 'minio'},

          'openwhisk':  {'endpoint': 'https://172.17.0.3:31001',
                          'api_key': '23bc46b1-71f6-4ed5-8c54-816aa4f8c502:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP',
                        'namespace': 'guest',
                         'insecure': 'True',
                          'runtime': 'python'},

          'minio': {'storage_bucket': 'incoming',
                          'endpoint': 'http://172.17.0.4:31113',
                     'access_key_id': 'minio',
                 'secret_access_key': 'minio123'}}

def hello_world(name):
    return 'Hello {}!'.format(name)

if __name__ == '__main__':
    fexec = lithops.FunctionExecutor(config=config)
    fexec.call_async(hello_world, 'World')
    print(fexec.get_result())

Here is the output:

2022-03-03 00:53:47,698 [INFO] lithops.config -- Lithops v2.5.8
2022-03-03 00:53:47,845 [INFO] lithops.storage.backends.minio.minio -- MinIO client created - Endpoint: http://172.17.0.4:31113
2022-03-03 00:53:47,845 [INFO] lithops.serverless.backends.openwhisk.openwhisk -- OpenWhisk client created - Namespace: guest
2022-03-03 00:53:47,846 [INFO] lithops.invokers -- ExecutorID d9372b-0 | JobID A000 - Selected Runtime: python - 256MB
        Traceback (most recent call last):
  File "/home/fedora/lithops/hello-wsk-minio.py", line 23, in <module>
    fexec.call_async(hello_world, 'World')
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/executors.py", line 205, in call_async
    runtime_meta = self.invoker.select_runtime(job_id, runtime_memory)
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/invokers.py", line 115, in select_runtime
    runtime_meta = self.internal_storage.get_runtime_meta(runtime_key)
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/storage/storage.py", line 402, in get_runtime_meta
    json_str = self.storage.get_object(self.bucket, obj_key)
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/storage/storage.py", line 111, in get_object
    return self.storage_handler.get_object(bucket, key, stream, extra_get_args)
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/storage/backends/minio/minio.py", line 118, in get_object
    raise e
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/storage/backends/minio/minio.py", line 109, in get_object
    r = self.cos_client.get_object(Bucket=bucket_name, Key=key, **extra_get_args)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
ibm_botocore.exceptions.ClientError: An error occurred (503) when calling the GetObject operation (reached max retries: 5): Service Unavailable

I have also tried the following example but get the same error:

"""
Simple Lithops example using the 'Storage' interface
"""
from lithops import FunctionExecutor, Storage

BUCKET_NAME = 'incoming'  # change-me
config = {'lithops': {'backend': 'openwhisk', 'storage': 'minio'},

          'openwhisk':  {'endpoint': 'https://172.17.0.3:31001',
                          'api_key': '23bc46b1-71f6-4ed5-8c54-816aa4f8c502:123zO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP',
                        'namespace': 'guest',
                         'insecure': 'True',
                          'runtime': 'python'},

          'minio': {'storage_bucket': 'incoming',
                          'endpoint': 'http://172.17.0.4:31113',
                     'access_key_id': 'minio',
                 'secret_access_key': 'minio123'}}

def my_function(obj_id, storage):
    print(obj_id)

    data = storage.get_cloudobject(obj_id)

    return data.decode()


if __name__ == '__main__':

    obj_key = 'cloudobject1.txt'
    storage = Storage()
    obj_id = storage.put_cloudobject('Hello World!', BUCKET_NAME, obj_key)
    print(obj_id)

#    fexec = FunctionExecutor()
    fexec = FunctionExecutor(config=config)
    fexec.call_async(my_function, obj_id)
    print(fexec.get_result())

Thanks!

[gilv]

may be @JosepSampe knows right away, but i would ask to see logs in DEBUG mode...can you add lithops.FunctionExecutor(log_level='DEBUG') and re-run?

[faraz-ahmd]

Thanks a lot for the quick response. I have tried running the example multiple times with the DEBUG mode. I got two different error logs for the multiple runs. After looking at the logs, I want to add that both Minio and Openwhisk are running on a Kubernetes cluster behind a proxy. Lithops, mcli, and wsk are in the host environment of the Kubernetes master node. Here is the first type of log with the DEBUG mode:

2022-03-03 18:33:14,459 [INFO] lithops.config -- Lithops v2.5.8
2022-03-03 18:33:14,459 [DEBUG] lithops.config -- Loading Serverless backend module: openwhisk
2022-03-03 18:33:14,461 [DEBUG] lithops.config -- Loading Storage backend module: minio
2022-03-03 18:33:14,527 [DEBUG] lithops.storage.backends.minio.minio -- Creating MinIO client
2022-03-03 18:33:14,527 [DEBUG] lithops.storage.backends.minio.minio -- Setting MinIO endpoint to http://172.17.0.4:31113
2022-03-03 18:33:14,577 [INFO] lithops.storage.backends.minio.minio -- MinIO client created - Endpoint: http://172.17.0.4:31113
2022-03-03 18:33:14,577 [DEBUG] lithops.serverless.backends.openwhisk.openwhisk -- Creating OpenWhisk client
2022-03-03 18:33:14,577 [DEBUG] lithops.serverless.backends.openwhisk.openwhisk -- Set OpenWhisk Endpoint to https://172.17.0.3:31001
2022-03-03 18:33:14,577 [DEBUG] lithops.serverless.backends.openwhisk.openwhisk -- Set OpenWhisk Namespace to guest
2022-03-03 18:33:14,577 [DEBUG] lithops.serverless.backends.openwhisk.openwhisk -- Set OpenWhisk Insecure to True
2022-03-03 18:33:14,578 [INFO] lithops.serverless.backends.openwhisk.openwhisk -- OpenWhisk client created - Namespace: guest
2022-03-03 18:33:14,578 [DEBUG] lithops.invokers -- ExecutorID ad7f8f-0 - Invoker initialized. Max workers: 100
2022-03-03 18:33:14,578 [DEBUG] lithops.invokers -- ExecutorID ad7f8f-0 - Serverless invoker created
2022-03-03 18:33:14,578 [DEBUG] lithops.executors -- Function executor for openwhisk created with ID: ad7f8f-0
2022-03-03 18:33:14,578 [INFO] lithops.invokers -- ExecutorID ad7f8f-0 | JobID A000 - Selected Runtime: python - 256MB
2022-03-03 18:33:14,578 [DEBUG] lithops.storage.storage -- Runtime metadata not found in local cache. Retrieving it from storage
2022-03-03 18:33:14,578 [DEBUG] lithops.storage.storage -- Trying to download runtime metadata from: minio://incoming/lithops.runtimes/2.5.8/openwhisk/guest/python_256MB.meta.json
Traceback (most recent call last):
  File "/home/fedora/lithops/hello-wsk-minio.py", line 23, in <module>
    fexec.call_async(hello_world, 'World')
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/executors.py", line 205, in call_async
    runtime_meta = self.invoker.select_runtime(job_id, runtime_memory)
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/invokers.py", line 115, in select_runtime
    runtime_meta = self.internal_storage.get_runtime_meta(runtime_key)
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/storage/storage.py", line 402, in get_runtime_meta
    json_str = self.storage.get_object(self.bucket, obj_key)
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/storage/storage.py", line 111, in get_object
    return self.storage_handler.get_object(bucket, key, stream, extra_get_args)
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/storage/backends/minio/minio.py", line 118, in get_object
    raise e
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/storage/backends/minio/minio.py", line 109, in get_object
    r = self.cos_client.get_object(Bucket=bucket_name, Key=key, **extra_get_args)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
ibm_botocore.exceptions.ClientError: An error occurred (503) when calling the GetObject operation (reached max retries: 5): Service Unavailable

Here is the second type of error log:

2022-03-03 18:46:12,334 [INFO] lithops.config -- Lithops v2.5.8
2022-03-03 18:46:12,334 [DEBUG] lithops.config -- Loading Serverless backend module: openwhisk
2022-03-03 18:46:12,336 [DEBUG] lithops.config -- Loading Storage backend module: minio
2022-03-03 18:46:12,408 [DEBUG] lithops.storage.backends.minio.minio -- Creating MinIO client
2022-03-03 18:46:12,408 [DEBUG] lithops.storage.backends.minio.minio -- Setting MinIO endpoint to http://172.17.0.4:31113
2022-03-03 18:46:12,468 [INFO] lithops.storage.backends.minio.minio -- MinIO client created - Endpoint: http://172.17.0.4:31113
2022-03-03 18:46:12,468 [DEBUG] lithops.serverless.backends.openwhisk.openwhisk -- Creating OpenWhisk client
2022-03-03 18:46:12,469 [DEBUG] lithops.serverless.backends.openwhisk.openwhisk -- Set OpenWhisk Endpoint to https://172.17.0.3:31001
2022-03-03 18:46:12,469 [DEBUG] lithops.serverless.backends.openwhisk.openwhisk -- Set OpenWhisk Namespace to guest
2022-03-03 18:46:12,469 [DEBUG] lithops.serverless.backends.openwhisk.openwhisk -- Set OpenWhisk Insecure to True
2022-03-03 18:46:12,469 [INFO] lithops.serverless.backends.openwhisk.openwhisk -- OpenWhisk client created - Namespace: guest
2022-03-03 18:46:12,469 [DEBUG] lithops.invokers -- ExecutorID 0a90da-0 - Invoker initialized. Max workers: 100
2022-03-03 18:46:12,469 [DEBUG] lithops.invokers -- ExecutorID 0a90da-0 - Serverless invoker created
2022-03-03 18:46:12,469 [DEBUG] lithops.executors -- Function executor for openwhisk created with ID: 0a90da-0
2022-03-03 18:46:12,469 [INFO] lithops.invokers -- ExecutorID 0a90da-0 | JobID A000 - Selected Runtime: python - 256MB
2022-03-03 18:46:12,470 [DEBUG] lithops.storage.storage -- Runtime metadata not found in local cache. Retrieving it from storage
2022-03-03 18:46:12,470 [DEBUG] lithops.storage.storage -- Trying to download runtime metadata from: minio://incoming/lithops.runtimes/2.5.8/openwhisk/guest/python_256MB.meta.json
2022-03-03 18:47:26,764 [DEBUG] lithops.storage.backends.minio.minio -- GET Object timeout. Retrying request
2022-03-03 18:48:24,904 [DEBUG] lithops.storage.backends.minio.minio -- GET Object timeout. Retrying request
2022-03-03 18:49:37,032 [DEBUG] lithops.storage.backends.minio.minio -- GET Object timeout. Retrying request
2022-03-03 18:50:46,860 [DEBUG] lithops.storage.backends.minio.minio -- GET Object timeout. Retrying request
2022-03-03 18:51:55,600 [DEBUG] lithops.storage.backends.minio.minio -- GET Object timeout. Retrying request
Traceback (most recent call last):
  File "/home/fedora/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 449, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/fedora/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 444, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib64/python3.9/http/client.py", line 1371, in getresponse
    response.begin()
  File "/usr/lib64/python3.9/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python3.9/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib64/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/httpsession.py", line 344, in send
    urllib_response = conn.urlopen(
  File "/home/fedora/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/home/fedora/.local/lib/python3.9/site-packages/urllib3/util/retry.py", line 525, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/fedora/.local/lib/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/home/fedora/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/fedora/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 451, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/fedora/.local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 340, in _raise_timeout
    raise ReadTimeoutError(
**urllib3.exceptions.ReadTimeoutError: AWSHTTPConnectionPool(host='my-proxy.net', port=8080): Read timed out. (read timeout=10)**

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fedora/lithops/hello-wsk-minio.py", line 23, in <module>
    fexec.call_async(hello_world, 'World')
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/executors.py", line 205, in call_async
    runtime_meta = self.invoker.select_runtime(job_id, runtime_memory)
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/invokers.py", line 115, in select_runtime
    runtime_meta = self.internal_storage.get_runtime_meta(runtime_key)
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/storage/storage.py", line 402, in get_runtime_meta
    json_str = self.storage.get_object(self.bucket, obj_key)
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/storage/storage.py", line 111, in get_object
    return self.storage_handler.get_object(bucket, key, stream, extra_get_args)
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/storage/backends/minio/minio.py", line 121, in get_object
    raise e
  File "/home/fedora/.local/lib/python3.9/site-packages/lithops/storage/backends/minio/minio.py", line 109, in get_object
    r = self.cos_client.get_object(Bucket=bucket_name, Key=key, **extra_get_args)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/client.py", line 691, in _make_api_call
    http, parsed_response = self._make_request(
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/client.py", line 711, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/endpoint.py", line 136, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/endpoint.py", line 253, in _needs_retry
    responses = self._event_emitter.emit(
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/hooks.py", line 357, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "/home/fedora/.local/lib/python3.9/site-packages/ibm_botocore/httpsession.py", line 379, in send
    raise ReadTimeoutError(endpoint_url=request.url, error=e)
ibm_botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "http://172.17.0.4:31113/incoming/lithops.runtimes/2.5.8/openwhisk/guest/python_256MB.meta.json"

Since I am running openwhisk on kubernetes, are there additional steps like for Knative provided here ?

[JosepSampe]

@gilv In this case it seems the MinIO deployment is not reachable/accessible.

@faraz-ahmd Does the bucket 'incoming' exists in MinIO? Could you verify that your MinIO deployment is accessible from python and without lithops? for example using the next script:

import boto3

bucket= 'incoming'
access_key_id = 'minio'
secret_access_key = 'minio123'
endpoint = 'http://172.17.0.4:31113'


ceph_client = boto3.client('s3',
                           aws_access_key_id=access_key_id,
                           aws_secret_access_key=secret_access_key,
                           endpoint_url=endpoint)

try:
    ceph_client.create_bucket(Bucket=bucket)
except:
    pass


# Call S3 to list current buckets
response = ceph_client.list_buckets()

# Get a list of all bucket names from the response
buckets = [bucket['Name'] for bucket in response['Buckets']]

# Print out the bucket list
print("Bucket List: %s" % buckets)

[faraz-ahmd]

Thanks a lot @JosepSampe for sharing the script. The bucket incoming existed and I could put objects using the minio client mcli. The issue was that my setup is behind a proxy. I can now run the above script and example scripts with Minio and Openwhisk. I specified the proxies as suggested here. Also, set no_proxy with the exact ip address of the host that has minio container (since I am using nodeport service).

from botocore.config import Config

proxy_definitions = {
    'http': 'http://proxy.amazon.com:6502',
    'https': 'https://proxy.amazon.org:2010'
}

my_config = Config(
    region_name='us-east-2',
    signature_version='v4',
    proxies=proxy_definitions
)

client = boto3.client('kinesis', config=my_config)