Resarch how Celery could be used

[jachym]

Seems, that Celery http://www.celeryproject.org/ might be of our scope.

Yes @jorgejesus already mentioned it

[cehbrecht]

just something a stumbled upon ... the multiproccessing module of pathos:

https://github.com/uqfoundation/pathos

I haven't used it by myself.

[tomLandry]

Hello Carsten! We are also looking closely at this. I guess we'll meet again!

[cehbrecht]

Just referencing this work with a pywps/celery demo:
https://github.com/fderue/pywps_celery_demo

This work was based on the scheduler extension included in pywps:
https://pywps.readthedocs.io/en/master/extensions.html#job-scheduler-extension

There was also a question on the pywps-dev mailing list concerning celery.

[tomLandry]

This implementation is described in OGC Testbed-13: Cloud Engineering Report. As you said, @cehbrecht , this was based on the sceduler extension.

I would also like to submit work done for the ESGF Compute Working Team API. While this implementation is not based on sceduler extension, it would probably be more sustainable going forward. I haven't compare architectures yet, but we will need to consider this also for ESGF Grand Compute Challenge.

[sbar13]

I would just like to comment on the pywps_celery_demo you describe above. https://github.com/fderue/pywps_celery_demo
I managed to get this simple demo to work in the docker container, and following its example, tried to implement it in my project using python 3.6. It failed to serialise the job. I then tried to reproduce this using python 2.7, and it also failed. Having a closer look, I found that the pywps source in the docker image does not contain the templates folder, or the response folder. The pywps source cloned for the demo is 225 commits behind master.

I have also read the http://docs.opengeospatial.org/per/17-035.html OGC Testbed-13 report, and do not believe that the celery implementation described in the report is the same, the report states that CRIM (Computer research Institute of Montreal) implemented a novel job scheduler - this works with the Birdhouse framework, which uses pywps but is current with master. So hopefully this opensource project will be useful but is currently closed to the public as it is under heavy development.

[tomLandry]

Hello @sbar13, thanks for your input! I work at CRIM and I am its OGC representative. Let me confirm that https://github.com/fderue/pywps_celery_demo is the latest version we have at hand.

[sbar13]

Hi @tomLandry,

Thank you for the information, that clears things up for me. So your celery implementation is based on the celery_demo, which is behind the PyWPS master branch. Do you have a working celery implementation based on the latest source or will you be updating the PyWPS source during this stage of development ?