[RFC/WIP] Asynchronous ParallelEvaluator

[alyst]

The PR changes the ParallelEvaluator to be asynchronous:

1. The candidate is submitted to the queue by async_update_fitness(), which immediately returns the fitness calculation job Id and the job gets submitted to one of the available worker processes.
2. The job status could be tested by the isready() function. Also, the candidates with the recently calculated fitness could be processed by calling process_completed() do job_id, candidate <custom code> end routine. (Note: if isready(job_id) is called and it returns true, this job would not be enumerated by a call to process_completed() anymore and it's up for the caller to take actions).

The old synchronous API (used by NES) is still supported.

BorgMOEA is updated to support asynchronous ParallelEvaluator: the algorithm runs on master, the new individual is generated by recombination and sent to the parallel evaluation, the further processing of the individual (updating the population and the archive) is postponed until its fitness is evaluated. This should improve the performance for the problems with computationally intensive fitness functions. I see the speed up for my problems, although it's not linear.

There are several reasons why it's RFC/WIP:

1. Unfortunately, the current julia parallelization scheme (remote channels/remote refs implemented using the messages sent through pipes) doesn't allow for an efficient intensive communication between the parallel processes (the profiling shows that the overhead of serializing/deserializing the message and writing/reading to UV pipe stream is too high), so after trying several implementations (this journey could be tracked as I haven't squashed the commits yet) I've ended up using shared arrays for the data and status exchange. Using threads (or maybe better messaging implementation) would be much more efficient, but it's not yet mainlined to Julia.
2. The logic of async_update_fitness()/isready()/process_completed() is somewhat confusing, but so far I had no better ideas given that Future{T} approach would create too much overhead.
3. As the communication is shmem-based, the workers use "busy wait" to continuously poll for the new jobs. To avoid 100% CPU usage, the workers have to be killed/suspended after the optimization is finished. This is done by the new shutdown!() call. However, OptController/OptRunController interface assumes the same optimization method (and its evaluator) could be reused in several OptRunController runs. With the ParallelEvaluator it is currently not possible, because all the workers are killed by the shutdown!() call at the end of the first run. I see two alternatives:
   • introduce start!(Evaluator)/shutdown!(Evaluator) methods that needs to be called immediately before/after method iterations (no-ops for the normal evaluators);
   • implement lazy initialization of the workers in ParallelEvaluator, detect idle periods, hybernate the workers when idle (using wait()), resume on the new fitness evaluation request. Given the state of the parallelism in julia it's not so easy to get it working nicely.

[coveralls]

Coverage increased (+3.9%) to 69.692% when pulling 24ea2a8 on alyst:async_parallel_eval into fbdd9ee on robertfeldt:master.

[coveralls]

Coverage increased (+4.1%) to 69.978% when pulling 17c2770 on alyst:async_parallel_eval into fbdd9ee on robertfeldt:master.

[coveralls]

Coverage increased (+4.1%) to 69.951% when pulling 17c2770 on alyst:async_parallel_eval into fbdd9ee on robertfeldt:master.

[coveralls]

Coverage increased (+4.3%) to 70.156% when pulling e55a8f4 on alyst:async_parallel_eval into bd0745c on robertfeldt:master.

[coveralls]

Coverage increased (+4.3%) to 70.156% when pulling e55a8f4 on alyst:async_parallel_eval into bd0745c on robertfeldt:master.

[coveralls]

Coverage increased (+4.4%) to 70.253% when pulling 88f9ff1 on alyst:async_parallel_eval into bd0745c on robertfeldt:master.

[coveralls]

Coverage increased (+3.6%) to 70.327% when pulling 3fd19d2 on alyst:async_parallel_eval into 4dd9ec2 on robertfeldt:master.

[coveralls]

Coverage increased (+3.1%) to 69.808% when pulling 9490577 on alyst:async_parallel_eval into 4dd9ec2 on robertfeldt:master.

[coveralls]

Coverage increased (+3.3%) to 71.416% when pulling b7b714d on alyst:async_parallel_eval into 572e647 on robertfeldt:master.

[coveralls]

Coverage increased (+2.7%) to 71.138% when pulling e9b2a91 on alyst:async_parallel_eval into 91440c1 on robertfeldt:master.

[coveralls]

Coverage decreased (-25.8%) to 69.371% when pulling 8225be4 on alyst:async_parallel_eval into cf37814 on robertfeldt:master.

[robertfeldt]

@alyst waht is the status of this branch now? In light of Julia 1.0 soon seems to be coming can we try to unify the different parallellization branches and ideas and merge with master?

[alyst]

I have a rebased version in my staging branch, I will update this one after #83 .

I'm using this branch and it works for me, with some caveats:

• sometimes during initialization I get ReadOnlyMemory() exceptions.
  Probably it's related to sharing of the arrays by different workers. Once the initialization of the workers succeeds, the optimization itself seems stable. The number of the workers may affect it.
• If the user interrupts julia (Ctrl+C) when the evaluator is in lock(), the whole julia crashes. Probably it could be fixed by wrapping lock() in try/catch at the expense of some performance loss.

I don't know how much 0.7 improves the situation with the workers. Maybe we can check this branch with 0.7alpha and merge it after making sure that Ctrl+C doesn't crash Julia so easily.