Use a generator on instance catalog read in to throttle memory usage.

[cwwalter]

If we used muti-processing parallel rendering of objects rather than rendering each sensor in parallel we would be able to use less shared memory and should be able to use more of the cores on the KNL node for simulation. This could increase the throughput by factors of 2-4.

GalSim has infrastructure to do this we could call.

[cwwalter]

This involves refactoring the code and we should consider how we call the GalSim API and whether some could be done directly though imSim itself. Currently much is split between imSim and the GalSim_interface code.

[cwwalter]

After discussion, we have decided to try a less radical approach to limiting the memory usage when running on multi-core systems like at NERSC in the current imSim version.

Instead of changing the parallelization to be object based in this interim version, thus reducing the amount of sky we need to hold in memory on each node, we will try using a python generator on read in. This way we can fix the amount of input being held in memory and the instance catalogs should effectively be read in chunks.

This will still increase throughput since we should be able to utilize more cores at once and process more sensors in parallel on each node, although the per sensor run time will now stay the same.

This is a less drastic and much more contained change. So, we think we can implement and confidently test it on a shorter time scale which is better matched to the DC2 re-run schedule we were requested to make this change for.

Note: this change has potential other benefits for even single core usage (like on the OSG) since the total memory usage can be reduced for the running job.

[cwwalter]

This was implemented with the merger of #227.