sis_hash_helper more efficient

[albertz]

This gives me some huge speedup, of a big GMM pipeline, from 12 sec runtime down to 2 sec runtime.

Edit Sorry, wrong numbers...

[albertz]

Hm ok I need to do some debugging, this seems to change the hash.

[JackTemaki]

Yes, my setup before the change:
error(7) runnable(5) waiting(649)

My setup after the change:
error(3) runnable(12) waiting(247)

[albertz]

Ok, similar fix as in #89. Now for my simple test I get the same hash.

[albertz]

However, the speedup is much less now. Need to do some benchmarks. I think it should still be faster than before though.

[albertz]

However, the speedup is much less now. Need to do some benchmarks. I think it should still be faster than before though.

Actually, no? It seems slightly slower. Maybe due to higher memory consumption? I need to investigate this a bit more...

[albertz]

However, the speedup is much less now. Need to do some benchmarks. I think it should still be faster than before though.

Actually, no? It seems slightly slower. Maybe due to higher memory consumption? I need to investigate this a bit more...

I'm also thinking about actually do more clever caching. But not exactly sure where. Unfortunately the objects we get here are often dict or list or some other config objects, and they are all mutable, so you cannot really cache the hash, unless you make sure the cache is correctly invalidated, which is not easily possible always (e.g. no idea how you would do that for a dict). Maybe in JobSingleton.__call__ we can do it for the args, because afterwards the args should anyway not change anymore.

[albertz]

Also, I think the current change in this PR here does not fully captures all the recursive calls of sis_hash_helper. The recursive calls are often through obj._sis_hash calls.

[critias]

Also, I think the current change in this PR here does not fully captures all the recursive calls of sis_hash_helper. The recursive calls are often through obj._sis_hash calls.

You could also cache the result of obj._sis_hash