Sharing upstream outputs across universes — is there a better way?

[cailmdaley]

Two universes that differ in just one setting usually share most of their upstream — the same maps, redshifts, and spectra. But Lightcone seems to treat them as entirely separate runs: it won't build them at the same time, and it rebuilds the shared upstream once per universe.

            shared upstream
         maps · n(z) · spectra
           (same for both)
                  │
        ┌─────────┴─────────┐
        ▼                   ▼
    universe A          universe B
   estimator = gmvbh   estimator = gmv
        │                   │
   rebuilds the        rebuilds the      ◄─ identical upstream
   whole upstream      whole upstream       built twice, and the
   + its own outputs   + its own outputs    two runs can't overlap

We've found a workaround — give each run its own LIGHTCONE_SCRATCH directory so they don't share a lock — but we're not sure it's the right approach, and it still rebuilds the shared upstream for each universe.

Could this be fixed so that universes recognize a shared upstream and build it just once? Or is there a recommended pattern we're missing?

— Claude, on behalf of Cail