MetaDetect Ingester for DP2#469
Conversation
|
It currently assumes 'ns' rather than '00' so we might need to merge that change at the same time |
|
so in the Schema for MetaDetect there are 2 different |
|
Good question. I guess for computing T / Tpsf we need the re-convolved one since the T will be re-convolved as well? But for null tests we probably want the other one. I guess this is why both were in there before. That's annoying. Could you ingest the non-reconvolved one as "original" or something like that? |
| } | ||
|
|
||
|
|
||
| class TXIngestMetaDetect(PipelineStage): |
There was a problem hiding this comment.
Can you add DP1 to the name of this class to distinguish from e.g. DESY6.
Also since all of this recaps some stuff that is in dp1.py, is there a reason it's not in there, or deleted from there?
There was a problem hiding this comment.
It is meant for DP2, but can add that to it (it should work on DP1 as well because they look the same)
| "ra": var_data["ra"], | ||
| "dec": var_data["dec"], | ||
| "id": var_data["shearObjectId"], | ||
| "metaStep": var_data["metaStep"].astype("S"), #might not be needed |
There was a problem hiding this comment.
Should all be the same I think?
There was a problem hiding this comment.
Sorry what do you mean?
There was a problem hiding this comment.
You wrote that it might not be needed, and I think that's right because this whole data set should have the same metaStep value because that's how they are selected. So no need to save it.
| "psf_g1": var_data["gauss_psfReconvolved_g1"], | ||
| "psf_g2": var_data["gauss_psfReconvolved_g1"], | ||
| "psf_T_mean": var_data["gauss_psfReconvolved_T"], | ||
| "flags": var_data["gauss_shape_flags"], # TO BE ADDRESSED! |
There was a problem hiding this comment.
Can you explain the comment more?
There was a problem hiding this comment.
Right, I wasn't sure if that was the only flag we needed, or if we need to add a bunch of the flags instead.
| if not created_files: | ||
| created_files = True | ||
| variants = { | ||
| "ns": len(shear_data["ns"]), |
There was a problem hiding this comment.
Let's leave this as "ns" and change the rest of the code before merging since we have to do it anyway.
There was a problem hiding this comment.
How do I/we update the sample catalog that the CI uses to reflect this change?
There was a problem hiding this comment.
Yes, will need to do a whole replacement process. A bit fiddly.
| # replace masked values with nans | ||
| if np.ma.isMaskedArray(col): | ||
| col = col.filled(np.nan) | ||
| k[name].append(col) #NOT SURE THIS WORKS EITHER TBD |
There was a problem hiding this comment.
Have you checked this as the comment says? Delete comment if so.
…into MetaDetect_ingest
2 participants
This is the ingestion stage for the Rubin MetaDetect catalog as we expect it to look for DP2