`IntegrityError` When Loading SEG File from NSCLC-Radiogenomics

[JoshuaSiraj]

I get the following IntegrityError, when trying to load the following SEG from the NSCLC-Radiogenomics dataset. Error indicates that there are duplicate SOPInstanceUIDs. However, I went through the DICOM and found no duplicates. I've included the DICOM dump of the SEG aswell.

Version:
highdicom 0.26.0
pydicom 3.0.1

In [5]: ds = dcmread("/NSCLC-Radiogenomics/R01-003/SEG
      ⋮ _Series-96510/1-1.dcm")

In [6]: seg = hd.seg.Segmentation.from_dataset(ds)
---------------------------------------------------------------------------
IntegrityError                            Traceback (most recent call last)
Cell In[6], line 1
----> 1 seg = hd.seg.Segmentation.from_dataset(ds)

File ~/bhklab/josh/med-imagetools/.pixi/envs/dev/lib/python3.12/site-packages/highdicom/seg/sop.py:2233, in Segmentation.from_dataset(cls, dataset, copy)
   2227             pixel_measures = PixelMeasuresSequence.from_sequence(
   2228                 pffg_item.PixelMeasuresSequence,
   2229                 copy=False,
   2230             )
   2231             pffg_item.PixelMeasuresSequence = pixel_measures
-> 2233 seg = super().from_dataset(seg, copy=False)
   2235 return cast(Self, seg)

File ~/bhklab/josh/med-imagetools/.pixi/envs/dev/lib/python3.12/site-packages/highdicom/image.py:1176, in _Image.from_dataset(cls, dataset, copy)
   1173 im.__class__ = cls
   1174 im = cast(Self, im)
-> 1176 im._build_luts()
   1177 return im

File ~/bhklab/josh/med-imagetools/.pixi/envs/dev/lib/python3.12/site-packages/highdicom/image.py:3307, in _Image._build_luts(self)
   3302 self._coordinate_system = get_image_coordinate_system(
   3303     self
   3304 )
   3306 if is_multiframe_image(self):
-> 3307     self._build_luts_multiframe()
   3308 else:
   3309     self._build_luts_single_frame()

File ~/bhklab/josh/med-imagetools/.pixi/envs/dev/lib/python3.12/site-packages/highdicom/image.py:3708, in _Image._build_luts_multiframe(self)
   3704                     shared_pixel_spacing = None
   3706 self._db_con = sqlite3.connect(":memory:")
-> 3708 self._create_ref_instance_table(referenced_uids)
   3710 # Construct the columns and values to put into a frame look-up table
   3711 # table within sqlite. There will be one row per frame in the
   3712 # image
   3713 col_defs = []  # SQL column definitions

File ~/bhklab/josh/med-imagetools/.pixi/envs/dev/lib/python3.12/site-packages/highdicom/image.py:4185, in _Image._create_ref_instance_table(self, referenced_uids)
   4177 with self._db_con:
   4178     self._db_con.execute(
   4179         "CREATE TABLE InstanceUIDs("
   4180         "StudyInstanceUID VARCHAR NOT NULL, "
   (...)   4183         ")"
   4184     )
-> 4185     self._db_con.executemany(
   4186         "INSERT INTO InstanceUIDs "
   4187         "(StudyInstanceUID, SeriesInstanceUID, SOPInstanceUID) "
   4188         "VALUES(?, ?, ?)",
   4189         referenced_uids,
   4190     )

IntegrityError: UNIQUE constraint failed: InstanceUIDs.SOPInstanceUID

1-1.dcm.dcmdump.txt

[CPBridge]

Hi @JoshuaSiraj thanks for reporting this

I looked into it quickly and this isn't an issue in highdicom, it's an issue in the actual data. There are duplicated referenced SOP instance UIDs in the dataset. It's very confusing actually.

Each SOPInstanceUID in the ReferencedSeriesSequence is duplicated in the StudiesContainingOtherReferencedInstancesSequence. For example:

print(dcm.ReferencedSeriesSequence[0].ReferencedInstanceSequence[0].ReferencedSOPInstanceUID)
# 1.3.6.1.4.1.14519.5.2.1.4334.1501.177050854000720628302957773144
print(dcm.StudiesContainingOtherReferencedInstancesSequence[0].ReferencedSeriesSequence[0].ReferencedInstanceSequence[0].ReferencedSOPInstanceUID)
# 1.3.6.1.4.1.14519.5.2.1.4334.1501.177050854000720628302957773144

Highdicom is tolerant to duplicates like this if all three UIDs (Study-, Series-, and SOP Instance) match. However, in this case the series instance UIDs do not match, implying that there are two source series that share all the same SOPInstanceUIDs but have different SeriesInstanceUIDs:

print(dcm.ReferencedSeriesSequence[0].SeriesInstanceUID)
# 1.3.6.1.4.1.14519.5.2.1.4334.1501.668993209202739148457939869868
print(dcm.StudiesContainingOtherReferencedInstancesSequence[0].ReferencedSeriesSequence[0].SeriesInstanceUID)
# 1.3.6.1.4.1.14519.5.2.1.4334.1501.250972837350611779650157070453

I do not think there is any situation in the DICOM standard where this is correct. I would consider this a data integrity problem with the data at least, and possibly a bug with the tool that created it.

@fedorov @pieper I see this metadata in the segmentation in question, so I figure one of you would probably know something about this. Any idea how this could have happened? Should I open an issue on the reporting repo, and should we perhaps fix this in the IDC?

(0008,103E) LO SeriesDescription = 3D Slicer segmentation result 
(0008,1090) LO ManufacturerModelName = https://github.com/fedorov/Reporting

@JoshuaSiraj As a workaround, I suggest you simply delete the StudiesContainingOtherReferencedInstancesSequence before having highdicom read it.

[CPBridge]

Also note that the second of these two Series UIDs (1.3.6.1.4.1.14519.5.2.1.4334.1501.250972837350611779650157070453) does not seem to exist in the IDC according to IDC index.

[fedorov]

@CPBridge it is the same collection we discussed earlier in #344 (comment). Yes, definitely looks like a problem with the data, and very likely with the tool, but it will be really difficult to trace it down to the tool, since it was so long ago. Plus the UIDs went through de-identification hashing, so who knows what was done to what....

I don't think opening an issue in https://github.com/fedorov/Reporting would help, since it is unlikely the issue is reproducible with the current version of the code.

I think the best would be to report the problem with the dataset to TCIA via https://help.cancerimagingarchive.net/servicedesk/customer/portal/1/create/1, but I would not hold the breath it will be fixed anytime soon, with everything else going on.

I would suggest the most practical way forward is to make that a warning, not an error, or let user force this kind of issue to be treated as a warning. Are there legitimate scenarios where this problem might interfere with interpretation of the segmentation?

[JoshuaSiraj]

@CPBridge Thanks for getting back to me so quickly—and thanks for the explanation!

[CPBridge]

@fedorov this is not a check I explicitly wrote unfortunately. I specified a schema for the underlying SQLite lookup tables that highdicom constructs when you load a segmentation, and the information in this file violates that schema because SOPInstanceUID is used as a primary key in one of the tables (quite reasonably I had thought) and therefore cannot contain duplicates... So it is not something that I can just turn into a warning straightforwardly. I would have to change (weaken) the whole schema

Generally my philosophy here is to construct these lookup tables upfront such that when they are used later by methods to respond to various user queries, these methods can all assume that the lookup tables are complete and valid. Introducing some sort of semi-valid or incomplete state considerably complicates all these other methods that may use the information, so I am generally rather reluctant to do it: the bug surface goes up considerably.

I could add logic to intercept and remove the duplicate UIDs before they are added to the lookup table without sacrificing the integrity of the table. Then they would not appear in response to user queries (which would be a bit unexpected but probably reasonable). In this case it is clear that the copy in StudiesContainingOtherReferencedInstancesSequence should be ignored. But I worry that this is not something that can generally be assumed. What should be done if the same SOPInstanceUID appears in two different series within the ReferencedSeriesSequence. Which then should be retained. It gets messy...

I will chew on this for a bit and see if there is a workaround that isn't too disruptive

[pieper]

This could be where the concept of "normalizers" could come into play. Basically a normalizer is identity at the superclass, but subclasses recognize specific abnormalities in input data and output a corrected version that is what the standard would require for that type of data. That way downstream code knows what to expect and isn't littered with special cases.

[CPBridge]

I did file an issue on the TCIA support desk for this. I also briefly checked that this seems to occur only for the file in question, not others in the collection (did you find the same @JoshuaSiraj?)

[CPBridge]

@pieper is there anything I could read about this? A web search doesn't seem to turn up the right things as "normalize" in such an overloaded term. It sounds like an interesting idea, but not sure how exactly it could be applied here

[JoshuaSiraj]

I did file an issue on the TCIA support desk for this. I also briefly checked that this seems to occur only for the file in question, not others in the collection (did you find the same @JoshuaSiraj?)

Yep what I've found as well.

[fedorov]

I specified a schema for the underlying SQLite lookup tables that highdicom constructs when you load a segmentation

@CPBridge what is the schema that you use? Where is StudiesContainingOtherReferencedInstancesSequence used to be of priority to be included in that schema? Or are you indexing all of the tags, including all of the nested sequences?

[CPBridge]

There's an internal table that stores all SOP Instances referenced anywhere in the segmentation, along with the series and study instance uids to which they belong, from the ReferencedSeriesSequence, StudiesContainingOtherReferencedInstancesSequence and SourceImageEvidenceSequence. This allows the user to ask for "full" information (study/series/instance) about referenced instance, and can be cross-referenced to the frame-level references found in the PerFrameFunctionalGroupsSequence. It is somewhat tolerant to things like duplicated or missing references, but having one SOPInstanceUID belong to two different series was not something I envisioned having to deal with

[CPBridge]

As things stand I am leaning towards ignoring this, we only know of one file in the world that has this problem (others in the same collection are fine) and it doesn't seem like it would an easy or common mistake to make when encoding a file (seemingly requiring you to make up incorrect study/series UIDs or the existence of two conflicting instances of the same UID that both happen to be relevant to the construction of a single segmentation)