Merge pull request #11 from ImagingDataCommons/update-iid-and-more

Update idc-index version and add functionality

Author: fedorov

.github/dependabot.yml

@@ -0,0 +1,6 @@
+version: 2
+updates:
+  - package-ecosystem: "github-actions"
+    directory: ".github/workflows"
+    schedule:
+      interval: "daily"

.github/workflows/check-links.yml

@@ -29,3 +29,4 @@ jobs:
             --config .lychee.toml
             '**/*.md'
           fail: true
+          debug: true

.github/workflows/test-snippets.yml

@@ -0,0 +1,42 @@
+name: Test Code Snippets
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - 'SKILL.md'
+      - 'references/**'
+      - 'tests/**'
+      - '.github/workflows/test-snippets.yml'
+  pull_request:
+    paths:
+      - 'SKILL.md'
+      - 'references/**'
+      - 'tests/**'
+      - '.github/workflows/test-snippets.yml'
+  workflow_dispatch:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+
+      - name: Cache idc-index data
+        uses: actions/cache@v4
+        with:
+          path: ~/.idc
+          key: idc-index-0.11.14
+
+      - name: Install test dependencies
+        run: pip install -r tests/requirements-test.txt
+
+      - name: Run snippet tests
+        run: pytest tests/test_snippets.py -v --timeout=300

CHANGELOG.md

@@ -5,7 +5,28 @@ All notable changes to the IDC Claude Skill are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/),
 and this project adheres to [Semantic Versioning](https://semver.org/).
 
-## [Unreleased]
+## [1.5.0] - 2026-04-08
+
+### Added
+
+- `volume_geometry_index` table documentation: 3D geometry validation for single-frame CT, MR, and PT series; boolean checks (orientation, spacing, dimensions, slice positions) and composite `regularly_spaced_3d_volume` flag; join via `SeriesInstanceUID`
+- `rtstruct_index` table documentation: RT Structure Set metadata (total ROIs, ROI names, generation algorithms, interpreted types, referenced image series UID); join via `SeriesInstanceUID`
+- New reference guide `references/parquet_access_guide.md`: direct DuckDB queries against public GCS Parquet files without installing idc-index; URL pattern, available files, and query examples for main index, `volume_geometry_index`, and `rtstruct_index`
+- SQL patterns for `volume_geometry_index` and `rtstruct_index` in `references/sql_patterns.md`
+- Detailed documentation for BigQuery-only derived tables in `references/bigquery_guide.md`:
+  - `segmentations`: per-segment anatomy with full schema, column descriptions, and queries for discovering structures, filtering by coded concept, and linking to SR measurements; note on gap vs `seg_index` in idc-index
+  - `quantitative_measurements`: radiomics and clinical numeric measurements from DICOM SR TID1500 (volume, diameter, shape descriptors, texture, intensity statistics); full schema with column descriptions and query examples
+  - `qualitative_measurements`: coded assessments from DICOM SR TID1500 (malignancy rating, calcification, texture, margin); full schema with column descriptions and query examples
+  - `measurement_groups`: parent grouping table for SR measurements
+  - Combined example joining all three derived tables for LIDC-IDRI nodule analysis (malignancy + volume + diameter)
+- SKILL.md section 7 now explicitly lists per-segment anatomy search, quantitative SR measurements, and qualitative SR measurements as BigQuery-only use cases with no idc-index equivalent
+
+### Changed
+
+- Updated to idc-index 0.11.14 (idc-index-data 23.10.1)
+- Added `SOPClassUID` and `TransferSyntaxUID` columns to Key Columns Reference in `references/index_tables_guide.md`
+- Added Direct Parquet Access entry to Data Access Options table and pointer in SKILL.md
+- Added `parquet_access_guide.md` to Quick Navigation table in SKILL.md
 
 ## [1.4.0] - 2026-03-04
 

SKILL.md

@@ -5,7 +5,7 @@ license: This skill is provided under the MIT License. IDC data itself has indiv
 metadata:
     version: 1.4.0
     skill-author: Andrey Fedorov, @fedorov
-    idc-index: "0.11.10"
+    idc-index: "0.11.14"
     idc-data-version: "v23"
     repository: https://github.com/ImagingDataCommons/idc-claude-skill
 ---
@@ -25,7 +25,7 @@ Use the `idc-index` Python package to query and download public cancer imaging d
 ```python
 import idc_index
 
-REQUIRED_VERSION = "0.11.10"  # Must match metadata.idc-index in this file
+REQUIRED_VERSION = "0.11.14"  # Must match metadata.idc-index in this file
 installed = idc_index.__version__
 
 if installed < REQUIRED_VERSION:
@@ -97,6 +97,7 @@ print(stats)
 | `digital_pathology_guide.md` | Slide microscopy (SM), annotations (ANN), pathology workflows |
 | `bigquery_guide.md` | Full DICOM metadata, private elements (requires GCP) |
 | `cli_guide.md` | Command-line tools (`idc download`, manifest files) |
+| `parquet_access_guide.md` | Direct Parquet queries via GCS (no idc-index install needed) |
 
 ## IDC Data Model
 
@@ -138,6 +139,8 @@ The `idc-index` package provides multiple metadata index tables, accessible via
 | `ann_index` | 1 row = 1 DICOM ANN series | fetch_index() | Microscopy Bulk Simple Annotations series metadata; references annotated image series |
 | `ann_group_index` | 1 row = 1 annotation group | fetch_index() | Detailed annotation group metadata: graphic type, annotation count, property codes, algorithm |
 | `contrast_index` | 1 row = 1 series with contrast info | fetch_index() | Contrast agent metadata: agent name, ingredient, administration route (CT, MR, PT, XA, RF) |
+| `volume_geometry_index` | 1 row = 1 CT/MR/PT series | fetch_index() | 3D volume geometry validation for single-frame CT, MR, and PT series; boolean checks for orientation, spacing, dimensions, and slice positions; composite `regularly_spaced_3d_volume` flag |
+| `rtstruct_index` | 1 row = 1 RTSTRUCT series | fetch_index() | RT Structure Set metadata: total ROI count, ROI names, generation algorithms, interpreted types, and the referenced image series UID |
 
 **Auto** = loaded automatically when `IDCClient()` is instantiated
 **fetch_index()** = requires `client.fetch_index("table_name")` to load
@@ -159,6 +162,8 @@ The `idc-index` package provides multiple metadata index tables, accessible via
 | `SeriesInstanceUID` | index, seg_index, ann_index, ann_group_index, contrast_index | Link segmentation/annotation/contrast series to its index metadata |
 | `segmented_SeriesInstanceUID` | seg_index → index | Link segmentation to its source image series (join seg_index.segmented_SeriesInstanceUID = index.SeriesInstanceUID) |
 | `referenced_SeriesInstanceUID` | ann_index → index | Link annotation to its source image series (join ann_index.referenced_SeriesInstanceUID = index.SeriesInstanceUID) |
+| `SeriesInstanceUID` | index, volume_geometry_index | Link series to its 3D geometry validation result (join index.SeriesInstanceUID = volume_geometry_index.SeriesInstanceUID) |
+| `SeriesInstanceUID` / `referenced_SeriesInstanceUID` | index, rtstruct_index | Join RTSTRUCT series to its metadata (index.SeriesInstanceUID = rtstruct_index.SeriesInstanceUID); use rtstruct_index.referenced_SeriesInstanceUID to find the source image series |
 
 **Note:** `Subjects`, `Updated`, and `Description` appear in multiple tables but have different meanings (counts vs identifiers, different update contexts).
 
@@ -184,6 +189,7 @@ See `references/clinical_data_guide.md` for detailed workflows including value m
 | Method | Auth Required | Best For |
 |--------|---------------|----------|
 | `idc-index` | No | Key queries and downloads (recommended) |
+| Direct Parquet (GCS) | No | Quick queries without installing idc-index; always uses latest data |
 | IDC Portal | No | Interactive exploration, manual selection, browser-based download |
 | BigQuery | Yes (GCP account) | Complex queries, full DICOM metadata |
 | DICOMweb proxy | No | Tool integration via DICOMweb API |
@@ -214,6 +220,12 @@ IDC data is available via DICOMweb interface (Google Cloud Healthcare API implem
 
 See `references/dicomweb_guide.md` for endpoint URLs, code examples, supported operations, and implementation details.
 
+**Direct Parquet access**
+
+All idc-index metadata tables are published as Parquet files to a public GCS bucket (`idc-index-data-artifacts`) with unrestricted CORS. This enables DuckDB or pandas queries without installing idc-index, including cross-table joins and queries against `volume_geometry_index` and `rtstruct_index`.
+
+See `references/parquet_access_guide.md` for URL patterns, available files, and DuckDB query examples.
+
 ## Installation and Setup
 
 **Required (for basic access):**
@@ -229,7 +241,7 @@ print(client.get_idc_version())  # Should return "v23"
 ```
 If you see an older version, upgrade with: `pip install --upgrade idc-index`
 
-**Tested with:** idc-index 0.11.10 (IDC data version v23)
+**Tested with:** idc-index 0.11.14 (IDC data version v23)
 
 **Optional (for data analysis):**
 ```bash
@@ -649,6 +661,13 @@ See `references/bigquery_guide.md` for setup, table schemas, query patterns, pri
 
 Common specialized indices: `seg_index` (segmentations), `ann_index` / `ann_group_index` (microscopy annotations), `sm_index` (slide microscopy), `collections_index` (collection metadata). Only use BigQuery if you need private DICOM elements or attributes not in any index.
 
+**Use cases that require BigQuery (no idc-index equivalent):**
+- **Per-segment anatomy search** — `seg_index` gives series-level SEG metadata, but the BigQuery `segmentations` table exposes each segment individually with its DICOM coded structure name (e.g., find all SEG series containing a "Liver" or "Neoplasm" segment)
+- **Quantitative measurements from SR** — the `quantitative_measurements` BigQuery table contains pre-extracted radiomics features (volume, diameter, shape descriptors, texture, intensity statistics) from DICOM SR TID1500 objects; no idc-index equivalent
+- **Qualitative measurements from SR** — the `qualitative_measurements` BigQuery table contains coded assessments (malignancy rating, calcification, texture, margin) from DICOM SR TID1500; no idc-index equivalent
+
+See `references/bigquery_guide.md` for schemas, column descriptions, and query examples for these tables.
+
 ### 8. Tool Selection Guide
 
 | Task | Tool | Reference |