Merge pull request #23 from bic-mac-challenge/dev/update_evaluation_metrics

Remove TAC Bias metric

Author: ChristianHinge

README.md

@@ -8,10 +8,11 @@
 
 ---
 ## Updates
-- July 6, 2026: BUG FIX: DRY-RUN and FINAL submission instructions listed a wrong email. All DRY-RUN and FINAL submissions should be sent to bic-mac-challenge@outlook.com. 
-- May 15, 2026: NIfTI validation and leaderboards now live - see submission instructions at [https://www.codabench.org/competitions/12555](https://www.codabench.org/competitions/12555)
-- April 8, 2026 (13:00 GMT+2): [BUG FIX] A bug in the Carney HU-to-MU conversion formula has been fixed. Please make sure to run `docker pull ghcr.io/bic-mac-challenge/recon:latest` and `git pull` for updated recon and evaluation logic. The 8 ground truth PET train images have been updated on huggingface. 
-- April 7, 2026: [NEW DATA POLICY] The use of public datasets for pretraining and other use-cases is now allowed under certain conditions. Please see [docs/rules.md](docs/rules.md) for details. 
+- **July 8, 2026: [CHANGE]**: The fifth metric, TAC-Bias, has been removed from the final evaluation. Final submissions are scored on the four metrics defined in [Evaluation](#evaluation-srcevaluation). Please see PR #23 for details.
+- **July 6, 2026: [BUG FIX]**: DRY-RUN and FINAL submission instructions listed a wrong email. All DRY-RUN and FINAL submissions should be sent to bic-mac-challenge@outlook.com. 
+- **May 15, 2026: [ANNOUNCEMENT]**: NIfTI validation and leaderboards now live - see submission instructions at [https://www.codabench.org/competitions/12555](https://www.codabench.org/competitions/12555)
+- **April 8, 2026 (13:00 GMT+2)**: [BUG FIX] A bug in the Carney HU-to-MU conversion formula has been fixed. Please make sure to run `docker pull ghcr.io/bic-mac-challenge/recon:latest` and `git pull` for updated recon and evaluation logic. The 8 ground truth PET train images have been updated on huggingface. 
+- **April 7, 2026: [NEW DATA POLICY]**: The use of public datasets for pretraining and other use-cases is now allowed under certain conditions. Please see [docs/rules.md](docs/rules.md) for details. 
 ## Table of Contents
 
 - [Overview](#overview)
@@ -185,15 +186,17 @@ The reconstructed PET is written to `/data/output/pet.nii.gz`.
 
 ## 📊 Evaluation (`src/evaluation/`)
 
-Five metrics compare predicted PET and CT outputs against the ground truth:
+> [!IMPORTANT]
+> The fifth metric `TAC Bias` has been removed from the final evaluation, please see announcement and PR #23. 
+
+Four metrics compare predicted PET and CT outputs against the ground truth:
 
 | Metric | Modality | Description | Region |
 |--------|------|-------------|--------|
 | Whole-body SUV MAE | `PET` | Mean absolute error in standardised uptake value (SUV = activity × weight / total dose) | Body mask, excluding ±4 cm around liver |
 | Brain Outlier Score | `PET` | AUC of fraction of brain voxels within relative error thresholds (5%, 10%, 15%) [Ladefoged 2017 (fig. 8)](https://www.sciencedirect.com/science/article/pii/S1053811916307170)| Brain |
 | Organ Bias | `PET` | Mean absolute relative error of mean SUV in 8 organs: brain, liver, spleen, heart, pancreas, muscle, adipose, extremities | TotalSegmentator organ labels |
 | CT MU MAE | `CT` | Mean absolute error of attenuation coefficients (μ at 511 keV) between predicted and ground-truth CT after HU→μ conversion | Body mask, excluding ±4 cm axial slices at top of liver|
-| TAC Bias | `Dynamic PET` | Absolute relative error of the integral of time-activity-curves (TACs) for the aorta and selected brain regions. NOTE: Metric is computed only for the final test set due to the size of the dynamic sinograms. | Brain regions and aorta|
 
 **Evaluate a single subject:**
 

docs/rules.md

@@ -55,7 +55,7 @@ Participating teams may publish their own results independently, subject to a **
 
 Performance is ranked using a rank-based aggregation across five evaluation metrics. Each metric is averaged across all test cases, submissions are ranked per metric (1 = best), and the final score is the mean of the five metric ranks. The lowest final score wins.
 
-Please see [src/evaluation/README.md](src/evaluation/README.md) for definitions of each metric. Note that fifth metric, TAC-bias, cannot be computed locally. It is used only for the final evaluation.
+Please see [src/evaluation/README.md](src/evaluation/README.md) for definitions of each metric.
 
 In the event of a tied aggregated rank, teams share the corresponding placement following Olympic-style conventions (e.g., two first places, no second place, then third place). Prize money is split equally between tied teams.
 

docs/tips-and-faq.md

@@ -121,7 +121,7 @@ Check `output_dir/intermediates/recon.log` for the full STIR log. Rerun with `VE
 
 **Which metrics are reported on the validation leaderboard?**
 
-Four metrics in total: Whole-body SUV MAE, Brain Outlier Score, Organ Bias, and CT μ-MAE. See the evaluation section of the main README for descriptions. The Brain Outlier Score is a dataset-level metric — it cannot be computed for a single subject. The fifth and final metric "TAC Bias" is only computed for the final test set. The metric calculation requires reconstruction using dynamic sinograms, which are unfortunately too large to share. 
+Four metrics in total: Whole-body SUV MAE, Brain Outlier Score, Organ Bias, and CT μ-MAE. See the evaluation section of the main README for descriptions. The Brain Outlier Score is a dataset-level metric — it cannot be computed for a single subject.
 
 **Can I evaluate without running reconstruction?**
 

src/evaluation/__init__.py

@@ -8,7 +8,6 @@
 - Whole-body SUV MAE
 - Brain outlier robustness score
 - Organ bias
-- TAC bias
 """
 
 from .eval_dataset import evaluate_dataset
@@ -17,6 +16,5 @@
     compute_whole_body_suv_mae,
     compute_brain_outlier_score,
     compute_organ_bias,
-    compute_tac_bias,
     compute_whole_body_mu_mae,
 )

src/evaluation/metrics/__init__.py

@@ -7,5 +7,4 @@
 from .pet_whole_body_mae import compute_whole_body_suv_mae
 from .pet_brain_outlier import compute_brain_outlier_score
 from .pet_organ_bias import compute_organ_bias
-from .pet_tac_bias import compute_tac_bias
 from .ct_whole_body_mae import compute_whole_body_mu_mae

src/evaluation/metrics/common.py

@@ -41,31 +41,4 @@ def compute_auc_of_K(k_values):
 
     Analytically, AUC = mean(k_values), since ∫₀¹ I(k > x) dx = k for k ∈ [0, 1].
     """
-    return np.mean(k_values)
-
-
-# =========================================================
-# TAC Utilities
-# =========================================================
-
-def integrate_tac(tac, frame_durations):
-    """
-    Compute time-integrated activity (AUC).
-    """
-    return np.sum(tac * frame_durations)
-
-
-def compute_region_auc(pet_4d, mask, frame_durations):
-    """
-    Compute integrated TAC (AUC) for one region.
-    """
-
-    T = pet_4d.shape[-1]
-    tac = []
-
-    for t in range(T):
-        tac.append(np.mean(pet_4d[..., t][mask]))
-
-    tac = np.array(tac)
-
-    return integrate_tac(tac, frame_durations)
+    return np.mean(k_values)