Function to calculate IDER-based metric

[sjspielman]

We should write a function that takes an integrated SCE object and calculates the IDER-based metric. The final output from this function should be an R object that can be saved to file, likely either TSV or RDS depending on the specific content of the calculations.

References:

• "Quick start" from the R package README: https://github.com/zhiyhu/CIDER#cider-as-an-evaluation-metric---quick-start
• Vignette: https://zhiyhu.github.io/CIDER/articles/evaluation.html

As described in the references above, the input will need to be a Seurat object, so this function will need to convert SCE --> Seurat to begin.

[sjspielman]

Here are some notes from my initial exploration of calculating this metric.

First, here's the quick overview (from their README, and I added comments) of how to run CIDER:

seu.integrated <- hdbscan.seurat(seu.integrated) # performs the clustering
ider <- getIDEr(seu.integrated, verbose = FALSE) # calculate ider metrics. 
seu.integrated <- estimateProb(seu.integrated, ider) # evaluate

Note that the ider object is, per ?getIDER: "A list of four objects: a similarity matrix, a numeric vector recording cells used and the data frame of combinations included."
I think "four objects" is a typo and they mean three!

Hurdles:

• The Seurat object will need to be renamed cca_PCA --> pca. Some of the CIDER functions, but not all!, allow you to specify a custom name for the PCs of interest. The functions that do not allow will assume it's called pca, so we just have to rename.
• The Seurat object needs to have a variable named Batch, not batch, giving the batches (Seuratobject$Batch is hardcoded 😑 )
• The estimateProb() function isn't working when I test this out, with the error "Cannot find 'Group' in this Seurat object". The traceback reveals the error is happening here: https://github.com/zhiyhu/CIDER/blob/68a08ae427a94fb783964e5709adeb904896e7f6/R/evaluation.R#L80-L82
  It seems that CIDER expects a variable named Group in the Seurat object that contains the ground truths. This is not clearly documented (in fact there is overall very little documentation for using IDER), and also confusing given that after this function is used and output explored in the vignette, the vignette states "So far the evaluation have completed and CIDER has not used the ground truth at all!".

We will probably want to file an issue to the CIDER github seeking clarification about estimateProb().

[sjspielman]

I filed an issue over in the CIDER repo. Let's see how it goes!! 🤞

zhiyhu/CIDER#4

Edit Friday 8/19 - no response yet.

[sjspielman]

After some offline discussion with @allyhawkins, we have decided to move forward here and just calculate the IDER similarity matrix.

I'll also note that I strongly suspect the CIDER package code is using their asCIDER algorithm, and not their dnCIDER algorithm, and there are some other concerns with how generalizable the package code itself is overall. We would like to use dnCIDER as asCIDER requires cell type annotations for initial clustering, but dnCIDER performs de novo clustering (though note, there is always clustering!!) at this step.

[sjspielman]

Here is the dnCIDER vignette as needed: https://github.com/zhiyhu/CIDER/blob/master/vignettes/dnCIDER_highlevel.Rmd

[jaclyn-taroni]

I'm going to remove this from the sprint. It got added automatically.

[sjspielman]

Slight progress here - they responded to the issue and said they fixed the bug with a new CRAN release, but actually now there's just a different bug. So, still no CIDER.
zhiyhu/CIDER#4 (comment)