Adding imputed sex labels

[erflynn]

Issue Number

This addresses issue number #2181.

Purpose/Implementation Notes

This update includes imputed sex labels for microarray data (mouse, rat, and human).

I will update with more labels soon; I am not sure if you want to integrate this pull request now or come back to it later when I have more labels? I am sending now so we can work through methods, format, etc.

Methods

Details are included in the below (this is also in the README within the config/externally_supplied_metadata/directory).
All code to produce this update is included in the sl_label repository and can be easily applied to other organisms who have XX/XY sex determination (provided sufficient metadata sex labels are available for model training).

The majority of gene expression data is missing metadata sex labels (see Table 1). This lack of labels prevents us from examining the sex breakdown of many studies. We trained a penalized logistic regression model that uses the expression of X and Y chromosome genes to impute the sex of a given sample. The model was trained using the glmnet R package, with the elastic net penalty. Lambda was selected in ten-fold cross validation. We used metadata sex labels for "ground truth" for the training and testing data. To construct the training and testing datasets, we filtered for samples with metadata sex labels that did not have a cell line annotation (using the refine-bio sex and cell_line tags), and grouped these samples into studies. We divided the set of all studies in half, and then within each half, sampled n=700 samples for training, and n=300 samples for testing for both males and females. This provided a balanced training and testing dataset, where none of the samples in the test data were from a study that had been seen in the training.

A previous study [1] indicated that there is widespread mis-annotation in metadata sex labels, because of this, we set up additional testing datasets: a high confidence mixed sex dataset and single sex datasets. The high confidence mixed sex dataset consists of all mixed sex studies with at least five male and five female samples where the metadata sex labels match expression-based labels from at least one of two clustering-based sex imputation methods [1,2] (we did not apply clustering based methods to the entire refine-bio dataset because they have poor performance on small studies, single sex studies, and studies with high class imbalance). The single sex datasets are all studies with at least ten samples and all male or all female labels as indicated by metadata sex labels.

Across all three organisms, our models achieve approximately 95% accuracy in a randomly selected held-out test set as compared to the metadata labels. Additionally, we assessed the accuracy of our model, on various subsets of the data; comparing to all metadata sex labels (agreement 93.5-94.8%), a random sample of single sex studies (agreement 92.6-96.5%), and, in human, manually annotated sex labels from a previous analysis [3] (94.2%) (see Table 2).

organism	Samples (n)	Samples missing metadata annotation	Studies (n)	Studies missing metadata annotation
human	430119	74.90%	14987	87.30%
mouse	228707	74.00%	12995	80.90%
rat	31361	55.50%	1295	65.30%

Table 1. Metadata missingness for sex labels.

dataset	Human	Mouse	Rat
Training (n=1400)	95.60%	96.10%	98.60%
Testing (n=600)	95.20%	95.70%	95.50%
Metadata	93.5% (107748)	93.5% (58473)	94.8% (13995)
High-confidence	97.3% (7301)	95.5% (3968)	n/a
Single sex - f	96.5% (12919)	93.4% (13243)	96.3% (2240)
Single sex - m	92.6% (8128)	93.6% (30225)	95.8% (12689)
Manual annotations	94.2% (8289)	n/a	n/a

Table 2. Concordance of sex labels. Numbers in parentheses indicate the total number of samples, percentages the number of samples that agree divided by the total number of samples. High confidence labels have matching metadata and clustering based expression labels.

The cleaned metadata sex labels are also included in the cleaned_metadata/ directory for microarray and RNA-seq. This process mapped all harmonized sex labels to "male", "female", "mixed", or "unknown". Code for this is included in 01_metadata within the erflynn/sl_label repository here

Types of changes

This includes externally supplied metadata files in .json format. These are under
config/externally_supplied_metadata/. The format is as follows (and as discussed in Issue #2127)

{"sample_accession": "<SAMPLE_ACCESSION_CODE>",
 "attributes": [{"PATO:0000047": {"value": <VALUE>,
                                          "probability": <PROBABILITY>}                } ]
}

where value is one of "PATO:0000383" (female) or "PATO:0000384" (male) and probability is P(imputed_sex=value) from the logistic regression model.

Cleaned harmonized metadata for sex is included in the cleaned_metadata/ directory with the columns "acc" (sample accession), "sex" (the harmonized sex label), and "mapped_sex" (the harmonized sex label mapped to "male", "female", "mixed", or "unknown").

Screenshot

Sex sample breakdown using metadata and (imputed) expression labels for each organism.
sex_breakdown_microarray.pdf

References

[1] Toker, L., et al. F1000Research. 2016, 5: 2103.
[2] Buckberry, S., et al. Bioinformatics. 2014, 30(14): 2084–2085.
[3] Giles, C. B, et al. BMC Bioinformatics. 2017,18(Suppl 14): 509.