Cell2location reads raw counts as normalized values

[deconvolute]

While loading the scRNA-seq reference data, I get a warning saying, "adata.X does not contain unnormalized count data" and when I proceed further to train the model to estimate the reference cell type signatures it throws an error.

adata_ref = sc.read("new.all.h5ad")

adata_ref.var['SYMBOL'] = adata_ref.var.index
del adata_ref.raw

from cell2location.utils.filtering import filter_genes
selected = filter_genes(adata_ref, cell_count_cutoff=5, cell_percentage_cutoff2=0.03, nonz_mean_cutoff=1.12)

# filter the object
adata_ref = adata_ref[:, selected].copy()
cell2location.models.RegressionModel.setup_anndata(adata_ref, batch_key='sample_id', labels_key = 'seurat_clusters')

# create the regression model
from cell2location.models import RegressionModel
mod = RegressionModel(adata_ref)

# view anndata_setup as a sanity check
mod.view_anndata_setup()
mod.train(max_epochs=250, use_gpu=False)

anaconda version 2020.07 loaded.
Global seed set to 0
/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/utilities/warnings.py:53: LightningDeprecationWarning: pytorch_lightning.utilities.warnings.rank_zero_deprecation has be
en deprecated in v1.6 and will be removed in v1.8. Use the equivalent function from the pytorch_lightning.utilities.rank_zero module instead.
  new_rank_zero_deprecation(
/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/utilities/warnings.py:58: LightningDeprecationWarning: The `pytorch_lightning.loggers.base.rank_zero_experiment` is depr
ecated in v1.7 and will be removed in v1.9. Please use `pytorch_lightning.loggers.logger.rank_zero_experiment` instead.
  return new_rank_zero_deprecation(*args, **kwargs)
/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pandas/core/arraylike.py:402: RuntimeWarning: divide by zero encountered in log10
  result = getattr(ufunc, method)(*inputs, **kwargs)
/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/scvi/data/fields/_layer_field.py:91: UserWarning: adata.X does not contain unnormalized count data. Are you sure this is what you want?
  warnings.warn(
Multiprocessing is handled by SLURM.
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/configuration_validator.py:105: UserWarning: You passed in a `val_dataloader` but have no `validation_step`. Ski
pping val loop.
  rank_zero_warn("You passed in a `val_dataloader` but have no `validation_step`. Skipping val loop.")
SLURM auto-requeueing enabled. Setting signal handlers.
Traceback (most recent call last):
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pyro/poutine/trace_struct.py", line 230, in compute_log_prob
    log_p = site["fn"].log_prob(
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pyro/distributions/conjugate.py", line 277, in log_prob
    self._validate_sample(value)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/torch/distributions/distribution.py", line 293, in _validate_sample
    raise ValueError(
ValueError: Expected value argument (Tensor of shape (2500, 21958)) to be within the support (IntegerGreaterThan(lower_bound=0)) of the distribution GammaPoisson(), but found invalid values:
tensor([[0.0000, 0.0000, 1.7781,  ..., 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 1.0603,  ..., 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 1.9427,  ..., 0.0000, 0.0000, 0.0000],
        ...,
        [0.0000, 0.0000, 2.2058,  ..., 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000]])

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/N/slate/merajam/Spatial_PDAC/cell2loc_2/New_Analysis/Scripts/cell2loc2_batch1.py", line 47, in <module>
    mod.train(max_epochs=250, use_gpu=False)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/cell2location/models/reference/_reference_model.py", line 157, in train
    super().train(**kwargs)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/scvi/model/base/_pyromixin.py", line 146, in train
    return runner()
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/scvi/train/_trainrunner.py", line 82, in __call__
    self.trainer.fit(self.training_plan, self.data_splitter)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/scvi/train/_trainer.py", line 188, in fit
    super().fit(*args, **kwargs)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 696, in fit
    self._call_and_handle_interrupt(
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1166, in _run
    results = self._run_stage()
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1252, in _run_stage
    return self._run_train()
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1283, in _run_train
    self.fit_loop.run()
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
 File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 271, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 203, in advance
    batch_output = self.batch_loop.run(kwargs)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 89, in advance
    outputs = self.manual_loop.run(kwargs)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/loops/loop.py", line 200, in run
    self.advance(*args, **kwargs)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/loops/optimization/manual_loop.py", line 110, in advance
    training_step_output = self.trainer._call_strategy_hook("training_step", *kwargs.values())
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1704, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pytorch_lightning/strategies/strategy.py", line 358, in training_step
    return self.model.training_step(*args, **kwargs)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/scvi/train/_trainingplans.py", line 809, in training_step
    loss = torch.Tensor([self.svi.step(*args, **kwargs)])
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pyro/infer/svi.py", line 145, in step
    loss = self.loss_and_grads(self.model, self.guide, *args, **kwargs)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pyro/infer/trace_elbo.py", line 140, in loss_and_grads
    for model_trace, guide_trace in self._get_traces(model, guide, args, kwargs):
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pyro/infer/elbo.py", line 182, in _get_traces
    yield self._get_trace(model, guide, args, kwargs)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pyro/infer/trace_elbo.py", line 57, in _get_trace
    model_trace, guide_trace = get_importance_trace(
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pyro/infer/enum.py", line 75, in get_importance_trace
    model_trace.compute_log_prob()
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pyro/poutine/trace_struct.py", line 236, in compute_log_prob
    raise ValueError(
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pyro/poutine/trace_struct.py", line 230, in compute_log_prob
    log_p = site["fn"].log_prob(
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/pyro/distributions/conjugate.py", line 277, in log_prob
    self._validate_sample(value)
  File "/N/u/merajam/Carbonate/.conda/envs/cell2loc_env/lib/python3.9/site-packages/torch/distributions/distribution.py", line 293, in _validate_sample
    raise ValueError(
ValueError: Error while computing log_prob at site 'data_target':
Expected value argument (Tensor of shape (2500, 21958)) to be within the support (IntegerGreaterThan(lower_bound=0)) of the distribution GammaPoisson(), but found invalid values:
tensor([[0.0000, 0.0000, 1.7781,  ..., 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 1.0603,  ..., 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 1.9427,  ..., 0.0000, 0.0000, 0.0000],
        ...,
        [0.0000, 0.0000, 2.2058,  ..., 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000],
        [0.0000, 0.0000, 0.0000,  ..., 0.0000, 0.0000, 0.0000]])
                Trace Shapes:                      
                 Param Sites:                      
                Sample Sites:                      
       per_cluster_mu_fg dist            | 25 21958
                        value            | 25 21958
                     log_prob            |         
      detection_mean_y_e dist            | 64     1
                        value            | 64     1
                     log_prob            |         
  s_g_gene_add_alpha_hyp dist    1     1 |         
                        value    1     1 |         
                     log_prob    1     1 |         
       s_g_gene_add_mean dist            | 64     1
                        value            | 64     1
                     log_prob            |         
                     log_prob            |         
            s_g_gene_add dist            | 64 21958
                        value            | 64 21958
                     log_prob            |         
         alpha_g_phi_hyp dist    1     1 |         
                        value    1     1 |         
                     log_prob    1     1 |         
         alpha_g_inverse dist            |  1 21958
                        value            |  1 21958
                     log_prob            |         
             data_target dist 2500 21958 |         
                        value 2500 21958 |         

[vitkl]

Hi @deconvolute

You see exactly what you should expect to see when values in adata_ref.X are non-integer counts. Please update your input data to have integer counts. I also find it confusing why many tutorials for other methods call normalised (and potentially log1p-transformed) data as "raw counts". Raw counts are integer counts.