Version Compatibility

Reference examples tested with: cell2location 0.1.4+, scvi-tools 1.0+, scanpy 1.10+, anndata 0.10+, numpy 1.26+

RCTD/SPOTlight/STdeconvolve are R packages (spacexr, SPOTlight, STdeconvolve via Bioconductor); call them from R or via rpy2.

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures
• R: packageVersion('<pkg>') then ?function_name to verify parameters

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Spatial Deconvolution

"What cell types are in each spot?" -> Decompose a multi-cell capture spot into the fractions (or absolute numbers) of each cell type, using an annotated scRNA-seq reference as the basis.

• Python: cell2location (RegressionModel -> Cell2location), stereoscope/DestVI (scvi-tools), Tangram (tg.mapcellsto_space), STdeconvolve (reference-free, R)
• R: RCTD (spacexr::create.RCTD/run.RCTD), SPOTlight, SpatialDWLS (Giotto), CARD

The operation BRANCHES on the platform before any tool is chosen. The first question is not "which method" but "does this data even contain mixtures?" -- answered by the resolution fork below.

The resolution fork

Deconvolution recovers the cell-type composition of a MIXTURE. Whether a mixture exists is set by the capture unit size relative to a mammalian cell (~8-30um diameter), and it sorts every dataset into three regimes. Choosing the wrong regime is the most expensive error in spatial analysis -- far more damaging than picking the second-best method within a regime.

• DECONVOLVE: a capture array has no per-transcript cell assignment, so the spot is an unavoidable mixture and only mixture modeling recovers composition. The H&E underlay can count nuclei but cannot say which transcript came from which cell.
• SEGMENT (imaging platforms): the data are already single cells. The work is segmentation (assign transcripts to cells) then annotation (cluster + markers, or label-transfer). Running deconvolution here is conceptually WRONG -- it fabricates fractional mixtures inside cells that are already pure. See image-analysis for segmentation and single-cell/cell-annotation for label transfer.
• AMBIGUOUS (the near-single-cell middle): bins/beads still hold partial or multiple cells. When a high-quality registered image exists (Visium HD), morphology-driven cell RECONSTRUCTION (Bin2cell, StarDist/Cellpose nuclei expansion) is increasingly preferred over treating bins as fixed mixtures; without per-bead morphology (Slide-seqV2), assignment/deconvolution (often RCTD doublet-mode) remains standard. No settled consensus -- see high-resolution-binning.

Governing Principle

A deconvolution result is a PROJECTION of the single-cell reference onto the spatial data. The reference is not a neutral input -- it is the dominant determinant of the answer, more than the algorithm.

The #1 trap is the missing or mismatched cell type. If a real type is ABSENT from the reference, its transcripts are forced onto the transcriptionally nearest type that IS present. The proportions still sum to 1 and look clean, and there is no internal warning -- garbage reference produces confident garbage proportions. The reference must match the TISSUE, the CONDITION (a healthy reference mis-estimates activated-immune or malignant states whose expression has shifted), and ideally the technology/protocol (snRNA-seq vs whole-cell dissociation bias propagates straight into the numbers). The reference is NOT ground truth.

Every benchmark reinforces the same lesson: reference quality and the cell-type abundance pattern swing results MORE than method choice, and a plain NNLS baseline outperforms nearly half the dedicated methods (Sang-Aram 2024 eLife 12:RP88431; Li 2022 Nat Methods 19:662-670). Method-shopping is the wrong lever; reference quality is the right one. Rare-type fractions (below a few percent) are the least reliable numbers in the output -- corroborate any rare type with its spatial marker genes before believing it. More tools is not more confidence: two NB-regression methods agreeing is pseudo-replication, not orthogonal validation. The defensible confidence move is reference-sensitivity analysis (perturb or swap the reference) plus an orthogonal modality.

Outputs are COMPOSITIONAL: per-spot proportions live on a simplex (sum to 1), so an increase in one type mechanically decreases the others. Downstream comparison must use CLR/ILR or compositional tests -- naive per-type t-tests/correlations on raw proportions are mis-specified.

Choosing a method

The first axis is output: cell2location uniquely returns ABSOLUTE cell abundance (expected cell numbers per spot); the rest return proportions only. "Number of cells of type X" and "fraction of the spot that is type X" answer different biological questions. The second axis is runtime and whether spatial coordinates or a platform-shift correction are modeled. When competing methods exist, verify current best practice against the latest benchmark before committing -- this field moves fast.

cell2location and RCTD are reliably top-tier across independent benchmarks (Li 2022; Sang-Aram 2024; Nat Commun 2023 14:1548). When no matched reference exists, STdeconvolve is the escape hatch -- it is immune to the missing-type trap because it uses no reference, but its topics are de novo and must be annotated afterward.

cell2location step 1: reference signatures

Goal: Learn each cell type's per-gene expression signature from the annotated scRNA-seq reference, correcting for the technical/batch structure of the reference.

Approach: Filter genes to informative ones, fit a negative-binomial RegressionModel with the cell-type label and any batch as covariates, then export the posterior per-cluster mean expression. cell2location consumes RAW integer counts -- do NOT pass log-normalized data.

import cell2location
import numpy as np
import scanpy as sc
from cell2location.utils.filtering import filter_genes
from cell2location.models import RegressionModel

adata_ref = sc.read_h5ad('reference_scrna.h5ad')           # raw counts in .X
adata_ref.obs['cell_type'] = adata_ref.obs['cell_type'].astype('category')

selected = filter_genes(adata_ref, cell_count_cutoff=5, cell_percentage_cutoff2=0.03, nonz_mean_cutoff=1.12)
adata_ref = adata_ref[:, selected].copy()

# batch_key absorbs technical structure across reference samples; drop if a single batch
RegressionModel.setup_anndata(adata_ref, labels_key='cell_type', batch_key='sample')
mod = RegressionModel(adata_ref)
mod.train(max_epochs=250, accelerator='gpu')               # use_gpu= is deprecated; accelerator in {'gpu','cpu','auto'}
adata_ref = mod.export_posterior(adata_ref, sample_kwargs={'num_samples': 1000})

factors = adata_ref.uns['mod']['factor_names']
if 'means_per_cluster_mu_fg' in adata_ref.varm:
    inf_aver = adata_ref.varm['means_per_cluster_mu_fg'][[f'means_per_cluster_mu_fg_{f}' for f in factors]].copy()
else:
    inf_aver = adata_ref.var[[f'means_per_cluster_mu_fg_{f}' for f in factors]].copy()
inf_aver.columns = factors                                 # genes x cell_types signature matrix

cell2location step 2: spatial mapping

Goal: Decompose each spatial spot into absolute cell-type abundances using the reference signatures.

Approach: Restrict both objects to shared genes, set up the Cell2location model with the signature matrix and the expected cells-per-spot, then train. Ncellsperlocation is a tissue-dependent prior (Visium ~10-30, denser tissue higher); detectionalpha controls within-experiment normalization (20 is the tutorial default; raise toward 200 if technical variability in total counts is high).

adata_vis = sc.read_h5ad('visium.h5ad')                    # raw counts
shared = np.intersect1d(adata_vis.var_names, inf_aver.index)
adata_vis = adata_vis[:, shared].copy()
inf_aver = inf_aver.loc[shared, :]

cell2location.models.Cell2location.setup_anndata(adata_vis, batch_key='sample')
mod_sp = cell2location.models.Cell2location(adata_vis, cell_state_df=inf_aver,
                                            N_cells_per_location=30, detection_alpha=20)
mod_sp.train(max_epochs=30000, batch_size=None, train_size=1, accelerator='gpu')
adata_vis = mod_sp.export_posterior(adata_vis, sample_kwargs={'num_samples': 1000, 'batch_size': mod_sp.adata.n_obs})

# q05 = 5% posterior quantile = 'at least this many cells of this type are present' (conservative)
abundance = adata_vis.obsm['q05_cell_abundance_w_sf']      # ABSOLUTE expected cell numbers per spot
abundance.columns = factors
adata_vis.obs[abundance.columns] = abundance.values

cell2location returns absolute abundances. Convert to proportions only if proportions are the question -- doing so discards the information that distinguishes cell2location from the proportion-only methods.

Handling compositional outputs

Goal: Compare cell-type composition across spots, regions, or conditions without the spurious negative correlations that the sum-to-1 constraint manufactures.

Approach: Map proportions out of the simplex with the centered log-ratio (CLR) before any correlation, t-test, or linear model. Add a small pseudocount because CLR is undefined at zero.

import numpy as np

def clr(proportions, pseudocount=1e-6):
    p = proportions + pseudocount
    p = p / p.sum(axis=1, keepdims=True)
    log_p = np.log(p)
    return log_p - log_p.mean(axis=1, keepdims=True)       # subtract per-spot geometric-mean log

abund = adata_vis.obsm['q05_cell_abundance_w_sf'].values
proportions = abund / abund.sum(axis=1, keepdims=True)
clr_comp = clr(proportions)                                # now safe for downstream correlation / DE / t-tests

For differential abundance between conditions, prefer a compositional method (ALDEx2, scCODA, or a Dirichlet-multinomial model) over per-type t-tests on raw fractions; the same closed-data logic that breaks naive correlations breaks naive DA.

Reference-free sanity check

Goal: Detect a missing-reference-type problem and recover composition when no matched scRNA-seq reference exists.

Approach: Run STdeconvolve (LDA topic model, R) on the spatial data alone, then check whether any de novo topic matches no reference cell type -- a topic with strong spatial structure and marker genes for a type absent from the reference is direct evidence the reference is incomplete.

library(STdeconvolve)
counts <- cleanCounts(spatial_counts_matrix, min.lib.size = 100)   # genes x spots
corpus <- restrictCorpus(counts, removeAbove = 1.0, removeBelow = 0.05)
ldas <- fitLDA(t(as.matrix(corpus)), Ks = seq(8, 15))              # K = number of topics, scan a range
opt <- optimalModel(models = ldas, opt = 'min')
res <- getBetaTheta(opt, perc.filt = 0.05)
deconv_prop <- res$theta                                          # spots x topics proportions
# annotate topics post hoc via res$beta (topic gene profiles) against known markers

Validating against marker genes

Goal: Confirm that an estimated cell-type fraction tracks the in-situ expression of that type's canonical markers, not the reference's wishful thinking.

Approach: For each type, correlate its estimated proportion across spots with the mean spatial expression of its marker genes; weak or negative correlation flags a misassignment or a reference mismatch.

markers = {'T_cell': ['CD3D', 'CD3E', 'CD8A'], 'Macrophage': ['CD68', 'CD14', 'CSF1R'], 'Epithelial': ['EPCAM', 'KRT8']}
for ct, genes in markers.items():
    present = [g for g in genes if g in adata_vis.var_names]
    if not present or ct not in proportions_df.columns:
        continue
    expr = np.asarray(adata_vis[:, present].X.mean(axis=1)).ravel()
    corr = np.corrcoef(expr, proportions_df[ct].values)[0, 1]
    print(f'{ct}: marker-vs-proportion r = {corr:.3f}')        # low r -> suspect reference or misassignment

Common Errors

Related Skills

• spatial-domains - group spots into tissue regions; a domain is a region, not a cell type or a niche
• high-resolution-binning - the AMBIGUOUS regime (Visium HD, Stereo-seq, Slide-seq): reconstruct cells vs deconvolve bins
• image-analysis - segment cells from imaging platforms, where deconvolution does not apply
• single-cell/cell-annotation - annotate the imaging cells after segmentation, and label the scRNA-seq reference
• single-cell/preprocessing - build a clean reference; the reference IS the result
• spatial-preprocessing - QC and normalize the spatial data before deconvolution
• spatial-visualization - map estimated proportions and abundances onto the tissue

References

• Kleshchevnikov V, Shmatko A, Dann E, et al. (2022) Cell2location maps fine-grained cell types in spatial transcriptomics. Nature Biotechnology 40:661-671. DOI 10.1038/s41587-021-01139-4
• Cable DM, Murray E, Zou LS, et al. (2022) Robust decomposition of cell type mixtures in spatial transcriptomics (RCTD). Nature Biotechnology 40:517-526. DOI 10.1038/s41587-021-00830-w
• Andersson A, Bergenstrahle J, Asp M, et al. (2020) Single-cell and spatial transcriptomics enables probabilistic inference of cell type topography (stereoscope). Communications Biology 3:565. DOI 10.1038/s42003-020-01247-y
• Elosua-Bayes M, Nieto P, Mereu E, Gut I, Heyn H (2021) SPOTlight: seeded NMF regression to deconvolute spatial transcriptomics spots with single-cell transcriptomes. Nucleic Acids Research 49(9):e50. DOI 10.1093/nar/gkab043
• Biancalani T, Scalia G, Buffoni L, et al. (2021) Deep learning and alignment of spatially resolved single-cell transcriptomes with Tangram. Nature Methods 18(11):1352-1362. DOI 10.1038/s41592-021-01264-7
• Ma Y, Zhou X (2022) Spatially informed cell-type deconvolution for spatial transcriptomics (CARD). Nature Biotechnology 40:1349-1359. DOI 10.1038/s41587-022-01273-7
• Dong R, Yuan GC (2021) SpatialDWLS: accurate deconvolution of spatial transcriptomic data. Genome Biology 22:145. DOI 10.1186/s13059-021-02362-7
• Miller BF, Huang F, Atta L, Sahoo A, Fan J (2022) Reference-free cell type deconvolution of multi-cellular pixel-resolution spatially resolved transcriptomics data (STdeconvolve). Nature Communications 13:2339. DOI 10.1038/s41467-022-30033-z
• Sang-Aram C, Browaeys R, Seurinck R, Saeys Y (2024) Spotless, a reproducible pipeline for benchmarking cell type deconvolution in spatial transcriptomics. eLife 12:RP88431. DOI 10.7554/eLife.88431
• Li B, Zhang W, Guo C, et al. (2022) Benchmarking spatial and single-cell transcriptomics integration methods for transcript distribution prediction and cell type deconvolution. Nature Methods 19:662-670. DOI 10.1038/s41592-022-01480-9