single_cell/process_integrate_annotate

Description

A pipeline to process, integrate and annotate single cell (multi-)omics data.
Available integration methods:

Harmony
scVI
Scanorama
BBKNN
Available annotation methods:
CellTypist
Harmony integration followed by KNN label transfer (harmony_knn)
scANVI (with scArches)
scVI integration followed by KNN label transfer (scvi_knn)
SingleR
When more than one annotation method is selected, a final consensus vote
combines the per-method predictions into a single consensus label.

Input (query) data arguments

Name	Type & Properties	Description
--id	string required	ID of the sample.
--input	file required	Input query dataset(s) to be annotated
--modality	string	Modality to be processed. Should match the modality in the --reference dataset, if provided.
--input_layer	string	The layer in the input data containing the raw RNA counts, if .X is not to be used.
--prot_layer	string	Input layer for the antibody capture (CITE-seq) modality, if .X is not to be used.
--gdo_layer	string	Input layer for the guide-derived oligonucleotide (GDO) modality, if .X is not to be used.
--input_var_gene_names	string	The name of the adata var column containing gene names; when no gene_name_layer is provided, the var index will be used.
--input_reference_gene_overlap	integer	The minimum number of genes present in both the reference and query datasets.
--sanitize_ensembl_ids	boolean	Whether to sanitize ensembl gene ids by removing version numbers. Read by the celltypist, scanvi_scarches, scvi_knn and singler annotation methods.
--input_obs_categorical_covariates	string multiple	.obs column(s) in the input query with categorical covariates to correct for in addition to the batch label. Used by the scvi integration method and by the scanvi_scarches annotation method. Treated as nuisance factors, so do not use them for biological signal you want to keep. For scanvi_scarches their order must match --reference_obs_categorical_covariates.
--input_obs_numerical_covariates	string multiple	.obs column(s) in the input query with continuous (numerical) covariates to correct for in addition to the batch label. Used by the scvi integration method and by the scanvi_scarches annotation method. Treated as nuisance factors, so do not use them for biological signal you want to keep. For scanvi_scarches their order must match --reference_obs_numerical_covariates.

Name

Type & Properties

--id

string

required

--input

file

required

--modality

string

--input_layer

string

--prot_layer

string

--gdo_layer

string

--input_var_gene_names

string

--input_reference_gene_overlap

integer

--sanitize_ensembl_ids

boolean

--input_obs_categorical_covariates

string

multiple

--input_obs_numerical_covariates

string

multiple

Reference data arguments

Name	Type & Properties	Description
--reference	file	The reference dataset in .h5mu format to be used as a reference mapper and to train annotation algorithms on.
--reference_layer_raw_counts	string	The layer in the reference dataset containing the raw counts, if .X is not to be used.
--reference_layer_lognormalized_counts	string	The layer in the reference dataset containing the log-normalized counts, if .X is not to be used.
--reference_var_gene_names	string	The name of the adata .var column containing gene names if the .var index is not to be used.
--reference_obs_batch	string	The .obs column of the reference dataset containing the batch information.
--reference_obs_label	string	The `.obs` key of the target labels to transfer.
--reference_obs_label_unlabeled_category	string	Value in the --reference_obs_label field that indicates unlabeled observations
--reference_var_input	string	.var column containing highly variable genes. By default, do not subset genes.
--reference_obs_categorical_covariates	string multiple	.obs column(s) in the reference with categorical covariates for the scanvi_scarches annotation method to correct for in addition to the batch label. Treated as nuisance factors; their order must match --input_obs_categorical_covariates.
--reference_obs_numerical_covariates	string multiple	.obs column(s) in the reference with continuous (numerical) covariates for the scanvi_scarches annotation method to correct for in addition to the batch label. Treated as nuisance factors; their order must match --input_obs_numerical_covariates.

Name

Type & Properties

--reference

file

--reference_layer_raw_counts

string

--reference_layer_lognormalized_counts

string

--reference_var_gene_names

string

--reference_obs_batch

string

--reference_obs_label

string

--reference_obs_label_unlabeled_category

string

--reference_var_input

string

--reference_obs_categorical_covariates

string

multiple

--reference_obs_numerical_covariates

string

multiple

Methods

Name	Type & Properties	Description
--integration_methods	string multiple	Integration methods to be executed.
--annotation_methods	string multiple	Annotation methods to be executed.

Name

Type & Properties

--integration_methods

string

multiple

--annotation_methods

string

multiple

Pre-processing options: RNA filtering

Name	Type & Properties	Description
--rna_min_counts	integer	Minimum number of counts captured per cell.
--rna_max_counts	integer	Maximum number of counts captured per cell.
--rna_min_genes_per_cell	integer	Minimum of non-zero values per cell.
--rna_max_genes_per_cell	integer	Maximum of non-zero values per cell.
--rna_min_cells_per_gene	integer	Minimum of non-zero values per gene.
--rna_min_fraction_mito	double	Minimum fraction of UMIs that are mitochondrial.
--rna_max_fraction_mito	double	Maximum fraction of UMIs that are mitochondrial.
--rna_min_fraction_ribo	double	Minimum fraction of UMIs that are ribosomal.
--rna_max_fraction_ribo	double	Maximum fraction of UMIs that are ribosomal.
--skip_scrublet_doublet_detection	boolean_true	Skip the scrublet doublet detection step.

Name

Type & Properties

--rna_min_counts

integer

--rna_max_counts

integer

--rna_min_genes_per_cell

integer

--rna_max_genes_per_cell

integer

--rna_min_cells_per_gene

integer

--rna_min_fraction_mito

double

--rna_max_fraction_mito

double

--rna_min_fraction_ribo

double

--rna_max_fraction_ribo

double

--skip_scrublet_doublet_detection

boolean_true

Pre-processing options: Protein (CITE-seq) filtering

Name	Type & Properties	Description
--prot_min_counts	integer	Minimum number of counts captured per cell.
--prot_max_counts	integer	Maximum number of counts captured per cell.
--prot_min_proteins_per_cell	integer	Minimum of non-zero values per cell.
--prot_max_proteins_per_cell	integer	Maximum of non-zero values per cell.
--prot_min_cells_per_protein	integer	Minimum of non-zero values per protein.

Name

Type & Properties

--prot_min_counts

integer

--prot_max_counts

integer

--prot_min_proteins_per_cell

integer

--prot_max_proteins_per_cell

integer

--prot_min_cells_per_protein

integer

Pre-processing options: GDO filtering

Name	Type & Properties	Description
--gdo_min_counts	integer	Minimum number of counts captured per cell.
--gdo_max_counts	integer	Maximum number of counts captured per cell.
--gdo_min_guides_per_cell	integer	Minimum of non-zero values per cell.
--gdo_max_guides_per_cell	integer	Maximum of non-zero values per cell.
--gdo_min_cells_per_guide	integer	Minimum of non-zero values per guide.

Name

Type & Properties

--gdo_min_counts

integer

--gdo_max_counts

integer

--gdo_min_guides_per_cell

integer

--gdo_max_guides_per_cell

integer

--gdo_min_cells_per_guide

integer

Pre-processing options: Cross-modality filtering

Name	Type & Properties	Description
--intersect_obs	boolean_true	After per-modality filtering and multisample processing, remove observations that are not present in all processed modalities so that each modality shares the same set of cells.

Name

Type & Properties

--intersect_obs

boolean_true

Pre-processing options: Sample ID handling

Name	Type & Properties	Description
--add_id_make_observation_keys_unique	boolean	Join the sample id to the .obs index (.obs_names) to make observation keys unique across samples.

Name

Type & Properties

--add_id_make_observation_keys_unique

boolean

Pre-processing options: Mitochondrial & Ribosomal Gene Detection

Name	Type & Properties	Description
--var_name_mitochondrial_genes	string	In which .var slot to store a boolean array corresponding the mitochondrial genes.
--var_name_ribosomal_genes	string	In which .var slot to store a boolean array corresponding the ribosomal genes.
--obs_name_mitochondrial_fraction	string	When specified, write the fraction of counts originating from mitochondrial genes (based on --mitochondrial_gene_regex) to an .obs column with the specified name. Requires --var_name_mitochondrial_genes.
--obs_name_ribosomal_fraction	string	When specified, write the fraction of counts originating from ribosomal genes (based on --ribosomal_gene_regex) to an .obs column with the specified name. Requires --var_name_ribosomal_genes.
--mitochondrial_gene_regex	string	Regex string that identifies mitochondrial genes from --var_gene_names. By default will detect human and mouse mitochondrial genes from a gene symbol.
--ribosomal_gene_regex	string	Regex string that identifies ribosomal genes from --var_gene_names. By default will detect human and mouse ribosomal genes from a gene symbol.

Name

Type & Properties

--var_name_mitochondrial_genes

string

--var_name_ribosomal_genes

string

--obs_name_mitochondrial_fraction

string

--obs_name_ribosomal_fraction

string

--mitochondrial_gene_regex

string

--ribosomal_gene_regex

string

Pre-processing options: QC metrics calculation options

Name	Type & Properties	Description
--var_qc_metrics	string multiple	Keys to select a boolean (containing only True or False) column from .var. For each cell, calculate the proportion of total values for genes which are labeled 'True', compared to the total sum of the values for all genes. Defaults to the combined values specified for --var_name_mitochondrial_genes and --highly_variable_features_var_output.
--top_n_vars	integer multiple	Number of top vars to be used to calculate cumulative proportions. If not specified, proportions are not calculated.

Name

Type & Properties

--var_qc_metrics

string

multiple

--top_n_vars

integer

multiple

Pre-processing options: Highly variable features detection

Name	Type & Properties	Description
--highly_variable_features_obs_batch_key	string	If specified, highly-variable genes are selected within each batch separately and merged. This avoids the selection of batch-specific genes and acts as a lightweight batch correction method.

Name

Type & Properties

--highly_variable_features_obs_batch_key

string

Pre-processing options: CLR (protein normalization)

Name	Type & Properties	Description
--clr_axis	integer	Axis to perform the CLR transformation on.

Name

Type & Properties

--clr_axis

integer

Pre-processing options: RNA scaling

Name	Type & Properties	Description
--rna_enable_scaling	boolean_true	Enable scaling for the RNA modality.
--rna_scaling_output_layer	string	Output layer where the scaled log-normalized data will be stored.
--rna_scaling_max_value	double	Clip (truncate) data to this value after scaling. If not specified, do not clip.
--rna_scaling_zero_center	boolean_false	If set, omit zero-centering variables, which allows to handle sparse input efficiently.
--rna_scaling_pca_obsm_output	string	Name of the .obsm key where the PCA representation of the scaled data is stored.
--rna_scaling_pca_loadings_varm_output	string	Name of the .varm key where the PCA loadings of the scaled data are stored.
--rna_scaling_pca_variance_uns_output	string	Name of the .uns key where the variance and variance ratio of the scaled data are stored as a map (keys: variance and variance_ratio).
--rna_scaling_umap_obsm_output	string	Name of the .obsm key where the UMAP representation of the scaled data is stored.

Name

Type & Properties

--rna_enable_scaling

boolean_true

--rna_scaling_output_layer

string

--rna_scaling_max_value

double

--rna_scaling_zero_center

boolean_false

--rna_scaling_pca_obsm_output

string

--rna_scaling_pca_loadings_varm_output

string

--rna_scaling_pca_variance_uns_output

string

--rna_scaling_umap_obsm_output

string

Harmony integration options

Name	Type & Properties	Description
--harmony_theta	double multiple	Diversity clustering penalty parameter. Specify for each variable in group.by.vars. theta=0 does not encourage any diversity. Larger values of theta result in more diverse clusters."
--harmony_obs_covariates	string required multiple	The .obs field(s) that define the covariate(s) to regress out.

Name

Type & Properties

--harmony_theta

double

multiple

--harmony_obs_covariates

string

required

multiple

Scanorama integration options

Name	Type & Properties	Description
--scanorama_knn	integer	Number of nearest neighbors used for matching.
--scanorama_batch_size	integer	Batch size used in the alignment vector computation.
--scanorama_sigma	double	Correction smoothing parameter on the Gaussian kernel.
--scanorama_approx	boolean	Use approximate nearest neighbors (annoy).
--scanorama_alpha	double	Alignment score minimum cutoff.

Name

Type & Properties

--scanorama_knn

integer

--scanorama_batch_size

integer

--scanorama_sigma

double

--scanorama_approx

boolean

--scanorama_alpha

double

BBKNN integration options

Name	Type & Properties	Description
--bbknn_n_neighbors_within_batch	integer	Top neighbours to report for each batch.
--bbknn_n_pcs	integer	Number of dimensions to use from the input embedding.
--bbknn_n_trim	integer	Trim each cell's neighbours to these many top connectivities. Set to 0 to skip; leave unset to use the BBKNN default (10 * n_neighbors_within_batch * n_batches).

Name

Type & Properties

--bbknn_n_neighbors_within_batch

integer

--bbknn_n_pcs

integer

--bbknn_n_trim

integer

scVI, scANVI and scArches training options

Name	Type & Properties	Description
--early_stopping	boolean	Whether to perform early stopping with respect to the validation set.
--early_stopping_monitor	string	Metric logged during validation set epoch.
--early_stopping_patience	integer	Number of validation epochs with no improvement after which training will be stopped.
--early_stopping_min_delta	double	Minimum change in the monitored quantity to qualify as an improvement, i.e. an absolute change of less than min_delta, will count as no improvement.
--max_epochs	integer	Number of passes through the dataset, defaults to (20000 / number of cells) * 400 or 400; whichever is smallest.
--reduce_lr_on_plateau	boolean	Whether to monitor validation loss and reduce learning rate when validation set `lr_scheduler_metric` plateaus.
--lr_factor	double	Factor to reduce learning rate.
--lr_patience	double	Number of epochs with no improvement after which learning rate will be reduced.

Name

Type & Properties

--early_stopping

boolean

--early_stopping_monitor

string

--early_stopping_patience

integer

--early_stopping_min_delta

double

--max_epochs

integer

--reduce_lr_on_plateau

boolean

--lr_factor

double

--lr_patience

double

CellTypist reference model

Name	Type & Properties	Description
--celltypist_model	file	Pretrained model in pkl format. If not provided, the model will be trained on the reference data and --reference should be provided.

Name

Type & Properties

--celltypist_model

file

CellTypist annotation options

Name	Type & Properties	Description
--celltypist_feature_selection	boolean	Whether to perform feature selection.
--celltypist_majority_voting	boolean	Whether to refine the predicted labels by running the majority voting classifier after over-clustering.
--celltypist_C	double	Inverse of regularization strength in logistic regression.
--celltypist_max_iter	integer	Maximum number of iterations before reaching the minimum of the cost function.
--celltypist_use_SGD	boolean_true	Whether to use the stochastic gradient descent algorithm.
--celltypist_min_prop	double	"For the dominant cell type within a subcluster, the minimum proportion of cells required to support naming of the subcluster by this cell type. Ignored if majority_voting is set to False. Subcluster that fails to pass this proportion threshold will be assigned 'Heterogeneous'."

Name

Type & Properties

--celltypist_feature_selection

boolean

--celltypist_majority_voting

boolean

--celltypist_C

double

--celltypist_max_iter

integer

--celltypist_use_SGD

boolean_true

--celltypist_min_prop

double

harmony_knn annotation options

Name	Type & Properties	Description
--harmony_knn_n_hvg	integer	Number of highly-variable genes to keep for harmony_knn.
--harmony_knn_pca_num_components	integer	Number of principal components to compute for harmony_knn.
--harmony_knn_theta	double multiple	Diversity clustering penalty for harmony.

Name

Type & Properties

--harmony_knn_n_hvg

integer

--harmony_knn_pca_num_components

integer

--harmony_knn_theta

double

multiple

scvi_knn annotation options

Name	Type & Properties	Description
--scvi_knn_n_hvg	integer	Number of highly-variable genes to keep for scvi_knn.

Name

Type & Properties

--scvi_knn_n_hvg

integer

SingleR annotation options

Name	Type & Properties	Description
--singler_input_obs_clusters	string	.obs column with cluster identities. If provided, SingleR annotates the aggregated cluster profiles; otherwise it annotates per observation.
--singler_de_method	string	Method to detect differentially expressed genes between pairs of labels.
--singler_de_n_genes	integer	Number of differentially expressed genes per label pair to use as markers. Defaults to a SingleR heuristic based on the number of labels.
--singler_quantile	double	Quantile of the correlation distribution used to score each label.
--singler_fine_tune	boolean	Whether to fine-tune label assignment after the initial classification.
--singler_fine_tuning_threshold	double	Maximum difference from the maximum correlation to use in fine-tuning.
--singler_prune	boolean	Whether to prune low-quality labels (replaced with NA in the pruned predictions output).

Name

Type & Properties

--singler_input_obs_clusters

string

--singler_de_method

string

--singler_de_n_genes

integer

--singler_quantile

double

--singler_fine_tune

boolean

--singler_fine_tuning_threshold

double

--singler_prune

boolean

Clustering options

Name	Type & Properties	Description
--leiden_resolution	double multiple	Control the coarseness of the clustering. Higher values lead to more clusters.

Name

Type & Properties

--leiden_resolution

double

multiple

Neighbor classifier arguments

Name	Type & Properties	Description
--knn_weights	string	Weight function used in prediction. Possible values are: `uniform` (all points in each neighborhood are weighted equally) or `distance` (weight points by the inverse of their distance)
--knn_n_neighbors	integer	The number of neighbors to use in k-neighbor graph structure used for fast approximate nearest neighbor search with PyNNDescent. Larger values will result in more accurate search results at the cost of computation time.

Name

Type & Properties

--knn_weights

string

--knn_n_neighbors

integer

Consensus vote options

Name	Type & Properties	Description
--run_consensus	boolean	Run a final consensus vote combining the per-method predictions. Only takes effect when more than one --annotation_methods is selected; with a single method there is nothing to combine and the step is skipped.
--consensus_obs_predictions	string	.obs slot in which to store the consensus predicted cell type.
--consensus_obs_score	string	.obs slot in which to store the consensus score, defined as the fraction of total weight assigned to the winning cell type.
--consensus_use_probabilities	boolean	Scale each method's vote by its per-cell prediction probability. When false, every method votes with equal weight regardless of confidence.
--consensus_tie_label	string	Label to assign when two or more cell types receive equal votes. If not provided, tied cells are assigned a missing value.

Name

Type & Properties

--run_consensus

boolean

--consensus_obs_predictions

string

--consensus_obs_score

string

--consensus_use_probabilities

boolean

--consensus_tie_label

string

Outputs

Name	Type & Properties	Description
--output	file required output	The output file.
--output_compression	string	Compression format for the output h5mu file.
--output_scvi_model	file output	Output folder with the trained scVI model. Only produced when scvi is among the selected --integration_methods.
--output_scanvi_model	file output	Output folder with the trained scANVI/scArches model. Only produced when scanvi_scarches is among the selected --annotation_methods.

Name

Type & Properties

--output

file

required

output

--output_compression

string

--output_scvi_model

file

output

--output_scanvi_model

file

output

Run this component

Run the following command to execute this component with Nextflow:

cat > params.yaml <<'EOM'  
id: "run"  
modality: [ "rna" ]  
input_reference_gene_overlap: [ 100 ]  
sanitize_ensembl_ids: [ true ]  
reference_layer_lognormalized_counts: [ "log_normalized" ]  
reference_obs_batch: [ "sample_id" ]  
reference_obs_label_unlabeled_category: [ "Unknown" ]  
reference_var_input: [ "filter_with_hvg" ]  
add_id_make_observation_keys_unique: [ true ]  
mitochondrial_gene_regex: [ "^[mM][tT]-" ]  
ribosomal_gene_regex: [ "^[Mm]?[Rr][Pp][LlSs]" ]  
top_n_vars: [ 50, 100, 200, 500 ]  
highly_variable_features_obs_batch_key: [ "sample_id" ]  
clr_axis: [ 0 ]  
rna_scaling_output_layer: [ "scaled" ]  
rna_scaling_pca_obsm_output: [ "scaled_pca" ]  
rna_scaling_pca_loadings_varm_output: [ "scaled_pca_loadings" ]  
rna_scaling_pca_variance_uns_output: [ "scaled_pca_variance" ]  
rna_scaling_umap_obsm_output: [ "scaled_umap" ]  
harmony_theta: [ 2 ]  
harmony_obs_covariates: [ "sample_id" ]  
scanorama_knn: [ 20 ]  
scanorama_batch_size: [ 5000 ]  
scanorama_sigma: [ 15 ]  
scanorama_approx: [ true ]  
scanorama_alpha: [ 0.1 ]  
bbknn_n_neighbors_within_batch: [ 3 ]  
bbknn_n_pcs: [ 50 ]  
early_stopping_monitor: [ "elbo_validation" ]  
early_stopping_patience: [ 45 ]  
early_stopping_min_delta: [ 0 ]  
reduce_lr_on_plateau: [ true ]  
lr_factor: [ 0.6 ]  
lr_patience: [ 30 ]  
celltypist_feature_selection: [ false ]  
celltypist_majority_voting: [ false ]  
celltypist_C: [ 1 ]  
celltypist_max_iter: [ 1000 ]  
celltypist_min_prop: [ 0 ]  
harmony_knn_n_hvg: [ 2000 ]  
harmony_knn_theta: [ 2 ]  
scvi_knn_n_hvg: [ 2000 ]  
singler_de_method: [ "classic" ]  
singler_quantile: [ 0.8 ]  
singler_fine_tune: [ true ]  
singler_fine_tuning_threshold: [ 0.05 ]  
singler_prune: [ true ]  
leiden_resolution: [ 1 ]  
knn_weights: [ "uniform" ]  
knn_n_neighbors: [ 15 ]  
run_consensus: [ true ]  
consensus_obs_predictions: [ "consensus_pred" ]  
consensus_obs_score: [ "consensus_score" ]  
consensus_use_probabilities: [ true ]  
output: "$id.$key.output.h5mu"  
output_scvi_model: "$id.$key.output_scvi_model.scvi_model"  
output_scanvi_model: "$id.$key.output_scanvi_model.scanvi_model"  
publish_dir: "output/"  
EOM

nextflow run https://packages.viash-hub.com/vsh/openpipeline_composed.git \  
  -revision v0.3.0 \  
  -main-script target/nextflow/single_cell/process_integrate_annotate/main.nf \  
  -params-file params.yaml

single_cell/process_integrate_annotate

Input (query) data arguments

Reference data arguments

Methods

Pre-processing options: RNA filtering

Pre-processing options: Protein (CITE-seq) filtering

Pre-processing options: GDO filtering

Pre-processing options: Cross-modality filtering

Pre-processing options: Sample ID handling

Pre-processing options: Mitochondrial & Ribosomal Gene Detection

Pre-processing options: QC metrics calculation options

Pre-processing options: Highly variable features detection

Pre-processing options: CLR (protein normalization)

Pre-processing options: RNA scaling

Harmony integration options

Scanorama integration options

BBKNN integration options

scVI, scANVI and scArches training options

CellTypist reference model

CellTypist annotation options

harmony_knn annotation options

scvi_knn annotation options

SingleR annotation options

Clustering options

Neighbor classifier arguments

Consensus vote options

Outputs

Run this component

Relationships

Used by

Uses