{
"cells": [
{
"cell_type": "markdown",
"id": "63f90e61-f6a1-440f-8745-b044e79e3261",
"metadata": {},
"source": [
"## Demonstration of Single-Cell Benchmark Tasks\n",
"### Using cz-benchmarks to evaluate a model on multiple tasks\n",
"\n",
"This notebook demonstrates how to leverage the `czbenchmarks` library to evaluate single-cell models on multiple benchmark tasks. `czbenchmarks` is a Python library designed to streamline benchmarking for single-cell analysis tasks, such as clustering, embedding, and cell type classification. \n",
"\n",
"\n",
"For demonstration purposes, we use `scVI`and showing performance of the published model weights for each of the following tasks: \n",
"* `Clustering`\n",
"* `Embedding`\n",
"* `Cell Type Classification`\n",
"\n",
"### Step 1: Setup and Imports\n",
"\n",
"This step sets up the environment and imports necessary libraries for benchmarking single-cell models. Ensure that the virtual environment is created and dependencies are installed before proceeding."
]
},
{
"cell_type": "code",
"execution_count": 74,
"id": "75fb3046-dce1-4eac-bb20-0a6448f88b86",
"metadata": {},
"outputs": [],
"source": [
"# # Create isolated virtual environment for scVI and czbenchmarks (run once)\n",
"\n",
"# !python3 -m venv .venv_scvi\n",
"\n",
"# # Install model required packages\n",
"# !.venv_scvi/bin/python -m pip install --upgrade pip\n",
"# !.venv_scvi/bin/python -m pip install ipykernel numpy pandas scvi-tools tabulate matplotlib seaborn\n",
"\n",
"# # Register the new environment as a Jupyter kernel (if not already registered)\n",
"# !.venv_scvi/bin/python -m ipykernel install --user --name venv_scvi --display-name \"Python (.venv_scvi)\"\n",
"\n",
"# print(\"Virtual environment '.venv_scvi' created, dependencies installed, and kernel registered.\")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "9d6c4dd6-c757-49f2-abbe-11497a0dfce1",
"metadata": {},
"outputs": [],
"source": [
"from czbenchmarks.datasets import load_dataset\n",
"from czbenchmarks.datasets.single_cell_labeled import SingleCellLabeledDataset\n",
"from czbenchmarks.tasks import (\n",
" ClusteringTask,\n",
" EmbeddingTask,\n",
" MetadataLabelPredictionTask,\n",
")\n",
"from czbenchmarks.tasks.clustering import ClusteringTaskInput\n",
"from czbenchmarks.tasks.embedding import EmbeddingTaskInput\n",
"from czbenchmarks.tasks.label_prediction import MetadataLabelPredictionTaskInput\n",
"\n",
"# Model specific imports\n",
"import scvi # other imports can be used as required by model\n",
"import functools\n",
"\n",
"# --- Visualization Imports ---\n",
"import matplotlib.pyplot as plt\n",
"import pandas as pd\n",
"import seaborn as sns\n",
"\n",
"# Notebook config imports\n",
"import warnings\n",
"\n",
"warnings.simplefilter(\"ignore\")\n",
"sns.set_theme(style=\"whitegrid\")\n",
"\n",
"\n",
"# utility functions\n",
"def summarize_results(results, source):\n",
" \"\"\"Summarize results into a DataFrame for easy comparison.\"\"\"\n",
" return pd.DataFrame(\n",
" {\n",
" \"Source\": source,\n",
" \"Metric\": [r.metric_type.name for r in results],\n",
" \"Value\": [r.value for r in results],\n",
" }\n",
" )\n",
"\n",
"\n",
"def plot_umap(embedding, labels, title):\n",
" import scanpy as sc\n",
"\n",
" adata_vis = sc.AnnData(embedding)\n",
" adata_vis.obs[\"cell_type\"] = labels.values.astype(str)\n",
" sc.pp.neighbors(adata_vis, use_rep=\"X\")\n",
" sc.tl.umap(adata_vis)\n",
" sc.pl.umap(adata_vis, color=\"cell_type\", title=title, frameon=False, show=False)\n",
"\n",
"\n",
"def plot_comparison(data, title, ylabel, palette):\n",
" plt.figure(figsize=(10, 5))\n",
" sns.barplot(data=data, x=\"Metric\", y=\"Value\", hue=\"Source\", palette=palette)\n",
" plt.title(title, fontsize=14)\n",
" plt.ylabel(ylabel, fontsize=10)\n",
" plt.xlabel(\"Metric\", fontsize=10)\n",
" plt.xticks(rotation=0, ha=\"right\")\n",
" plt.legend(title=\"Source\")\n",
" plt.tight_layout()\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"id": "548178de-86f1-4066-82eb-770c365dc3af",
"metadata": {},
"source": [
"### Step 2: Load a Dataset\n",
"\n",
"Load the pre-configured `tsv2_prostate` dataset. The library handles automatic download, caching, and loading as a `SingleCellLabeledDataset` for streamlined reuse.\n",
"\n",
"**Loaded dataset provides:**\n",
"- `dataset.adata`: AnnData object with gene expression data.\n",
"- `dataset.labels`: pandas Series of cell type labels."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "34e11f12-12d7-4a20-a2fb-0f96a45fba1f",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:czbenchmarks.file_utils:File already exists in cache: /Users/sgupta/.cz-benchmarks/datasets/homo_sapiens_10df7690-6d10-4029-a47e-0f071bb2df83_Prostate_v2_curated.h5ad\n",
"INFO:czbenchmarks.datasets.single_cell:Loading dataset from /Users/sgupta/.cz-benchmarks/datasets/homo_sapiens_10df7690-6d10-4029-a47e-0f071bb2df83_Prostate_v2_curated.h5ad in memory mode.\n"
]
},
{
"data": {
"text/plain": [
"TSP25_Prostate_NA_10X_1_1_AAACCCAAGTGGTTAA endothelial cell\n",
"TSP25_Prostate_NA_10X_1_1_AAACCCACATGCACTA luminal cell of prostate epithelium\n",
"TSP25_Prostate_NA_10X_1_1_AAACGAAGTTCTGACA endothelial cell\n",
"TSP25_Prostate_NA_10X_1_1_AAACGCTTCTACCCAC erythrocyte\n",
"TSP25_Prostate_NA_10X_1_1_AAAGAACCAGTTGTCA smooth muscle cell\n",
" ... \n",
"TSP25_Prostate_NA_10X_1_2_TTTATGCTCTTGGTCC fibroblast\n",
"TSP25_Prostate_NA_10X_1_2_TTTCACAAGATCGGTG basal cell of prostate epithelium\n",
"TSP25_Prostate_NA_10X_1_2_TTTCACAGTGCCTTCT fibroblast\n",
"TSP25_Prostate_NA_10X_1_2_TTTCATGCAATAGTAG CD8-positive, alpha-beta T cell\n",
"TSP25_Prostate_NA_10X_1_2_TTTCCTCAGGTGATCG fibroblast\n",
"Name: cell_type, Length: 2044, dtype: category\n",
"Categories (14, object): ['fibroblast', 'T cell', 'mast cell', 'endothelial cell', ..., 'neutrophil', 'mature NK T cell', 'luminal cell of prostate epithelium', 'basal cell of prostate epithelium']"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# The 'dataset' object is a validated AnnData wrapper, ensuring efficient downstream processing.\n",
"dataset: SingleCellLabeledDataset = load_dataset(\"tsv2_prostate\")\n",
"dataset.adata\n",
"dataset.labels"
]
},
{
"cell_type": "markdown",
"id": "7f803094",
"metadata": {},
"source": [
"#### Optionally Transform Data\n",
"\n",
"After loading the dataset, you may need to transform the data to meet the requirements of your model."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "7da9bbce",
"metadata": {},
"outputs": [],
"source": [
"required_obs_keys = [\"dataset_id\", \"assay\", \"suspension_type\", \"donor_id\"]\n",
"adata = dataset.adata.copy()\n",
"\n",
"batch_keys = required_obs_keys\n",
"adata.obs[\"batch\"] = functools.reduce(\n",
" lambda a, b: a + b, [adata.obs[c].astype(str) for c in batch_keys]\n",
")"
]
},
{
"cell_type": "markdown",
"id": "23e4b045-5c5b-4277-9bc5-98d4f3926dcd",
"metadata": {},
"source": [
"### Step 3: Model Inference\n",
"Obtain the model weights associated with the pre-trained scVI model to use as a reference. Generate cell embeddings for evaluation within the benchmarking framework.\n",
"\n",
"---\n",
"> **Note:** Ensure the model weights directory exists and contains the required `model.pt` file. If the weights are missing, refer to `examples/scvi_model_dev_workflow.ipynb` for instructions on obtaining them."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "0bd77a63-74a0-4d0f-8aba-ba2cae56cb08",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[34mINFO \u001b[0m File czbenchmarks_scvi_model/model.pt already downloaded \n",
"\u001b[34mINFO \u001b[0m Found \u001b[1;36m44.05\u001b[0m% reference vars in query data. \n",
"\u001b[34mINFO \u001b[0m File czbenchmarks_scvi_model/model.pt already downloaded \n",
"Generated scVI embedding with shape: (2044, 50)\n"
]
}
],
"source": [
"# Ensure model weights directory exists and model.pt file is present. Check `examples/scvi_model_dev_workflow.ipynb` for details on how to get the model weights.\n",
"model_weights_dir = \"czbenchmarks_scvi_model\"\n",
"scvi.model.SCVI.prepare_query_anndata(adata, model_weights_dir)\n",
"scvi_model = scvi.model.SCVI.load_query_data(adata, model_weights_dir)\n",
"scvi_model.is_trained = True\n",
"\n",
"# Now, generate the latent representation (the embedding)\n",
"model_output = scvi_model.get_latent_representation()\n",
"print(f\"Generated scVI embedding with shape: {model_output.shape}\")"
]
},
{
"cell_type": "markdown",
"id": "89d4a60a",
"metadata": {},
"source": [
"### Step 4. Benchmark Tasks\n",
"\n",
"Evaluate the scVI model and PCA baseline across clustering, embedding, and cell type classification tasks."
]
},
{
"cell_type": "markdown",
"id": "aa27b5d4-cd94-4d17-a5aa-c071287acef3",
"metadata": {},
"source": [
"#### Clustering Task\n",
"\n",
"Evaluate clustering performance using metrics such as Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI)."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "98829130-4a6c-44bf-b2c4-35f3218519e4",
"metadata": {},
"outputs": [],
"source": [
"clustering_task = ClusteringTask()\n",
"clustering_task_input = ClusteringTaskInput(\n",
" obs=dataset.adata.obs, input_labels=dataset.labels\n",
")\n",
"\n",
"# Run clustering task\n",
"clustering_results = clustering_task.run(\n",
" cell_representation=model_output, task_input=clustering_task_input\n",
")\n",
"\n",
"# Compute baseline\n",
"expression_data = dataset.adata.X\n",
"clustering_baseline = clustering_task.compute_baseline(expression_data)\n",
"clustering_baseline_results = clustering_task.run(\n",
" cell_representation=clustering_baseline, task_input=clustering_task_input\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "917eb14c",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Clustering Results Comparison:\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" 0 | \n",
" 1 | \n",
" 2 | \n",
" 3 | \n",
"
\n",
" \n",
" \n",
" \n",
" | Source | \n",
" scVI Model | \n",
" scVI Model | \n",
" PCA Baseline | \n",
" PCA Baseline | \n",
"
\n",
" \n",
" | Metric | \n",
" ADJUSTED_RAND_INDEX | \n",
" NORMALIZED_MUTUAL_INFO | \n",
" ADJUSTED_RAND_INDEX | \n",
" NORMALIZED_MUTUAL_INFO | \n",
"
\n",
" \n",
" | Value | \n",
" 0.71937 | \n",
" 0.86654 | \n",
" 0.642149 | \n",
" 0.833138 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" Source | \n",
" Metric | \n",
" Value | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" scVI Model | \n",
" SILHOUETTE_SCORE | \n",
" 0.627913 | \n",
"
\n",
" \n",
" | 1 | \n",
" PCA Baseline | \n",
" SILHOUETTE_SCORE | \n",
" 0.650182 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" 0 | \n",
" 1 | \n",
" 2 | \n",
" 3 | \n",
" 4 | \n",
" 5 | \n",
" 6 | \n",
" 7 | \n",
" 8 | \n",
" 9 | \n",
" ... | \n",
" 30 | \n",
" 31 | \n",
" 32 | \n",
" 33 | \n",
" 34 | \n",
" 35 | \n",
" 36 | \n",
" 37 | \n",
" 38 | \n",
" 39 | \n",
"
\n",
" \n",
" \n",
" \n",
" | Source | \n",
" scVI Model | \n",
" scVI Model | \n",
" scVI Model | \n",
" scVI Model | \n",
" scVI Model | \n",
" scVI Model | \n",
" scVI Model | \n",
" scVI Model | \n",
" scVI Model | \n",
" scVI Model | \n",
" ... | \n",
" PCA Baseline | \n",
" PCA Baseline | \n",
" PCA Baseline | \n",
" PCA Baseline | \n",
" PCA Baseline | \n",
" PCA Baseline | \n",
" PCA Baseline | \n",
" PCA Baseline | \n",
" PCA Baseline | \n",
" PCA Baseline | \n",
"
\n",
" \n",
" | Metric | \n",
" MEAN_FOLD_ACCURACY | \n",
" MEAN_FOLD_F1_SCORE | \n",
" MEAN_FOLD_PRECISION | \n",
" MEAN_FOLD_RECALL | \n",
" MEAN_FOLD_AUROC | \n",
" MEAN_FOLD_ACCURACY | \n",
" MEAN_FOLD_F1_SCORE | \n",
" MEAN_FOLD_PRECISION | \n",
" MEAN_FOLD_RECALL | \n",
" MEAN_FOLD_AUROC | \n",
" ... | \n",
" MEAN_FOLD_ACCURACY | \n",
" MEAN_FOLD_F1_SCORE | \n",
" MEAN_FOLD_PRECISION | \n",
" MEAN_FOLD_RECALL | \n",
" MEAN_FOLD_AUROC | \n",
" MEAN_FOLD_ACCURACY | \n",
" MEAN_FOLD_F1_SCORE | \n",
" MEAN_FOLD_PRECISION | \n",
" MEAN_FOLD_RECALL | \n",
" MEAN_FOLD_AUROC | \n",
"
\n",
" \n",
" | Value | \n",
" 0.958237 | \n",
" 0.905656 | \n",
" 0.914681 | \n",
" 0.905266 | \n",
" 0.992342 | \n",
" 0.964759 | \n",
" 0.918456 | \n",
" 0.922952 | \n",
" 0.919351 | \n",
" 0.997509 | \n",
" ... | \n",
" 0.453757 | \n",
" 0.367207 | \n",
" 0.566225 | \n",
" 0.370051 | \n",
" 0.738066 | \n",
" 0.96476 | \n",
" 0.889146 | \n",
" 0.945736 | \n",
" 0.876686 | \n",
" 0.997666 | \n",
"
\n",
" \n",
"
\n",
"
3 rows × 40 columns
\n",
"