Learning about the CZ CELLxGENE Census

This notebook showcases the Census contents and how to obtain high-level information about it. It covers the organization of data within the Census, what cell and gene metadata are available, and it provides simple demonstrations to summarize cell counts across cell metadata.


  • Opening the census

  • Census organization

  • Cell metadata

  • Gene metadata

  • Census summary content tables

  • Understanding Census contents beyond the summary tables

⚠️ Note that the Census RNA data includes duplicate cells present across multiple datasets. Duplicate cells can be filtered in or out using the cell metadata variable is_primary_data which is described in the Census schema.

Opening the Census

The cellxgene_census python package contains a convenient API to open the latest version of the Census. If you open the census, you should close it. open_soma() returns a context, so you can open/close it in several ways, like a Python file handle. The context manager is preferred, as it will automatically close upon an error raise.

import cellxgene_census

# Preferred: use a Python context manager
with cellxgene_census.open_soma() as census:

# or
census = cellxgene_census.open_soma()
The "stable" release is currently 2023-07-25. Specify 'census_version="2023-07-25"' in future calls to open_soma() to ensure data consistency.
The "stable" release is currently 2023-07-25. Specify 'census_version="2023-07-25"' in future calls to open_soma() to ensure data consistency.

You can learn more about the cellxgene_census methods by accessing their corresponding documentation via help(). For example help(cellxgene_census.open_soma).

census = cellxgene_census.open_soma()
The "stable" release is currently 2023-07-25. Specify 'census_version="2023-07-25"' in future calls to open_soma() to ensure data consistency.

Census organization

The Census schema defines the structure of the Census. In short, you can think of the Census as a structured collection of items that stores different pieces of information. All of these items and the parent collection are SOMA objects of various types and can all be accessed with the TileDB-SOMA API (documentation).

The cellxgene_census package contains some convenient wrappers of the TileDB-SOMA API. An example of this is the function we used to open the Census: cellxgene_census.open_soma()

Main Census components

With the command above you created census, which is a SOMACollection. It is analogous to a Python dictionary, and it has two items: census_info and census_data.

Census summary info

  • census["census_info"] A collection of tables providing information of the census as a whole.

    • census["census_info"]["summary"]: A data frame with high-level information of this Census, e.g. build date, total cell count, etc.

    • census["census_info"]["datasets"]: A data frame with all datasets from CELLxGENE Discover used to create the Census.

    • census["census_info"]["summary_cell_counts"]: A data frame with cell counts stratified by relevant cell metadata

Census data

Data for each organism is stored in independent SOMAExperiment objects which are a specialized form of a SOMACollection. Each of these store a data matrix (cell by genes), cell metadata, gene metadata, and some other useful components not covered in this notebook.

This is how the data is organized for one organism – Homo sapiens:

  • census_obj["census_data"]["homo_sapiens"].obs: Cell metadata

  • census_obj["census_data"]["homo_sapiens"].ms["RNA"].X: Data matrices, currently only raw counts exist X["raw"]

  • census_obj["census_data"]["homo_sapiens"].ms["RNA"].var: Gene Metadata

Cell metadata

You can obtain all cell metadata variables by directly querying the columns of the corresponding SOMADataFrame.

All of these variables can be used for querying the Census in case you want to work with specific cells.

keys = list(census["census_data"]["homo_sapiens"].obs.keys())


All of these variables are defined in the CELLxGENE dataset schema except for the following:

  • soma_joinid: a SOMA-defined value use for join operations.

  • dataset_id: the dataset id as encoded in census["census-info"]["datasets"].

  • tissue_general and tissue_general_ontology_term_id: the high-level tissue mapping.

Gene metadata

Similarly, we can obtain all gene metadata variables by directly querying the columns of the corresponding SOMADataFrame.

These are the variables you can use for querying the Census in case there are specific genes you are interested in.

keys = list(census["census_data"]["homo_sapiens"].ms["RNA"].var.keys())

['soma_joinid', 'feature_id', 'feature_name', 'feature_length']

All of these variables are defined in the CELLxGENE dataset schema except for the following:

  • soma_joinid: a SOMA-defined value use for join operations.

  • feature_length: the length in base pairs of the gene.

census_info = census["census_info"]["summary"].read().concat().to_pandas()

soma_joinid label value
0 0 census_schema_version 1.0.0
1 1 census_build_date 2023-07-25
2 2 dataset_schema_version 3.0.0
3 3 total_cell_count 61656118
4 4 unique_cell_count 37447773
5 5 number_donors_homo_sapiens 13035
6 6 number_donors_mus_musculus 1417

Census summary content tables

You can take a quick look at the high-level Census information by looking at census["census_info"]["summary"]

Of special interest are the label-value combinations for :

  • total_cell_count is the total number of cells in the Census.

  • unique_cell_count is the number of unique cells, as some cells may be present twice due to meta-analysis or consortia-like data.

  • number_donors_homo_sapiens and number_donors_mus_musculus are the number of individuals for human and mouse. These are not guaranteed to be unique as one individual ID may be present or identical in different datasets.

Cell counts by cell metadata

By looking at census["summary_cell_counts"] you can get a general idea of cell counts stratified by some relevant cell metadata. Not all cell metadata is included in this table, you can take a look at all cell and gene metadata available in the sections below “Cell metadata” and “Gene metadata”.

The line below retrieves this table and casts it into a pandas.DataFrame.

census_counts = census["census_info"]["summary_cell_counts"].read().concat().to_pandas()

soma_joinid organism category ontology_term_id unique_cell_count total_cell_count label
0 0 Homo sapiens all na 33364242 56400873 na
1 1 Homo sapiens assay EFO:0008722 264166 279635 Drop-seq
2 2 Homo sapiens assay EFO:0008780 25652 51304 inDrop
3 3 Homo sapiens assay EFO:0008919 89477 206754 Seq-Well
4 4 Homo sapiens assay EFO:0008931 78750 188248 Smart-seq2
... ... ... ... ... ... ... ...
1357 1357 Mus musculus tissue_general UBERON:0002113 179684 208324 kidney
1358 1358 Mus musculus tissue_general UBERON:0002365 15577 31154 exocrine gland
1359 1359 Mus musculus tissue_general UBERON:0002367 37715 130135 prostate gland
1360 1360 Mus musculus tissue_general UBERON:0002368 13322 26644 endocrine gland
1361 1361 Mus musculus tissue_general UBERON:0002371 90225 144962 bone marrow

1362 rows × 7 columns

For each combination of organism and values for each category of cell metadata you can take a look at total_cell_count and unique_cell_count for the cell counts of that combination.

The values for each category are specified in ontology_term_id and label, which are the value’s IDs and labels, respectively.

Example: cell metadata included in the summary counts table

To get all the available cell metadata in the summary counts table you can do the following. Remember this is not all the cell metadata available, as some variables were omitted in the creation of this table.

census_counts[["organism", "category"]].value_counts(sort=False)
organism      category
Homo sapiens  all                          1
              assay                       19
              cell_type                  613
              disease                     64
              self_reported_ethnicity     26
              sex                          3
              suspension_type              1
              tissue                     220
              tissue_general              54
Mus musculus  all                          1
              assay                        9
              cell_type                  248
              disease                      5
              self_reported_ethnicity      1
              sex                          3
              suspension_type              1
              tissue                      66
              tissue_general              27
Name: count, dtype: int64

Example: cell counts for each sequencing assay in human data

To get the cell counts for each sequencing assay type in human data, you can perform the following pandas.DataFrame operations:

census_human_assays = census_counts.query("organism == 'Homo sapiens' & category == 'assay'")
census_human_assays.sort_values("total_cell_count", ascending=False)
soma_joinid organism category ontology_term_id unique_cell_count total_cell_count label
10 10 Homo sapiens assay EFO:0009922 11845077 25597563 10x 3' v3
7 7 Homo sapiens assay EFO:0009899 7559102 12638794 10x 3' v2
14 14 Homo sapiens assay EFO:0011025 3872375 6139786 10x 5' v1
13 13 Homo sapiens assay EFO:0010550 4062980 5064268 sci-RNA-seq
8 8 Homo sapiens assay EFO:0009900 2930054 3139770 10x 5' v2
17 17 Homo sapiens assay EFO:0030004 915037 1084235 10x 5' transcription profiling
16 16 Homo sapiens assay EFO:0030003 744798 811422 10x 3' transcription profiling
15 15 Homo sapiens assay EFO:0030002 625175 642559 microwell-seq
1 1 Homo sapiens assay EFO:0008722 264166 279635 Drop-seq
3 3 Homo sapiens assay EFO:0008919 89477 206754 Seq-Well
4 4 Homo sapiens assay EFO:0008931 78750 188248 Smart-seq2
18 18 Homo sapiens assay EFO:0700003 146278 177276 BD Rhapsody Whole Transcriptome Analysis
9 9 Homo sapiens assay EFO:0009901 42397 121394 10x 3' v1
12 12 Homo sapiens assay EFO:0010183 58981 117962 single cell library construction
19 19 Homo sapiens assay EFO:0700004 96145 96145 BD Rhapsody Targeted mRNA
2 2 Homo sapiens assay EFO:0008780 25652 51304 inDrop
6 6 Homo sapiens assay EFO:0008995 0 29128 10x technology
5 5 Homo sapiens assay EFO:0008953 4693 9386 STRT-seq
11 11 Homo sapiens assay EFO:0010010 3105 5244 CEL-seq2

Example: number of microglial cells in the Census

If you have a specific term from any of the categories shown above you can directly find out the number of cells for that term.

census_counts.query("label == 'microglial cell'")
soma_joinid organism category ontology_term_id unique_cell_count total_cell_count label
69 69 Homo sapiens cell_type CL:0000129 268114 370771 microglial cell
1038 1038 Mus musculus cell_type CL:0000129 48998 62617 microglial cell

Understanding Census contents beyond the summary tables

While using the pre-computed tables in census["census_info"] is an easy and quick way to understand the contents of the Census, it falls short if you want to learn more about certain slices of the Census.

For example, you may want to learn more about:

  • What are the cell types available for human liver?

  • What are the total number of cells in all lung datasets stratified by sequencing technology?

  • What is the sex distribution of all cells from brain in mouse?

  • What are the diseases available for T cells?

All of these questions can be answered by directly querying the cell metadata as shown in the examples below.

Example: all cell types available in human

To exemplify the process of accessing and slicing cell metadata for summary stats, let’s start with a trivial example and take a look at all human cell types available in the Census:

human_cell_types = (
    census["census_data"]["homo_sapiens"].obs.read(column_names=["cell_type", "is_primary_data"]).concat().to_pandas()
cell_type is_primary_data
0 syncytiotrophoblast cell False
1 placental villous trophoblast False
2 syncytiotrophoblast cell False
3 syncytiotrophoblast cell False
4 extravillous trophoblast False
... ... ...
56400868 pericyte True
56400869 pericyte True
56400870 pericyte True
56400871 pericyte True
56400872 pericyte True

56400873 rows × 2 columns

The number of rows is the total number of cells for humans. Now, if you wish to get the cell counts per cell type we can perform some pandas operations on this object.

In addition, we will only focus on cells that are marked with is_primary_data=True as this ensures we de-duplicate cells that appear more than once in CELLxGENE Discover.

human_cell_types = (
    .obs.read(column_names=["cell_type"], value_filter="is_primary_data == True")

human_cell_types = human_cell_types[["cell_type"]]
(33364242, 1)

This is the number of unique cells. Now let’s look at the counts per cell type:

human_cell_type_counts = human_cell_types.value_counts()
neuron                                             2673669
glutamatergic neuron                               1541605
CD4-positive, alpha-beta T cell                    1258976
CD8-positive, alpha-beta T cell                    1235987
classical monocyte                                 1030996
microfold cell of epithelium of small intestine         19
mature conventional dendritic cell                      17
serous cell of epithelium of bronchus                   15
sperm                                                   11
type N enteroendocrine cell                             10
Name: count, Length: 599, dtype: int64

This shows you that the most abundant cell types are “glutamatergic neuron”, “CD8-positive, alpha-beta T cell”, and “CD4-positive, alpha-beta T cell”.

Now let’s take a look at the number of unique cell types:


That is the total number of different cell types for human.

All the information in this example can be quickly obtained from the summary table at census["census-info"]["summary_cell_counts"].

The examples below are more complex and can only be achieved by accessing the cell metadata.

Example: cell types available in human liver

Similar to the example above, we can learn what cell types are available for a specific tissue, e.g. liver.

To achieve this goal we just need to limit our cell metadata to that tissue. We will use the information in the cell metadata variable tissue_general. This variable contains the high-level tissue label for all cells in the Census:

human_liver_cell_types = (
    .obs.read(column_names=["cell_type"], value_filter="is_primary_data == True and tissue_general == 'liver'")

T cell                               85739
hepatoblast                          58447
neoplastic cell                      52431
erythroblast                         45605
monocyte                             31388
pulmonary artery endothelial cell        1
germinal center B cell                   1
enteroendocrine cell                     1
type I pneumocyte                        1
group 2 innate lymphoid cell             1
Name: count, Length: 126, dtype: int64

These are the cell types and their cell counts in the human liver.

Example: diseased T cells in human tissues

In this example we are going to get the counts for all diseased cells annotated as T cells. For the sake of the example we will focus on “CD8-positive, alpha-beta T cell” and “CD4-positive, alpha-beta T cell”:

t_cells_list = ["CD8-positive, alpha-beta T cell", "CD4-positive, alpha-beta T cell"]

t_cells_diseased = (
        column_names=["disease", "tissue_general"],
        value_filter=f"is_primary_data == True and cell_type in {t_cells_list} and disease != 'normal'",

t_cells_diseased = t_cells_diseased[["disease", "tissue_general"]].value_counts(sort=False)
disease                                tissue_general
B-cell non-Hodgkin lymphoma            blood                  62499
COVID-19                               blood                 819428
                                       lung                   30578
                                       nose                      13
                                       respiratory system         4
                                       saliva                    41
Crohn disease                          colon                  17490
                                       small intestine        52029
Down syndrome                          bone marrow              181
breast cancer                          breast                  1850
chronic obstructive pulmonary disease  lung                    9382
chronic rhinitis                       nose                     909
clear cell renal carcinoma             blood                   6548
                                       kidney                 20540
                                       lymph node                36
cystic fibrosis                        lung                       7
follicular lymphoma                    lymph node              1089
influenza                              blood                   8871
interstitial lung disease              lung                    1803
kidney benign neoplasm                 blood                     20
                                       kidney                    10
kidney oncocytoma                      blood                     16
                                       kidney                  2408
lung adenocarcinoma                    adrenal gland            205
                                       brain                   3274
                                       liver                    507
                                       lung                  215013
                                       lymph node             24969
                                       pleural fluid          11558
lung large cell carcinoma              lung                    5922
lymphangioleiomyomatosis               lung                     513
non-small cell lung carcinoma          lung                   36573
nonpapillary renal cell carcinoma      adipose tissue           243
                                       adrenal gland           4828
                                       blood                    288
                                       blood clot              1717
                                       kidney                 69136
pleomorphic carcinoma                  lung                    1715
pneumonia                              lung                     856
pulmonary fibrosis                     lung                    1671
respiratory system disorder            blood                  34301
squamous cell lung carcinoma           lung                   52053
                                       lymph node               100
systemic lupus erythematosus           blood                 355471
Name: count, dtype: int64

These are the cell counts annotated with the indicated disease across human tissues for “CD8-positive, alpha-beta T cell” or “CD4-positive, alpha-beta T cell”.

And, don’t forget to close the census!

del census