CZ CELLxGENE Discover Census Schema

Document Status: Drafting

Version: 2.4.0

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED” “MAY”, and “OPTIONAL” in this document are to be interpreted as described in BCP 14, RFC2119, and RFC8174 when, and only when, they appear in all capitals, as shown here.

Census overview

The CZ CELLxGENE Discover Census is a versioned data object and API for most of the single-cell data hosted at CZ CELLxGENE Discover. It is referred to throughout this document as the Census.

The reader should be familiar with the CELLxGENE Discover dataset schema and the SOMA (“stack of matrices, annotated”) specification.

Definitions

The following terms are used throughout this document:

EDITORIAL NOTE: Review and remove as needed.

  • adata – generic variable name that refers to an AnnData object.

  • CELLxGENE Discover dataset schema – the data schema for h5ad files served by CELLxGENE Discover, for this Census schema: CELLxGENE dataset schema version is 7.0.0

  • census_obj – the Census root object, a SOMACollection.

  • Census data release – a versioned Census object deposited in a public bucket and accessible by APIs.

  • tissue – original tissue annotation.

  • tissue_general – high-level mapping of a tissue, e.g. “Heart” is the tissue_general of “Heart left ventricle” .

  • scene – a collection of spatially resolved data that can be registered to a single coordinate space.

Census Schema versioning

The Census Schema follows Semver for its versioning:

  • Major: any schema changes that make the Census incompatible with the Census API or SOMA API. Examples:

    • Column deletion in Census obs

    • Addition of new modality

  • Minor: schema additions that are compatible with public API(s) and SOMA. Examples:

    • New column to Census obs is added

    • tissue/tissue_general mapping changes

  • Patch: schema fixes. Examples:

    • Editorial schema changes

Changes MUST be documented in the schema Changelog at the end of this document.

Census data releases are versioned separately from the schema.

Schema

Data included

All datasets included in the Census MUST be of CELLxGENE Discover dataset schema version 7.0.0. The following data constraints are imposed on top of the CELLxGENE Discover dataset schema.

Organisms

The CELLxGENE Discover dataset schema requires one organism_ontology_term_id per dataset. Each feature_reference MUST contain a matching value or:


The Census MUST only include observations (cells) corresponding to the following values for organism_ontology_term_id and MUST only include features corresponding to the following values for feature_reference:

Value for Organism
"NCBITaxon:9483" Callithrix jacchus
"NCBITaxon:9606" Homo sapiens
"NCBITaxon:9544" Macaca mulatta
"NCBITaxon:10090" Mus musculus
"NCBITaxon:9598" Pan troglodytes

The following values for feature_reference MUST NOT be included:

Value for Organism
"NCBITaxon:2697049" SARS-CoV-2
"NCBITaxon:32630" ERCC Spike-Ins

Assays

CELLxGENE Discover requirements for assays are defined in assay_ontology_term_id.

The Census MUST only include observations (cells) corresponding to the following values for assay_ontology_term_id if present in CELLxGENE Discover.

Accepted Assays

EFO:0003755
for FL-cDNA

EFO:0008640
for 3’T-fill

EFO:0008641
for 3’-end-seq

EFO:0008643
for 3′-Seq

EFO:0008661
for Bru-Seq

EFO:0008669
for CAGEscan

EFO:0008673
for CapSeq

EFO:0008675
for CaptureSeq

EFO:0008679
for CEL-seq

EFO:0008694
for ClickSeq

EFO:0008697
for cP-RNA-Seq

EFO:0008703
for CytoSeq

EFO:0008708
for DeepCAGE

EFO:0008710
for Digital RNA

EFO:0008718
for DP-Seq

EFO:0008720
for DroNc-seq

EFO:0008722
for Drop-seq

EFO:0008735
for FACS-seq

EFO:0008747
for FRISCR

EFO:0008748
for FRT-Seq

EFO:0008752
for GMUCT 1.0

EFO:0008753
for GMUCT 2.0

EFO:0008756
for GRO-CAP

EFO:0008763
for Hi-SCL

EFO:0008780
for inDrop

EFO:0008796
for MARS-seq

EFO:0008797
for MATQ-seq

EFO:0008824
for NanoCAGE

EFO:0008825
for Nanogrid RNA-Seq

EFO:0008826
for NET-Seq

EFO:0008850
for PAS-Seq

EFO:0008859
for PEAT

EFO:0008863
for PLATE-Seq

EFO:0008868
for PRO-cap

EFO:0008869
for PRO-seq

EFO:0008877
for Quartz-seq

EFO:0008896
for RNA-Seq

EFO:0008897
for RNAtag-Seq

EFO:0008898
for RNET-seq

EFO:0008903
for SC3-seq

EFO:0008919
for Seq-Well

EFO:0008929
for SMA

EFO:0008930
for Smart-seq

EFO:0008931
for Smart-seq2

EFO:0008937
for snDrop-seq

EFO:0008941
for sNuc-Seq

EFO:0008945
for SPET-seq

EFO:0008953
for STRT-seq

EFO:0008954
for STRT-seq-2i

EFO:0008956
for SUPeR-seq

EFO:0008962
for TARDIS

EFO:0008966
for TCR Chain Paring

EFO:0008967
for TCR-LA-MC PCR

EFO:0008972
for TL-seq

EFO:0008974
for Tomo-Seq

EFO:0008975
for TRAP-Seq

EFO:0008978
for TSS Sequencing

EFO:0008980
for UMI Method

EFO:0009309
for Div-Seq

EFO:0009899
for 10x 3’ v2

EFO:0009900
for 10x 5’ v2

EFO:0009901
for 10x 3’ v1

EFO:0009919
for SPLiT-seq

EFO:0009922
for 10x 3’ v3

EFO:0009991
for Nuc-Seq

EFO:0009999
for G&T-Seq

EFO:0010003
for RASL-seq

EFO:0010004
for SCRB-seq

EFO:0010005
for DR-Seq

EFO:0010006
for scM&T-seq

EFO:0010007
for scTrio-seq

EFO:0010010
for CEL-seq2

EFO:0010022
for Smart-3Seq

EFO:0010034
for Cappable-Seq

EFO:0010041
for Nascent-Seq

EFO:0010058
for Fluidigm C1-based SMARTer library preparation

EFO:0010184
for Smart-like

EFO:0010550
for sci-RNA-seq

EFO:0011025
for 10x 5’ v1

EFO:0022396
for TruSeq

EFO:0022488
for Smart-seq3

EFO:0022490
for ScaleBio single cell RNA sequencing

EFO:0022600
for Parse Evercode Whole Transcriptome v1

EFO:0022601
for Parse Evercode Whole Transcriptome v2

EFO:0022602
for Parse Evercode Whole Transcriptome v3

EFO:0022604
for 10x 3’ v4

EFO:0022605
for 10x 5’ v3

EFO:0022606
for 10x gene expression flex

EFO:0022839
for STORM-seq

EFO:0022845
for modified STRT-seq

EFO:0022846
for 5’ STRT-seq

EFO:0022857
for Visium Spatial Gene Expression V1

EFO:0022962
for SHARE-seq

EFO:0030001
for sci-CAR

EFO:0030002
for microwell-seq

EFO:0030003
for 10x 3’ transcription profiling

EFO:0030004
for 10x 5’ transcription profiling

EFO:0030019
for Seq-Well S3

EFO:0030021
for Nx1-seq

EFO:0030026
for sci-Plex

EFO:0030028
for sci-RNA-seq3

EFO:0030030
for Quant-seq

EFO:0030031
for SCOPE-chip

EFO:0030059
for 10x multiome

EFO:0030060
for mCT-seq

EFO:0030061
for mcSCRB-seq

EFO:0030062
for Slide-seqV2

EFO:0030074
for SORT-seq

EFO:0700003
for BD Rhapsody Whole Transcriptome Analysis

EFO:0700004
for BD Rhapsody Targeted mRNA

EFO:0700010
for TruDrop

EFO:0700011
for GEXSCOPE technology

EFO:0700016
for Smart-seq v4

EFO:0900000
for particle-templated instant partition sequencing

EFO:0900001
for Asteria scRNA-seq kit

EFO:0900002
for HIVE CLX Single-Cell RNAseq Solution

Spatial Assays

There are further requirements for two spatial assays:

Otherwise, Census MUST NOT include the corresponding observations (cells).

Full-gene sequencing assays

Full-gene sequencing assays represent the subset of accepted assays that perform full-gene sequencing when used at the single-cell level. Such data MUST be normalized by gene length for downstream analysis.

Full-gene sequencing assays

EFO:0003755
for FL-cDNA

EFO:0008747
for FRISCR

EFO:0008763
for Hi-SCL

EFO:0008797
for MATQ-seq

EFO:0008877
for Quartz-seq

EFO:0008930
for Smart-seq

EFO:0008931
for Smart-seq2

EFO:0008956
for SUPeR-seq

EFO:0009999
for G&T-Seq

EFO:0010004
for SCRB-seq

EFO:0010006
for scM&T-seq

EFO:0010022
for Smart-3Seq

EFO:0010058
for Fluidigm C1-based
SMARTer library preparation

EFO:0010184
for Smart-like

EFO:0022396
for TruSeq

EFO:0022488
for Smart-seq3

EFO:0022839
for STORM-seq

EFO:0030031
for SCOPE-chip

EFO:0030060
for mCT-seq

EFO:0030061
for mcSCRB-seq

EFO:0700016
for Smart-seq v4

Data matrix types

Per the CELLxGENE dataset schema, all RNA assays MUST include UMI or read counts. Author-normalized data layers as defined in the CELLxGENE dataset schema MUST NOT be included in the Census.

Sample types

Observations (cells) with a tissue_type value equal to “tissue” or “organoid” MUST be included in the Census. Observations with all other values of tissue_type such as “primary cell culture” MUST NOT be included.

Repeated data

When a cell is represented multiple times in CELLxGENE Discover, only one is marked as the primary cell. This is defined in the CELLxGENE dataset schema under is_primary_data. This information MUST be included in the Census cell metadata to enable queries that retrieve datasets (see cell metadata below), and all cells MUST be included in the Census.

Data encoding and organization

The Census MUST be encoded as a SOMACollection which is referenced as census_obj in the following sections. census_obj MUST have two keys "census_info" and "census_data" whose contents are defined in the sections below.

Census information census_obj["census_info"] - SOMACollection

A series of summary and metadata tables MUST be included in this SOMACollection:

Census metadata – census_obj​​["census_info"]["summary"] – SOMADataFrame

Census metadata MUST be stored as a SOMADataFrame with two columns:

Column Encoding Description
label string Human readable label of metadata variable
value string Value associated to metadata variable

This SOMADataFrame MUST have the following rows:

  1. Census schema version:

    1. label: "census_schema_version"

    2. value: Semver schema version.

  2. Census build date:

    1. label: "census_build_date"

    2. value: The date this Census was built in ISO 8601 date format

  3. Dataset schema version:

    1. label: "dataset_schema_version"

    2. value: The CELLxGENE Discover schema version of the source H5AD files.

  4. Total number of cells or spatial spots included in this Census build:

    1. label: "total_cell_count"

    2. value: Cell count

  5. Unique number of cells or spatial spots included in this Census build (is_primary_data == True)

    1. label: "unique_cell_count"

    2. value: Cell count

An example of this SOMADataFrame is shown below:

label value
census_schema_version 2.0.0
census_build_date 2024-04-01
dataset_schema_version 5.1.0
total_cell_count 10000
unique_cell_count 1000

Census table of CELLxGENE Discover datasets – census_obj["census_info"]["datasets"] – SOMADataFrame

All datasets used to build the Census MUST be included in a table modeled as a SOMADataFrame. Each row MUST correspond to an individual dataset with the following columns:

Column Encoding Description
citation string As defined in the CELLxGENE schema.
collection_id string As defined in CELLxGENE Discover data schema (see "Schemas" section for field definitions)".
collection_name string
collection_doi string
collection_doi_label string
dataset_id string
dataset_title string
dataset_h5ad_path string Relative path to the source h5ad file in the Census storage bucket.
dataset_total_cell_count int Total number of cells from the dataset included in the Census.
dataset_version_id string As defined in CELLxGENE Discover data schema (see "Schemas" section for field definitions)".

Census summary cell counts – census_obj["census_info"]["summary_cell_counts"] – SOMADataframe

Summary cell counts grouped by organism and relevant cell metadata MUST be modeled as a SOMADataFrame in census_obj["census_info"]["summary_cell_counts"]. Each row MUST correspond to a combination of organism and metadata variables with the following columns:

Column Encoding Description
organism string The value of an organism_label defined in Census table of organisms – `census_obj["census_info"]["organisms"]` – `SOMADataframe`
category string Cell metadata used for grouping, one of the following:
  • all (special case, no grouping)
  • cell_type
  • assay
  • tissue
  • tissue_general (high-level mapping of a tissue)
  • disease
  • self_reported_ethnicity
  • sex
  • suspension_type
label string Label associated with an instance of metadata (e.g. "lung" if category is "tissue"). "na" if none.
ontology_term_id string ID associated with an instance of metadata (e.g. "UBERON:0002048" if category is "tissue"). "na" if none.
total_cell_count int Total number of cell counts for the combination of values of all other fields above.
unique_cell_count int Unique number of cells for the combination of values of all other fields above. Unique number of cells refers to the cell count, for this group, when is_primary_data == True

Example of this SOMADataFrame:

organism category label ontology_term_id total_cell_count unique_cell_count
Homo sapiens all na na x x
Homo sapiens cell_type cell_type_a CL:XXXXX x x
Homo sapiens cell_type … … x x
Homo sapiens cell_type cell_type_N CL:XXXXX x x
Homo sapiens assay assay_a EFO:XXXXX x x
Homo sapiens assay … … x x
Homo sapiens assay assay_N EFO:XXXXX x x
Homo sapiens tissue tissue_a UBERON:XXXXX x x
Homo sapiens tissue … … x x
Homo sapiens tissue tissue_N UBERON:XXXXX x x
Homo sapiens tissue_general tissue_general_a UBERON:XXXXX x x
Homo sapiens tissue_general … … x x
Homo sapiens tissue_general tissue_general_N UBERON:XXXXX x x
Homo sapiens disease disease_a MONDO:XXXXX x x
Homo sapiens disease … … x x
Homo sapiens disease disease_N MONDO:XXXXX x x
Homo sapiens self_reported_ethnicity self_reported_ethnicity_a HANCESTRO:XXXXX x x
Homo sapiens self_reported_ethnicity … … x x
Homo sapiens self_reported_ethnicity self_reported_ethnicity_N HANCESTRO:XXXXX x x
Homo sapiens sex sex_a PATO:XXXXX x x
Homo sapiens sex … … x x
Homo sapiens sex sex_N PATO:XXXXX x x
Homo sapiens suspension_type suspension_type_a na x x
Homo sapiens suspension_type … … x x
Homo sapiens suspension_type suspension_type_N na x x

Census table of organisms – census_obj["census_info"]["organisms"] – SOMADataframe

Information about organisms whose cells are included in the Census MUST be included in a table modeled as a SOMADataFrame. Each row MUST correspond to an individual organism with the following columns:

Column Encoding Description
organism_ontology_term_id string As defined in the CELLxGENE dataset schema.
organism_label string As defined in the CELLxGENE dataset schema.
organism string Machine-friendly name for Single Cell Census Data – `census_obj["census_data"][organism]` – `SOMAExperiment`. Its value MUST be the result of:
  • Converting the organism_label to lowercase
  • Replacing one or more consecutive spaces in the organism_label with a single underscore

An example of this SOMADataFrame:

organism_ontology_term_id organism_label organism
NCBITaxon:9483 Callithrix jacchus callithrix_jacchus
NCBITaxon:9606 Homo sapiens homo_sapiens
NCBITaxon:10090 Mus musculus mus_musculus

Single Cell Census Data – census_obj["census_data"][organism] – SOMAExperiment

Non-spatial data for organisms MUST be stored as a SOMAExperiment in census_obj["census_data"][organism] where the value of organism matches an organism defined in Census table of organisms – census_obj["census_info"]["organisms"] – SOMADataframe.

For example, non-spatial data for Homo sapiens MUST be stored as a SOMAExperiment in census_obj["census_data"]["homo_sapiens"].

For each organism with qualifying data, the SOMAExperiment MUST contain the following:

  • Cell metadata – census_obj["census_data"][organism].obs – SOMADataFrame

  • Data – census_obj["census_data"][organism].ms – SOMACollection. This SOMACollection MUST only contain one SOMAMeasurement in census_obj["census_data"][organism].ms["RNA"] with the following:

    • Matrix data – census_obj["census_data"][organism].ms["RNA"].X – SOMACollection. It MUST contain exactly one layer:

      • Count matrix – census_obj["census_data"][organism].ms["RNA"].X["raw"] – SOMASparseNDArray

      • Normalized count matrix – census_obj["census_data"][organism].ms["RNA"].X["normalized"] – SOMASparseNDArray

    • Feature metadata – census_obj["census_data"][organism].ms["RNA"].var – SOMAIndexedDataFrame

    • Feature dataset presence matrix – census_obj["census_data"][organism].ms["RNA"]["feature_dataset_presence_matrix"] – SOMASparseNDArray

Matrix Data, count (raw) matrix – census_obj["census_data"][organism].ms["RNA"].X["raw"] – SOMASparseNDArray

Per the CELLxGENE dataset schema, all RNA assays MUST include UMI or read counts. These counts MUST be encoded as float32 in this SOMASparseNDArray with a fill value of zero (0), and no explicitly stored zero values.

Matrix Data, normalized count matrix – census_obj["census_data"][organism].ms["RNA"].X["normalized"] – SOMASparseNDArray

This is an experimental data artifact - it may be removed at any time.

A library-sized normalized layer, containing a normalized variant of the count (raw) matrix. For full-gene sequencing assays, given a value X[i,j] in the counts (raw) matrix, library-size normalized values are defined as normalized[i,j] = (X[i,j] / var[j].feature_length) / sum(X[i, ] / var.feature_length[j]). For all other assays, for a value X[i,j] in the counts (raw) matrix, library-size normalized values are defined as normalized[i,j] = X[i,j] / sum(X[i, ]).

Feature metadata – census_obj["census_data"][organism].ms["RNA"].var – SOMADataFrame

The Census MUST only contain features with a feature_biotype value of “gene”.

The gene references are pinned as defined in the CELLxGENE dataset schema.

The following columns MUST be included:

Column Encoding Description
feature_id str Index of adata.var as defined in CELLxGENE dataset schema
feature_name str As defined in CELLxGENE dataset schema
feature_length int As defined in CELLxGENE dataset schema.
nnz int64 For this feature, the number of non-zero values present in the `X['raw']` counts (raw) matrix.
n_measured_obs int64 For this feature, the number of observations present in the source H5AD (sum of feature presence matrix).

Feature dataset presence matrix – census_obj["census_data"][organism].ms["RNA"]["feature_dataset_presence_matrix"] – SOMASparseNDArray

In some datasets, there are features not included in the source data. To clarify the difference between features that were not included and features that were not measured, for each SOMAExperiment the Census MUST include a presence matrix encoded as a SOMASparseNDArray.

For all features included in the Census, the dataset presence matrix MUST indicate what features are included in each dataset of the Census. This information MUST be encoded as a boolean matrix, True indicates the feature was included in the dataset, False otherwise. This is a two-dimensional matrix and it MUST be N x M where N is the number of datasets in the SOMAExperiment and M is the number of features. The matrix is indexed by the soma_joinid value of census_obj["census_info"]["datasets"] and census_obj["census_data"][organism].ms["RNA"].var.

If the feature has at least one cell with a value greater than zero in the count data matrix X in the dataset of origin, the value MUST be True; otherwise, it MUST be False.

An example of this matrix is shown below:

Feature_1 … Feature_M
Dataset_soma_joinid_1 [True|False] … [True|False]
… … … …
Dataset_soma_joinid_N [True|False] … [True|False]

Cell metadata – census_obj["census_data"][organism].obs – SOMADataFrame

Cell metadata MUST be encoded as a SOMADataFrame with the following columns:

Column Encoding Description
dataset_id string CELLxGENE dataset ID
tissue_general_ontology_term_id string UBERON ontology term identifier for the high-level tissue mapping assigned by the TissueMapper.
tissue_general string UBERON ontology label for the high-level tissue mapping assigned by the TissueMapper.
assay_ontology_term_id As defined in CELLxGENE dataset schema
assay
cell_type_ontology_term_id
cell_type
development_stage_ontology_term_id
development_stage
disease_ontology_term_id
disease
donor_id
is_primary_data
observation_joinid
self_reported_ethnicity_ontology_term_id
self_reported_ethnicity
sex_ontology_term_id
sex
suspension_type
tissue_ontology_term_id
tissue
tissue_type
nnz int64 For this observation, the number of non-zero measurements in the `X['raw']` counts (raw) matrix.
n_measured_vars int64 For this observation, the number of features present in the source H5AD (sum of feature presence matrix).
raw_sum float32 For this observation, the sum of the `X['raw']` counts (raw) matrix values.
raw_mean_nnz float32 For this observation, the mean of the `X['raw']` counts (raw) matrix values. Zeroes are excluded from the calculation.
raw_variance_nnz float32 For this observation, the variance of the `X['raw']` counts (raw) matrix values. Zeroes are excluded from the calculation.

Census Spatial Sequencing Data – census_obj["census_spatial_sequencing"][organism] – SOMAExperiment

Spatial data for organisms MUST be stored as a SOMAExperiment in census_obj["census_data"][organism] where the value of organism matches an organism defined in Census table of organisms – census_obj["census_info"]["organisms"] – SOMADataframe.

For example, spatial data for Homo sapiens MUST be stored as a SOMAExperiment in census_obj["census_spatial_sequencing"]["homo_sapiens"].

EDITORIAL NOTE: The assay references/restatement will be addressed when this section is rewritten.

Only Visium Spatial Gene Expression V1 (“EFO:0022857”) and Slide-seqV2 (“EFO:0030062”) are supported for spatial data. See the “assays included” section above.

For each organism the SOMAExperiment MUST contain the following:

  • Cell metadata – census_obj["census_spatial_sequencing"][organism].obs – SOMADataFrame

  • Non-spatial data – census_obj["census_spatial_sequencing"][organism].ms – SOMACollection. This SOMACollection MUST only contain one SOMAMeasurement in census_obj["census_spatial_sequencing"][organism].ms["RNA"] with the following:

    • Matrix data – census_obj["census_spatial_sequencing"][organism].ms["RNA"].X – SOMACollection. It MUST contain exactly one layer:

      • Count matrix – census_obj["census_spatial_sequencing"][organism].ms["RNA"].X["raw"] – SOMASparseNDArray

    • Feature metadata – census_obj["census_spatial_sequencing"][organism].ms["RNA"].var – SOMAIndexedDataFrame

    • Feature dataset presence matrix – census_obj["census_spatial_sequencing"][organism].ms["RNA"]["feature_dataset_presence_matrix"] – SOMASparseNDArray

  • Obs to spatial data mapping:

    • Obs to spatial data – census_obj["census_spatial_sequencing"][organism].obs_spatial_presence – SOMADataFrame

  • Spatial data – census_obj["census_spatial_sequencing"][organism].spatial – SOMACollection.

    • Spatial Scenes with spatial data – census_obj["census_spatial_sequencing"][organism].spatial[scene_id] – SOMAScene. There will be as many Spatial Scenes as spatial datasets. EachSOMAScene MUST contain the following:

      • Positions array – census_obj["census_spatial_sequencing"][organism].spatial[scene_id].obsl["loc"] – SOMAPointCloudDataFrame.

      • High resolution image – census_obj["census_spatial_sequencing"][organism].spatial[scene_id].img[library_id]["highres_image"] – MultiscaleImage.

Matrix Data, count (raw) matrix – census_obj["census_spatial_sequencing"][organism].ms["RNA"].X["raw"] – SOMASparseNDArray

Spatial and non-spatial data share the same requirements.

Feature metadata – census_obj["census_spatial_sequencing"][organism].ms["RNA"].var – SOMADataFrame

Spatial and non-spatial data share the same requirements.

Feature dataset presence matrix – census_obj["census_spatial_sequencing"][organism].ms["RNA"]["feature_dataset_presence_matrix"] – SOMASparseNDArray

Spatial and non-spatial data share the same requirements.

Cell metadata – census_obj["census_spatial_sequencing"][organism].obs – SOMADataFrame

Spatial and non-spatial data share the same requirements.

Important note: In addition, the following spatial obs columns from the CELLxGENE dataset schema MUST be included in this SOMADataFrame

Column Encoding Description
array_col As defined in CELLxGENE dataset schema
array_row
in_tissue

Obs to spatial mapping – census_obj["census_spatial_sequencing"][organism].obs_presence_matrix – SOMADataFrame

It indicates the link between an observation and a scene. Each row corresponds to an observation with the following columns:

Column Encoding Description
soma_joinid int It MUST be valid soma_joinid from census_obj["census_spatial_sequencing"][organism].obs.
scene_id string It MUST be valid scene_id from census_obj["census_spatial_sequencing"][organism].spatial.
data bool It MUST be True if the scene contains spatial information about the observation, otherwise it MUST be False.

Positions array of a Scene – census_obj["census_spatial_sequencing"][organism].spatial[scene_id].obsl["loc"] – SOMAPointCloudDataFrame

scene_id MUST correspond to the values scene_id in census_obj["census_spatial_sequencing"][organism].obs_presence_matrix.

For each observation in each Scene, spatial array positions and additional positional metadata MUST be encoded as a SOMAPointCloudDataFrame. Each row corresponds to an observation with the following columns:

Column Encoding Description
x float It MUST be the corresponding value in the first column of obsm["spatial"]. As defined in the CELLxGENE dataset schema.
y float It MUST be the corresponding value in the second column of obsm["spatial"]. As defined in the CELLxGENE dataset schema.
soma_joinid integer It MUST be valid soma_joinid from census_obj["census_spatial_sequencing"][organism].obs.

If Visium Spatial Gene Expression V1 (“EFO:0022857”), the units for the spatial array positions are pixels from the full-resolution image.

The location dataframe MUST have the metadata fields:

Key Encoding Description
soma_geometry_type str MUST be "radius"
soma_geometry float Radius of points: diameter/2. If Visium Spatial Gene Expression V1 ("EFO:0022857") diameter MUST be .uns["spatial"][library_id]['spot_diameter_fullres'], as defined in the CELLxGENE dataset schema. For Slide-seqV2 ("EFO:0030062") the radius is a small constant.

Images of a Scene - census_obj["census_spatial_sequencing"][organism].spatial[scene_soma_joinid].img[library_id] – Collection of MultiscaleImage

Images of a Visium Spatial Gene Expression V1 (“EFO:0022857”) scene MUST adhere to the following specifications. Other assays MUST NOT have images, and MUST NOT include the img collection.

library_id MUST be the corresponding value in the source H5AD slot .uns["spatial"][library_id], as defined in the CELLxGENE dataset schema.

High resolution image of a Scene – census_obj["census_spatial_sequencing"][organism].spatial[scene_soma_joinid].img[library_id]["highres_image"] – MultiscaleImage

The high resolution image of a Visium Spatial Gene Expression V1(“EFO:0022857”) scene MUST be included and MUST be encoded as a SOMAImageNDArray.

Value: the image from uns["spatial"][library_id]['images']['hires'] as defined in the CELLxGENE dataset schema.

Changelog

Version 2.4.0

  • Updated all CELLxGENE Discover dataset schema references from 5.2.0 to 7.0.0

  • Updated all Visium Spatial Gene Expression references to Visium Spatial Gene Expression V1

  • Species

    • Renamed section from Species to Organisms for consistency

    • Added Callithrix jacchus

    • Added Macaca mulatta

    • Added Pan troglodytes

  • Multi-species data constraints

    • Deleted section due to deprecated requirements for datasets containing multiple species or orthologous gene references

  • Assays

    • Inlined and updated the Census Accepted Assays (CSV) file

      • Added EFO:0008703 for CytoSeq

      • Added EFO:0009999 for G&T-Seq

      • Added EFO:0010005 for DR-Seq

      • Added EFO:0010006 for scM&T-seq

      • Added EFO:0010007 for scTrio-seq

      • Added EFO:0022604 for 10x 3’ v4

      • Added EFO:0022605 for 10x 5’ v3

      • Added EFO:0022857 for Visium Spatial Gene Expression V1

      • Added EFO:0022962 for SHARE-seq

      • Added EFO:0030001 for sci-CAR

      • Added EFO:0030026 for sci-Plex

      • Added EFO:0030059 for 10x multiome

      • Added EFO:0030060 for mCT-seq

      • Added EFO:0900000 for particle-templated instant partition sequencing

      • Added EFO:0900001 for Asteria scRNA-seq kit

      • Added EFO:0900002 for HIVE CLX Single-Cell RNAseq Solution

      • Removed EFO:0008908 for SCI-seq

      • Removed EFO:0010713 for 10x immune profiling

      • Removed EFO:0010714 for 10x TCR enrichment

      • Removed EFO:0010715 for 10x Ig enrichment

      • Removed EFO:0010961 for Visium Spatial Gene Expression

      • Removed EFO:0010964 for barcoded plate-based single cell RNA-seq

      • Removed EFO:0030078 for droplet-based single-cell RNA library preparation

    • Inlined and updated the Census Accepted Assays Full Gene (CSV) file

      • Added EFO:0009999 for G&T-Seq

      • Added EFO:0010006 for scM&T-seq

      • Added EFO:0030060 for mCT-seq

      • Removed EFO:0008441 for full length single cell RNA sequencing

      • Removed EFO:0009810 for full length single nucleus RNA sequencing

  • Census metadata – census_obj​​["census_info"]["summary"]

    • Deprecated "number_donors_homo_sapiens"

    • Deprecated "number_donors_mus_musculus"

  • Census table of organisms – census_obj["census_info"]["organisms"]

    • Replaced the code reference that documented the value of organism with its requirements

  • Cell metadata – census_obj["census_data"][organism].obs

    • Corrected code references for tissue_general_ontology_term_id and tissue_general

Version 2.3.0

  • Added new top level object for holding spatial information, census_spatial_sequencing.

Version 2.2.0

  • Allow organoid data, i.e. tissue_type can now be organoid other than tissue.

Version 2.1.0

  • Update to require CELLxGENE schema version 5.2.0

  • Adds collection_doi_label to “Census table of CELLxGENE Discover datasets – census_obj["census_info"]["datasets"]”

Version 2.0.1

  • Update accepted assays for Census based on guidance from curators.

Version 2.0.0

  • Update to require CELLxGENE schema version 5.0.0

  • Expanded list of assays included in the Census.

  • Expanded the list of assays defined as full-gene sequencing assays, which have special normalized layer handling.

  • Clarified handling of datasets which are multi-species on the obs or var axis.

Version 1.3.0

  • Update to require CELLxGENE schema version 4.0.0

  • Adds citation to “Census table of CELLxGENE Discover datasets – census_obj["census_info"]["datasets"]”

  • Adds observation_joinid and tissue_type to obs dataframe

Version 1.2.0

Version 1.1.0

  • Adds dataset_version_id to “Census table of CELLxGENE Discover datasets – census_obj["census_info"]["datasets"]”

  • Add X["normalized"] layer

  • Add nnz and n_measured_obs columns to ms["RNA"].var dataframe

  • Add nnz, n_measured_vars, raw_sum, raw_mean_nnz and raw_variance_nnz columns to obs dataframe

Version 1.0.0

  • Updates text to reflect official name: CZ CELLxGENE Discover Census.

  • Updates census["census_info"]["summary"] to reflect official name in the column label:

    • From "cell_census_build_date" to "census_build_date".

    • From "cell_census_schema_version" to "census_schema_version".

  • Adds the following row to census["census_info"]["summary"]:

    • "dataset_schema_version"

Version 0.1.1

  • Adds clarifying text for “Feature Dataset Presence Matrix”

Version 0.1.0

  • The “Dataset Presence Matrix” was renamed to “Feature Dataset Presence Matrix” and moved from census_obj["census_data"][organism].ms["RNA"].varp["dataset_presence_matrix"] to census_obj["census_data"][organism].ms["RNA"]["feature_dataset_presence_matrix"].

  • Editorial: changes all double quotes in the schema to ASCII quotes 0x22.

Version 0.0.1

  • Initial Census schema is published.