Design Overview

cz-benchmarks is designed with modularity and reproducibility in mind. Its core components include:

• Datasets:

  Manage input data (AnnData objects, metadata) and ensure data integrity through type checking with custom DataType definitions. Images are supported in the future. See Datasets for more details.

• Models:

  Models are packaged in Docker containers and follow the BaseModelImplementation interface. Each model is checked for correctness using dedicated validator classes. For more information, see Models.

• Tasks:

  Define evaluation operations such as clustering, embedding evaluation, label prediction, and perturbation assessment. Tasks extend the BaseTask class and serve as blueprints for benchmarking. See Tasks for more details.

• Metrics:

  A central MetricRegistry handles the registration and computation of metrics, enabling consistent and reusable evaluation criteria. See Metrics for more details.

• Runner:

  Orchestrates the workflow by handling containerized execution, automatic serialization, and seamless integration of datasets, models, and tasks.

• Configuration Management:

  Uses Hydra and OmegaConf to dynamically compose configurations for datasets, models, and tasks.

Key Design Concepts

• Declarative Configuration: Use Hydra and OmegaConf to centralize and manage configuration for datasets, models, and tasks.

• Loose Coupling: Components communicate through well-defined interfaces. This minimizes dependencies and makes testing easier.

• Validation and Type Safety: Custom type definitions in the datasets and validators enforce that the data and model outputs meet expected standards.

Class Diagrams

        classDiagram
  ABC <|-- BaseDataset
  BaseDataset <|-- SingleCellDataset
  Enum <|-- DataType
  Enum <|-- Organism
  SingleCellDataset <|-- PerturbationSingleCellDataset
    

        classDiagram
  ABC <|-- BaseModelImplementation
  ABC <|-- BaseModelValidator
  BaseModelValidator <|-- BaseModelImplementation
  BaseModelValidator <|-- BaseSingleCellValidator
    

        classDiagram
  BaseTask <|-- BatchIntegrationTask
  BaseTask <|-- ClusteringTask
  BaseTask <|-- CrossSpeciesIntegrationTask
  BaseTask <|-- EmbeddingTask
  BaseTask <|-- MetadataLabelPredictionTask
  BaseTask <|-- PerturbationTask
    

        classDiagram
  BaseModel <|-- AggregatedMetricResult
  BaseModel <|-- MetricInfo
  BaseModel <|-- MetricResult
  Enum <|-- MetricType