API Reference

The cz-benchmarks package consists of several core modules, each designed to work independently while contributing to a cohesive benchmarking workflow. Below is an overview of these modules, along with links to their detailed documentation.

Core Modules

• Datasets (czbenchmarks.datasets):

  Contains classes for loading and validating datasets (e.g., SingleCellDataset), with support for AnnData and custom metadata. See the full documentation: czbenchmarks.datasets.

• Models (czbenchmarks.models):

  • Implementations:

    Contains the concrete model inference logic in Docker container form. The base class is BaseModelImplementation.

  • Validators:

    Enforces that datasets meet the requirements of particular models. Validators extend from BaseModelValidator or BaseSingleCellValidator. See the full documentation: czbenchmarks.models.

• Tasks (czbenchmarks.tasks):

  Provides evaluation tasks (e.g., clustering, embedding, perturbation prediction) by extending the BaseTask class. See the full documentation: czbenchmarks.tasks.

• Metrics (czbenchmarks.metrics):

  Maintains a registry of metric functions through the MetricRegistry interface and organizes metrics into categories (clustering, embedding, etc.). See the full documentation: czbenchmarks.metrics.

• Runner (czbenchmarks.runner):

  Orchestrates the overall workflow: loading datasets, running model inference, executing tasks, and serializing results. See the full documentation: czbenchmarks.runner.

Additional Utilities

• CLI (czbenchmarks.cli):

  Command-line interface for interacting with the cz-benchmarks package. See the full documentation: czbenchmarks.cli.

• Utils (czbenchmarks.utils):

  Contains utility functions and helpers used across the package. See the full documentation: czbenchmarks.utils.