czbenchmarks.cli.cli_run

Attributes

`log`
`VALID_OUTPUT_FORMATS`
`DEFAULT_OUTPUT_FORMAT`
`TaskType`
`ModelArgsDict`
`RuntimeMetricsDict`

Classes

`ModelArgs`	!!! abstract "Usage Documentation"
`TaskArgs`	!!! abstract "Usage Documentation"
`TaskResult`	!!! abstract "Usage Documentation"
`CacheOptions`	!!! abstract "Usage Documentation"

Functions

`add_arguments`(→ None)	Add run command arguments to the parser.
`main`(→ None)	Execute a series of tasks using multiple models on a collection of datasets.
`run`(→ list[TaskResult])	Run a set of tasks against a set of datasets. Runs inference if any model_args are specified.
`run_with_inference`(→ list[TaskResult])	Execute a series of tasks using multiple models on a collection of datasets.
`run_inference_or_load_from_cache`(...)	Load the processed dataset from the cache if it exists, else run inference and save to cache.
`run_without_inference`(→ list[TaskResult])	Run a set of tasks directly against raw datasets without first running model inference.
`run_multi_dataset_task`(→ list[TaskResult])	Run a task and return the results.
`run_task`(→ list[TaskResult])	Run a task and return the results.
`get_model_arg_permutations`(→ dict[str, ...)	Generate all the "permutations" of model arguments we want to run for each dataset:
`write_results`(→ None)	Format and write results to the given directory or file.
`get_result_url_for_remote`(→ str)
`set_processed_datasets_cache`(→ None)	Write a dataset to the cache
`try_processed_datasets_cache`(...)	Deserialize and return a processed dataset from the cache if it exists, else return None.
`get_remote_cache_prefix`(cache_options)	get the prefix ending in '/' that the remote processed datasets go under
`get_processed_dataset_cache_filename`(→ str)	generate a unique filename for the given dataset and model arguments
`get_processed_dataset_cache_path`(→ pathlib.Path)	Return a unique file path in the cache directory for the given dataset and model arguments.
`parse_model_args`(→ ModelArgs)	Populate a ModelArgs instance from the given argparse namespace.
`parse_task_args`(→ TaskArgs)	Populate a TaskArgs instance from the given argparse namespace.
`parse_batch_json`(→ list[dict[str, Any]])	Parse the --batch-json argument.

Module Contents

czbenchmarks.cli.cli_run.log[source]

czbenchmarks.cli.cli_run.VALID_OUTPUT_FORMATS = ['json', 'yaml']

czbenchmarks.cli.cli_run.DEFAULT_OUTPUT_FORMAT = 'json'

czbenchmarks.cli.cli_run.TaskType

czbenchmarks.cli.cli_run.ModelArgsDict

czbenchmarks.cli.cli_run.RuntimeMetricsDict

class czbenchmarks.cli.cli_run.ModelArgs(/, **data: Any)[source]

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

__class_vars__: The names of the class variables defined on the model.

__private_attributes__: Metadata about the private attributes of the model.

__signature__: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__: Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__: The core schema of the model.

__pydantic_custom_init__: Whether the model has a custom __init__ function.

__pydantic_decorators__: Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__: Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__: The name of the post-init method for the model, if defined.

__pydantic_root_model__: Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__: The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__: The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__: A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__: A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__: The names of fields explicitly set during instantiation.

__pydantic_private__: Values of private attributes set on the model instance.

name: str

args: dict[str, list[str | int]]

class czbenchmarks.cli.cli_run.TaskArgs(/, **data: Any)[source]

Bases: pydantic.BaseModel, Generic[TaskType]

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

__class_vars__: The names of the class variables defined on the model.

__private_attributes__: Metadata about the private attributes of the model.

__signature__: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__: Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__: The core schema of the model.

__pydantic_custom_init__: Whether the model has a custom __init__ function.

__pydantic_decorators__: Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__: Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__: The name of the post-init method for the model, if defined.

__pydantic_root_model__: Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__: The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__: The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__: A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__: A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__: The names of fields explicitly set during instantiation.

__pydantic_private__: Values of private attributes set on the model instance.

model_config: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str

task: TaskType

set_baseline: bool

baseline_args: dict[str, Any]

class czbenchmarks.cli.cli_run.TaskResult(/, **data: Any)[source]

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

__class_vars__: The names of the class variables defined on the model.

__private_attributes__: Metadata about the private attributes of the model.

__signature__: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__: Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__: The core schema of the model.

__pydantic_custom_init__: Whether the model has a custom __init__ function.

__pydantic_decorators__: Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__: Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__: The name of the post-init method for the model, if defined.

__pydantic_root_model__: Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__: The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__: The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__: A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__: A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__: The names of fields explicitly set during instantiation.

__pydantic_private__: Values of private attributes set on the model instance.

task_name: str

task_name_display: str

model_type: czbenchmarks.models.types.ModelType

dataset_names: list[str]

dataset_names_display: list[str]

model_args: ModelArgsDict

metrics: list[czbenchmarks.metrics.types.MetricResult]

runtime_metrics: RuntimeMetricsDict

property model_name_display: str

class czbenchmarks.cli.cli_run.CacheOptions(/, **data: Any)[source]

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

__class_vars__: The names of the class variables defined on the model.

__private_attributes__: Metadata about the private attributes of the model.

__signature__: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__: Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__: The core schema of the model.

__pydantic_custom_init__: Whether the model has a custom __init__ function.

__pydantic_decorators__: Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__: Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__: Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__: The name of the post-init method for the model, if defined.

__pydantic_root_model__: Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__: The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__: The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__: A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__: A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__: The names of fields explicitly set during instantiation.

__pydantic_private__: Values of private attributes set on the model instance.

download_embeddings: bool

upload_embeddings: bool

upload_results: bool

remote_cache_url: str

classmethod from_args(args: argparse.Namespace) → CacheOptions[source]

czbenchmarks.cli.cli_run.add_arguments(parser: argparse.ArgumentParser) → None[source]: Add run command arguments to the parser.

czbenchmarks.cli.cli_run.main(parsed_args: argparse.Namespace) → None[source]

Execute a series of tasks using multiple models on a collection of datasets.

This function handles the benchmarking process by iterating over the specified datasets, running inference with the provided models to generate results, and running the tasks to evaluate the generated outputs.

czbenchmarks.cli.cli_run.run(dataset_names: list[str], model_args: list[ModelArgs], task_args: list[TaskArgs], cache_options: CacheOptions) → list[TaskResult][source]: Run a set of tasks against a set of datasets. Runs inference if any model_args are specified.

czbenchmarks.cli.cli_run.run_with_inference(dataset_names: list[str], model_args: list[ModelArgs], task_args: list[TaskArgs], cache_options: CacheOptions) → list[TaskResult][source]

Execute a series of tasks using multiple models on a collection of datasets.

This function handles the benchmarking process by iterating over the specified datasets, running inference with the provided models to generate results, and running the tasks to evaluate the generated outputs.

czbenchmarks.cli.cli_run.run_inference_or_load_from_cache(dataset_name: str, *, model_name: str, model_args: ModelArgsDict, cache_options: CacheOptions) → czbenchmarks.datasets.base.BaseDataset[source]: Load the processed dataset from the cache if it exists, else run inference and save to cache.

czbenchmarks.cli.cli_run.run_without_inference(dataset_names: list[str], task_args: list[TaskArgs]) → list[TaskResult][source]: Run a set of tasks directly against raw datasets without first running model inference.

czbenchmarks.cli.cli_run.run_multi_dataset_task(dataset_names: list[str], embeddings: list[czbenchmarks.datasets.base.BaseDataset], model_args: dict[str, ModelArgsDict], task_args: TaskArgs) → list[TaskResult][source]: Run a task and return the results.

czbenchmarks.cli.cli_run.run_task(dataset_name: str, dataset: czbenchmarks.datasets.base.BaseDataset, model_args: dict[str, ModelArgsDict], task_args: TaskArgs) → list[TaskResult][source]: Run a task and return the results.

czbenchmarks.cli.cli_run.get_model_arg_permutations(model_args: list[ModelArgs]) → dict[str, list[ModelArgsDict]][source]: Generate all the “permutations” of model arguments we want to run for each dataset: E.g. Running 2 variants of scgenept at 2 chunk sizes results in 4 permutations

czbenchmarks.cli.cli_run.write_results(task_results: list[TaskResult], *, cache_options: CacheOptions, output_format: str = DEFAULT_OUTPUT_FORMAT, output_file: str | None = None) → None[source]: Format and write results to the given directory or file.

czbenchmarks.cli.cli_run.get_result_url_for_remote(remote_prefix_url: str) → str[source]

czbenchmarks.cli.cli_run.set_processed_datasets_cache(dataset: czbenchmarks.datasets.base.BaseDataset, dataset_name: str, *, model_name: str, model_args: ModelArgsDict, cache_options: CacheOptions) → None[source]: Write a dataset to the cache A “processed” dataset has been run with model inference for the given arguments.

czbenchmarks.cli.cli_run.try_processed_datasets_cache(dataset_name: str, *, model_name: str, model_args: ModelArgsDict, cache_options: CacheOptions) → czbenchmarks.datasets.base.BaseDataset | None[source]: Deserialize and return a processed dataset from the cache if it exists, else return None.

czbenchmarks.cli.cli_run.get_remote_cache_prefix(cache_options: CacheOptions)[source]: get the prefix ending in ‘/’ that the remote processed datasets go under

czbenchmarks.cli.cli_run.get_processed_dataset_cache_filename(dataset_name: str, *, model_name: str, model_args: ModelArgsDict) → str[source]: generate a unique filename for the given dataset and model arguments

czbenchmarks.cli.cli_run.get_processed_dataset_cache_path(dataset_name: str, *, model_name: str, model_args: ModelArgsDict) → pathlib.Path[source]: Return a unique file path in the cache directory for the given dataset and model arguments.

czbenchmarks.cli.cli_run.parse_model_args(model_name: str, args: argparse.Namespace) → ModelArgs[source]: Populate a ModelArgs instance from the given argparse namespace.

czbenchmarks.cli.cli_run.parse_task_args(task_name: str, TaskCls: type[TaskType], args: argparse.Namespace) → TaskArgs[source]: Populate a TaskArgs instance from the given argparse namespace.

czbenchmarks.cli.cli_run.parse_batch_json(batch_json_list: list[str]) → list[dict[str, Any]][source]: Parse the –batch-json argument. Returns a list of dicts where each entry is a batch of CLI arguments.