czbenchmarks.cli.cli_run

Attributes

log

VALID_OUTPUT_FORMATS

DEFAULT_OUTPUT_FORMAT

TaskType

ModelArgsDict

RuntimeMetricsDict

Classes

ModelArgs

!!! abstract "Usage Documentation"

TaskArgs

!!! abstract "Usage Documentation"

TaskResult

!!! abstract "Usage Documentation"

CacheOptions

!!! abstract "Usage Documentation"

Functions

add_arguments(→ None)

Add run command arguments to the parser.

main(→ None)

Execute a series of tasks using multiple models on a collection of datasets.

run(→ list[TaskResult])

Run a set of tasks against a set of datasets. Runs inference if any model_args are specified.

run_with_inference(→ list[TaskResult])

Execute a series of tasks using multiple models on a collection of datasets.

run_inference_or_load_from_cache(...)

Load the processed dataset from the cache if it exists, else run inference and save to cache.

run_without_inference(→ list[TaskResult])

Run a set of tasks directly against raw datasets without first running model inference.

run_multi_dataset_task(→ list[TaskResult])

Run a task and return the results.

run_task(→ list[TaskResult])

Run a task and return the results.

get_model_arg_permutations(→ dict[str, ...)

Generate all the "permutations" of model arguments we want to run for each dataset:

write_results(→ None)

Format and write results to the given directory or file.

get_result_url_for_remote(→ str)

set_processed_datasets_cache(→ None)

Write a dataset to the cache

try_processed_datasets_cache(...)

Deserialize and return a processed dataset from the cache if it exists, else return None.

get_remote_cache_prefix(cache_options)

get the prefix ending in '/' that the remote processed datasets go under

get_processed_dataset_cache_filename(→ str)

generate a unique filename for the given dataset and model arguments

get_processed_dataset_cache_path(→ pathlib.Path)

Return a unique file path in the cache directory for the given dataset and model arguments.

parse_model_args(→ ModelArgs)

Populate a ModelArgs instance from the given argparse namespace.

parse_task_args(→ TaskArgs)

Populate a TaskArgs instance from the given argparse namespace.

parse_batch_json(→ list[dict[str, Any]])

Parse the --batch-json argument.

Module Contents

czbenchmarks.cli.cli_run.log[source]
czbenchmarks.cli.cli_run.VALID_OUTPUT_FORMATS = ['json', 'yaml']
czbenchmarks.cli.cli_run.DEFAULT_OUTPUT_FORMAT = 'json'
czbenchmarks.cli.cli_run.TaskType
czbenchmarks.cli.cli_run.ModelArgsDict
czbenchmarks.cli.cli_run.RuntimeMetricsDict
class czbenchmarks.cli.cli_run.ModelArgs(/, **data: Any)[source]

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”

[Models](../concepts/models.md)

A base class for creating Pydantic models.

__class_vars__

The names of the class variables defined on the model.

__private_attributes__

Metadata about the private attributes of the model.

__signature__

The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__

Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__

The core schema of the model.

__pydantic_custom_init__

Whether the model has a custom __init__ function.

__pydantic_decorators__

Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__

Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__

Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__

The name of the post-init method for the model, if defined.

__pydantic_root_model__

Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__

The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__

The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__

A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__

A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__

A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__

The names of fields explicitly set during instantiation.

__pydantic_private__

Values of private attributes set on the model instance.

name: str
args: dict[str, list[str | int]]
class czbenchmarks.cli.cli_run.TaskArgs(/, **data: Any)[source]

Bases: pydantic.BaseModel, Generic[TaskType]

!!! abstract “Usage Documentation”

[Models](../concepts/models.md)

A base class for creating Pydantic models.

__class_vars__

The names of the class variables defined on the model.

__private_attributes__

Metadata about the private attributes of the model.

__signature__

The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__

Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__

The core schema of the model.

__pydantic_custom_init__

Whether the model has a custom __init__ function.

__pydantic_decorators__

Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__

Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__

Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__

The name of the post-init method for the model, if defined.

__pydantic_root_model__

Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__

The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__

The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__

A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__

A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__

A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__

The names of fields explicitly set during instantiation.

__pydantic_private__

Values of private attributes set on the model instance.

model_config

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

name: str
task: TaskType
set_baseline: bool
baseline_args: dict[str, Any]
class czbenchmarks.cli.cli_run.TaskResult(/, **data: Any)[source]

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”

[Models](../concepts/models.md)

A base class for creating Pydantic models.

__class_vars__

The names of the class variables defined on the model.

__private_attributes__

Metadata about the private attributes of the model.

__signature__

The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__

Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__

The core schema of the model.

__pydantic_custom_init__

Whether the model has a custom __init__ function.

__pydantic_decorators__

Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__

Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__

Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__

The name of the post-init method for the model, if defined.

__pydantic_root_model__

Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__

The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__

The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__

A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__

A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__

A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__

The names of fields explicitly set during instantiation.

__pydantic_private__

Values of private attributes set on the model instance.

task_name: str
task_name_display: str
model_type: czbenchmarks.models.types.ModelType
dataset_names: list[str]
dataset_names_display: list[str]
model_args: ModelArgsDict
metrics: list[czbenchmarks.metrics.types.MetricResult]
runtime_metrics: RuntimeMetricsDict
property model_name_display: str
class czbenchmarks.cli.cli_run.CacheOptions(/, **data: Any)[source]

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”

[Models](../concepts/models.md)

A base class for creating Pydantic models.

__class_vars__

The names of the class variables defined on the model.

__private_attributes__

Metadata about the private attributes of the model.

__signature__

The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__

Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__

The core schema of the model.

__pydantic_custom_init__

Whether the model has a custom __init__ function.

__pydantic_decorators__

Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__

Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__

Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__

The name of the post-init method for the model, if defined.

__pydantic_root_model__

Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__

The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__

The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__

A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__

A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__

A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__

The names of fields explicitly set during instantiation.

__pydantic_private__

Values of private attributes set on the model instance.

download_embeddings: bool
upload_embeddings: bool
upload_results: bool
remote_cache_url: str
classmethod from_args(args: argparse.Namespace) CacheOptions[source]
czbenchmarks.cli.cli_run.add_arguments(parser: argparse.ArgumentParser) None[source]

Add run command arguments to the parser.

czbenchmarks.cli.cli_run.main(parsed_args: argparse.Namespace) None[source]

Execute a series of tasks using multiple models on a collection of datasets.

This function handles the benchmarking process by iterating over the specified datasets, running inference with the provided models to generate results, and running the tasks to evaluate the generated outputs.

czbenchmarks.cli.cli_run.run(dataset_names: list[str], model_args: list[ModelArgs], task_args: list[TaskArgs], cache_options: CacheOptions) list[TaskResult][source]

Run a set of tasks against a set of datasets. Runs inference if any model_args are specified.

czbenchmarks.cli.cli_run.run_with_inference(dataset_names: list[str], model_args: list[ModelArgs], task_args: list[TaskArgs], cache_options: CacheOptions) list[TaskResult][source]

Execute a series of tasks using multiple models on a collection of datasets.

This function handles the benchmarking process by iterating over the specified datasets, running inference with the provided models to generate results, and running the tasks to evaluate the generated outputs.

czbenchmarks.cli.cli_run.run_inference_or_load_from_cache(dataset_name: str, *, model_name: str, model_args: ModelArgsDict, cache_options: CacheOptions) czbenchmarks.datasets.base.BaseDataset[source]

Load the processed dataset from the cache if it exists, else run inference and save to cache.

czbenchmarks.cli.cli_run.run_without_inference(dataset_names: list[str], task_args: list[TaskArgs]) list[TaskResult][source]

Run a set of tasks directly against raw datasets without first running model inference.

czbenchmarks.cli.cli_run.run_multi_dataset_task(dataset_names: list[str], embeddings: list[czbenchmarks.datasets.base.BaseDataset], model_args: dict[str, ModelArgsDict], task_args: TaskArgs) list[TaskResult][source]

Run a task and return the results.

czbenchmarks.cli.cli_run.run_task(dataset_name: str, dataset: czbenchmarks.datasets.base.BaseDataset, model_args: dict[str, ModelArgsDict], task_args: TaskArgs) list[TaskResult][source]

Run a task and return the results.

czbenchmarks.cli.cli_run.get_model_arg_permutations(model_args: list[ModelArgs]) dict[str, list[ModelArgsDict]][source]

Generate all the “permutations” of model arguments we want to run for each dataset: E.g. Running 2 variants of scgenept at 2 chunk sizes results in 4 permutations

czbenchmarks.cli.cli_run.write_results(task_results: list[TaskResult], *, cache_options: CacheOptions, output_format: str = DEFAULT_OUTPUT_FORMAT, output_file: str | None = None) None[source]

Format and write results to the given directory or file.

czbenchmarks.cli.cli_run.get_result_url_for_remote(remote_prefix_url: str) str[source]
czbenchmarks.cli.cli_run.set_processed_datasets_cache(dataset: czbenchmarks.datasets.base.BaseDataset, dataset_name: str, *, model_name: str, model_args: ModelArgsDict, cache_options: CacheOptions) None[source]

Write a dataset to the cache A “processed” dataset has been run with model inference for the given arguments.

czbenchmarks.cli.cli_run.try_processed_datasets_cache(dataset_name: str, *, model_name: str, model_args: ModelArgsDict, cache_options: CacheOptions) czbenchmarks.datasets.base.BaseDataset | None[source]

Deserialize and return a processed dataset from the cache if it exists, else return None.

czbenchmarks.cli.cli_run.get_remote_cache_prefix(cache_options: CacheOptions)[source]

get the prefix ending in ‘/’ that the remote processed datasets go under

czbenchmarks.cli.cli_run.get_processed_dataset_cache_filename(dataset_name: str, *, model_name: str, model_args: ModelArgsDict) str[source]

generate a unique filename for the given dataset and model arguments

czbenchmarks.cli.cli_run.get_processed_dataset_cache_path(dataset_name: str, *, model_name: str, model_args: ModelArgsDict) pathlib.Path[source]

Return a unique file path in the cache directory for the given dataset and model arguments.

czbenchmarks.cli.cli_run.parse_model_args(model_name: str, args: argparse.Namespace) ModelArgs[source]

Populate a ModelArgs instance from the given argparse namespace.

czbenchmarks.cli.cli_run.parse_task_args(task_name: str, TaskCls: type[TaskType], args: argparse.Namespace) TaskArgs[source]

Populate a TaskArgs instance from the given argparse namespace.

czbenchmarks.cli.cli_run.parse_batch_json(batch_json_list: list[str]) list[dict[str, Any]][source]

Parse the –batch-json argument. Returns a list of dicts where each entry is a batch of CLI arguments.