czbenchmarks.cli.cli_run
Attributes
Classes
!!! abstract "Usage Documentation" |
|
!!! abstract "Usage Documentation" |
|
!!! abstract "Usage Documentation" |
|
!!! abstract "Usage Documentation" |
Functions
|
Add run command arguments to the parser. |
|
Execute a series of tasks using multiple models on a collection of datasets. |
|
Run a set of tasks against a set of datasets. Runs inference if any model_args are specified. |
|
Execute a series of tasks using multiple models on a collection of datasets. |
Load the processed dataset from the cache if it exists, else run inference and save to cache. |
|
|
Run a set of tasks directly against raw datasets without first running model inference. |
|
Run a task and return the results. |
|
Run a task and return the results. |
|
Generate all the "permutations" of model arguments we want to run for each dataset: |
|
Format and write results to the given directory or file. |
|
|
|
Write a dataset to the cache |
Deserialize and return a processed dataset from the cache if it exists, else return None. |
|
|
get the prefix ending in '/' that the remote processed datasets go under |
generate a unique filename for the given dataset and model arguments |
|
|
Return a unique file path in the cache directory for the given dataset and model arguments. |
|
Populate a ModelArgs instance from the given argparse namespace. |
|
Populate a TaskArgs instance from the given argparse namespace. |
|
Parse the --batch-json argument. |
Module Contents
- czbenchmarks.cli.cli_run.VALID_OUTPUT_FORMATS = ['json', 'yaml']
- czbenchmarks.cli.cli_run.DEFAULT_OUTPUT_FORMAT = 'json'
- czbenchmarks.cli.cli_run.TaskType
- czbenchmarks.cli.cli_run.ModelArgsDict
- czbenchmarks.cli.cli_run.RuntimeMetricsDict
- class czbenchmarks.cli.cli_run.ModelArgs(/, **data: Any)[source]
Bases:
pydantic.BaseModel
- !!! abstract “Usage Documentation”
[Models](../concepts/models.md)
A base class for creating Pydantic models.
- __class_vars__
The names of the class variables defined on the model.
- __private_attributes__
Metadata about the private attributes of the model.
- __signature__
The synthesized __init__ [Signature][inspect.Signature] of the model.
- __pydantic_complete__
Whether model building is completed, or if there are still undefined fields.
- __pydantic_core_schema__
The core schema of the model.
- __pydantic_custom_init__
Whether the model has a custom __init__ function.
- __pydantic_decorators__
Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.
- __pydantic_generic_metadata__
Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.
- __pydantic_parent_namespace__
Parent namespace of the model, used for automatic rebuilding of models.
- __pydantic_post_init__
The name of the post-init method for the model, if defined.
- __pydantic_root_model__
Whether the model is a [RootModel][pydantic.root_model.RootModel].
- __pydantic_serializer__
The pydantic-core SchemaSerializer used to dump instances of the model.
- __pydantic_validator__
The pydantic-core SchemaValidator used to validate instances of the model.
- __pydantic_fields__
A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.
- __pydantic_computed_fields__
A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.
- __pydantic_extra__
A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.
- __pydantic_fields_set__
The names of fields explicitly set during instantiation.
- __pydantic_private__
Values of private attributes set on the model instance.
- class czbenchmarks.cli.cli_run.TaskArgs(/, **data: Any)[source]
Bases:
pydantic.BaseModel
,Generic
[TaskType
]- !!! abstract “Usage Documentation”
[Models](../concepts/models.md)
A base class for creating Pydantic models.
- __class_vars__
The names of the class variables defined on the model.
- __private_attributes__
Metadata about the private attributes of the model.
- __signature__
The synthesized __init__ [Signature][inspect.Signature] of the model.
- __pydantic_complete__
Whether model building is completed, or if there are still undefined fields.
- __pydantic_core_schema__
The core schema of the model.
- __pydantic_custom_init__
Whether the model has a custom __init__ function.
- __pydantic_decorators__
Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.
- __pydantic_generic_metadata__
Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.
- __pydantic_parent_namespace__
Parent namespace of the model, used for automatic rebuilding of models.
- __pydantic_post_init__
The name of the post-init method for the model, if defined.
- __pydantic_root_model__
Whether the model is a [RootModel][pydantic.root_model.RootModel].
- __pydantic_serializer__
The pydantic-core SchemaSerializer used to dump instances of the model.
- __pydantic_validator__
The pydantic-core SchemaValidator used to validate instances of the model.
- __pydantic_fields__
A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.
- __pydantic_computed_fields__
A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.
- __pydantic_extra__
A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.
- __pydantic_fields_set__
The names of fields explicitly set during instantiation.
- __pydantic_private__
Values of private attributes set on the model instance.
- model_config
Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].
- task: TaskType
- class czbenchmarks.cli.cli_run.TaskResult(/, **data: Any)[source]
Bases:
pydantic.BaseModel
- !!! abstract “Usage Documentation”
[Models](../concepts/models.md)
A base class for creating Pydantic models.
- __class_vars__
The names of the class variables defined on the model.
- __private_attributes__
Metadata about the private attributes of the model.
- __signature__
The synthesized __init__ [Signature][inspect.Signature] of the model.
- __pydantic_complete__
Whether model building is completed, or if there are still undefined fields.
- __pydantic_core_schema__
The core schema of the model.
- __pydantic_custom_init__
Whether the model has a custom __init__ function.
- __pydantic_decorators__
Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.
- __pydantic_generic_metadata__
Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.
- __pydantic_parent_namespace__
Parent namespace of the model, used for automatic rebuilding of models.
- __pydantic_post_init__
The name of the post-init method for the model, if defined.
- __pydantic_root_model__
Whether the model is a [RootModel][pydantic.root_model.RootModel].
- __pydantic_serializer__
The pydantic-core SchemaSerializer used to dump instances of the model.
- __pydantic_validator__
The pydantic-core SchemaValidator used to validate instances of the model.
- __pydantic_fields__
A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.
- __pydantic_computed_fields__
A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.
- __pydantic_extra__
A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.
- __pydantic_fields_set__
The names of fields explicitly set during instantiation.
- __pydantic_private__
Values of private attributes set on the model instance.
- model_type: czbenchmarks.models.types.ModelType
- model_args: ModelArgsDict
- metrics: list[czbenchmarks.metrics.types.MetricResult]
- runtime_metrics: RuntimeMetricsDict
- class czbenchmarks.cli.cli_run.CacheOptions(/, **data: Any)[source]
Bases:
pydantic.BaseModel
- !!! abstract “Usage Documentation”
[Models](../concepts/models.md)
A base class for creating Pydantic models.
- __class_vars__
The names of the class variables defined on the model.
- __private_attributes__
Metadata about the private attributes of the model.
- __signature__
The synthesized __init__ [Signature][inspect.Signature] of the model.
- __pydantic_complete__
Whether model building is completed, or if there are still undefined fields.
- __pydantic_core_schema__
The core schema of the model.
- __pydantic_custom_init__
Whether the model has a custom __init__ function.
- __pydantic_decorators__
Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.
- __pydantic_generic_metadata__
Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.
- __pydantic_parent_namespace__
Parent namespace of the model, used for automatic rebuilding of models.
- __pydantic_post_init__
The name of the post-init method for the model, if defined.
- __pydantic_root_model__
Whether the model is a [RootModel][pydantic.root_model.RootModel].
- __pydantic_serializer__
The pydantic-core SchemaSerializer used to dump instances of the model.
- __pydantic_validator__
The pydantic-core SchemaValidator used to validate instances of the model.
- __pydantic_fields__
A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.
- __pydantic_computed_fields__
A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.
- __pydantic_extra__
A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.
- __pydantic_fields_set__
The names of fields explicitly set during instantiation.
- __pydantic_private__
Values of private attributes set on the model instance.
- classmethod from_args(args: argparse.Namespace) CacheOptions [source]
- czbenchmarks.cli.cli_run.add_arguments(parser: argparse.ArgumentParser) None [source]
Add run command arguments to the parser.
- czbenchmarks.cli.cli_run.main(parsed_args: argparse.Namespace) None [source]
Execute a series of tasks using multiple models on a collection of datasets.
This function handles the benchmarking process by iterating over the specified datasets, running inference with the provided models to generate results, and running the tasks to evaluate the generated outputs.
- czbenchmarks.cli.cli_run.run(dataset_names: list[str], model_args: list[ModelArgs], task_args: list[TaskArgs], cache_options: CacheOptions) list[TaskResult] [source]
Run a set of tasks against a set of datasets. Runs inference if any model_args are specified.
- czbenchmarks.cli.cli_run.run_with_inference(dataset_names: list[str], model_args: list[ModelArgs], task_args: list[TaskArgs], cache_options: CacheOptions) list[TaskResult] [source]
Execute a series of tasks using multiple models on a collection of datasets.
This function handles the benchmarking process by iterating over the specified datasets, running inference with the provided models to generate results, and running the tasks to evaluate the generated outputs.
- czbenchmarks.cli.cli_run.run_inference_or_load_from_cache(dataset_name: str, *, model_name: str, model_args: ModelArgsDict, cache_options: CacheOptions) czbenchmarks.datasets.base.BaseDataset [source]
Load the processed dataset from the cache if it exists, else run inference and save to cache.
- czbenchmarks.cli.cli_run.run_without_inference(dataset_names: list[str], task_args: list[TaskArgs]) list[TaskResult] [source]
Run a set of tasks directly against raw datasets without first running model inference.
- czbenchmarks.cli.cli_run.run_multi_dataset_task(dataset_names: list[str], embeddings: list[czbenchmarks.datasets.base.BaseDataset], model_args: dict[str, ModelArgsDict], task_args: TaskArgs) list[TaskResult] [source]
Run a task and return the results.
- czbenchmarks.cli.cli_run.run_task(dataset_name: str, dataset: czbenchmarks.datasets.base.BaseDataset, model_args: dict[str, ModelArgsDict], task_args: TaskArgs) list[TaskResult] [source]
Run a task and return the results.
- czbenchmarks.cli.cli_run.get_model_arg_permutations(model_args: list[ModelArgs]) dict[str, list[ModelArgsDict]] [source]
Generate all the “permutations” of model arguments we want to run for each dataset: E.g. Running 2 variants of scgenept at 2 chunk sizes results in 4 permutations
- czbenchmarks.cli.cli_run.write_results(task_results: list[TaskResult], *, cache_options: CacheOptions, output_format: str = DEFAULT_OUTPUT_FORMAT, output_file: str | None = None) None [source]
Format and write results to the given directory or file.
- czbenchmarks.cli.cli_run.set_processed_datasets_cache(dataset: czbenchmarks.datasets.base.BaseDataset, dataset_name: str, *, model_name: str, model_args: ModelArgsDict, cache_options: CacheOptions) None [source]
Write a dataset to the cache A “processed” dataset has been run with model inference for the given arguments.
- czbenchmarks.cli.cli_run.try_processed_datasets_cache(dataset_name: str, *, model_name: str, model_args: ModelArgsDict, cache_options: CacheOptions) czbenchmarks.datasets.base.BaseDataset | None [source]
Deserialize and return a processed dataset from the cache if it exists, else return None.
- czbenchmarks.cli.cli_run.get_remote_cache_prefix(cache_options: CacheOptions)[source]
get the prefix ending in ‘/’ that the remote processed datasets go under
- czbenchmarks.cli.cli_run.get_processed_dataset_cache_filename(dataset_name: str, *, model_name: str, model_args: ModelArgsDict) str [source]
generate a unique filename for the given dataset and model arguments
- czbenchmarks.cli.cli_run.get_processed_dataset_cache_path(dataset_name: str, *, model_name: str, model_args: ModelArgsDict) pathlib.Path [source]
Return a unique file path in the cache directory for the given dataset and model arguments.
- czbenchmarks.cli.cli_run.parse_model_args(model_name: str, args: argparse.Namespace) ModelArgs [source]
Populate a ModelArgs instance from the given argparse namespace.