czbenchmarks.metrics.types

Classes

MetricType	Enumeration of all supported metric types.
MetricInfo	Stores metadata about a metric.
MetricRegistry	Central registry for all available metrics.
MetricResult	!!! abstract "Usage Documentation"
AggregatedMetricResult	!!! abstract "Usage Documentation"

Module Contents

class czbenchmarks.metrics.types.MetricType(*args, **kwds)

Bases: enum.Enum

Enumeration of all supported metric types.

Each metric type corresponds to a specific evaluation metric that can be computed. The value is the string identifier used in results dictionaries.

ADJUSTED_RAND_INDEX = 'adjusted_rand_index'

NORMALIZED_MUTUAL_INFO = 'normalized_mutual_info'

SILHOUETTE_SCORE = 'silhouette_score'

ENTROPY_PER_CELL = 'entropy_per_cell'

BATCH_SILHOUETTE = 'batch_silhouette'

MEAN_FOLD_ACCURACY = 'mean_fold_accuracy'

MEAN_FOLD_F1_SCORE = 'mean_fold_f1'

MEAN_FOLD_PRECISION = 'mean_fold_precision'

MEAN_FOLD_RECALL = 'mean_fold_recall'

MEAN_FOLD_AUROC = 'mean_fold_auroc'

MEAN_SQUARED_ERROR = 'mean_squared_error'

PEARSON_CORRELATION = 'PEARSON_CORRELATION'

JACCARD = 'jaccard'

class czbenchmarks.metrics.types.MetricInfo(/, **data: Any)

Bases: pydantic.BaseModel

Stores metadata about a metric.

func: Callable

The function that computes the metric

required_args: Set[str]

Set of required argument names

default_params: Dict[str, Any]

Default parameters for the metric function

description: str | None = None

Optional documentation string for custom metrics

tags: Set[str] = None

Set of tags for grouping related metrics

class czbenchmarks.metrics.types.MetricRegistry

Central registry for all available metrics.

Handles registration and computation of metrics with proper validation.

register(metric_type: MetricType, func: Callable, required_args: Set[str] | None = None, default_params: Dict[str, Any] | None = None, description: str = '', tags: Set[str] | None = None) → None

Register a new metric.

Parameters:

• metric_type – Type of metric to register

• func – Function that computes the metric

• required_args – Set of required argument names

• default_params – Default parameters for the metric function

• description – Documentation string

• tags – Set of tags for grouping metrics

compute(metric_type: MetricType, **kwargs) → float

Compute a metric with the given parameters.

Parameters:

• metric_type – Type of metric to compute

• **kwargs – Arguments to pass to metric function

Returns:

Computed metric value

Raises:

ValueError – If metric type unknown or missing required args

get_info(metric_type: MetricType) → MetricInfo

Get metadata about a metric.

Parameters:

metric_type – Type of metric

Returns:

MetricInfo object with metric metadata

Raises:

ValueError – If metric type unknown

list_metrics(tags: Set[str] | None = None) → Set[MetricType]

List available metrics, optionally filtered by tags.

Parameters:

tags – If provided, only return metrics with all these tags

Returns:

Set of matching MetricType values

class czbenchmarks.metrics.types.MetricResult(/, **data: Any)

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”

[Models](../concepts/models.md)

A base class for creating Pydantic models.

__class_vars__

The names of the class variables defined on the model.

__private_attributes__

Metadata about the private attributes of the model.

__signature__

The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__

Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__

The core schema of the model.

__pydantic_custom_init__

Whether the model has a custom __init__ function.

__pydantic_decorators__

Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__

Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__

Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__

The name of the post-init method for the model, if defined.

__pydantic_root_model__

Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__

The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__

The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__

A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__

A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__

A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__

The names of fields explicitly set during instantiation.

__pydantic_private__

Values of private attributes set on the model instance.

metric_type: MetricType

value: float

params: Dict[str, Any] | None = None

property aggregation_key: str

return a key based on the metric type and params in order to aggregate the same metrics together

class czbenchmarks.metrics.types.AggregatedMetricResult(/, **data: Any)

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”

[Models](../concepts/models.md)

A base class for creating Pydantic models.

__class_vars__

The names of the class variables defined on the model.

__private_attributes__

Metadata about the private attributes of the model.

__signature__

The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__

Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__

The core schema of the model.

__pydantic_custom_init__

Whether the model has a custom __init__ function.

__pydantic_decorators__

Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__

Metadata for generic models; contains data used for a similar purpose to __args__, __origin__, __parameters__ in typing-module generics. May eventually be replaced by these.

__pydantic_parent_namespace__

Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__

The name of the post-init method for the model, if defined.

__pydantic_root_model__

Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__

The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__

The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__

A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__

A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__

A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__

The names of fields explicitly set during instantiation.

__pydantic_private__

Values of private attributes set on the model instance.

metric_type: MetricType

params: Dict[str, Any] | None = None

n_values: int

value: float

value_std_dev: float | None

values_raw: list[float]