Add a Custom Metric

This guide explains how to add your own metric to the cz-benchmarks metrics system. Follow the steps below to implement and register your custom metric.


Step-by-Step Guide

1. Define a New Metric Type

First, define your new metric type in the MetricType enum.

  1. Open the file: czbenchmarks/metrics/types.py.

  2. Add a new entry to the MetricType enum. For example:

    from enum import Enum
    
    class MetricType(Enum):
         ...
         YOUR_NEW_METRIC = "your_new_metric"
    

2. Implement the Metric Function

Next, register a function that computes the value of your metric.

  1. Define the function in the czbenchmarks/metrics/implementations.py. For example:

    def your_new_metric_func(y_true, y_pred):
         """
         Computes the value of your new metric.
    
         Args:
              y_true (list): Ground truth values.
              y_pred (list): Predicted values.
    
         Returns:
              float: Computed metric value.
         """
         # Replace with your metric calculation logic
         return float(...)
    
  2. Ensure the function:

    • Accepts any required arguments it needs to compute its output (e.g., y_true, y_pred). These can be any parameters.

    • Returns a float value representing the metric.


3. Register the Metric

Register your metric with the MetricRegistry to make it available for use.

  1. Open the file: czbenchmarks/metrics/implementations.py.

  2. Add the registration code:

    from czbenchmarks.metrics.registry import metrics_registry
    from czbenchmarks.metrics.types import MetricType
    
    metrics_registry.register(
         MetricType.YOUR_NEW_METRIC,
         func=your_new_metric_func,
         required_args={"y_true", "y_pred"},
         default_params={},  # Add any default parameters if needed
         description="A brief description of your new metric.",
         tags={"custom"}  # Add relevant tags for categorization
    )
    

4. Use the Metric

Once your metric is registered, you can use it within the czbenchmarks/tasks directory by integrating it into a relevant task file. Here’s how:

  1. Navigate to the appropriate task file in the czbenchmarks/tasks directory or create a new task file.

  2. Import the necessary components:

    from czbenchmarks.metrics.types import MetricType, metrics_registry
    
  3. Compute your custom metric by calling the compute method:

    metric_value = metrics_registry.compute(
        MetricType.YOUR_NEW_METRIC,
        y_true=true_values,
        y_pred=pred_values
    )
    
  4. Use the computed metric_value as needed in your task logic.

This allows you to seamlessly integrate your custom metric into the existing benchmarking workflow.


Best Practices for Adding Metrics

  • Documentation: Provide clear and concise documentation for your metric, including its purpose and usage.

  • Tags: Use descriptive tags to categorize your metric (e.g., {"custom", "classification"}).

  • Error Handling: Ensure your metric function handles edge cases gracefully and raises meaningful errors.

  • Testing: Write unit tests to validate the correctness of your metric implementation.