czbenchmarks.file_cache

File caching utilities for remote storage operations.

This module provides functions for caching datasets and model outputs to/from remote storage, primarily AWS S3. It includes functionality for downloading, uploading, and managing cached processed datasets.

Attributes

log

Classes

CacheOptions

Configuration options for caching datasets and model outputs.

Module Contents

czbenchmarks.file_cache.log[source]

class czbenchmarks.file_cache.CacheOptions(/, **data: Any)[source]

Bases: pydantic.BaseModel

Configuration options for caching datasets and model outputs.

download_embeddings

Whether to download embeddings from the remote cache.

Type:: bool

upload_embeddings

Whether to upload embeddings to the remote cache.

Type:: bool

upload_results

Whether to upload results to the remote cache.

Type:: bool

remote_cache_url

URL of the remote cache.

Type:: str

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

download_embeddings: bool

upload_embeddings: bool

upload_results: bool

remote_cache_url: str

classmethod from_args(args: argparse.Namespace) → CacheOptions[source]