czbenchmarks.utils

Attributes

logger

Functions

initialize_hydra([config_path])	Initialize Hydra configuration system.
import_class_from_config(config_path)	Import a class based on the _target_ field in a configuration file.
sync_s3_to_local(bucket, prefix, local_dir[, unsigned])	Syncs files from an S3 bucket prefix to a local directory.
get_remote_last_modified(→ datetime.datetime | None)	Return the LastModified timestamp of the remote S3 object, or None if it doesn't exist.
upload_file_to_remote(→ None)	Upload a local file to an s3 prefix, preserving the filename remotely
upload_blob_to_remote(→ None)	Upload the contents of a text buffer to the exact S3 location given by remote_url.
download_file_from_remote(→ None)	Download a remote file from s3 to a local directory, preserving the filename if not specified

Module Contents

czbenchmarks.utils.logger

czbenchmarks.utils.initialize_hydra(config_path='./conf')

Initialize Hydra configuration system.

Parameters:

config_path – Path to the configuration directory

czbenchmarks.utils.import_class_from_config(config_path: str)

Import a class based on the _target_ field in a configuration file.

Parameters:

config_path – Path to the configuration file

Returns:

The imported class object

Return type:

class_obj

czbenchmarks.utils.sync_s3_to_local(bucket, prefix, local_dir, unsigned=True)

Syncs files from an S3 bucket prefix to a local directory.

Parameters:

• bucket – S3 bucket name

• prefix – S3 prefix (directory path or file) to sync from

• local_dir – Local directory path to sync to

• unsigned – Whether to use unsigned requests (default: True)

czbenchmarks.utils.get_remote_last_modified(remote_url: str, make_unsigned_request: bool = True) → datetime.datetime | None

Return the LastModified timestamp of the remote S3 object, or None if it doesn’t exist.

czbenchmarks.utils.upload_file_to_remote(local_file: str | pathlib.Path, remote_prefix_url: str, make_unsigned_request: bool = False, overwrite_existing: bool = False) → None

Upload a local file to an s3 prefix, preserving the filename remotely

czbenchmarks.utils.upload_blob_to_remote(blob: bytes, remote_url: str, make_unsigned_request: bool = False, overwrite_existing: bool = False) → None

Upload the contents of a text buffer to the exact S3 location given by remote_url.

czbenchmarks.utils.download_file_from_remote(remote_url: str, local_directory: str | pathlib.Path, local_filename: str | None = None, make_unsigned_request: bool = True) → None

Download a remote file from s3 to a local directory, preserving the filename if not specified