czbenchmarks.utils

Attributes

logger

Functions

initialize_hydra([config_path])

Initialize Hydra configuration system.

import_class_from_config(config_path)

Import a class based on the _target_ field in a configuration file.

sync_s3_to_local(bucket, prefix, local_dir[, unsigned])

Syncs files from an S3 bucket prefix to a local directory.

get_remote_last_modified(→ datetime.datetime | None)

Return the LastModified timestamp of the remote S3 object, or None if it doesn't exist.

upload_file_to_remote(→ None)

Upload a local file to an s3 prefix, preserving the filename remotely

upload_blob_to_remote(→ None)

Upload the contents of a text buffer to the exact S3 location given by remote_url.

download_file_from_remote(→ None)

Download a remote file from s3 to a local directory, preserving the filename if not specified

Module Contents

czbenchmarks.utils.logger
czbenchmarks.utils.initialize_hydra(config_path='./conf')[source]

Initialize Hydra configuration system.

Parameters:

config_path – Path to the configuration directory

czbenchmarks.utils.import_class_from_config(config_path: str)[source]

Import a class based on the _target_ field in a configuration file.

Parameters:

config_path – Path to the configuration file

Returns:

The imported class object

Return type:

class_obj

czbenchmarks.utils.sync_s3_to_local(bucket, prefix, local_dir, unsigned=True)[source]

Syncs files from an S3 bucket prefix to a local directory.

Parameters:
  • bucket – S3 bucket name

  • prefix – S3 prefix (directory path or file) to sync from

  • local_dir – Local directory path to sync to

  • unsigned – Whether to use unsigned requests (default: True)

czbenchmarks.utils.get_remote_last_modified(remote_url: str, make_unsigned_request: bool = True) datetime.datetime | None[source]

Return the LastModified timestamp of the remote S3 object, or None if it doesn’t exist.

czbenchmarks.utils.upload_file_to_remote(local_file: str | pathlib.Path, remote_prefix_url: str, make_unsigned_request: bool = False, overwrite_existing: bool = False) None[source]

Upload a local file to an s3 prefix, preserving the filename remotely

czbenchmarks.utils.upload_blob_to_remote(blob: bytes, remote_url: str, make_unsigned_request: bool = False, overwrite_existing: bool = False) None[source]

Upload the contents of a text buffer to the exact S3 location given by remote_url.

czbenchmarks.utils.download_file_from_remote(remote_url: str, local_directory: str | pathlib.Path, local_filename: str | None = None, make_unsigned_request: bool = True) None[source]

Download a remote file from s3 to a local directory, preserving the filename if not specified