czbenchmarks.utils
Attributes
Functions
|
Initialize Hydra configuration system. |
|
Import a class based on the _target_ field in a configuration file. |
|
Syncs files from an S3 bucket prefix to a local directory. |
|
Return the LastModified timestamp of the remote S3 object, or None if it doesn't exist. |
|
Upload a local file to an s3 prefix, preserving the filename remotely |
|
Upload the contents of a text buffer to the exact S3 location given by remote_url. |
|
Download a remote file from s3 to a local directory, preserving the filename if not specified |
Module Contents
- czbenchmarks.utils.logger
- czbenchmarks.utils.initialize_hydra(config_path='./conf')[source]
Initialize Hydra configuration system.
- Parameters:
config_path – Path to the configuration directory
- czbenchmarks.utils.import_class_from_config(config_path: str)[source]
Import a class based on the _target_ field in a configuration file.
- Parameters:
config_path – Path to the configuration file
- Returns:
The imported class object
- Return type:
class_obj
- czbenchmarks.utils.sync_s3_to_local(bucket, prefix, local_dir, unsigned=True)[source]
Syncs files from an S3 bucket prefix to a local directory.
- Parameters:
bucket – S3 bucket name
prefix – S3 prefix (directory path or file) to sync from
local_dir – Local directory path to sync to
unsigned – Whether to use unsigned requests (default: True)
- czbenchmarks.utils.get_remote_last_modified(remote_url: str, make_unsigned_request: bool = True) datetime.datetime | None [source]
Return the LastModified timestamp of the remote S3 object, or None if it doesn’t exist.
- czbenchmarks.utils.upload_file_to_remote(local_file: str | pathlib.Path, remote_prefix_url: str, make_unsigned_request: bool = False, overwrite_existing: bool = False) None [source]
Upload a local file to an s3 prefix, preserving the filename remotely