Cloud¶
Boto3x¶
adele.cloud.aws.boto3x.Boto3x
¶
Boto3x is a set of functionalities layered onto top of Boto3, AWS's Python SDK.
It is easily extendable to either client or resource based services that are
in the SDK.
The boto3 session is initialized as part of the Boto3x client, and available as
part of the session property. That session can be used to easily switch to a
a regular boto3 client or resource, for example in the following way.
>>> boto3x = Boto3x()
>>> s3 = boto3x.session.client("s3")
In that sense, boto3x can be used anywhere that boto3 is required, with the additional functionality received via boto3x.
list_objects_all(self, **base_kwargs)
¶
Retrieve information from all objects in an S3 bucket. base_kwargs can take any of the parameters accepted by s3 boto client, Bucket, Prefix, etc.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
Bucket |
str |
The reference bucket name |
required |
Prefix |
str |
The folder name within the bucket |
required |
Examples:
>>> boto3x = Boto3x()
>>> boto3x.list_objects_all(Bucket="mybucketname", Prefix="folderinmybucket")
Returns:
| Type | Description |
|---|---|
Generator |
A generator of the list of objects found at the specified location |
get_secret(self, secret_name)
¶
Retrieves the specified secret from secrets manager
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
secret_name |
str |
The name of the secret in secrets manager |
required |
Examples:
>>> boto3x = Boto3x()
>>> boto3x.get_secret("GH_TOKEN")
Returns:
| Type | Description |
|---|---|
dict |
A decoded json as a set of key value pairs |
create_presigned_url(self, bucket_name, object_key, event='get_object', expiration=3600)
¶
Generate a presigned URL to share an S3 object
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
bucket_name |
str |
Container name. |
required |
object_key |
str |
Object of interest within the container. |
required |
event |
str |
Type of event. Defaults to 'get_object'. |
'get_object' |
expiration |
int |
Defaults to 3600. |
3600 |
Returns:
| Type | Description |
|---|---|
response |
The presigned url or None if failed. msg (str): Empty or reason for failure. |
record_transaction(self, bucket_name, log_path, payload={})
¶
Drop a .json file into a bucket with some information in it
BigQuery¶
adele.cloud.gcp.bigquery.BigQuery
¶
BigQuery is a class object to be used to generate a BigQuery API and Data Transfer Service API clients. Treat this object as a container for all API clients requierd to interact with Google Cloud Platform. The "secret" string is generated when BigQuery is initialized. The various clients are evaluated by accessing their respective attributes. For example:
>>> bigquery = BigQuery("dev")
>>> bq_client = bigquery.client
>>> dt_client = bigquery.transfer_client
Use the clients as if you were accessing them directly.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
environment |
str |
Valid values are "dev" and "prod" |
required |
GitHub¶
adele.cloud.github.release.GithubRelease
¶
Client to interact with Github API to manage reading and writing releases to a specific repository.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
owner |
str |
The GitHub account name |
required |
repo |
str |
The name the repository |
required |
Attributes:
| Name | Type | Description |
|---|---|---|
token |
str |
A 40 character Personal Access Token retrieved from GitHub |
branch |
str |
The specific branch on the repository to read from and and write to |
Examples:
>>> from adele.cloud.github.release import GithubRelease
>>> from adele.cloud.aws.boto3x import Boto3x
>>> boto3x = Boto3x()
>>> tkn = boto3x.get_secret("GH_TOKEN")
>>> gh = GithubRelease("Material-Dev","lrw-insights-portal-ba-api")
>>> setattr(gh,"token",tkn)
>>> setattr(gh,"branch","develop")
>>> gh.latest_branch_release()
>>> gh.create_release()