gupy-io / target-s3-parquet Goto Github PK

• Make it possible to define a partition columns from configuration (verifying if the column exists)
• Remove partition key from the schema definition.

Since this project uses awswrangler to manage creds that means it follows the default boto3 session procedure to find creds. See below

boto3 (by relation awswrangler) looks through the following locations to find credentials

1. Passing credentials as parameters in the boto.client() method
2. Passing credentials as parameters when creating a Session object
3. Environment variables
4. Shared credential file (~/.aws/credentials)
5. AWS config file (~/.aws/config)
6. Assume Role provider
7. Boto2 config file (/etc/boto.cfg and ~/.boto)
8. Instance metadata service on an Amazon EC2 instance that has an IAM role configured.

My use cases are all below #3 in that list. So they should be configurable, but not required if the machine I am running this on has other means of getting credentials.

Resources:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html#configuring-credentials
https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/002%20-%20Sessions.ipynb
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html

Hey @ndrluis @lorransr @Marcos314 - this is more of a question that an issue but I was just checking out this target and based on the requirement to have athena_database and using aws wrangler I was wondering if this is meant only to be used specifically with Athena or is it intending to support writing plain parquet files to S3 as well. Similar to what https://github.com/transferwise/pipelinewise-target-s3-csv does with CSVs but in Parquet format instead. No problem either way but wanted to clarify the use cases for this target 😄 .