jacopotagliabue / you-dont-need-a-bigger-boat Goto Github PK

hi all!

Thank you for these incredible resources :)

how would I run the local flow if I don't have access to any GPUs?

https://github.com/jacopotagliabue/you-dont-need-a-bigger-boat/tree/main/local_flow

Hey there! Thanks for this amazing work! I was wondering if you /any users here had done a comparison between these projects. Seems like this repo just describes a general set of tools and how they link up together whereas the others take an opinionated stance and do the linking themselves?

I'm in the process of evaluating tools / frameworks.

I'm currently a single person looking to set up the groundwork for things to come. So far my plans have been to stay local for as long as possible before moving to some distributed computing framework (Dask gave me a lot of trouble in the past). I'm also looking to avoid using tools such as AWS or GCP for as long as possible so ideally the discussion would revolve around local machines. I'd love to hear thoughts and opinions.

Hey there!

I was recently looking at the remote_flow workflow in the repo. However at the step where data needs to be uploaded into a Snowflake instance, I have been running into an error when calling make upload:

Traceback (most recent call last):
  File "push_data_to_sf.py", line 83, in <module>
    write_chunks(table=sku_to_content_table,
  File "push_data_to_sf.py", line 64, in write_chunks
    conn.upload_file(f"{output_prefix}*", table_name)
  File "/app/connectors/sf_connector.py", line 79, in upload_file
    self._cs.execute(f"COPY INTO {table} FILE_FORMAT = ESCAPED_DQ")
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/cursor.py", line 693, in execute
    Error.errorhandler_wrapper(
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/errors.py", line 258, in errorhandler_wrapper
    cursor.errorhandler(connection, cursor, error_class, error_value)
  File "/usr/local/lib/python3.8/site-packages/snowflake/connector/errors.py", line 188, in default_errorhandler
    raise error_class(
snowflake.connector.errors.ProgrammingError: 002003 (02000): SQL compilation error:
File format 'ESCAPED_DQ' does not exist or not authorized.

It appears that ESCAPED_DQ might be a custom defined file format for Snowflake data ingestion? Where is its definition?

Thanks for putting this repo together!

Thanks a lot for your work. Love your repos and podcasts. I wanted to suggest adding atleast one open source alternative for each stage. It would be helpful for folks that was to quickly build a full system with freely available open source tools. Thanks!