andrewcstewart / ds4fnp

Data Stacks For Fun & Nonprofits!

Hi there 👋

Social

Techs

Misc

ds4fnp's Issues

Meltano, dbt have package dependency conflicts

Running into dependency conflicts with the most recent versions of dbt, meltano. My requirements.txt file attached. Do you know which versions to pin these to? (With the dependency issues, might help to use a requirements.txt file for the project?)

requirements.txt

$ pip install meltano
...
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
dbt-core 1.0.2 requires click<9,>=8, but you have click 7.1.2 which is incompatible.

$ pip install dbt-postgres
...
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
meltano 1.96.0 requires click<8.0,>=7.0, but you have click 8.0.4 which is incompatible.
flask 1.1.4 requires click<8.0,>=5.1, but you have click 8.0.4 which is incompatible.

Post v1.0, dbt package now installs with dbt-postgres

Getting this error when trying to install via pip install dbt directly. Seems like choosing pip install dbt-postgres would fix it.

$ pip install dbt
Collecting dbt
  Downloading dbt-1.0.0.tar.gz (2.9 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      
      As of v1.0.0, `pip install dbt` is no longer supported.
      Instead, please use one of the following.
      
      **To use dbt with your specific database, platform, or query engine:**
      
          pip install dbt-<adapter>
      
          See full list: https://docs.getdbt.com/docs/available-adapters
      
      **For developers of integrations with dbt Core:**
      
          pip install dbt-core
      
          Be advised, dbt Cores python API is not yet stable or documented
          (https://docs.getdbt.com/docs/running-a-dbt-project/dbt-api)
      
      **For the previous behavior of `pip install dbt`:**
      
          pip install dbt-core dbt-postgres dbt-redshift dbt-snowflake dbt-bigquery
      
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Deprecation warnings for dbt project YAML file

Receiving the following deprecation warnings when trying dbt run with current version of dbt.

$ dbt run
14:40:53  [WARNING]: Deprecated functionality
The `source-paths` config has been renamed to `model-paths`. Please update your
`dbt_project.yml` configuration to reflect this change.
14:40:53  [WARNING]: Deprecated functionality
The `data-paths` config has been renamed to `seed-paths`. Please update your
`dbt_project.yml` configuration to reflect this change.
14:40:53  Running with dbt=1.0.2

Demonstration of kedro meltano plugin

Meltano custom plugin examples

ie: meltano add utility sqlfluff

Ideas:

• sqlfluff
• https://github.com/Squarespace/pgbedrock
• pgcli
• ideas from https://datacoves.com/dbt-libs
• https://pypyr.io/

Investigate DuckDB as local teaching dw

mention https://github.com/victorcouste/data-tools

Consider incorporating Grouparoo

https://www.grouparoo.com/

Create section to link off to similar resources

Examples:

• https://duckdb.org/2022/10/12/modern-data-stack-in-a-box.html