GithubHelp home page GithubHelp logo

Comments (6)

findinpath avatar findinpath commented on August 15, 2024 2

I think we should make the batch size configurable

https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/include/global_project/macros/materializations/seeds/helpers.sql#L60-L62

I see that by default it is 10000. Either we make it 1000 or maybe get it to a configurable value.
I'll ask on the chat about the possibility of making this setting configurable by the user.

from dbt-trino.

findinpath avatar findinpath commented on August 15, 2024 1

@bachng2017 thank you for reporting this issue.
This is likely caused by the fact that the INSERT query which is executed against Trino is too long.
We'll address this issue by limiting/making configurable the batch size of the values inserted into Trino.

from dbt-trino.

jtcohen6 avatar jtcohen6 commented on August 15, 2024 1

Reimplementing the default batch size makes a lot of sense:

{% macro trino__get_batch_size() %}
  {{ return(1000) }}
{% endmacro %}

Users can still override that in their own projects if they want a different value, by defining a local macro named get_batch_size(), default__get_batch_size(), or trino__get_batch_size().

from dbt-trino.

hovaesco avatar hovaesco commented on August 15, 2024

It's probably related to: ef05a4e

I will verify that and come back to you.

from dbt-trino.

hovaesco avatar hovaesco commented on August 15, 2024

PR with the change: #18

from dbt-trino.

hovaesco avatar hovaesco commented on August 15, 2024

Closing the issue. The fix will be available in the next release. Default value of batch_size changed to 1000. There is an instruction in README how to override default value if needed.

from dbt-trino.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.