Comments (6)
I think we should make the batch size configurable
I see that by default it is 10000
. Either we make it 1000
or maybe get it to a configurable value.
I'll ask on the chat about the possibility of making this setting configurable by the user.
from dbt-trino.
@bachng2017 thank you for reporting this issue.
This is likely caused by the fact that the INSERT
query which is executed against Trino is too long.
We'll address this issue by limiting/making configurable the batch size of the values inserted into Trino.
from dbt-trino.
Reimplementing the default batch size makes a lot of sense:
{% macro trino__get_batch_size() %}
{{ return(1000) }}
{% endmacro %}
Users can still override that in their own projects if they want a different value, by defining a local macro named get_batch_size()
, default__get_batch_size()
, or trino__get_batch_size()
.
from dbt-trino.
It's probably related to: ef05a4e
I will verify that and come back to you.
from dbt-trino.
PR with the change: #18
from dbt-trino.
Closing the issue. The fix will be available in the next release. Default value of batch_size changed to 1000. There is an instruction in README how to override default value if needed.
from dbt-trino.
Related Issues (20)
- Support CASCADE dropping relations
- Solving for large stage depths HOT 3
- Support `CREATE OR REPLACE` HOT 2
- get_relation not working as already_exists HOT 4
- deltalake rename managed table not allowed arised HOT 1
- Failed to connect to Trino cluster using LDAP auth and HTTP connection HOT 3
- Extra credentials in connection HOT 3
- Not able to create snapshot of a model using dbt-trino==1.7.1 HOT 4
- Refactor to use dbt-adapters interface layer HOT 2
- merge_exclude_columns doesn't work
- upgrade to support dbt-core v1.8.0 HOT 1
- Add REPLACE strategy to incremental materialization
- Executing 'dbt docs generate' when the 'where' clause has different combinations of schemas > produces a permission error HOT 3
- Running codegen to generate source yaml fails HOT 1
- Add partition by in configs for incremental materialization HOT 2
- Custom TrinoCredentials support
- Incremental Materialization Macro Uses Variables Before Defined HOT 3
- When running "dbt test --store-fallures" trino reports "destination table XXX already exists" HOT 3
- Support for persist_docs in Starburst Galaxy HOT 1
- __dbt_tmp suffix in table location HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt-trino.