Cannot generate observability report about elementary

[sphinks]

Same here. Have the issue with 0.14.1 version as well.
As workaround I'm recreating elementary schema:

dbt run -t prod --select elementary --vars '{"elementary_full_refresh": "true"}'

But couple days after it stops working and fails with the same issue.

[sphinks]

Any hints about the issue? What could be wrong? It keeps failing each several days.

[haritamar]

Hi @naomicarrillo5 @sphinks ,
Thanks for opening this issue and apologies for the delay.

The issue above seems potentially like a resources issue (likely memory). @sphinks are you also getting the "died with SIGTERM" error?
To better understand if this is an issue in Elementary, It would be helpful if you can share details on the setup on which you are running the Elementary CLI:

• Is it happening on a local run or in production?
• How much memory is provided to the machine running Elementary?
• What is the memory consumption by Elementary? (if you can obtain that)

Thanks,
Itamar

[sphinks]

@haritamar

1. it is happening in production (have no chance to reproduce the error locally)
2. 4 Gb of memory is used.
3. Consumption up to 100%, but that happens with previous version of elementary and it was working. But I got your point - try to increase memory up to 8Gb and will check if that will help.

Any idea why does new version consumes more memory?

[haritamar]

@sphinks - I'm actually not aware of a specific issue with newer versions that would cause this - did you notice it only happened to you in 0.14.1?
From which version did you upgrade?

[sphinks]

@haritamar i've notice that on 0.14.1. It was updated from 0.13.0.
As in traceback there is a reference to test API and the issue with get_all_test_results_db_rows I wonder could it be that table layour has changed in new veriosn? Should be drop and recreate tables from the scratch?

[haritamar]

Thanks @sphinks .
I don't believe this has anything to do with the table schemas - it's likely that the new version consumes more memory than before (which is interesting, we'll keep track of more evidence of this and see if we can reduce the memory footprint)

Did increasing memory work for you though?