get {:ok, ...} when in fact it was an insert error about elixir-google-api

[dazuma]

This is expected behavior; it's how the BigQuery API is defined. If there are insertion errors not related to transport, the call itself will succeed, and the errors are returned as part of the response. See https://cloud.google.com/bigquery/docs/reference/rest/v2/tabledata/insertAll#response for more info on the response data type. (If there are transport-related errors such as authentication failed or network problems, then you should get an {:error, reason} response.)

[henry-hz]

Thanks @dazuma for the fast feeback. The question is, can I rely on "insert_errors: nil" to be sure that no data was inserted on BigQuery ? Or... mabye, are there cases that we can insert data, even with errors ? I ask, because I am writing a transaction feature to have sync data between BQ and Postgres, and in case BQ fails, I have to rollback Postgres

[dazuma]

If you get a success response, and insert_errors is set to an empty list, then yes, you can rely on the rows having been inserted correctly. Otherwise, the insert_errors list will have entries for the rows that failed. Note that it is possible for a partial success to have taken place: rows that failed will have a corresponding entry in the insert_errors list, while rows were successfully inserted will not. For more details, I suggest studying https://cloud.google.com/bigquery/troubleshooting-errors#streaming

One other thing to be aware of if data consistency is critical. If you get a HTTP error or some other condition that results in an {:error, reason} response, then the actual end state may be indeterminate. (For example, it is possible that all the inserts succeeded, but due to network issues, the response never made it back to you.) See https://cloud.google.com/bigquery/streaming-data-into-bigquery#dataconsistency for some techniques you can use to handle this case.

[dazuma]

Closing. Feel free to re-open if your question hasn't been answered yet.