.tif file size exceeds regular tif limits about rpops

[nkruskamp]

When creating a new GeoTIFF with no compression, GDAL computes in advance the size of the resulting file. If that computed file size is over 4GiB, GDAL will automatically decide to create a BigTIFF file. However, when compression is used, it is not possible in advance to known the final size of the file, so classical TIFF will be chosen. In that case, the user must explicitly require the creation of a BigTIFF with BIGTIFF=YES if the final file is anticipated to be too big for classical TIFF format. If BigTIFF creation is not explicitly asked or guessed and the resulting file is too big for classical TIFF, libtiff will fail with an error message like “TIFFAppendToStrip:Maximum TIFF file size exceeded”. from https://gdal.org/drivers/raster/gtiff.html

Underlying challenge with using compression in GTiff format. My saving data with compression leads to this issue.

[nkruskamp]

50: C:/Users/nfkruska/AppData/Local/Temp/RtmpIpJMC7/spat_4rR77HZD0RcFisK.tif, band 48: An error occurred while writing a dirty block from FlushCache (GDAL error 1)

See here for an explanation of the issue.
https://gis.stackexchange.com/questions/368251/an-error-occurred-while-writing-a-dirty-block-from-gdalrasterbandirasterio

I now have my source data uncompressed with BIGTIFF set, but somewhere in our code or terra the files are being written without the correct configuration.

Admittedly, without compressing my weather coef file is now 40GB, so we may be reaching the limits of what we can do in single files?