libcurl and libxml2 are missing in amazonlinux 2 about docker-lambda

[vincentsarago]

🤔

ghcr.io/lambgeo/lambda-gdal:3.5-python3.9 image

$ docker run --entrypoint bash --platform=linux/amd64 -it --rm ghcr.io/lambgeo/lambda-gdal:3.5-python3.9
$ ldd /opt/bin/gdalinfo
        libgdal.so.31 => /opt/bin/../lib/libgdal.so.31 (0x0000004001a2b000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000004003236000)
        libm.so.6 => /lib64/libm.so.6 (0x00000040035b8000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000040038f8000)
        libc.so.6 => /lib64/libc.so.6 (0x0000004003b0e000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000004003eb9000)
        libz.so.1 => /lib64/libz.so.1 (0x00000040040bf000)
        libcurl.so.4 => /lib64/libcurl.so.4 (0x00000040042d4000)
        libxml2.so.2 => /lib64/libxml2.so.2 (0x000000400456b000)
        libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00000040048c4000)
        libssl.so.10 => /lib64/libssl.so.10 (0x0000004004d1a000)
        libzstd.so.1 => /opt/bin/../lib/libzstd.so.1 (0x0000004004f89000)
        liblzma.so.5 => /var/lang/lib/liblzma.so.5 (0x0000004005213000)
        libdeflate.so.0 => /opt/bin/../lib/libdeflate.so.0 (0x0000004005438000)
        libblosc.so.1 => /opt/bin/../lib/libblosc.so.1 (0x0000004005648000)
        libproj.so.22 => /opt/bin/../lib/libproj.so.22 (0x000000400586e000)
        libexpat.so.1 => /opt/bin/../lib/libexpat.so.1 (0x0000004005d61000)
        libjpeg.so.62 => /opt/bin/../lib/libjpeg.so.62 (0x0000004005f91000)
        libtiff.so.5 => /opt/bin/../lib/libtiff.so.5 (0x000000400623a000)
        libgeotiff.so.5 => /opt/bin/../lib/libgeotiff.so.5 (0x00000040064ba000)
        libpng16.so.16 => /opt/bin/../lib/libpng16.so.16 (0x00000040066ee000)
        libLercLib.so => /opt/bin/../lib/libLercLib.so (0x000000400691f000)
        libnetcdf.so.18 => /opt/bin/../lib/libnetcdf.so.18 (0x0000004006b83000)
        libhdf5.so.200 => /opt/bin/../lib/libhdf5.so.200 (0x0000004006eb6000)
        libsz.so.2 => /opt/bin/../lib/libsz.so.2 (0x000000400758a000)
        libwebp.so.7 => /opt/bin/../lib/libwebp.so.7 (0x000000400779e000)
        libsqlite3.so.0 => /opt/bin/../lib/libsqlite3.so.0 (0x0000004007a08000)
        libpq.so.5 => /opt/bin/../lib/libpq.so.5 (0x0000004007d19000)
        libopenjp2.so.7 => /opt/bin/../lib/libopenjp2.so.7 (0x0000004007f62000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00000040081b6000)
        libgeos_c.so.1 => /opt/bin/../lib/libgeos_c.so.1 (0x000000400841c000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000004008659000)
        libdf.so.0 => /opt/bin/../lib/libdf.so.0 (0x0000004008877000)
        libmfhdf.so.0 => /opt/bin/../lib/libmfhdf.so.0 (0x0000004008b28000)
        /lib64/ld-linux-x86-64.so.2 (0x0000004000000000)
        libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x0000004008d52000)
        libidn2.so.0 => /lib64/libidn2.so.0 (0x0000004008f7b000)
        libssh2.so.1 => /lib64/libssh2.so.1 (0x00000040091ca000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00000040093f2000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x000000400963e000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x0000004009922000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000004009b53000)
        libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x0000004009d59000)
        liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x0000004009fab000)
        libhdf5_hl.so.200 => /opt/lib/libhdf5_hl.so.200 (0x000000400a1ba000)
        libgeos.so.3.10.2 => /opt/bin/../lib/libgeos.so.3.10.2 (0x000000400a3dc000)
        libunistring.so.0 => /lib64/libunistring.so.0 (0x000000400a841000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x000000400ab59000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x000000400ad68000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x000000400af6c000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x000000400b182000)
        libssl3.so => /lib64/libssl3.so (0x000000400b39e000)
        libsmime3.so => /lib64/libsmime3.so (0x000000400b5ff000)
        libnss3.so => /lib64/libnss3.so (0x000000400b825000)
        libnssutil3.so => /lib64/libnssutil3.so (0x000000400bb52000)
        libplds4.so => /lib64/libplds4.so (0x000000400bd81000)
        libplc4.so => /lib64/libplc4.so (0x000000400bf85000)
        libnspr4.so => /lib64/libnspr4.so (0x000000400c18a000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x000000400c3c8000)
        libcrypt.so.1 => /var/lang/lib/libcrypt.so.1 (0x000000400c5ef000)
        librt.so.1 => /lib64/librt.so.1 (0x000000400c826000)

$ ls /lib64/ | grep "curl"
libcurl.so.4
libcurl.so.4.7.0
$ ls /lib64/ | grep "xml2"
libxml2.so.2
libxml2.so.2.9.1

public.ecr.aws/lambda/python:3.9

docker run --entrypoint bash --platform=linux/amd64 -it --rm public.ecr.aws/lambda/python:3.9
$  ls /lib64 | grep "curl"
libcurl.so.4
libcurl.so.4.7.0
$ ls /lib64 | grep "xml2"
libxml2.so.2
libxml2.so.2.9.1

@ThiagoDiasV ☝️ seems to tell the libraries a present 🤷

[vincentsarago]

What do you mean by public.ecr.aws/lambda/python:3.9 doesn't match the AWS lambda env?

as shown in previous comment we see /lib64/libcurl.so.4 in public.ecr.aws/lambda/python:3.9 but when doing ls /lib64 | grep "lib curl" in AWS Lambda it returns empty

[vincentsarago]

🙏

But I think you forgot to update the layer versions in layer.json

Yes I went to bed before the layer were published 😄, I'll do it now

[vincentsarago]

fixed in 0978231

[ThiagoDiasV]

After the last updates and the new Dockerfile to use gdal3.5 we are missing libcurl and libxml2 again.

Here's the output of ldd /opt/bin/ogr2ogr in a amazon linux 2 instance running python 3.9 version:

\tlinux-vdso.so.1 (0x00007fff173fd000)
\tlibgdal.so.31 => /opt/bin/../lib/libgdal.so.31 (0x00007fb82bf5d000)
\tlibstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fb82bbdb000)
\tlibm.so.6 => /lib64/libm.so.6 (0x00007fb82b89b000)
\tlibgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb82b685000)
\tlibc.so.6 => /lib64/libc.so.6 (0x00007fb82b2da000)
\tlibdl.so.2 => /lib64/libdl.so.2 (0x00007fb82b0d6000)
\tlibz.so.1 => /lib64/libz.so.1 (0x00007fb82aec1000)
\tlibcurl.so.4 => not found
\tlibxml2.so.2 => not found
\tlibcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fb82aa6b000)
\tlibssl.so.10 => /lib64/libssl.so.10 (0x00007fb82a7fc000)
\tlibzstd.so.1 => /opt/bin/../lib/libzstd.so.1 (0x00007fb82a574000)
\tliblzma.so.5 => /var/lang/lib/liblzma.so.5 (0x00007fb82a34e000)
\tlibdeflate.so.0 => /opt/bin/../lib/libdeflate.so.0 (0x00007fb82a13e000)
\tlibblosc.so.1 => /opt/bin/../lib/libblosc.so.1 (0x00007fb829f18000)
\tlibproj.so.22 => /opt/bin/../lib/libproj.so.22 (0x00007fb829a25000)
\tlibexpat.so.1 => /opt/bin/../lib/libexpat.so.1 (0x00007fb8297f5000)
\tlibjpeg.so.62 => /opt/bin/../lib/libjpeg.so.62 (0x00007fb82954e000)
\tlibtiff.so.5 => /opt/bin/../lib/libtiff.so.5 (0x00007fb8292ce000)
\tlibgeotiff.so.5 => /opt/bin/../lib/libgeotiff.so.5 (0x00007fb82909a000)
\tlibpng16.so.16 => /opt/bin/../lib/libpng16.so.16 (0x00007fb828e69000)
\tlibLercLib.so => /opt/bin/../lib/libLercLib.so (0x00007fb828c05000)
\tlibnetcdf.so.18 => /opt/bin/../lib/libnetcdf.so.18 (0x00007fb8288d2000)
\tlibhdf5.so.200 => /opt/bin/../lib/libhdf5.so.200 (0x00007fb828200000)
\tlibsz.so.2 => /opt/bin/../lib/libsz.so.2 (0x00007fb827fec000)
\tlibwebp.so.7 => /opt/bin/../lib/libwebp.so.7 (0x00007fb827d82000)
\tlibsqlite3.so.0 => /opt/bin/../lib/libsqlite3.so.0 (0x00007fb827a71000)
\tlibpq.so.5 => /opt/bin/../lib/libpq.so.5 (0x00007fb827828000)
\tlibopenjp2.so.7 => /opt/bin/../lib/libopenjp2.so.7 (0x00007fb8275d4000)
\tlibpcre.so.1 => /lib64/libpcre.so.1 (0x00007fb827370000)
\tlibgeos_c.so.1 => /opt/bin/../lib/libgeos_c.so.1 (0x00007fb827133000)
\tlibpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb826f15000)
\tlibdf.so.0 => /opt/bin/../lib/libdf.so.0 (0x00007fb826c64000)
\tlibmfhdf.so.0 => /opt/bin/../lib/libmfhdf.so.0 (0x00007fb826a3a000)
\t/lib64/ld-linux-x86-64.so.2 (0x00007fb82d768000)
\tlibgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fb8267ee000)
\tlibkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fb82650a000)
\tlibcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fb826306000)
\tlibk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fb8260d5000)
\tlibcurl.so.4 => not found
\tlibcurl.so.4 => not found
\tlibhdf5_hl.so.200 => /opt/lib/libhdf5_hl.so.200 (0x00007fb825eb3000)
\tlibcurl.so.4 => not found
\tlibgeos.so.3.10.2 => /opt/bin/../lib/libgeos.so.3.10.2 (0x00007fb825a50000)
\tlibkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fb825841000)
\tlibkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fb82563d000)
\tlibresolv.so.2 => /lib64/libresolv.so.2 (0x00007fb825427000)
\tlibselinux.so.1 => /lib64/libselinux.so.1 (0x00007fb825200000)

[vincentsarago]

Oh no I though I check this 😬

[vincentsarago]

😬 public.ecr.aws/lambda/python:3.9 do not match the AWS Lambda environment 🤬🤬🤬🤬🤬

[ThiagoDiasV]

What do you mean by public.ecr.aws/lambda/python:3.9 doesn't match the AWS lambda env?

[ThiagoDiasV]

Thank you mister for the quick fix. I already tested and everything went fine.

But I think you forgot to update the layer versions in layer.json file. I deployed a new lambda successfully but using arn:aws:lambda:{region}:524387336408:layer:gdal35:3 instead of arn:aws:lambda:{region}:524387336408:layer:gdal35:2