Microsoft.OSTCExtensions.LinuxDiagnostic extension failing: libicuuc.so.52: cannot open shared object file about walinuxagent

[XVII]

Upon further observation, it seems it's looking for version 52.
I tried installing v52 (vivid), and uninstalling v55.

It still tries to load v55...

mdsd stopped:MDSD crash:/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.0.92/mdsd/mdsd: error while loading shared libraries: libicuuc.so.55: cannot open shared object file: No such file or directory

[karataliu]

@ShadowXVII

This is an issue for Diagnostic Extension v2.0.92, we've got it fixed it in v2.2.

We're working on the update recently and basically it should work for a new created VM (ub15.10) now.

[XVII]

Cool, will it be available for existing VMs? I can uninstall and reinstall.

What's the timeframe?

[karataliu]

For existing VMs, it will still goes to the 2.0 version when turning on Diagnostics. We're actively working on this since it could involve upgrade for existing extensions. Hopefully that would be available earlier next month.

[kasnol]

Hi Just would like to know any update to this issue, as I have created the VM few months ago and still running without any VM metrics/statistics....

[kiratp]

I deployed a new Ubuntu 15.10 VM a couple of days ago and still got agent version 2.1.2 (being < 2.2, still no metrics.

[karataliu]

I just tried enabling diagnostic on existing VM, it would now turn to latest version 2.3.3, which includes the fix.

Would you please try that (VM portal->Settings->Diagnostics-> Turn On, you need to Turn off first if it is already On)?

[XVII]

I now get

STATUS MESSAGEmdsd stopped:MDSD crash:
icrosoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin/mdsd) 
/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin/mdsd: /opt/omi/lib/libcrypto.so.1.0.0: version `OPENSSL_1.0.0' not found (required by /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin/libcpprest.so.2.7) 
/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin/mdsd: /opt/omi/lib/libssl.so.1.0.0: version `OPENSSL_1.0.0' not found (required by /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin/libcpprest.so.2.7) 
/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin/mdsd: /opt/omi/lib/libcrypto.so.1.0.0: version `OPENSSL_1.0.0' not found (required by /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin/libazurestorage.so.2.2) 
/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin/mdsd: /opt/omi/lib/libssl.so.1.0.0: version `OPENSSL_1.0.0' not found (required by /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin/libazurestorage.so.2.2)

The Ubuntu openssl packages appear to be up to date. I haven't dug any deeper into this yet.

Regards,
Jake

[karataliu]

Can you help check the output of following 2 commands?

#check openssl version
dpkg -l openssl

#check library linkage
LD_LIBRARY_PATH=/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin ldd /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin/libazurestorage.so.2.2

Thanks

[XVII]

#check openssl version
dpkg -l openssl

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                          Version                     Architecture                Description
+++-=============================================-===========================-===========================-===============================================================================================
ii  openssl                                       1.0.2f-2+deb.sury.org~wily+ amd64                       Secure Sockets Layer toolkit - cryptographic utility

#check library linkage
sudo echo $LD_LIBRARY_PATH. Should this be set?

There isn't any environment variables set for LD_LIBRARY_PATH.

#ldd
sudo ldd /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin/libazurestorage.so.2.2

        linux-vdso.so.1 =>  (0x00007ffc2f1d7000)
        libcpprest.so.2.7 => not found
        libboost_log.so.1.55.0 => not found
        libboost_system.so.1.55.0 => not found
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0d5d2d4000)
        libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f0d5d06b000)
        libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f0d5cc28000)
        libxml++-2.6.so.2 => not found
        libglibmm-2.4.so.1 => /usr/lib/x86_64-linux-gnu/libglibmm-2.4.so.1 (0x00007f0d5c9b2000)
        libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f0d5c7ac000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f0d5c42a000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0d5c122000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0d5bf0a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0d5bb40000)
        /lib64/ld-linux-x86-64.so.2 (0x00005575de543000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0d5b93c000)
        libsigc-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libsigc-2.0.so.0 (0x00007f0d5b735000)
        libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007f0d5b4e2000)
        libgmodule-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007f0d5b2de000)
        libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f0d5afce000)
        libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f0d5adc6000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f0d5ab58000)

[karataliu]

Yes, that's needed, placing LD_LIBRARY_PATH and ldd in same line would cause that to be set for the ldd command:

'LD_LIBRARY_PATH=/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin ldd /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.3.3/bin/libazurestorage.so.2.2'

[iandow]

I reproduced this problem with WALinuxAgent-2.0.14 provisioned in ARM.

[karataliu]

@ShadowXVII

Looks like you're using the libssl package from a custom PPA source(deb.sury.org~wily, ondrej/php), which doesn't provider ABI for OPENSSL_1.0.0:

1. In that case:
   $ ldd openssl
   ldd /usr/bin/openssl
   linux-vdso.so.1 => (0x00007fffb51cd000)
   libssl.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2 (0x00007fed3e3a7000)
   libcrypto.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2 (0x00007fed3df5d000)
   libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fed3db92000)
   libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fed3d98e000)
   /lib64/ld-linux-x86-64.so.2 (0x000055c8be92e000)

$ strings /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2 |grep OPENSSL_1
OPENSSL_1.0.2d

1. With official repo:
   $ ldd /usr/bin/openssl
   linux-vdso.so.1 => (0x00007ffdbcb43000)
   libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fae6d898000)
   libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fae6d456000)
   libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fae6d08b000)
   libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fae6ce87000)
   /lib64/ld-linux-x86-64.so.2 (0x0000560752623000)

$ strings /lib/x86_64-linux-gnu/libssl.so.1.0.0 |grep OPENSSL_1
OPENSSL_1.0.0
OPENSSL_1.0.1
OPENSSL_1.0.1d
OPENSSL_1.0.2

For now, one dependency of the diagnostic extension, the omi package requires the OPENSSL_1.0.0, that's could be why it does not work.

[iandow]

We are also seeing mdsd failing due to missing library:

cat /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.2.0/mdsd.log 
/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.2.0/bin/mdsd: error while loading shared libraries: libglibmm-2.4.so.1: cannot open shared object file: No such file or directory

There are two missing libraries:

LD_LIBRARY_PATH=/var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.2.0/bin ldd /var/lib/waagent/Microsoft.OSTCExtensions.LinuxDiagnostic-2.2.0/bin/libazurestorage.so.1.0 | grep -i not | sort -u
    libglibmm-2.4.so.1 => not found
    libsigc-2.0.so.0 => not found

The azure linux agent defines a custom rsyslog action queue for something called, "omazuremds". Its configuration is in /etc/rsyslog.d/omazurelinuxmds.conf.

However, that omazuremds action queue fails to connect because mdsd failed to start. It attempts to connect 10 times, then stops.

[root@scai10demo nimda]# grep -i omazuremds /var/log/messages
Feb 11 13:58:44 scai10demo rsyslogd-3000: omazuremds error at connect(). errno=Permission denied
Feb 11 13:58:45 scai10demo rsyslogd-3000: omazuremds error at connect(). errno=Permission denied
Feb 11 13:58:46 scai10demo rsyslogd-3000: omazuremds error at connect(). errno=Permission denied
Feb 11 13:58:47 scai10demo rsyslogd-3000: omazuremds error at connect(). errno=Permission denied
Feb 11 13:58:48 scai10demo rsyslogd-3000: omazuremds error at connect(). errno=Permission denied
Feb 11 13:58:50 scai10demo rsyslogd-3000: omazuremds error at connect(). errno=Permission denied
Feb 11 13:58:51 scai10demo rsyslogd-3000: omazuremds error at connect(). errno=Permission denied
Feb 11 13:58:52 scai10demo rsyslogd-3000: omazuremds error at connect(). errno=Permission denied
Feb 11 13:58:53 scai10demo rsyslogd-3000: omazuremds error at connect(). errno=Permission denied
Feb 11 13:58:54 scai10demo rsyslogd-3000: omazuremds error at connect(). errno=Permission denied

We see that the messages in /var/log/messages keep getting older and older. After a few days of the system running, anything that wants to write a message to /var/log/messages is delayed by as much as 60 seconds. After several more days, that delay is on the order of 5 minutes.

In omazurelinuxmds.conf, we see the following setting:

actionresumeretrcycount = -1 

which I think will cause infinite retries on insert failure and also avoids dropping any of the mds messages. I think this could should not be -1.

I haven't tested the effects of changing actionresumeretrcycount, but I do know we can workaround this problem by removing the omazuremds rsyslog queue altogether, like this:

rm /etc/rsyslog.d/omazurelinuxmds_fileom.conf
rm /etc/rsyslog.d/omazurelinuxmds.conf

This won't fix the azure resource graphs, but it helps prevent the mdsd failure from effecting everything else that uses /var/log/messages (such as, our bread-and-butter web service).

I'm eagerly looking forward to a fix for this.

[XVII]

@iandow, I had that originally (see post op). Try apt-get installing 'libglibmm', then uninstalled diagnostics from portal and re-activate logging and got past that.

[jasonzio]

Current version of the Linux Diagnostics extension is 2.3.4. Compared to 2.2, this update resolves at least one issue that was preventing clean install and startup of omazuremds and mdsd. Unfortunately the dependency on glibmm remains, as the Azure Storage C++ SDK (used by mdsd) requires it.

[XVII]

@karataliu, is there anyway to co-install both versions to satisfy the older dependency from OMI? As far as I understand, 1.0.0 has been superseded by security updates?

Is there an ETA for when the OMI package will be updated to use a supported version?

[karataliu]

@iandow Looks v2.3.4 should have fixed the dependency issue you mentioned, would you please try that out?

[karataliu]

@hosungsmsft Can you help with the question regarding openssl and omi versioning issue?

[hosungsmsft]

@ShadowXVII @karataliu As far as I understand, the OMI installer just creates links to the system's libcrypto.so.* and libssl.so.*, and mdsd is just linked with the OpenSSL libs through the links. It could be that the OMI binaries may require specifically OpenSSL 1.0.0, but it looks like it's the CPPREST libs that require 1.0.0. Since we don't own either OMI nor CPPREST, we can only forward the request to the relevant teams. And I'd suggest the same: Removing the custom OpenSSL package with the official one (that support all ABI versions including 1.0.0) would solve the issue. I see that on our mdsd build machines, OpenSSL 1.0.1f is used and it works fine with all dependencies, so I'd suggest the same again.

[XVII]

@hosungsmsft , These are the openssl versions available to my VM;

sudo apt-cache policy openssl

 *** 1.0.2f-2+deb.sury.org~wily+1 0
        500 http://ppa.launchpad.net/ondrej/php/ubuntu/ wily/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.2d-0ubuntu1.4 0
        500 http://azure.archive.ubuntu.com/ubuntu/ wily-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ wily-security/main amd64 Packages
     1.0.2d-0ubuntu1 0
        500 http://azure.archive.ubuntu.com/ubuntu/ wily/main amd64 Packages

You mentioned 1.0.1f, but that's not available so I've downgraded to 1.0.2d-0ubuntu1.4 (the closest provided by the standard repos).

It still errors with the same error.

[hosungsmsft]

@ShadowXVII , I spun up a fresh Ubuntu 15.10 VM from Azure portal, and below is what I get. It's also OpenSSL 1.0.2, but 1.0.0 ABI is still available as seen below. Even though mine is 1.0.2d-0ubuntu1.3.0, I'm pretty sure it should be the same with 1.0.2d-0ubuntu1.4.0. Could you please show the results of the same commands as below, showing your installed version is 1.0.2d-0ubuntu1.x and OPENSSL_1.0.0 ABI is available?

azureuser@ub1510:~$ apt-cache policy openssl
openssl:
  Installed: 1.0.2d-0ubuntu1.3
  Candidate: 1.0.2d-0ubuntu1.3
  Version table:
 *** 1.0.2d-0ubuntu1.3 0
        500 http://azure.archive.ubuntu.com/ubuntu/ wily-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ wily-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.2d-0ubuntu1 0
        500 http://azure.archive.ubuntu.com/ubuntu/ wily/main amd64 Packages
azureuser@ub1510:~$ strings /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 | grep OPENSSL_1
OPENSSL_1.0.0
OPENSSL_1.0.1
OPENSSL_1.0.1d
OPENSSL_1.0.2
citest@ladtest-ub1510:~$ 

[XVII]

@hosungsmsft ,

user@vm1:~$ sudo apt-cache policy openssl
openssl:
  Installed: 1.0.2d-0ubuntu1.4
  Candidate: 1.0.2f-2+deb.sury.org~wily+1
  Version table:
     1.0.2f-2+deb.sury.org~wily+1 0
        500 http://ppa.launchpad.net/ondrej/php/ubuntu/ wily/main amd64 Packages
 *** 1.0.2d-0ubuntu1.4 0
        500 http://azure.archive.ubuntu.com/ubuntu/ wily-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu/ wily-security/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.2d-0ubuntu1 0
        500 http://azure.archive.ubuntu.com/ubuntu/ wily/main amd64 Packages
user@vm1:~$ strings /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 | grep OPENSSL_1
OPENSSL_1.0.0
OPENSSL_1.0.1
OPENSSL_1.0.1d
OPENSSL_1.0.2
user@vm1:~$

[hosungsmsft]

@ShadowXVII Thanks much for the info. This is getting pretty strange. The only thing I think we can look is if /opt/omi/lib/libcrypto.so.1.0.0 and /opt/omi/lib/libssl.so.1.0.0 are linking to the correct/updated /lib/x86_64-linux-gnu/lib{crypto,ssl}.so.* files. That's because the mdsd binary is linked with /opt/omi/lib as the path for those libraries, and OMI package just creates links to the actual lib{crypto,ssl}.so.* files in the system directory. Since the OpenSSL package was reinstalled, I can only guess that those links are now stale, causing this issue. I hope this makes sense to you, and ask you to check if the symlinks in /opt/omi/lib are valid.

[XVII]

@hosungsmsft
Symlinks in /opt/omi/lib/;

lrwxrwxrwx 1 root root      44 Mar  2 06:26 libcrypto.so.1.0.0 -> /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.2
-rwxr-xr-x 1 root root  899702 Feb  6 12:05 libmicxx.so
-rwxr-xr-x 1 root root 1520680 Feb  6 12:05 libmi.so
lrwxrwxrwx 1 root root      52 Jan 18 12:11 libmysqlProvider.so -> /opt/microsoft/mysql-cimprov/lib/libmysqlProvider.so
-rwxr-xr-x 1 root root 1110135 Feb  6 12:05 libomiclient.so
-rwxr-xr-x 1 root root  266385 Feb  6 12:05 libomiidentify.so
lrwxrwxrwx 1 root root      50 Feb  6 12:07 libSCXCoreProviderModule.so -> /opt/microsoft/scx/lib/libSCXCoreProviderModule.so
lrwxrwxrwx 1 root root      41 Mar  2 06:26 libssl.so.1.0.0 -> /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2

The link targets do exist;

/usr/lib/x86_64-linux-gnu$ ls -la | grep 'libssl\|libcrypto'
-rw-r--r--  1 root root  2386400 Jan 29 09:47 libcrypto.so.1.0.2
-rw-r--r--  1 root root   324120 Feb  5 05:22 libssl3.so
-rw-r--r--  1 root root   425592 Jan 29 09:47 libssl.so.1.0.2

[hosungsmsft]

@ShadowXVII Thanks again for your confirmation and patience. I'm wondering if we can discuss this in some other way? Please feel free to email me at [email protected].

[XVII]

Thanks for your help @hosungsmsft .

Turned out to be the PHP ppa by ondrej (https://launchpad.net/~ondrej/+archive/ubuntu/php) which was causing some funniness with OMI. An issue should be raised with the relevant Github repo if that's something they need to fix (I'm not versed well enough to advise them) -- https://github.com/oerdnj/deb.sury.org/issues

As an unsupported, temporary workaround, I've replaced the links for the libs in /opt/omi/lib/ with the version from libssl1.0.0 for the wily distro (http://packages.ubuntu.com/wily/amd64/libssl1.0.0/download). You can grab the libraries out by extracting the .deb.

[anatolinicolae]

If you've already installed ondrej's version do this

apt-get install openssl=1.0.1f-1ubuntu2.18

Then or if you're on the right version, keep that one with this

apt-mark hold openssl