KeyError: 'size_total' about ansible-cmdb

[fboender]

Hmm, strange. There seem to be some mount points with no available size_total. Could you try to find out the mount point that's causing the problem? If you save the following to a file called find.py:

import json
import os
import sys
for f in os.listdir(sys.argv[1]):
    j = json.load(open(os.path.join(sys.argv[1], f), 'r'));
    for m in j['ansible_facts']['ansible_mounts']:
        if not 'size_total' in m:
            print f
            print m

and run it like:

$ python find.py /path/to/ansible/out

It should print the host and mount points causing the problems. If you could copy-paste the results, I can probably figure out what's wrong.

Thanks for your help Cory!

[coryrwagner]

I ran find.py against the fact files and it kept failing, but it didn't tell me which file was the problem, so I moved "print f" after the first for loop. The find script then threw errors on files that had SSH or json error messages, rather than inventory info. After I removed the files w/o inventory info, the find.py script ran through all of the files w/o any errors, but almost every mount was listed. Only 105 devices have size_total vs 13,847 that don't.

It looks like Linux hosts are OK, but Solaris hosts are not. I will send you some Solaris inventory files to your gmail account.

Here's a sample of some of the mounts from a Solaris host:

        "ansible_mounts": [
            {
                "device": "rpool/ROOT/20150113_host1",
                "fstype": "zfs",
                "mount": "/",
                "options": "dev=4010002",
                "time": "0"
            },
            {
                "device": "/devices",
                "fstype": "devfs",
                "mount": "/devices",
                "options": "dev=62c0000",
                "time": "1437988345"
            },
            {
                "device": "swap",
                "fstype": "tmpfs",
                "mount": "/var/run",
                "options": "xattr,dev=63c0003",
                "time": "1437988368"
            },
            {
                "device": "rpool/export",
                "fstype": "zfs",
                "mount": "/export",
                "options": "rw,devices,setuid,nonbmand,exec,rstchown,xattr,atime,dev=4010003",
                "time": "1437988370"
            },
            {
                "device": "rpool/export/home",
                "fstype": "zfs",
                "mount": "/export/home",
                "options": "rw,devices,setuid,nonbmand,exec,rstchown,xattr,atime,dev=4010004",
                "time": "1437988370"
            },
            {
                "device": "rpool",
                "fstype": "zfs",
                "mount": "/rpool",
                "options": "rw,devices,setuid,nonbmand,exec,rstchown,xattr,atime,dev=4010005",
                "time": "1437988370"
            },
            {
                "device": "rpool/usr_log",
                "fstype": "zfs",
                "mount": "/usr/log",
                "options": "rw,devices,setuid,nonbmand,exec,rstchown,xattr,atime,dev=4010006",
                "time": "1437988370"
            },
            {
                "device": "host1:vold(pid444)",
                "fstype": "nfs",
                "mount": "/vol",
                "options": "ignore,noquota,dev=6600002",
                "time": "1437988374"
            },
            {
                "device": "strg-01:/cnos",
                "fstype": "nfs",
                "mount": "/cnos",
                "options": "bg,intr,xattr,dev=6600001",
                "time": "1437988374"
            },
            {
                "device": "auto_home",
                "fstype": "autofs",
                "mount": "/home",
                "options": "indirect,ignore,nobrowse,dev=6640001",
                "time": "1437988379"
            },
        ],

[muffl0n]

It looks like the facts-module just does not get the sizes of the mounts in SunOSHardware.get_mount_facts():
https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts.py#L1186

Compare with the method LinuxHardware.get_mount_facts(): https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts.py#L938

[fboender]

Nice find Sven!

I received some samples from Cory, and it looks like the disk size information is not available anywhere in the generated facts files. I'd consider this a bug (shortcoming?) in Ansible.

For now I'm going to write a fix so that ansible-cmdb doens't barf on missing sizes.

[muffl0n]

The code was added in PR ansible/ansible#10292. It does not look like there were any real problems that prevented those measurements to be included.
Normally I would get a PR ready to fix that. But we are finally ditching our last Solaris machines soon, so I won't invest time in that. :)

[fboender]

I've commited a fix for the KeyError. I'm going to close this issue, even though I don't feel it's been fixed properly. But the problem is in Ansible, not ansible-cmdb. I'm going to look into fixing the problem in Ansible itself and making a PR. I'll have to set up a solaris machine first though.

Thank you all for your help and information!

[muffl0n]

At least there seems to be the same possibility to get the fs sizes like on Linux:

> python
Python 2.6.4 (r264:75706, Jun 26 2012, 21:27:36) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.statvfs('/var/log')
posix.statvfs_result(f_bsize=131072, f_frsize=512, f_blocks=708532116L, f_bfree=703980079L, f_bavail=703980079L, f_files=703980116L, f_ffree=703980079L, f_favail=703980079L, f_flag=4, f_namemax=255)
>>> ^D

> df /var/log
/var/log           (/var/log          ):703980079 blocks 703980079 files

[h0lzi]

@fboender did you have the time to open a PR to sort this. I pinged @dago (colleague of mine) to have a look and improve his code when he has time so we get some size back :)

[dago]

In Solaris mnttab(4) data does not include filesystem sizes:
special mount_point fstype options time
As in the Linux implementation the fact gathering would need to go through all the filesystems to get the usage data.

[fboender]

@h0lzi, @dago I haven't had time to open a PR to Ansible for this yet. I've got some time right now, so I'll look into it. Depending on how difficult it's going to be it might have to wait until after christmas, when I've got some vacation time coming.

[fboender]

Looks like solaris support is now completely broken in the last (master) version of ansible:

./ansible -m ping -i ~/Projects/ansible-solaris/hosts all
sol.local | UNREACHABLE! => {
    "changed": false, 
    "msg": "ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh", 
    "unreachable": true
}

With an older version, the same hosts file still works though:

ansible -m ping -i ~/Projects/ansible-solaris/hosts all
sol.local | success >> {
    "changed": false,
    "ping": "pong"
}

I'm gonna see if I can fix this first.

[fboender]

ansible/ansible#13410