cvescan fails due to lack of networking when confined about sec-cvescan

[jdstrand]

I'm not sure what is special about this VM; we're hitting a security policy, perhaps?

What is the output of $ sudo journalctl |grep audit at the time of the failure?

[markmorlino]

For me it works fine in xenial VMs but I am able to reproduce the problem in trusty and here is the output:

Feb 19 14:41:49 sec-trusty-amd64 kernel: audit: type=1400 audit(1582141309.662:32): apparmor="DENIED" operation="create" profile="snap.cvescan.cvescan" pid=1800 comm="curl" family="inet6" sock_type="dgram" protocol=0 requested_mask="create" denied_mask="create"
Feb 19 14:41:49 sec-trusty-amd64 kernel: audit: type=1400 audit(1582141309.662:33): apparmor="DENIED" operation="open" profile="snap.cvescan.cvescan" name="/etc/host.conf" pid=1801 comm="curl" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Feb 19 14:41:49 sec-trusty-amd64 kernel: audit: type=1400 audit(1582141309.662:34): apparmor="DENIED" operation="open" profile="snap.cvescan.cvescan" name="/run/resolvconf/resolv.conf" pid=1801 comm="curl" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Feb 19 14:41:49 sec-trusty-amd64 kernel: audit: type=1400 audit(1582141309.666:35): apparmor="DENIED" operation="open" profile="snap.cvescan.cvescan" name="/etc/hosts" pid=1801 comm="curl" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
Feb 19 14:41:49 sec-trusty-amd64 kernel: audit: type=1400 audit(1582141309.666:36): apparmor="DENIED" operation="create" profile="snap.cvescan.cvescan" pid=1801 comm="curl" family="inet" sock_type="dgram" protocol=0 requested_mask="create" denied_mask="create"

[jdstrand]

These are all included when plugging 'network', which is auto-connected. What is the output of:

$ snap connections cvescan

[markmorlino]

Interface Plug Slot Notes
home cvescan:home - -
network cvescan:network - -
system-files cvescan:apt-dpkg-db - -

[jdstrand]

@markmorlino - in your case the network interface is not connected. Please use sudo snap connect cvescan:network.

That said, what is the output of snap version.

[markmorlino]

$ sudo snap connect cvescan:network
error: cannot resolve connection, slot snap name is empty

$ sudo snap connect cvescan:home
error: cannot resolve connection, slot snap name is empty

$ sudo snap connect cvescan:apt-dpkg-db
error: cannot resolve connection, slot snap name is empty

$ snap version
snap    2.38~14.04
snapd   2.38~14.04
series  16
ubuntu  14.04
kernel  4.4.0-174-generic

[jdstrand]

@markmorlino - your snapd is too old. What happens if you do:

$ sudo snap refresh
$ snap version

[markmorlino]

$ sudo snap refresh
All snaps up to date.

snap version output is the same

[jdstrand]

@markmorlino - that seems odd. This seems like something for the snapd team. Perhaps @mvo5 or @anonymouse64 can comment?

[jdstrand]

@markmorlino - what is the output of cat /proc/version_signature?

[markmorlino]

$ cat /proc/version_signature 
Ubuntu 4.4.0-174.204~14.04.1-generic 4.4.211

[jdstrand]

Oh, maybe you don't have the core snap installed. What is the output of snap list?

[markmorlino]

$ snap list
Name     Version   Rev   Tracking  Publisher    Notes
core18   20200124  1668  stable    canonical✓   base
cvescan  1.0.1     6     stable    markmorlino  -

[jdstrand]

Right, so, locally, I installed snapd, then rebooted into the new kernel and snap list showed no snaps. If I 'snap install cvescan', that pulls down the core18 snap since cvescan uses 'base: core18', but it doesn't pull down the 'core' snap. The snapd in trusty (2.38~14.04) only knows about re-execing into the 'core' snap, not 'core18'. So I did:

$ sudo snap install core
$ sudo snap install cvescan
$ snap connections cvescan
Interface     Plug                 Slot           Notes
home          cvescan:home         :home          -
network       cvescan:network      :network       -
system-files  cvescan:apt-dpkg-db  :system-files  -

and everything worked. I think all you need to do is:

$ sudo snap remove cvescan
$ sudo snap install core
$ sudo snap install cvescan

and you'll find that the 3 interfaces auto-connect (from the state you are currently in, you probably have to manually connect them if you just install core, which is why I suggested first removing cvescan).

[markmorlino]

💯 👍 that seems to make it work @jdstrand thanks!

Do you think @kiko-canonical was having the same issue under xenial (the behavior seemed the same)?

Do you still think this is something for the snapd team? It seems like core should get installed automatically if it needs to be there.

[jdstrand]

Could be if cvescan was the only snap that was downloaded. On a fresh install:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.6 LTS
Release:	16.04
Codename:	xenial
$ snap list
No snaps are installed yet. Try 'snap install hello-world'.

I know @mvo5 and the snapd team are aware of the issue, and I thought that they might've fixed it, but it could be that the trusty snapd deb needs an update to know to pull down core. @mvo5 - the issue in a nutshell is that trusty and xenial don't rexec since there is no core snap installed, and snap install cvescan only pulls in core18, but this snap needs things from newer snapds to work. Is it possible to adjust trusty/xenial to pull down core in these situations?

[jdstrand]

@markmorlino - a totally not great workaround would be to use 'base: core' in cvescan.

[anonymouse64]

@jdstrand yes I agree I think we need a new snapd deb in xenial and trusty because AFAIK we fixed this bug in newer snapd versions, but since the deb package is too old it doesn't have the fix.

[jdstrand]

Actually, xenial's deb is smart enough to pull down the 'snapd' snap and reexec into it:

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.6 LTS
Release:	16.04
Codename:	xenial
$ snap list
No snaps are installed yet. Try 'snap install hello-world'.
$ sudo snap install cvescan
$ sudo snap list
Name     Version   Rev   Tracking  Publisher    Notes
core18   20200124  1668  stable    canonical✓   base
cvescan  1.0.1     6     stable    markmorlino  -
snapd    2.43.3    6434  stable    canonical✓   snapd
$ snap version
snap    2.43.3
snapd   2.43.3
series  16
ubuntu  16.04
kernel  4.4.0-173-generic
$ snap connections cvescan
Interface     Plug                 Slot           Notes
home          cvescan:home         :home          -
network       cvescan:network      :network       -
system-files  cvescan:apt-dpkg-db  :system-files  -

That said, my vm has xenial-updates. @kiko-canonical - what is the output of apt-cache policy snapd? The one that is in xenial-security is 2.37.4ubuntu0.1 (ie, older than trusty). If that machine only has xenial-security and not xenial-updates, we would need to do a no change rebuild and push through xenial-security.

[jdstrand]

If that machine only has xenial-security and not xenial-updates, we would need to do a no change rebuild and push through xenial-security.

This is also theoretically true for trusty. I say theoretically because I'm not sure that newer snapd's will build on trusty at this point. We would have to weigh the cost of putting a whole new snapd in ESM (and supporting toolchain) against cherrypicking the new behavior.

[kiko-canonical]

Here's what I get: Feb 20 10:19:30 secured-ghost-on-ubuntu-16-04-lts-1-vm audit[24854]: AVC apparmor="DENIED" operation="create" profile="snap.cvescan.cvescan" pid=24854 comm="curl" family="inet6" sock_type="dgram" protocol=0 requested_mask="create" denied_mask="create" Feb 20 10:19:30 secured-ghost-on-ubuntu-16-04-lts-1-vm audit[24854]: SYSCALL arch=c000003e syscall=41 success=no exit=-13 a0=a a1=2 a2=0 a3=7f8c3db17d10 items=0 ppid=24833 pid=24854 auid=1001 uid=1001 gid=1002 euid=1001 suid=1001 fsuid=1001 egid=1002 sgid=1002 fsgid=1002 tty=pts0 ses=2 comm="curl" exe="/snap/cvescan/6/usr/bin/curl" key=(null) Feb 20 10:19:30 secured-ghost-on-ubuntu-16-04-lts-1-vm audit[24854]: AVC apparmor="DENIED" operation="open" profile="snap.cvescan.cvescan" name="/etc/host.conf" pid=24854 comm="curl" requested_mask="r" denied_mask="r" fsuid=1001 ouid=0 Feb 20 10:19:30 secured-ghost-on-ubuntu-16-04-lts-1-vm audit[24854]: SYSCALL arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=7f8c3db48e54 a2=80000 a3=0 items=1 ppid=24833 pid=24854 auid=1001 uid=1001 gid=1002 euid=1001 suid=1001 fsuid=1001 egid=1002 sgid=1002 fsgid=1002 tty=pts0 ses=2 comm="curl" exe="/snap/cvescan/6/usr/bin/curl" key=(null) Feb 20 10:19:30 secured-ghost-on-ubuntu-16-04-lts-1-vm audit: CWD cwd="/home/kiko/snap/cvescan/common" Feb 20 10:19:30 secured-ghost-on-ubuntu-16-04-lts-1-vm audit[24854]: AVC apparmor="DENIED" operation="open" profile="snap.cvescan.cvescan" name="/run/resolvconf/resolv.conf" pid=24854 comm="curl" requested_mask="r" denied_mask="r" fsuid=1001 ouid=0 Feb 20 10:19:30 secured-ghost-on-ubuntu-16-04-lts-1-vm audit[24854]: SYSCALL arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=7f8c3db48e09 a2=80000 a3=0 items=1 ppid=24833 pid=24854 auid=1001 uid=1001 gid=1002 euid=1001 suid=1001 fsuid=1001 egid=1002 sgid=1002 fsgid=1002 tty=pts0 ses=2 comm="curl" exe="/snap/cvescan/6/usr/bin/curl" key=(null) Feb 20 10:19:30 secured-ghost-on-ubuntu-16-04-lts-1-vm audit: CWD cwd="/home/kiko/snap/cvescan/common" Feb 20 10:19:30 secured-ghost-on-ubuntu-16-04-lts-1-vm audit[24854]: AVC apparmor="DENIED" operation="open" profile="snap.cvescan.cvescan" name="/etc/hosts" pid=24854 comm="curl" requested_mask="r" denied_mask="r" fsuid=1001 ouid=0 Feb 20 10:19:30 secured-ghost-on-ubuntu-16-04-lts-1-vm audit[24854]: SYSCALL arch=c000003e syscall=257 success=no exit=-13 a0=ffffff9c a1=7f8c37585e07 a2=80000 a3=0 items=1 ppid=24833 pid=24854 auid=1001 uid=1001 gid=1002 euid=1001 suid=1001 fsuid=1001 egid=1002 sgid=1002 fsgid=1002 tty=pts0 ses=2 comm="curl" exe="/snap/cvescan/6/usr/bin/curl" key=(null) Feb 20 10:19:30 secured-ghost-on-ubuntu-16-04-lts-1-vm audit: CWD cwd="/home/kiko/snap/cvescan/common" Feb 20 10:19:30 secured-ghost-on-ubuntu-16-04-lts-1-vm audit[24854]: AVC apparmor="DENIED" operation="create" profile="snap.cvescan.cvescan" pid=24854 comm="curl" family="inet" sock_type="dgram" protocol=0 requested_mask="create" denied_mask="create" Feb 20 10:19:30 secured-ghost-on-ubuntu-16-04-lts-1-vm audit[24854]: SYSCALL arch=c000003e syscall=41 success=no exit=-13 a0=2 a1=80802 a2=0 a3=2b items=0 ppid=24833 pid=24854 auid=1001 uid=1001 gid=1002 euid=1001 suid=1001 fsuid=1001 egid=1002 sgid=1002 fsgid=1002 tty=pts0 ses=2 comm="curl" exe="/snap/cvescan/6/usr/bin/curl" key=(null)

…

On Wed, Feb 19, 2020 at 12:58 PM jdstrand ***@***.***> wrote: I'm not sure what is special about this VM; we're hitting a security policy, perhaps? What is the output of $ sudo journalctl |grep audit at the time of the failure? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AOQWRDNG6U7XJ3C7DOYXZ5DRDVJJDA5CNFSM4KVVYDN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMIKFKA#issuecomment-588292776>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOQWRDPH4YEIYT3K7DBJY3DRDVJJDANCNFSM4KVVYDNQ> .

[kiko-canonical]

For reference, yes, installing core in this Xenial instance does address the issue. I'm not sure what makes this instance special, however. Here is some output: kiko@secured-ghost-on-ubuntu-16-04-lts-1-vm:~$ snap version snap 2.37.4ubuntu0.1 snapd 2.37.4ubuntu0.1 series 16 ubuntu 16.04 kernel 4.15.0-1052-gcp kiko@secured-ghost-on-ubuntu-16-04-lts-1-vm:~$ sudo snap refresh All snaps up to date. kiko@secured-ghost-on-ubuntu-16-04-lts-1-vm:~$ cat /proc/version_signature Ubuntu 4.15.0-1052.56-gcp 4.15.18 I can give ssh access to it if it will help investigate. Thanks!

…

On Wed, Feb 19, 2020 at 6:02 PM Mark Morlino ***@***.***> wrote: 💯 👍 that seems to make it work @jdstrand <https://github.com/jdstrand> thanks! Do you think @kiko-canonical <https://github.com/kiko-canonical> was having the same issue under xenial (the behavior seemed the same)? Do you still think this is something for the snapd team? It seems like core should get installed automatically if it needs to be there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AOQWRDN367YZPAVOULGKK6LRDWM5TA5CNFSM4KVVYDN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMJUW3A#issuecomment-588467052>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOQWRDIDP3BNMWGNFEB524DRDWM5TANCNFSM4KVVYDNQ> .

[jdstrand]

@kiko-canonical - thanks for the info. One more question, what is the output of apt-cache policy snapd?

[kiko-canonical]

kiko@secured-ghost-on-ubuntu-16-04-lts-1-vm:~$ apt-cache policy snapd snapd: Installed: 2.37.4ubuntu0.1 Candidate: 2.40 Version table: 2.40 500 500 http://us-east4-a.gce.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages *** 2.37.4ubuntu0.1 500 500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages 100 /var/lib/dpkg/status 2.0.2 500 500 http://us-east4-a.gce.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages I haven't tried upgrading, now that I see that one is available.

…

On Thu, Feb 20, 2020 at 9:11 AM jdstrand ***@***.***> wrote: @kiko-canonical <https://github.com/kiko-canonical> - thanks for the info. One more question, what is the output of apt-cache policy snapd? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AOQWRDLGMYA7ANIT7QJYY4TRDZXQFA5CNFSM4KVVYDN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMNUN7Q#issuecomment-588990206>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOQWRDO44KM5T5C2WZHVXALRDZXQFANCNFSM4KVVYDNQ> .

[mvo5]

Thanks for this bugreport and update. It looks like the VM is not having the updated snapd installed, only the older version from security. I wonder if we can refresh the cloud image to have a newer snapd by default? About trusty, we still build/test snapd on trusty, I can ensure we do another upload so that the latest fixes for this get in (we do not SRU to trusty anymore by default as it's EOL but we still keep it working and make exceptions for SRUs).

[mvo5]

Fwiw, I see snapfuse consume cpu after "maas init" has finished, looking further it looks like there are a ton of "supervisorctl status" and "supervisorctl restart" comands in ps. When I count them I see >200 supervisorctl restart commands for various services like syslog, ntp, proxy. The number also seems to be growing, i..e it's 240 now (so grew by 40 in the time I needed to write these sentences).

[mvo5]

Just to ensure things are fine on 14.04 as well I did a new SRU of snapd on trusty with 2.43.3. This will install the snapd snap automatically so everyone wil lget the benefit of re-exec and auto-refreshes of the snapd snap. This should fix the manual steps that @markmorlino had to do.

[kiko-canonical]

I've upgraded to snapd 2.40 on the VM instance I had and the problem is resolved; snapd upgraded itself upon installing cvescan, and it works now out of the box on a fresh VM, without needing to install the core stnap: kiko@secured-ghost-on-ubuntu-16-04-lts-2-vm:~$ snap list Name Version Rev Tracking Publisher Notes core18 20200124 1668 stable canonical✓ base cvescan 1.0.1 6 stable markmorlino - snapd 2.43.3 6434 stable canonical✓ snapd I think this is addressed for me on old Xenial now.

…

On Thu, Feb 20, 2020 at 2:30 PM Michael Vogt ***@***.***> wrote: Just to ensure things are fine on 14.04 as well I did a new SRU of snapd on trusty with 2.43.3. This will install the snapd snap automatically so everyone wil lget the benefit of re-exec and auto-refreshes of the snapd snap. This should fix the manual steps that @markmorlino <https://github.com/markmorlino> had to do. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AOQWRDOXTNV4RBSSDG64XCDRD24Z5A5CNFSM4KVVYDN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMPLC6Y#issuecomment-589214075>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOQWRDN3SH5OGWTXVAYHKHTRD24Z5ANCNFSM4KVVYDNQ> .