Using multiarch s390x (and ppc64) on little endian systems - seg fault? about qemu-user-static

[jeking3]

I also tried grabbing the qemu-user-static package v3.1 from Ubuntu Disco and installed it on Bionic:

    # wget http://lug.mtu.edu/ubuntu/pool/universe/q/qemu/qemu-user-static_3.1+dfsg-2ubuntu3.1_amd64.deb
    # dpkg -i qemu-user-static_3.1+dfsg-2ubuntu3.1_amd64.deb

Then I registered with binfmt using the docker container, and I can start a s390x container but I cannot do anything that involves the network:

jking@ubuntu:~$ cat /proc/sys/fs/binfmt_misc/qemu-s390x
enabled
interpreter /usr/bin/qemu-s390x-static
flags:
offset 0
magic 7f454c4602020100000000000000000000020016
mask ffffffffffffff00fffffffffffffffffffeffff
jking@ubuntu:~$ /usr/bin/qemu-s390x-static --version
qemu-s390x version 3.1.0 (Debian 1:3.1+dfsg-2ubuntu3.1)
Copyright (c) 2003-2018 Fabrice Bellard and the QEMU Project developers
jking@ubuntu:~$ docker run --rm -it multiarch/fedora:29-s390x
[root@61f0e74c7574 /]# yum update
Illegal instruction (core dumped)                 [===                                         ] ---  B/s |   0  B     --:-- ETA

I wasn't able to build qemu 4.0 from source properly (see last comment). In one example I found someone using docker volumes to push the /usr/bin/qemu-s390x-static file through to the container. Does that help or make sense? Well, I tried it, and it seems to make things work on the 25-ppc64 image:

jking@ubuntu:~$ docker run --rm -v /usr/bin/qemu-ppc64-static:/usr/bin/qemu-ppc64-static -it multiarch/fedora:25-ppc64
[root@5b29bd257351 /]# dnf update
Fedora 25 - ppc64                                                                                                             2.2 MB/s |  44 MB     00:19
...

So it looks like I have a solution, although it's pretty fragile, I can do what I need to do in this case, but not with the s390x images.

[jeking3]

Running a big-endian container on Ubuntu Bionic

This procedure allows you to run a Fedora PowerPC container on an x86_64 host with Ubuntu Bionic.
(Could not find any debian based containers for multiarch that worked)

1. Install binfmt-support:

    root@ubuntu:~# apt-get install -y binfmt-support
   

2. Get the disco qemu 3.1 qemu-user-static package:

    root@ubuntu:~# wget http://lug.mtu.edu/ubuntu/pool/universe/q/qemu/qemu-user-static_3.1+dfsg-2ubuntu3.1_amd64.deb
    root@ubuntu:~# dpkg -i qemu-user-static_3.1+dfsg-2ubuntu3.1_amd64.deb
   

3. Register the qemu translators:

    user@ubuntu:~$ docker run --rm --privileged multiarch/qemu-user-static:register --reset
   

4. Run a big endian container:

   The pass-through of /usr/bin/qemu-ppc64-static seems to be necessary for proper operation.
   The overall performance it pretty sluggish inside the container - it should be sufficient
   to do some small task like debugging

    user@ubuntu:~$ docker run --rm -v /usr/bin/qemu-ppc64-static:/usr/bin/qemu-ppc64-static -it multiarch/fedora:25-ppc64
    [root@5b29bd257351 /]# dnf update
    Fedora 25 - ppc64                                                 2.2 MB/s |  44 MB     00:19
    Fedora 25 - ppc64 - Updates                                       6.2 MB/s |  18 MB     00:02
    Last metadata expiration check: 0:01:06 ago on Tue Jun 11 12:50:38 2019.
    Dependencies resolved.
    ...
    Transaction Summary
    ===================
    Install    8 Packages
    Upgrade  122 Packages
    
    Total download size: 71 M
    Is this ok [y/N]: N
   

[junaruga]

@jeking3 Did you use multiarch/fedora:30-s390x image, right?
Right now qemu v3.1.0 is used for the compatible images for the fedora.
https://github.com/multiarch/fedora/blob/master/.travis.yml#L19
As the latest version is 4.0.0, we can upgrade it

if we can reproduce this bug with simple script, we might be able to report it qemu project.
https://github.com/qemu/
https://bugs.launchpad.net/qemu/
https://www.qemu.org/contribute/report-a-bug/

[junaruga]

You are using 'yum', so, are you using CentOS?
If CentOS, here is a similar issue.
multiarch/centos#1

If you use Fedora, you can use dnf instead of yum.

dnf -y update
dnf -y install something

[junaruga]

Good news:
After this PR will be merged, multiarch/fedora:29-s390x works.
multiarch/fedora#15

[jeking3]

Perhaps the issue is qemu 2.11 from Ubuntu Bionic. I'll remove that and build a newer one into the system and we'll see what happens.

[junaruga]

Yeah, old version qemu causes some issues. On qemu 2, we saw an issue about that Java did not work on it in this repository.

[junaruga]

On this repository, Fedora Project's built binary qemu is used.
https://github.com/multiarch/qemu-user-static/blob/master/.travis.yml#L19
https://src.fedoraproject.org/rpms/qemu/blob/master/f/qemu.spec
The qemu is new, and well-maintained with patch files in the qemu.spec file.

[jeking3]

How do I build qemu-4.0.0 from source and produce /usr/local/bin/qemu-s390x-static ? I have tried running configure --static but the binaries do not come out with -static in them. The docker image that registers the multiarch binfmt hooks uses -static as a suffix.

After building qemu 4.0 and installing into /usr/local/bin, I made a symlink, but I still cannot run anything:

root@ubuntu:/usr/local/bin# ln -s qemu-s390x qemu-s390x-static
root@ubuntu:/usr/local/bin# cat /proc/sys/fs/binfmt_misc/qemu-s390x
enabled
interpreter /usr/local/bin/qemu-s390x-static
flags:
offset 0
magic 7f454c4602020100000000000000000000020016
mask ffffffffffffff00fffffffffffffffffffeffff
root@ubuntu:/usr/local/bin# /usr/local/bin/qemu-s390x-static --version
qemu-s390x version 4.0.0
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
root@ubuntu:/usr/local/bin# exit
logout
jking@ubuntu:~$ docker run --rm -it multiarch:fedora/29-s390x /bin/bash
docker: invalid reference format.
See 'docker run --help'.
jking@ubuntu:~$ docker run --rm -it multiarch:fedora/30-s390x
docker: invalid reference format.
See 'docker run --help'.

[junaruga]

In one example I found someone using docker volumes to push the /usr/bin/qemu-s390x-static file through to the container. Does that help or make sense?

Yeah, it makes sense!

[junaruga]

I just share one more case to run Fedora 29 s390x (Big endian) on x86_64 (little endian) with the compatible container image.

$ uname -m
x86_64

$ docker run --rm --privileged multiarch/qemu-user-static:register --reset

$ docker run --rm -t multiarch/fedora:29-s390x bash -cx "
uname -m
dnf -y update
dnf -y install gcc
gcc --version
"
...
+ uname -m
s390x
...
+ gcc --version
gcc (GCC) 8.3.1 20190223 (Red Hat 8.3.1-2)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[junaruga]

Hi @jeking3 , 2 days ago, we released a new feature (new image multiarch/qemu-user-static) to use a standard architecture specific container image that creates /proc/sys/fs/binfmt_misc/qemu-$arch files with flags: F to use QEMU without specifying the interpreter /usr/bin/qemu-$arch-static for each run time.

Now you can use the standard s390 (and ppc64) container image. Below commands work.
See the updated document https://github.com/multiarch/qemu-user-static for detail.

$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

$ cat /proc/sys/fs/binfmt_misc/qemu-s390x
enabled
interpreter /usr/bin/qemu-s390x-static
flags: F
offset 0
magic 7f454c4602020100000000000000000000020016
mask ffffffffffffff00fffffffffffffffffffeffff

$ docker run --rm -t s390x/fedora uname -m
s390x

$ docker run --rm -t s390x/fedora:30 uname -m
s390x

$ docker run --rm -t s390x/fedora:30 bash -cx '
uname -m && \
dnf -y upgrade && \
dnf -y install gcc
'
$ echo $?
0

$ docker run --rm -t s390x/ubuntu uname -m
s390x

$ docker run --rm -t s390x/ubuntu:bionic uname -m
s390x

$ docker run --rm -t s390x/ubuntu:bionic bash -cx '
apt-get -y update && \
apt-get -y install gcc
'

$ echo $?
0