multiarch / qemu-user-static Goto Github PK
View Code? Open in Web Editor NEW:earth_africa: `/usr/bin/qemu-*-static`
Home Page: https://hub.docker.com/r/multiarch/qemu-user-static/
License: MIT License
:earth_africa: `/usr/bin/qemu-*-static`
Home Page: https://hub.docker.com/r/multiarch/qemu-user-static/
License: MIT License
My host is an Ubuntu Bionic VM in ESXi 6.7 on a system with a Xeon processor.
I am using a relatively new version of docker:
Docker version 18.09.6, build 481bc77
I have installed qemu-user-static:
jking@ubuntu:~/bdde$ dpkg -l | grep qemu
ii qemu-user-static 1:2.11+dfsg-1ubuntu7.14 amd64 QEMU user mode emulation binaries (static version)
I have run the required register and I can see things in /proc. Let's take s390x for example:
jking@ubuntu:~/bdde$ cat /proc/sys/fs/binfmt_misc/qemu-s390x
enabled
interpreter /usr/bin/qemu-s390x-static
flags:
offset 0
magic 7f454c4602020100000000000000000000020016
mask ffffffffffffff00fffffffffffffffffffeffff
All the examples I could find do trivial checks to see if things are working. When I try to do a simple "yum list" it fails:
[root@5f54f1622a90 bdde]# yum list
Illegal instruction (core dumped) [=== ] --- B/s | 0 B --:-- ETA
When I try to install something like gcc it fails:
[root@f91e5a0023fa bdde]# yum install gcc
Illegal instruction (core dumped) [=== ] --- B/s | 0 B --:-- ETA
I am a member of the Boost Community Maintenance Team and I am working on adding a big-endian docker container to CI resources so we can test big and little endian issues in CI. For example there's an open bug in boostorg/uuid about endian handling, and I have no access to a big endian system. I'd like to use a docker container since it is portable and can run in Travis CI.
How can I diagnose and correct this issue? I know this may not be specific to the Fedora image; I could not find any Ubuntu based images that work either (most are too old to be useful).
Thanks.
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind enhancement
Description:
We recently released new feature to add version tag images (#74 (comment)) by the PR #84 .
I added the version tag vX.Y.Z-R
with people's agreements.
The philosophy came from fX.Y.Z-R
and dX.Y.Z-R
of https://github.com/dbhi/qus .
I thought if our version tag meant common version tag, I thought vX.Y.Z-R
was good to mean the common version.
But in case of this project, I think X.Y.Z-R
is better than vX.Y.Z-R
, after rethinking it. Because the version string without v
is very common in the container version rule such as https://hub.docker.com/_/alpine .
I admit I made mistake.
So, you like I like to change the image version tag to change from "vX.Y.Z-R" to "X.Y.Z-R".
Then I want to run the pipeline for 3.1.1-2
, 4.0.0-4
and 4.0.0-5
again.
How do you think?
$ docker run --rm --privileged multiarch/qemu-user-static:register --reset
sh: write error: File exists
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
sh: write error: File exists
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
sh: write error: File exists
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
sh: write error: File exists
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
sh: write error: File exists
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
sh: write error: File exists
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
sh: write error: File exists
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
sh: write error: File exists
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
sh: write error: File exists
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
sh: write error: File exists
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
sh: write error: File exists
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
sh: write error: File exists
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
sh: write error: File exists
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
sh: write error: File exists
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
sh: write error: File exists
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
sh: write error: File exists
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
sh: write error: File exists
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
sh: write error: File exists
$ docker info
Containers: 3
Running: 0
Paused: 0
Stopped: 3
Images: 26
Server Version: 17.09.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.13.0-16-generic
Operating System: Ubuntu 17.10
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.15GiB
Name: me
ID: UGF7:3IR1:SSKZ:BOMH:N667:5IDN:TYYC:VHBG:BHCX:RAQV:NHRY:3V49
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: aledbf
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Any idea how can be fixed?
Thanks in advance
Since QEMU 3.0 is released, it would be nice to update it. Thanks!
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind question
Possible issue or bug with QEMU - need help identifying where issue is
Description:
I'm using latest multiarch/qemu-user-static installed via Docker container.
I run some builds using QEMUs via Docker buildx. Docker builds work fine for AArch64, but I consistently get a failure like below when attempting to build for 32-bit ARM (hard-float).
The containers are successfully built for AMD64 (ie. with no emulation) and Aarch64 (using QEMU emulation), but, fail for the ARMHF (ARMv7 32-bit) build again using QEMU emulation. I am using Focal 20.04 as the host and also building a Focal 20.04 container.
I have specifically used cmake 3.16.3 (from Focal 20.04 repo) and also compiled and used cmake 3.17.1
For reference, I am also using Conan as the C++ package manager - Version 1.24.1 / Docker 19.03.8 with the very latest buildx / Moby-Buildkit v0.7.1... All I believe are current
Please find attached the output of my failure
Finally, I have done my best to enable the--trace-expand
option on cmake and redirect stdout and stderr - As you can see its not perfect but I'm hoping it will give you something to go on..
stdout.out.txt
stderr.out (1).txt
Steps to reproduce the issue:
Generally following this guide using multiarch/qemu-user-static - https://medium.com/@artur.klauser/building-multi-architecture-docker-images-with-buildx-27d80f7e2408
I can provide access to a Git containing the build Dockerfile - Please let me know the required Git account
Typically setting buildx to support linux ARM/v7, Aarch64 and amd64
Run using: docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 -t rhastie/container .
Describe the results you received:
Container is successfully built for amd64 (ie. x86_64) no QEMU emulation
Container is successfully built for Aarch64 via QEMU emulation
Container build errors and terminates for Arm/v7 via QEMU emulation
Describe the results you expected:
I would expect the container to be built for both architectures - This may be either cmake issue or a QEMU issue. Unusual that it works for Aarch64 and not armhf under QEMU.
Environment:
x86-64 Running Ubuntu Focal 20.04
Docker 19.03.8
Moby-Buildkit v0.7.1
cmake 3.16.3 and cmake 3.17.1 tested
latest buildx CLI from - https://github.com/docker/buildx
latest QEMU-user-static from - https://github.com/multiarch/qemu-user-static
Container being built using Focal 20.04
docker version
, podman version
or singularity version
Client:
Version: 19.03.8
API version: 1.40
Go version: go1.13.8
Git commit: afacb8b7f0
Built: Wed Mar 11 23:42:35 2020
OS/Arch: linux/amd64
Experimental: true
Server:
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.13.8
Git commit: afacb8b7f0
Built: Wed Mar 11 22:48:33 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.3-0ubuntu2
GitCommit:
runc:
Version: spec: 1.0.1-dev
GitCommit:
docker-init:
Version: 0.18.0
GitCommit:
Additional information optionally:
I also have an issue raised here: https://gitlab.kitware.com/cmake/cmake/-/issues/20568
When I executed any of qemu
Hi, on my x86_64 server:
$ uname -a
Linux a010291 4.4.0-127-generic #153-Ubuntu SMP Sat May 19 10:58:46 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ docker version
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:17:20 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:15:30 2018
OS/Arch: linux/amd64
Run the command "docker run --rm --privileged multiarch/qemu-user-static:register" encounters errors:
sh: write error: Invalid argument
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
sh: write error: Invalid argument
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
sh: write error: Invalid argument
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
sh: write error: Invalid argument
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
sh: write error: Invalid argument
But run command ""docker run --rm --privileged multiarch/qemu-user-static:register --reset" actually installs all the interpreters successfully:
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind question
Description:
Using qemu-mips64el-static
and image mips64el aoqi/debian-mips64el:oldstable
to compile bazel application, it will very slow with javac ...
in debug output(it's not still finished in 10m above). However, it will costs 1m ~ 2m to finish javac command in x86_64 container). In mips64el container, speed is too slow.
Steps to reproduce the issue:
start x86_64 debian:stretch
(since debian:stretch has java 8), and start mips64el aoqi/debian-mips64el:oldstable
(https://hub.docker.com/r/aoqi/debian-mips64el) with qemu-mips64el-static
in another terminal
install openjdk-8-jdk etc. development tools, and download it from https://github.com/bazelbuild/bazel/releases/download/2.0.0/bazel-2.0.0-dist.zip and then extract it
start compile with below similar command
root@container:/home/bazel-2.0# pwd
/home/bazel-2.0
root@container:/home/bazel-2.0# PATH=/usr/lib/ccache:$PATH EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk --explain /tmp/LOG --verbose_explanations --compilation_mode fastbuild --verbose_failures --show_progress --subcommands --show_timestamps --color yes" VERBOSE=yes BAZEL_DEBUG_JAVA_COMPILATION=1 BAZEL_JAVAC_OPTS="-J-Xmx6g -J-Xms4g" bash compile.sh
Describe the results you received:
Using mips64el aoqi/debian-mips64el:oldstable
to compile it, it will very slow with javac ...
in debug output. However, it will costs 1m ~ 2m to finish javac command in x86_64 container):
root@container:/home/bazel-2.0# ...... bash compile
... ...
tools/java/runfiles/Runfiles.java
tools/java/runfiles/Util.java
/usr/lib/jvm/java-8-openjdk-amd64/bin/javac -classpath ... ...
... ...
Describe the results you expected:
I know about that performance will decrease using qemu user mode, but it's abnormal for so slow speed in qemu-mips64el-static. It is a normal phenomenon?
Environment:
[yancy@asus github]$ cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
[yancy@asus github]$ uname -a
Linux asus 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
[yancy@asus github]$ qemu-mips64el-static --version
qemu-mips64el version 4.0.0 (qemu-4.0.0-5.fc31)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
[yancy@asus github]$ cat /proc/sys/fs/binfmt_misc/qemu-mips64el
enabled
interpreter /usr/bin/qemu-mips64el-static
flags:
offset 0
magic 7f454c4602010100000000000000000002000800
mask ffffffffffffff000000000000000000feffffff
[yancy@asus github]$ rpm -qa docker-ce
docker-ce-19.03.5-3.el7.x86_64
[yancy@asus github]$ docker version
Client: Docker Engine - Community
Version: 19.03.5
API version: 1.40
Go version: go1.12.12
Git commit: 633a0ea
Built: Wed Nov 13 07:25:41 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.12.12
Git commit: 633a0ea
Built: Wed Nov 13 07:24:18 2019
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
root@container:/home/bazel-2.0-mips64# uname -m
mips64
root@container:/home/bazel-2.0-mips64# java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-8u222-b10-1~deb9u1-b10)
OpenJDK 64-Bit Zero VM (build 25.222-b10, interpreted mode)
root@container:/home/bazel-2.0-mips64# qemu-mips64el-static --version
qemu-mips64el version 4.0.0 (qemu-4.0.0-5.fc31)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind enhancement
Description:
I think it worth mention in README, that Docker Hub and QEMU have different notion of architecture. And I found that by abusing docker volume, I can register a single qemu-user-static binfmt_misc entry easily.
I came up with GitHub Actions workflow like this:
on: push
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
docker_arch: [amd64, i386, arm32v6, arm32v7, arm64v8, ppc64le, s390x] # arm32v5 is not supported by alpine
steps:
- name: Register binfmt_misc entry for qemu-user-static
env:
DOCKER_ARCH: ${{ matrix.docker_arch }}
run: |
case ${DOCKER_ARCH} in
amd64|i386)
QEMU_ARCH=
;;
arm32*)
QEMU_ARCH=arm
;;
arm64*)
QEMU_ARCH=aarch64
;;
*)
QEMU_ARCH=${DOCKER_ARCH}
;;
esac
if [ -n "${QEMU_ARCH}" ]; then
docker rm $(docker create --volume qemu-user-static:/usr/bin multiarch/qemu-user-static:${QEMU_ARCH} dummy)
docker run --rm --privileged --volume qemu-user-static:/usr/bin:ro multiarch/qemu-user-static:register --persistent yes
fi
- name: Build
env:
DOCKER_ARCH: ${{ matrix.docker_arch }}
run: |
docker run --rm --interactive "${DOCKER_ARCH}/alpine:latest" <<-'EOF'
echo 'Hello World!'
EOF
Hi, I'm a big fan of your QEMU builds, thank you!
Now that v2.9.0 was released this week, would you mind upgrading to the latest version?
Cheers!
I'm trying to use qemu-arm-static
in a debian stretch based docker image, built from a debootstrap, using raspbian mirrors.
I've noticed that qemu-arm-static
versions newer than 2.9.1 are really slow. It takes 10x or 20x as long for apt-get
operations to run, for example. My host machine is x86_64, Ubuntu 16.04 (4.13.0-37-generic).
Just wondering if anyone else has seen this regression? I can provide more information if it's useful.
x86_64 does not get registered.
Can you please add x86_64?
Right now these images contain one file /usr/bin which is the tar.gz of the corresponding qemu-user-static binary. I don't understand the reason behind this. My thoughts on this:
Line 34 in 5ee00f7
In my dockerfile I would love to do something like:
FROM multiarch/qemu-user-static:x86_64-aarch64 as qemu
FROM arm64v8/ubuntu
COPY --from=qemu /usr/bin/qemu-* /usr/bin
#continue with my now qemu equipped docker image
What I have to do right now to get the same effect:
FROM multiarch/qemu-user-static:x86_64-aarch64 as qemu
FROM alpine:3.5 as qemu_extract
COPY --from=qemu /usr/bin qemu_user_static.tgz
RUN tar -xzvf qemu_user_static.tgz
FROM arm64v8/ubuntu
COPY --from=qemu_extract qemu-* /usr/bin
#continue with my now qemu equipped docker image
Maybe I'm just missing something obvious or didn't understand the purpose of these images.
Regards,
Stefan
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind announcement
Description:
Last week, I had a talk session at Flock event (= Fedora project community event) to share how multiarch/qemu-user-static
is great. Yes. this project :)
https://flock2019.sched.com/event/SJLx/lets-add-fedora-multiarch-containers-to-your-ci
When we want to know how multiarch/qemu-user-static
works, we might need to know below tech topics.
qemu-$arch-static
)multiarch/qemu-user-static
image works.I believe below my slide with some pictures for the talk would help you to understand the topics.
As people in Fedora would tend to prefer podman
than docker
for the container command, the examples in the slides are for podman
. But you can read it replacing it docker
.
As the talk was recorded, when it is prepared, I would share it here.
I could not accomplish it without people's help in this project and dbhi/qus
project.
Thanks!
Cheers.
We are trying to run sudo inside the alpine docker image and it fails with:
sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges
This may be related:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=683205
/kind question
Description:
the :register
image seems to be missing the qemu-*
binaries
Steps to reproduce the issue:
~ 👺 diff -Nru <(docker pull multiarch/qemu-user-static:register && docker run --rm --privileged --entryp
oint /bin/sh multiarch/qemu-user-static:register -c find) <(docker pull multiarch/qemu-user-static && doc
ker run --rm --privileged --entrypoint /bin/sh multiarch/qemu-user-static -c find)
--- /dev/fd/63 2019-10-01 10:34:37.885057387 +0900
+++ /dev/fd/62 2019-10-01 10:34:37.885057387 +0900
@@ -1,9 +1,45 @@
-register: Pulling from multiarch/qemu-user-static
-Digest: sha256:c77eb2da3597aa370f07ef970e2e0adf155172eb9d3c40e43d97aa43eef6b0c9
-Status: Image is up to date for multiarch/qemu-user-static:register
+Using default tag: latest
+latest: Pulling from multiarch/qemu-user-static
+Digest: sha256:0d5897acf5f822cec80fe40031856d417fa3c8c8cfa5d4dc66e30c74a72a450e
+Status: Image is up to date for multiarch/qemu-user-static:latest
.
./usr
./usr/sbin
+./usr/bin
+./usr/bin/qemu-sh4eb-static
+./usr/bin/qemu-ppc64le-static
+./usr/bin/qemu-microblaze-static
+./usr/bin/qemu-sh4-static
+./usr/bin/qemu-nios2-static
+./usr/bin/qemu-mips-static
+./usr/bin/qemu-aarch64-static
+./usr/bin/qemu-sparc64-static
+./usr/bin/qemu-s390x-static
+./usr/bin/qemu-sparc-static
+./usr/bin/qemu-xtensa-static
+./usr/bin/qemu-ppc-static
+./usr/bin/qemu-arm-static
+./usr/bin/qemu-tilegx-static
+./usr/bin/qemu-i386-static
+./usr/bin/qemu-cris-static
+./usr/bin/qemu-alpha-static
+./usr/bin/qemu-armeb-static
+./usr/bin/qemu-sparc32plus-static
+./usr/bin/qemu-mips64-static
+./usr/bin/qemu-ppc64abi32-static
+./usr/bin/qemu-mipsn32el-static
+./usr/bin/qemu-xtensaeb-static
+./usr/bin/qemu-riscv64-static
+./usr/bin/qemu-mips64el-static
+./usr/bin/qemu-hppa-static
+./usr/bin/qemu-ppc64-static
+./usr/bin/qemu-mipsn32-static
+./usr/bin/qemu-m68k-static
+./usr/bin/qemu-microblazeel-static
+./usr/bin/qemu-riscv32-static
+./usr/bin/qemu-mipsel-static
+./usr/bin/qemu-aarch64_be-static
+./usr/bin/qemu-or1k-static
./root
./dev
./dev/core
Describe the results you received:
./usr/bin/qemu-*
files are missing in the :register
image
Describe the results you expected:
./usr/bin/qemu-*
files are included in the :register
image
Environment:
Output of docker version
~ 🍊 docker version
Client:
Version: 18.09.3
API version: 1.39
Go version: go1.10.8
Git commit: 774a1f4
Built: Thu Feb 28 06:34:04 2019
OS/Arch: linux/amd64
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 18.09.3
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 774a1f4
Built: Thu Feb 28 05:59:55 2019
OS/Arch: linux/amd64
Experimental: false
The message #75 seems that they should be included:
Add all the binaries in multiarch/qemu-user-static:* images to image multiarch/qemu-user-static:register.
This will cause :register --reset -p true
to fail w/ sh: write error: No such file or directory
is this the expected behavior?
qemu-s390x-static release 2.9.1 and other released binaries have a release version of 2.9.0 within the version string. Is this intended?
I am attempting to bump to Debian stretch within kubernetes and am hitting an issue with GNU/Linux vs SYSV binaries. 2.9.1 of qemu is supposed to fix this issue.
~/Downloads $ ./qemu-s390x-static --version
qemu-s390x version 2.9.0(qemu-2.9.0-1.fc27)
Steps to replicate:
Fedora 25
wine-systemd installed
Contents of /proc/sys/fs/binfmt_misc
should be:
status register windows windowsPE
Run docker run --rm --privileged multiarch/qemu-user-static:register --reset
Contents of /proc/sys/fs/binfmt_misc
now missing windows
and windowsPE
when running --reset, you will effectively remove all but status and register in /proc/sys/fs/binfmt_misc
- if you like me have a setup that use the same folder --reset
will remove all those files.
--reset flag should just remove qemu files, right ? If so here is a fix:
entries="aarch64 arm m68k mips64 mipsel mipsn32el ppc64 sh4 sparc alpha armeb mips mips64el mipsn32 ppc ppc64le s390x sh4eb"
if [ "${1}" = "--reset" ]; then
(
cd /proc/sys/fs/binfmt_misc
for file in $entries; do
if [ -f $file ]; then
echo -1 > "$file"
fi
done
)
fi
Hi, is it possible to soon do a new release for v2.7?
That would be cool
/kind bug
Description:
Similar error to #38, but different circumstances.
Operating system info:
$ cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)
$ uname -a
Linux saturn 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ ll /proc/sys/fs/binfmt_misc
.rw-r--r-- 0 root root 16 Dec 15:06 jexec
.-w------- 0 root root 16 Dec 15:28 register
.rw-r--r-- 0 root root 16 Dec 15:05 status
$ rpm -qa | grep qemu
# nothing, there are no pre-existing qemu packages installed
Then when I run this as root:
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
I get:
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
sh: write error: Invalid argument
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
sh: write error: Invalid argument
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
sh: write error: Invalid argument
sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc-static as binfmt interpreter for sparc
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
sh: write error: Invalid argument
Setting /usr/bin/qemu-sparc64-static as binfmt interpreter for sparc64
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
sh: write error: Invalid argument
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
sh: write error: Invalid argument
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
sh: write error: Invalid argument
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
sh: write error: Invalid argument
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
sh: write error: Invalid argument
sh: write error: Invalid argument
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
sh: write error: Invalid argument
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
sh: write error: Invalid argument
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
sh: write error: Invalid argument
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
sh: write error: Invalid argument
sh: write error: Invalid argument
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
sh: write error: Invalid argument
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
sh: write error: Invalid argument
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
sh: write error: Invalid argument
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
sh: write error: Invalid argument
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
sh: write error: Invalid argument
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
sh: write error: Invalid argument
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
sh: write error: Invalid argument
What am I missing? This command works on my other linux mint distro, but my CI nodes are running Centos7.
I like to create a GitHub issue template for this repository.
Because maybe we want to know
https://help.github.com/en/articles/about-issue-and-pull-request-templates
https://help.github.com/en/articles/creating-issue-templates-for-your-repository
I think below is a good example about the content though it is a old version of the template.
https://github.com/containers/libpod/blob/master/.github/ISSUE_TEMPLATE.md
Below is a case for the new version.
https://github.com/WorksOnArm/cluster/tree/master/.github/ISSUE_TEMPLATE
Shall we create it?
I believe this makes our work better.
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind bug
Description:
After I included 4.0.0-5 into my base images, some future builds like for python fails to install gcc with an xattrs set attributes errors. After I revert to previus version, it works.
Steps to reproduce the issue:
I saw the issue only on Azure Pipelines, but I had no time to investigate the issue deeper. I leaf only a issue that maybe other known that it could have an issue with this version.
I have one application that I compile in an ubuntu-core:arm64 container that, when run inside that container, causes the following errors:
Unknown host QEMU_IFLA type: 47
Unknown host QEMU_IFLA type: 48
Unknown host QEMU_IFLA type: 43
Copying the executable to an arm64 allows me to run it correctly, but on the x86_64 host, it runs but causes this error (and seems to have problems with network communication, although I am not sure it is related to this problem).
Hi,
We have systems that would stop working if we want to use the qemu-arm-static v2.8.0 that is used in this project.
I've created a project to demonstrate the problem.
I'm not quite sure where the problem is to be honest, is it with Docker, qemu or Java ? I don't know for sure. But what I do know, is that it was working with 2.5, 2.6 and 2.7 and stopped working with 2.8
The demonstration project linked above is a very simplified version of what we use and was created specifically to help the debugging of this issue. So feel free to clone it and let me know if you need more information.
The Dockerfile start from "multiarch/debian-debootstrap:armhf-jessie" and simply add oracle-java and maven. The qemu-arm-static binaries are overridden using volume bind (-v).
The test.sh will take care of everything to demonstrate the problem
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind bug
Description:
Steps to reproduce the issue:
Register x86_64 host with the latest qemu-user-static.
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
Build the following Docker Image using following Dockerfile.s390x using command docker build -t test/crossbuild:latest-s390x -f Dockerfile.s390x .
Dockerfile.s390x
FROM alpine:3.11 as qemu
ARG QEMU_VERSION=5.0.0-2
ARG QEMU_ARCHS="s390x"
RUN apk --update add curl
#Enable non-native runs on amd64 architecture hosts
RUN for i in ${QEMU_ARCHS}; do curl -L https://github.com/multiarch/qemu-user-static/releases/download/v${QEMU_VERSION}/qemu-${i}-static.tar.gz | tar zxvf - -C /usr/bin; done
RUN chmod +x /usr/bin/qemu-*
FROM s390x/golang:1.14.2-alpine3.11
MAINTAINER LoZ Open Source Ecosystem (https://www.ibm.com/developerworks/community/groups/community/lozopensource)
ARG MANIFEST_TOOL_VERSION=v1.0.2
#Enable non-native builds of this image on an amd64 hosts.
#This must be the first RUN command in this file!
COPY --from=qemu /usr/bin/qemu-*-static /usr/bin/
#Install su-exec for use in the entrypoint.sh (so processes run as the right user)
#Install bash for the entry script (and because it's generally useful)
#Install curl to download glide
#Install git for fetching Go dependencies
#Install ssh for fetching Go dependencies
#Install mercurial for fetching go dependencies
#Install wget since it's useful for fetching
#Install make for building things
#Install util-linux for column command (used for output formatting).
#Install grep and sed for use in some Makefiles (e.g. pulling versions out of glide.yaml)
#Install shadow for useradd (it allows to use big UID)
RUN apk update && apk add --no-cache su-exec curl bash git openssh mercurial make wget util-linux tini file grep sed shadow
RUN apk upgrade --no-cache
#Disable ssh host key checking
RUN echo 'Host *' >> /etc/ssh/ssh_config \
&& echo ' StrictHostKeyChecking no' >> /etc/ssh/ssh_config
#Disable cgo so that binaries we build will be fully static.
ENV CGO_ENABLED=0
#Recompile the standard library with cgo disabled. This prevents the standard library from being
#marked stale, causing full rebuilds every time.
RUN go install -v std
#Install glide
RUN go get github.com/Masterminds/glide
ENV GLIDE_HOME /home/user/.glide
#Install dep
RUN go get github.com/golang/dep/cmd/dep
#Install ginkgo CLI tool for running tests
RUN go get github.com/onsi/ginkgo/ginkgo
#Install linting tools.
RUN wget -O - -q https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s v1.20.0
RUN golangci-lint --version
#Install license checking tool.
RUN go get github.com/pmezard/licenses
#Install tool to merge coverage reports.
RUN go get github.com/wadey/gocovmerge
#Install CLI tool for working with yaml files
RUN go get github.com/mikefarah/yaml
#Delete all the Go sources that were downloaded, we only rely on the binaries
RUN rm -rf /go/src/*
#Install vgo (should be removed once we take Go 1.11)
RUN go get -u golang.org/x/vgo
#Ensure that everything under the GOPATH is writable by everyone
RUN chmod -R 777 $GOPATH
RUN curl -sSL https://github.com/estesp/manifest-tool/releases/download/${MANIFEST_TOOL_VERSION}/manifest-tool-linux-s390x > manifest-tool && \
chmod +x manifest-tool && \
mv manifest-tool /usr/bin/
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/entrypoint.sh"]
The build just hangs at RUN go install -v std
Illegal instruction (core dumped)
is thrown.Register x86_64 host with the latest qemu-user-static.
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker run -it -v /home/test/qemu-s390x-static:/usr/bin/qemu-s390x-static s390x/golang:1.14.2-alpine3.11
Inside s390x container:
apk update && apk add --no-cache su-exec curl bash git openssh mercurial make wget util-linux tini file grep sed shadow
apk upgrade --no-cache
#Disable ssh host key checking
echo 'Host *' >> /etc/ssh/ssh_config
echo ' StrictHostKeyChecking no' >> /etc/ssh/ssh_config
#Disable cgo so that binaries we build will be fully static.
export CGO_ENABLED=0
#Recompile the standard library with cgo disabled. This prevents the standard library from being
#marked stale, causing full rebuilds every time.
go install -v std
Describe the results you received:
Step 2 just halts while running step RUN go install -v std
while building Dockerfile.s390x.
Step 3 gives the following error:
Illegal instruction (core dumped)
Describe the results you expected:
The command should have been successful without any output.
Environment:
x86_64 Ub18.04 4.15.0-101-generic Ubuntu SMP x86_64 GNU/Linux
Output of docker version
, podman version
or singularity version
Client: Docker Engine - Community
Version: 19.03.11
API version: 1.40
Go version: go1.13.10
Git commit: 42e35e61f3
Built: Mon Jun 1 09:12:22 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.11
API version: 1.40 (minimum version 1.12)
Go version: go1.13.10
Git commit: 42e35e61f3
Built: Mon Jun 1 09:10:54 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Additional information optionally:
When I build the same Dockerfile.s390x on an s390x machine, it is built successfully.
When trying to run mips
/mipsel
images, docker can't start the container.
Matt$ docker run -it --rm multiarch/debian-debootstrap:mipsel-jessie /bin/uname -a
panic: standard_init_linux.go:175: exec user process caused "no such file or directory" [recovered]
panic: standard_init_linux.go:175: exec user process caused "no such file or directory"
goroutine 1 [running, locked to thread]:
panic(0x88f8a0, 0xc820152d90)
/usr/local/go/src/runtime/panic.go:481 +0x3e6
After a while thinking about an error with the images, I finally found that the no such file or directory
was related to the qemu
binary.
We can check that the expected one is present in the image:
Matt$ docker run -it --rm multiarch/debian-debootstrap:mipsel-jessie /usr/bin/qemu-mipsel-static /bin/uname -a
Linux b9de7767a6d1 4.4.39-moby #1 SMP Fri Dec 16 07:34:12 UTC 2016 mips GNU/Linux
What happens in fact is that when running docker run --privileged --rm multiarch/qemu-user-static:register
, 2 qemu binaries get registered for the mips
magic/mask.The same goes for mipsel
.
I moved a bit the lines from register.sh
to emphasize this:
echo ':mips:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:'${QEMU_BIN_DIR}'/qemu-mips-static:' > /proc/sys/fs/binfmt_misc/register
echo ':mipsn32:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:'${QEMU_BIN_DIR}'/qemu-mipsn32-static:' > /proc/sys/fs/binfmt_misc/register
echo ':mipsel:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:'${QEMU_BIN_DIR}'/qemu-mipsel-static:' > /proc/sys/fs/binfmt_misc/register
echo ':mipsn32el:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:'${QEMU_BIN_DIR}'/qemu-mipsn32el-static:' > /proc/sys/fs/binfmt_misc/register
What happens when starting a mips
image is that qemu-mipsn32-static
is not found.
I worked around that issue (c.f. libjpeg-turbo/libjpeg-turbo#106) by creating a new image that adds another copy of qemu-mipsel-static
with the name qemu-mipsn32el-static
.
I don't know how this should/could be fixed (well, other than removing/commenting the n32
variants). Adding the real copy of qemu-mipsn32el-static
to the image leads to qemu
exiting with an error.
Hello @lafin,
When running my travis job with qemu version 2.12.0 (arm) the following error occured, while running with 2.9.1-1 it succeeds.
Step 19/23 : RUN npm install
---> Running in 0a4abea8c736
#
# Fatal error in , line 0
# Missing deoptimization information for OptimizedFrame::Summarize.
#
qemu: uncaught target signal 5 (Trace/breakpoint trap) - core dumped
Trace/breakpoint trap (core dumped)
The command '/bin/sh -c npm install' returned a non-zero code: 133
Can you please have a look?
Thanks in advance,
Ray
I want to add big endian arch environment (ppc64
(not ppc64le
), s390x
) for this project.
But I have no idea how to add it.
Can we add the big endian arch docker container to the DockerHub [1]?
Thank you.
[1] https://hub.docker.com/r/multiarch/ubuntu-debootstrap/tags/
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind question
Description:
How can I register the qemu-static-i386?
Steps to reproduce the issue:
$ docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
Describe the results you received:
List of different architectures that works perfectly
Describe the results you expected:
The i386 arch
Environment:
Output of docker version
, podman version
or singularity version
docker version ✔ 10066 17:36:04
Client: Docker Engine - Community
Version: 19.03.1
API version: 1.40
Go version: go1.12.5
Git commit: 74b1e89e8a
Built: Thu Jul 25 21:17:37 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.1
API version: 1.40 (minimum version 1.12)
Go version: go1.12.5
Git commit: 74b1e89e8a
Built: Thu Jul 25 21:27:55 2019
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: v1.2.7.m
GitCommit: 85f6aa58b8a3170aec9824568f7a31832878b603.m
runc:
Version: 1.0.0-rc8
GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f
docker-init:
Version: 0.18.0
GitCommit: fec3683
Additional information optionally:
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind enhancement
Description:
There are tickets that has an error because the executed host architecture is not x86_64. Right now only x86_64 are supported. Because we only use qemu-user-static
RPM x86_64 package in .travis.yml
.
https://github.com/multiarch/qemu-user-static/blob/master/.travis.yml#L25
- PACKAGE_URI="https://kojipkgs.fedoraproject.org/packages/qemu/4.0.0/5.fc31/x86_64/qemu-user-static-4.0.0-5.fc31.x86_64.rpm"
To support other architectures, we need to add new piple line to .travis.yml
, downloading the arch's qemu-user-static as input.
I think a similar project might support ARM architectures as a host architecture. I am not sure.
https://github.com/dbhi/qus
Additional information optionally:
I'm trying to run the following code:
package main
import "fmt"
func main() {
fmt.Println("Hello, world!")
}
# GOARCH=arm go build ./hello.go
# qemu-arm-static ./hello
Which works with v2.7 and older.
/tmp/qemu-arm-static2.7 ./hello
Hello, world!
However, on v2.8 and newer I'm seeing
/tmp/qemu-arm-static2.8.0 ./hello
runtime: failed to create new OS thread (have 2 already; errno=22)
fatal error: newosproc
runtime stack:
runtime.throw(0xf6458, 0x9)
/usr/lib/golang/src/runtime/panic.go:547 +0x78
runtime.newosproc(0x10328000, 0x10337fe0)
/usr/lib/golang/src/runtime/os1_linux.go:149 +0x180
runtime.newm(0x1103fc, 0x0)
/usr/lib/golang/src/runtime/proc.go:1516 +0x12c
runtime.main.func1()
/usr/lib/golang/src/runtime/proc.go:125 +0x24
runtime.systemstack(0x155e00)
/usr/lib/golang/src/runtime/asm_arm.s:247 +0x80
runtime.mstart()
/usr/lib/golang/src/runtime/proc.go:1051
goroutine 1 [running]:
runtime.systemstack_switch()
/usr/lib/golang/src/runtime/asm_arm.s:192 +0x4 fp=0x103247ac sp=0x103247a8
runtime.main()
/usr/lib/golang/src/runtime/proc.go:126 +0x5c fp=0x103247d4 sp=0x103247ac
runtime.goexit()
/usr/lib/golang/src/runtime/asm_arm.s:990 +0x4 fp=0x103247d4 sp=0x103247d4
I'm running on go version
go version go1.6.3 linux/amd64
on centos 7.3.1611
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind bug
Description:
Steps to reproduce the issue:
Run docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
Observe error:
Unable to find image 'multiarch/qemu-user-static:latest' locally
latest: Pulling from multiarch/qemu-user-static
bdbbaa22dec6: Pull complete
42399a41a764: Pull complete
ed8a5179ae11: Pull complete
1ec39da9c97d: Pull complete
df7dd9470aac: Pull complete
Digest: sha256:25d6e8bb037094525cd70da43edc06a62122028cb9ad434605affbd4fffb3a4f
Status: Downloaded newer image for multiarch/qemu-user-static:latest
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
Setting /usr/bin/qemu-sparc-static as binfmt interpreter for sparc
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-sparc64-static as binfmt interpreter for sparc64
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
ERRO[0024] Error waiting for container: container 8daa8a39dacdae857a57bbebe300c5871efe08626931741205b6e50a0a83dbd4: driver "zfs" failed to remove root filesystem: exit status 1: "/usr/bin/zfs fs destroy -r nerv/ROOT/arch/caeeb050ad4ee9a4c3c9b00e69472f75f09287ac95264dca8863801842880e84" => cannot destroy 'nerv/ROOT/arch/caeeb050ad4ee9a4c3c9b00e69472f75f09287ac95264dca8863801842880e84': dataset is busy
Describe the results you received:
Above error is displayed when attempting to clean up the container.
Describe the results you expected:
Clean container removal and no error upon running the command.
Environment:
Output of docker version
, podman version
or singularity version
Client:
Version: 19.03.5-ce
API version: 1.40
Go version: go1.13.4
Git commit: 633a0ea838
Built: Fri Nov 15 03:19:09 2019
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 19.03.5-ce
API version: 1.40 (minimum version 1.12)
Go version: go1.13.4
Git commit: 633a0ea838
Built: Fri Nov 15 03:17:51 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.3.2.m
GitCommit: d50db0a42053864a270f648048f9a8b4f24eced3.m
runc:
Version: 1.0.0-rc9
GitCommit: d736ef14f0288d6993a1845745d6756cfc9ddd5a
docker-init:
Version: 0.18.0
GitCommit: fec3683
Additional information optionally:
I have attempted this on multiple machines with a zfs storage driver and they all appear to be experiencing the same issue, I've even attempted older versions of QEMU as far back as v4.1.0-1 and can reproduce this issue.
Hi,
I am trying run qemu-user-static:register on arm based machine (aarch64) and I am getting the below error please take a look
root@arm-ucpe-test:# uname -a16.04.1-Ubuntu SMP Tue Oct 10 16:33:57 UTC 2017 aarch64 aarch64 aarch64 GNU/Linux
Linux arm-ucpe-test 4.10.0-38-generic #42
root@arm-ucpe-test:~# docker run --rm --privileged multiarch/qemu-user-static:register
standard_init_linux.go:178: exec user process caused "no such file or directory"
So is this tested on aarch64?
Please let me know if you need more information.
Thanks,
Anand
docker image pull multiarch/qemu-aarch64-static
Error response from daemon: pull access denied for multiarch/qemu-aarch64-static, repository does not exist or may require 'docker login'
Does I need to set some configuration?
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind bug
Description:
Not able to install docker inside s390x container under qemu on amd64
Steps to reproduce the issue:
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker run --it --privileged s390x/ubuntu:18.04
apt-get update
apt-get install -y
apt-transport-https
ca-certificates
curl
gnupg-agent
software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
add-apt-repository
"deb [arch=s390x] https://download.docker.com/linux/ubuntu
$(lsb_release -cs)
stable"
apt-get update
apt-get install -y --no-install-recommends docker-ce=18.06.*
Describe the results you received:
These errors are seen while installing docker
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of start.
docker does not come up after running service docker start
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Describe the results you expected:
Docker is expected to come up after running service docker start
Environment:
Output of docker version
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:25:46 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b7f0
Built: Wed Mar 11 01:24:19 2020
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
Additional information optionally:
I tried the following fixes before installing docker inside the s390x container but they didn't work.
printf '#!/bin/sh\nexit 0' > /usr/sbin/policy-rc.d
export RUNLEVEL=1
/kind bug
Description:
Steps to reproduce the issue:
as the title, when I runs Armv7 Container
root@a4cea7e5c656:/# tunctl
Unsupported ioctl: cmd=0x400454ca
TUNSETIFF: Function not implemented
Is the architecture not yet supported ?
or I need to add ioctl mapping manually?
Environment:
linux version : Linux 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:20:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux (ubuntu 18.04)
qemu version : lastest
docker image : armv7/armhf-ubuntu tag: 16.04
docker version :
Client: Docker Engine - Community
Version: 19.03.5
API version: 1.40
Go version: go1.12.12
Git commit: 633a0ea838
Built: Wed Nov 13 07:29:52 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.12.12
Git commit: 633a0ea838
Built: Wed Nov 13 07:28:22 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
Hello,
I was wondering if the maintainers of this project would consider releasing updates for qemu 2.10 and/or 2.11? The support for various powerPC architectures is improved.
Thanks!
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind bug
Description:
When I was running the pipeline for old qemu versions, then when I tried git pull --tags
, the command showed the error "would clobber existing tag"
Steps to reproduce the issue:
I did qemu v4.0.0-4 that is new release. But the current latest released version is v4.0.0-5. #87
$ git pull --tags
Describe the results you received:
$ git pull --tags
From github.com:multiarch/qemu-user-static
! [rejected] v4.0.0-4 -> v4.0.0-4 (would clobber existing tag)
Describe the results you expected:
The pull is success.
Environment:
$ git --version
git version 2.21.0
$ git remote -v
jaruga [email protected]:junaruga/qemu-user-static.git (fetch)
jaruga [email protected]:junaruga/qemu-user-static.git (push)
meeDamian [email protected]:meeDamian/qemu-user-static.git (fetch)
meeDamian [email protected]:meeDamian/qemu-user-static.git (push)
origin [email protected]:multiarch/qemu-user-static.git (fetch)
origin [email protected]:multiarch/qemu-user-static.git (push)
Docker
But the issue is not related to my local environment's docker command.
Output of docker version
, podman version
or singularity version
$ docker version
Client: Docker Engine - Community
Version: 19.03.0
API version: 1.40
Go version: go1.12.5
Git commit: aeac9490dc
Built: Wed Jul 17 18:16:02 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.0
API version: 1.40 (minimum version 1.12)
Go version: go1.12.5
Git commit: aeac9490dc
Built: Wed Jul 17 18:14:40 2019
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.6
GitCommit: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc:
Version: 1.0.0-rc8
GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f
docker-init:
Version: 0.18.0
GitCommit: fec3683
Additional information optionally:
on x86 , I am trying to cross compile for s390x Alpine. The apk
command fails while fetch IO errors.
Note: Alpine image works perfectly well on an s390x host.
**Issue: (Below commands executed on x86 host) **
Dockerfile:
FROM s390x/alpine
COPY qemu-s390x-static /usr/bin/
CMD uname -a
uname -a
Linux dk-vm 4.4.0-134-generic #160-Ubuntu SMP Wed Aug 15 14:58:00 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
docker build -t test-alpine -f Dockerfile-test .
Sending build context to Docker daemon 23.27MB
Step 1/3 : FROM s390x/alpine
latest: Pulling from s390x/alpine
cdf21ace9418: Already exists
ea178433f2f0: Already exists
Digest: sha256:903e1eeceeb7208949673dcb557659b5f4c8e9065a45432a0668acdd0069268c
Status: Downloaded newer image for s390x/alpine:latest
---> 2fc01d06b47d
Step 2/3 : COPY qemu-s390x-static /usr/bin/
---> 0dfcc829de99
Step 3/3 : CMD uname -a
---> Running in 1034e895153a
Removing intermediate container 1034e895153a
---> e8d1f3431d7a
Successfully built e8d1f3431d7a
Successfully tagged test-alpine:latest
root@dk-vm:~/jenkins/docker# docker run -it --name alpine test-alpine sh
/ # clear
/ # apk add --no-cache git openssh-client curl unzip bash ttf-dejavu coreutils tini
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/s390x/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/main/s390x/APKINDEX.tar.gz: IO ERROR
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/s390x/APKINDEX.tar.gz
WARNING: Ignoring http://dl-cdn.alpinelinux.org/alpine/v3.8/community/s390x/APKINDEX.tar.gz: IO ERROR
ERROR: unsatisfiable constraints:
bash (missing):
required by: world[bash]
coreutils (missing):
required by: world[coreutils]
curl (missing):
required by: world[curl]
git (missing):
required by: world[git]
openssh-client (missing):
required by: world[openssh-client]
tini (missing):
required by: world[tini]
ttf-dejavu (missing):
required by: world[ttf-dejavu]
unzip (missing):
required by: world[unzip]
musl-1.1.19-r10:
breaks: musl-utils-1.1.19-r10[musl?1.1.19-r10] musl-utils-1.1.19-r10[so:libc.musl-s390x.so.1] busybox-1.28.4-r0[so:libc.musl-s390x.so.1]
libressl2.7-libcrypto-2.7.4-r0[so:libc.musl-s390x.so.1] apk-tools-2.10.0-r0[so:libc.musl-s390x.so.1]
libressl2.7-libssl-2.7.4-r0[so:libc.musl-s390x.so.1] scanelf-1.2.3-r0[so:libc.musl-s390x.so.1]
zlib-1.2.11-r1[so:libc.musl-s390x.so.1] alpine-baselayout-3.1.0-r0[so:libc.musl-s390x.so.1]
busybox-1.28.4-r0:
breaks: world[busybox] alpine-baselayout-3.1.0-r0[/bin/sh]
alpine-baselayout-3.1.0-r0:
breaks: world[alpine-baselayout]
alpine-keys-2.1-r1:
breaks: world[alpine-keys]
libressl2.7-libcrypto-2.7.4-r0:
breaks: apk-tools-2.10.0-r0[so:libcrypto.so.43] libressl2.7-libssl-2.7.4-r0[so:libcrypto.so.43]
libressl2.7-libssl-2.7.4-r0:
breaks: apk-tools-2.10.0-r0[so:libssl.so.45]
zlib-1.2.11-r1:
breaks: apk-tools-2.10.0-r0[so:libz.so.1]
apk-tools-2.10.0-r0:
breaks: world[apk-tools]
scanelf-1.2.3-r0:
breaks: musl-utils-1.1.19-r10[scanelf]
musl-utils-1.1.19-r10:
breaks: libc-utils-0.7.1-r0[musl-utils]
libc-utils-0.7.1-r0:
breaks: world[libc-utils]
/ # exit
Hi there. For some time now I have had a couple of errors pop up when running with qemu-arm-static inside an armv7h chroot for crosscompiling armv7h software.
Most notable and annoying ones are:
QT
I get missing modules (these vary depending on project) when compiling QT applications:
Project ERROR: Unknown module(s) in QT: core gui widgets
Project ERROR: Unknown module(s) in QT: core
The exact same procedure, with qemu-aarch64-static in an aarch64 chroot works fine.
It also builds fine on an armv7h device without qemu at all.
GTK
I get unrecognised image formats when compiling some GTK apps. Like this one:
[3/71] Generating manager_resources_h with a custom command.
FAILED: resources/manager_resources.h
glib-compile-resources ../resources/pamac.manager.gresource.xml --sourcedir ../resources --internal --generate --target resources/manager_resources.h
failed to load "../resources/package-available.png": Couldn?t recognize the image file format for file ?../resources/package-available.png?
../resources/pamac.manager.gresource.xml: Child process exited with code 1.
[4/71] Generating manager_resources_c with a custom command.
FAILED: resources/manager_resources.c
glib-compile-resources ../resources/pamac.manager.gresource.xml --sourcedir ../resources --internal --generate --target resources/manager_resources.c --dependency-file resources/manager_resources.c.d
failed to load "../resources/package-available.png": Couldn?t recognize the image file format for file ?../resources/package-available.png?
../resources/pamac.manager.gresource.xml: Child process exited with code 1.
[11/71] Compiling Vala source ../src/common_daemon.vala ../src/package.vala ../src/pamac_config.vala ../src/alpm_config.vala ../src/aur.vala ../src/system_daemon.vala.
ninja: build stopped: subcommand failed.
Again, it builds fine with aarch64 instead and on the armv7h device directly.
I am at a loss here, I even tried different versions of the qemu-arm-static file, but they all had the same issues.
I'm using qemu 2.12, qemu-user-static 2.12 and systemd 239 (for systemd-nspawn).
QEMU 2.8 is available, could you please update the images?
See http://ftp.fr.debian.org/debian/pool/main/q/qemu/qemu-user-static_2.8+dfsg-1_amd64.deb
Hi and thanks for the awesome work!
I'm trying to build an image FROM ppc64le/debian:jessie
that only installs iptables.
COPY qemu-ppc64le-static /usr/bin/
is there but docker run --rm --privileged multiarch/qemu-user-static:register --reset
doesn't register ppc64le
, only ppc which obviously doesn't work.
Is there any issues with registering ppc64le
or could we do it?
I'm trying to understand how to build docker images on x86 cups for different architectures like ARM.
I tried the following command on my macbook:
sudo docker run --rm --privileged multiarch/qemu-user-static:register
and one of the output is:
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
sh: write error: File exists
It looks like the generated file already exists. My problem is that I can't find qemu-arm-static
in /usr/bin/
.
So, where was it saved?
Thanks
On a fresh C2M with docker run --rm --privileged multiarch/qemu-user-static:register
done I get this error /build/qemu-Ee59aw/qemu-2.0.0+dfsg/tcg/optimize.c:362: tcg fatal error
(qemu bug for sure) but it seems the version is 2.0.0 and not 2.5.0
I'm trying to build/test a Docker image on travis. The build fails on an ARM alpine image:
The command "docker run --rm --privileged multiarch/qemu-user-static:register" exited with 0.
12.37s$ docker run --rm -v $PWD:/usr/src/myapp -w /usr/src/myapp -v go:/go ${GOIMG} bash -c "cd /usr/src/myapp && go get -d -v -t && go test --cover -v ./... --run UnitTest && go build -v -o docker-flow-proxy"
Unable to find image 'kutsudock/rpi-alpine-go:latest' locally
latest: Pulling from kutsudock/rpi-alpine-go
Status: Downloaded newer image for kutsudock/rpi-alpine-go:latest
standard_init_linux.go:175: exec user process caused "no such file or directory"
I suspect it's something that's missing from the alpine image. Any ideas how I can get this running?
Environment: multicore ARMv8 machine running Ubuntu 16.04.
Trying to follow directions, and I get this:
emv@armv8hello:~$ docker run --rm --privileged multiarch/qemu-user-static:register
Unable to find image 'multiarch/qemu-user-static:register' locally
register: Pulling from multiarch/qemu-user-static
8ddc19f16526: Already exists
5ac0057b3331: Pull complete
Digest: sha256:ee56b87e8debafb7c6ed2a2d2b7c2b5f9d471e4bee0cb84956f97d0316f4bc59
Status: Downloaded newer image for multiarch/qemu-user-static:register
write pipe: bad file descriptor
I have a feeling I'm missing something, perhaps a longer README about getting started.
thanks!
Ed
/kind bug
[root@localhost ~]# file mips
mips: ELF 64-bit LSB executable, MIPS, MIPS-III version 1 (SYSV), statically linked, not stripped
[root@localhost ~]#
[root@localhost ~]# ./mips
Hello, MIPS!
[root@localhost ~]#
[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# docker run --rm -it --net=host --privileged -v /usr/bin/qemu-mips64el-static:/usr/bin/qemu-mips64el-static rhel_mips:7.4 date
qemu: uncaught target signal 4 (Illegal instruction) - core dumped
[root@localhost ~]# docker version
Client:
Version: 1.12.6
API version: 1.24
Go version: go1.6.4
Git commit: 78d1802
Built: Tue Jan 10 20:20:01 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Go version: go1.6.4
Git commit: 78d1802
Built: Tue Jan 10 20:20:01 2017
OS/Arch: linux/amd64
same error info on Docker CE 17.12
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind enhancement
Description:
Related to #15 .
I want to add examples to use mips and riscv64 images to README.md.
Are there some official images for those architectures, such as arm64v8/ubuntu
for aarch64?
Thanks.
Is this a bug report, feature (enhancement) request or question? (leave only one on its own line)
/kind bug
Description:
I'm trying to create dockerfile for base image to be built by CI (drone) for multiple architectures. I want to have a single dockerfile so I'm using ARGs to pass arch value into dockerfile.
Steps to reproduce the issue:
test.dockerfile
ARG arch
ARG qemu
FROM multiarch/qemu-user-static:x86_64-${qemu} as qemu
FROM ${arch}/mono
ARG qemu
COPY --from=qemu /usr/bin/qemu-${qemu}-static /usr/bin
docker build -f test.dockerfile --build-arg arch=amd64 --build-arg qemu=x86_64 -t test:amd64 .
Describe the results you received:
Step 6/6 : COPY --from=qemu /usr/bin/qemu-${qemu}-static /usr/bin
COPY failed: Forbidden path outside the build context: usr/bin/qemu-x86_64-static ()
Describe the results you expected:
qemu-x86_64-static
is being copied to /usr/bin
If I run docker build -f test.dockerfile --build-arg arch=arm32v7 --build-arg qemu=arm -t test:amd64 .
everything works fine.
Environment:
Output of docker version
, podman version
or singularity version
Client: Docker Engine - Community
Version: 19.03.5
API version: 1.40
Go version: go1.12.12
Git commit: 633a0ea
Built: Wed Nov 13 07:22:34 2019
OS/Arch: darwin/amd64
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 19.03.5
API version: 1.40 (minimum version 1.12)
Go version: go1.12.12
Git commit: 633a0ea
Built: Wed Nov 13 07:29:19 2019
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: v1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
Additional information optionally:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.