Building in Anaconda3 environment about xprec

[shinaoka]

Debug code for setup.py

        print(self.compiler.compiler_so)

        if platform.system() == 'unix':
            new_flags = {"-march": "native", "-mtune": "native",
                         "-Wextra": None, "-std": "c11"}

Contents of compiler_so:

    ['gcc', '-pthread', '-B', '/opt/conda/compiler_compat', '-Wno-unused-result', '-Wsign-compare', '-DNDEBUG', '-O2', '-Wall', '-fPIC', '-O2', '-n1', '.2-a+fp16+rcpc+dotprod+crypto', '-isystem', '/opt/conda/include', '-I/opt/conda/include', '-fPIC', '-O2', '-n1', '.2-a+fp16+rcpc+dotprod+crypto', '-isystem', '/opt/conda/include', '-fPIC']

[mwallerb]

Something seems to go horribly wrong here with self.compiler.compiler_so before we even touch that variable:
The option string does not make any sense at all ... why would we pass -n to gcc, and then the linker, thereby disabling linking to shared libraries, when we are building a shared library? But even if it did, -n does not take any argument, so what is that 1 doing there? Plus what is that: .2-a+fp16+rcpc+dotprod+crypto? We do not even load anything crypto-related ...

So this compiler invocation reads completely nonsensical to me ... can you build anything else with this?

[mwallerb]

I have a hunch: Can you check if anaconda passing -march=neoverse-n1.2-a+fp16+rcpc+dotprod+crypto or similar stuff into setuptools? You can check build flags etc. in python -m sysconfig ...

[shinaoka]

Yes, after removing -n1 .2-a+fp16+rcpc+dotprod+crypto , gcc stops complaining.
These flags may come from the compilation flags used to build python.

> python3 -m sysconfig|grep crypto
CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG  -O2 -Wall  -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include -I/opt/conda/include  -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include "        CONFIGURE_CFLAGS = " -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include -I/opt/conda/include"
CONFIG_ARGS = "'--prefix=/opt/conda' '--build=aarch64-conda_cos7-linux-gnu' '--host=aarch64-conda_cos7-linux-gnu''--enable-ipv6' '--with-ensurepip=no' '--with-tzpath=/opt/conda/share/zoneinfo:/opt/conda/share/tzinfo' '--with-computed-gotos''--with-system-ffi' '--enable-loadable-sqlite-extensions' '--with-tcltk-includes=-I/opt/conda/include' '--with-tcltk-libs=-L/opt/conda/lib -ltcl8.6 -ltk8.6''--with-platlibdir=lib' '--with-openssl=/opt/conda' 'PKG_CONFIG_LIBDIR=/opt/conda/lib' 'PKG_CONFIG_PATH=/opt/conda/lib' 'CPPFLAGS=-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/conda/include -I/opt/conda/include -I/opt/conda/include' 'CXXFLAGS=-fvisibility-inlines-hidden -std=c++17 -fmessage-length=0  -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include -I/opt/conda/include' 'CFLAGS= -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include -I/opt/conda/include' 'LDFLAGS=   -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib -L/opt/conda/lib' 'CC= gcc' 'CXX=g++''--with-lto' '--enable-optimizations' '-oldincludedir=/usr/include' '--disable-shared''PROFILE_TASK=-m test --pgo' 'build_alias=aarch64-conda_cos7-linux-gnu' 'host_alias=aarch64-conda_cos7-linux-gnu''MACHDEP=linux' 'CPP=/home/ec2-user/miniconda3/envs/prefect/conda-bld/python-split_1631809412910/_build_env/bin/cpp'"
OPENSSL_LIBS = "-lssl -lcrypto"
PY_BUILTIN_MODULE_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG  -O2 -Wall  -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include -I/opt/conda/include  -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include  -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I/home/ec2-user/miniconda3/envs/prefect/conda-bld/python-split_1631809412910/work/Include/internal -IObjects -IInclude -IPython -I. -I/home/ec2-user/miniconda3/envs/prefect/conda-bld/python-split_1631809412910/work/Include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/conda/include -I/opt/conda/include -I/opt/conda/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/conda/include -I/opt/conda/include -DPy_BUILD_CORE_BUILTIN"
PY_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG  -O2 -Wall  -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include -I/opt/conda/include  -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include "
PY_CORE_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG  -O2 -Wall  -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include -I/opt/conda/include  -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include  -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I/home/ec2-user/miniconda3/envs/prefect/conda-bld/python-split_1631809412910/work/Include/internal -IObjects -IInclude -IPython -I. -I/home/ec2-user/miniconda3/envs/prefect/conda-bld/python-split_1631809412910/work/Include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/conda/include -I/opt/conda/include -I/opt/conda/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/conda/include -I/opt/conda/include -DPy_BUILD_CORE"
PY_STDMODULE_CFLAGS = "-Wno-unused-result -Wsign-compare -DNDEBUG  -O2 -Wall  -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include -I/opt/conda/include  -fPIC   -O2  -n1 .2-a+fp16+rcpc+dotprod+crypto -isystem /opt/conda/include   -g -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I/home/ec2-user/miniconda3/envs/prefect/conda-bld/python-split_1631809412910/work/Include/internal -IObjects -IInclude -IPython -I. -I/home/ec2-user/miniconda3/envs/prefect/conda-bld/python-split_1631809412910/work/Include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/conda/include -I/opt/conda/include -I/opt/conda/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /opt/conda/include -I/opt/conda/include"

Full output:
sysconfig.txt

[mwallerb]

Not much we can do here ... if python reports to having been built with flags that it could not have been built since the compiler does not understand it, the config is simply broken. Possible workaround is to remove such flags, but I don't know how general this is...

[shinaoka]

A general way may be to distribute precompiled binary packages....

[mwallerb]

We can try, though it does have one downside: performance in double-double arithmetic relies heavily on a fast FMA instruction, which is not universally available. If we build binary packages, can we fine-tune it or do we have to take a performance hit?

[shinaoka]

Do you mean FMA3? Reasonably new x86 processors support FMA3, I think.
https://en.wikipedia.org/wiki/FMA_instruction_set
I do not know ARM64.

FYI:
https://github.com/marketplace/actions/build-and-publish-conda-packages-to-anaconda-org

[mwallerb]

Please have a go at this if you want ...

I have to say I hate setuptools so much that I don't want to touch it unless absolutely necessary :-D

[shinaoka]

I completely agree.

Anaconda has its own compilers bundled.
https://docs.conda.io/projects/conda-build/en/latest/resources/compiler-tools.html

[shinaoka]

Do you know any other libraries that have numpy C extensions and are available on conda?

[shinaoka]

The flags may be related to Graviton2, arm CPUs from Amazon.

[mwallerb]

Closed via #4