OpenMP issue in Windows (doesn't like `int64_t` in for loop initialization) about graphblas

[eriknw]

btw, we also tried to build on OS X with cpu_features enabled. It failed during linking:

[100%] Linking C shared library libgraphblas.dylib
$BUILD_PREFIX/bin/cmake -E cmake_link_script CMakeFiles/graphblas.dir/link.txt --verbose=1
$BUILD_PREFIX/bin/x86_64-apple-darwin13.4.0-clang -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/graphblas-6.1.4 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -Wno-pointer-sign  -O3 -DNDEBUG   -fopenmp=libomp  -O3 -DNDEBUG -isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk -mmacosx-version-min=10.9 -dynamiclib -Wl,-headerpad_max_install_names -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,$PREFIX/lib -L$PREFIX/lib -compatibility_version 6.0.0 -current_version 6.1.4 -o libgraphblas.6.1.4.dylib -install_name @rpath/libgraphblas.6.dylib @CMakeFiles/graphblas.dir/objects1.rsp @CMakeFiles/graphblas.dir/objects2.rsp  -lm $PREFIX/lib/libomp.dylib 
ld: warning: -pie being ignored. It is only used when linking a main executable
Undefined symbols for architecture x86_64:
  "_GetX86Info", referenced from:
      _GB_Global_cpu_features_query in GB_Global.c.o
ld: symbol(s) not found for architecture x86_64
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)
make[3]: *** [CMakeFiles/graphblas.dir/build.make:42948: libgraphblas.6.1.4.dylib] Error 1
make[3]: Leaving directory '$SRC_DIR/build'
make[2]: *** [CMakeFiles/Makefile2:83: CMakeFiles/graphblas.dir/all] Error 2
make[2]: Leaving directory '$SRC_DIR/build'
make[1]: *** [Makefile:136: all] Error 2
make[1]: Leaving directory '$SRC_DIR/build'
make: *** [Makefile:31: library] Error 2

as seen here: https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=436931&view=logs&j=58ac6aab-c4bc-5de2-2894-98e408cc8ec9&t=933f325c-924e-533d-4d95-e93b5843ce8b&l=5603

Building with cpu_features disabled on OS X works for us.

[jim22k]

More to add to the Windows failure. Just to make sure, I explicitly set GBNCPUFEAT=0, GBAVX2=0, and GBAVX512F=0.
I'm fairly certain the cmake parameters are being read correctly. I see these line in the logs:

-- cpu_features (by google.com): disabled
-- CMAKE C flags: /DWIN32 /D_WINDOWS /W3 -DGBNCPUFEAT=1  -DGBAVX2=0  -DGBAVX512F=0  /O2 -wd"4244" -wd"4146" -wd"4018" -wd"4996" -wd"4047" -wd"4554" /MD /O2 /Ob2 /DNDEBUG   -openmp 

But we're still getting the failures that Erik pasted above.

And even crazier is that all the auto-generated files above this one compile just fine. But we consistently error out on GB_AxB__plus_times_fp32.c. It makes no sense.

[DrTimothyAldenDavis]

Oops. I took out the "for (int64_t k ...". MS Visual Studio is so brain-dead; they only support OpenMP 2.0. I have to do that throughout the code, all the time, but missed this.

For the GBNCPUFEAT, I just check if it is defined,. I don't check its value, the same way -DNDEBUG is used to control assert statements. So -DGBNCPUFEAT=0 is the same as -GBNCPUFEAT=1, and both say "do not use cpu_features".

There are only 2 files that make use of the check for AVX2 and AVX512F: GB_AxB_plus_times_fp32.c and GB_AxB__plus_times_fp64.c.

[DrTimothyAldenDavis]

I'm unsure why GetX86Info doesn't show up. I can compile it just fine on my Mac OSX system. Maybe it's a clang issue. I've used icc and gcc-11.2.0 on my Mac, but not clang since it breaks my interface for Octave and MATLAB. I think it has something to do with different OpenMP libraries used in the library as compared to the ones in MATLAB.

[DrTimothyAldenDavis]

My user guide describes the GBNCPUFEAT option as "-DGNBNCPUFEAT=1" since you can't pass just "-DGBNCPUFEAT" to cmake as a cmake option.

[DrTimothyAldenDavis]

I'm able to compile GraphBLAS on the Mac with clang 13.0.0 (MacOS Monterey), with cpu_features enabled. I added a printf in GB_Global.c just after calling GetX86Info, and it works and returns the right values. "nm libgraphblas.dylib" shows that GetX86Info is present inside the dylib. I'm not sure why it doesn't work for you. I can't replicate the failure you're seeing.

[eriknw]

Alright, here's our current status when building SuiteSparse:GraphBLAS on conda-forge as a shared library:

Linux

• Works with google/cpu_features!

OS X (x86_64)

• Does not work with google/cpu_features enables (error in previous message). I haven't reproduced this locally either.
• Works with google/cpu_features disabled via -DGBNCPUFEAT=1

Windows (with OpenMP fix)

• Does not work with google/cpu_features
• Does not work with google/cpu_features disabled via -DGBNCPUFEAT=1
• Works with google/cpu_features disabled via -DGBNCPUFEAT=1 and applying the following patch

diff --git a/Source/GB_compiler.h b/Source/GB_compiler.h
index 560fc93947..27207f3d12 100644
--- a/Source/GB_compiler.h
+++ b/Source/GB_compiler.h
@@ -261,7 +261,7 @@
             #define GB_COMPILER_SUPPORTS_AVX2 0
         #endif
     #elif GB_COMPILER_ICX || GB_COMPILER_ICC || GB_COMPILER_CLANG || \
-          GB_COMPILER_GCC || GB_COMPILER_MSC
+          GB_COMPILER_GCC
         // all these compilers can handle AVX512F and AVX2 on x86
         #define GB_COMPILER_SUPPORTS_AVX512F 1
         #define GB_COMPILER_SUPPORTS_AVX2 1

Let's dive into why we tried this patch.

As shown above, the error we were seeing is this:

[ 54%] Building C object CMakeFiles/graphblas.dir/Source/Generated2/GB_AxB__plus_times_fp32.c.obj
GB_AxB__plus_times_fp32.c
%SRC_DIR%\Source\Generated2\GB_AxB__plus_times_fp32.c(447): error C2485: 'target': unrecognized extended attribute
%SRC_DIR%\Source\Generated2\GB_AxB__plus_times_fp32.c(447): error C2059: syntax error: ')'
%SRC_DIR%\Source\Generated2\GB_AxB__plus_times_fp32.c(447): error C2059: syntax error: 'type'
%SRC_DIR%\Source\Generated2\GB_AxB__plus_times_fp32.c(485): error C2485: 'target': unrecognized extended attribute
%SRC_DIR%\Source\Generated2\GB_AxB__plus_times_fp32.c(485): error C2059: syntax error: ')'
%SRC_DIR%\Source\Generated2\GB_AxB__plus_times_fp32.c(485): error C2059: syntax error: 'type'
%SRC_DIR%\Source\Template\GB_AxB_saxpy5_meta.c(118): warning C4013: 'GB_AxB_saxpy5_unrolled_avx512f' undefined; assuming extern returning int
%SRC_DIR%\Source\Template\GB_AxB_saxpy5_meta.c(129): warning C4013: 'GB_AxB_saxpy5_unrolled_avx2' undefined; assuming extern returning int
NMAKE : fatal error U1077: 'D:\bld\graphblas_1641583385327\_build_env\Library\bin\cmake.exe' : return code '0x2'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\nmake.exe"' : return code '0x2'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64\nmake.exe"' : return code '0x2'
Stop.

It's complaining about these lines:

GraphBLAS/Source/Generated2/GB_AxB__plus_times_fp32.c

Lines 444 to 447 in 1b97623

	#if GB_SEMIRING_HAS_AVX_IMPLEMENTATION && GB_COMPILER_SUPPORTS_AVX512F \
	&& GB_V4_512
	GB_TARGET_AVX512F static inline void GB_AxB_saxpy5_unrolled_avx512f

So, even though we disabled google/cpu_features, AVX2, and AVX512F, the pre-compiler condition #if GB_SEMIRING_HAS_AVX_IMPLEMENTATION && GB_COMPILER_SUPPORTS_AVX512F && GB_V4_512 is True. Apparently, our Windows compiler doesn't like compiling the code that follows this condition.

Our patch above ensures that GB_COMPILER_SUPPORTS_AVX512F and GB_COMPILER_SUPPORTS_AVX2 are False when we compile on Windows. Perhaps a better approach would be to set these to False when GBNCPUFEAT is defined and e.g. GBAVX2/GBAVX512F are not defined or are set to False.

For now, I don't feel compelled to get google/cpu_features working on OS X or Windows, which means v6.1.4beta1 is "good enough for us" (which means AVX will only be enabled for x86_64 Linux). We can apply the patch to make it work on Windows. Of course, you're welcome to update your code so our patch is no longer needed.

p.s., yes, we do have Windows users!

[DrTimothyAldenDavis]

Thanks for the update. I got the Windows version here: https://www.intel.com/content/www/us/en/develop/documentation/cpp-compiler-developer-guide-and-reference/top/compiler-reference/attributes/target.html and I had assumed that would work on any compiler on Windows. I guess not.

For Windows with MS Visual Studio, I will add an #ifdef internally so that GBNCPUFEAT is #defined as 1, regardless of the setting made from cmake. For the Intel compilers on Windows, I'll use the __declspec(target("avx512f")) as describe in the url.

[DrTimothyAldenDavis]

On the Mac, I would like to keep the code as-is, since it works for me at least on clang and icc on my Mac (OSX Monterey). Maybe it is this part, trying to compile for older OS:

-isysroot /Applications/Xcode_12.4.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk -mmacosx-version-min=10.9

If you can simply set -DGBNCPUFEAT=1 on your copy on the Mac, that would work for you, and also for my compilation and perhaps others on the latest MacOS.

[DrTimothyAldenDavis]

Can you give the v6.1.4.beta2 tag a try on Windows?

[eriknw]

Thanks. Trying now.

[eriknw]

Everything builds for us for v6.1.4.beta2! 🚀

We only enable google/cpu_features and AVX support for Linux. Perhaps we can revisit other OSes in the future when compilers and/or build environments are upgraded (or when somebody asks).

As always, thanks for your careful attention to detail and quick turnaround 😄

[DrTimothyAldenDavis]

Sounds great. I will polish it up and post the v6.1.4 stable shortly.