Include envoy in (SmartOS) pkgsrc about pkgsrc-joyent

[siepkes]

Woohoo! I've got a working Bazel binary on SmartOS!

I've created a gist with the patch (which applies to bazel-0.11.1-dist.zip). I'm now going to test if it actually builds something (like Envoy) to see if it needs tweaking of some sort. I've created the toolchain with GCC 7 even though I know pkgsrc uses GCC 4.9. But I need atleast GCC 5 to compile envoy. We should probably think about how to deal with these multiple toolchains in Bazel (higher GCC versions, clang, etc.). But first I'm going to focus on getting the thing to actually do something and be useful.

# ./bazel/output/bazel
Extracting Bazel installation...
...........................
                                              [bazel release 0.11.1- (@non-git)]
Usage: bazel <command> <options> ...

Available commands:
  analyze-profile     Analyzes build profile data.
  build               Builds the specified targets.
  canonicalize-flags  Canonicalizes a list of bazel options.
----8<----------

[mamash]

In that case I'd just patch out any attempts to link in -lgcc and the like. Unless Bazel also "helpfully" uses something like -nostdlib or -nodefaultlibs, the standard GCC libs like libgcc_s or libstdc++ should be linked in automatically.

The entire "-lgcc -lgcc_eh -lc -lgcc -lgcc_eh" part sounds fishy.

[mamash]

Well, bazel is the main problem really. It only works on sanctioned and explicitly listed platforms, and - not surprisingly - SunOS is not one of them.

There's probably no significant reason why it's so, but upstream is pretty clear on not wanting the job, and I'm not sure spending much time on fixing yet another crazy build system is really worth it. Remember, Bazel source tarball comes with close to 10.000 files.

[mamash]

I should add I'm interested in hearing how you ever got close to running bazel on SmartOS.

[siepkes]

I think I'm actually pretty close to getting Bazel to work. I've looked at FreeBSD and Darwin support (which were both shoe-horned in later on). For some reason Bazel still complains about not detecting the correct OS but I suspect thats just a minor error I made somewhere. Since I have reviewed FreeBSD and Darwin support and I don't see any additional changes they made that I haven't.

Got sidetracked by a bunch of other work but I hope to get back to it in a couple of days.

[mamash]

Cool, let me know if you need a second pair of eyes for anything.

[siepkes]

I definitely underestimated the complexity of Bazel. It seems a lot of functionality was never part of the original design and were more or less duct-tapped and shoehorned in later.

For example 90% the time Linux is called K8 (probably from Bazel's Kubernetes build tool roots) and other OS'es, like FreeBSD, are regarded as a "CPU". Oddly enough "arm" is the only real cpu regarded as a "CPU" by Bazel. I'm guessing thats actually Linux on ARM. Oddly enough for Windows there is an explicit x64 prefix in it's CPU name ("x64_windows") while FreeBSD x64 is just called "freebsd" (so what about FreeBSD arm support? Or ARM64?). Also Linux (K8) is kind of the default which is implied everywhere and all other things are then explicitly mentioned (Darwin, Windows, FreeBSD, etc.).

I got as far as this diff, unfortunely Bazel doesn't compile with it (by running compile.sh). The patch can be applied to bazel-0.11.1-dist.zip. I might take a look at it in the future but for now I'm going to let this rest.

--- bazel-orig/scripts/bootstrap/buildenv.sh    2018-04-01 14:52:36.150686900 +0000
+++ bazel/scripts/bootstrap/buildenv.sh 2018-04-01 14:53:02.524832025 +0000
@@ -83,6 +83,10 @@
   JAVA_HOME="${JAVA_HOME:-/usr/local/openjdk8}"
   ;;
 
+solaris)
+  JAVA_HOME="${JAVA_HOME:-/opt/local/openjdk8}"
+  ;;
+
 darwin)
   if [[ -z "$JAVA_HOME" ]]; then
     JAVA_HOME="$(/usr/libexec/java_home -v ${JAVA_VERSION}+ 2> /dev/null)" \
diff -uar bazel-orig/scripts/packages/BUILD bazel/scripts/packages/BUILD
--- bazel-orig/scripts/packages/BUILD   2018-04-01 14:52:36.152851396 +0000
+++ bazel/scripts/packages/BUILD        2018-04-01 14:53:02.525363493 +0000
@@ -20,6 +20,7 @@
             ":zip-bazel-exe_with_jdk",
         ],
         "//src/conditions:freebsd": [],
+        "//src/conditions:solaris": [],
         "//src/conditions:darwin": [
             ":with-jdk/install.sh",
             ":without-jdk/install.sh",
diff -uar bazel-orig/src/conditions/BUILD bazel/src/conditions/BUILD
--- bazel-orig/src/conditions/BUILD     2018-04-01 14:52:36.170223015 +0000
+++ bazel/src/conditions/BUILD  2018-04-01 14:53:02.525807667 +0000
@@ -41,6 +41,12 @@
 )
 
 config_setting(
+    name = "solaris",
+    values = {"cpu": "solaris"},
+    visibility = ["//visibility:public"],
+)
+
+config_setting(
     name = "windows",
     values = {"cpu": "x64_windows"},
     visibility = ["//visibility:public"],
diff -uar bazel-orig/src/conditions/BUILD.tools bazel/src/conditions/BUILD.tools
--- bazel-orig/src/conditions/BUILD.tools       2018-04-01 14:52:36.170282808 +0000
+++ bazel/src/conditions/BUILD.tools    2018-04-01 14:53:02.526128210 +0000
@@ -5,6 +5,12 @@
 )
 
 config_setting(
+    name = "solaris",
+    values = {"cpu": "solaris"},
+    visibility = ["//visibility:public"],
+)
+
+config_setting(
     name = "darwin",
     values = {"cpu": "darwin"},
     visibility = ["//visibility:public"],
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/analysis/config/AutoCpuConverter.java bazel/src/main/java/com/google/devtools/build/lib/analysis/config/AutoCpuConverter.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/analysis/config/AutoCpuConverter.java        2018-04-01 14:52:36.216707875 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/analysis/config/AutoCpuConverter.java     2018-04-01 14:53:02.526694805 +0000
@@ -37,6 +37,8 @@
           return "darwin";
         case FREEBSD:
           return "freebsd";
+        case SOLARIS:
+          return "solaris";
         case WINDOWS:
           switch (CPU.getCurrent()) {
             case X86_64:
@@ -79,6 +81,8 @@
       return Pair.of(CPU.getCurrent(), OS.DARWIN);
     } else if (input.startsWith("freebsd")) {
       return Pair.of(CPU.getCurrent(), OS.FREEBSD);
+    } else if (input.startsWith("solaris")) {
+      return Pair.of(CPU.getCurrent(), OS.SOLARIS);
     } else if (input.startsWith("x64_windows")) {
       return Pair.of(CPU.getCurrent(), OS.WINDOWS);
     }
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CppActionConfigs.java bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CppActionConfigs.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CppActionConfigs.java      2018-04-01 14:52:36.280856404 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CppActionConfigs.java   2018-04-01 14:53:02.527808835 +0000
@@ -1064,7 +1064,7 @@
   }
 
   private static String ifLinux(CppPlatform platform, String... lines) {
-    // Platform `LINUX` also includes FreeBSD.
+    // Platform `LINUX` also includes FreeBSD and Solaris.
     return ifTrue(platform == CppPlatform.LINUX, lines);
   }
 
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfiguration.java bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfiguration.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfiguration.java      2018-04-01 14:52:36.281580812 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfiguration.java   2018-04-01 14:53:02.528615420 +0000
@@ -214,6 +214,7 @@
 
   static CppConfiguration create(CppConfigurationParameters params)
       throws InvalidConfigurationException {
+    System.out.println("Target CPU:" + params.toolchain.getTargetCpu());
     CrosstoolConfig.CToolchain toolchain = params.toolchain;
     CppOptions cppOptions = params.cppOptions;
     PathFragment crosstoolTopPathFragment =
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfigurationLoader.java bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfigurationLoader.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfigurationLoader.java        2018-04-01 14:52:36.281668789 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CppConfigurationLoader.java     2018-04-01 15:04:50.380820354 +0000
@@ -143,8 +143,10 @@
     CrosstoolConfigurationLoader.CrosstoolFile file =
         CrosstoolConfigurationLoader.readCrosstool(env, crosstoolTopLabel);
     if (file == null) {
+      System.out.println("No Crosstool file");
       return null;
     }
+    System.out.println("Crosstool file: '" + file.getProto().toString() + "'.");
     CrosstoolConfig.CToolchain toolchain =
         CrosstoolConfigurationLoader.selectToolchain(
             file.getProto(), options, cpuTransformer.getTransformer());
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationIdentifier.java bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationIdentifier.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationIdentifier.java      2018-04-01 14:52:36.283483130 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationIdentifier.java   2018-04-01 14:53:02.529455917 +0000
@@ -40,6 +40,12 @@
 
   /** Creates a new {@link CrosstoolConfigurationIdentifier} with the given parameters. */
   CrosstoolConfigurationIdentifier(String cpu, String compiler, String libc) {
+    System.out.println("cpu: '" + cpu + "' compiler: '" + compiler + "' libc: '" + libc + "'");
+
+    for (StackTraceElement ste : Thread.currentThread().getStackTrace()) {
+        System.out.println(ste);
+    }
+
     this.cpu = Preconditions.checkNotNull(cpu);
     this.compiler = compiler;
     this.libc = libc;
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationLoader.java bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationLoader.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationLoader.java  2018-04-01 14:52:36.283562611 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/rules/cpp/CrosstoolConfigurationLoader.java       2018-04-01 14:53:02.530014889 +0000
@@ -324,6 +324,7 @@
       boolean convertLipoToThinLto,
       Function<String, String> cpuTransformer)
       throws InvalidConfigurationException {
+    System.out.println("config compiler:" + config.getCompiler() + "config libc: " + config.getLibc());
     if ((config.getCompiler() != null) || (config.getLibc() != null)) {
       ArrayList<CrosstoolConfig.CToolchain> candidateToolchains = new ArrayList<>();
       for (CrosstoolConfig.CToolchain toolchain : release.getToolchainList()) {
@@ -356,6 +357,7 @@
     // We use fake CPU values to allow cross-platform builds for other languages that use the
     // C++ toolchain. Translate to the actual target architecture.
     String desiredCpu = cpuTransformer.apply(config.getCpu());
+    System.out.println("CPU: '" + config.getCpu() + "' desired CPU: '" + desiredCpu + "'");
     boolean needsLipo = lipoMode != LipoMode.OFF && !convertLipoToThinLto;
     for (CrosstoolConfig.DefaultCpuToolchain selector : release.getDefaultToolchainList()) {
       if (needsLipo && !selector.getSupportsLipo()) {
@@ -367,6 +369,7 @@
       }
     }
 
+    System.out.println("selectedIdentifier: '" + selectedIdentifier + "'");
     if (selectedIdentifier == null) {
       StringBuilder cpuBuilder = new StringBuilder();
       for (CrosstoolConfig.DefaultCpuToolchain selector : release.getDefaultToolchainList()) {
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/util/OS.java bazel/src/main/java/com/google/devtools/build/lib/util/OS.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/util/OS.java 2018-04-01 14:52:36.327916523 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/util/OS.java      2018-04-01 14:53:02.530468640 +0000
@@ -21,11 +21,12 @@
 public enum OS {
   DARWIN("osx", "Mac OS X"),
   FREEBSD("freebsd", "FreeBSD"),
+  SOLARIS("solaris","SunOS"),
   LINUX("linux", "Linux"),
   WINDOWS("windows", "Windows"),
   UNKNOWN("unknown", "");
 
-  private static final EnumSet<OS> POSIX_COMPATIBLE = EnumSet.of(DARWIN, FREEBSD, LINUX);
+  private static final EnumSet<OS> POSIX_COMPATIBLE = EnumSet.of(DARWIN, FREEBSD, SOLARIS, LINUX);
 
   private final String canonicalName;
   private final String detectionName;
diff -uar bazel-orig/src/main/java/com/google/devtools/build/lib/vfs/LocalPath.java bazel/src/main/java/com/google/devtools/build/lib/vfs/LocalPath.java
--- bazel-orig/src/main/java/com/google/devtools/build/lib/vfs/LocalPath.java   2018-04-01 14:52:36.363892385 +0000
+++ bazel/src/main/java/com/google/devtools/build/lib/vfs/LocalPath.java        2018-04-01 14:53:02.531108306 +0000
@@ -667,6 +667,7 @@
     switch (OS.getCurrent()) {
       case LINUX:
       case FREEBSD:
+      case SOLARIS:
       case UNKNOWN:
         return new UnixOsPathPolicy();
       case DARWIN:
diff -uar bazel-orig/src/main/tools/jdk.BUILD bazel/src/main/tools/jdk.BUILD
--- bazel-orig/src/main/tools/jdk.BUILD 2018-04-01 14:52:36.380117028 +0000
+++ bazel/src/main/tools/jdk.BUILD      2018-04-01 14:53:02.531508432 +0000
@@ -21,6 +21,11 @@
 )
 
 filegroup(
+    name = "jni_md_header-solaris",
+    srcs = ["include/solaris/jni_md.h"],
+)
+
+filegroup(
     name = "jni_md_header-windows",
     srcs = ["include/win32/jni_md.h"],
 )
diff -uar bazel-orig/third_party/BUILD bazel/third_party/BUILD
--- bazel-orig/third_party/BUILD        2018-04-01 14:52:36.555267787 +0000
+++ bazel/third_party/BUILD     2018-04-01 14:53:02.532119695 +0000
@@ -662,6 +662,11 @@
 )
 
 config_setting(
+    name = "solaris",
+    values = {"host_cpu": "solaris"},
+)
+
+config_setting(
     name = "s390x",
     values = {"host_cpu": "s390x"},
 )
diff -uar bazel-orig/tools/cpp/BUILD bazel/tools/cpp/BUILD
--- bazel-orig/tools/cpp/BUILD  2018-04-01 14:52:36.549549685 +0000
+++ bazel/tools/cpp/BUILD       2018-04-01 14:53:02.532556059 +0000
@@ -40,6 +40,7 @@
         "armeabi-v7a|compiler": ":cc-compiler-armeabi-v7a",
         "darwin|compiler": ":cc-compiler-darwin",
         "freebsd|compiler": ":cc-compiler-freebsd",
+        "solaris|compiler": ":cc-compiler-solaris",
         "local|compiler": ":cc-compiler-local",
         "x64_windows|compiler": ":cc-compiler-x64_windows",
         "x64_windows_msvc|compiler": ":cc-compiler-x64_windows_msvc",
@@ -123,6 +124,20 @@
     all_files = ":empty",
     compiler_files = ":empty",
     cpu = "local",
+    dwp_files = ":empty",
+    dynamic_runtime_libs = [":empty"],
+    linker_files = ":empty",
+    objcopy_files = ":empty",
+    static_runtime_libs = [":empty"],
+    strip_files = ":empty",
+    supports_param_files = 0,
+)
+
+cc_toolchain(
+    name = "cc-compiler-solaris",
+    all_files = ":empty",
+    compiler_files = ":empty",
+    cpu = "local",
     dwp_files = ":empty",
     dynamic_runtime_libs = [":empty"],
     linker_files = ":empty",
diff -uar bazel-orig/tools/cpp/BUILD.static bazel/tools/cpp/BUILD.static
--- bazel-orig/tools/cpp/BUILD.static   2018-04-01 14:52:36.549606668 +0000
+++ bazel/tools/cpp/BUILD.static        2018-04-01 14:53:02.532895942 +0000
@@ -19,6 +19,7 @@
     toolchains = {
         "armeabi-v7a|compiler": ":cc-compiler-armeabi-v7a",
         "freebsd|compiler": ":cc-compiler-freebsd",
+        "solaris|compiler": ":cc-compiler-solaris",
         "x64_windows|msvc-cl": ":cc-compiler-x64_windows",
         "x64_windows|msys-gcc": ":cc-compiler-x64_windows_msys",
         "x64_windows|mingw-gcc": ":cc-compiler-x64_windows_mingw",
@@ -45,6 +46,20 @@
     all_files = ":empty",
     compiler_files = ":empty",
     cpu = "local",
+    dwp_files = ":empty",
+    dynamic_runtime_libs = [":empty"],
+    linker_files = ":empty",
+    objcopy_files = ":empty",
+    static_runtime_libs = [":empty"],
+    strip_files = ":empty",
+    supports_param_files = 0,
+)
+
+cc_toolchain(
+    name = "cc-compiler-solaris",
+    all_files = ":empty",
+    compiler_files = ":empty",
+    cpu = "local",
     dwp_files = ":empty",
     dynamic_runtime_libs = [":empty"],
     linker_files = ":empty",
diff -uar bazel-orig/tools/cpp/CROSSTOOL bazel/tools/cpp/CROSSTOOL
--- bazel-orig/tools/cpp/CROSSTOOL      2018-04-01 14:52:36.549774320 +0000
+++ bazel/tools/cpp/CROSSTOOL   2018-04-01 14:53:02.533700016 +0000
@@ -19,6 +19,10 @@
   toolchain_identifier: "local_freebsd"
 }
 default_toolchain {
+  cpu: "solaris"
+  toolchain_identifier: "local_solaris"
+}
+default_toolchain {
   cpu: "armeabi-v7a"
   toolchain_identifier: "stub_armeabi-v7a"
 }
@@ -388,6 +392,126 @@
   # Gold linker only? Can we enable this by default?
   # linker_flag: "-Wl,--warn-execstack"
   # linker_flag: "-Wl,--detect-odr-violations"
+
+  compilation_mode_flags {
+    mode: DBG
+    # Enable debug symbols.
+    compiler_flag: "-g"
+  }
+  compilation_mode_flags {
+    mode: OPT
+
+    # No debug symbols.
+    # Maybe we should enable https://gcc.gnu.org/wiki/DebugFission for opt or
+    # even generally? However, that can't happen here, as it requires special
+    # handling in Bazel.
+    compiler_flag: "-g0"
+
+    # Conservative choice for -O
+    # -O3 can increase binary size and even slow down the resulting binaries.
+    # Profile first and / or use FDO if you need better performance than this.
+    compiler_flag: "-O2"
+
+    # Disable assertions
+    compiler_flag: "-DNDEBUG"
+
+    # Removal of unused code and data at link time (can this increase binary size in some cases?).
+    compiler_flag: "-ffunction-sections"
+    compiler_flag: "-fdata-sections"
+    linker_flag: "-Wl,--gc-sections"
+  }
+  linking_mode_flags { mode: DYNAMIC }
+}
+
+toolchain {
+  abi_version: "local"
+  abi_libc_version: "local"
+  builtin_sysroot: ""
+  compiler: "compiler"
+  host_system_name: "local"
+  needsPic: true
+  supports_gold_linker: false
+  supports_incremental_linker: false
+  supports_fission: false
+  supports_interface_shared_objects: false
+  supports_normalizing_ar: false
+  supports_start_end_lib: false
+  target_libc: "local"
+  target_cpu: "solaris"
+  target_system_name: "local"
+  toolchain_identifier: "local_solaris"
+
+  tool_path { name: "ar" path: "/opt/local/bin/ar" }
+  tool_path { name: "compat-ld" path: "/usr/bin/ld" }
+  tool_path { name: "cpp" path: "/opt/local/gcc7/bin/cpp" }
+  tool_path { name: "dwp" path: "/usr/bin/dwp" }
+  tool_path { name: "gcc" path: "/opt/local/gcc7/bin/gcc" }
+  cxx_flag: "-std=c++0x"
+  linker_flag: "-lstdc++"
+  linker_flag: "-B/usr/bin/"
+
+  # TODO(bazel-team): In theory, the path here ought to exactly match the path
+  # used by gcc. That works because bazel currently doesn't track files at
+  # absolute locations and has no remote execution, yet. However, this will need
+  # to be fixed, maybe with auto-detection?
+  cxx_builtin_include_directory: "/opt/local/gcc7/lib"
+  cxx_builtin_include_directory: "/opt/local/include"
+  cxx_builtin_include_directory: "/usr/include"
+  tool_path { name: "gcov" path: "/opt/local/gcc7/bin/gcov" }
+
+  # C(++) compiles invoke the compiler (as that is the one knowing where
+  # to find libraries), but we provide LD so other rules can invoke the linker.
+  tool_path { name: "ld" path: "/usr/bin/ld" }
+
+  tool_path { name: "nm" path: "/opt/local/bin/nm" }
+  tool_path { name: "objcopy" path: "/usr/bin/objcopy" }
+  objcopy_embed_flag: "-I"
+  objcopy_embed_flag: "binary"
+  tool_path { name: "objdump" path: "/usr/bin/objdump" }
+  tool_path { name: "strip" path: "/usr/bin/strip" }
+
+  # Anticipated future default.
+  unfiltered_cxx_flag: "-no-canonical-prefixes"
+  unfiltered_cxx_flag: "-fno-canonical-system-headers"
+
+  # Make C++ compilation deterministic. Use linkstamping instead of these
+  # compiler symbols.
+  unfiltered_cxx_flag: "-Wno-builtin-macro-redefined"
+  unfiltered_cxx_flag: "-D__DATE__=\"redacted\""
+  unfiltered_cxx_flag: "-D__TIMESTAMP__=\"redacted\""
+  unfiltered_cxx_flag: "-D__TIME__=\"redacted\""
+
+  # Security hardening on by default.
+  # Conservative choice; -D_FORTIFY_SOURCE=2 may be unsafe in some cases.
+  # We need to undef it before redefining it as some distributions now have
+  # it enabled by default.
+  compiler_flag: "-U_FORTIFY_SOURCE"
+  compiler_flag: "-D_FORTIFY_SOURCE=1"
+  compiler_flag: "-fstack-protector"
+  linker_flag: "-Wl,-z,relro,-z,now"
+
+  # Enable coloring even if there's no attached terminal. Bazel removes the
+  # escape sequences if --nocolor is specified. This isn't supported by gcc
+  # on Ubuntu 14.04.
+  # compiler_flag: "-fcolor-diagnostics"
+
+  # All warnings are enabled. Maybe enable -Werror as well?
+  compiler_flag: "-Wall"
+  # Enable a few more warnings that aren't part of -Wall.
+  compiler_flag: "-Wunused-but-set-parameter"
+  # But disable some that are problematic.
+  compiler_flag: "-Wno-free-nonheap-object" # has false positives
+
+  # Keep stack frames for debugging, even in opt mode.
+  compiler_flag: "-fno-omit-frame-pointer"
+
+  # Anticipated future default.
+  linker_flag: "-no-canonical-prefixes"
+  # Have gcc return the exit code from ld.
+  linker_flag: "-pass-exit-codes"
+  # Gold linker only? Can we enable this by default?
+  # linker_flag: "-Wl,--warn-execstack"
+  # linker_flag: "-Wl,--detect-odr-violations"
 
   compilation_mode_flags {
     mode: DBG
diff -uar bazel-orig/tools/cpp/cc_configure.bzl bazel/tools/cpp/cc_configure.bzl
--- bazel-orig/tools/cpp/cc_configure.bzl       2018-04-01 14:52:36.550063824 +0000
+++ bazel/tools/cpp/cc_configure.bzl    2018-04-01 14:53:02.534036692 +0000
@@ -24,7 +24,7 @@
       Label("@bazel_tools//tools/cpp:dummy_toolchain.bzl"), "dummy_toolchain.bzl")
   env = repository_ctx.os.environ
   cpu_value = get_cpu_value(repository_ctx)
-  if cpu_value == "freebsd":
+  if (cpu_value == "freebsd" or cpu_value == "solaris"):
     # This is defaulting to the static crosstool, we should eventually
     # autoconfigure this platform too.  Theorically, FreeBSD should be
     # straightforward to add but we cannot run it in a docker container so
diff -uar bazel-orig/tools/jdk/BUILD bazel/tools/jdk/BUILD
--- bazel-orig/tools/jdk/BUILD  2018-04-01 14:52:36.548335741 +0000
+++ bazel/tools/jdk/BUILD       2018-04-01 14:53:02.534423986 +0000
@@ -77,6 +77,11 @@
 )
 
 alias(
+    name = "jni_md_header-solaris",
+    actual = "@local_jdk//:jni_md_header-solaris",
+)
+
+alias(
     name = "java",
     actual = "@local_jdk//:java",
 )
diff -uar bazel-orig/tools/platforms/BUILD bazel/tools/platforms/BUILD
--- bazel-orig/tools/platforms/BUILD    2018-04-01 14:52:36.549193491 +0000
+++ bazel/tools/platforms/BUILD 2018-04-01 14:53:02.534848733 +0000
@@ -56,6 +56,11 @@
 )
 
 constraint_value(
+    name = "solaris",
+    constraint_setting = ":os",
+)
+
+constraint_value(
     name = "linux",
     constraint_setting = ":os",
 )
@@ -83,6 +88,7 @@
     os_constraints = [
         ":osx",
         ":freebsd",
+        ":solaris",
         ":linux",
         ":windows",
     ],
@@ -100,6 +106,7 @@
     os_constraints = [
         ":osx",
         ":freebsd",
+        ":solaris",
         ":linux",
         ":windows",
     ],
diff -uar bazel-orig/tools/platforms/platforms.BUILD bazel/tools/platforms/platforms.BUILD
--- bazel-orig/tools/platforms/platforms.BUILD  2018-04-01 14:52:36.549245186 +0000
+++ bazel/tools/platforms/platforms.BUILD       2018-04-01 14:53:02.535174740 +0000
@@ -46,6 +46,11 @@
 )
 
 constraint_value(
+    name = "solaris",
+    constraint_setting = ":os",
+)
+
+constraint_value(
     name = "linux",
     constraint_setting = ":os",
 )
@@ -73,6 +78,7 @@
     os_constraints = [
         ":osx",
         ":freebsd",
+        ":solaris",
         ":linux",
         ":windows",
     ],
@@ -90,6 +96,7 @@
     os_constraints = [
         ":osx",
         ":freebsd",
+        ":solaris",
         ":linux",
         ":windows",
     ],

[siepkes]

So turns out I couldn't let this rest ;-) I've gotten past the point I was stuck and I've created bazelbuild/bazel#4969 and bazelbuild/bazel#4967 to fix some POSIX incompatibilities in Bazel to test the waters to see if they're open to such PR's. Other communities like NetBSD might also be interested in those changes.

[mamash]

You truly are determined! :)

[siepkes]

I came across this gem in the Bazel Wiki:

Let's say that for some reason we need to configure our C++ toolchain in Bazel. This is currently quite a frustrating task. To quote one of many:

This is a fabulously difficult project that causes hardened engineers to stare blankly at screens in defeat.

But after a whole bunch of changes I'm getting somewhere! However I could use your help with the following error I'm bumping in to (see below). gcc in pkgsrc is configured (compile time) to use the SmartOS (aka Solaris, aka Illumos) linker instead of the GNU linker. The SmartOS linker does not know how how to find the GCC shared libs (-lgcc). How should I deal with this? Is this something that shouldn't have occurred in the first place (I don't specify the use of -lgcc anywhere so I'm assuming GCC does it)? I'm assuming using the SmartOS linker is preferred (since gcc is compiled that way in pkgsrc)?

ERROR: /root/bazel/third_party/protobuf/3.4.0/BUILD:399:1: Linking of rule '//third_party/protobuf/3.4.0:js_embed' failed (Exit 1): gcc failed: error executing command 
  (cd /tmp/bazel_NJvKzmPd/out/execroot/io_bazel && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin:/data/pkgbuild/scripts \
    PWD=/proc/self/cwd \
  /opt/local/gcc49/bin/gcc -o bazel-out/host/bin/third_party/protobuf/3.4.0/js_embed bazel-out/host/bin/third_party/protobuf/3.4.0/_objs/js_embed/third_party/protobuf/3.4.0/src/google/protobuf/compiler/js/embed.o -lstdc++ -B/usr/bin/ -no-canonical-prefixes -pass-exit-codes -Wl,-S)
ld: fatal: dlopen() of support library (-lgcc) failed with error: ld.so.1: ld: fatal: -lgcc: open failed: No such file or directory
collect2: error: ld returned 1 exit status
Target //src:bazel failed to build
ERROR: /root/bazel/src/main/cpp/BUILD:74:1 Linking of rule '//third_party/protobuf/3.4.0:js_embed' failed (Exit 1): gcc failed: error executing command 
  (cd /tmp/bazel_NJvKzmPd/out/execroot/io_bazel && \
  exec env - \
    PATH=/usr/local/sbin:/usr/local/bin:/opt/local/sbin:/opt/local/bin:/usr/sbin:/usr/bin:/sbin:/data/pkgbuild/scripts \
    PWD=/proc/self/cwd \
  /opt/local/gcc49/bin/gcc -o bazel-out/host/bin/third_party/protobuf/3.4.0/js_embed bazel-out/host/bin/third_party/protobuf/3.4.0/_objs/js_embed/third_party/protobuf/3.4.0/src/google/protobuf/compiler/js/embed.o -lstdc++ -B/usr/bin/ -no-canonical-prefixes -pass-exit-codes -Wl,-S)
INFO: Elapsed time: 546.197s, Critical Path: 405.28s
FAILED: Build did NOT complete successfully

ERROR: Could not build Bazel

[jperkin]

You shouldn't need to, GCC figures out by itself where to find libgcc, unless it is told not to look in the default locations. All of the -lstdc++ -B/usr/bin/ -no-canonical-prefixes -pass-exit-codes -Wl,-S arguments do not look correct, I'd start by trying to remove them all (-lstdc++ is valid but GCC should be able to figure out if when it needs it, and hardcoding will only introduce problems for building with clang).

[siepkes]

@jperkin Thanks for the assist! I'll investigate where those flags are coming from.

I think the issue with Bazel is that it is (unfortunately for us) not just another build system. It is different from other build systems in the regard that Bazel defines the entire toolchain (compiler flags, linker flags, etc.). That's a fundamentally different view of things then for example autoconf which will try and detect the environment it is in and adapt to that. Bazel's root's are with Google which uses it to build all projects in their giant GIT mono-repo. I have currently defined a SmartOS toolchain with GCC 4.9 in Bazel but for clang support a new toolchain will have to be defined inside the Bazel config. So getting it to work is one thing but getting it into pkgsrc in such a way that it will be maintainable is going to be another.

[siepkes]

The paths seem to be correct (see https://gist.github.com/siepkes/956b3bbd17072e8553b4e8f2104025f7). One of the paths is /opt/local/gcc49/x86_64-sun-solaris2.11/lib/amd64 where libgcc_s.so.1 actually lives. Could it be that -lgcc must actually be -lgcc_s for the Solaris linker to work? Or should -lgcc work? I would test it but I can't seem to find how Bazel gets the idea it should use -lgcc...

BTW I tried cheating by creating a symlink called libgcc.so but that didn't fool the linker.

[mamash]

Yes, it's libgcc_s, but one thing is that ideally the linker should not be called directly but rather through GCC. That's where GCC automatically links in any required objects with proper paths. If Bazel tries to call ld directly than things can get ugly.

[siepkes]

As far as I can tell the linker is called via gcc (see: https://gist.github.com/siepkes/956b3bbd17072e8553b4e8f2104025f7 ) or am I missing something?

[siepkes]

I think I figured out where those flags are coming from. Because gcc in pkgsrc is configured with -static-libgcc (see gcc -v) gcc expands -static-libgcc to -lgcc -lgcc_eh -lc -lgcc -lgcc_eh it seems.

-static-libgcc is not used in Bazel itself but in one of its depdencies, protobuf.

[mamash]

That's not about how GCC was configured, but yes, if something specifies -static or -static-libgcc, it gets expanded that way. Where's this particular protobuf coming from? The one from pkgsrc doesn't call for static linking - and nothing really should ideally.

[siepkes]

I read here that if GCC itself was configured with -static-libgcc GCC expands the -static-libgcc differently (ie. to the -lgcc -lgcc_eh -lc -lgcc -lgcc_eh flags I was observing). Though I must admit that my compiler foo is relatively weak and might have misinterpreted it.

Building Bazel starts with Bazel bootstrapping a (smaller?) Bazel binary. With this bootstrapped Bazel binary it will build the actual Bazel project (the BUILD file in the Bazel project). This includes building it's protobuf depedency. However Bazel will then use the BUILD file inside the protobuf project instead of the Makefile in the project (which pkgsrc uses). Apparently the flag gets added in the Bazel protobuf build but it doesn't happen in the Makefile build. Could BTW also be because I botched the Solaris toolchain config in Bazel.

[mamash]

It's OK to mix shared and static linking when building a binary, so it should be possible to let Bazel link a static protobuf archive into its binary, but link libgcc_s dynamically. You can use the -Bstatic and -Bdynamic linker args to switch behavior (e.g. -Bstatic turns on static linking for all subsequent -l args until -Bdynamic is encountered). Or - have Bazel use 'our' system protobuf lib (from pkgsrc) and link to it dynamically. (In the pkgsrc world, dynamic linking is should be used when possible.)

Back to the previous problem - it should be possible to build an entirely static binary that has the libgcc and libgcc_eh objects linked in. The compiler does have those libs - just look at pkg_info -L gcc49 | grep libgcc_eh.

[siepkes]

How rude of me! In my excitement I forget to thank you guys for the help in getting this to work!...So thanks! ;-)

[mamash]

You don't really have to worry about multiple compiler versions. The only sane pkgsrc scenario makes you have a single compiler version installed and used for everything, otherwise you're facing hell with libs linking against each other and against different compiler libs.

So if somebody needs to a build a (future) pkgsrc Envoy package that requires GCC 5, they need to start with a GCC 5+ toolchain in the beginning.

[mamash]

Oh and that's a ugly big patch. Looks good though (the /opt/local paths should not be hardcoded but that's a different thing of course). Let's hope it works and upstream will accept most of it. Good job!

[siepkes]

Yeah the patch will definitely be chopped up in to some smaller pieces. I already have 2 POSIX compatibility PR's which got merged separately in to Bazel. These are also still in this patch. There is one other POSIX compatibility change which I need to create a PR for.

There are also fixes in it to get gRPC to build on SmartOS. These will obviously need to be upstreamed to gRPC.

I will also look in to rebasing to the latest Bazel version.

[siepkes]

Bit of radio silence but I'm still working on this. I'm currently trying to build Envoy with my Solaris Bazel. Good way to test if I can actually build a reallife application with my Bazel port and didn't miss anything. After which I can try to package Bazel in to pkgsrc.

I need the help of you guys on what I suspect is an incompatibility between Solaris and Illumos (goes a bit beyond my current knowledge). One of the dependencies Envoy uses is gperftools. Specifically it requires the archive: libtcmalloc_and_profiler.a. I tried building gperftools but by default it didn't build this specific archive file. I did some digging and discovered that the configure script did not detect the correct way to use the program counter. I also managed to fix this ( see siepkes/gperftools@06d7ca2 ). Basically I need to explicitly include sys/regset.h to get access to REG_PC. I would like to upstream this fix however now comes the weird part; I don't think this is needed on Solaris? Only on Illumos / SmartOS? I came to this hypothesis because here: http://www.oracle.com/technetwork/articles/servers-storage-dev/signal-handlers-studio-2282526.html is an example on how to access the program counter for Solaris and the example program does not explicitly include sys/regset.h. Like I said I would like to upstream this fix but I think I need to be able to understand what is happening here before I can do that (and not break Solaris support for example). Could you guys point me in the right direction?

[mamash]

Yep, see this Illumos ticket.

[siepkes]

Thanks! That explains whats happening. From what I understand in the ticket if I submit a PR to upstream gperftools to get Illumos working properly (ie. by explicitly including sys/regset.h) this shouldn't impact Solaris at all, right?

[mamash]

See this comment in ucontext.h. It'd be better if the configure script didn't prefer sys/ucontext.h (which it does for the sake of Darwin systems, if I understand correctly), but rather through the ucontext.h wrapper, then no futher include should be needed:

https://github.com/illumos/illumos-gate/blob/master/usr/src/head/ucontext.h#L36

[siepkes]

I agree that it's nicer (as far as I can tell) but I think that is going to be a hard sell to upstream. For one I can't oversee what changing the order might break on other platforms?

I think the best we can get into upstream if we want to go the ucontext.h way is to add a elif (see snippet below) specific for Solaris (below the RedHat exception) with something like autoconf's host_os variable. As far as I can tell switching Solaris to ucontext.h does not break stuff for them?

AC_DEFUN([AC_PC_FROM_UCONTEXT],
  [AC_CHECK_HEADERS(ucontext.h)
   # Redhat 7 has <sys/ucontext.h>, but it barfs if we #include it directly
   # (this was fixed in later redhats).  <ucontext.h> works fine, so use that.
   if grep "Red Hat Linux release 7" /etc/redhat-release >/dev/null 2>&1; then
     AC_DEFINE(HAVE_SYS_UCONTEXT_H, 0, [<sys/ucontext.h> is broken on redhat 7])
     ac_cv_header_sys_ucontext_h=no
   else
     AC_CHECK_HEADERS(sys/ucontext.h)       # ucontext on OS X 10.6 (at least)
   fi
   AC_CHECK_HEADERS(cygwin/signal.h)        # ucontext on cywgin
   AC_MSG_CHECKING([how to access the program counter from a struct ucontext])

Sorry for all the questions but this part is definitely something I still need to gain some experience.

[mamash]

I believe the standard practice on SunOS should be to include <ucontext.h> rather than the other one. There should be no harm in doing it this way on legacy SunOS platforms.

[siepkes]

Sorry for the silence (life got in the way of my Envoy ambitions ;-) but I finally got Envoy working on SmartOS!

This is the output for the Envoy getting started tutorial:

./bazel-bin/source/exe/envoy-static --disable-hot-restart -c /root/simplest.yaml --v2-config-only -l debug
[2018-05-27 08:55:25.704][2168849798097841941][info][main] source/server/server.cc:178] initializing epoch 0 (hot restart version=disabled)
[2018-05-27 08:55:25.717][2168849798097841941][debug][main] source/server/server.cc:206] admin address: 0.0.0.0:9901
[2018-05-27 08:55:25.723][5009289261590690597][debug][grpc] source/common/grpc/google_async_client_impl.cc:37] completionThread running
[2018-05-27 08:55:25.726][2168849798097841941][debug][upstream] source/common/upstream/cluster_manager_impl.cc:628] adding TLS initial cluster service_google
[2018-05-27 08:55:25.726][2168849798097841941][debug][upstream] source/common/upstream/logical_dns_cluster.cc:70] starting async DNS resolution for google.com
[2018-05-27 08:55:25.727][2168849798097841941][debug][upstream] source/common/network/dns_impl.cc:132] Setting DNS resolution timer for 4687 milliseconds
[2018-05-27 08:55:25.727][2168849798097841941][debug][upstream] source/common/upstream/cluster_manager_impl.cc:59] cm init: adding: cluster=service_google primary=1 secondary=0
[2018-05-27 08:55:25.727][2168849798097841941][info][config] source/server/configuration_impl.cc:52] loading 1 listener(s)
[2018-05-27 08:55:25.727][2168849798097841941][debug][config] source/server/configuration_impl.cc:54] listener #0:
[2018-05-27 08:55:25.727][2168849798097841941][debug][config] source/server/listener_manager_impl.cc:314] begin add/update listener: name=listener_0 hash=9681839072681943295
[2018-05-27 08:55:25.727][2168849798097841941][debug][config] source/server/listener_manager_impl.cc:29]   filter #0:
[2018-05-27 08:55:25.727][2168849798097841941][debug][config] source/server/listener_manager_impl.cc:30]     name: envoy.http_connection_manager
[2018-05-27 08:55:25.727][2168849798097841941][debug][config] source/server/listener_manager_impl.cc:33]   config: {"codec_type":"AUTO","route_config":{"virtual_hosts":[{"domains":["*"],"name":"local_service","routes":[{"match":{"prefix":"/"},"route":{"host_rewrite":"www.google.com","cluster":"service_google"}}]}],"name":"local_route"},"http_filters":[{"name":"envoy.router"}],"stat_prefix":"ingress_http"}
[2018-05-27 08:55:25.734][2168849798097841941][debug][config] source/server/config/network/http_connection_manager.cc:239]     filter #0
[2018-05-27 08:55:25.734][2168849798097841941][debug][config] source/server/config/network/http_connection_manager.cc:240]       name: envoy.router
[2018-05-27 08:55:25.734][2168849798097841941][debug][config] source/server/config/network/http_connection_manager.cc:244]     config: {}
[2018-05-27 08:55:25.735][2168849798097841941][debug][config] source/server/listener_manager_impl.cc:248] add active listener: name=listener_0, hash=9681839072681943295, address=0.0.0.0:10000
[2018-05-27 08:55:25.735][2168849798097841941][info][config] source/server/configuration_impl.cc:92] loading tracing configuration
[2018-05-27 08:55:25.735][2168849798097841941][info][config] source/server/configuration_impl.cc:119] loading stats sink configuration
[2018-05-27 08:55:25.735][2168849798097841941][info][main] source/server/server.cc:353] starting main dispatch loop
[2018-05-27 08:55:25.735][2168849798097841941][debug][upstream] source/common/upstream/logical_dns_cluster.cc:78] async DNS resolution complete for google.com

Had to disable hot restart for now since there is apparently some issue how Envoy tries to bind the socket:

./bazel-bin/source/exe/envoy-static -c /root/simplest.yaml --v2-config-only -l debug
terminate called after throwing an instance of 'Envoy::EnvoyException'
  what():  unable to bind domain socket with id=0 (see --base-id option)
Abort (core dumped)

But thats just some minor fixing and improving that needs to be done. Overall the thing seems to work!

I had to fork libeevent, boringssl, gperftools, grpc and (obviously) envoy itself to make changes to make them Solaris compatible. So now that the fun is over I'll need to polish these changes and get them upstream.

@mamash @jperkin Bazel for Envoy is configured to create a statically linked Envoy binary. While I guess it is possible to create a Bazel config which creates a shared linked Envoy binary it is going to be hard and a lot of work to maintain. The first hurdle being that Bazel is basically designed like Go; It wants to create statically linked binaries. Just like one might be able to use a tennis racket to drive a nail into a wall it's still going to be a painful experience. Envoy itself cares even less about shared libraries then Bazel. Which from their perspective kinda makes sense because Envoy is often deployed inside a docker container as a sidecar; You don't want to run in to dependency hell with your sidecar. We can probably debate if this is a good idea or not but it doesn't really matter what our opinion on it is since thats just the way Envoy (and the container world for a large part, certainly the parties backing Envoy like Google) currently work.

So aside from getting stuff upstream, etc. how do you guys want to approach this from a pkgsrc perspective? I'm no pkgsrc expert but I would advise to just let Bazel and Envoy generate their statically linked things and package that.

[mamash]

Yes, I don't think it's worth the effort to insist on dynamic linking if it involves fighting a principle design of a tool.

So right now how do your bazel and envoy binaries look like linking wise - no dynamic libs linked beyond libc/libm and maybe libgcc_s?

It's not entirely unheard of in pkgsrc to have a package that builds some dependecies embedded from source bundled with the distribution (e.g. lately I had to switch lang/nodejs to use OpenSSL embedded, because it requires 1.1.x).

It may get tricky if a package requires dependencies to exist, but insists on linking them statically.

[gdt]

Static linking of dependencies should not be a problem in building. The problem is that when there's an issue in a dependency, and it's updated, it requires rebuilding the depending package to get the fix. This is why statically linking -- whether of an actual dependency package or an included copy -- is discouraged in pkgsrc.

Stepping back, if I read correctly (too quickly), it seems like this package is almost turning into a mild fork, with changes that upstream is unwilling to take. If that's true, it may be better to actually do that (a following fork) and then package the fork, rather than trying to maintain a fork in a package. (I don't speak for joyent's pkgsrc, but that's what I'd say about pkgsrc proper.)

[siepkes]

@mamash Below is how the binary currently looks. It's currently in the im-still-surprised-I-even-got-it-to-work-and-still-needs-polishing state ;-). Very open to feedback though!

I'm currently a bit swamped at work but I'm going to plan a couple of days of from work in 1 or 2 weeks so I can get this all properly wrapped up. I'm 80% done, now I just need to finish the other 80% ;-).

$ ldd ./bazel-bin/source/exe/envoy-static
        librt.so.1 =>    /lib/64/librt.so.1
        libdl.so.1 =>    /lib/64/libdl.so.1
        libpthread.so.1 =>       /lib/64/libpthread.so.1
        libm.so.2 =>     /lib/64/libm.so.2
        libstdc++.so.6 =>        /opt/local/gcc7//lib/amd64/libstdc++.so.6
        libxnet.so.1 =>  /lib/64/libxnet.so.1
        libsocket.so.1 =>        /lib/64/libsocket.so.1
        libnsl.so.1 =>   /lib/64/libnsl.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libgcc_s.so.1 =>         /opt/local/gcc7//lib/amd64/libgcc_s.so.1
        libmp.so.2 =>    /lib/64/libmp.so.2
        libmd.so.1 =>    /lib/64/libmd.so.1

[siepkes]

Just to let you guys know I'm still on this. I'm actually trying to get the SmartOS / Illumos / Solaris patches for gRPC upstreamed which is proving somewhat harder then I had hoped. I might just try and get the Bazel things upstreamed or if they don't want them at least the stuff that fixes a bunch of POSIX compliance.

I've rebased my patch to Bazel 0.16 in case your interested: https://gist.github.com/siepkes/0d144125974b2a84a92433e0282804b2

[siepkes]

For reference here are the repositories with (working) ports which I keep updated with releases of upstream:

• bazel-smartos port repo.
• envoy-smartos port repo.

Solaris gRPC support was actually upstreamed.

[Smithx10]

@siepkes Now that Hashicorp Consul is using Envoy as a Service Mesh Proxy I could see this being used more by the "DevOpers" of the world. Any word on getting packages added to pkgsrc? Great Job doing this. Bazel is brutal.

[siepkes]

@Smithx10 I share the feeling that Envoy will only become more relevant in the future. It's also slowly eating away edge proxy market share of NGINX, haproxy and the likes.

I've dropped the ball a bit (ok a lot) on getting Envoy included in pkgsrc so there isn't much progress there. I am however keeping Envoy up to date with upstream and Bazel as required (ie. I bump Bazel when Envoy needs it). Bazel is mostly hard to keep up because upstream Bazel is well...let's say....not afraid to refactor their code base 😉. I'm hoping that since they are now in the 1.X version range the internals will stabilize a bit, making porting easier. Ideally I would like to upstream it to Bazel as Illumos support however there are a bunch of upstream dependencies that will first require accepting certain patches (for example madler/zlib#395).

I'm not really a pkgsrc packaging pro and both Bazel and Envoy aren't the easiest thing to package. So if anyone wants to take a crack at putting Bazel and Envoy in pkgsrc I'd be happy to assist!

[Smithx10]

@siepkes @danmcd noted to me in IRC that the following bug will really make using Envoy SmartOS problematic. envoyproxy/envoy#1446 Did you resolve this in your fork?

[danmcd]

@siepkes @danmcd noted to me in IRC that the following bug will really make using Envoy SmartOS problematic. envoyproxy/envoy#1446 Did you resolve this in your fork?

NOTE: There's #ifdef APPLE in the relevant code that just needs to include illumos as well.

[siepkes]

@Smithx10 Yes, that particular issue is fixed in the fork. The fix which @danmcd mentions is located here: siepkes/envoy-smartos@a70767a#diff-3315c7b291f78a482ebd4cbfc0545362