Explorer NIF broken on FreeBSD about explorer

[leonardorame]

Great @philss I'll try it later today.

[philss]

Hey, @leonardorame 👋

José added the instructions for manual compilation to the README.md, but just to complement: we are building for FreeBSD using the cross-compilation feature of the Rust compiler. I think it's not possible to target a specific version of that OS, so you need to compile in your machine.

Another tip is that you can specify the environment var in your Mix.install/2 call:

Mix.install([
  {:explorer, "~> 0.8.2"},
  {:rustler, ">= 0.0.0"} # Rustler is needed for the build.
], system_env: %{"EXPLORER_BUILD" => "1"})

Please check the README.md for more details about the dependencies (mostly the Rust toolchain).

[leonardorame]

As I didn't have rust nor cmake I installed both:

pkg install rust-nightly-1.79.0.20240317
pkg install cmake

Then clicked on Reconnect and setup and it started compiling. Then, after about 200seconds I've got this:

   Compiling polars-io v0.38.3
   Compiling explorer v0.1.0 (/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/deps/explorer/native/explorer)
error: could not compile `explorer` (lib)

Caused by:
  process didn't exit successfully: `rustc --crate-name explorer --edition=2021 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type cdylib --emit=dep-info,link -C opt-level=3 -C lto --cfg 'feature="aws"' --cfg 'feature="cloud"' --cfg 'feature="default"' --cfg 'feature="ndjson"' --cfg 'feature="nif_version_2_15"' --cfg 'feature="object_store"' --cfg 'feature="tokio"' --cfg 'feature="tokio-util"' -C metadata=af3605e760517818 --out-dir /usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps -C strip=debuginfo -L dependency=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps --extern anyhow=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libanyhow-aa7010fbb529b211.rlib --extern chrono=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libchrono-9afb8f29bfaf3df7.rlib --extern either=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libeither-2de17266c51dccd6.rlib --extern mimalloc=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libmimalloc-29e6df8377a9d59e.rlib --extern object_store=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libobject_store-4aea81ad9dfc2ff9.rlib --extern polars=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libpolars-2aed453d564afe76.rlib --extern polars_ops=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libpolars_ops-e8791d90f9468b58.rlib --extern rand=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/librand-4b2107d15797d59f.rlib --extern rand_pcg=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/librand_pcg-d094ddec11919880.rlib --extern rustler=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/librustler-0213b10aec1582e4.rlib --extern smartstring=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libsmartstring-d403afceab17b606.rlib --extern thiserror=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libthiserror-02c7149448432c30.rlib --extern tokio=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libtokio-45d4963c954defeb.rlib --extern tokio_util=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libtokio_util-6aa64607e6d7991d.rlib -L native=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/build/libmimalloc-sys-80094c8c79d90f5e/out -L native=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/build/ring-153dc47b30099ac3/out -L native=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/build/lz4-sys-8a9a0bb2bd29dabd/out -L native=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/build/zstd-sys-eb22deeff51b996f/out -L native=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/build/libz-ng-sys-7321d91231aeb2ba/out/lib -L native=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/build/libz-ng-sys-7321d91231aeb2ba/out/lib64` (signal: 9, SIGKILL: kill)

== Compilation error in file lib/explorer/polars_backend/native.ex ==
** (RuntimeError) Rust NIF compile error (rustc exit code 101)
    (rustler 0.32.1) lib/rustler/compiler.ex:36: Rustler.Compiler.compile_crate/3
    lib/explorer/polars_backend/native.ex:32: (module)
could not compile dependency :explorer, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile explorer --force", update it with "mix deps.update explorer" or clean it with "mix deps.clean explorer"

[leonardorame]

Hi, I was able to compile it from the command line. rustc needed 11gb of Ram.

The build directory is ~/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps, now, I would like to prevent a re-compilation. Should I leave this directory as is?

[philss]

@leonardorame yeah, you shouldn't remove the _build directory of your project. But if you are using Mix.install/2, be aware that changing any of the deps (or adding a new one) will trigger a recompilation. If you need something more stable, you can fork explorer, make a release, publish your artifact that was compiled, and point the RustlerPrecompiled config to your fork.

[leonardorame]

Let me see if I understood it.

The original Mix.install is:

Mix.install([
  {:explorer, "~> 0.8.2"},
  {:rustler, ">= 0.0.0"} # Rustler is needed for the build.
], system_env: %{"EXPLORER_BUILD" => "1"})

I must convert to this?:

Mix.install([
  {:explorer, "~> 0.8.2"},
  {:rustler, ">= 0.0.0"} # Rustler is needed for the build.
]

[philss]

@leonardorame no, the idea is to do something like I did in the example project with a fork of explorer, and them use that fork in your notebooks. It's kind of hard to maintain, but at least you won't have to recompile it every time.

Your installation would look like this:

Mix.install([
  {:explorer, git: "yourgitrepo"}
])

But if you do not plan to change dependencies in your current project, I believe it's better to not go through all that work. I would suggest to maintain a fork just in case you need to use Explorer in production, or use in different notebooks.

[leonardorame]

Hi @philss I forked explorer, then did a mix compile, the dependency adbc didn't compile on FreeBSD so I commented it on mix.exs. Then it started to build, until I've got this error:

   ...  
   Compiling polars-time v0.38.3
   Compiling polars-io v0.38.3
   Compiling explorer v0.1.0 (/usr/home/informemedico/explorer/native/explorer)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 00s

11:13:20.530 [warning] The on_load function for module Elixir.Explorer.PolarsBackend.Native returned:
{:error,
 {:load_failed,
  ~c"Failed to load NIF library: '/usr/home/informemedico/explorer/_build/dev/lib/explorer/priv/native/libexplorer.so: No space available for static Thread Local Storage'"}}

Generated explorer app

Any hint?

[leonardorame]

I see Finished dev profile [unoptimized + debuginfo] target(s) in 2m 00s, how can I build a release (optimized without debug info)?

[josevalim]

Set MIN_ENV=prod env var perhaps?

[leonardorame]

Thanks @josevalim with MIX_ENV=prod I was able to build a release, but I still get the "No space available for static Thread Local Storage" error.

By reading this: https://lists.freebsd.org/archives/freebsd-hackers/2024-April/003182.html and exporting the LD_STATIC_TLS_EXTRA env var with a higher number than 128 it didn't fix the error.

Now I'm looking at this: https://doc.rust-lang.org/beta/unstable-book/compiler-flags/tls-model.html but I don't know where can I set the -Z tls-model param for the build, could you point me to the right place?.

[leonardorame]

RUSTFLAGS="-Z tls-model=global-dynamic" mix compile -> Same error
RUSTFLAGS="-Z tls-model=local-dynamic" mix compile -> Same error
RUSTFLAGS="-Z tls-model=initial-exec" mix compile -> can't find crate for tracing_attributes
RUSTFLAGS="-Z tls-model=local-exec" mix compile -> many errors
RUSTFLAGS="-Z tls-model=emulated" mix compile -> many errors

It looks like there's no way to build explorer on FreeBSD.