Comments (12)
Great @philss I'll try it later today.
from explorer.
Hey, @leonardorame 👋
José added the instructions for manual compilation to the README.md
, but just to complement: we are building for FreeBSD using the cross-compilation feature of the Rust compiler. I think it's not possible to target a specific version of that OS, so you need to compile in your machine.
Another tip is that you can specify the environment var in your Mix.install/2
call:
Mix.install([
{:explorer, "~> 0.8.2"},
{:rustler, ">= 0.0.0"} # Rustler is needed for the build.
], system_env: %{"EXPLORER_BUILD" => "1"})
Please check the README.md for more details about the dependencies (mostly the Rust toolchain).
from explorer.
As I didn't have rust nor cmake I installed both:
pkg install rust-nightly-1.79.0.20240317
pkg install cmake
Then clicked on Reconnect and setup and it started compiling. Then, after about 200seconds I've got this:
Compiling polars-io v0.38.3
Compiling explorer v0.1.0 (/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/deps/explorer/native/explorer)
error: could not compile `explorer` (lib)
Caused by:
process didn't exit successfully: `rustc --crate-name explorer --edition=2021 src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type cdylib --emit=dep-info,link -C opt-level=3 -C lto --cfg 'feature="aws"' --cfg 'feature="cloud"' --cfg 'feature="default"' --cfg 'feature="ndjson"' --cfg 'feature="nif_version_2_15"' --cfg 'feature="object_store"' --cfg 'feature="tokio"' --cfg 'feature="tokio-util"' -C metadata=af3605e760517818 --out-dir /usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps -C strip=debuginfo -L dependency=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps --extern anyhow=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libanyhow-aa7010fbb529b211.rlib --extern chrono=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libchrono-9afb8f29bfaf3df7.rlib --extern either=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libeither-2de17266c51dccd6.rlib --extern mimalloc=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libmimalloc-29e6df8377a9d59e.rlib --extern object_store=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libobject_store-4aea81ad9dfc2ff9.rlib --extern polars=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libpolars-2aed453d564afe76.rlib --extern polars_ops=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libpolars_ops-e8791d90f9468b58.rlib --extern rand=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/librand-4b2107d15797d59f.rlib --extern rand_pcg=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/librand_pcg-d094ddec11919880.rlib --extern rustler=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/librustler-0213b10aec1582e4.rlib --extern smartstring=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libsmartstring-d403afceab17b606.rlib --extern thiserror=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libthiserror-02c7149448432c30.rlib --extern tokio=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libtokio-45d4963c954defeb.rlib --extern tokio_util=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps/libtokio_util-6aa64607e6d7991d.rlib -L native=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/build/libmimalloc-sys-80094c8c79d90f5e/out -L native=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/build/ring-153dc47b30099ac3/out -L native=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/build/lz4-sys-8a9a0bb2bd29dabd/out -L native=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/build/zstd-sys-eb22deeff51b996f/out -L native=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/build/libz-ng-sys-7321d91231aeb2ba/out/lib -L native=/usr/home/informemedico/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/build/libz-ng-sys-7321d91231aeb2ba/out/lib64` (signal: 9, SIGKILL: kill)
== Compilation error in file lib/explorer/polars_backend/native.ex ==
** (RuntimeError) Rust NIF compile error (rustc exit code 101)
(rustler 0.32.1) lib/rustler/compiler.ex:36: Rustler.Compiler.compile_crate/3
lib/explorer/polars_backend/native.ex:32: (module)
could not compile dependency :explorer, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile explorer --force", update it with "mix deps.update explorer" or clean it with "mix deps.clean explorer"
from explorer.
Hi, I was able to compile it from the command line. rustc needed 11gb of Ram.
The build directory is ~/.cache/mix/installs/elixir-1.16.2-erts-14.2.5/b4bfc151806c51196f3b62c267bb60b4/_build/dev/lib/explorer/native/explorer/release/deps
, now, I would like to prevent a re-compilation. Should I leave this directory as is?
from explorer.
@leonardorame yeah, you shouldn't remove the _build
directory of your project. But if you are using Mix.install/2
, be aware that changing any of the deps (or adding a new one) will trigger a recompilation. If you need something more stable, you can fork explorer, make a release, publish your artifact that was compiled, and point the RustlerPrecompiled config to your fork.
from explorer.
Let me see if I understood it.
The original Mix.install is:
Mix.install([
{:explorer, "~> 0.8.2"},
{:rustler, ">= 0.0.0"} # Rustler is needed for the build.
], system_env: %{"EXPLORER_BUILD" => "1"})
I must convert to this?:
Mix.install([
{:explorer, "~> 0.8.2"},
{:rustler, ">= 0.0.0"} # Rustler is needed for the build.
]
from explorer.
@leonardorame no, the idea is to do something like I did in the example project with a fork of explorer, and them use that fork in your notebooks. It's kind of hard to maintain, but at least you won't have to recompile it every time.
Your installation would look like this:
Mix.install([
{:explorer, git: "yourgitrepo"}
])
But if you do not plan to change dependencies in your current project, I believe it's better to not go through all that work. I would suggest to maintain a fork just in case you need to use Explorer in production, or use in different notebooks.
from explorer.
Hi @philss I forked explorer, then did a mix compile
, the dependency adbc
didn't compile on FreeBSD so I commented it on mix.exs
. Then it started to build, until I've got this error:
...
Compiling polars-time v0.38.3
Compiling polars-io v0.38.3
Compiling explorer v0.1.0 (/usr/home/informemedico/explorer/native/explorer)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 2m 00s
11:13:20.530 [warning] The on_load function for module Elixir.Explorer.PolarsBackend.Native returned:
{:error,
{:load_failed,
~c"Failed to load NIF library: '/usr/home/informemedico/explorer/_build/dev/lib/explorer/priv/native/libexplorer.so: No space available for static Thread Local Storage'"}}
Generated explorer app
Any hint?
from explorer.
I see Finished
dev profile [unoptimized + debuginfo] target(s) in 2m 00s
, how can I build a release (optimized without debug info)?
from explorer.
Set MIN_ENV=prod
env var perhaps?
from explorer.
Thanks @josevalim with MIX_ENV=prod
I was able to build a release
, but I still get the "No space available for static Thread Local Storage" error.
By reading this: https://lists.freebsd.org/archives/freebsd-hackers/2024-April/003182.html and exporting the LD_STATIC_TLS_EXTRA env var with a higher number than 128 it didn't fix the error.
Now I'm looking at this: https://doc.rust-lang.org/beta/unstable-book/compiler-flags/tls-model.html but I don't know where can I set the -Z tls-model
param for the build, could you point me to the right place?.
from explorer.
RUSTFLAGS="-Z tls-model=global-dynamic" mix compile -> Same error
RUSTFLAGS="-Z tls-model=local-dynamic" mix compile -> Same error
RUSTFLAGS="-Z tls-model=initial-exec" mix compile -> can't find crate for tracing_attributes
RUSTFLAGS="-Z tls-model=local-exec" mix compile -> many errors
RUSTFLAGS="-Z tls-model=emulated" mix compile -> many errors
It looks like there's no way to build explorer on FreeBSD.
from explorer.
Related Issues (20)
- Seeing `:nif_not_loaded` error for `Series.split/2` when mutating a dataframe HOT 1
- [Feature request] Add support for read_database in Polars backend. HOT 1
- Using `sort_by` with a grouped data frame doesn't respect `nils:` option HOT 1
- `{:datetime, :second}` dtype support HOT 2
- Add :streaming option to DataFrame.to_csv/3 HOT 1
- Exporting to CSV with a duration column returns an error
- Regression in `DataFrame.concat_rows/2` in v0.8.2 HOT 1
- Filter throwing undefined variable error HOT 1
- Error using is_finite and is_infinite within mutate HOT 1
- Support Elixir built in Duration struct HOT 1
- Bug: Rounding Error in Tests HOT 1
- exposing the `fold` expressions from Polars HOT 7
- :nif_panicked "Chunk require all its arrays to have an equal number of rows" HOT 1
- Sorting an empty DataFrame results in a runtime Polars error HOT 1
- Performance of `DataFrame.new/2` on dataframes containing list columns HOT 7
- `Series.filter` should work inside `DataFrame.summarise` HOT 5
- Large memory usage when using `Explorer.Dataframe.concat_columns` on 30k (small) data frames. Memory leak? HOT 4
- [Not Issue] - Are the plans to use duckdb as an alternative backend? HOT 2
- Support streaming: true on collect HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from explorer.