dtolnay / linkme Goto Github PK
View Code? Open in Web Editor NEWSafe cross-platform linker shenanigans
License: Apache License 2.0
Safe cross-platform linker shenanigans
License: Apache License 2.0
It seems like
#[distributed_slice(FOO)]
#[linkme(crate=::mycrate::linkme)]
static BENCH_DESERIALIZE: fn(&mut Bencher) = bench_deserialize;
causes distributed_slice
to refer to the linkme crate as ::mycrate::linkme
. But, I don't see this documented anywhere. I don't quite understand it well enough to write those docs myself :)
https://travis-ci.com/dtolnay/linkme/jobs/263379561
error: linking with `rust-lld` failed: exit code: 1
|
= note: "rust-lld" "rust-lld" "-flavor" "gnu" "-L" "/home/travis/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7m-none-eabi/lib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/cortex_m_linkme_test-e6719463d731e9f0.cortex_m_linkme_test.dqximbec-cgu.0.rcgu.o" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/cortex_m_linkme_test-e6719463d731e9f0.cortex_m_linkme_test.dqximbec-cgu.1.rcgu.o" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/cortex_m_linkme_test-e6719463d731e9f0.cortex_m_linkme_test.dqximbec-cgu.2.rcgu.o" "-o" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/cortex_m_linkme_test-e6719463d731e9f0" "--gc-sections" "-L" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps" "-L" "/home/travis/build/dtolnay/linkme/target/release/deps" "-L" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/build/cortex-m-linkme-test-200ea53de4bd4e09/out" "-L" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/build/cortex-m-31349c7505c00923/out" "-L" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/build/cortex-m-rt-4c8df37bd6f5b5e1/out" "-L" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/build/cortex-m-semihosting-2445422be57c0b84/out" "-L" "/home/travis/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7m-none-eabi/lib" "-Bstatic" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/liblinkme-cf1aa9b254dec488.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libcortex_m_rt-4e1c1e14bbef8ced.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libr0-1f53d375f373e972.rlib" "--start-group" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libpanic_semihosting-458c7d50d4f6de89.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libcortex_m_semihosting-8944461358d63d4a.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libcortex_m-7f4503edc7f5e4d5.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libvolatile_register-255db70b159ec0a4.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libvcell-b3554e83a40b7ef5.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libbare_metal-1297e03344968a12.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libaligned-7926c2ef4e454092.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libas_slice-7ff6256bbd72fde4.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libstable_deref_trait-550b77b954e6d921.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libgeneric_array-685219a2762e8d8f.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libgeneric_array-5204fbf5f82b726a.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libtypenum-94476904ccd9d85f.rlib" "/home/travis/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7m-none-eabi/lib/librustc_std_workspace_core-eafb71f8805df5a7.rlib" "/home/travis/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7m-none-eabi/lib/libcore-bfb368382075f384.rlib" "--end-group" "/home/travis/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7m-none-eabi/lib/libcompiler_builtins-c6415c30ba62782a.rlib" "-Tlink.x" "-Bdynamic"
= note: rust-lld: error: section linkme_SHENANIGANS file range overlaps with .rodata
>>> linkme_SHENANIGANS range is [0x2604, 0x260F]
>>> .rodata range is [0x2604, 0x29BF]
rust-lld: error: section linkme_SHENANIGANS virtual address range overlaps with .rodata
>>> linkme_SHENANIGANS range is [0x1604, 0x160F]
>>> .rodata range is [0x1604, 0x19BF]
rust-lld: error: section linkme_SHENANIGANS load address range overlaps with .rodata
>>> linkme_SHENANIGANS range is [0x1604, 0x160F]
>>> .rodata range is [0x1604, 0x19BF]
The only unique identifier in the link section name currently is the name of the static. Multiple crates could make distributed slices with the same name. We should do one of:
I'm trying to use linkme as part of writing extensions for other languages, such as PHP. It's working very well for this case, with one exception: currently, I have to clone the slice to add a null item on the end for lists that must be null-terminated on the PHP end. It would be nice if there was a way to provide a guaranteed 'last item' when defining the list, or something similar.
The crate currently doesn't really support ZSTs the way one might expect. We should just have these fail to compile if the size is zero.
use linkme::distributed_slice;
use other_crate::BENCHMARKS;
#[distributed_slice(BENCHMARKS)]
static BENCH_DESERIALIZE: fn(&mut Bencher) = bench_deserialize;
fn bench_deserialize(b: &mut Bencher) {
/* ... */
}
If BENCHMARKS
is initialized in external crate the call to #[distributed_slice(BENCHMARKS)]
doesn't add bench_deserialize
function to the array. But if I turn on lto=true
in Cargo.toml of the builded project (where BENCHMARKS
is placed) lib works as expected.
I also tested this on ubuntu and it worked without lto
it actually ends up crashing lld-link, but the errors come first, so I think it's also doing something wrong.
I tried with and without used_linker
and get the same thing either way, whereas used_linker
fixed lld on other platforms.
The GTESTS
here is the distributed_slice, in the rust_gtest_interop
crate.
[2/2] LINK rust_gtest_interop_unittests.exe rust_gtest_interop_unittests.exe.pdb
FAILED: rust_gtest_interop_unittests.exe rust_gtest_interop_unittests.exe.pdb
..\..\third_party\llvm-build\Release+Asserts\bin\lld-link.exe "/OUT:./rust_gtest_interop_unittests.exe" /nologo -libpath:..\..\third_party\llvm-build\Release+Asserts\lib\clang\17\lib\windows /winsysroot:../../third_party/depot_tools/win_toolchain/vs_files/27370823e7 /MACHINE:X64 "/PDB:./rust_gtest_interop_unittests.exe.pdb" "@./rust_gtest_interop_unittests.exe.rsp"
lld-link: error: relocation against symbol in discarded section: _ZN18rust_gtest_interop6GTESTS12LINKME_START17h75470cf2db5c947aE
>>> referenced by librust_gtest_interop.rlib(librust_gtest_interop.rust_gtest_interop.11482080eb92b0cf-cgu.0.rcgu.o):(rust_gtest_interop::GTESTS::h78f8c0ca7ebcfebd)
>>> referenced by librust_gtest_interop.rlib(librust_gtest_interop.rust_gtest_interop.11482080eb92b0cf-cgu.0.rcgu.o):(__declspec(dllimport) rust_gtest_interop::GTESTS::LINKME_START::h75470cf2db5c947a)
lld-link: error: relocation against symbol in discarded section: _ZN18rust_gtest_interop6GTESTS11LINKME_STOP17h3f94993507f6c1d0E
>>> referenced by librust_gtest_interop.rlib(librust_gtest_interop.rust_gtest_interop.11482080eb92b0cf-cgu.0.rcgu.o):(rust_gtest_interop::GTESTS::h78f8c0ca7ebcfebd)
>>> referenced by librust_gtest_interop.rlib(librust_gtest_interop.rust_gtest_interop.11482080eb92b0cf-cgu.0.rcgu.o):(__declspec(dllimport) rust_gtest_interop::GTESTS::LINKME_STOP::h3f94993507f6c1d0)
This should be reproducible on Windows outside of Chromium with --target=x86_64-pc-windows-msvc -Clinker=clang-cl -Clink-arg=-fuse-ld=lld
.
I investigated if it would be possible to add WASM support and it looks like there might be a way. While I have not found a similar start/end convention that works, WASM supports custom data sections which can be accessed as an array from JS. wasm_bindgen
has a hidden and somewhat unsupported function that can get a handle to the WebAssembly.Module, while js-sys
exposes bindings for fetching data from a custom section. This code compiles:
let module = js_sys::WebAssembly::Module::new(&wasm_bindgen::module()).unwrap();
let data = js_sys::WebAssembly::Module::custom_sections(&module, "thingies");
@ibaryshnikov created a more complete and runnable example here:
https://github.com/ibaryshnikov/wasm-custom-sections/blob/master/src/lib.rs
Unfortunately, webpack does not expose the WebAssembly.Module object in javascript:
webpack/webpack#8157
Here's another similar issue:
WebAssembly/esm-integration#14
So this approach seems to work, but the WebAssembly.Module handle is not exposed in certain cases. Better than nothing I guess? ๐
This probably isn't an issue with this crate per-se, but I wanted to bring up this use case just in case someone more versed in rust than me has a solution.
I have noticed that when I declare and construct a distributed_slice
in a dependency crate, the entries in the slice declared in the dependency crates do not appear. This is probably due to rust-lang/rust#67209, which seems to point to some kind of linker issue that causes custom sections to disappear across crate boundaries. I don't fully understand that issue and it has been open for some time. I suspect that the issue I'm providing an example for here is just a consequence of linkme
's use of custom sections (via link_section
) and that bug. I've done a little inspecting of the output elf when I don't have a distributed slice member in a root binary crate (so all members are declared in dependency crates) and the custom section doesn't even appear in the output.
Again, I don't really expect a solution to appear in this crate since it is probably a deeper linker issue, but I wanted to make sure this was mentioned here and possibly advocate for this caveat to be called out on the documentation until it is fixed.
[workspace]
members = [
"toplevel",
"dependency",
]
[package]
name = "dependency"
version = "0.1.0"
edition = "2018"
[dependencies]
linkme = "0.2"
use linkme::distributed_slice;
#[distributed_slice]
pub static TEST: [fn()] = [..];
mod other;
pub fn do_stuff() {
println!("Doing stuff");
println!("There are {} items.", TEST.len());
for func in TEST {
func();
}
}
#[cfg(test)]
mod tests {
#[test]
fn it_works() {
crate::do_stuff();
}
}
use linkme::distributed_slice;
use crate::TEST;
#[distributed_slice(TEST)]
fn dependency_test() {
println!("Dependency");
}
[package]
name = "toplevel"
version = "0.1.0"
edition = "2018"
[dependencies]
dependency = { path = "../dependency" }
linkme = "0.2"
use dependency::do_stuff;
mod other;
fn main() {
do_stuff();
}
#[cfg(test)]
mod tests {
#[test]
fn it_works() {
dependency::do_stuff();
}
}
use linkme::distributed_slice;
use dependency::TEST;
#[distributed_slice(TEST)]
fn root_test() {
println!("Root");
}
As I have referenced the same distributed slice between crates, I would expect that running my toplevel should print:
Doing stuff
There are 2 items.
Dependency
Root
Running the toplevel gives me:
Doing stuff
There are 1 items.
Root
And running the tests gives me:
Compiling dependency v0.1.0 (/home/kcuzner/Projects/Rust/linktest/dependency)
Compiling toplevel v0.1.0 (/home/kcuzner/Projects/Rust/linktest/toplevel)
Finished test [unoptimized + debuginfo] target(s) in 0.41s
Running target/debug/deps/dependency-eafa826fb0a16142
running 1 test
Doing stuff
There are 1 items.
Dependency
test tests::it_works ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Running target/debug/deps/toplevel-48a62a711f6372bb
running 1 test
Doing stuff
There are 1 items.
Root
test tests::it_works ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Doc-tests dependency
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
running 2 tests
test declaration::test_slice ... FAILED
test declaration::test_functions ... FAILED
failures:
---- declaration::test_slice stdout ----
thread 'declaration::test_slice' panicked at 'assertion failed: !SLICE.is_empty()', tests/custom_linkme_path.rs:12:9
---- declaration::test_functions stdout ----
thread 'declaration::test_functions' panicked at 'assertion failed: !FUNCTIONS.is_empty()', tests/custom_linkme_path.rs:21:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
declaration::test_functions
declaration::test_slice
For reasons, backwards compatibility mainly, rather than have my list of plugins be a list of function pointers, I would like them to be a list of PluginFactories and I'd like them to be created lazily. In code terms:
#[linkme::distributed_slice]
pub static PLUGINS: [Lazy<PluginFactory>] = [..];
is my list of plugins.
Each plugin registers from a macro with something like:
#[linkme::distributed_slice(PLUGINS)]
static REGISTER_PLUGIN: Lazy<PluginFactory> =
Lazy::new(|| PluginFactory::new::<$plugin_type>($group, $name));
I then work with PLUGINS whenever I need access to my plugins. If I never access my PLUGINS, then the Lazy code is never invoked. (BTW: Lazy is nice from a performance point of view, but I'm mainly using it to workaround the limitation that the distributed slice must be const/static.)
That works fine on OS X, Linux (arm and amd), but doesn't work on windows. The code compiles and runs, but on windows none of my plugins are registered and so I get runtime errors about missing plugins.
Before I dig into this too much. Am I expecting too much of linkme and once_cell? Is this a reasonable thing to do or am I just lucky that it works on non-windows platforms and really this is just a silly thing to try?
I'm concerned that trying to finesse away the requirement for const/static
via Lazy is a potential source of problems. I wonder if I'm just lucky with the initialisation order on non-windows platforms.
I am able to trigger a STATUS_ACCESS_VIOLATION
or panic
in some cases on my Windows machine. It is highly dependent upon the surrounding code. I have only been able to trigger it in release mode so far, not debug mode. The simplest reproduction case I've found so far was to add the following to a file in this repo's tests
directory and run cargo test --release
. If I find a simpler, stand alone case, I'll post it.
use linkme::distributed_slice;
pub struct Item {
pub name: &'static str,
}
#[distributed_slice]
pub static ITEMS: [Item] = [..];
macro_rules! make_items {
($($name:ident = $str:literal), +) => {
$(
#[distributed_slice(ITEMS)]
static $name: Item = Item {
name: $str,
};
)+
}
}
make_items! {
ITEM1 = "item1",
ITEM2 = "item2",
ITEM3 = "item3"
}
#[test]
fn test_failure() {
let mut last_address = None;
for item in ITEMS {
let address = item as *const Item as usize;
// If these 3 lines are commented out there is no STATUS_ACCESS_VIOLATION
// below. Instead you get an invalid utf8 error.
use std::io::Write as _;
println!("{:x} {}", address, item.name);
std::io::stdout().flush().unwrap();
if let Some(last) = last_address {
assert_eq!(address - last, std::mem::size_of::<Item>());
}
last_address = Some(address);
// When run with `cargo test --release`, this causes either a
// STATUS_ACCESS_VIOLATION or panic depending on what lines are commented
// out above. Note, that if it panics, it prints out the name properly.
// When this is commented out, everything runs fine and the test passes.
let _ = std::str::from_utf8(item.name.as_bytes())
.unwrap_or_else(|err| panic!("invalid utf8: {:?}: {:?}", item.name, err));
}
}
The errors I get when running cargo test --release
look like either of the following:
running 1 test
error: test failed, to rerun pass `--test status_access_violation`
Caused by:
process didn't exit successfully: `target\release\deps\status_access_violation-0071020cc9b8bc0c.exe` (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)
running 1 test
test test_failure ... FAILED
failures:
---- test_failure stdout ----
thread 'test_failure' panicked at 'invalid utf8: "item3": Utf8Error { valid_up_to: 6, error_len: Some(1) }', tests\status_access_violation.rs:47:33
stack backtrace:
...
I've seen it with linkme
versions 0.3.9 and master and Rust versions 1.70 and 1.71.0. I haven't looked at any other versions.
Information about my machine:
OS Name: Microsoft Windows 11 Home
OS Version: 10.0.22621 N/A Build 22621
System Model: ZenBook UX434IQ_UM433IQ
System Type: x64-based PC
Processor(s): AMD Ryzen 7 4000 series
I am happy to help debug/test things.
I stumbled upon this crate on r/rust. I haven't tried it because I'm currently not on a machine that supports ASAN (arm64), but quickly looking at the code, combined with my experience maintaining something similar in Firefox, this crate likely fails in bad ways when building with ASAN. What happens in that case is that each item actually ends up larger than their size, to allow ASAN to do its magic. Which means a) the list is actually larger (in bytes) than expected b) there's a bunch of 0 bytes in between items. In Firefox, what was being aggregated was pointers, so the nulls between items were the same size as the items themselves, but I wouldn't be surprised if the size of the nulls doesn't match with larger items.
This works:
static ELEMENT: &str = "...";
and this works:
#[distributed_slice(MYSLICE)]
static ELEMENT: &str = "...";
but this does not work:
#[distributed_slice]
pub static MYSLICE: [&str] = [..];
error[E0106]: missing lifetime specifier
--> src/main.rs:5:22
|
5 | pub static MYSLICE: [&str] = [..];
| ^ expected named lifetime parameter
We should figure out the context in the generated code in which elision is not kicking in, and explicitly insert 'static
on all references there.
I'm trying to write a macro that put elements into a distribute_slice. One hard thing is that i have to name the individual items...
I feel it would be more convenient if using _ as the name of anonymous slice element is supported (distribute_slice(XYZ) can replace it with whatever name it feels best i think).
The generated code for an distributed slice:
#[distributed_slice]
pub static MYSLICE: [&'static str] = [..];
currently contains something like:
#[cfg(any(target_os = "none", target_os = "linux", target_os = "macos", target_os = "ios", target_os = "tvos", target_os = "illumos", target_os = "freebsd"))]
extern "C" {
#[link_name = "__start_linkme_MYSLICE"]
static LINKME_START: <[&'static str] as ::linkme::private::Slice>::Element;
#[link_name = "__stop_linkme_MYSLICE"]
static LINKME_STOP: <[&'static str] as ::linkme::private::Slice>::Element;
}
For identical code written by hand, ordinarily Rust would warn about the use of a repr(Rust) type in an extern "C" block. Maybe this warning is suppressed in macro-generated code, but anyway changing the block from extern "C"
to extern "Rust"
would eliminate it. These statics here only exist for interaction with other Rust compilation units, not C.
warning: `extern` block uses type `str`, which is not FFI-safe
--> src/main.rs
|
| static LINKME_START: <[&'static str] as ::linkme::private::Slice>::Element;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not FFI-safe
|
= note: `#[warn(improper_ctypes)]` on by default
= help: consider using `*const u8` and a length instead
= note: string slices have no C equivalent
Reproduction:
#[distributed_slice]
pub static SLICE: [&'static str];
#[distributed_slice(SLICE)]
static ELEMENT1: &&str = &"uhoh";
#[distributed_slice(SLICE)]
static ELEMENT2: &&str = &"ohno";
fn main() {
eprintln!("{:?}", SLICE[0].as_bytes().as_ptr_range());
// Possible output: 0x7ff75a903c88..0xffeeb5207928
// That's 140 TB of memory just available to read
}
Culprit: DistributedSlice::private_typecheck(self, T)
accepts an argument of type &&str
and coerces it to &str
.
Following the example in the readme results in a warning like:
warning: the type `for<'r> fn(...)` does not permit being left uninitialized
this code causes undefined behavior when executed
help: use `MaybeUninit<T>` instead
I'm using a recent nightly rustc. Presumably this doesn't actually cause any issues because the typecheck part is never called?
I created a minimal repro project at https://github.com/crlf0710/linkme_bug_repro/
To reproduce: run cargo run
, the program will panic (the distributed slice is empty), while it should be not (there is one data entry in module sub
within the inner
crate.)
Interestingly, if you comment the module sub
part out (lifting the definition to toplevel), the program will work fine.
On my machine the example test fails to link on lld 8.0:
note: ld.lld: error: can't create dynamic relocation R_X86_64_64 against local symbol in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in /.../debug/deps/example-56d39e659ef87805.3p5k1gwcovuwpn4a.rcgu.o
>>> referenced by 3p5k1gwcovuwpn4a
>>> /.../debug/deps/example-56d39e659ef87805.3p5k1gwcovuwpn4a.rcgu.o:(example::BENCH_DESERIALIZE::h28653d3b093a8cd3)
There's some nuance to this that I'm not really up to diving into right now.
In my ~/.cargo/config.toml
, I have the following entry:
[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-Clink-arg=-fuse-ld=lld"]
to improve link times and iteration speed. However, this seems to break linkme compilation, badly.
Here's a snippet from the error message during compilation:
= note: ld.lld: error: undefined symbol: __start_linkme_BENCHMARKS
>>> referenced by 29qlfu0bgl5pi1ii
>>> /home/meatball/coding/linkme/target/debug/deps/example-d6ac051010cd24ec.29qlfu0bgl5pi1ii.rcgu.o:(example::BENCHMARKS::h84fb1b0801cd0cb3)
>>> the encapsulation symbol needs to be retained under --gc-sections properly; consider -z nostart-stop-gc (see https://lld.llvm.org/ELF/start-stop-gc)
ld.lld: error: undefined symbol: __stop_linkme_BENCHMARKS
>>> referenced by 29qlfu0bgl5pi1ii
>>> /home/meatball/coding/linkme/target/debug/deps/example-d6ac051010cd24ec.29qlfu0bgl5pi1ii.rcgu.o:(example::BENCHMARKS::h84fb1b0801cd0cb3)
ld.lld: error: undefined symbol: __start_linkm2_BENCHMARKS
>>> referenced by 29qlfu0bgl5pi1ii
>>> /home/meatball/coding/linkme/target/debug/deps/example-d6ac051010cd24ec.29qlfu0bgl5pi1ii.rcgu.o:(example::BENCHMARKS::h84fb1b0801cd0cb3)
>>> the encapsulation symbol needs to be retained under --gc-sections properly; consider -z nostart-stop-gc (see https://lld.llvm.org/ELF/start-stop-gc)
ld.lld: error: undefined symbol: __stop_linkm2_BENCHMARKS
>>> referenced by 29qlfu0bgl5pi1ii
>>> /home/meatball/coding/linkme/target/debug/deps/example-d6ac051010cd24ec.29qlfu0bgl5pi1ii.rcgu.o:(example::BENCHMARKS::h84fb1b0801cd0cb3)
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
The current master branch fails to build on FreeBSD 14.0-CURRENT amd64, even though it still works on FreeBSD stable/13.
$ cargo test
...
= note: ld: error: undefined symbol: __start_linkme_SHENANIGANS
>>> referenced by x4z07wsgcl2z4i4
>>> /usr/home/somers/src/rust/linkme/target/debug/deps/distributed_slice-425b465e5440679a.x4z07wsgcl2z4i4.rcgu.o:(distributed_slice::SHENANIGANS::hca25c6519cd4df0f)
>>> the encapsulation symbol needs to be retained under --gc-sections properly; consider -z nostart-stop-gc (see https://lld.llvm.org/ELF/start-stop-gc)
ld: error: undefined symbol: __stop_linkme_SHENANIGANS
>>> referenced by x4z07wsgcl2z4i4
>>> /usr/home/somers/src/rust/linkme/target/debug/deps/distributed_slice-425b465e5440679a.x4z07wsgcl2z4i4.rcgu.o:(distributed_slice::SHENANIGANS::hca25c6519cd4df0f)
ld: error: undefined symbol: __start_linkme_EMPTY
>>> referenced by x4z07wsgcl2z4i4
>>> /usr/home/somers/src/rust/linkme/target/debug/deps/distributed_slice-425b465e5440679a.x4z07wsgcl2z4i4.rcgu.o:(distributed_slice::test_empty::EMPTY::h31c7f9e37773eb8f)
>>> the encapsulation symbol needs to be retained under --gc-sections properly; consider -z nostart-stop-gc (see https://lld.llvm.org/ELF/start-stop-gc)
ld: error: undefined symbol: __stop_linkme_EMPTY
>>> referenced by x4z07wsgcl2z4i4
>>> /usr/home/somers/src/rust/linkme/target/debug/deps/distributed_slice-425b465e5440679a.x4z07wsgcl2z4i4.rcgu.o:(distributed_slice::test_empty::EMPTY::h31c7f9e37773eb8f)
ld: error: undefined symbol: __start_linkme_NONCOPY
>>> referenced by x4z07wsgcl2z4i4
>>> /usr/home/somers/src/rust/linkme/target/debug/deps/distributed_slice-425b465e5440679a.x4z07wsgcl2z4i4.rcgu.o:(distributed_slice::test_non_copy::NONCOPY::h920eb5b4a050a94d)
>>> the encapsulation symbol needs to be retained under --gc-sections properly; consider -z nostart-stop-gc (see https://lld.llvm.org/ELF/start-stop-gc)
ld: error: undefined symbol: __stop_linkme_NONCOPY
>>> referenced by x4z07wsgcl2z4i4
>>> /usr/home/somers/src/rust/linkme/target/debug/deps/distributed_slice-425b465e5440679a.x4z07wsgcl2z4i4.rcgu.o:(distributed_slice::test_non_copy::NONCOPY::h920eb5b4a050a94d)
cc: error: linker command failed with exit code 1 (use -v to see invocation)
error: could not compile `linkme` due to previous error
error: could not compile `linkme` due to previous error
error: could not compile `linkme` due to previous error
Currently accessing a distributed slice under Miri makes the interpreter abort.
test readme ... error: unsupported operation: `extern` static `BENCHMARKS::LINKME_START` from crate `example` is not supported by Miri
--> tests/example.rs:22:19
|
22 | for _bench in BENCHMARKS { /* ... */ }
| ^^^^^^^^^^ `extern` static `BENCHMARKS::LINKME_START` from crate `example` is not supported by Miri
|
= help: this is likely not a bug in the program; it indicates that the program performed an operation that the interpreter does not support
= note: BACKTRACE:
= note: inside `readme` at tests/example.rs:22:19: 22:29
It would be reasonable to use cfg(miri)
to produce a Miri-compatible stub implementation that always contains no elements.
Currently, the linkme macro accepts an usize sort key. Although it is not documented, I assume this controls the ordering of the elements in the final slice. I was planing to use this to guarantee a deterministic ordering of entries between builds by supplying a hash of the item as the sort key. Otherwise, I end up with different orderings between release and debug builds. Unfortunately, the sort key is limited to a maximum of 9999, which is too small to avoid hash collisions.
It would be neat to have more control over the ordering of items.
Declaring a static
to be defined downstream seems right in the "linker shenanigans" alley of the linkme crate, and can be made (mostly) typesafe using essentially the same linker techniques as with #[distributed_slice]
.
I'm most interested in the static RESOURCE: dyn Trait
case (c.f. #[distributed_slice]
using static SLICE: [T]
), but any concrete type that can be put in a static
could of course be used.
If you agree that this would be a reasonable fit for the linkme crate to provide, I'm willing to work on getting a working implementation.
// upstream
pub trait Log {
fn record(&self, entry: &Record<'_>);
fn flush(&self);
}
#[global_resource]
pub static GLOBAL_LOGGER: dyn Log;
// downstream
#[global_resource(GLOBAL_LOGGER)]
static LOGGER: MyLogger = MyLogger::new();
There are two main potential implementation strategies. The first is to use a DistributedSlice<&dyn Log>
with a runtime assert_eq!(len(), 1)
(like the distributed slice dupcheck). Reusing the #[distibuted_slice]
machinery is a reason for putting this in linkme.
The second one is to use a single symbol instead, e.g.
// upstream
static GLOBAL_LOGGER: GlobalResource<dyn Log> = {
extern {
#[link_name = "__linkme.resource.GLOBAL_LOGGER"]
static LINKME: &dyn Log;
}
GlobalResource::private_new(LINKME, todo!())
};
// downstream
static LOGGER: MyLogger = MyLogger::new();
const _: () = {
#[link_name = "__linkme.resource.GLOBAL_LOGGER"]
static LINKME: &dyn Log = &LOGGER;
};
(attributes aren't exactly correct). An advantage of this approach is that it (potentially) works under Miri, but a big disadvantage is that it relies on linker errors for missing or duplicated definition of the linked symbol. Using weak symbols might be able to avoid this.
As a variation, instead of linking static: &dyn Log
, link fn() -> &dyn Log
instead. That could be more well supported and allows the downstream to more directly do lazy initialization, which could be an upside or drawback, depending on what you want.
In either case, the ability to provide a default implementation is useful (e.g. how #[global_allocator]
defaults to as-if Global
is used) and would theoretically be possible: a strongly ordered slice member for the former implementation strategy, and weak symbol definition for the latter.
As far as I can tell, if a distributed slice is declared anywhere beside the crate root, the element macro can't be used from any other scope.
Here's a simple example, adapted from the distributed_slice
test:
mod declaration {
use linkme::distributed_slice;
#[distributed_slice]
pub static SLICE: [i32] = [..];
}
mod usage {
use linkme::distributed_slice;
#[distributed_slice(crate::declaration::SLICE)]
pub static N: i32 = 9;
}
mod alt_usage {
use linkme::distributed_slice;
use crate::declaration::SLICE;
#[distributed_slice(SLICE)]
pub static N: i32 = 9;
}
Both usage
and alt_usage
fail to compile:
$ cargo test
Compiling linkmetest v0.1.0 (/home/daboross/linkmetest)
error[E0433]: failed to resolve: could not find `SLICE` in `declaration`
--> src/main.rs:11:45
|
11 | #[distributed_slice(crate::declaration::SLICE)]
| ^^^^^ could not find `SLICE` in `declaration`
error: cannot find macro `SLICE!` in this scope
--> src/main.rs:20:25
|
20 | #[distributed_slice(SLICE)]
| ^^^^^
|
= help: have you added the `#[macro_use]` on the module/import?
warning: unused import: `crate::declaration::SLICE`
--> src/main.rs:18:9
|
18 | use crate::declaration::SLICE;
| ^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: `#[warn(unused_imports)]` on by default
error: aborting due to 2 previous errors
For more information about this error, try `rustc --explain E0433`.
error: Could not compile `linkmetest`.
To learn more, run the command again with --verbose.
I'm assuming this case simply isn't supported. But since this seems potentially quite useful, I'm opening this issue to track it.
I believe the cause for this is the usage of macro_rules!
macros, and the fact that when exported using #[macro_export]
, they're always exported from the crate root, regardless of the module they're declared in. This works as long as the static is also exported from the crate root, but as soon as it's declared in a sub-module, it's hard to make this work.
error: distributed_slice is not implemented for this platform
Fuchsia is much like linux, so hopefully this is kind of straightforward. I think Chromium folks can provide a PR to do this (in Sept).
As of the most recent nightly, rustc 1.42.0-nightly (01a46509a 2019-12-20), our Windows builder in Travis has begun failing. https://travis-ci.com/dtolnay/linkme/jobs/269503401
Both cargo test
and cargo test --release
fail on Windows with the following:
thread 'test' panicked at 'assertion failed: `(left == right)`
left: `0`,
right: `3`', tests\distributed_slice.rs:17:5
The relevant code in the test file is:
linkme/tests/distributed_slice.rs
Lines 3 to 17 in 5376b49
Mentioning @kabergstrom in case this is something you might be able to look into.
Allow this syntax:
#[distributed_slice]
static RES: [Res; 0] = [];
as equivalent to
#[distributed_slice]
static RES: [Res];
so CLion won't complain about syntax it does not understand.
It's unclear whether WASM is supported. I assume it's probably not, but it would be informative if this was mentioned either way
Running `qemu-system-arm -cpu cortex-m3 -machine lm3s6965evb -nographic -semihosting-config enable=on,target=native -kernel /home/runner/work/linkme/linkme/target/thumbv7m-none-eabi/release/cortex-m-linkme-test`
panicked at 'assertion failed: `(left == right)`
left: `0`,
right: `3`', tests/cortex/src/main.rs:25:5
This appears to be a regression from nightly-2021-08-21 to nightly-2021-08-22. A couple older nightlies I spot checked all work, whereas 2021-08-22 fails consistently.
This is the error message:
LLVM ERROR: Global variable '_ZN3tex12section_026570_LINKME_ELEMENT_put_each_of_tex_s_primitivies_into_the_hash_table_026517hd67dd207d79119b3E' has an invalid section specifier '__DATA,__PUT_EACH_OF_TEX_S_PRIMITIVIES_INTO_THE_HASH_TABLE': mach-o section specifier requires a section whose length is between 1 and 16 characters.
It seems there's a limit on the section name length.
Building registries of function pointers is probably the most common use case for distributed_slice.
The benchmark harness use case in the readme is one example, or the following registry of error type conversions for crates to register semantically meaningful conversions into Python errors for their custom error types.
use cpython::{exc, PyErr, Python};
use distributed_slice::distributed_slice;
use std::io;
#[distributed_slice]
pub static INTO_PYERR: [fn(&anyhow::Error, py: Python<'_>) -> Option<PyErr>] = [..];
#[distributed_slice(INTO_PYERR)]
static IO_INTO_PYERR: fn(&anyhow::Error, py: Python<'_>) -> Option<PyErr> = io_into_pyerr;
fn io_into_pyerr(err: &anyhow::Error, py: Python<'_>) -> Option<PyErr> {
let err = err.downcast_ref::<io::Error>()?;
Some(PyErr::new::<exc::IOError, _>(
py,
(e.raw_os_error(), e.to_string()),
))
}
It would be great to handle function pointers as a special case in our attribute macro to allow this more concise invocation:
#[distributed_slice(INTO_PYERR)]
fn io_into_pyerr(err: &anyhow::Error, py: Python<'_>) -> Option<PyErr> {
...
}
which would expand to the pair of IO_INTO_PYERR static + io_into_pyerr function as above.
@arcnmx do you have any tips for how to set this up? Having some kind of test would help me avoid accidentally breaking your code.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.