dtolnay / linkme Goto Github PK

View Code? Open in Web Editor NEW

607.0 607.0 42.0 453 KB

Safe cross-platform linker shenanigans

License: Apache License 2.0

Rust 100.00%

linkme's People

Contributors

Stargazers

Watchers

linkme's Issues

Document follow-up attributes

It seems like

#[distributed_slice(FOO)]
#[linkme(crate=::mycrate::linkme)]
static BENCH_DESERIALIZE: fn(&mut Bencher) = bench_deserialize;

causes distributed_slice to refer to the linkme crate as ::mycrate::linkme. But, I don't see this documented anywhere. I don't quite understand it well enough to write those docs myself :)

Cortex-M builds is failing with link error: section overlaps with .rodata

https://travis-ci.com/dtolnay/linkme/jobs/263379561

error: linking with `rust-lld` failed: exit code: 1
  |
  = note: "rust-lld" "rust-lld" "-flavor" "gnu" "-L" "/home/travis/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7m-none-eabi/lib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/cortex_m_linkme_test-e6719463d731e9f0.cortex_m_linkme_test.dqximbec-cgu.0.rcgu.o" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/cortex_m_linkme_test-e6719463d731e9f0.cortex_m_linkme_test.dqximbec-cgu.1.rcgu.o" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/cortex_m_linkme_test-e6719463d731e9f0.cortex_m_linkme_test.dqximbec-cgu.2.rcgu.o" "-o" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/cortex_m_linkme_test-e6719463d731e9f0" "--gc-sections" "-L" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps" "-L" "/home/travis/build/dtolnay/linkme/target/release/deps" "-L" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/build/cortex-m-linkme-test-200ea53de4bd4e09/out" "-L" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/build/cortex-m-31349c7505c00923/out" "-L" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/build/cortex-m-rt-4c8df37bd6f5b5e1/out" "-L" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/build/cortex-m-semihosting-2445422be57c0b84/out" "-L" "/home/travis/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7m-none-eabi/lib" "-Bstatic" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/liblinkme-cf1aa9b254dec488.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libcortex_m_rt-4e1c1e14bbef8ced.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libr0-1f53d375f373e972.rlib" "--start-group" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libpanic_semihosting-458c7d50d4f6de89.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libcortex_m_semihosting-8944461358d63d4a.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libcortex_m-7f4503edc7f5e4d5.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libvolatile_register-255db70b159ec0a4.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libvcell-b3554e83a40b7ef5.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libbare_metal-1297e03344968a12.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libaligned-7926c2ef4e454092.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libas_slice-7ff6256bbd72fde4.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libstable_deref_trait-550b77b954e6d921.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libgeneric_array-685219a2762e8d8f.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libgeneric_array-5204fbf5f82b726a.rlib" "/home/travis/build/dtolnay/linkme/target/thumbv7m-none-eabi/release/deps/libtypenum-94476904ccd9d85f.rlib" "/home/travis/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7m-none-eabi/lib/librustc_std_workspace_core-eafb71f8805df5a7.rlib" "/home/travis/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7m-none-eabi/lib/libcore-bfb368382075f384.rlib" "--end-group" "/home/travis/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/thumbv7m-none-eabi/lib/libcompiler_builtins-c6415c30ba62782a.rlib" "-Tlink.x" "-Bdynamic"
  = note: rust-lld: error: section linkme_SHENANIGANS file range overlaps with .rodata
          >>> linkme_SHENANIGANS range is [0x2604, 0x260F]
          >>> .rodata range is [0x2604, 0x29BF]
          
          rust-lld: error: section linkme_SHENANIGANS virtual address range overlaps with .rodata
          >>> linkme_SHENANIGANS range is [0x1604, 0x160F]
          >>> .rodata range is [0x1604, 0x19BF]
          
          rust-lld: error: section linkme_SHENANIGANS load address range overlaps with .rodata
          >>> linkme_SHENANIGANS range is [0x1604, 0x160F]
          >>> .rodata range is [0x1604, 0x19BF]

Disambiguate distributed slices with the same name

The only unique identifier in the link section name currently is the name of the static. Multiple crates could make distributed slices with the same name. We should do one of:

disambiguate using the crate name in which the static is declared,
trigger failure during compilation or linking,
detect the duplication with a runtime panic.

Feature: Way to force a 'last item'

I'm trying to use linkme as part of writing extensions for other languages, such as PHP. It's working very well for this case, with one exception: currently, I have to clone the slice to add a null item on the end for lists that must be null-terminated on the PHP end. It would be nice if there was a way to provide a guaranteed 'last item' when defining the list, or something similar.

Reject ZST

The crate currently doesn't really support ZSTs the way one might expect. We should just have these fail to compile if the size is zero.

Doesn't add elements to distributed slice from external crate on macOS

use linkme::distributed_slice;
use other_crate::BENCHMARKS;

#[distributed_slice(BENCHMARKS)]
static BENCH_DESERIALIZE: fn(&mut Bencher) = bench_deserialize;

fn bench_deserialize(b: &mut Bencher) {
    /* ... */
}

If BENCHMARKS is initialized in external crate the call to #[distributed_slice(BENCHMARKS)] doesn't add bench_deserialize function to the array. But if I turn on lto=true in Cargo.toml of the builded project (where BENCHMARKS is placed) lib works as expected.
I also tested this on ubuntu and it worked without lto

Linker error on Windows with lld-link: relocation against symbol in discarded section

it actually ends up crashing lld-link, but the errors come first, so I think it's also doing something wrong.

I tried with and without used_linker and get the same thing either way, whereas used_linker fixed lld on other platforms.

The GTESTS here is the distributed_slice, in the rust_gtest_interop crate.

[2/2] LINK rust_gtest_interop_unittests.exe rust_gtest_interop_unittests.exe.pdb
FAILED: rust_gtest_interop_unittests.exe rust_gtest_interop_unittests.exe.pdb
..\..\third_party\llvm-build\Release+Asserts\bin\lld-link.exe "/OUT:./rust_gtest_interop_unittests.exe" /nologo -libpath:..\..\third_party\llvm-build\Release+Asserts\lib\clang\17\lib\windows /winsysroot:../../third_party/depot_tools/win_toolchain/vs_files/27370823e7 /MACHINE:X64  "/PDB:./rust_gtest_interop_unittests.exe.pdb" "@./rust_gtest_interop_unittests.exe.rsp"
lld-link: error: relocation against symbol in discarded section: _ZN18rust_gtest_interop6GTESTS12LINKME_START17h75470cf2db5c947aE
>>> referenced by librust_gtest_interop.rlib(librust_gtest_interop.rust_gtest_interop.11482080eb92b0cf-cgu.0.rcgu.o):(rust_gtest_interop::GTESTS::h78f8c0ca7ebcfebd)
>>> referenced by librust_gtest_interop.rlib(librust_gtest_interop.rust_gtest_interop.11482080eb92b0cf-cgu.0.rcgu.o):(__declspec(dllimport) rust_gtest_interop::GTESTS::LINKME_START::h75470cf2db5c947a)

lld-link: error: relocation against symbol in discarded section: _ZN18rust_gtest_interop6GTESTS11LINKME_STOP17h3f94993507f6c1d0E
>>> referenced by librust_gtest_interop.rlib(librust_gtest_interop.rust_gtest_interop.11482080eb92b0cf-cgu.0.rcgu.o):(rust_gtest_interop::GTESTS::h78f8c0ca7ebcfebd)
>>> referenced by librust_gtest_interop.rlib(librust_gtest_interop.rust_gtest_interop.11482080eb92b0cf-cgu.0.rcgu.o):(__declspec(dllimport) rust_gtest_interop::GTESTS::LINKME_STOP::h3f94993507f6c1d0)

This should be reproducible on Windows outside of Chromium with --target=x86_64-pc-windows-msvc -Clinker=clang-cl -Clink-arg=-fuse-ld=lld.

WASM support

I investigated if it would be possible to add WASM support and it looks like there might be a way. While I have not found a similar start/end convention that works, WASM supports custom data sections which can be accessed as an array from JS. wasm_bindgen has a hidden and somewhat unsupported function that can get a handle to the WebAssembly.Module, while js-sys exposes bindings for fetching data from a custom section. This code compiles:

    let module = js_sys::WebAssembly::Module::new(&wasm_bindgen::module()).unwrap(); 
    let data = js_sys::WebAssembly::Module::custom_sections(&module, "thingies");

@ibaryshnikov created a more complete and runnable example here:
https://github.com/ibaryshnikov/wasm-custom-sections/blob/master/src/lib.rs

Unfortunately, webpack does not expose the WebAssembly.Module object in javascript:
webpack/webpack#8157

Here's another similar issue:
WebAssembly/esm-integration#14

So this approach seems to work, but the WebAssembly.Module handle is not exposed in certain cases. Better than nothing I guess? 😁

Distributed slice members in dependency crates are discarded

This probably isn't an issue with this crate per-se, but I wanted to bring up this use case just in case someone more versed in rust than me has a solution.

I have noticed that when I declare and construct a distributed_slice in a dependency crate, the entries in the slice declared in the dependency crates do not appear. This is probably due to rust-lang/rust#67209, which seems to point to some kind of linker issue that causes custom sections to disappear across crate boundaries. I don't fully understand that issue and it has been open for some time. I suspect that the issue I'm providing an example for here is just a consequence of linkme's use of custom sections (via link_section) and that bug. I've done a little inspecting of the output elf when I don't have a distributed slice member in a root binary crate (so all members are declared in dependency crates) and the custom section doesn't even appear in the output.

Again, I don't really expect a solution to appear in this crate since it is probably a deeper linker issue, but I wanted to make sure this was mentioned here and possibly advocate for this caveat to be called out on the documentation until it is fixed.

Example

Cargo.toml

[workspace]
members = [
    "toplevel",
    "dependency",
]

dependency/Cargo.toml

[package]
name = "dependency"
version = "0.1.0"
edition = "2018"

[dependencies]
linkme = "0.2"

dependency/src/lib.rs

use linkme::distributed_slice;

#[distributed_slice]
pub static TEST: [fn()] = [..];

mod other;

pub fn do_stuff() {
    println!("Doing stuff");
    println!("There are {} items.", TEST.len());
    for func in TEST {
        func();
    }
}


#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
        crate::do_stuff();
    }
}

dependency/src/other.rs

use linkme::distributed_slice;

use crate::TEST;

#[distributed_slice(TEST)]
fn dependency_test() {
    println!("Dependency");
}

toplevel/Cargo.toml

[package]
name = "toplevel"
version = "0.1.0"
edition = "2018"

[dependencies]
dependency = { path = "../dependency" }
linkme = "0.2"

toplevel/src/main.rs

use dependency::do_stuff;

mod other;

fn main() {
    do_stuff();
}

#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
        dependency::do_stuff();
    }
}

toplevel/src/other.rs

use linkme::distributed_slice;

use dependency::TEST;

#[distributed_slice(TEST)]
fn root_test() {
    println!("Root");
}

Expected Behavior

As I have referenced the same distributed slice between crates, I would expect that running my toplevel should print:

Doing stuff
There are 2 items.
Dependency
Root

Actual Behavior

Running the toplevel gives me:

Doing stuff
There are 1 items.
Root

And running the tests gives me:

   Compiling dependency v0.1.0 (/home/kcuzner/Projects/Rust/linktest/dependency)
   Compiling toplevel v0.1.0 (/home/kcuzner/Projects/Rust/linktest/toplevel)
    Finished test [unoptimized + debuginfo] target(s) in 0.41s
     Running target/debug/deps/dependency-eafa826fb0a16142

running 1 test
Doing stuff
There are 1 items.
Dependency
test tests::it_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

     Running target/debug/deps/toplevel-48a62a711f6372bb

running 1 test
Doing stuff
There are 1 items.
Root
test tests::it_works ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests dependency

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

macOS build is failing: SLICE.is_empty()

running 2 tests
test declaration::test_slice ... FAILED
test declaration::test_functions ... FAILED

failures:

---- declaration::test_slice stdout ----
thread 'declaration::test_slice' panicked at 'assertion failed: !SLICE.is_empty()', tests/custom_linkme_path.rs:12:9

---- declaration::test_functions stdout ----
thread 'declaration::test_functions' panicked at 'assertion failed: !FUNCTIONS.is_empty()', tests/custom_linkme_path.rs:21:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    declaration::test_functions
    declaration::test_slice

Should once_cell::sync::Lazy work with linkme?

For reasons, backwards compatibility mainly, rather than have my list of plugins be a list of function pointers, I would like them to be a list of PluginFactories and I'd like them to be created lazily. In code terms:

#[linkme::distributed_slice]
pub static PLUGINS: [Lazy<PluginFactory>] = [..];

is my list of plugins.

Each plugin registers from a macro with something like:

#[linkme::distributed_slice(PLUGINS)]
static REGISTER_PLUGIN: Lazy<PluginFactory> =
    Lazy::new(|| PluginFactory::new::<$plugin_type>($group, $name));

I then work with PLUGINS whenever I need access to my plugins. If I never access my PLUGINS, then the Lazy code is never invoked. (BTW: Lazy is nice from a performance point of view, but I'm mainly using it to workaround the limitation that the distributed slice must be const/static.)

That works fine on OS X, Linux (arm and amd), but doesn't work on windows. The code compiles and runs, but on windows none of my plugins are registered and so I get runtime errors about missing plugins.

Before I dig into this too much. Am I expecting too much of linkme and once_cell? Is this a reasonable thing to do or am I just lucky that it works on non-windows platforms and really this is just a silly thing to try?

I'm concerned that trying to finesse away the requirement for const/static via Lazy is a potential source of problems. I wonder if I'm just lucky with the initialisation order on non-windows platforms.

STATUS_ACCESS_VIOLATION on Windows MSVC

I am able to trigger a STATUS_ACCESS_VIOLATION or panic in some cases on my Windows machine. It is highly dependent upon the surrounding code. I have only been able to trigger it in release mode so far, not debug mode. The simplest reproduction case I've found so far was to add the following to a file in this repo's tests directory and run cargo test --release. If I find a simpler, stand alone case, I'll post it.

use linkme::distributed_slice;

pub struct Item {
    pub name: &'static str,
}

#[distributed_slice]
pub static ITEMS: [Item] = [..];

macro_rules! make_items {
    ($($name:ident = $str:literal), +) => {
        $(
            #[distributed_slice(ITEMS)]
            static $name: Item = Item {
                name: $str,
            };
        )+
    }
}
make_items! {
    ITEM1 = "item1",
    ITEM2 = "item2",
    ITEM3 = "item3"
}

#[test]
fn test_failure() {
    let mut last_address = None;
    for item in ITEMS {
        let address = item as *const Item as usize;

        // If these 3 lines are commented out there is no STATUS_ACCESS_VIOLATION
        // below. Instead you get an invalid utf8 error.
        use std::io::Write as _;
        println!("{:x} {}", address, item.name);
        std::io::stdout().flush().unwrap();

        if let Some(last) = last_address {
            assert_eq!(address - last, std::mem::size_of::<Item>());
        }
        last_address = Some(address);

        // When run with `cargo test --release`, this causes either a
        // STATUS_ACCESS_VIOLATION or panic depending on what lines are commented
        // out above. Note, that if it panics, it prints out the name properly. 
        // When this is commented out, everything runs fine and the test passes.
        let _ = std::str::from_utf8(item.name.as_bytes())
            .unwrap_or_else(|err| panic!("invalid utf8: {:?}: {:?}", item.name, err));
    }
}

The errors I get when running cargo test --release look like either of the following:

running 1 test
error: test failed, to rerun pass `--test status_access_violation`

Caused by:
  process didn't exit successfully: `target\release\deps\status_access_violation-0071020cc9b8bc0c.exe` (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)

running 1 test
test test_failure ... FAILED

failures:

---- test_failure stdout ----
thread 'test_failure' panicked at 'invalid utf8: "item3": Utf8Error { valid_up_to: 6, error_len: Some(1) }', tests\status_access_violation.rs:47:33
stack backtrace:
...

I've seen it with linkme versions 0.3.9 and master and Rust versions 1.70 and 1.71.0. I haven't looked at any other versions.

Information about my machine:

OS Name:                   Microsoft Windows 11 Home
OS Version:                10.0.22621 N/A Build 22621
System Model:              ZenBook UX434IQ_UM433IQ
System Type:               x64-based PC
Processor(s):              AMD Ryzen 7 4000 series

I am happy to help debug/test things.

Probable problem with ASAN

I stumbled upon this crate on r/rust. I haven't tried it because I'm currently not on a machine that supports ASAN (arm64), but quickly looking at the code, combined with my experience maintaining something similar in Firefox, this crate likely fails in bad ways when building with ASAN. What happens in that case is that each item actually ends up larger than their size, to allow ASAN to do its magic. Which means a) the list is actually larger (in bytes) than expected b) there's a bunch of 0 bytes in between items. In Firefox, what was being aggregated was pointers, so the nulls between items were the same size as the items themselves, but I wouldn't be surprised if the size of the nulls doesn't match with larger items.

Static lifetime elision

This works:

static ELEMENT: &str = "...";

and this works:

#[distributed_slice(MYSLICE)]
static ELEMENT: &str = "...";

but this does not work:

#[distributed_slice]
pub static MYSLICE: [&str] = [..];

error[E0106]: missing lifetime specifier
 --> src/main.rs:5:22
  |
5 | pub static MYSLICE: [&str] = [..];
  |                      ^ expected named lifetime parameter

We should figure out the context in the generated code in which elision is not kicking in, and explicitly insert 'static on all references there.

Feature request: support anonymous slice element.

I'm trying to write a macro that put elements into a distribute_slice. One hard thing is that i have to name the individual items...

I feel it would be more convenient if using _ as the name of anonymous slice element is supported (distribute_slice(XYZ) can replace it with whatever name it feels best i think).

Try using `extern "Rust"` for extern statics

The generated code for an distributed slice:

#[distributed_slice]
pub static MYSLICE: [&'static str] = [..];

currently contains something like:

#[cfg(any(target_os = "none", target_os = "linux", target_os = "macos", target_os = "ios", target_os = "tvos", target_os = "illumos", target_os = "freebsd"))]
extern "C" {
    #[link_name = "__start_linkme_MYSLICE"]
    static LINKME_START: <[&'static str] as ::linkme::private::Slice>::Element;
    #[link_name = "__stop_linkme_MYSLICE"]
    static LINKME_STOP: <[&'static str] as ::linkme::private::Slice>::Element;
}

For identical code written by hand, ordinarily Rust would warn about the use of a repr(Rust) type in an extern "C" block. Maybe this warning is suppressed in macro-generated code, but anyway changing the block from extern "C" to extern "Rust" would eliminate it. These statics here only exist for interaction with other Rust compilation units, not C.

warning: `extern` block uses type `str`, which is not FFI-safe
  --> src/main.rs
   |
   |         static LINKME_START: <[&'static str] as ::linkme::private::Slice>::Element;
   |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not FFI-safe
   |
   = note: `#[warn(improper_ctypes)]` on by default
   = help: consider using `*const u8` and a length instead
   = note: string slices have no C equivalent

Coercion bypasses DistributedSlice typecheck

Reproduction:

#[distributed_slice]
pub static SLICE: [&'static str];

#[distributed_slice(SLICE)]
static ELEMENT1: &&str = &"uhoh";

#[distributed_slice(SLICE)]
static ELEMENT2: &&str = &"ohno";

fn main() {
    eprintln!("{:?}", SLICE[0].as_bytes().as_ptr_range());
    // Possible output: 0x7ff75a903c88..0xffeeb5207928
    // That's 140 TB of memory just available to read
}

Culprit: DistributedSlice::private_typecheck(self, T) accepts an argument of type &&str and coerces it to &str.

example warning: the type does not permit being left uninitialized

Following the example in the readme results in a warning like:

warning: the type `for<'r> fn(...)` does not permit being left uninitialized
this code causes undefined behavior when executed
help: use `MaybeUninit<T>` instead

I'm using a recent nightly rustc. Presumably this doesn't actually cause any issues because the typecheck part is never called?

Data silently disappeared with certain configuration on Linux

I created a minimal repro project at https://github.com/crlf0710/linkme_bug_repro/

To reproduce: run cargo run, the program will panic (the distributed slice is empty), while it should be not (there is one data entry in module sub within the inner crate.)

Interestingly, if you comment the module sub part out (lifting the definition to toplevel), the program will work fine.

Can't create dynamic relocation against local symbol in readonly segment

On my machine the example test fails to link on lld 8.0:

note: ld.lld: error: can't create dynamic relocation R_X86_64_64 against local symbol in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in /.../debug/deps/example-56d39e659ef87805.3p5k1gwcovuwpn4a.rcgu.o
>>> referenced by 3p5k1gwcovuwpn4a
>>>               /.../debug/deps/example-56d39e659ef87805.3p5k1gwcovuwpn4a.rcgu.o:(example::BENCH_DESERIALIZE::h28653d3b093a8cd3)

There's some nuance to this that I'm not really up to diving into right now.

Compilation fails if using lld

In my ~/.cargo/config.toml, I have the following entry:

[target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-Clink-arg=-fuse-ld=lld"]

to improve link times and iteration speed. However, this seems to break linkme compilation, badly.

Here's a snippet from the error message during compilation:

  = note: ld.lld: error: undefined symbol: __start_linkme_BENCHMARKS                                                                                                                                                                                                                                                  
          >>> referenced by 29qlfu0bgl5pi1ii                                                                                                                                                                                                                                                                          
          >>>               /home/meatball/coding/linkme/target/debug/deps/example-d6ac051010cd24ec.29qlfu0bgl5pi1ii.rcgu.o:(example::BENCHMARKS::h84fb1b0801cd0cb3)                                                                                                                                                  
          >>> the encapsulation symbol needs to be retained under --gc-sections properly; consider -z nostart-stop-gc (see https://lld.llvm.org/ELF/start-stop-gc)                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                      
          ld.lld: error: undefined symbol: __stop_linkme_BENCHMARKS                                                                                                                                                                                                                                                   
          >>> referenced by 29qlfu0bgl5pi1ii                                                                                                                                                                                                                                                                          
          >>>               /home/meatball/coding/linkme/target/debug/deps/example-d6ac051010cd24ec.29qlfu0bgl5pi1ii.rcgu.o:(example::BENCHMARKS::h84fb1b0801cd0cb3)                                                                                                                                                  
                                                                                                                                                                                                                                                                                                                      
          ld.lld: error: undefined symbol: __start_linkm2_BENCHMARKS                                                                                                                                                                                                                                                  
          >>> referenced by 29qlfu0bgl5pi1ii                                                                                                                                                                                                                                                                          
          >>>               /home/meatball/coding/linkme/target/debug/deps/example-d6ac051010cd24ec.29qlfu0bgl5pi1ii.rcgu.o:(example::BENCHMARKS::h84fb1b0801cd0cb3)                                                                                                                                                  
          >>> the encapsulation symbol needs to be retained under --gc-sections properly; consider -z nostart-stop-gc (see https://lld.llvm.org/ELF/start-stop-gc)                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                      
          ld.lld: error: undefined symbol: __stop_linkm2_BENCHMARKS                                                                                                                                                                                                                                                   
          >>> referenced by 29qlfu0bgl5pi1ii                                                                                                                                                                                                                                                                          
          >>>               /home/meatball/coding/linkme/target/debug/deps/example-d6ac051010cd24ec.29qlfu0bgl5pi1ii.rcgu.o:(example::BENCHMARKS::h84fb1b0801cd0cb3)                                                                                                                                                  
          clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

"The encapsulation symbol needs to be retained under --gc-sections properly"

The current master branch fails to build on FreeBSD 14.0-CURRENT amd64, even though it still works on FreeBSD stable/13.

$ cargo test
...
  = note: ld: error: undefined symbol: __start_linkme_SHENANIGANS
          >>> referenced by x4z07wsgcl2z4i4
          >>>               /usr/home/somers/src/rust/linkme/target/debug/deps/distributed_slice-425b465e5440679a.x4z07wsgcl2z4i4.rcgu.o:(distributed_slice::SHENANIGANS::hca25c6519cd4df0f)
          >>> the encapsulation symbol needs to be retained under --gc-sections properly; consider -z nostart-stop-gc (see https://lld.llvm.org/ELF/start-stop-gc)
          
          ld: error: undefined symbol: __stop_linkme_SHENANIGANS
          >>> referenced by x4z07wsgcl2z4i4
          >>>               /usr/home/somers/src/rust/linkme/target/debug/deps/distributed_slice-425b465e5440679a.x4z07wsgcl2z4i4.rcgu.o:(distributed_slice::SHENANIGANS::hca25c6519cd4df0f)
          
          ld: error: undefined symbol: __start_linkme_EMPTY
          >>> referenced by x4z07wsgcl2z4i4
          >>>               /usr/home/somers/src/rust/linkme/target/debug/deps/distributed_slice-425b465e5440679a.x4z07wsgcl2z4i4.rcgu.o:(distributed_slice::test_empty::EMPTY::h31c7f9e37773eb8f)
          >>> the encapsulation symbol needs to be retained under --gc-sections properly; consider -z nostart-stop-gc (see https://lld.llvm.org/ELF/start-stop-gc)
          
          ld: error: undefined symbol: __stop_linkme_EMPTY
          >>> referenced by x4z07wsgcl2z4i4
          >>>               /usr/home/somers/src/rust/linkme/target/debug/deps/distributed_slice-425b465e5440679a.x4z07wsgcl2z4i4.rcgu.o:(distributed_slice::test_empty::EMPTY::h31c7f9e37773eb8f)
          
          ld: error: undefined symbol: __start_linkme_NONCOPY
          >>> referenced by x4z07wsgcl2z4i4
          >>>               /usr/home/somers/src/rust/linkme/target/debug/deps/distributed_slice-425b465e5440679a.x4z07wsgcl2z4i4.rcgu.o:(distributed_slice::test_non_copy::NONCOPY::h920eb5b4a050a94d)
          >>> the encapsulation symbol needs to be retained under --gc-sections properly; consider -z nostart-stop-gc (see https://lld.llvm.org/ELF/start-stop-gc)
          
          ld: error: undefined symbol: __stop_linkme_NONCOPY
          >>> referenced by x4z07wsgcl2z4i4
          >>>               /usr/home/somers/src/rust/linkme/target/debug/deps/distributed_slice-425b465e5440679a.x4z07wsgcl2z4i4.rcgu.o:(distributed_slice::test_non_copy::NONCOPY::h920eb5b4a050a94d)
          cc: error: linker command failed with exit code 1 (use -v to see invocation)
          

error: could not compile `linkme` due to previous error
error: could not compile `linkme` due to previous error
error: could not compile `linkme` due to previous error

Consider producing empty slices under cfg(miri)

Currently accessing a distributed slice under Miri makes the interpreter abort.

test readme ... error: unsupported operation: `extern` static `BENCHMARKS::LINKME_START` from crate `example` is not supported by Miri
  --> tests/example.rs:22:19
   |
22 |     for _bench in BENCHMARKS { /* ... */ }
   |                   ^^^^^^^^^^ `extern` static `BENCHMARKS::LINKME_START` from crate `example` is not supported by Miri
   |
   = help: this is likely not a bug in the program; it indicates that the program performed an operation that the interpreter does not support
   = note: BACKTRACE:
   = note: inside `readme` at tests/example.rs:22:19: 22:29

It would be reasonable to use cfg(miri) to produce a Miri-compatible stub implementation that always contains no elements.

Item ordering

Currently, the linkme macro accepts an usize sort key. Although it is not documented, I assume this controls the ordering of the elements in the final slice. I was planing to use this to guarantee a deterministic ordering of entries between builds by supplying a hash of the item as the sort key. Otherwise, I end up with different orderings between release and debug builds. Unfortunately, the sort key is limited to a maximum of 9999, which is too small to avoid hash collisions.

It would be neat to have more control over the ordering of items.

macOS support for distributed slice

Proposal: add a `#[disjointed_static]` attribute

Declaring a static to be defined downstream seems right in the "linker shenanigans" alley of the linkme crate, and can be made (mostly) typesafe using essentially the same linker techniques as with #[distributed_slice].

I'm most interested in the static RESOURCE: dyn Trait case (c.f. #[distributed_slice] using static SLICE: [T]), but any concrete type that can be put in a static could of course be used.

If you agree that this would be a reasonable fit for the linkme crate to provide, I'm willing to work on getting a working implementation.

Example

// upstream
pub trait Log {
    fn record(&self, entry: &Record<'_>);
    fn flush(&self);
}

#[global_resource]
pub static GLOBAL_LOGGER: dyn Log;

// downstream
#[global_resource(GLOBAL_LOGGER)]
static LOGGER: MyLogger = MyLogger::new();

Implementation strategy

There are two main potential implementation strategies. The first is to use a DistributedSlice<&dyn Log> with a runtime assert_eq!(len(), 1) (like the distributed slice dupcheck). Reusing the #[distibuted_slice] machinery is a reason for putting this in linkme.

The second one is to use a single symbol instead, e.g.

// upstream
static GLOBAL_LOGGER: GlobalResource<dyn Log> = {
    extern {
        #[link_name = "__linkme.resource.GLOBAL_LOGGER"]
        static LINKME: &dyn Log;
    }
    GlobalResource::private_new(LINKME, todo!())
};

// downstream
static LOGGER: MyLogger = MyLogger::new();
const _: () = {
    #[link_name = "__linkme.resource.GLOBAL_LOGGER"]
    static LINKME: &dyn Log = &LOGGER;
};

(attributes aren't exactly correct). An advantage of this approach is that it (potentially) works under Miri, but a big disadvantage is that it relies on linker errors for missing or duplicated definition of the linked symbol. Using weak symbols might be able to avoid this.

As a variation, instead of linking static: &dyn Log, link fn() -> &dyn Log instead. That could be more well supported and allows the downstream to more directly do lazy initialization, which could be an upside or drawback, depending on what you want.

In either case, the ability to provide a default implementation is useful (e.g. how #[global_allocator] defaults to as-if Global is used) and would theoretically be possible: a strongly ordered slice member for the former implementation strategy, and weak symbol definition for the latter.

Support referring to distributed slices declared in sub-modules

As far as I can tell, if a distributed slice is declared anywhere beside the crate root, the element macro can't be used from any other scope.

Here's a simple example, adapted from the distributed_slice test:

mod declaration {
    use linkme::distributed_slice;

    #[distributed_slice]
    pub static SLICE: [i32] = [..];
}

mod usage {
    use linkme::distributed_slice;

    #[distributed_slice(crate::declaration::SLICE)]
    pub static N: i32 = 9;
}

mod alt_usage {
    use linkme::distributed_slice;

    use crate::declaration::SLICE;

    #[distributed_slice(SLICE)]
    pub static N: i32 = 9;
}

Both usage and alt_usage fail to compile:

$ cargo test
   Compiling linkmetest v0.1.0 (/home/daboross/linkmetest)
error[E0433]: failed to resolve: could not find `SLICE` in `declaration`
  --> src/main.rs:11:45
   |
11 |     #[distributed_slice(crate::declaration::SLICE)]
   |                                             ^^^^^ could not find `SLICE` in `declaration`

error: cannot find macro `SLICE!` in this scope
  --> src/main.rs:20:25
   |
20 |     #[distributed_slice(SLICE)]
   |                         ^^^^^
   |
   = help: have you added the `#[macro_use]` on the module/import?

warning: unused import: `crate::declaration::SLICE`
  --> src/main.rs:18:9
   |
18 |     use crate::declaration::SLICE;
   |         ^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: `#[warn(unused_imports)]` on by default

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0433`.
error: Could not compile `linkmetest`.

To learn more, run the command again with --verbose.

I'm assuming this case simply isn't supported. But since this seems potentially quite useful, I'm opening this issue to track it.

I believe the cause for this is the usage of macro_rules! macros, and the fact that when exported using #[macro_export], they're always exported from the crate root, regardless of the module they're declared in. This works as long as the static is also exported from the crate root, but as soon as it's declared in a sub-module, it's hard to make this work.

Missing support for target_os = "fuchsia"

error: distributed_slice is not implemented for this platform

Fuchsia is much like linux, so hopefully this is kind of straightforward. I think Chromium folks can provide a PR to do this (in Sept).

Test failures on pc-windows-gnu

As of the most recent nightly, rustc 1.42.0-nightly (01a46509a 2019-12-20), our Windows builder in Travis has begun failing. https://travis-ci.com/dtolnay/linkme/jobs/269503401

Both cargo test and cargo test --release fail on Windows with the following:

thread 'test' panicked at 'assertion failed: `(left == right)`
  left: `0`,
 right: `3`', tests\distributed_slice.rs:17:5

The relevant code in the test file is:

linkme/tests/distributed_slice.rs

Lines 3 to 17 in 5376b49

 #[distributed_slice] 

 static SHENANIGANS: [i32] = [..]; 

 #[distributed_slice(SHENANIGANS)] 

 static N: i32 = 9; 

 #[distributed_slice(SHENANIGANS)] 

 static NN: i32 = 99; 

 #[distributed_slice(SHENANIGANS)] 

 static NNN: i32 = 999; 

 #[test] 

 fn test() { 

 assert_eq!(SHENANIGANS.len(), 3);

Mentioning @kabergstrom in case this is something you might be able to look into.

Allow valid Rust syntax

Allow this syntax:

#[distributed_slice]
static RES: [Res; 0] = [];

as equivalent to

#[distributed_slice]
static RES: [Res];

so CLion won't complain about syntax it does not understand.

Add compile-fail test to test the "mismatched types" error

Windows support for distributed slice

Document whether there is WASM support

It's unclear whether WASM is supported. I assume it's probably not, but it would be informative if this was mentioned either way

Cortex-M build is failing: 0 vs 3

     Running `qemu-system-arm -cpu cortex-m3 -machine lm3s6965evb -nographic -semihosting-config enable=on,target=native -kernel /home/runner/work/linkme/linkme/target/thumbv7m-none-eabi/release/cortex-m-linkme-test`
panicked at 'assertion failed: `(left == right)`
  left: `0`,
 right: `3`', tests/cortex/src/main.rs:25:5

This appears to be a regression from nightly-2021-08-21 to nightly-2021-08-22. A couple older nightlies I spot checked all work, whereas 2021-08-22 fails consistently.

Limit on section name length on macOS

This is the error message:

LLVM ERROR: Global variable '_ZN3tex12section_026570_LINKME_ELEMENT_put_each_of_tex_s_primitivies_into_the_hash_table_026517hd67dd207d79119b3E' has an invalid section specifier '__DATA,__PUT_EACH_OF_TEX_S_PRIMITIVIES_INTO_THE_HASH_TABLE': mach-o section specifier requires a section whose length is between 1 and 16 characters.

It seems there's a limit on the section name length.

Recognize distributed_slice on functions as a special case

Building registries of function pointers is probably the most common use case for distributed_slice.

The benchmark harness use case in the readme is one example, or the following registry of error type conversions for crates to register semantically meaningful conversions into Python errors for their custom error types.

use cpython::{exc, PyErr, Python};
use distributed_slice::distributed_slice;
use std::io;

#[distributed_slice]
pub static INTO_PYERR: [fn(&anyhow::Error, py: Python<'_>) -> Option<PyErr>] = [..];

#[distributed_slice(INTO_PYERR)]
static IO_INTO_PYERR: fn(&anyhow::Error, py: Python<'_>) -> Option<PyErr> = io_into_pyerr;

fn io_into_pyerr(err: &anyhow::Error, py: Python<'_>) -> Option<PyErr> {
    let err = err.downcast_ref::<io::Error>()?;
    Some(PyErr::new::<exc::IOError, _>(
        py,
        (e.raw_os_error(), e.to_string()),
    ))
}

It would be great to handle function pointers as a special case in our attribute macro to allow this more concise invocation:

#[distributed_slice(INTO_PYERR)]
fn io_into_pyerr(err: &anyhow::Error, py: Python<'_>) -> Option<PyErr> {
    ...
}

which would expand to the pair of IO_INTO_PYERR static + io_into_pyerr function as above.

Add CI build with target_os = "none"

@arcnmx do you have any tips for how to set this up? Having some kind of test would help me avoid accidentally breaking your code.

	#[distributed_slice]
	static SHENANIGANS: [i32] = [..];

	#[distributed_slice(SHENANIGANS)]
	static N: i32 = 9;

	#[distributed_slice(SHENANIGANS)]
	static NN: i32 = 99;

	#[distributed_slice(SHENANIGANS)]
	static NNN: i32 = 999;

	#[test]
	fn test() {
	assert_eq!(SHENANIGANS.len(), 3);

dtolnay / linkme Goto Github PK

linkme's People

Contributors

Stargazers

Watchers

Forkers

linkme's Issues

Example

Expected Behavior

Actual Behavior

Example

Implementation strategy

Recommend Projects

Recommend Topics

Recommend Org

Jobs