CI: MemorySanitizer is erroring (ASAN job) about rust-bitcoin

[tcharding]

I'm debugging now, posted the issue for any tips or pointers. I've never debugged memory sanitizer bugs before.

[tcharding]

Putting llvm-symbolizer in my path gives better output (I just symlinked to llvm-symbolizer-15)

[tcharding]

I was unable to reslove this today, here are my findings:

• The call chain is bitcoin::PublicKey::from_slice -> secp256k1::PublicKey::from_slice -> ffi::secp256k1_ec_pubkey_parse
• secp256k1_ec_pubkey_parse uses data.as_c_ptr() to get a pointer to the data array (unsigned char *)
• Accessing the first element of the data array triggers the MSAN error
• Right before the FFI call the data slice is non-empty

            // Sanity, just ensure we can access the first element here.
            if data[0] == 0xff {
                panic!("first element of data is 0xff")
            }
            let pointer = data.as_c_ptr();
            if pointer.is_null() {
                panic!("pointer is null");
            }

No further ideas ATM.

[tcharding]

Had another go, I'm hesitant to say it but it might be a bug in the MemorySanitazer? Or else something odd is going on. I changed rust-secp256k1/secp256k1-sys/depend/secp256k/src/secp256k1.c to be:

int rustsecp256k1_v0_9_2_ec_pubkey_parse(const rustsecp256k1_v0_9_2_context* ctx, rustsecp256k1_v0_9_2_pubkey* _pubkey, const unsigned char *input, size_t inputlen) {
    rustsecp256k1_v0_9_2_ge Q;

    VERIFY_CHECK(ctx != NULL);

    rustsecp256k1_v0_9_2_pubkey pk;
    rustsecp256k1_v0_9_2_pubkey *pubkey = &pk;

    if (pubkey == NULL) {
        return 1;
    }

    ARG_CHECK(pubkey != NULL);
    memset(pubkey, 0, sizeof(*pubkey));
    ARG_CHECK(input != NULL);

    if (input == NULL) {
        return 20;
    }

    if (input[0] == 4) {
        return 0;
    }

    /*
      * Array access inside this function call is causing MSAN error but the array access above is ok
      * 
      * This makes me think the bug is not ours because there should be no problem passing 
      * an arg to another function within C code.
      * */
    if (!rustsecp256k1_v0_9_2_eckey_pubkey_parse(&Q, input, inputlen)) {
        return 0;
    }
    if (!rustsecp256k1_v0_9_2_ge_is_in_correct_subgroup(&Q)) {
        return 0;
    }
    rustsecp256k1_v0_9_2_pubkey_save(pubkey, &Q);
    rustsecp256k1_v0_9_2_ge_clear(&Q);
    return 1;
}

[tcharding]

@apoelstra if you get a chance can you give this ten minutes and give some pointers on which direction to look please?

[tcharding]

ooo, it might not be the pub arg at all - debugging now. Posting incase you get here at the same time.

[tcharding]

MSAN also errors for size_t arguments, even when I replace the usize on the otherside of the FFI boundry with a local variable.

Change key.rs lines 556 - 567 to be (ie, line 560 is the FFI function call)

        let mut ret = [0_u8; 66];
        let mut ret_len = 66;

        let res = unsafe {
            ffi::secp256k1_ec_pubkey_serialize(
                ffi::secp256k1_context_no_precomp,
                ret.as_mut_c_ptr(),
                &mut ret_len,
                self.as_c_ptr(),
                flag,
            )
        };

Still gives

==892789==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x5591f749a2b7 in rustsecp256k1_v0_9_2_ec_pubkey_serialize /home/tobin/build/github.com/tcharding/rust-secp256k1/master/secp256k1-sys/depend/secp256k1/src/secp256k1.c:279:5
    #1 0x5591f6881b09 in secp256k1::key::PublicKey::serialize_internal::hd1155a157cd04844 /home/tobin/build/github.com/tcharding/rust-secp256k1/master/src/key.rs:560:13
    #2 0x5591f6881b09 in secp256k1::key::PublicKey::serialize_uncompressed::hc4d3025786944379 /home/tobin/build/github.com/tcharding/rust-secp256k1/master/src/key.rs:550:9

[apoelstra]

I cannot repro. When I try to run the test I get

FATAL: Code 0x60c539d809b0 is out of application range. Non-PIE build?
FATAL: MemorySanitizer can not mmap the shadow memory.
FATAL: Make sure to compile with -fPIE and to link with -pie.
FATAL: Disabling ASLR is known to cause this error.
FATAL: If running under GDB, try 'set disable-randomization off'.

[apoelstra]

Running in gdb everything seems fine. If I run in valgrind it says ==1124717== Warning: set address range perms: large range [0x10000000000, 0x100000000000) (defined) then apperas to lock up.

[apoelstra]

I wonder if we are hitting bitcoin-core/secp256k1#1506

[real-or-random]

I wonder if we are hitting bitcoin-core/secp256k1#1506

Yes, this really looks like google/sanitizers#1614

[apoelstra]

In the linked libsecp issue fanquake says that the upstream problem should be fixed (in github actions), so maybe we can re-enable msan.

But can maybe wait til after the release because futzing with CI is time-consuming and annoying.