matklad / arbtest Goto Github PK

Hi there, I have some async code which I want to call with data generated from arbtest. Is this supported - I can't figure out how it could be right now because the closure to run needs to be async.

For example, I'm trying to write:

            #[tokio::test]
            async fn prop_does_not_crash() {
                arbtest::builder().budget_ms(1_000).run(|u| {
                    let mut sut = $sut();
                    let i: usize = u.arbitrary()?;
                    // sanity - skip if it already exists
                    if !sut.fetch(&[i]).unwrap().is_empty() {
                        return Ok(());
                    }
                    let expected: Vec<u8> = u.arbitrary().unwrap();

                    // ***************** ASYNC 
                    sut.persist(i, expected.clone()).await.unwrap();
                    // ***************** ASYNC 
                    let actual = sut.fetch(&[i]).unwrap();
                    assert_eq!(HashMap::from([(i, expected)]), actual);
                    Ok(())
                })
            }

this obviously fails because the closure to run isn't async and so can't have await inside it. Changing it to an async |u| isn't on stable and I'm not sure that would actually help anyway as I would think (but don't really know) that arbtest should be involved in driving the async workflow as well....

Help :-)

Hi, thanks for the nice library!
I've started using it recently and now expanding it to my work related and private projects.

What's desired

Would it be possible to make builder.run() accept a real closure instead of

pub type Property = fn(u: &mut Unstructured<'_>) -> arbitrary::Result<()>;

In other words, something like:

pub trait Property: FnMut(&mut Unstructured<'_>) -> arbitrary::Result<()> {
}

pub fn run<P: Property>(mut self, prop: P) {
    // implementation..
}

Why it would be useful

One of my tests needs to be ran with some predefined IDs, that come from another place (in particular from DB).

Why it's not possible at the moment

I've tried to do the desired change and open a PR, but I see that it narrows down to catch_unwind, which I could not workaround:

    fn fails(&mut self, seed: Seed, prop: Property) -> bool {
        let this = AssertUnwindSafe(self);
        std::panic::catch_unwind(move || {            // <- FnMut cannot be moved here
            let that = this;
            that.0.single_run(seed, prop)
        })
        .is_err()
    }

Hi - is there a reason arbtest::arbitrary is publicly re-exported? I keep running into an issue where my IDE producesuse arbtest::arbitrary::{Self, Unstructured} instead of use arbitrary::Unstructured. I tried to work around this by fully qualifying the #[derive(arbitrary::Arbitrary)] but it just pulls in use arbtest::arbitrary::Self.

Is the intention for me to only depend on arbtest as oppose to both arbtest and arbitrary? At the moment I'm pulling in arbitrary as a dependency as well.

Thanks! (It's entirely possible I'm doing something dumb...)

To my surprise this test doesn't fail (I ran it ~10 times):

#[test]
fn my_test() {
    arbtest::arbtest(|u| {
        let x: Vec<u8> = u.arbitrary()?;
        assert!(x.len() < 10);
        Ok(())
    })
    .budget_ms(10000) // Try really hard
    .size_min(100000) // Provide a lot of bytes 
    .size_max(100000000);
}

I know, property-based testing is very limited. But this limit seems to be quite small (in comparison to something like quickcheck). And the limit can't be extended easily by increasing the budget or size parameters.

I checked the implementation of Arbitrary for Vec in the arbitrary crate:

impl<'a, A: Arbitrary<'a>> Arbitrary<'a> for Vec<A> {
    fn arbitrary(u: &mut Unstructured<'a>) -> Result<Self> {
        u.arbitrary_iter()?.collect()
    }
    ...
}

And here the implementation of the Iterator returned by arbitrary_iter:

impl<'a, 'b, ElementType: Arbitrary<'a>> Iterator for ArbitraryIter<'a, 'b, ElementType> {
    type Item = Result<ElementType>;
    fn next(&mut self) -> Option<Result<ElementType>> {
        let keep_going = self.u.arbitrary().unwrap_or(false);
        if keep_going {
            Some(Arbitrary::arbitrary(self.u))
        } else {
            None
        }
    }
}

As far as I know arbtest is using random bytes for Unstructured (instead of using a feedback loop like a fuzzer). Based on the implementation above I can understand why the generated Vecs are so small. But still, it doesn't feel right. Did I do something wrong?

Tested with arbtest 0.3.1 and arbitrary 1.3.2.

I've noticed that if I define a tests as a prop and run it in arbtest, my tracing setup with the test-log crate does not appear when running a test like this RUST_LOG=trace cargo test -- --nocapture, tests that are not run within arbtest produce output just fine with the same setup.

Is there something to configure with arbtest to allow the output to be sent to stdout?