Write the witness file for concurrent programs about cbmc

[tautschnig]

Am I right that a9e31b6 is a fix for this issue?

[peterschrammel]

That was a first attempt. Looking at the current results, this requires more work. Currently, I see only two witnesses confirmed. I'm looking into this right now.

[peterschrammel]

Stripping the witness produced by

./cbmc ../sv-benchmarks/c/pthread/fib_bench_false-unreach-call.i --propertyfile ../sv-benchmarks/c/ConcurrencySafety.prp --graphml-witness witness.graphml --32

down to

witness_works2.graphml.txt

seems to give something acceptable for cpachecker. I've also seen witness by other tools that just have entry and violation nodes... which is also accepted...

[marek-trtik]

I am currently working on witness validation of pthread-wmm/mix000_power.oepc_false-unreach-call.i.

[marek-trtik]

I'll try to fix these 2 issue first:

1. Missing <data key="specification">CHECK( init(main()), LTL(G ! call(__VERIFIER_error())) )</data> (CBMC does not write it, while ESBMC does)
2. Strange assumptions <data key="assumption">\result = FALSE;</data>

[tautschnig]

The specification field is inserted by the wrapper script, so it should really be there?

For the "strange" assumptions: which part of it do you consider "strange"?

[peterschrammel]

In my experience the strategy to solve witness problems is to manually modify the witnesses that we produce until they are accepted and only then to start implementing the changes necessary to produce the modified witness.

[peterschrammel]

Typical modifications that might lead to success are

• suppressing assumptions
• suppressing nodes

For example, in my example above, I removed all assumptions and kept only one node per context switch.

[marek-trtik]

@tautschnig :

1. The "missing specification" issue is not an issue. It is present. I did not know it is generated in the wrapper script.
2. The condition https://github.com/marek-trtik/cbmc/blob/sv-comp-2018-patches/src/goto-programs/graphml_witness.cpp#L164 does not for benchmarks containing variables with the $ character. For exmaple, int y$mem_tmp; in pthread-wmm/mix000_power.oepc_false-unreach-call.i. I fixed that by replacing $ by #return_value, according to description of assumption.resultfunction in page https://github.com/sosy-lab/sv-witnesses

[marek-trtik]

@peterschrammel : I applied the changes you proposed. And, for example, for pthread-wmm/mix000_power.oepc_false-unreach-call.i we now get this result
Verification result: TRUE. No property violation found by chosen configuration.
But the correct answer should be:
Verification result: FALSE. Property violation (WitnessAutomaton) found by chosen configuration.

[tautschnig]

I fixed that by replacing $ by #return_value

Will this also work for functions that don't have a body, such as __VERIFIER_nondet_*?

[marek-trtik]

@tautschnig : If you mean variables of the form return_value___VERIFIER_nondet_*$N, like is this GOTO program:

        // 168 file ../sv-benchmarks/c/pthread-wmm/mix000_power.oepc_false-unreach-call.i line 781 function P1
        void *return_value___VERIFIER_nondet_pointer$1;
        // 169 file ../sv-benchmarks/c/pthread-wmm/mix000_power.oepc_false-unreach-call.i line 781 function P1
        return_value___VERIFIER_nondet_pointer$1 = NONDET(void *);
        // 170 file ../sv-benchmarks/c/pthread-wmm/mix000_power.oepc_false-unreach-call.i line 781 function P1
        P1#return_value = return_value___VERIFIER_nondet_pointer$1;

then no. I can upgrade the condition to capture these variables as well.

[tautschnig]

It's ok if either of 169 or 170 results in a \result = ... assumption in the witness. It's not necessary to cover both of them, but please do make sure that one of them ends up in there.

[marek-trtik]

Thanks for the notes. To be sure we do not miss anything I upgraded the condition to capture both cases.

[marek-trtik]

Here is the commit that provides the improved validation of our witnesses:
68fee5b

I checked the improvement on these benchmarks:

"pthread-wmm/mix000_power.oepc_false-unreach-call.i",     # OK!
"pthread-wmm/mix000_power.opt_false-unreach-call.i",     # OK!
"pthread-wmm/rfi008_tso.opt_false-unreach-call.i",    # OK!
"pthread-lit/fkp2013_variant_false-unreach-call.i",       # PROBLEM! Verification result: UNKNOWN, incomplete analysis.
"ldv-races/race-1_3-join_false-unreach-call.i",       # OK!
"pthread-driver-races/char_generic_nvram_read_nvram_write_nvram_false-unreach-call.i",      # OK!
"pthread-C-DAC/pthread-demo-datarace_false-unreach-call.i",           # OK!
"pthread/lazy01_false-unreach-call.i",        # OK!
"pthread/stack_false-unreach-call.i",     # PROBLEM! Verification result: UNKNOWN, incomplete analysis. Error: Unsupported C feature (BDD-analysis does not support arrays: stack[__CPAchecker_TMP_1]) (BDDVectorCExpressionVisitor.visit, SEVERE)
"pthread-atomic/qrcu_false-unreach-call.i",           # OK!
"pthread-ext/28_buggy_simple_loop1_vf_false-unreach-call.i",      # PROBLEM! Verification result: UNKNOWN, incomplete analysis. Error: Unrecognized code (multiple thread assignments to same LHS not supported: t) (ThreadingTransferRelation.getNewThreadId, SEVERE)

We should re-evaluate all benchmarks before this commit is merged to the main branch.

[marek-trtik]

The fix/improvement of the \result guard condition is here:
9bdb376
Please, look at it, and if you have any suspicion that it might cause troubles for some benchmark, then comment here. Thanks.

[tautschnig]

Looks ok to me!

[tautschnig]

We should re-evaluate all benchmarks before this commit is merged to the main branch.

Do you have some reasonable way of doing so?

[marek-trtik]

Yes, I am in the progress of writing a Python script which will take all the benchmarks of that category, and for each of them it will call benchexec with 30s timeout and then CPAchecker to approve the witness. Hopefully it will work.

In this evaluation I'd like to use all today fixed we have (those to be merged to the main branch).

[tautschnig]

In an attempt to rule out at least one possible cause of trouble, I've added 476e65d to output 0 and 1 instead of FALSE/TRUE.

Yet I'm not sure how much of a difference that makes.