conditional expressions, repeated lines and matrix columns bugs about cbmc

[theyoucheng]

The errors are as follows: 1. Unless I've missed something here, Line 17 is executed just in case line 16 is (the conditional expression is executed just in case its body is), and so the suspiciousness should be the same - this is not the case however given the first two reports of the fault localisation scores report the body as having different scores (impossible if they always execute together). Note that correcting this will also mean sbo performs better by putting the bug in top equal place.

No. Line 17 and line 16 are not always executed at the same time. ``` line 16: while((i>=0) && (v[i]>key)) { line 17: if (i<2) // BUG: should be eliminated line 17 line 18: v[i+1] = v[i]; line 19: i = i - 1; line 20: } ``` Image the scenario when "v[i]<=key". In this case, the evaluation result of "(i>=0) && (v[i]>key)" will be false, and line 17 will not be executed. However, where does the evaluation result come from? That is, the "execution" of line 16. Right? With best regards, Youcheng

…

-- Youcheng Sun Research Assistant Department of Computer Science University of Oxford Wolfson Building, Parks Road Oxford OX1 3QD https://sites.google.com/site/theyoucheng/

[theyoucheng]

1. Each line of code should only refer to one column in the finalised matrix. This is not the case for lines 16 and 23 for example. This property is essential for the correct functioning of the measures (in particular pfl). The correct specification for the matrix is as follows: for each row in the matrix, a line of code should have a 1/0 in that row just in case that line of code was executed at some (any) point during the execution which in turn corresponds to that row.

Let's consider the following piece of code ``` line 16: while((i>=0) && (v[i]>key)) { line 17: if (i<2) // BUG: should be eliminated line 17 line 18: v[i+1] = v[i]; line 19: i = i - 1; line 20: } ``` If we unwind this while loop 3 times, we obtain if ((i>=0) && (v[i]>key)) { if (i<2) // BUG: should be eliminated line 17 v[i+1] = v[i]; i = i - 1; if ((i>=0) && (v[i]>key)) { if (i<2) // BUG: should be eliminated line 17 v[i+1] = v[i]; i = i - 1; if ((i>=0) && (v[i]>key)) { if (i<2) // BUG: should be eliminated line 17 v[i+1] = v[i]; i = i - 1; } } } As a matter of fact, the fault localization is applied in this unwound copy of the program. It seems that the same line repeats multiple times in the matrix. However, they actually represent different lines of code in the unwound program. In the final report of fault localization results, to avoid confusing users, when there are multiple appearances of the same line, only the one with the highest faulty probability is displayed. With best regards, Youcheng

…

-- Youcheng Sun Research Assistant Department of Computer Science University of Oxford Wolfson Building, Parks Road Oxford OX1 3QD https://sites.google.com/site/theyoucheng/

[quiveringlemon]

WRT problem 1 - i meant to say line 18 executes just in case line 17 is true (this is true because line 18 is in the body of the conditional statement given at line 17) - they should then have the same suspiciousness which they don't.

WRT problem 2 - i'm worried about this. As we are still at the testing phase, we still want to test pfl according to specification (it has been shown to be a lot more accurate than sbo in my HVC paper). However, pfl makes its assignments of fault probability according to the coverage matrix, where this coverage matrix must conform to the specification I described where each line of code gets one column in the coverage matrix. I am presuming then with the current setup that pfl will make its assessments of fault probability given the matrix provided - in which each unwinding of the loop gets a new column. Now, very roughly speaking (and without having to get into the statistical particularities of pfl), the probability a covered program artefact will be the bug in a given row of the matrix will be 1/n where n is the number of 1s in the row of that matrix (that's at least how i wrote the C++ code which i passed to you and I gather is implemented in the current branch). Suppose the execution in question only covers a single faulty line of code, and that happens to be a line of code in the loop. Then, if we have unwound a given loop 1000 times, and each of these unwindings features in the matrix and each unwinding is executed, then the current setup will output a fault probability of that line being faulty as 1/1000 - which is completely wrong (the correct probability is 1, as it is the only line executed in an error trace). Thus, in general it is very important the coverage matrix which is input to the fault localisation measures behaves according to specification - simply removing repeated lines in the suspiciousness report won't solve this problem.

[theyoucheng]

WRT problem 1 - i meant to say line 18 executes just in case line 17 is true (this is true because line 18 is in the body of the conditional statement given at line 17) - they should then have the same suspiciousness which they don't.

What's the reason for them to have the same suspiciousness? Line 17 and Line 18 are not inside the same block. It may happen that Line 17 is executed but Line 18 is not.

…

-- Youcheng Sun Research Assistant Department of Computer Science University of Oxford Wolfson Building, Parks Road Oxford OX1 3QD https://sites.google.com/site/theyoucheng/

[theyoucheng]

WRT problem 2 - i'm worried about this. As we are still at the testing phase, we still want to test pfl according to specification (it has been shown to be a lot more accurate than sbo in my HVC paper). However, pfl makes its assignments of fault probability according to the coverage matrix, where this coverage matrix must conform to the specification I described where each line of code gets one column in the coverage matrix. I am presuming then with the current setup that pfl will make its assessments of fault probability given the matrix provided - in which each unwinding of the loop gets a new column. Now, very roughly speaking (and without having to get into the statistical particularities of pfl), the probability a covered program artefact will be the bug in a given row of the matrix will be 1/n where n is the number of 1s in the row of that matrix (that's at least how i wrote the C++ code which i passed to you and I gather is implemented in the current branch). Suppose the execution in question only covers a single faulty line of code, and that happens to be a line of code in the loop. Then, if we have unwound a given loop 1000 times, and each of these unwindings features in the matrix and each unwinding is executed, then the current setup will output a fault probability of that line being faulty as 1/1000 - which is completely wrong (the correct probability is 1, as it is the only line executed in an error trace). Thus, in general it is very important the coverage matrix which is input to the fault localisation measures behaves according to specification - simply removing repeated lines in the suspiciousness report won't solve this problem.

If so, I'll then merge these repeated lines when constructing the matrix.

…

-- Youcheng Sun Research Assistant Department of Computer Science University of Oxford Wolfson Building, Parks Road Oxford OX1 3QD https://sites.google.com/site/theyoucheng/

[quiveringlemon]

WRT problem 1. I think I see what is going on now - I was for some reason presuming that line 17 (and indeed any conditional expression) gets a 1 in the matrix just in case i) flow of control evaluates the truth value of the condition in the given execution and ii) that condition evaluates to true, and 0 if i) but not ii) holds. I see what is actually happening is it gets a 1 in the matrix whenever i) holds simpliciter.

If I have understood what is going on in the implementation I think your solution is probably the better one, but it suffers from the problem that any condition which is always evaluated simply turns up as a vertical column of 1s in the coverage matrix and therefore fault localisation data becomes pretty uninformative.
In contrast there is also a problem with the alternative proposal above: that is, if a programmer creates a bug in a conditional expression which makes it unsatisfiable, then it will never turn up in the matrix at all and any fault localisation method will be ineffective.

I think for the purposes of our experiments we can leave this issue perhaps. But in the future I think the ideal solution would be to have two columns in the matrix for each conditional expression - one column gets 1 just in case the given expression evaluates to true and is executed, the other gets 1 just in case it evaluates to false and is executed. This way different evaluations of conditional expressions can have different degrees of suspiciousness, and both problems outlined above are avoided. However, for the moment I imagine adding this distinction to the implementation is more trouble that it is worth.

[theyoucheng]

On 6/22/17 3:02 PM, Youcheng Sun wrote: > WRT problem 2 - i'm worried about this. As we are still at the > testing phase, we still want to test pfl according to specification > (it has been shown to be a lot more accurate than sbo in my HVC > paper). However, pfl makes its assignments of fault probability > according to the coverage matrix, where this coverage matrix must > conform to the specification I described where each line of code gets > one column in the coverage matrix. I am presuming then with the > current setup that pfl will make its assessments of fault probability > given the matrix provided - in which each unwinding of the loop gets > a new column. Now, very roughly speaking (and without having to get > into the statistical particularities of pfl), the probability a > covered program artefact will be the bug in a given row of the matrix > will be 1/n where n is the number of 1s in the row of that matrix > (that's at least how i wrote the C++ code which i passed to you and I > gather is implemented in the current branch). Suppose the execution > in question only covers a single faulty line of code, and that > happens to be a line of code in the loop. Then, if we have unwound a > given loop 1000 times, and each of these unwindings features in the > matrix and each unwinding is executed, then the current setup will > output a fault probability of that line being faulty as 1/1000 - > which is completely wrong (the correct probability is 1, as it is the > only line executed in an error trace). Thus, in general it is very > important the coverage matrix which is input to the fault > localisation measures behaves according to specification - simply > removing repeated lines in the suspiciousness report won't solve this > problem. > > If so, I'll then merge these repeated lines when constructing the matrix.

Done. Please try it now. Interestingly, this time line 17 (the BUG) is indeed ranked as the most suspicious! Thanks. Youcheng

…

-- Youcheng Sun Research Assistant Department of Computer Science University of Oxford Wolfson Building, Parks Road Oxford OX1 3QD https://sites.google.com/site/theyoucheng/

[quiveringlemon]

WRT conditional expressions - some of my benchmarks now perform substantially worse, where previously the bug was reported as the most suspicious. Here is the command

cbmc byte_add_false-unreach-call_true-no-overflow.c --incremental --stop-on-fail --unwind 8 --localize-faults --localize-faults-max-traces 20 --localize-faults-method sbo --localize-faults-max-display 100 --verbosity 0 --slice-formula

Here is the output (the bug is the conditional expression at 76):

** Most likely fault location:
Fault localization scores:
[main.assertion.1]: Single Bug Optimal Fault Localization
[score: 8.55] ##file byte_add_false-unreach-call_true-no-overflow.c line 77 function mp_add
[score: 8.1] ##file byte_add_false-unreach-call_true-no-overflow.c line 62 function mp_add
[score: 8.05] ##file byte_add_false-unreach-call_true-no-overflow.c line 65 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 66 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 67 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 68 function mp_add
[score: 8.05] ##file byte_add_false-unreach-call_true-no-overflow.c line 71 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 72 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 73 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 74 function mp_add
[score: 8.05] ##file byte_add_false-unreach-call_true-no-overflow.c line 107 function main ##file byte_add_false-unreach-call_true-no-overflow.c line 108 function main ##file byte_add_false-unreach-call_true-no-overflow.c line 110 function main ##file byte_add_false-unreach-call_true-no-overflow.c line 31 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 32 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 33 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 34 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 35 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 36 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 37 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 38 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 40 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 50 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 61 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 64 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 70 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 76

The bug is the conditional expression at line 76, which is now very low on the report. However, in a previous implementation, it was top equal (1st) along with 77. 77 is executed just one line after the conditional expression is executed at line 76.

To solve this (and problems like it), I suggested two possible solutions earlier in this thread:

1. we would have two columns in the matrix for each conditional expression, one for when it evaluates to true, the other for false. If this was implemented then 77 would have the same suspiciousness as 76 and the bug would be reported top equal suspicious. In general, this seems like the right thing to do in any case. In the current scenario, conditional expressions which are always evaluated turn up as always true in the matrix, where presumably we want to measure the suspiciousness of the events in which the conditional expression are evaluated to true/false respectively.

2. the conditional expression only gets 1 in the matrix if it evaluates to true in the execution. We ignore it when it is false.

1 i think is the preferred and correct solution, but 2 might be more of a quick fix - but that said I don't know how that will perform down the line. Would either of the these take a long time to implement?

[quiveringlemon]

On further analysis, either of the above solutions would mean 17 and 18 get the same suspiciousness for insertion_sort, which means the results aren't significantly affected for that benchmark (both would be top equal which is still an excellent result)

[theyoucheng]

On 6/28/17 10:56 PM, quiveringlemon wrote: WRT conditional expressions - some of my benchmarks now perform substantially worse, where previously the bug was reported as the most suspicious. Here is the command cbmc byte_add_false-unreach-call_true-no-overflow.c --incremental --stop-on-fail --unwind 8 --localize-faults --localize-faults-max-traces 20 --localize-faults-method sbo --localize-faults-max-display 100 --verbosity 0 --slice-formula Here is the output (the bug is the conditional expression at 76): ** Most likely fault location: Fault localization scores: [main.assertion.1]: Single Bug Optimal Fault Localization [score: 8.55] ##file byte_add_false-unreach-call_true-no-overflow.c line 77 function mp_add [score: 8.1] ##file byte_add_false-unreach-call_true-no-overflow.c line 62 function mp_add [score: 8.05] ##file byte_add_false-unreach-call_true-no-overflow.c line 65 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 66 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 67 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 68 function mp_add [score: 8.05] ##file byte_add_false-unreach-call_true-no-overflow.c line 71 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 72 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 73 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 74 function mp_add [score: 8.05] ##file byte_add_false-unreach-call_true-no-overflow.c line 107 function main ##file byte_add_false-unreach-call_true-no-overflow.c line 108 function main ##file byte_add_false-unreach-call_true-no-overflow.c line 110 function main ##file byte_add_false-unreach-call_true-no-overflow.c line 31 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 32 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 33 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 34 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 35 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 36 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 37 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 38 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 40 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 50 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 61 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 64 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 70 function mp_add ##file byte_add_false-unreach-call_true-no-overflow.c line 76 The bug is the conditional expression at line 76, which is now very low on the report. However, in a previous implementation, it was top equal (1st) along with 77. 77 is executed just one line after the conditional expression is executed at line 76. To solve this (and problems like it), I suggested two possible solutions earlier in this thread: 1. we would have two columns in the matrix for each conditional expression, one for when it evaluates to true, the other for false. If this was implemented then 77 would have the same suspiciousness as 76 and the bug would be reported top equal suspicious. In general this seems like the right thing to do in any case. In the current scenario, conditional expressions which are always evaluated turn up as always true in the matrix, where presumably we want to measure the suspiciousness of the events in which the conditional expression are evaluated to true/false respectively. 2. the conditional expression only gets 1 in the matrix if it evaluates to true in the execution. We ignore it when it is false.

Option 2 is now implemented: 68da68a Now for two benchmarks we have, the BUG condition is ranked the most suspicious in both examples.

…

1. 1 i think is the preferred and correct solution, but 2 might be more of a quick fix - but that said I don't know how that will perform down the line. Would either of the these take a long time to implement? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATYfmxWsrhpdibswZtNOnZzCpavpxBTeks5sIswPgaJpZM4OAJO0>.

-- Youcheng Sun Research Assistant Department of Computer Science University of Oxford Wolfson Building, Parks Road Oxford OX1 3QD https://sites.google.com/site/theyoucheng/