efficiency issue - calling model checker to find new traces which execute same LOC about cbmc

[quiveringlemon]

(exception to 3rd para above: the model checker is in fact allowed to find a trace which covers the same LOC as a previously found trace but only in the case when one is a failing trace and the other is a passing trace - this allows us to that rows of the matrix which are otherwise identical apart from one being passing and one being failing).

[quiveringlemon]

Also - how is it possible that there is a passing trace in the unsimplified matrix covers only one element, but all passing traces in the simplified matrix cover more than one element?

[theyoucheng]

OK. Now the same LOC (are merged in advance and) will only appear once when constructing the matrix : 3d2cf83 I do not know to what extent it can improve the efficiency. But it does seem to improve the accuracy. See the "kundu2" example: now the bug line is ranked among the most suspicious. Youcheng

…

On 6/23/17 8:43 PM, quiveringlemon wrote: I noticed that FCBMC was running a touch slowly on one of the simplest benchmarks when compared to CBMC (insertion_sort_safe.c : FCBMC = 14 secs, CBMC = 0.6 secs) - especially given the final matrix included only 4 traces - so I began investigating why... It seems as if the implementation is often calling the model checker to return traces which execute the same LOC as a previous trace returned by it, but differ insofar as the number of unwindings executed differ. Then, in the final simplified coverage matrix the data about these traces are then completely deleted. This can potentially be highly inefficient - for instance in the below example 14 traces are generated but only 4 are used in our actual analysis, which means that most of the traces should not be called for in the first place by the model checker. In general (and following the implementation described in my thesis), the specified implementation of the single bug optimal data generation algorithm should only ever call the model checker in order to find /a new trace which is different to all previous traces generated - where different means different in terms of LOC executed in the original faulty benchmark version/ (importantly - different does not mean different in terms of LOC executed in the unwound version!). In other words, it should never be possible to call the model checker and it return a trace which executes the same LOC as a trace which was previously returned by it. The command and output is as follows: cbmc insertion_sort_unsafe.c --incremental --stop-on-fail --unwind 4 --localize-faults --localize-faults-method sbo --localize-faults-max-display 100 --verbosity 0yes, got a failing trace failing traces: 1 1 1 1 1 1 1 1 1 0 0 0 0 1 0 0 0 1 0 1 1 1 0 1 1 1 1 1 1 1 0 0 0 0 1 0 0 0 1 0 1 1 1 0 1 1 0 1 1 1 0 0 0 0 0 1 0 0 0 1 0 1 1 1 0 0 1 0 1 1 1 0 0 0 0 0 1 0 0 0 1 0 1 1 passing traces: 1 0 1 1 1 1 1 1 0 0 1 0 0 1 0 1 0 1 0 1 1 1 0 1 1 0 1 1 1 0 0 1 0 0 1 0 1 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 s traces: 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 1 1 1 1 0 1 1 1 0 0 1 0 0 1 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1 0 0 1 0 0 1 0 1 0 1 0 1 1 1 1 1 1 0 1 1 1 1 0 1 0 0 1 0 1 0 1 0 1 1 1 0 1 1 0 1 1 1 1 0 1 0 0 1 0 1 0 1 0 1 1 1 0 1 1 1 1 1 1 1 0 1 0 0 1 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1 1 0 1 0 0 1 0 1 0 1 0 1 1 1 0 0 1 0 1 1 0 0 1 1 1 0 1 0 1 0 1 0 1 1 *** after simplification (blocks) *** [1] ##file insertion_sort_unsafe.c line 13 function main ##file insertion_sort_unsafe.c line 14 function main ##file insertion_sort_unsafe.c line 15 function main ##file insertion_sort_unsafe.c line 16 function main ##file insertion_sort_unsafe.c line 21 function main [3] ##file insertion_sort_unsafe.c line 18 function main ##file insertion_sort_unsafe.c line 19 function main [4972] ##file insertion_sort_unsafe.c line 17 function main [47627] ##file insertion_sort_unsafe.c line 23 function main ##file insertion_sort_unsafe.c line 24 function main ##file insertion_sort_unsafe.c line 3 function __VERIFIER_assert [true] ##file insertion_sort_unsafe.c line 10 function main failing traces: 1 1 1 1 1 passing traces: 1 1 1 1 1 1 0 0 1 1 1 1 0 1 1 /*after compute spectra*/ **ef: 1 1 1 1 1 **ep: 3 2 1 3 3 **nf: 0 0 0 0 0 **np: 0 1 2 0 0 /*after measure_sb*/ 0.25 0.5 0.75 0.25 0.25 ** Most likely fault location: Fault localization scores: [__VERIFIER_assert.assertion.1]: Single Bug Optimal Fault Localization [score: 0.75] ##file insertion_sort_unsafe.c line 17 function main [score: 0.5] ##file insertion_sort_unsafe.c line 18 function main ##file insertion_sort_unsafe.c line 19 function main [score: 0.25] ##file insertion_sort_unsafe.c line 13 function main ##file insertion_sort_unsafe.c line 14 function main ##file insertion_sort_unsafe.c line 15 function main ##file insertion_sort_unsafe.c line 16 function main ##file insertion_sort_unsafe.c line 21 function main [score: 0.25] ##file insertion_sort_unsafe.c line 23 function main ##file insertion_sort_unsafe.c line 24 function main ##file insertion_sort_unsafe.c line 3 function __VERIFIER_assert [score: 0.25] ##file insertion_sort_unsafe.c line 10 function main — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATYfm-tlR66GLaglP6WCaywFVcpMBH3Sks5sHBVSgaJpZM4OD9-_>.

-- Youcheng Sun Research Assistant Department of Computer Science University of Oxford Wolfson Building, Parks Road Oxford OX1 3QD https://sites.google.com/site/theyoucheng/

[theyoucheng]

On 6/23/17 8:59 PM, quiveringlemon wrote: Also - how is it possible that there is a passing trace in the unsimplified matrix covers only one element, but all passing traces in the simplified matrix cover more than one element?

Very good observation. In that case, some "merge" operation happened in between the two outputs. Now, we "merge" all repeated lines before constructing the matrix and there is no need to "merge" anything afterwards. Youcheng

…

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATYfm3sJgl0sIuIw_-09vJgUrrg_rTwfks5sHBk7gaJpZM4OD9-_>.

-- Youcheng Sun Research Assistant Department of Computer Science University of Oxford Wolfson Building, Parks Road Oxford OX1 3QD https://sites.google.com/site/theyoucheng/

[quiveringlemon]

WRT to the unwindings issue: I have updated FCBMC and the symptom of the problem remains - insertion sort is still 14 secs - this suggests to me that the problem hasn't been fixed - if FCBMC really is making fewer calls to the model checker (It appears there need to be at least ten less calls) we'd expect some reduction to this time. So, just to be clear - have you have fixed this problem by ensuring the model checker is not being called to find the ten erroneous executions of the program originally discussed, or is it simply the case that these original traces are now no longer reported but are still in fact generated by the model checker?

Just to emphasise the importance of this problem - Reporting 14 seconds for a such a tiny program (24 lines!) is a big problem, and undercuts the main claim of any paper (that fault localisation adds negligible time cost to CBMC), especially when CBMC takes under a second to generate 1 trace. It also doesn't seem to me to make much intuitive sense to me - there are only 4 traces reported, and around 3 branches of the program itself, which means a maximum of (2^3) 8 calls to a model checker to check for all failing traces (and an additional 8 to find all possible passing traces). Obviously if the problem hasn't been solved as per the solution above, and the model checker is finding different executions on different paths of the unwound program (unwound to lvl 4) then it can potentially be searching overs hundreds of different paths which is precisely what we want to avoid.

Another test is as follows: As there are only 4 possible executions of the program wrt the final matrix, we would expect FCBMC to work easily with 100 unwindings (in the same way as CBMC works almost instantly with 100 unwindings) - however this isn't happening - suggesting the original problem hasn't been fixed. Can you elaborate on how FCBMC was changed to solve the problem please?

[quiveringlemon]

WRT to the passing trace - i don't really understand what is going on here, but so long as the model checker never generates an execution which is not reported by the matrix (this is an extremely important condition required by the algorithm and efficiency considerations), then it is fine. Can we be assured that this is actually the case?

[quiveringlemon]

I've just had a look at the merge function in your code, if i understand what is going on here all that is going on is that traces that have been generated by the model checker have been merged. However that's the wrong solution and precisely what we want to avoid. The correct solution is to not call the model checker for these traces at all. In short, nothing should require merging.

[quiveringlemon]

WRT kundu2. My output is as follows (the fault is 428), so is quite far down the list. This must mean our outputs are different or you have a different buggy line, what is your output?

** Most likely fault location:
Fault localization scores:
[error.assertion.1]: Single Bug Optimal Fault Localization
[score: 1.71429] ##file kundu2_unsafe.cil.c line 75 function P_1
[score: 1.66667] ##file kundu2_unsafe.cil.c line 133 function P_2
[score: 1.42857] ##file kundu2_unsafe.cil.c line 71 function P_1
[score: 1.42857] ##file kundu2_unsafe.cil.c line 82 function P_1
[score: 1.38095] ##file kundu2_unsafe.cil.c line 129 function P_2
[score: 1.38095] ##file kundu2_unsafe.cil.c line 140 function P_2
[score: 1.19048] ##file kundu2_unsafe.cil.c line 475 function activate_threads
[score: 1.19048] ##file kundu2_unsafe.cil.c line 439 function reset_time_events ##file kundu2_unsafe.cil.c line 444 function reset_time_events ##file kundu2_unsafe.cil.c line 449 function reset_time_events
[score: 1.19048] ##file kundu2_unsafe.cil.c line 268 function is_C_1_triggered ##file kundu2_unsafe.cil.c line 269 function is_C_1_triggered
[score: 1.19048] ##file kundu2_unsafe.cil.c line 483 function activate_threads
[score: 1.19048] ##file kundu2_unsafe.cil.c line 267 function is_C_1_triggered
[score: 1.19048] ##file kundu2_unsafe.cil.c line 176 function is_P_2_triggered ##file kundu2_unsafe.cil.c line 177 function is_P_2_triggered
[score: 1.19048] ##file kundu2_unsafe.cil.c line 467 function activate_threads
[score: 1.19048] ##file kundu2_unsafe.cil.c line 332 function exists_runnable_thread
[score: 1.19048] ##file kundu2_unsafe.cil.c line 226 function C_1 ##file kundu2_unsafe.cil.c line 227 function C_1
[score: 1.19048] ##file kundu2_unsafe.cil.c line 235 function C_1 ##file kundu2_unsafe.cil.c line 236 function C_1 ##file kundu2_unsafe.cil.c line 237 function C_1 ##file kundu2_unsafe.cil.c line 238 function C_1 ##file kundu2_unsafe.cil.c line 241 function C_1 ##file kundu2_unsafe.cil.c line 26 function read_data
[score: 1.19048] ##file kundu2_unsafe.cil.c line 108 function is_P_1_triggered ##file kundu2_unsafe.cil.c line 109 function is_P_1_triggered
[score: 1.19048] ##file kundu2_unsafe.cil.c line 427 function fire_time_events ##file kundu2_unsafe.cil.c line 428 function fire_time_events ##file kundu2_unsafe.cil.c line 429 function fire_time_events ##file kundu2_unsafe.cil.c line 557 function start_simulation

[theyoucheng]

On 6/28/17 7:41 PM, quiveringlemon wrote: I've just had a look at the merge function in your code, if i understand what is going on here all that is going on is that traces that have been generated by the model checker have been merged. However that's the wrong solution and precisely what we want to avoid. The correct solution is to not call the model checker for these traces at all. In short, nothing should require merging.

Please have a look at the "pre_merge" method. The "merge" function is deprecated now.

…

— You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATYfm51hZEx_dt8ovJmaafkFg05tENEdks5sIp5LgaJpZM4OD9-_>.

-- Youcheng Sun Research Assistant Department of Computer Science University of Oxford Wolfson Building, Parks Road Oxford OX1 3QD https://sites.google.com/site/theyoucheng/

[theyoucheng]

Attached. ./cbmc-incr/src/cbmc/cbmc kundu2-origin.c --incremental --stop-on-fail --unwind 2 --localize-faults --localize-faults-method sbo --localize-faults-max-display 100 --localize-faults-max-traces 100 --verbosity 0 --beautify The bug is at line 581 ** Most likely fault location: Fault localization scores: [error.assertion.1]: Single Bug Optimal Fault Localization [score: 0.944444] ##file kundu2-origin.c line 131 function error ##file kundu2-origin.c line 188 function write_data [score: 0.916667] ##file kundu2-origin.c line 508 function eval ##file kundu2-origin.c line 509 function eval [score: 0.833333] ##file kundu2-origin.c line 274 function P_2 ##file kundu2-origin.c line 275 function P_2 [score: 0.777778] ##file kundu2-origin.c line 216 function P_1 ##file kundu2-origin.c line 217 function P_1 [score: 0.777778] ##file kundu2-origin.c line 218 function P_1 [score: 0.722222] ##file kundu2-origin.c line 156 function read_data ##file kundu2-origin.c line 157 function read_data [score: 0.722222] ##file kundu2-origin.c line 154 function read_data ##file kundu2-origin.c line 155 function read_data [score: 0.555556] ##file kundu2-origin.c line 581 function fire_time_events ##file kundu2-origin.c line 582 function fire_time_events ##file kundu2-origin.c line 583 function fire_time_events ##file kundu2-origin.c line 584 function fire_time_events ##file kundu2-origin.c line 712 function start_simulation [score: 0.555556] ##file kundu2-origin.c line 630 function activate_threads [score: 0.555556] ##file kundu2-origin.c line 422 function is_C_1_triggered ##file kundu2-origin.c line 423 function is_C_1_triggered [score: 0.555556] ##file kundu2-origin.c line 251 function is_P_1_triggered ##file kundu2-origin.c line 252 function is_P_1_triggered [score: 0.555556] ##file kundu2-origin.c line 622 function activate_threads [score: 0.555556] ##file kundu2-origin.c line 320 function is_P_2_triggered ##file kundu2-origin.c line 321 function is_P_2_triggered [score: 0.555556] ##file kundu2-origin.c line 481 function exists_runnable_thread ##file kundu2-origin.c line 486 function exists_runnable_thread [score: 0.555556] ##file kundu2-origin.c line 591 function reset_time_events ##file kundu2-origin.c line 593 function reset_time_events ##file kundu2-origin.c line 594 function reset_time_events ##file kundu2-origin.c line 598 function reset_time_events ##file kundu2-origin.c line 599 function reset_time_events ##file kundu2-origin.c line 603 function reset_time_events ##file kundu2-origin.c line 604 function reset_time_events ##file kundu2-origin.c line 715 function start_simulation [score: 0.472222] ##file kundu2-origin.c line 477 function exists_runnable_thread ##file kundu2-origin.c line 478 function exists_runnable_thread [score: 0.444444] ##file kundu2-origin.c line 638 function activate_threads [score: 0.416667] ##file kundu2-origin.c line 375 function C_1 [score: 0.416667] ##file kundu2-origin.c line 421 function is_C_1_triggered ##file kundu2-origin.c line 424 function is_C_1_triggered [score: 0.416667] ##file kundu2-origin.c line 145 function read_data ##file kundu2-origin.c line 151 function read_data ##file kundu2-origin.c line 378 function C_1 ##file kundu2-origin.c line 379 function C_1 ##file kundu2-origin.c line 380 function C_1 ##file kundu2-origin.c line 381 function C_1 ##file kundu2-origin.c line 382 function C_1 ##file kundu2-origin.c line 393 function C_1 [score: 0.416667] ##file kundu2-origin.c line 367 function C_1 ##file kundu2-origin.c line 370 function C_1 ##file kundu2-origin.c line 371 function C_1 [score: 0.388889] ##file kundu2-origin.c line 187 function write_data [score: 0.388889] ##file kundu2-origin.c line 276 function P_2 ##file kundu2-origin.c line 283 function P_2 [score: 0.388889] ##file kundu2-origin.c line 185 function write_data ##file kundu2-origin.c line 186 function write_data [score: 0.361111] ##file kundu2-origin.c line 473 function exists_runnable_thread ##file kundu2-origin.c line 474 function exists_runnable_thread [score: 0.361111] ##file kundu2-origin.c line 250 function is_P_1_triggered ##file kundu2-origin.c line 253 function is_P_1_triggered [score: 0.222222] ##file kundu2-origin.c line 319 function is_P_2_triggered ##file kundu2-origin.c line 322 function is_P_2_triggered [score: 0.0277778] ##file kundu2-origin.c line 666 function stop_simulation ##file kundu2-origin.c line 667 function stop_simulation [score: 0.0277778] ##file kundu2-origin.c line 660 function stop_simulation [score: 0.0277778] ##file kundu2-origin.c line 663 function stop_simulation ##file kundu2-origin.c line 665 function stop_simulation ##file kundu2-origin.c line 721 function start_simulation ##file kundu2-origin.c line 723 function start_simulation [score: 0.0277778] ##file kundu2-origin.c line 180 function write_data ##file kundu2-origin.c line 183 function write_data ##file kundu2-origin.c line 184 function write_data ##file kundu2-origin.c line 211 function P_1 ##file kundu2-origin.c line 214 function P_1 ##file kundu2-origin.c line 215 function P_1 ##file kundu2-origin.c line 225 function P_1 ##file kundu2-origin.c line 227 function P_1 ##file kundu2-origin.c line 228 function P_1 ##file kundu2-origin.c line 229 function P_1 ##file kundu2-origin.c line 230 function P_1 ##file kundu2-origin.c line 231 function P_1 ##file kundu2-origin.c line 234 function P_1 ##file kundu2-origin.c line 241 function P_1 ##file kundu2-origin.c line 247 function is_P_1_triggered ##file kundu2-origin.c line 249 function is_P_1_triggered ##file kundu2-origin.c line 256 function is_P_1_triggered ##file kundu2-origin.c line 259 function is_P_1_triggered ##file kundu2-origin.c line 269 function P_2 ##file kundu2-origin.c line 272 function P_2 ##file kundu2-origin.c line 273 function P_2 ##file kundu2-origin.c line 282 function P_2 ##file kundu2-origin.c line 285 function P_2 ##file kundu2-origin.c line 286 function P_2 ##file kundu2-origin.c line 287 function P_2 ##file kundu2-origin.c line 289 function P_2 ##file kundu2-origin.c line 291 function P_2 ##file kundu2-origin.c line 292 function P_2 ##file kundu2-origin.c line 294 function P_2 ##file kundu2-origin.c line 296 function P_2 ##file kundu2-origin.c line 299 function P_2 ##file kundu2-origin.c line 300 function P_2 ##file kundu2-origin.c line 302 function P_2 ##file kundu2-origin.c line 310 function P_2 ##file kundu2-origin.c line 315 function is_P_2_triggered ##file kundu2-origin.c line 318 function is_P_2_triggered ##file kundu2-origin.c line 325 function is_P_2_triggered ##file kundu2-origin.c line 328 function is_P_2_triggered ##file kundu2-origin.c line 340 function C_1 ##file kundu2-origin.c line 342 function C_1 ##file kundu2-origin.c line 343 function C_1 ##file kundu2-origin.c line 357 function C_1 ##file kundu2-origin.c line 359 function C_1 ##file kundu2-origin.c line 407 function is_C_1_triggered ##file kundu2-origin.c line 410 function is_C_1_triggered ##file kundu2-origin.c line 420 function is_C_1_triggered ##file kundu2-origin.c line 427 function is_C_1_triggered ##file kundu2-origin.c line 430 function is_C_1_triggered ##file kundu2-origin.c line 437 function update_channels ##file kundu2-origin.c line 444 function init_threads ##file kundu2-origin.c line 447 function init_threads ##file kundu2-origin.c line 448 function init_threads ##file kundu2-origin.c line 452 function init_threads ##file kundu2-origin.c line 453 function init_threads ##file kundu2-origin.c line 457 function init_threads ##file kundu2-origin.c line 458 function init_threads ##file kundu2-origin.c line 470 function exists_runnable_thread ##file kundu2-origin.c line 471 function exists_runnable_thread ##file kundu2-origin.c line 472 function exists_runnable_thread ##file kundu2-origin.c line 496 function eval ##file kundu2-origin.c line 501 function eval ##file kundu2-origin.c line 504 function eval ##file kundu2-origin.c line 506 function eval ##file kundu2-origin.c line 511 function eval ##file kundu2-origin.c line 513 function eval ##file kundu2-origin.c line 515 function eval ##file kundu2-origin.c line 517 function eval ##file kundu2-origin.c line 518 function eval ##file kundu2-origin.c line 526 function eval ##file kundu2-origin.c line 528 function eval ##file kundu2-origin.c line 530 function eval ##file kundu2-origin.c line 532 function eval ##file kundu2-origin.c line 533 function eval ##file kundu2-origin.c line 541 function eval ##file kundu2-origin.c line 543 function eval ##file kundu2-origin.c line 545 function eval ##file kundu2-origin.c line 547 function eval ##file kundu2-origin.c line 548 function eval ##file kundu2-origin.c line 566 function fire_delta_events ##file kundu2-origin.c line 574 function reset_delta_events ##file kundu2-origin.c line 616 function activate_threads ##file kundu2-origin.c line 619 function activate_threads ##file kundu2-origin.c line 621 function activate_threads ##file kundu2-origin.c line 623 function activate_threads ##file kundu2-origin.c line 627 function activate_threads ##file kundu2-origin.c line 629 function activate_threads ##file kundu2-origin.c line 631 function activate_threads ##file kundu2-origin.c line 635 function activate_threads ##file kundu2-origin.c line 637 function activate_threads ##file kundu2-origin.c line 639 function activate_threads ##file kundu2-origin.c line 647 function immediate_notify ##file kundu2-origin.c line 680 function start_simulation ##file kundu2-origin.c line 683 function start_simulation ##file kundu2-origin.c line 691 function start_simulation ##file kundu2-origin.c line 694 function start_simulation ##file kundu2-origin.c line 698 function start_simulation ##file kundu2-origin.c line 702 function start_simulation ##file kundu2-origin.c line 708 function start_simulation ##file kundu2-origin.c line 710 function start_simulation ##file kundu2-origin.c line 725 function start_simulation ##file kundu2-origin.c line 738 function init_model ##file kundu2-origin.c line 739 function init_model ##file kundu2-origin.c line 740 function init_model ##file kundu2-origin.c line 741 function init_model ##file kundu2-origin.c line 749 function main ##file kundu2-origin.c line 752 function main ##file kundu2-origin.c line 753 function main ##file kundu2-origin.c line 754 function main ##file kundu2-origin.c line 756 function main ##file kundu2-origin.c line 757 function main ##file kundu2-origin.c line 758 function main ##file kundu2-origin.c line 759 function main ##file kundu2-origin.c line 761 function main ##

…

On 6/28/17 7:44 PM, quiveringlemon wrote: WRT kundu2. My output is as follows (the fault is 428), so is quite far down the list. This must mean our outputs are different or you have a different buggy line, what is your output? ** Most likely fault location: Fault localization scores: [error.assertion.1]: Single Bug Optimal Fault Localization [score: 1.71429] ##file kundu2_unsafe.cil.c line 75 function P_1 [score: 1.66667] ##file kundu2_unsafe.cil.c line 133 function P_2 [score: 1.42857] ##file kundu2_unsafe.cil.c line 71 function P_1 [score: 1.42857] ##file kundu2_unsafe.cil.c line 82 function P_1 [score: 1.38095] ##file kundu2_unsafe.cil.c line 129 function P_2 [score: 1.38095] ##file kundu2_unsafe.cil.c line 140 function P_2 [score: 1.19048] ##file kundu2_unsafe.cil.c line 475 function activate_threads [score: 1.19048] ##file kundu2_unsafe.cil.c line 439 function reset_time_events ##file kundu2_unsafe.cil.c line 444 function reset_time_events ##file kundu2_unsafe.cil.c line 449 function reset_time_events [score: 1.19048] ##file kundu2_unsafe.cil.c line 268 function is_C_1_triggered ##file kundu2_unsafe.cil.c line 269 function is_C_1_triggered [score: 1.19048] ##file kundu2_unsafe.cil.c line 483 function activate_threads [score: 1.19048] ##file kundu2_unsafe.cil.c line 267 function is_C_1_triggered [score: 1.19048] ##file kundu2_unsafe.cil.c line 176 function is_P_2_triggered ##file kundu2_unsafe.cil.c line 177 function is_P_2_triggered [score: 1.19048] ##file kundu2_unsafe.cil.c line 467 function activate_threads [score: 1.19048] ##file kundu2_unsafe.cil.c line 332 function exists_runnable_thread [score: 1.19048] ##file kundu2_unsafe.cil.c line 226 function C_1 ##file kundu2_unsafe.cil.c line 227 function C_1 [score: 1.19048] ##file kundu2_unsafe.cil.c line 235 function C_1 ##file kundu2_unsafe.cil.c line 236 function C_1 ##file kundu2_unsafe.cil.c line 237 function C_1 ##file kundu2_unsafe.cil.c line 238 function C_1 ##file kundu2_unsafe.cil.c line 241 function C_1 ##file kundu2_unsafe.cil.c line 26 function read_data [score: 1.19048] ##file kundu2_unsafe.cil.c line 108 function is_P_1_triggered ##file kundu2_unsafe.cil.c line 109 function is_P_1_triggered [score: 1.19048] ##file kundu2_unsafe.cil.c line 427 function fire_time_events ##file kundu2_unsafe.cil.c line 428 function fire_time_events ##file kundu2_unsafe.cil.c line 429 function fire_time_events ##file kundu2_unsafe.cil.c line 557 function start_simulation — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATYfm52GaMmbLU5Fcavme2OxCWtPP8K0ks5sIp70gaJpZM4OD9-_>.

-- Youcheng Sun Research Assistant Department of Computer Science University of Oxford Wolfson Building, Parks Road Oxford OX1 3QD https://sites.google.com/site/theyoucheng/ #define FAIL #include <assert.h> int BLOCK1 = 0; int BLOCK2 = 0; int BLOCK3 = 0; int BLOCK4 = 0; int BLOCK5 = 0; int BLOCK6 = 0; int BLOCK7 = 0; int BLOCK8 = 0; int BLOCK9 = 0; int BLOCK10 = 0; int BLOCK11 = 0; int BLOCK12 = 0; int BLOCK13 = 0; int BLOCK14 = 0; int BLOCK15 = 0; int BLOCK16 = 0; int BLOCK17 = 0; int BLOCK18 = 0; int BLOCK19 = 0; int BLOCK20 = 0; int BLOCK21 = 0; int BLOCK22 = 0; int BLOCK23 = 0; int BLOCK24 = 0; int BLOCK25 = 0; int BLOCK26 = 0; int BLOCK27 = 0; int BLOCK28 = 0; int BLOCK29 = 0; int BLOCK30 = 0; int BLOCK31 = 0; int BLOCK32 = 0; int BLOCK33 = 0; int BLOCK34 = 0; int BLOCK35 = 0; int BLOCK36 = 0; int BLOCK37 = 0; int BLOCK38 = 0; int BLOCK39 = 0; int BLOCK40 = 0; int BLOCK41 = 0; int BLOCK42 = 0; int BLOCK43 = 0; int BLOCK44 = 0; int BLOCK45 = 0; int BLOCK46 = 0; int BLOCK47 = 0; int BLOCK48 = 0; int BLOCK49 = 0; int BLOCK50 = 0; int BLOCK51 = 0; int BLOCK52 = 0; int BLOCK53 = 0; int BLOCK54 = 0; int BLOCK55 = 0; int BLOCK56 = 0; int BLOCK57 = 0; int BLOCK58 = 0; int BLOCK59 = 0; int BLOCK60 = 0; int BLOCK61 = 0; int BLOCK62 = 0; int BLOCK63 = 0; int BLOCK64 = 0; int BLOCK65 = 0; int BLOCK66 = 0; int BLOCK67 = 0; int BLOCK68 = 0; int BLOCK69 = 0; int BLOCK70 = 0; int BLOCK71 = 0; int BLOCK72 = 0; int BLOCK73 = 0; int BLOCK74 = 0; int BLOCK75 = 0; int BLOCK76 = 0; int BLOCK77 = 0; int BLOCK78 = 0; int BLOCK79 = 0; int BLOCK80 = 0; int BLOCK81 = 0; int BLOCK82 = 0; int BLOCK83 = 0; int BLOCK84 = 0; int BLOCK85 = 0; int BLOCK86 = 0; int BLOCK87 = 0; int BLOCK88 = 0; int BLOCK89 = 0; int BLOCK90 = 0; int BLOCK91 = 0; int BLOCK92 = 0; int BLOCK93 = 0; int BLOCK94 = 0; int BLOCK95 = 0; int BLOCK96 = 0; int BLOCK97 = 0; int BLOCK98 = 0; int BLOCK99 = 0; int BLOCK100 = 0; int BLOCK101 = 0; int BLOCK102 = 0; int BLOCK103 = 0; int BLOCK104 = 0; int BLOCK105 = 0; int BLOCK106 = 0; int BLOCK107 = 0; int BLOCK108 = 0; int BLOCK109 = 0; int BLOCK110 = 0; int BLOCK111 = 0; int BLOCK112 = 0; int BLOCK113 = 0; int BLOCK114 = 0; int BLOCK115 = 0; int BLOCK116 = 0; int BLOCK117 = 0; // NB there are three calls to error in the program. 91 is the bug . 5 10 48 are the error statments. // paths always go through 5|10|48, but not always thru 5, and not always through 48, but always through 7 // NB PASSING TRACES are anything that doesnt violate at least one spec. nb "degrees" of passing and failing? // BLOCK5==1 && BLOCK7==1 && BLOCK48==1 void error(void) { #ifdef FAIL assert(0);//SUT #endif } void immediate_notify(void) ; int max_loop ; int num ; int i ; int e ; int timer ; char data_0 ; char data_1 ; char read_data(int i___0 ) { BLOCK1 = 1; char c ; char __retres3 ; { if (i___0 == 0) { BLOCK2 = 1; __retres3 = data_0; goto return_label; } else { BLOCK3 = 1; if (i___0 == 1) { BLOCK4 = 1; __retres3 = data_1; goto return_label; } else { BLOCK5 = 1; { error(); } } } __retres3 = c; return_label: /* CIL Label */ // PASS ASSERTION1 #ifdef PASS1 assert(BLOCK5!=0); //if (BLOCK5==0) //{ // assert(0);//SUT //} #endif return (__retres3); } } void write_data(int i___0 , char c ) { BLOCK6 = 1; { if (i___0 == 0) { BLOCK7 = 1; data_0 = c; } else { BLOCK8 = 1; if (i___0 == 1) { BLOCK9 = 1; data_1 = c; } else { BLOCK10 = 1; { error(); } } } // PASS ASSERTION2 #ifdef PASS2 if (BLOCK10==0) { assert(0);//SUT } #endif return; } } int P_1_pc; int P_1_st ; int P_1_i ; int P_1_ev ; void P_1(void) { BLOCK11 = 1; { if ((int )P_1_pc == 0) { BLOCK12 = 1; goto P_1_ENTRY_LOC; } else { BLOCK13 = 1; if ((int )P_1_pc == 1) { BLOCK14 = 1; goto P_1_WAIT_LOC; } else { BLOCK15 = 1; } } P_1_ENTRY_LOC: { while (i < max_loop) { BLOCK16 = 1; while_0_continue: /* CIL Label */ ; { BLOCK17 = 1; write_data(num, 'A'); num += 1; P_1_pc = 1; P_1_st = 2; } goto return_label; P_1_WAIT_LOC: ; } while_0_break: BLOCK18 = 1; /* CIL Label */ ; } P_1_st = 2; return_label: BLOCK19 = 1; /* CIL Label */ return; } } int is_P_1_triggered(void) { int __retres1 ; BLOCK20 = 1; { if ((int )P_1_pc == 1) { BLOCK21 = 1; if ((int )P_1_ev == 1) { BLOCK22 = 1; __retres1 = 1; goto return_label; } else { BLOCK23 = 1; } } else { BLOCK23 = 1; } __retres1 = 0; return_label: /* CIL Label */ return (__retres1); } } int P_2_pc ; int P_2_st ; int P_2_i ; int P_2_ev ; void P_2(void) { BLOCK24 = 1; { if ((int )P_2_pc == 0) { BLOCK25 = 1; goto P_2_ENTRY_LOC; } else { BLOCK26 = 1; if ((int )P_2_pc == 1) { BLOCK27 = 1; goto P_2_WAIT_LOC; } else { BLOCK28 = 1; } } P_2_ENTRY_LOC: { BLOCK29 = 1; while (i < max_loop) { while_1_continue: /* CIL Label */ ; {BLOCK30 = 1; write_data(num, 'B'); num += 1; } if (timer) { BLOCK31 = 1; { timer = 0; e = 1; immediate_notify(); e = 2; } } else { BLOCK32 = 1; } P_2_pc = 1; P_2_st = 2; goto return_label; P_2_WAIT_LOC: ; } while_1_break: /* CIL Label */ ; } P_2_st = 2; return_label: /* CIL Label */ return; } } int is_P_2_triggered(void) { int __retres1 ; BLOCK33 = 1; { if ((int )P_2_pc == 1) { BLOCK34 = 1; if ((int )P_2_ev == 1) { BLOCK35 = 1; __retres1 = 1; goto return_label; } else { BLOCK36 = 1; } } else { BLOCK37 = 1; } __retres1 = 0; return_label: /* CIL Label */ return (__retres1); } } int C_1_pc ; int C_1_st ; int C_1_i ; int C_1_ev ; int C_1_pr ; void C_1(void) { char c ; BLOCK38 = 1; { if ((int )C_1_pc == 0) {BLOCK39 = 1; goto C_1_ENTRY_LOC; } else { BLOCK40 = 1; if ((int )C_1_pc == 1) {BLOCK41 = 1; goto C_1_WAIT_1_LOC; } else { BLOCK42 = 1; if ((int )C_1_pc == 2) { BLOCK43 = 1; goto C_1_WAIT_2_LOC; } else { BLOCK44 = 1; } } } C_1_ENTRY_LOC: { while (i < max_loop) {BLOCK45 = 1; while_2_continue: /* CIL Label */ ; if (num == 0) {BLOCK46 = 1; timer = 1; i += 1; C_1_pc = 1; C_1_st = 2; goto return_label; C_1_WAIT_1_LOC: ; } else {BLOCK47 = 1; } num -= 1; if (! (num >= 0)) { {BLOCK48 = 1; error(); } } else {BLOCK49 = 1; } {BLOCK50 = 1; c = read_data(num); i += 1; C_1_pc = 2; C_1_st = 2; } // PASS ASSERTION3 #ifdef PASS3 if (BLOCK48==0) { assert(0);//SUT } #endif goto return_label; C_1_WAIT_2_LOC: ; } while_2_break: /* CIL Label */ ; } C_1_st = 2; return_label: /* CIL Label */ return; } } int is_C_1_triggered(void) { int __retres1 ; BLOCK51 = 1; { if ((int )C_1_pc == 1) { BLOCK52 = 1; if ((int )e == 1) { BLOCK53 = 1; __retres1 = 1; goto return_label; } else { BLOCK54 = 1; } } else { } if ((int )C_1_pc == 2) { BLOCK55 = 1; if ((int )C_1_ev == 1) { BLOCK56 = 1; __retres1 = 1; goto return_label; } else { BLOCK57 = 1; } } else { BLOCK58 = 1; } __retres1 = 0; return_label: /* CIL Label */ return (__retres1); } } void update_channels(void) { BLOCK59 = 1; return; } void init_threads(void) { BLOCK60 = 1; { if ((int )P_1_i == 1) { BLOCK61 = 1; P_1_st = 0; } else { BLOCK62 = 1; P_1_st = 2; } if ((int )P_2_i == 1) { BLOCK63 = 1; P_2_st = 0; } else { BLOCK64 = 1; P_2_st = 2; } if ((int )C_1_i == 1) { BLOCK65 = 1; C_1_st = 0; } else { BLOCK66 = 1; C_1_st = 2; } return; } } int exists_runnable_thread(void) { int __retres1 ; { if ((int )P_1_st == 0) { BLOCK67 = 1; __retres1 = 1; goto return_label; } else { BLOCK68 = 1; if ((int )P_2_st == 0) { BLOCK69 = 1; __retres1 = 1; goto return_label; } else { BLOCK70 = 1; if ((int )C_1_st == 0) { BLOCK71 = 1; __retres1 = 1; goto return_label; } else { BLOCK72 = 1; } } } __retres1 = 0; return_label: /* CIL Label */ return (__retres1); } } void eval(void) { int tmp ; int tmp___0 ; int tmp___1 ; int tmp___2 ; BLOCK73 = 1; // int __VERIFIER_nondet_int(); { { while (1) { BLOCK74 = 1; while_3_continue: /* CIL Label */ ; { tmp___2 = exists_runnable_thread(); } if (tmp___2) { BLOCK75 = 1; } else { BLOCK76 = 1; goto while_3_break; } if ((int )P_1_st == 0) { BLOCK77 = 1; { tmp = __VERIFIER_nondet_int(); } if (tmp) { BLOCK78 = 1; { P_1_st = 1; P_1(); } } else { BLOCK79 = 1; } } else { BLOCK80 = 1; } if ((int )P_2_st == 0) { BLOCK81 = 1; { tmp___0 = __VERIFIER_nondet_int(); } if (tmp___0) { BLOCK82 = 1; { P_2_st = 1; P_2(); } } else { BLOCK83 = 1; } } else { BLOCK84 = 1; } if ((int )C_1_st == 0) { BLOCK85 = 1; { tmp___1 = __VERIFIER_nondet_int(); } if (tmp___1) { BLOCK86 = 1; { C_1_st = 1; C_1(); } } else { BLOCK87 = 1; } } else { BLOCK88 = 1; } } while_3_break: /* CIL Label */ ; } return; } } void fire_delta_events(void) { BLOCK89 = 1; return; } void reset_delta_events(void) { BLOCK90 = 1; return; } void fire_time_events(void) { BLOCK91 = 1; // BUG 1/1, should be the body of a conditional statement. C_1_ev = 1; P_1_ev = 1; P_2_ev = 1; return; } void reset_time_events(void) { BLOCK92 = 1; { if ((int )P_1_ev == 1) { BLOCK93 = 1; P_1_ev = 2; } else { BLOCK94 = 1; } if ((int )P_2_ev == 1) { BLOCK95 = 1; P_2_ev = 2; } else { BLOCK96 = 1; } if ((int )C_1_ev == 1) { BLOCK97 = 1; C_1_ev = 2; } else { BLOCK98 = 1; } return; } } void activate_threads(void) { int tmp ; int tmp___0 ; int tmp___1 ; BLOCK99 = 1; { { tmp = is_P_1_triggered(); } if (tmp) { BLOCK100 = 1; P_1_st = 0; } else { BLOCK101 = 1; } { tmp___0 = is_P_2_triggered(); } if (tmp___0) { BLOCK102 = 1; P_2_st = 0; } else { BLOCK103 = 1; } { tmp___1 = is_C_1_triggered(); } if (tmp___1) { BLOCK104 = 1; C_1_st = 0; } else { BLOCK105 = 1; } return; } } void immediate_notify(void) { BLOCK106 = 1; activate_threads(); return; } int stop_simulation(void) { int tmp ; int __retres2 ; BLOCK107 = 1; { { tmp = exists_runnable_thread(); } if (tmp) { BLOCK108 = 1; __retres2 = 0; goto return_label; } else { BLOCK109 = 1; } __retres2 = 1; return_label: /* CIL Label */ return (__retres2); } } void start_simulation(void) { int kernel_st ; int tmp ; int tmp___0 ; BLOCK110 = 1; { { kernel_st = 0; update_channels(); init_threads(); fire_delta_events(); activate_threads(); reset_delta_events(); } { while (1) { BLOCK111 = 1; while_4_continue: /* CIL Label */ ; { kernel_st = 1; eval(); } { kernel_st = 2; update_channels(); } { kernel_st = 3; fire_delta_events(); activate_threads(); reset_delta_events(); } { tmp = exists_runnable_thread(); } if (tmp == 0) { BLOCK112 = 1; { kernel_st = 4; fire_time_events(); activate_threads(); reset_time_events(); } } else { BLOCK113= 1; } { tmp___0 = stop_simulation(); } if (tmp___0) { BLOCK114 = 1; goto while_4_break; } else { BLOCK115 = 1; } } while_4_break: /* CIL Label */ ; } return; } } void init_model(void) { { BLOCK116 = 1; P_1_i = 1; P_2_i = 1; C_1_i = 1; return; } } int main(void) { int count ; int __retres2 ; BLOCK117 = 1; { { num = 0; i = 0; max_loop = 2; e ; timer = 0; P_1_pc = 0; P_2_pc = 0; C_1_pc = 0; count = 0; init_model(); start_simulation(); } __retres2 = 0; return (__retres2); } }

[quiveringlemon]

That's good news about kundu - my results are similar (the version of kundu i gave you is annotated and different to mine a tiny bit).

[quiveringlemon]

So we can discuss this tomorrow, but I've just done a lengthy analysis of FCBMC and it's time inefficiency problem with insert_sort. I decided to compare FCBMC against a brute force manual check in which i manually generated each possible failing/passing trace by making individual calls to CBMC... the results along with the program I used are pasted below:

// REPORT: In theory the FCBMC implementation will be more efficient than a brute force algorithm in which all possible paths are checked for (this is because in theory it is clever about how it calls the model checker and uses every trace which the model checker returns in the matrix). Consequently, FCBMC should outperform a brute force algorithm on insert_sort. So I did a manual check- calling CBMC to check whether a given path was satisfiable and passing/failing for all possible paths. I did this as follows: to generate a failing trace I added the following conditional expressions (below) before the assertion (meaning a failing trace would only be generated if it covered the specified path). To generate a passing trace i did a similar thing but added ! to the assertion statement (negating the specification). I have listed the conditions, along with the time it took for CBMC to come back with its response below. Calling CBMC to do this brute force task took a cumulative 11.424 seconds, which is several seconds quicker than FCBMC (both approaches generate the same matrix - see below). There is consequently something seriously wrong in the implementation of FCBMC - not only is the single bug data algorithm guaranteed to have a much lower complexity than brute force, brute force will in practice be additionally slower because it has to unwind the program an additional 32 times to FCBMC.

// conditional expressions for FAILING TRACES, following by time in secs. - means it generated a failing trace.
// (C1 == 1 && C2 == 1 && C3 == 1 && C4 == 1), 0.660 -
// (C1 == 1 && C2 == 1 && C3 == 1 && C4 == 0), 0.796
// (C1 == 1 && C2 == 1 && C3 == 0 && C4 == 1), 0.720
// (C1 == 1 && C2 == 1 && C3 == 0 && C4 == 0), 0.848
// (C1 == 1 && C2 == 0 && C3 == 1 && C4 == 1), 0.400
// (C1 == 1 && C2 == 0 && C3 == 1 && C4 == 0), 0.392
// (C1 == 1 && C2 == 0 && C3 == 0 && C4 == 1), 0.384
// (C1 == 1 && C2 == 0 && C3 == 0 && C4 == 0), 0.632
// (C1 == 0 && C2 == 1 && C3 == 1 && C4 == 1), 0.132
// (C1 == 0 && C2 == 1 && C3 == 1 && C4 == 0), 0.108
// (C1 == 0 && C2 == 1 && C3 == 0 && C4 == 1), 0.124
// (C1 == 0 && C2 == 1 && C3 == 0 && C4 == 0), 0.124
// (C1 == 0 && C2 == 0 && C3 == 1 && C4 == 1), 0.124
// (C1 == 0 && C2 == 0 && C3 == 1 && C4 == 0), 0.128
// (C1 == 0 && C2 == 0 && C3 == 0 && C4 == 1), 0.132
// (C1 == 0 && C2 == 0 && C3 == 0 && C4 == 0), 0.128

// conditional expressions for PASSING TRACES, following by time in secs. + means it generated a passing trace.
// (C1 == 1 && C2 == 1 && C3 == 1 && C4 == 1), 0.692 +
// (C1 == 1 && C2 == 1 && C3 == 1 && C4 == 0), 0.728 +
// (C1 == 1 && C2 == 1 && C3 == 0 && C4 == 1), 0.660
// (C1 == 1 && C2 == 1 && C3 == 0 && C4 == 0), 0.692 +
// (C1 == 1 && C2 == 0 && C3 == 1 && C4 == 1), 0.408
// (C1 == 1 && C2 == 0 && C3 == 1 && C4 == 0), 0.380
// (C1 == 1 && C2 == 0 && C3 == 0 && C4 == 1), 0.408
// (C1 == 1 && C2 == 0 && C3 == 0 && C4 == 0), 0.628
// (C1 == 0 && C2 == 1 && C3 == 1 && C4 == 1), 0.120
// (C1 == 0 && C2 == 1 && C3 == 1 && C4 == 0), 0.136
// (C1 == 0 && C2 == 1 && C3 == 0 && C4 == 1), 0.128
// (C1 == 0 && C2 == 1 && C3 == 0 && C4 == 0), 0.128
// (C1 == 0 && C2 == 0 && C3 == 1 && C4 == 1), 0.128
// (C1 == 0 && C2 == 0 && C3 == 1 && C4 == 0), 0.128
// (C1 == 0 && C2 == 0 && C3 == 0 && C4 == 1), 0.116
// (C1 == 0 && C2 == 0 && C3 == 0 && C4 == 0), 0.112

// MATRIX
// 1 1 1 1 -
// 1 1 1 1 +
// 1 1 1 0 +
// 1 1 0 0 +
//
// TRIMMED:
// 1 1 1 -
// 1 1 1 +
// 1 1 0 +
// 1 0 0 +
// BRUTE FORCE TIME = 11.424

// FCBMC

// MATRIX - Note this is the same matrix as the one done above except the columns are swapped for implementation reasons. This confirms that the matrix produced by the brute force algorithm is equivalent to the FCBMC implementation.

// 1 1 1 -
// 1 0 1 +
// 0 0 1 +
// 1 1 1 +

// FCBMC TIME = 15.040s!
#include <assert.h>

void __VERIFIER_assert(int cond) {
if (!(cond)) {
assert(0);
}
return;
}
unsigned int __VERIFIER_nondet_uint();
int main() {

int C1 = 0;
int C2 = 0;
int C3 = 0;
int C4 = 0;

unsigned int SIZE=__VERIFIER_nondet_uint();
int i, j, k, key;
int v[SIZE];

C1 = 1;

for (j=1;j<SIZE;j++) {

  C2 = 1;

  key = v[j];
  i = j - 1;
  while((i>=0) && (v[i]>key)) {

     C3 = 1;

     if (i<2) // BUG: should be eliminated line 17
{
     C4 = 1;

     v[i+1] = v[i];
}

     i = i - 1;
  }
  v[i+1] = key;	        

}

for (k=1;k<SIZE;k++)
{
if (1) // assertion condition
{
__VERIFIER_assert(v[k-1]<=v[k]);
}
}

return 0;
}

[theyoucheng]

I investigated a bit. For some reason, the "most time consuming" part comes at the generation of S traces in the algorithm. I simplified this procedure 0e0be40 Now it's faster.

…

On 6/29/17 12:41 AM, quiveringlemon wrote: So we can discuss this tomorrow, but I've just done a lengthy analysis of FCBMC and it's time inefficiency problem with insert_sort. I decided to compare FCBMC against a brute force manual check in which i manually generated each possible failing/passing trace by making individual calls to CBMC... the results along with the program I used are pasted below: // REPORT: In theory the FCBMC implementation will be more efficient than a brute force algorithm in which all possible paths are checked for (this is because in theory it is clever about how it calls the model checker and uses every trace which the model checker returns in the matrix). Consequently, FCBMC should outperform a brute force algorithm on insert_sort. So I did a manual check- calling CBMC to check whether a given path was satisfiable and passing/failing for all possible paths. I did this as follows: to generate a failing trace I added the following conditional expressions (below) before the assertion (meaning a failing trace would only be generated if it covered the specified path). To generate a passing trace i did a similar thing but added ! to the assertion statement (negating the specification). I have listed the conditions, along with the time it took for CBMC to come back with its response below. Calling CBMC to do this brute force task took a cumulative 11.424 seconds, which is several seconds quicker than FCBMC (both approaches generate the same matrix - see below). There is consequently something seriously wrong in the implementation of FCBMC - not only is the single bug data algorithm /guaranteed/ to have a much lower complexity than brute force, brute force will in practice be additionally slower because it has to unwind the program an additional 32 times to FCBMC. // conditional expressions for FAILING TRACES, following by time in secs. - means it generated a failing trace. // (C1 == 1 && C2 == 1 && C3 == 1 && C4 == 1), 0.660 - // (C1 == 1 && C2 == 1 && C3 == 1 && C4 == 0), 0.796 // (C1 == 1 && C2 == 1 && C3 == 0 && C4 == 1), 0.720 // (C1 == 1 && C2 == 1 && C3 == 0 && C4 == 0), 0.848 // (C1 == 1 && C2 == 0 && C3 == 1 && C4 == 1), 0.400 // (C1 == 1 && C2 == 0 && C3 == 1 && C4 == 0), 0.392 // (C1 == 1 && C2 == 0 && C3 == 0 && C4 == 1), 0.384 // (C1 == 1 && C2 == 0 && C3 == 0 && C4 == 0), 0.632 // (C1 == 0 && C2 == 1 && C3 == 1 && C4 == 1), 0.132 // (C1 == 0 && C2 == 1 && C3 == 1 && C4 == 0), 0.108 // (C1 == 0 && C2 == 1 && C3 == 0 && C4 == 1), 0.124 // (C1 == 0 && C2 == 1 && C3 == 0 && C4 == 0), 0.124 // (C1 == 0 && C2 == 0 && C3 == 1 && C4 == 1), 0.124 // (C1 == 0 && C2 == 0 && C3 == 1 && C4 == 0), 0.128 // (C1 == 0 && C2 == 0 && C3 == 0 && C4 == 1), 0.132 // (C1 == 0 && C2 == 0 && C3 == 0 && C4 == 0), 0.128 // conditional expressions for PASSING TRACES, following by time in secs. + means it generated a passing trace. // (C1 == 1 && C2 == 1 && C3 == 1 && C4 == 1), 0.692 + // (C1 == 1 && C2 == 1 && C3 == 1 && C4 == 0), 0.728 + // (C1 == 1 && C2 == 1 && C3 == 0 && C4 == 1), 0.660 // (C1 == 1 && C2 == 1 && C3 == 0 && C4 == 0), 0.692 + // (C1 == 1 && C2 == 0 && C3 == 1 && C4 == 1), 0.408 // (C1 == 1 && C2 == 0 && C3 == 1 && C4 == 0), 0.380 // (C1 == 1 && C2 == 0 && C3 == 0 && C4 == 1), 0.408 // (C1 == 1 && C2 == 0 && C3 == 0 && C4 == 0), 0.628 // (C1 == 0 && C2 == 1 && C3 == 1 && C4 == 1), 0.120 // (C1 == 0 && C2 == 1 && C3 == 1 && C4 == 0), 0.136 // (C1 == 0 && C2 == 1 && C3 == 0 && C4 == 1), 0.128 // (C1 == 0 && C2 == 1 && C3 == 0 && C4 == 0), 0.128 // (C1 == 0 && C2 == 0 && C3 == 1 && C4 == 1), 0.128 // (C1 == 0 && C2 == 0 && C3 == 1 && C4 == 0), 0.128 // (C1 == 0 && C2 == 0 && C3 == 0 && C4 == 1), 0.116 // (C1 == 0 && C2 == 0 && C3 == 0 && C4 == 0), 0.112 // MATRIX // 1 1 1 1 - // 1 1 1 1 + // 1 1 1 0 + // 1 1 0 0 + // // TRIMMED: // 1 1 1 - // 1 1 1 + // 1 1 0 + // 1 0 0 + // BRUTE FORCE TIME = 11.424 // FCBMC // MATRIX - Note this is the same matrix as the one done above except the columns are swapped for implementation reasons. This confirms that the matrix produced by the brute force algorithm is equivalent to the FCBMC implementation. // 1 1 1 - // 1 0 1 + // 0 0 1 + // 1 1 1 + // FCBMC TIME = 15.040s! #include <assert.h> void __VERIFIER_assert(int cond) { if (!(cond)) { assert(0); } return; } unsigned int __VERIFIER_nondet_uint(); int main() { int C1 = 0; int C2 = 0; int C3 = 0; int C4 = 0; unsigned int SIZE=__VERIFIER_nondet_uint(); int i, j, k, key; int v[SIZE]; C1 = 1; for (j=1;j<SIZE;j++) { |C2 = 1; key = v[j]; i = j - 1; while((i>=0) && (v[i]>key)) { C3 = 1; if (i<2) // BUG: should be eliminated line 17 { C4 = 1; v[i+1] = v[i]; } i = i - 1; } v[i+1] = key; | } for (k=1;k<SIZE;k++) { if (1) // assertion condition { __VERIFIER_assert(v[k-1]<=v[k]); } } return 0; } — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ATYfm8KBqqp9XYC9v70bt1yOF0D9varyks5sIuSkgaJpZM4OD9-_>.

-- Youcheng Sun Research Assistant Department of Computer Science University of Oxford Wolfson Building, Parks Road Oxford OX1 3QD https://sites.google.com/site/theyoucheng/

[theyoucheng]

Another thing to remind. I realize that when you specify the unwinding number (let's say 4) for the incremental BMC, using *"--unwind-min 4 --unwind-max 4" *is much more efficient than using *"--unwind 4"*. This is due to different approaches that CBMC and CBMC-incremental proceeding the loop unwinding.

…

On 6/29/17 12:24 PM, Youcheng Sun wrote: I investigated a bit. For some reason, the "most time consuming" part comes at the generation of S traces in the algorithm. I simplified this procedure 0e0be40 Now it's faster. On 6/29/17 12:41 AM, quiveringlemon wrote: > > So we can discuss this tomorrow, but I've just done a lengthy > analysis of FCBMC and it's time inefficiency problem with > insert_sort. I decided to compare FCBMC against a brute force manual > check in which i manually generated each possible failing/passing > trace by making individual calls to CBMC... the results along with > the program I used are pasted below: > > // REPORT: In theory the FCBMC implementation will be more efficient > than a brute force algorithm in which all possible paths are checked > for (this is because in theory it is clever about how it calls the > model checker and uses every trace which the model checker returns in > the matrix). Consequently, FCBMC should outperform a brute force > algorithm on insert_sort. So I did a manual check- calling CBMC to > check whether a given path was satisfiable and passing/failing for > all possible paths. I did this as follows: to generate a failing > trace I added the following conditional expressions (below) before > the assertion (meaning a failing trace would only be generated if it > covered the specified path). To generate a passing trace i did a > similar thing but added ! to the assertion statement (negating the > specification). I have listed the conditions, along with the time it > took for CBMC to come back with its response below. Calling CBMC to > do this brute force task took a cumulative 11.424 seconds, which is > several seconds quicker than FCBMC (both approaches generate the same > matrix - see below). There is consequently something seriously wrong > in the implementation of FCBMC - not only is the single bug data > algorithm /guaranteed/ to have a much lower complexity than brute > force, brute force will in practice be additionally slower because it > has to unwind the program an additional 32 times to FCBMC. > > // conditional expressions for FAILING TRACES, following by time in > secs. - means it generated a failing trace. > // (C1 == 1 && C2 == 1 && C3 == 1 && C4 == 1), 0.660 - > // (C1 == 1 && C2 == 1 && C3 == 1 && C4 == 0), 0.796 > // (C1 == 1 && C2 == 1 && C3 == 0 && C4 == 1), 0.720 > // (C1 == 1 && C2 == 1 && C3 == 0 && C4 == 0), 0.848 > // (C1 == 1 && C2 == 0 && C3 == 1 && C4 == 1), 0.400 > // (C1 == 1 && C2 == 0 && C3 == 1 && C4 == 0), 0.392 > // (C1 == 1 && C2 == 0 && C3 == 0 && C4 == 1), 0.384 > // (C1 == 1 && C2 == 0 && C3 == 0 && C4 == 0), 0.632 > // (C1 == 0 && C2 == 1 && C3 == 1 && C4 == 1), 0.132 > // (C1 == 0 && C2 == 1 && C3 == 1 && C4 == 0), 0.108 > // (C1 == 0 && C2 == 1 && C3 == 0 && C4 == 1), 0.124 > // (C1 == 0 && C2 == 1 && C3 == 0 && C4 == 0), 0.124 > // (C1 == 0 && C2 == 0 && C3 == 1 && C4 == 1), 0.124 > // (C1 == 0 && C2 == 0 && C3 == 1 && C4 == 0), 0.128 > // (C1 == 0 && C2 == 0 && C3 == 0 && C4 == 1), 0.132 > // (C1 == 0 && C2 == 0 && C3 == 0 && C4 == 0), 0.128 > > // conditional expressions for PASSING TRACES, following by time in > secs. + means it generated a passing trace. > // (C1 == 1 && C2 == 1 && C3 == 1 && C4 == 1), 0.692 + > // (C1 == 1 && C2 == 1 && C3 == 1 && C4 == 0), 0.728 + > // (C1 == 1 && C2 == 1 && C3 == 0 && C4 == 1), 0.660 > // (C1 == 1 && C2 == 1 && C3 == 0 && C4 == 0), 0.692 + > // (C1 == 1 && C2 == 0 && C3 == 1 && C4 == 1), 0.408 > // (C1 == 1 && C2 == 0 && C3 == 1 && C4 == 0), 0.380 > // (C1 == 1 && C2 == 0 && C3 == 0 && C4 == 1), 0.408 > // (C1 == 1 && C2 == 0 && C3 == 0 && C4 == 0), 0.628 > // (C1 == 0 && C2 == 1 && C3 == 1 && C4 == 1), 0.120 > // (C1 == 0 && C2 == 1 && C3 == 1 && C4 == 0), 0.136 > // (C1 == 0 && C2 == 1 && C3 == 0 && C4 == 1), 0.128 > // (C1 == 0 && C2 == 1 && C3 == 0 && C4 == 0), 0.128 > // (C1 == 0 && C2 == 0 && C3 == 1 && C4 == 1), 0.128 > // (C1 == 0 && C2 == 0 && C3 == 1 && C4 == 0), 0.128 > // (C1 == 0 && C2 == 0 && C3 == 0 && C4 == 1), 0.116 > // (C1 == 0 && C2 == 0 && C3 == 0 && C4 == 0), 0.112 > > // MATRIX > // 1 1 1 1 - > // 1 1 1 1 + > // 1 1 1 0 + > // 1 1 0 0 + > // > // TRIMMED: > // 1 1 1 - > // 1 1 1 + > // 1 1 0 + > // 1 0 0 + > // BRUTE FORCE TIME = 11.424 > > // FCBMC > > // MATRIX - Note this is the same matrix as the one done above except > the columns are swapped for implementation reasons. This confirms > that the matrix produced by the brute force algorithm is equivalent > to the FCBMC implementation. > > // 1 1 1 - > // 1 0 1 + > // 0 0 1 + > // 1 1 1 + > > // FCBMC TIME = 15.040s! > #include <assert.h> > > void __VERIFIER_assert(int cond) { > if (!(cond)) { > assert(0); > } > return; > } > unsigned int __VERIFIER_nondet_uint(); > int main() { > > int C1 = 0; > int C2 = 0; > int C3 = 0; > int C4 = 0; > > unsigned int SIZE=__VERIFIER_nondet_uint(); > int i, j, k, key; > int v[SIZE]; > > C1 = 1; > > for (j=1;j<SIZE;j++) { > > |C2 = 1; key = v[j]; i = j - 1; while((i>=0) && (v[i]>key)) { C3 = 1; > if (i<2) // BUG: should be eliminated line 17 { C4 = 1; v[i+1] = > v[i]; } i = i - 1; } v[i+1] = key; | > > } > > for (k=1;k<SIZE;k++) > { > if (1) // assertion condition > { > __VERIFIER_assert(v[k-1]<=v[k]); > } > } > > return 0; > } > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <#5 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ATYfm8KBqqp9XYC9v70bt1yOF0D9varyks5sIuSkgaJpZM4OD9-_>. > -- Youcheng Sun Research Assistant Department of Computer Science University of Oxford Wolfson Building, Parks Road Oxford OX1 3QD https://sites.google.com/site/theyoucheng/

-- Youcheng Sun Research Assistant Department of Computer Science University of Oxford Wolfson Building, Parks Road Oxford OX1 3QD https://sites.google.com/site/theyoucheng/