CHC mode with `pf` timeout but BMC gives `sat` result. Why are they giving different results? about seahorn

[caballa]

I didn't look at your example but the behavior you are seeing is quite possible.
The BMC engine is typically much faster than the CHC mode finding counterexamples.
There are at least two scenarios where this might happen:

• There are several loops and the CHC mode "gets stuck" in one of them searching for invariants while BMC finds quickly the cex since all loops are unrolled.
• Even with one loop, the CHC mode might run out of time unrolling the loop since each unroll done by the CHC mode might be quite expensive (find spurious cex, find lemma that blocks the spurious cex, etc).

[caballa]

Another reason is that the BMC models faithfully machine arithmetic semantics while the CHC uses mathematical integers. So it's possible that BMC finds a cex due to (e.g., overflow) that CHC would miss. I would first look at the cex found by BMC and convince yourself that the cex should be also found by the CHC mode.

[caballa]

You can only run one or the other.
I would try to understand first the cex. If the cex doesn't depend on machine arithmetic or very low-level memory reasoning then the most plausible reason is that the CHC mode cannot find a safe invariant for some loop. In this case, I would try to identify which loop is causing problem and see how the loop can be abstracted.

[caballa]

bpf calls the LLVM loop unroller to unroll loops but it doesn't pick between CHC mode or BMC.

If --bmc=none or option --bmc is skipped then it will still run the CHC mode on the unroll program.
If --bmc != none then BMC engine will be used.

[caballa]

In both cases, you do not have loops because you are using bpf option.
So then, it looks like the reason is because BMC uses a more precise semantics for LLVM than the CHC mode.
You need to understand the cex generated by BMC. Again, if the cex is due to some overflow or very low-level memory reasoning then BMC would catch it but CHC mode wouldn't.

Siddharth gave you a link to #537 (comment) that can be useful.

[lahiri-phdworks]

Logs

Mode-1 (sea with pf, no bpf)

/home/usea/seahorn/bin/seapp -o /tmp/sea-4y3wd57m/IntegerOverflowMultiTxMultiFuncFeasible.pp.bc --simplifycfg-sink-common=false --strip-extern=false --promote-assumptions=false --kill-vaarg=true --ignore-def-verifier-fn=false --horn-keep-arith-overflow=false --promote-nondet-undef=true --horn-replace-loops-with-nd-funcs=false IntegerOverflowMultiTxMultiFuncFeasible.ll

/home/usea/seahorn/bin/seapp --simplifycfg-sink-common=false -o /tmp/sea-4y3wd57m/IntegerOverflowMultiTxMultiFuncFeasible.pp.ms.bc --horn-mixed-sem --ms-reduce-main /tmp/sea-4y3wd57m/IntegerOverflowMultiTxMultiFuncFeasible.pp.bc

/home/usea/seahorn/bin/seaopt -f --simplifycfg-sink-common=false -o /tmp/sea-4y3wd57m/IntegerOverflowMultiTxMultiFuncFeasible.pp.ms.o.bc -O3 --seaopt-enable-indvar=false --seaopt-enable-loop-idiom=false --unroll-threshold=150 --unroll-allow-partial=false --unroll-partial-threshold=0 --vectorize-slp=false /tmp/sea-4y3wd57m
/IntegerOverflowMultiTxMultiFuncFeasible.pp.ms.bc

/home/usea/seahorn/bin/seahorn --keep-shadows=true --sea-dsa=ci --horn-solve -horn-cex-pass -horn-cex=cex.ll -horn-inter-proc -horn-sem-lvl=mem --horn-step=large /tmp/sea-4y3wd57m/IntegerOverflowMultiTxMultiFuncFeasible.pp.ms.o.bc

WARNING: initializing DSA node due to memset()
WARNING: initializing DSA node due to memset()
****

Mode-2 (sea bpf with BMC)

OVERFLOW

/home/usea/seahorn/bin/seapp -o /tmp/sea-3grgf61i/IntegerOverflowMultiTxMultiFuncFeasible_b.pp.bc --simplifycfg-sink-common=false --horn-inline-all --strip-extern=false --promote-assumptions=false --kill-vaarg=true --ignore-def-verifier-fn=false --horn-keep-arith-overflow=false --promote-nondet-undef=true --horn-replace-loops-with-nd-funcs=false IntegerOverflowMultiTxMultiFuncFeasible_b.ll
/home/usea/seahorn/bin/seapp --simplifycfg-sink-common=false -o /tmp/sea-3grgf61i/IntegerOverflowMultiTxMultiFuncFeasible_b.pp.ms.bc --horn-mixed-sem --ms-reduce-main /tmp/sea-3grgf61i/IntegerOverflowMultiTxMultiFuncFeasible_b.pp.bc
/home/usea/seahorn/bin/seaopt -f --simplifycfg-sink-common=false -o /tmp/sea-3grgf61i/IntegerOverflowMultiTxMultiFuncFeasible_b.pp.ms.o.bc -O3 --seaopt-enable-indvar=false --seaopt-enable-loop-idiom=false --unroll-threshold=150 --unroll-allow-partial=false --unroll-partial-threshold=0 --vectorize-slp=false /tmp/sea-3grgf61i/IntegerOverflowMultiTxMultiFuncFeasible_b.pp.ms.bc
/home/usea/seahorn/bin/seaopt -f -o /tmp/sea-3grgf61i/IntegerOverflowMultiTxMultiFuncFeasible_b.pp.ms.o.ul.bc -loop-simplify -fake-latch-exit -sea-loop-unroll /tmp/sea-3grgf61i/IntegerOverflowMultiTxMultiFuncFeasible_b.pp.ms.o.bc
/home/usea/seahorn/bin/seapp -o /tmp/sea-3grgf61i/IntegerOverflowMultiTxMultiFuncFeasible_b.pp.ms.o.ul.cut.bc --horn-cut-loops --back-edge-cutter-with-asserts=false /tmp/sea-3grgf61i/IntegerOverflowMultiTxMultiFuncFeasible_b.pp.ms.o.ul.bc
/home/usea/seahorn/bin/seaopt -f --simplifycfg-sink-common=false -o /tmp/sea-3grgf61i/IntegerOverflowMultiTxMultiFuncFeasible_b.pp.ms.o.ul.cut.o.bc -O3 --seaopt-enable-indvar=false --seaopt-enable-loop-idiom=false --unroll-threshold=150 --unroll-allow-partial=false --unroll-partial-threshold=0 --vectorize-slp=false /tmp/sea-3grgf61i/IntegerOverflowMultiTxMultiFuncFeasible_b.pp.ms.o.ul.cut.bc
/home/usea/seahorn/bin/seahorn --horn-bmc --keep-shadows=true --sea-dsa=ci --horn-solve -horn-cex-pass -horn-cex=cex.ll -horn-inter-proc -horn-sem-lvl=mem --horn-step=large /tmp/sea-3grgf61i/IntegerOverflowMultiTxMultiFuncFeasible_b.pp.ms.o.ul.cut.o.bc

sat
****

[lahiri-phdworks]

@agurfinkel How do I share the code here, it's a bit long. Will a drive link work?

[priyasiddharth]

A good way to run bmc engine is using the verify script from https://github.com/seahorn/verify-c-common/blob/master/verify.py.in

To see how this executes... look at how the CI runs: https://github.com/seahorn/verify-c-common/blob/master/.github/workflows/main.yml

Specifically if you want to get a counterexample trace, use verify --cex <...>. This will pick up options in https://github.com/seahorn/verify-c-common/blob/master/seahorn/sea_cex_base.yaml

[lahiri-phdworks]

Thanks for the clarification.
Should I run the CHC mode with bpf enabled and with a higher timeout? I guess not supplying the BMC flag with bpf is still CHC mode.

[lahiri-phdworks]

I see. Does bpf activate the bmc mode? What if I try with sea bpf --bmc=none ...?

[lahiri-phdworks]

When I run my example now with SeaHorn enabling bpf mode with sea bpf ... without the bmc mode, I get a UNSAT result with an invariant (--show-invars). However, when running in bmc mode I get a SAT result with a counter example. I am not sure why that is the case. Cann you please clarify if this is possible?

[lahiri-phdworks]

Logs and the example are here. https://drive.google.com/drive/folders/1nWla3lFMn2_5hxC-sxRoqXB4ki5aPl3F?usp=sharing

[lahiri-phdworks]

I shall check that. Thanks for the clarification. Thanks