swtv-kaist / cs458-spring23 Goto Github PK

Hello!

I am curious how we should define "def-clearness" with sidetrips.

For the sake of easier understand, let me just define a graph first:

N = {0, 1, 2, 3}
N_0 = {0}  
N_f = {3}  
E = {(0, 1), (1, 2), (1, 3), (2,3)}  
def(0) = def(2) = use(3) = {x}  

That is,

In this graph, [0, 1, 3] is du-path. And if we allow sidetrips, [0, 1, 2, 3] tours this [0, 1, 3] path, but I'm not sure it also du-tours this path. The point I'm comfusing is the fact that although [0, 1, 2, 3] defines x twice so it is not def-clear, but still the subpath [0, 1, 3] contains only one definition of x (since 2 is just a sidetrip).

In this case, can we say that [0, 1, 2, 3] du-tours [0, 1, 3]?

In other words, should we consider sidetrips as well when determining def-clearness?

Thank you for reading!

Hello,

I'm wondering if I can assume the 12 arguments to be non-negative values.

It is because if I do not, CROWN generates some inputs as "Own_Tracked_Alt = -2147483647", which seems to me invalid.

Thank you!

In HW02, I used the -f [file] option in grep to obtain high coverage, which allows to use patterns written in the [file].

May I use -f option in Problem 1, 2, and 3?
Specifically, I would like to include some file named pattern.txt and fuzz with afl-fuzz ... -- grep -f pattern.txt @@.

If it is okay, should I submit the pattern.txt that I have used also?

According to the material, the definition of "Def-clear" is as follows:
A path from l_i to l_j is def-clear with respect to variable v, if v is not given another value on any of the nodes or edges in the path.

Can def-clear path can have several use nodes with respect to v?

Thank you!

Dear TA,
sorry for too many questions..
I'm almost done implementing, but have some minor issues.

1. in question 5, grep.i file contins below enum declaration.
   enum = {_ISupper = ((0) < 8 ? ((1 << (0)) << 8) : ((1 << (0)) >> 8)), ... } ;
   my kcov changed this like below
   _ISupper = ((((0) < 8)?(mycov_onTrueCondition(135,4782),1):(mycov_onFalseCondition(135,4782),0)) ? ((1 << (0)) << 8) : ((1 << (0)) >> 8)),
   I tried build gcc grep.i, but I got an error
   error: enumerator value for ‘_ISupper’ is not an integer constant
   and this is because function call is not allowed in a constant expressionC/C++(59)

so I tried to check wheather this ? : is in enum declaration, but I didn't find a good way.
what should I study for this?

2. I inject a code that save result to *.dat file at the end of target source code's main file.
   I find this location by check "return" statment's location.
   but in the case of grep.c, I does not have a return statement, but calls 'exit' function.
   can you give me some help about this?

Thank you

조교님 안녕하세요
우선 너무 자주 질문을 올려 죄송합니다 ㅠㅠ
example-kcov.i 에 대해서는 잘 작동하여 이를 grep.i 파일에 대해서도 적용해보는 중입니다.
제가 작성한 kcov 로 변환은 잘 되는데 몇가지 문제가 있습니다.

1. enum 열거형 내부에서 ? : 삼항 연산자를 사용한 경우가 있는데 이게 branch 개수에는 포함이 되는데 저걸 위의 코드와 같이 변환하게되면 선언안에 함수가 들어갈 수 없다고 에러가 나옵니다. 그래서 enum 안에 있는건지 맥락을 좀 확인해보려했는데 좋은 방법을 찾지 못했습니다. 혹시 뭘 공부하거나 찾아보면 될까요?
2. 타겟 소스코드의 메인문 마지막에 데이터를 다시 *.dat 파일에 적어주는 부분 코드를 심었습니다. 이 위치를 판단하는 기준으로 리턴문의 위치를 생각했는데 grep.c 의 경우 메인 함수가 return 문 없이 exit() 호출만 하고 종료되어 버려서 저장하는 코드가 호출되지 않는 상황입니다.
   역시 이런 부분을 핸들링하려면 어디를 좀 더 찾아봐야할까요?
   감사합니다

Hello, I have a question about the Example in the Logic Coverage slide, specifically about GICC.

From this example, to my best understanding, GICC for major clause b should be {5, 6} x {7, 8} for p=F, and {(1, 3), (1, 4)} for p=T.
It is because for p=T (and F), b should be covered with T and F while minor clause a and c doesn't have to be the same for each b (i.e., b=T and b=F).

Similarly, for major clause c, GICC should be {5,7} x {6, 8} for p=F, and {(1, 4), (3, 4)} for p=T.

However, the answer in the slide doesn't include (1,4) for p=T.

Am I missing something?

Thank you :)

Welcome to Q&A Session for CS458 Spring 2023!

You can freely ask your questions on the lecture slides and/or homework here. Please, attach proper labels to your question so that other students can quickly find similar questions on the same topic.

Also, you can look at the past CS458 Q&A sessions to quickly find answers to the similar questions:

• Q&A session for CS458 Fall 2022

I have a question regarding homework 2.
I might have misunderstood the instruction in last Tuesday's class, but as far as I remember the number of test cases must be exactly 10 while the pdf instruction says "at least 10".

Would it be okay if the number of test cases exceed 10?

When I run gcov -b kcov-branch-identify, it also generates output and .gcov file of every header files.

1. Am I doing it right?
2. If so, should I report coverage of all those headers or only coverage of kcov-branch-identify.cpp?

Thank you.

Hello,

I have a simple question about fuzzing grep.

I've been provided the empty seed named "init_input_empty".

However, I'm not sure what to grep from this seed.

For example, if I want to grep "apple" in a text file named "fruits.txt", I may grep ./grep apple ./fruits.txt

In our homework 5, we would need to have "something" to grep from the input file as ./grep something ./init_input_empty

What do we want to grep? Is it specified? Or is it anything I want?

Dear CS458 students,

I am sorry that I confused you regarding labeling your questions/issues.
I forgot that Github did not support a non-collaborator of a repo to label an issue ;-(

Thus, since you are not enrolled as a collaborator of the CS458 repo, you cannot label your questions.
TAs and I will do it for you.

Thanks

Sincerely
-Moonzoo

Tmux is not installed in verifier18, may you install it for the ease of testing? I would really appreciate it! :)

Dear TA
I have some questions about example about example-kcov.c.

1. what is the rule for output print format?
   for example, there are 17 print of branches in example, but total branch number is 30.

• where is missing 13 branches? (ex. ELSE branch is missing)
• should I determine the unique number for ID field?
• should print format sort by line number?

2. how can example-kcov.c have 30 branches?
   if mine and example have different rule for count branch, then is it okay to have different number of branch?
   Thank you!

조교님 안녕하세요 과제 진행중 궁금한점이 있어 질문드립니다.

1. 결과 출력형식에 대해 질문이 있습니다. 예시를 보면 else절 같은건 출력에 안나와있는데 전체 카운트에는 포함되는 것 같습니다.
   어떤 것은 출력하고 어떤 것은 안하는지 출력형식에 대한 명확한 기준이 있을까요? 그리고 ID 필드의 값은 제가 임의로 할당해주는 건가요?
   마지막으로 출력 형식은 라인넘버에 맞게 정렬되어야하는 걸까요? 작성하다보니 코드상의 순서와 AST로 변환시 내부노드안에 들어가는 순서가 달라지는 것 같아서 처리가 필요할지 질문드립니다.
2. example-kcov.c를 분석한 결과 예시에는 브랜치가 30개가 있다고 나옵니다. 제가 작성한 결과와는 차이가 있는데 브랜치를
   판정하는 기준이 달라지면 달라질 수 있을 것 같습니다. 달라도 문제 없는 건가요? 아니면 무조건 30개를 맞추도록 기준을 정해야하는지 질문드립니다.
   감사합니다.

Hello,

While doing homework #6, I've captured that the gap of branch coverage reported by AFL++ vs gcov is somehow large than expected.

Specifically, AFL++ gave 16.83% as the total branch coverage while gcov only gave 5.36%.

Is this amount of gap a normal phenomenon?

If there is node N1 which have self loop,

The path [N0, N1, N1] is TR for EPC. But it is not a simple path because there is internal loop [N1, N1].
Thus PPC doesn't subsume EPC in this case.

Did we ever talked let's ignore it?
Sorry for late question...

Dear TA,
I'm trying to complie kcov-branch-identify.cpp file with gcc to use gcov.
but I got an complie error that it can't find #include header files.
the command I tried is like below
gcc -xc -E -v -coverage kcov-branch-identify.cpp -o kcov-branch-identify

I thought below code blocks deal with about header files, but It does not work.

>     const char *include_paths[] = {"/usr/local/include",
        "/usr/include/x86_64-linux-gnu",
        "/usr/lib/gcc/x86_64-linux-gnu/7/include",
        "/usr/include"};
    
    for (int i = 0; i < (sizeof(include_paths) / sizeof(include_paths[0])); i++) {
        hso.AddPath(include_paths[i], clang::frontend::Angled, false, false);
    }

maybe I'm missing something. please let me know
Thank you!

조교님 안녕하세요
gcov 사용을 위해 gcc를 통해 kcov~~.cpp 파일을 컴파일하려고 시도중인데 계속해서 컴파일 에러가 발생해서 질문드립니다.
헤더 파일을 못찾아오는 것 같은데 혹시 어떤 커맨드를 사용해야할지 도움을 주실 수 있을까요?
일단 make로 빌드해서 실행하는데에는 아무 문제 없는 상황입니다.
감사합니다

Hello,

While fuzzing grep, I've got an error as below:

I guess this is a disk space issue.

Could you check if there is sufficient memory left in the server?

FYI: My allocated server is verifier18 :)

Thank you!

Dear TA,
I need some functions in "string.h" file to read and write result data to *.dat file.
however, if given *.i file does not include "string.h", then I can't use this functions.

what can I do in this situation?
should I manually implement function like strlen(), strtok() or something?
as I intentionally include "string.h" file to *.i file and test it, it works well.

Thank you!

안녕하세요 조교님
결과 파일을 읽고 쓰려면 스트링 관련 함수가 조금 필요한 상황입니다.
제가 이해한바로는 주어진 *.i 파일에서 #include <string.h> 내용이 없었으면 제가 따로 추가하는건 어려운 걸로 이해했습니다.
일단 일부러 *.c 파일에 #include <string.h> 를 추가하고 .i 파일을 만들어서 테스트했을땐 잘 작동하는 것 확인했습니다.
혹시 관련 함수들(strlen(), strtok() 등등)까지 제가 수동으로 구현해서 사용해야하나요?
감사합니다.

We announced the server account information on the KLMS board:
https://klms.kaist.ac.kr/mod/courseboard/article.php?id=784710&bwid=314607
Please check it before starting next homework.

Dear CS458 students,

Since a lot of CPU power is required for further assignments (e.g. fuzzing), we decided to open two more servers to balance the load for this course. You can use the new servers from now on. We will close the old remaining accounts at March 29th 00:00 KST (Wednesday).

For the detailed information, we posted an announcement on KLMS. Please check it.
https://klms.kaist.ac.kr/mod/courseboard/article.php?id=784710&bid=119014&bwid=315163

Sorry in advance. Thank you.

Dear TA,
as I understood, to solve prob 4

1. get input *.i file and insert needed code
2. convert *.file to new *.c file
3. compile and execute new *.c file

I'm managed to change given source code, insert probe, and make a new file.
but I'm stuck at converting input *.i file to new *.c file.
I googles it, but couldn't find a good answer.
I think I'm missing something.
what should I study to solve this problem?
Thank you!

조교님 안녕하세요
입력으로 들어온 파일을 수정해서 필요한 코드를 심고, 새로운 파일로 만드는것까지는 그럭저럭 해냈습니다.
그런데 *.i 파일을 *.c 파일로 변환하는게 문제의 조건이라고 이해했는데 이 부분에서 조금 어려움을 겪고 있습니다.
혹시 제가 이부분에서 뭔가 놓치고 있거나 공부해야할 부분이 있다면 알려주시면 감사하겠습니다.
감사합니다.

Hi! I have a question about designing probes, which are resistant to enums in C/C++.

As the professor said during the lecture, in C/C++, every values should be calculated in complie-time, but not runtime.
Therefore, we have to carefully insert probes not to violate this requirements.

However, I ran out of my own ideas and I feel frustrated... DX

At first, I tried to count the case which the conditional operatiors are nested in enum definitions. But it faield(my implementation failed to count such cases exhaustively), so I took another approach: if I met enum definitions, just 'ignore' nested enum definitions. Again, if failed.. I couldn't think up when to restore...(i.e. when to stop ignoring)

Could anyone give me some hints? Any tiny one is okay...!

Hi, I have some questions regarding the format of the test script and text files to submit.

1. Format of test_script

For test_script, can we safely assume that the binary grep20 in the testing environment can be executed like,

$ grep20 "foo" bar.txt

as written in the Homework description, or should we write the commands like

$ ./grep20 "foo" bar.txt

2. Format of text files

Can I submit a directory containing the text files to be used in my commands?

To be more specific, I would like to submit a directory organized as follows,

dir0
| - subdir0
  | - 0.txt, 1.txt, 2.txt, ...
| - subdir1
  | - a.txt, b.txt, 3.txt, ...

such that I can use the following command.

$ grep20 "foo" dir0/subdir0/0.txt

Am I allowed to submit such a directory containing text files?

Sorry for the late question, and thank you in advance :)

Hi, I have one question about using the _replay binary for measuring branch coverage with gcov.

When tcas-mod-crown_replay is run with crown_replay, and its coverage information is retrieved by gcov,

it seems like the conditions used in SYM_assume are considered as branches also.

For example, SYM_assume(a >= 1 && a <= 5) is regarded as 4 branches in gcov.

So for the main function where the instrumentation for CROWN is inserted, the reported branch coverage doesn't seem so accurate.

Is there a way to exclude the CROWN instrumentations from the coverage measurement in gcov?
Or, is it okay to report the coverage as it is (including the SYM_assume instrumentations)?

Hello, I have one question on logic coverage criteria.

If a predicate P has a single clause, say A, can we define its ICC?

A is the one and only major clause and there is no minor clause, so RICC and GICC would have the same meaning here.

The definition of ICC requires four cases:

• A is true when P is true
• A is false when P is true
• A is true when P is false
• A is false when P is false

But it is nonsense that A is false when P is true, and A is true when P is false. So I think for a predicate P = A, ICC is infeasible and its TR is an empty set.

How can we handle such case, to ensure that variants of ICC subsumes PC in the coverage subsumption hierarchy?

In HW1 2.a, What does it mean to 'Design a simple fault'? Should we insert or change some code?

Hi, I have some questions regarding the odd behavior of afl-fuzz.

1. Can the queue directory contain non-terminating tests?

After fuzzing with my custom driver, I have some fuzzed test case in queue directory, that doesn't terminate when I actually run them manually.

For example, there is one test in queue that is equivalent to executing ./grep -f- tmp.
(i.e., the driver eventually runs that command, where ./grep here is the non-instrumented, original grep binary)

The odd part is that, ./grep -f- tmp falls into an infinite loop, while parsing the command line options in a while loop in grep.c::main.

So I wonder how this testcase is in queue, not in hang. Does queue also hold non-terminating inputs?

2. Is the implementation choice of the grep driver up to us?

I tried implementing the custom driver following the suggestion in the class, regarding the first line as command-line options and the rest as the target of grep.

However, there are some problems, including the one mentioned above.

Instead of such a general approach, I want to implement a driver that is aware of grep command-line options, to make sure that the command-line options are at least, valid.

Is the implementation choice up to me as long as afl++ reports a coverage over 60%?

Hello,

I have a brief question for hw05.

When counting the total branches, the file "hw-kcov-llvm-v5.pdf" introduces me to double the branch conditions in a target program.

As far I understand, for the code given (i.e., example.c), I guess the total branch should be 16 rather than 18.

It is because we can say that a switch instruction has one branch condition without considering how many case statements there are.

The example code seems to have 6 br instructions, 2 switch instructions. And each instruction has one branch condition.

This leads me (6 + 2) * 2 = 16 total branches.

Am I thinking something wrong?
Quite confused..

Thank you :)

Hello!

Well, I tagged this question as HW3, but it might not be related to it directly.
It is about Clang's ASTVisitor.

In the given sample code, MyASTVisitor is defined as follows:

class MyASTVisitor : public RecursiveASTVisitor<MyASTVisitor>
{
public:
...
    bool VisitFunctionDecl(FunctionDecl *f) {
    ...

As we can see, the function VisitFunctionDecl(FunctionDecl *f) is defined & works well. To check, I defined even more specific function such as VisitSwitchStmt(SwitchStmt *s), and it works well...

However, the template clang::RecursiveASTVisitor does not define such functions, as its document shows. It only defines somewhat general functions including VisitStmt(Stmt *S).

Then how is it possible to such functions actually work?
What is the connecting link between VisitSwitchStmt(SwitchStmt *s) and RecursiveASTVisitor?

In other words, how the Clang detect such callback functions & invoke them?

Hello, here are some questions on the expected behavior of kcov and the submission template.

Behavior of kcov

1. For the following switch statement, how should kcov measure its coverage?

int a = 100;
switch (a) {
  case 100:
    a += 1;
  case 200:
    a += 2;
}

Since there is no break; in case 100, the code reaches case 200. Then can we consider that a = 100 covers the branch case 200 also?

2. What if multiple binaries happen to share coverage.dat?

Suppose that we have two (independent) source files a.i and b.i. They are instrumented and compiled to a-cov and b-cov, respectively (they are in the same directory).

The first command ./a-cov generates coverage.dat. Now, if we type command ./b-cov, what is the expected behavior?

• Should it check that coverage.dat belongs to a-cov and overwrite the contents?
• If so, what should be the criteria for validating coverage.dat? The format of coverage.dat doesn't hold the file name that it came from.

Submission Template

3. In kcov-branch-identify, can we implement outside VisitStmt and VisitFunctionDecl?

The given template file says // Fill out this function for your homework in those two functions.

But can I implement some functions in MyASTConsumer and define some class fields in MyASTVisitor?

Thank you in advance!

Hello,

I have a question when concolic testing grep with various search strategies.

As in the above lecture slide, the DFS strategy takes a very long time iterating 10000 times.

As far as I remember, it was due to some loop related reason, but I can't remember it in detail :(

Could you explain the reason why DFS took such a long time when running CROWN with grep?

Also, I think DFS is the best strategy for finding bugs as in the case of grep.

However, to reach a high coverage in a limited time budget, DFS looks like a bad choice.

So, my question is, is DFS the best strategy when the time budget is unlimited?

Then, what is the best strategy when the time budget is limited?

Thank you :)

In lecture note 10, there are examples

if ((a&&b)||c){
  s1;
}
else {
  s2;
}

transforms into

d = a&&b;
e = d||c;
if (e) {
  s1;
}
else {
  s2;
}

Considering the way we deal with clause and predicate before,
I think we should look at the predicate "e" as "e" itself rather than "d||c".

But the lecture note said transformed code has 2 clauses d and c.
So, it said to satisfy CACC, transformed code requires 3 tests.
However, if we consider "e" as "e" itself, we need 2 tests.

Am I thinking something wrong?

Hi, I have some questions regarding the definitions of EC and EPC in our lecture slides.

• Definition of EC

We have revised the definition of EPC as, TR contains each reachable path of LENGTH UP TO 2, INCLUSIVE, in G to handle some corner cases.

Then, shouldn't we also revise the definition of EC to, TR contains each reachable path of LENGTH UP TO 1, INCLUSIVE, in G.?

A counterexample would be a graph with only one node, where EC cannot subsume NC without the condition, "LENGTH UP TO 1, INCLUSIVE".

• Definition of TR

In class, we have discussed the TR of EPC for the following graph G as,

TR_EPC = { (0, 1, 2), (0, 2) }

Then all test paths that cover TR_EPC will also achieve NC and EC.
But following the definition of EPC STRICTLY, I think the TR should rather be

TR_EPC = { (0), (1), (2), (0, 1), (0, 2), (1, 2), (0, 1, 2) }

since this TR indeed contains all reachable path of LENGTH UP TO 2, INCLUSIVE, in G.

So I wonder which TR_EPC is correct.

In the homework, since tcas-mod receives arguments as input, I modified it like a triangle program we saw in the lecture. Is that okay? Or is there a way to give input as an argument using CROWN? Thank you

Hi, I have three questions regarding the measurement of branch coverage of switch in LLVM IR.

1. As we did in kcov, when counting the total number of branches, does each branch (or case) of switch count as 1 branch, not 2?

2. In relation to the first question, do we consider that visits to each case of switch counts as taking the true branch?

3. How should we measure the branch coverage of a switch statement without breaks between the cases?

Consider a switch statement in a C program,

int main() {
  char a = 'a';
  int y = 0;
  switch (a) {
    case 'a':
      y = 1;
    case 'b':
      y = 2;
    case 'c':
      y = 3;
    default:
      y = 4;
  }
}

In kcov, since a is 'a', we have considered it as taking all of the case 'a', case 'b', case 'c', and default branches.

But when the program is translated into a switch instruction in LLVM IR,

; Function Attrs: noinline nounwind optnone uwtable
define dso_local i32 @main() #0 {
entry:
  %retval = alloca i32, align 4
  %a = alloca i8, align 1
  %y = alloca i32, align 4
  store i32 0, i32* %retval, align 4
  store i8 97, i8* %a, align 1
  store i32 0, i32* %y, align 4
  %0 = load i8, i8* %a, align 1
  %conv = sext i8 %0 to i32
  switch i32 %conv, label %sw.default [
    i32 97, label %sw.bb
    i32 98, label %sw.bb1
    i32 99, label %sw.bb2
  ]

sw.bb:                                            ; preds = %entry
  store i32 1, i32* %y, align 4
  br label %sw.bb1

sw.bb1:                                           ; preds = %entry, %sw.bb
  store i32 2, i32* %y, align 4
  br label %sw.bb2

sw.bb2:                                           ; preds = %entry, %sw.bb1
  store i32 3, i32* %y, align 4
  br label %sw.default

sw.default:                                       ; preds = %entry, %sw.bb2
  store i32 4, i32* %y, align 4
  br label %sw.epilog

sw.epilog:                                        ; preds = %sw.default
  %1 = load i32, i32* %retval, align 4
  ret i32 %1
}

Here, %conv evaluates to 97, so we first switch from entry to sw.bb, then br unconditionally from sw.bb to sw.bb1, sw.bb1 to sw.bb2, and sw.bb2 to sw.default.

In our ccov assignment, we are supposed to measure the branch coverage for conditional brs and switch.

So in this case, can we consider that the program only takes the entry to sw.bb branch, and do not increment the counts for the subsequent sw.bb1, sw.bb2, sw.default that are visited via unconditional brs from sw.bb?