push_count and pop_count overflow about c-utils

[SG7]

Good solution! I have run new code without any problems and it is passing the submitted test.

[sidcha]

@fovea1959, very good catch. Thanks for your effort, but the proposed solution isn't thread safe. Please have a look at my solution and how it passes your test case.

The rationale behind using push/pop count as opposed to total element count (as you have done) is to avoid sharing variables across push and pop code paths (threads). In your solution, if you see, circ_buf->count is being modified by push and pop. This would mean that this variable has to be locked (when integer access is non-atomic) to guarantee integrity and locking is a pain.

[fovea1959]

hmmm. test fails for me, using commit 6f65e6e and attached test program. the check in circ_gbuf_push for fullness indicates that there is room in the circular buffer even if there is not if the push_count has wrapped around size*2 and been reset to 0 and pop_count has not.

W_PROB = 1.000000, R_PROB = 0.010000
* total =    5, push_count =      0, pop_count =      0
Pushed 1
* total =    5, push_count =      1, pop_count =      0
Pushed 2
* total =    5, push_count =      2, pop_count =      0
Pushed 3
* total =    5, push_count =      3, pop_count =      0
Pushed 4
* total =    5, push_count =      4, pop_count =      0
Pushed 5
* total =    5, push_count =      5, pop_count =      0
trying to pop
Popped 1
* total =    5, push_count =      5, pop_count =      1
Pushed 6
* total =    5, push_count =      6, pop_count =      1
trying to pop
Popped 2
* total =    5, push_count =      6, pop_count =      2
Pushed 7
* total =    5, push_count =      7, pop_count =      2
trying to pop
Popped 3
* total =    5, push_count =      7, pop_count =      3
Pushed 8
* total =    5, push_count =      8, pop_count =      3
trying to pop
Popped 4
* total =    5, push_count =      8, pop_count =      4
Pushed 9
* total =    5, push_count =      9, pop_count =      4
trying to pop
Popped 5
* total =    5, push_count =      9, pop_count =      5
Pushed 10
* total =    5, push_count =     10, pop_count =      5
trying to pop
Popped 6
* total =    5, push_count =     10, pop_count =      6
Pushed 11
* total =    5, push_count =     11, pop_count =      6
trying to pop
Popped 7
* total =    5, push_count =     11, pop_count =      7
Pushed 12
* total =    5, push_count =     12, pop_count =      7
trying to pop
Popped 8
* total =    5, push_count =     12, pop_count =      8
Pushed 13
* total =    5, push_count =     13, pop_count =      8
trying to pop
Popped 9
* total =    5, push_count =     13, pop_count =      9
Pushed 14
* total =    5, push_count =     14, pop_count =      9
trying to pop
Popped 10
* total =    5, push_count =     14, pop_count =      0
trying to pop
Popped 11
* total =    5, push_count =     14, pop_count =      1
trying to pop
Popped 12
* total =    5, push_count =     14, pop_count =      2
trying to pop
Popped 13
* total =    5, push_count =     14, pop_count =      3
trying to pop
Popped 14
* total =    5, push_count =     14, pop_count =      4
trying to pop
Popped 10
* total =    5, push_count =     14, pop_count =      5
Invalid data, got 10, expected 15

test.zip

[SG7]

@cbsiddharth Are you claiming that your solution is thread safe? That would be great!

However, I am afraid that simple separation of push and pop variables may not be enough. In multi threaded environment all functions would have to be mutex protected.

In int circ_gbuf_push function we have many push_count variables.

If many threads are doing the push simultaneously then the push_count value may vary from line to line. That would not be good for the calculations.

Even, if we have only two threads - one producer and one consumer - the simple separation may not be good enough.

For example circ_gbuf_free_space could return wrong value if during its execution the pop_count or push_count was changed by the other thread.

[sidcha]

In a lot of embedded use cases, there is one producer (ISRs) and one consumer (application). The above code works well there.

circ_gbuf_free_space is read only and no real harm comes from it. I agree it's no super thread safe but it's a step in that direction.

[fovea1959]

It is *not* working with single consumer/single producer; that is the test case, and it is failing.,... On Wednesday, December 19, 2018, 10:48:19 PM EST, Siddharth Chandrasekaran <[email protected]> wrote: In a lot of embedded use cases, there is one producer (ISRs) and one consumer (application). The above code works well there. circ_gbuf_free_space is read only and no real harm comes from it. I agree it's no super thread safe but it's a step in that direction. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[SG7]

@fovea1959 Your test exposed the problem with single consumer and single producer in ONE thread!

That as you indicated is a problem with the lack of the synchronization regarding wrapping push_count and pop_count.

In the worst case the problem may not be fixable by the way how push and pop operates now. One cannot run two separate entities operating in totally independent way over the same container. Push and pop has to know more about themselves.

[sidcha]

@fovea1959, Yes. I see you mention it's not working. That's why I reopened this issue. I. Will get to it ASAP.

The explanation was for @SG7 on his comment on thread safety, which is valid with or without the issue you mention.

[sidcha]

@SG7, don't loose hope just yet. It can be done. This was a work in progress that got interrupted. I will finish this soon.

[sidcha]

@fovea1959, this issue is resolved. I have added a new tests directory to have more thorough testing. I have also added your test case into test/test-circular-buffer.c under test_probabilistic().