Solve the thread completion/destruction problem without cost on every context switch about osv

[nyh]

One downside with solution 2 proposed above is that it adds more assembly code, which we'll need to port in the future to different architectures.

To avoid this, we can use actual threads instead of bare stacks - have a per-cpu thread with a small stack, terminator_thread(), similar to how we have a per-cpu load-balancer and idle thread.

So this proposal becomes:

Solution 2a:

complete() will:

1. preempt_disable() - so the terminator_thread doesn't run in the middle of this part
2. set thread's state to "terminating" (as we do now)
3. add to this thread to a PERCPU variable termination_list (can also use a struct cpu field). No locking is needed. [but note: don't use an STL list, an malloc can cause context switch so we need to control where happens]
4. wake this CPU's terminator_thread
5. call schedule()

Each terminator thread will:

1. wait_until there's something in this cpu's termination_list
2. preempt_disable(), to prevent competition with more complete()s
3. for each thread in termination_list, assert() its status is "terminating", and unref() it.
4. preempt_enable()

[loudinthecloud]

Solution 2a is better than unrefing from within schedule() but it adds X more threads, they don't consume much (do they?) but we'll have X idle threads X load balancer and X terminators. If 2a is also an option, maybe we can unify all those threads to a single scheduler_helper thread or something.

Besides that, adding a set_stack() function for each arch is not a big problem in my opinion, we are doing it anyway in switch_to so we can reuse it.

[nyh]

On Mon, Jun 24, 2013 at 4:48 PM, loudinthecloud [email protected]:

Solution 2a is better than unrefing from within schedule() but it adds X
more threads, they don't consume much (do they?) but we'll have X idle
threads X load balancer and X terminators.

Waiting threads do not consume any resources beyond their memory use -
mostly the stack.
The scheduler doesn't even consider them, because they're not on the
runqueue of any CPU.

One thing that is "ugly" about all these extra threads is that they
"consume" a thread number, and visible on "osv info threads". To make it
less ugly, perhaps we should consider giving these "internal" threads
negative numbers, and have "osv info threads" not show them.

If 2a is also an option, maybe we can unify all those threads to a single

scheduler_helper thread or something.

A good idea. Since idle_thread, load_balancer, and terminator_thread, are
not meant to run concurrently (they run on the same CPU, and do no
waiting), there is no reason why they need to be different threads, with
different stacks. We can save a bit of memory if we use the same stack for
all three, and just have one helper thread that checks all three conditions.

Besides that, adding a set_stack() function for each arch is not a big
problem in my opinion, we are doing it anyway in switch_to so we can reuse
it.

I'm not sure that all it takes to change the stack in the middle of C code
without a disaster is a set_stack() function (I'm not even sure what it
means to change the stack in a function, maybe it needs to be a macro...).
I'm not saying it can't work, but since we already have the same mechanism
for normal context-switching, maybe it's better to just use exactly the
same mechanism, instead of duplicating it.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/10#issuecomment-19907628
.

Nadav Har'El
[email protected]

[loudinthecloud]

On 06/24/2013 05:27 PM, nyh wrote:

I'm not sure that all it takes to change the stack in the middle of C
code
without a disaster is a set_stack() function (I'm not even sure what it
means to change the stack in a function, maybe it needs to be a
macro...).
I'm not saying it can't work, but since we already have the same
mechanism
for normal context-switching, maybe it's better to just use exactly the
same mechanism, instead of duplicating it.

It's possible that set_stack() will be used both by switch_to() and
complete() and that's what I was suggesting.

Obviously, set_stack() will not work if it's not inlined...

Besides that, I support option 2a as well especially if we plan to unify
idle_thread, load_balance and "the terminator".

[avikivity]

On Mon, Jun 24, 2013 at 5:56 PM, loudinthecloud [email protected]:

On 06/24/2013 05:27 PM, nyh wrote:

I'm not sure that all it takes to change the stack in the middle of C
code
without a disaster is a set_stack() function (I'm not even sure what it
means to change the stack in a function, maybe it needs to be a
macro...).
I'm not saying it can't work, but since we already have the same
mechanism
for normal context-switching, maybe it's better to just use exactly the
same mechanism, instead of duplicating it.

It's possible that set_stack() will be used both by switch_to() and
complete() and that's what I was suggesting.

Obviously, set_stack() will not work if it's not inlined...

Besides that, I support option 2a as well especially if we plan to unify
idle_thread, load_balance and "the terminator".

Inlining is not sufficient, as the compiler may/will spill variables on
the stack.