Kernel panic in `mmu_unmap_user` in threaded app about toaruos

[klange]

Ooh, cross-thread unmap refcount mismatch… fun. Sounds like two threads probably tried to unmap the same page simultaneously and some locking or other synchronization is missing, or something neglected to reference the right page directory…

[klange]

@Vir-BryanQ Can you provide your build of ffmpeg and the test app? I would like to perform some interactive debugging with them.

[Vir-BryanQ]

Sure.

1. You can find the port of ffmpeg in http://q3z8400525.oicp.vip:25587/ffmpeg.tar.gz. I have opened this link agian.
   Just use these commands to install ffmpeg in toaruos:
   * fetch -vh -o ffmpeg.tar.gz 'http://q3z8400525.oicp.vip:25587/ffmpeg.tar.gz'
   * tar -zxvf ffmpeg.tar.gz
   * sudo cp -r ffmpeg/* /usr
   If you can't open this link, just let me know and I will check it.

2. About the test app, I have changed its source code and it may be difficult to find a built one. But I have shown the same source code in my first commend of this issue 3 days ago. You can copy it and build it in toaruos.
   Just use these commands:

• gcc -o test test.c -ltoaru_graphics -ltoaru_yutani -ltoaru_list -ltoaru_decorations -ltoaru_menu -lavformat -lavcodec -lswscale -lavutil

3. In fact, the kernel panic will not always happen and you may need to run it several times to make it happen.
   Though the video to play doesn't matter, I still give it below:
   http://q3z8400525.oicp.vip:25587/i.mp4

[klange]

Thanks, I was able to install ffmpeg and build the video player, and was able to reproduce the kernel panic. This will be very helpful in tracking down the root cause.

Also, bugs aside, very nice to see ffmpeg running under my libc. I will spend some time, possibly after tracking down this page refcount issue, to improve the video player and get this all packaged.

[klange]

By the way, I noticed a bug in the video player source you supplied; fixing it does not affect the kernel issue, but I thought I'd point it out:

                         GFX(ctx, x, y) = *((uint32_t *)frame + i);

This should reference frame->data. Currently it's interpreting the frame header data as a few pixels, which causes everything to be shifted a bit, which is why your screenshot shows the scrollbar on the left. Also, this could be improved with a memcpy for each line. I see my original source, without support for decorations, was copying the entire buffer to the window (https://github.com/klange/toaru-vidplayer/blob/master/vidplayer.c#L103).

[klange]

After some digging into tracebacks and memory state at the time of the panic, I believe this stems from some missing resource locking between threads when modifying page tables - both in the free() case calling mmu_unmap_user and in thread teardown later on. As mentioned in #263 (comment), this is an area that needs a lot of improvement.

[Vir-BryanQ]

1. Thanks for pointing out this bug in the video player. I have noticed it and fixed it two days ago. As you can see, this video player is weak and it doesn't even support audio and I have rewritten it two days ago. The new player has five threads: demuxer, audio decoder, audio player, video decoder, video player. It support almost all generic audio format and video format as it's based on ffmpeg. But it's still too weak to be a player as I haven't found a good way to implement the synchronization between video and audio.
   new player: http://q3z8400525.oicp.vip:25587/test.c
2. Abount the port of ffmpeg, ffmpeg has worked well since it's ported to toaruos but it's not perfect. As you can see, in order to build it, I got libm function ceilf() stubbed and I don't know what potantial problems it will cause. Besides, this port doesn't support pthreads as pthread_cond_xxx functions are missing.
3. About the timer support in SDL1.2, this port has a powerful player called ffplay in ffmpeg/bin and it can't work under toaruos. The ffplay is based on SDL and I get an error "SDL is not built with timer support" when I try to run it. So I think the port of SDL1.2 https://github.com/klange/SDL may be incomplete? (That's why I don't use SDL to implement the video player)
4. After I rewrote the video player, it seems that the new player won't cause the kernel panic. However, there is a new problem.
   Sometimes the toaruos will get stucked when the player is running (dead lock in kernel?). The mouse and keyboard doesn't work at all and I can do nothing and have to reboot the toaruos.

[klange]

I pushed a small patch a week ago that should improve the stability of threaded applications unmapping pages, at least during runtime. There is a lot that needs to be fixed around thread cleanup when processes exit, still.

So I think the port of SDL1.2 https://github.com/klange/SDL may be incomplete?

You're right - we are missing the timer interfaces. I think a variety of the 'standard' Unix timer implementation should be usable and will take a look at getting it into the SDL builds.

Sometimes the toaruos will get stucked when the player is running (dead lock in kernel?).

Hm, if there's no crash log from the kernel, then most likely the compositor has failed in a strange way and the system is still running but with no GUI. If that is the case, a serial console (sudo getty /dev/ttyS0) would still work if you had one running, otherwise it's a bit difficult to recover through a debugger.