fatal firmware crash on attempt to play video on chromium-browser which leads to complete system freeze after some time about firmware

[pelwell]

Is this replacing raspberrypi/linux#6170? You seem to have found a page (or class of pages) that causes Chromium to exhaust the firmware's resources, and are now mining a rich seam of similar but non-identical problems.

[qrp73]

I'm not sure, but at a glance yes - it seems that this issue leads to issue raspberrypi/linux#6170. I discovered it because I'm tired from system freeze issue which started to happens very often since kernel 6.6.28 or something like that. So I tried to find a short steps to reproduce it and found this issue.

Note, that when this firmware crash happens for a first time it don't leads to a system freeze, like it described in #6170, but I believe that this firmware crash is a root of cause for system freeze which happens later when this firmware crash happens (maybe several times). Also after this firmware crash something is going wrong in firmware, because it leads to problems with some video playback and no way to properly reboot the system with sudo reboot, because it hung the system on waiting for process end.

Hope it will be fixed soon, because it leads to unexpected system freeze and crash, which leads to lose unsaved work.

[qrp73]

looks like today update for chromium-codecs-ffmpeg-extra:arm64 (124.0.6367.73-rpt1, 125.0.6422.133-rpt1), chromium-browser:arm64 (124.0.6367.73-rpt1, 125.0.6422.133-rpt1), chromium-browser-l10n:arm64 (124.0.6367.73-rpt1, 125.0.6422.133-rpt1) solved the issue.

At least with original steps to reproduce I can play videos with no crash.

[qrp73]

After some runtime with new updates, I confirm that system crash issue due to playing video on browser disappears.

Now there is no crash, but sometimes the following error appears in the log, but now it happens rare and don't lead to system crash:

[14696.596892] vc4-drm gpu: swiotlb buffer is full (sz: 491520 bytes), total 32768 (slots), used 418 (slots)
[14696.606252] vc4-drm gpu: swiotlb buffer is full (sz: 524288 bytes), total 32768 (slots), used 934 (slots)
[14698.303095] vc4-drm gpu: swiotlb buffer is full (sz: 491520 bytes), total 32768 (slots), used 418 (slots)
[14698.908934] vc4-drm gpu: swiotlb buffer is full (sz: 524288 bytes), total 32768 (slots), used 934 (slots)
[14708.127736] v3d fec00000.v3d: MMU error from client CLE (4) at 0x5201000, pte invalid
[14747.703863] vc4-drm gpu: swiotlb buffer is full (sz: 524288 bytes), total 32768 (slots), used 552 (slots)
[14747.748102] vc4-drm gpu: swiotlb buffer is full (sz: 524288 bytes), total 32768 (slots), used 552 (slots)
[31434.235549] ------------[ cut here ]------------
[31434.235570] WARNING: CPU: 0 PID: 40051 at drivers/media/common/videobuf2/videobuf2-core.c:2024 __vb2_queue_cancel+0x220/0x2a0 [videobuf2_common]
[31434.235633] Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq cmac algif_hash aes_arm64 aes_generic algif_skcipher af_alg bnep brcmfmac_wcc vc4 brcmfmac snd_soc_hdmi_codec joydev drm_display_helper snd_usb_audio cec binfmt_misc snd_hwdep drm_dma_helper bcm2835_codec(C) snd_usbmidi_lib drm_kms_helper rpivid_hevc(C) snd_rawmidi snd_seq_device bcm2835_isp(C) hci_uart brcmutil v4l2_mem2mem snd_soc_core btbcm cfg80211 bluetooth v3d bcm2835_v4l2(C) gpu_sched bcm2835_mmal_vchiq(C) videobuf2_vmalloc ecdh_generic snd_compress videobuf2_dma_contig ecc drm_shmem_helper videobuf2_memops snd_bcm2835(C) snd_pcm_dmaengine rfkill videobuf2_v4l2 snd_pcm videodev raspberrypi_hwmon libaes snd_timer i2c_bcm2835 i2c_brcmstb vc_sm_cma(C) videobuf2_common snd raspberrypi_gpiomem mc nvmem_rmem uio_pdrv_genirq uio drm i2c_dev fuse dm_mod drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[31434.235966] CPU: 0 PID: 40051 Comm: ThreadPoolSingl Tainted: G         C         6.6.32-v8+ #1
[31434.235979] Hardware name: Raspberry Pi 4 Model B Rev 1.5 (DT)
[31434.235986] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[31434.235999] pc : __vb2_queue_cancel+0x220/0x2a0 [videobuf2_common]
[31434.236038] lr : __vb2_queue_cancel+0x38/0x2a0 [videobuf2_common]
[31434.236074] sp : ffffffc081dc3ae0
[31434.236080] x29: ffffffc081dc3ae0 x28: ffffffd729731e18 x27: ffffff8042158000
[31434.236102] x26: ffffff804264ad30 x25: 0000000000000000 x24: 0000000040045613
[31434.236122] x23: 0000000000000000 x22: ffffff809a4ac028 x21: ffffff809a4ac028
[31434.236141] x20: 0000000000000009 x19: ffffff809a4ac028 x18: ffffffffffffffff
[31434.236160] x17: 0000000000000005 x16: ffffffd7944f8df8 x15: ffffffc101dc3787
[31434.236178] x14: 000000000000000c x13: ffffff8043b80028 x12: 0000000000000000
[31434.236197] x11: ffffff808b44b990 x10: ffffff808b44b908 x9 : ffffffd7297909f4
[31434.236216] x8 : ffffff808b44b930 x7 : 0000000000000001 x6 : ffffc00000000000
[31434.236234] x5 : ffffff804264b01c x4 : 0000000000000000 x3 : 0000000000000000
[31434.236252] x2 : ffffff80eb888000 x1 : 0000000000000000 x0 : 0000000000000002
[31434.236271] Call trace:
[31434.236278]  __vb2_queue_cancel+0x220/0x2a0 [videobuf2_common]
[31434.236315]  vb2_core_streamoff+0x2c/0xe0 [videobuf2_common]
[31434.236350]  vb2_streamoff+0x24/0x80 [videobuf2_v4l2]
[31434.236383]  v4l2_m2m_streamoff+0x80/0x160 [v4l2_mem2mem]
[31434.236419]  v4l2_m2m_ioctl_streamoff+0x20/0x38 [v4l2_mem2mem]
[31434.236452]  v4l_streamoff+0x2c/0x40 [videodev]
[31434.236552]  __video_do_ioctl+0x194/0x400 [videodev]
[31434.236636]  video_usercopy+0x1e4/0x780 [videodev]
[31434.236719]  video_ioctl2+0x20/0x40 [videodev]
[31434.236801]  v4l2_ioctl+0x48/0x70 [videodev]
[31434.236883]  __arm64_sys_ioctl+0xb0/0x100
[31434.236906]  invoke_syscall+0x50/0x128
[31434.236922]  el0_svc_common.constprop.0+0x48/0xf8
[31434.236937]  do_el0_svc+0x28/0x40
[31434.236950]  el0_svc+0x40/0xf8
[31434.236967]  el0t_64_sync_handler+0x13c/0x158
[31434.236976]  el0t_64_sync+0x190/0x198
[31434.236987] ---[ end trace 0000000000000000 ]---
[31434.236998] videobuf2_common: driver bug: stop_streaming operation is leaving buf 000000000b086321 in active state
[31434.237009] videobuf2_common: driver bug: stop_streaming operation is leaving buf 000000009f9af9a2 in active state
[31434.237279] ------------[ cut here ]------------
[31434.237293] WARNING: CPU: 0 PID: 57920 at drivers/media/common/videobuf2/videobuf2-core.c:1049 vb2_buffer_done+0x29c/0x2c0 [videobuf2_common]
[31434.237348] Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq cmac algif_hash aes_arm64 aes_generic algif_skcipher af_alg bnep brcmfmac_wcc vc4 brcmfmac snd_soc_hdmi_codec joydev drm_display_helper snd_usb_audio cec binfmt_misc snd_hwdep drm_dma_helper bcm2835_codec(C) snd_usbmidi_lib drm_kms_helper rpivid_hevc(C) snd_rawmidi snd_seq_device bcm2835_isp(C) hci_uart brcmutil v4l2_mem2mem snd_soc_core btbcm cfg80211 bluetooth v3d bcm2835_v4l2(C) gpu_sched bcm2835_mmal_vchiq(C) videobuf2_vmalloc ecdh_generic snd_compress videobuf2_dma_contig ecc drm_shmem_helper videobuf2_memops snd_bcm2835(C) snd_pcm_dmaengine rfkill videobuf2_v4l2 snd_pcm videodev raspberrypi_hwmon libaes snd_timer i2c_bcm2835 i2c_brcmstb vc_sm_cma(C) videobuf2_common snd raspberrypi_gpiomem mc nvmem_rmem uio_pdrv_genirq uio drm i2c_dev fuse dm_mod drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[31434.237673] CPU: 0 PID: 57920 Comm: kworker/0:0 Tainted: G        WC         6.6.32-v8+ #1
[31434.237685] Hardware name: Raspberry Pi 4 Model B Rev 1.5 (DT)
[31434.237693] Workqueue: events buffer_work_cb [bcm2835_mmal_vchiq]
[31434.237725] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[31434.237737] pc : vb2_buffer_done+0x29c/0x2c0 [videobuf2_common]
[31434.237773] lr : op_buffer_cb+0x3c4/0x710 [bcm2835_codec]
[31434.237798] sp : ffffffc082c73cf0
[31434.237804] x29: ffffffc082c73cf0 x28: ffffffd794e269c0 x27: ffffffd794e26000
[31434.237824] x26: ffffff80458287c0 x25: ffffff8045828780 x24: ffffff8042158000
[31434.237843] x23: ffffffd729a87780 x22: ffffff809a4ac028 x21: 0000000000000000
[31434.237862] x20: ffffff8043b80550 x19: ffffff80cfb78800 x18: ffffffffffffffff
[31434.237880] x17: 0000000000000000 x16: ffffffd7938f5a18 x15: ffffffc101dc37c7
[31434.237899] x14: 0000000000000003 x13: ffffffc081dc37cd x12: 000000000000000d
[31434.237917] x11: 000000000000033b x10: 0000000000001a30 x9 : ffffffd729a8323c
[31434.237936] x8 : ffffff804316b950 x7 : 0000000000000002 x6 : 00001c96529d1800
[31434.237954] x5 : 00000000410fd080 x4 : 0000000000000000 x3 : 0000000000000000
[31434.237972] x2 : 0000000000000000 x1 : 0000000000000003 x0 : 0000000000000000
[31434.237990] Call trace:
[31434.237996]  vb2_buffer_done+0x29c/0x2c0 [videobuf2_common]
[31434.238033]  op_buffer_cb+0x3c4/0x710 [bcm2835_codec]
[31434.238056]  buffer_work_cb+0x78/0xf0 [bcm2835_mmal_vchiq]
[31434.238082]  process_one_work+0x148/0x388
[31434.238094]  worker_thread+0x338/0x450
[31434.238103]  kthread+0x120/0x130
[31434.238118]  ret_from_fork+0x10/0x20
[31434.238131] ---[ end trace 0000000000000000 ]---
[31434.238677] ------------[ cut here ]------------
[31434.238711] WARNING: CPU: 0 PID: 57920 at drivers/media/common/videobuf2/videobuf2-core.c:1049 vb2_buffer_done+0x29c/0x2c0 [videobuf2_common]
[31434.238784] Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq cmac algif_hash aes_arm64 aes_generic algif_skcipher af_alg bnep brcmfmac_wcc vc4 brcmfmac snd_soc_hdmi_codec joydev drm_display_helper snd_usb_audio cec binfmt_misc snd_hwdep drm_dma_helper bcm2835_codec(C) snd_usbmidi_lib drm_kms_helper rpivid_hevc(C) snd_rawmidi snd_seq_device bcm2835_isp(C) hci_uart brcmutil v4l2_mem2mem snd_soc_core btbcm cfg80211 bluetooth v3d bcm2835_v4l2(C) gpu_sched bcm2835_mmal_vchiq(C) videobuf2_vmalloc ecdh_generic snd_compress videobuf2_dma_contig ecc drm_shmem_helper videobuf2_memops snd_bcm2835(C) snd_pcm_dmaengine rfkill videobuf2_v4l2 snd_pcm videodev raspberrypi_hwmon libaes snd_timer i2c_bcm2835 i2c_brcmstb vc_sm_cma(C) videobuf2_common snd raspberrypi_gpiomem mc nvmem_rmem uio_pdrv_genirq uio drm i2c_dev fuse dm_mod drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[31434.239122] CPU: 0 PID: 57920 Comm: kworker/0:0 Tainted: G        WC         6.6.32-v8+ #1
[31434.239135] Hardware name: Raspberry Pi 4 Model B Rev 1.5 (DT)
[31434.239143] Workqueue: events buffer_work_cb [bcm2835_mmal_vchiq]
[31434.239179] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[31434.239192] pc : vb2_buffer_done+0x29c/0x2c0 [videobuf2_common]
[31434.239228] lr : op_buffer_cb+0x3c4/0x710 [bcm2835_codec]
[31434.239254] sp : ffffffc082c73cf0
[31434.239260] x29: ffffffc082c73cf0 x28: ffffffd794e269c0 x27: ffffffd794e26000
[31434.239281] x26: ffffff80458287c0 x25: ffffff8045828780 x24: ffffff8042158000
[31434.239299] x23: ffffffd729a87780 x22: ffffff809a4ac028 x21: 0000000000000000
[31434.239318] x20: ffffff8043b80550 x19: ffffff8042126800 x18: ffffffffffffffff
[31434.239336] x17: 0000000000000000 x16: ffffffd7938f5a18 x15: ffffffc101dc37c7
[31434.239355] x14: 0000000000000003 x13: ffffffc081dc37cd x12: 000000000000000d
[31434.239373] x11: 000000000000033b x10: 0000000000001a30 x9 : ffffffd729a8323c
[31434.239392] x8 : fefefefefefefeff x7 : 0000000000000018 x6 : ffffff8045828800
[31434.239410] x5 : 000073746e657665 x4 : 0000000000000000 x3 : 0000000000000000
[31434.239429] x2 : 0000000000000000 x1 : 0000000000000003 x0 : 0000000000000000
[31434.239447] Call trace:
[31434.239454]  vb2_buffer_done+0x29c/0x2c0 [videobuf2_common]
[31434.239491]  op_buffer_cb+0x3c4/0x710 [bcm2835_codec]
[31434.239513]  buffer_work_cb+0x78/0xf0 [bcm2835_mmal_vchiq]
[31434.239540]  process_one_work+0x148/0x388
[31434.239555]  worker_thread+0x338/0x450
[31434.239564]  kthread+0x120/0x130
[31434.239580]  ret_from_fork+0x10/0x20
[31434.239595] ---[ end trace 0000000000000000 ]---
[49387.692236] ------------[ cut here ]------------
[49387.692258] WARNING: CPU: 3 PID: 40051 at drivers/media/common/videobuf2/videobuf2-core.c:2024 __vb2_queue_cancel+0x220/0x2a0 [videobuf2_common]
[49387.692324] Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq cmac algif_hash aes_arm64 aes_generic algif_skcipher af_alg bnep brcmfmac_wcc vc4 brcmfmac snd_soc_hdmi_codec joydev drm_display_helper snd_usb_audio cec binfmt_misc snd_hwdep drm_dma_helper bcm2835_codec(C) snd_usbmidi_lib drm_kms_helper rpivid_hevc(C) snd_rawmidi snd_seq_device bcm2835_isp(C) hci_uart brcmutil v4l2_mem2mem snd_soc_core btbcm cfg80211 bluetooth v3d bcm2835_v4l2(C) gpu_sched bcm2835_mmal_vchiq(C) videobuf2_vmalloc ecdh_generic snd_compress videobuf2_dma_contig ecc drm_shmem_helper videobuf2_memops snd_bcm2835(C) snd_pcm_dmaengine rfkill videobuf2_v4l2 snd_pcm videodev raspberrypi_hwmon libaes snd_timer i2c_bcm2835 i2c_brcmstb vc_sm_cma(C) videobuf2_common snd raspberrypi_gpiomem mc nvmem_rmem uio_pdrv_genirq uio drm i2c_dev fuse dm_mod drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
[49387.692656] CPU: 3 PID: 40051 Comm: ThreadPoolSingl Tainted: G        WC         6.6.32-v8+ #1
[49387.692670] Hardware name: Raspberry Pi 4 Model B Rev 1.5 (DT)
[49387.692678] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[49387.692691] pc : __vb2_queue_cancel+0x220/0x2a0 [videobuf2_common]
[49387.692730] lr : __vb2_queue_cancel+0x38/0x2a0 [videobuf2_common]
[49387.692765] sp : ffffffc081dc3ae0
[49387.692771] x29: ffffffc081dc3ae0 x28: ffffffd729731e18 x27: ffffff8042158000
[49387.692793] x26: ffffff804264ad30 x25: 0000000000000000 x24: 0000000040045613
[49387.692813] x23: 0000000000000000 x22: ffffff809a4ac028 x21: ffffff809a4ac028
[49387.692832] x20: 0000000000000009 x19: ffffff809a4ac028 x18: ffffffffffffffff
[49387.692851] x17: 00000001fb7e0880 x16: ffffffd7944f8df8 x15: ffffffc101dc3787
[49387.692870] x14: 000000000000000c x13: ffffff8043b80028 x12: 0000000000000000
[49387.692888] x11: ffffff808b44b990 x10: ffffff808b44b908 x9 : ffffffd7297909f4
[49387.692907] x8 : ffffff808b44b930 x7 : 0000000000000001 x6 : ffffc00000000000
[49387.692925] x5 : ffffff804264b01c x4 : 0000000000000000 x3 : 0000000000000000
[49387.692943] x2 : ffffff80eb888000 x1 : 0000000000000000 x0 : 0000000000000002
[49387.692962] Call trace:
[49387.692969]  __vb2_queue_cancel+0x220/0x2a0 [videobuf2_common]
[49387.693006]  vb2_core_streamoff+0x2c/0xe0 [videobuf2_common]
[49387.693042]  vb2_streamoff+0x24/0x80 [videobuf2_v4l2]
[49387.693074]  v4l2_m2m_streamoff+0x80/0x160 [v4l2_mem2mem]
[49387.693111]  v4l2_m2m_ioctl_streamoff+0x20/0x38 [v4l2_mem2mem]
[49387.693144]  v4l_streamoff+0x2c/0x40 [videodev]
[49387.693246]  __video_do_ioctl+0x194/0x400 [videodev]
[49387.693334]  video_usercopy+0x1e4/0x780 [videodev]
[49387.693424]  video_ioctl2+0x20/0x40 [videodev]
[49387.693507]  v4l2_ioctl+0x48/0x70 [videodev]
[49387.693595]  __arm64_sys_ioctl+0xb0/0x100
[49387.693623]  invoke_syscall+0x50/0x128
[49387.693639]  el0_svc_common.constprop.0+0x48/0xf8
[49387.693655]  do_el0_svc+0x28/0x40
[49387.693669]  el0_svc+0x40/0xf8
[49387.693687]  el0t_64_sync_handler+0x13c/0x158
[49387.693696]  el0t_64_sync+0x190/0x198
[49387.693708] ---[ end trace 0000000000000000 ]---

[qrp73]

it looks like issue is fixed with updates, running for several days with no issue. So, I close the issue