dmx_receive returns too early and data size is wrong about esp_dmx

[arneboe]

I did some digging and it seems that this bug/behavior was introduced in commit 834c131.

Commit 7f24799 is the last one that is still working as expected.

[someweisguy]

Great catch! I'll investigate.

[someweisguy]

I did some poking around, and it appears this problem is much older than 834c131. I'm very frustrated with myself for this!

First, there was an issue where the function dmx_receive() was waiting for a task notification for much too long due to being passed an inappropriate variable. This would cause occasional watchdog crashes. I've fixed this error.

The second error which I think I've fixed has to do with calling dmx_receive() after you already have called dmx_receive() but before actually receiving new DMX data. The expected behavior is that the function blocks until a new packet has been received. The function was instead returning early to report that a new packet has been received when it hadn't. This is why we were seeing 159Hz.

The third error may need a bit of background first. A standard DMX packet is 513 bytes long. According to the DMX specification, however, there is no minimum packet size. I've used lighting consoles that output 197 bytes per packet instead of the typical 513. There is some logic in the DMX driver that is intended to allow for shorter DMX packets to be processed. However, it appears this logic is not working as intended. The result is that the first several packets received are reporting a much shorter packet size than the actual size received. Eventually the DMX driver reports the correct packet size.

I'm going to merge this branch with the dev branch and do some work to correct these errors.

[arneboe]

wow! Thanks for the quick reply :-)
I will do some more digging as well and try how to understand how the dmx driver works.

[arneboe]

A question that has nothing to do with this ticket, feel free to ignore it :D (I am still new to the whole esp32 world, coming from stm32 :D)

I am trying to understand the dmx_uart_isr and was wondering if there is a reason why you set uart_ll_set_rxfifo_full_thr to 1.
If I understand the code correctly this means that you will get an interrupt for every byte that the uart receives. That seems somewhat cpu-time consuming. Wouldn't it be possible to set the buffer size to 513 and ignore the DMX_INTR_RX_DATA?
If the buffer is big enough it should be possible to just listen to the DMX_INTR_RX_ERR and DMX_INTR_RX_BREAK interrupts to get all dmx data. This would free a massive amount of cpu-time.

[someweisguy]

Great question! What you're describing would work too. This library already uses a large FIFO for sending data, just not receiving data. The ESP32 UART is unusual because it has that UART FIFO. Most other microcontroller UARTs just have a single byte FIFO, as far as I know. It would be possible to setup the ESP32 so that it can receive multiple bytes of data before the interrupt needs to fire.

If I recall correctly, the base ESP32 has somewhere around 1KB of FIFO space shared between 3 UARTS. Each UART can be configured to use different sizes of FIFO. By default, each UART's FIFO is 128 bytes. I think it is possible to configure a UART so that its FIFO is 513 bytes long. The problem with this is that ESP-IDF doesn't actually support this feature in software. The 128 byte FIFO is actually a hardcoded integer literal in ESP-IDF. I am not sure why they've done this. Perhaps this feature doesn't actually work in the silicon? Furthermore, the ESP32 technical reference manual doesn't explain very well how to expand the UART FIFOs. The registers controlling the FIFOs can be seen on page 364, but page 338 mentions that expanding the FIFO sizes can also overwrite the FIFOs of other UARTs. So the whole thing just seems messy and difficult.

It would be possible to set the RX FIFO full threshold to the full 128 bytes, though! As you know, you can set the full threshold to whatever number you want. The ESP-IDF UART implementation uses 120 by default. When you have 120 bytes or more in your FIFO, the interrupt fires and you can read your data. If you only receive 119 bytes the interrupt doesn't fire. What happens instead is the ESP32 waits until a user-specified timeout occurs. When that timeout occurs the interrupt fires and you read your data. I think I went with the 1 byte full threshold because I concerned about latency - particularly when it comes to using RDM. Now that I am thinking about it again though, I'm not sure that latency would be a problem. The minimum RX FIFO timeout length would be somewhere around 44 microseconds. This would mean that you receive 119 bytes and then 44 microseconds later you can read data. I might play with this and see what happens!

[someweisguy]

This appears to be fixed. I've merged changes back into the release/v3.0 branch. Let me know if you have any questions or if you find any more issues! :)

[arneboe]

Thanks for the detailed explanation.
The 128 byte limit seems a bit strange and smells like a workaround for a bug in silicone :)
I don't understand why you would need the timeout interrupt at all in the dmx use-case.
From my understanding in dmx you either get another byte or you get a break. The break (or the frame error) should be detected when the stop bits of the 514th byte are low, this happens ~44us after the last byte was received.

And thanks again for the quick fix! I am back at work but will continue playing with the esp during the weekend :)

[someweisguy]

Yep, you're right about not needing the timeout interrupt for DMX as well. It really was just a choice between low latency and high CPU efficiency. I went with the low latency option. Though I think in v2 of the library I went with the high CPU efficiency option? I can't recall.

In my testing, I found that some lighting consoles only send DMX packets every 25ms. So waiting for the next DMX break to report receiving a new packet would mean that the library would be reporting the packet approximately 24.97ms late. I felt that this was too long of a delay.

In the RDM use case, it may be possible to wait until the next DMX break. Some RDM packets don't actually send a break though so it would be tricky to decide when to wait for a break or not. Sounds like it would be worth looking into! But I'm going to finish getting RDM working first and then try to optimize later. :)

Excellent questions!