UTP_Write not writing? about libutp

[ghazel]

From the README:

The write side of the socket is proactive, and you call UTP_Write to indicate the number of bytes
you wish to write. As packets are created, the on_write callback is called for each packet, so you
can fill the buffers with data.

And from the header:

// The uTP socket layer calls this to fill the outgoing buffer with bytes.
// The uTP layer takes responsibility that those bytes will be delivered.
typedef void UTPOnWriteProc(void *userdata, byte *bytes, size_t count);

So you would copy count bytes into the buffer pointed to by bytes when the on_write callback occurred. Data is not written until the on_write callback asks for it -- UTP_Write is simply to indicate how many bytes you have ready for writing.

[jacobmetrick]

Ok, I think I understand this, but if I call UTP_Write(utp_socket, 100), I would expect a call of my UTPOnWriteProc, but the function that I register never seems to be called (I put printf statements in my UTPOnWriteProc). My UTPOnWriteProc looks like this:

void utp_write(void* socket, byte* bytes, size_t count)
{
        int i;
        // copy data over to bytes
        for (i = 0; i < count; i++) {
                bytes[i] = *(g_dbuf++);
        }
        printf("did some writn\n");
        g_total_count += count;
}

where g_dbuf is a global variable holding the bytes, and g_total_count is a global variable counting how many bytes have been sent total.

Why doesn't the utp_write get called?

[ghazel]

It's possible the socket is not writable at this time. What does UTP_Write return?

To answer your previous question; you call UTP_IsIncomingUTP to process the received packet with libutp. This is necessary as all uTP packets are UDP packets, including the connection handshakes, ACKs, etc. You can't send on a uTP socket until it is connected, and you can't connect without processing the replied ACK.

So my guess is that you aren't calling UTP_Connect at all. When forming an outgoing connection you must call UTP_Connect and then wait for the on_state callback to give you UTP_STATE_CONNECT (and in the future, UTP_STATE_WRITABLE as the write buffer has more room available).

[jacobmetrick]

Thanks so much for the help thus far.

I have set up a recvfrom / UTP_IsIncomingUTP loop on both my send side and receive side. I think it's in some sort of deadlock state however. My program needs to send chunks of data at a time.

From what I can tell, I call UTP_Connect, my send side goes into the recvfrom loop, acknowledges the handshake from the receive side, the state changes to UTP_STATE_CONNECT which triggers a UTP_Write (a la utp_file/utp_send.cpp), and I write a chunk. However, whenever I call UTP_Write outside of the utp_state function, it returns 0.

Is the state changing without me knowing it? While I'm connected, shouldn't I be fine? Why are these UTP_Writes failing? Is it possible the buffer is full, and if so, is there a way to check that state, or increase the buffer?

Thanks again.

[ghazel]

UTP_Write returning 0 does mean the buffer is full. When there is space in the buffer again, on_state will be called with UTP_STATE_WRITABLE.

[jacobmetrick]

I can't understand why the buffer would be full though. The utp_write callback only gets called once (corresponding, I suppose, with the one UTP_Write function that actually works). Do the failed UTP_Write calls fill up the buffer as well? Is there a way to flush the buffer?

[ghazel]

The buffer automatically flushes. It can also be quite small to start - it is related to the number of packets in flight on the network, which is adjusted according to latency and packet loss. The odd write system libutp has is designed to minimize the size of buffers, and fetch data only when it needs it, with excessive copying.

[jacobmetrick]

Can this buffer flush or change size while blocking in the recv call? If not, that may be causing the problem I have where both the receiver and the sender are stuck in their recv calls at the same time, and neither can break out because the sender is not sending. If so, it still doesn't seem to be automatically flushing or changing size for me.

Is it possible that the receive buffer is the problem? Because I don't do anything with the received bytes except count them. Do *bytes in on_read need to be freed?

[ghazel]

The on_read call doesn't need to do anything at all. It is your opportunity to read the bytes from the network, but you can just ignore them if you like.

See the utp_file directory for an example of a sender and a receiver.

[jacobmetrick]

Sorry for all the questions: I found the solution to my problems. I needed to alternate between UTP_CheckTimeouts() as well as checking my socket for new data, which I wasn't doing previously. Making sure there was no blocking on the recv call as well as using UTP_CheckTimeouts() made everything work like a charm. I should have known that since the library isn't threadsafe, something would need to call something actively for the previously registered callbacks to work.

Many thanks once again.

[jacobmetrick]

Hi;

Largely got everything to work, as I mentioned in my previous comments. However, in situations with high packet loss, I am running up against an issue.

I am running my two programs on VMs connected by a bridge. I place a delay of 50ms +- 10ms and a 10% chance of packet drop. They send and receive normally, but then a weird state is achieved. My code is in the select/CheckTimeouts loop, and, after no reaction for a few cycles of the loop, it will keep calling utp_state with a state of UTP_STATE_WRITABLE. In response, and similarly to the utp_file example, in such cases, I call UTP_Write(). However, utp_write is never called, and utp_state keeps getting called and calling UTP_Write, but no progress is ever made.

Thanks

[ghazel]

What happens if you let it sit for minutes? Does the connection eventually time out?

[jacobmetrick]

I've let it run for circa 10 minutes now and it hasn't yet timed out. Occasionally acks are sent/received, but as far as I can tell, nothing else is happening.

[ghazel]

Those might be the keep-alives. What does UTP_Write return?

[jacobmetrick]

It's returning 0. But shouldn't it be writable?

[ghazel]

Returning 0 just means it can accept some, but not all, of your data. You should get a corresponding utp_write later.

Are you passing <= 0 to UTP_Write by accident? Can you trace into UTP_Write and see how many bytes it's figured out it can take?

[jacobmetrick]

After a little investigation, it seems as if it is never entering the loop in line 2727,

while(conn->is_writeable(num_to_send) {

When looking into that function, I have used a few helpful print statements for diagnosis when in the looped state:

6910 < 100 (send_quota / 100 < (int32)to_write) satisfaction returns false
0 >= 510 (cur_window_packets >= OUTGOING_BUFFER_MAX_SIZE - 1) satisfaction returns false
0 + 1382 <= 926 (cur_window + packet_size <= max_send) satisfaction returns true
wasn't true off of packet pacing conditional... function will return false

These results repeat without variation.

[ghazel]

Why is max_send so low? What are the values of max_window, opt_sndbuf and max_window_user?

[jacobmetrick]

opt_sndbuf and max_window_user are far larger. max_window on the other hand, on a recent run, repeated at the value 565. Immediately before this its value was 10. I have seen it at multitude of values. It seems the take away here is that the value is less than 1382, which is the packet size I have always encountered.

I decided to investigate where the value of max_window is set immediately before the bad state. It's set in the function apply_ledbat_ccontrol. Here's a standard run before it fails.

scaled gain is -2915.819943 and max_window is 13747
max_window changed to 10831 in UTPSocket::apply_ledbat_ccontrol
scaled gain is -3671.784264 and max_window is 10831
max_window changed to 7159 in UTPSocket::apply_ledbat_ccontrol
scaled gain is -2321.702087 and max_window is 7159
max_window changed to 4837 in UTPSocket::apply_ledbat_ccontrol
scaled gain is -3770.194286 and max_window is 4837
max_window changed to 1066 in UTPSocket::apply_ledbat_ccontrol
scaled gain is -2544.608336 and max_window is 1066
max_window changed to 10 in UTPSocket::apply_ledbat_ccontrol
scaled gain is 291.638298 and max_window is 10
max_window changed to 301 in UTPSocket::apply_ledbat_ccontrol

[ghazel]

Hm. Well, that seems like a bug. However if packet pacing is on, it seems it should allow you to send anyway.

What are the values of USE_PACKET_PACING, max_window, to_write, cur_window, and cur_window_packets at the end of is_writable?

[jacobmetrick]

USE_PACKET_PACING: 1, max_window: 607, to_write: 100, cur_window: 0, cur_window_packets:0
USE_PACKET_PACING: 1, max_window: 404, to_write: 100, cur_window: 0, cur_window_packets:0
USE_PACKET_PACING: 1, max_window: 594, to_write: 100, cur_window: 0, cur_window_packets:0

are all various values that I have gotten stuck in. Each one repeats ad nauseum.

[ghazel]

Hm. The error seems to be that is_writable is checking to see if we have room for a full packet, not the amount to be written.

If you replace instances of packet_size in is_writable with to_write instead, does the problem go away?