Mc socket is keep reading but no data is readed about memcached-java-client

[newroot]

which version are you using for the memcached client?

[esah]

I have the same problem! My application server (jetty 7) stopped the same way with the same thread dump:

"qtp248786184-1889" prio=10 tid=0x00007fa57fadd800 nid=0x246d runnable [0x00007fa581480000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
at sun.nio.ch.IOUtil.read(IOUtil.java:169)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
- locked <0x00000000be622510> (a java.lang.Object)
at com.schooner.MemCached.SockInputStream.readFromChannel(Unknown Source)
at com.schooner.MemCached.SockInputStream.read(Unknown Source)
at com.schooner.MemCached.SockInputStream.getLine(Unknown Source)
at com.schooner.MemCached.AscIIClient.get(Unknown Source)
at com.schooner.MemCached.AscIIClient.get(Unknown Source)
at com.schooner.MemCached.AscIIClient.get(Unknown Source)
at com.danga.MemCached.MemCachedClient.get(Unknown Source)

The bug is appeared after upgrading memcache client from 2.0.1 to 2.6.1.

It's a real problem, so i could't find solution and enforced to switch to spymemcache client.

[newroot]

HI esah,

Since we could not repeat the issue in our environment, can you switch to 2.6.0 or even previous version to verify the same issue?

[esah]

Unfortunately i can't repeat it cause the issue is appeared only in production under high load next day after release.
All web requests to my app server are stuck. The machine has 55 tcp open connections to memcached servers. Cpu is 0% and huge free memory.
I really don't know what stay behind the bug and how to repeat it in test environment.
Memcached version is 1.2.8-1
Protocol is text.

[jpvirolainen]

Hi, we had the same issue happen with our app cluster today. 8 out of 10 servers were stuck in the same place as the original issue in this thread.
We're running version 2.6.0.

[PiouseLeo]

Unfortunately I met the same problem.
client version 2.5.1
memcached version 1.4.5
web server: resin-3.0.21

my stack trace:

"xxx-1-thread-1" daemon prio=10 tid=0x0000000053962000 nid=0x5cd9 runnable [0x00000000520fd000..0x00000000520fdb10]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:200)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
- locked <0x00002aab0f1e05e0> (a java.lang.Object)
at com.schooner.MemCached.SockInputStream.readFromChannel(SockInputStream.java:124)
at com.schooner.MemCached.SockInputStream.read(SockInputStream.java:109)
at com.schooner.MemCached.SockInputStream.getLine(SockInputStream.java:173)
at com.schooner.MemCached.AscIIClient.get(AscIIClient.java:770)
at com.schooner.MemCached.AscIIClient.get(AscIIClient.java:612)
at com.schooner.MemCached.AscIIClient.get(AscIIClient.java:608)
at com.danga.MemCached.MemCachedClient.get(MemCachedClient.java:1047)

I am trying to repeat this bug.

[joongonn]

In our case (using 2.6.2), our web server threads got stucked at:

"2076090123@qtp-1696446139-91312" prio=10 tid=0x00007f4970178800 nid=0x559c runnable [0x00007f493eae9000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:202)
at sun.nio.ch.IOUtil.read(IOUtil.java:169)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
- locked <0x00000007d44dfc48> (a java.lang.Object)
at com.schooner.MemCached.SockInputStream.readFromChannel(SockInputStream.java:124)
at com.schooner.MemCached.SockInputStream.read(SockInputStream.java:109)
at com.schooner.MemCached.SockInputStream.getLine(SockInputStream.java:173)
at com.schooner.MemCached.AscIIClient.get(AscIIClient.java:790)
at com.schooner.MemCached.AscIIClient.get(AscIIClient.java:630)
at com.schooner.MemCached.AscIIClient.get(AscIIClient.java:626)
at com.danga.MemCached.MemCachedClient.get(MemCachedClient.java:1061)

Can't figure out why, as it appears, the AsciiClient didn't get back the \r\n it is waiting for from the server.

Drawing reference from http://technfun.wordpress.com/2009/01/29/networking-in-java-non-blocking-nio-blocking-nio-and-io/ , would you guys recommend a workaround, something along the lines of :

private final void readFromChannel() throws IOException {
    sock.readBuf.clear();
    sock.getChannel().socket().setSoTimeout(60000);
    InputStream is = sock.getChannel().socket().getInputStream();
    ReadableByteChannel wrappedChannel = Channels.newChannel(is);
    wrappedChannel.read(sock.readBuf);
    sock.readBuf.flip();
}

[PiouseLeo]

Hi，

I guess that it's not the real reason.

Code "return b & 0xff;" and "return (b & 0xff;)" will return the same value.

thx

On Fri, Nov 25, 2011 at 12:31 PM, Xingen Wang <
[email protected]

wrote:

Recently, I found a potential bug in the ChannelInputStream implementation.
I modified the read implementation from
@OverRide
public int read() throws IOException {
byte b = 0;
try {
b = buf.get();
} catch (BufferUnderflowException e) {
if (readFromChannel() > 0)
b = buf.get();
else
return -1;
}
return b & 0xff;
}

to:
@OverRide
public int read() throws IOException {
byte b = 0;
try {
b = buf.get();
} catch (BufferUnderflowException e) {
if (readFromChannel() > 0)
b = buf.get();
else
return -1;
}
return (b & 0xff);
}

And actually, it is small issue but critical bug.

Could you please retry the issue in version 2.6.3, here is the link to
download:

https://github.com/downloads/gwhalin/Memcached-Java-Client/java_memcached-release_2.6.3.zip

---

Reply to this email directly or view it on GitHub:

#20 (comment)

[hjdongxuxu]

Hi,

I guess that the reason maybe: When you analyse the response you don't handle the error case like:
GET (key is a blank string)
ERROR

All the contents have been readed out, but you don't handle this error case and continue to read the response. So when call input.getLine();-->read()-->readFromChannel() no content will come and the thread will hang at:"sock.getChannel().read(sock.readBuf);".

[newroot]

I think the hang issue is caused by http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4614802.

We are using the nio in incorrect way. I will figure out a small patch, but perhaps decrease the performance a little bit.

Thanks for joongonn.

Add lines in readFromChannel

ReadableByteChannel wrappedChannel = Channels.newChannel(is);
wrappedChannel.read(sock.readBuf);