ning / async-http-client Goto Github PK

See http://is.gd/eLIj8 for the description

As described here.

From the NettyAsyncHttpProvider:

        case PUT:
            if (request.getByteData() != null) {
                nettyRequest.setContent(ChannelBuffers.copiedBuffer(request.getByteData()));
            } else if (request.getStringData() != null) {
                nettyRequest.setContent(ChannelBuffers.copiedBuffer(request.getStringData(), "UTF-8"));
            }
            break;

In contrast to the POST case, this does not set the content length header, causing the server to effectively loose any message body transmitted. In case it matters, the request was built using BoundRequestBuilder.setBody(byte[]).

query param support on the request builder is presently on of the setQueryParam(String, String). It should include receiving a map (multimap, really), and should probably be renamed to addQueryParam once it uses a multimap internally,

If you're issuing a GET request, you don't necessarily know or want to know what content type you're going to get back. Some servers treat content-type on GET as a request for a particular content-type, and will redirect you to the same url with the correct content-type that's available from that url.

The construct() method in NettyAsyncHttpProvider wrongly includes a default Content-Type of "text/html; charset=utf-8" for messages that don't have content, and as a result, is also incapable of correctly resolving the redirect it receives from servers that interpret Content-Type as described above. What should probably be in place instead is a default "Accept", for "/".

HEAD request fails when response is 302
Please see neotyk/async-http-client@f50e1d8 for test case.

com.ning.http.client.Headers is semantically a java.util.Map.
Will propose a fix in short.

in the NettyAsyncHttpProvider,has the code :
StringBuilder path = new StringBuilder(uri.getPath());
if (uri.getQuery() != null) {
path.append("?").append(uri.getQuery());
}
uri.getQuery() return the decoded query component of this URI
should use getRawQuery?

Hi Jean-Francois :)

I have this code:

AsyncHttpClientConfig asyncHttpClientConfig = new AsyncHttpClientConfig.Builder().
setKeepAlive(true).setConnectionTimeoutInMs(1000).setMaximumConnectionsTotal(1).build();
AsyncHttpClient asyncHttpClient = new AsyncHttpClient(asyncHttpClientConfig);

    for (int i = 0; i < cacheOperations; i++) {
        String url = new StringBuffer(cacheUrl).append('/').append(keyBase).append(i).toString();
        final int finalI = i;

// LOG.info("Request " + i);
// openConnections.incrementAndGet();
// if (openConnections.get() <= 1) {
asyncHttpClient.prepareGet(url).execute(new AsyncCompletionHandler() {

                @Override
                public Response onCompleted(Response response) throws Exception {
                    assertEquals(200, response.getStatusCode());
                    openConnections.decrementAndGet();
                    return response;
                }

                @Override
                public void onThrowable(Throwable t) {
                    openConnections.decrementAndGet();
                    LOG.error("On " + finalI + "th request" + t.getMessage());
                }
            });

// }

    }

I get:

java.io.IOException: Too many connections
at com.ning.http.client.providers.NettyAsyncHttpProvider.execute(NettyAsyncHttpProvider.java:495)
at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:291)
at com.ning.http.client.AsyncHttpClient$BoundRequestBuilder.execute(AsyncHttpClient.java:192)
at net.sf.ehcache.server.rest.resources.SpeedTest.testMemCachedBench(SpeedTest.java:213)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

But when I control myself the maximum number of open connections and limit those to 10 it works fine. So it seems to me that setMaximumConnectionsTotal() is not working.

12:40:00.296 [New I/O client worker #1-1] DEBUG c.n.h.c.p.n.NettyAsyncHttpProvider - Closing Channel [id: 0x01a62c31, /127.0.0.1:3865 => localhost/127.0.0.1:8080]
12:40:00.296 [New I/O client worker #1-1] DEBUG c.n.h.c.p.n.NettyAsyncHttpProvider - Channel Closed: [id: 0x01a62c31, /127.0.0.1:3865 :> localhost/127.0.0.1:8080] with attachment com.ning.http.client.providers.netty.NettyAsyncHttpProvider$DiscardEvent@1fe1feb
12:40:15.232 [New I/O client worker #1-2] DEBUG c.n.h.c.p.n.NettyAsyncHttpProvider - Unexpected I/O exception on channel [id: 0x01f6ba0f, /127.0.0.1:3870 => localhost/127.0.0.1:8080]
java.lang.NullPointerException: null
at org.jboss.netty.handler.codec.http.HttpClientCodec$Decoder.isContentAlwaysEmpty(HttpClientCodec.java:124) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.readHeaders(HttpMessageDecoder.java:450) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:191) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:108) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:96) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:465) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:438) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:72) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:343) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:274) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:194) ~[netty-3.3.1.Final.jar:na]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [na:1.6.0_07]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.6.0_07]
at java.lang.Thread.run(Unknown Source) [na:1.6.0_07]

RequestType is an enum meaning you can't use custom http methods, which are allowable via spec. Please support custom methods besides POST, GET, etc.

The current support for connection pool is hardcoded inside the Netty Provider. We should define an API and make that mechanism pluggable.

Target: 1.2.0

This works:

AsyncHttpClient asyncHttpClient = new AsyncHttpClient();
Future f = asyncHttpClient.prepareGet("http://www.ning.com/").execute();
Response r = f.get();
System.out.println("r = " + r.getResponseBody());

This doesn't
AsyncHttpClient asyncHttpClient = new AsyncHttpClient();
Future f = asyncHttpClient.prepareGet("http://www.ning.com").execute();
Response r = f.get();
System.out.println("r = " + r.getResponseBody());

I want to use Sourcegraph for async-http-client code search, browsing, and usage examples. Can an admin enable Sourcegraph for this repository? Just go to https://sourcegraph.com/github.com/ning/async-http-client. (It should only take 30 seconds.)

Thank you!

I noticed that if a URL doesn't have trailing slash right after the hostname part (when path = ""), an exception will be thrown. I know that the best practice is to always include it; therefore, not sure what the best behavior of a http client would be.

Examples:
http://ning.com # bad
http://ning.com/ # good
Discovered it when sending 302 redirects from Jetty. The exception is thrown at NettyAsyncHttpProvider#createUrl(NettyAsyncHttpProvider.java:325)
} else if (path.length() == 0) {
throw new IllegalArgumentException("The URI path, of the URI " + uri
+ ", must be present");

Thanks,
-alex yarmula

Linger, Buffer etc. which are available on the Socket class.

NettyAsyncHttpProvider.doConnect() always calls activeConnectionsCount.getAndIncrement() (see if-statement around line 465 ). Later on, it calls lookupInCache() to see if an existing connection exists. When this succeeds, however, activeConnectionsCount is not decremented.

In general, keeping these two pieces of state in sync seems tricky and error prone. Maybe it's worth introducing a generic object pooling component?

java.lang.NullPointerException
at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onInitialLineParsed(GrizzlyAsyncHttpProvider.java:1148)
at org.glassfish.grizzly.http.HttpClientFilter.decodeInitialLine(HttpClientFilter.java:336)
at org.glassfish.grizzly.http.HttpCodecFilter.decodeHttpPacket(HttpCodecFilter.java:688)
at org.glassfish.grizzly.http.HttpCodecFilter.handleRead(HttpCodecFilter.java:443)
at org.glassfish.grizzly.http.HttpClientFilter.handleRead(HttpClientFilter.java:157)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:814)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:111)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2012-05-14 11:16:20 org.glassfish.grizzly.filterchain.DefaultFilterChain execute
ATTENTION: Exception during FilterChain execution

curently, as it is, Response object does not have getCookies method.
The only way to get access to cookies is to retrieve the Set-Cookie header (rfc2965) and manually do the parsing.

While this shouldn't be a problem according to the spec, in the real world it's an annoying issue: you don't have to append ":80" to the host in the Host header when making an http request, and when you do, some servers in the real world will redirect you to the same host without the port appended. In the current code, this will result in an infinite redirect loop, as the port keeps being re-appended in the construct method on each redirect.

The fix is to not append the port number when it matches the default port for the requested service.

Currently the Channel is being cached in a map based on the full Url, which includes Path and QueryString.
In reality, the connection is defined only by domain:port between the two endpoints, the rest is a logical resource identifier that is being parsed by the receiving webserver to know which resource to return. It does not play part in the over-the-wire connection handling.
It would be more accurate to define the connectionPool as ConcurrentHash<String from Url.getBaseUrl(), Channel>

The README states that development has moved to https://github.com/sonatype/async-http-client ... but that URL returns a 404.

It would be cleaner if the core client dealt only with raw http. Value-add features such as cookie management, caching, form-encoding, multipart, etc should be left to higher-level layers.

One of the issues with the current design, for instance, is that it's not clear what happens if you add a bunch of request (body) parameters and then set a content type other than application/x-www-form-urlencoded. This could easily be solved via builders designed for each particular scenario that assemble proper low-level requests to be handed to the barebones http client.

Multiple headers are trimmed to one header as seen on test case: neotyk/async-http-client@d78460c

This is fixed by: neotyk/async-http-client@020c66c

Very nice and robust HTTP client.
The problem:
The following URL will be truncated after the "?".

Future f = c.preparePost("http://localhost:8085/configuration?action=update").setBody(bytes).execute();

Combining GET parameters with POST query is very common and very important. Every HTTP client allows it. It will be noce if you can fix it.

When calling

client.prepareGet("http://localhost:123/spa%20ce").execute(),

I get
"java.net.URISyntaxException: Illegal character in path at index 24: http://localhost:123/spa ce"

As far as I can tell, the problem is in com.ning.http.client.RequestBuilderBase#getUrl, where Url.valueOf(this.url).toString() gets called and the URL encoding gets lost.

Thanks,
-alex yarmula

Just API polish, right now there is

• setConnectionTimeoutInMs
• setIdleConnectionTimeout
• setRequestTimeout

Note the different occurrence of the suffic "InMs" in the method name.

Test case is here: http://github.com/alaz/async-http-client/commit/372776cfa49c0365dd1d5483b9a65f6ed56c0bc0

The exception looks like

java.lang.StringIndexOutOfBoundsException: String index out of range: 56
at java.lang.String.substring(String.java:1934)
at com.ning.http.client.providers.NettyAsyncHttpProvider.getBaseUrl(NettyAsyncHttpProvider.java:668)
at com.ning.http.client.providers.NettyAsyncHttpProvider.lookupInCache(NettyAsyncHttpProvider.java:160)
at com.ning.http.client.providers.NettyAsyncHttpProvider.doConnect(NettyAsyncHttpProvider.java:474)
at com.ning.http.client.providers.NettyAsyncHttpProvider.execute(NettyAsyncHttpProvider.java:451)
at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:401)
at com.ning.http.client.AsyncHttpClient$BoundRequestBuilder.execute(AsyncHttpClient.java:202)
at com.ning.http.client.async.ComplexClientTest.urlWithColonTest(ComplexClientTest.java:143)

With Grizzly provider, I could able to connect websocket through proxy.
But when I try to set SSLContext to the client, it starting throwing exception:

javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at org.glassfish.grizzly.ssl.SSLUtils.getSSLPacketSize(SSLUtils.java:172)
at org.glassfish.grizzly.ssl.SSLUtils.handshakeUnwrap(SSLUtils.java:203)
at org.glassfish.grizzly.ssl.SSLFilter.doHandshakeStep(SSLFilter.java:327)
at org.glassfish.grizzly.ssl.SSLFilter.handleRead(SSLFilter.java:177)
at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$SwitchingSSLFilter.handleRead(GrizzlyAsyncHttpProvider.java:2532)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:770)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

The Squid Proxy access log is showing the following request log line:
1356520015.018 0 192.168.125.1 NONE/400 3501 �e - NONE/- text/html
Note the binary data in the request line.

Notes:

1. Websocket connection through Proxy is working fine.
2. Secured Websocket connection with the same SSLContext with out any proxy is working fine.
3. Secured Websocket connection with the same SSLContext through proxy is not working.

Posted this issue in the forum and was asked to create issue for the same:
https://groups.google.com/d/msg/asynchttpclient/-/FvoakS5P3OwJ.

It would be cool if the library provided something similar to java.net.Authenticator such that the caller could provide user credentials for a request to a server and/or proxy.

I was trying to send a PUT request with InputStream body, and it resulted in infinite blocking with request not being sent.
Looking into the code (NettyAsyncHttpProvider#construct), I noticed that for PUT request type, only byteData and stringData are handled. In contrast, for POST requests, more types are supported, including streamData (InputStream).
I'm talking about the code that starts with:
switch (request.getType()) {
case POST:
In my code, I was able to work around the issue:
requestBuilder.setBody(ByteStreams.toByteArray(someInputStream));
but why not support InputStreams for PUT?

Thanks,
-alex yarmula

It would be nice to have addFormParameter() call (or similar) in RequestBuilder. It should handle encoding (similar to how other add methods work), and can override / be overridden if there are conflicting setBody() calls.
While it is possible to construct these manually, it would be convenient to add basic support

As described in http://groups.google.com/group/asynchttpclient/browse_thread/thread/e9a7b44fc08766e2

Instead of allowing only to configure proxy when starting AHC enable per request proxy configuration as well.
For now there is only global proxy configuration.
This would enable per request configuration.

I observed HEAD requests timing out and investigation suggests this is caused when a Content-Length header with positive value is present in the server response. E.g. this response works

HTTP/1.1 200 OK
Content-Length: 0
Server: Jetty(6.1.11)

while

HTTP/1.1 200 OK
Content-Length: 1
Server: Jetty(6.1.11)

makes the client timeout (probably waiting for the non-existing response body).

Expose protocol details like:

• protocol name
• protocol major version
• protocol minor version

That come from status line.

OOM is thrown when requesting a file size around 500 mb.
I thought the getResponseBodyAsStream supposed to be used for this exact case where we are requesting a big size data and use stream for it. It seems to me that we are actually writing out the entire data here.

Is there any other ways to get this data using a synchronous call?

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /home/y/logs/yjava_tomcat/java_pid29120.hprof ...
Heap dump file created [1674393227 bytes in 3.792 secs]
Exception in thread "ajp-bio-/127.0.0.1-8009-exec-1" java.lang.OutOfMemoryError: Java heap space
at org.jboss.netty.buffer.HeapChannelBuffer.(HeapChannelBuffer.java:42)
at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.(BigEndianHeapChannelBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:69)
at org.jboss.netty.buffer.DynamicChannelBuffer.ensureWritableBytes(DynamicChannelBuffer.java:78)
at org.jboss.netty.buffer.DynamicChannelBuffer.writeBytes(DynamicChannelBuffer.java:227)
at org.jboss.netty.buffer.AbstractChannelBuffer.writeBytes(AbstractChannelBuffer.java:441)
at com.ning.http.client.providers.netty.NettyResponse.getResponseBodyAsStream(NettyResponse.java:106)

From AsyncHttpClientConfig:

private final long connectionTimeOutInMs;
private final long idleConnectionTimeoutInMs;
private final int requestTimeoutInMs;

i.e. some timeouts are given as int while others are given as long. int is probably big enough.

Looking at the code, I first thought that:
if(isNotNull(response.getResponseBody())) {
doSomethingWith(response.getResponseBodyAsStream)
}
would work, as it's just a different representation of the same response body. However, #getResponseBodyAsStream throws a NPE, because ResponseBodyPart#chunk = null.
The string-based #getResponseBody returns a valid content, the problem's in #getResponseBodyAsStream.
Please let me know if it's my misunderstanding or a bug.

Thank you,
-alex yarmula

does asynch client support socks protocol?

When a server sends 302 redirect, the following code takes place in com.ning.http.client.providers.NettyAsyncHttpProvider#messageReceived:

HttpRequest r = construct(config,request, map(request.getType()),      createUrl(response.getHeader(HttpHeaders.Names.LOCATION)));
ctx.getChannel().write(r); 

If I have request A sent to serverA port 1000, and then redirected to serverA port 1001, the request "r" will still go on port 1000 (with correct path, but to the incorrect port).
I think the reason is because the channel is already bound to another socket and forwards everything to that socket. If I replace the above with
new AsyncHttpClient().prepareGet(response.getHeader(HttpHeaders.Names.LOCATION)).execute();
the correct port gets hit.

Thanks,
-Alex Yarmula

looking through RequestBuilderBase.java I see a lot of (T)this where T is a generic. Java doesn't have reified generics so they cannot be used to cast like this. I am surprised the code even compiles.

a post on reified generics: http://gafter.blogspot.com/2006/11/reified-generics-for-java.html

It appears as if a "blocking" .get() method marks the future as done if its not and always checks for the exEx reference to see if it has an exception... Naturally, one would expect the behavior of .get(long, TimeUnit) to be exactly identical, except for the different timeout period, which is, however, not the case. internally the future is never marked as "done" and the exEx checking is done only if the future is not done and not canceled. It seems as if the .get(long, TimeUnit) should behave identically to the .get() with the exception of the latch.await timeout value and time unit.

as it specifically sets a www-url-form-encoded content body parameter, should probably be renamed to something like addContentParam or addBodyParam or something to indicate. The confusion comes from the srvlet specs horrid overlading of parameter to be both query param (in url) and form param in content body.

While I know that the HTTP spec mandates an absolute URL for the Location header of redirects, both FF, IE and curl also support relative URLs like "/moved-to.html". Given that users will likely report any differences in behavior compared to a popular HTTP client as a bug, it appears the async-http-client should support this as well. Better safe than sorry.

com.ning.http.util.DateUtil$DateParseException: Unable to parse the date Mon, 04 Aug 2014 01:08:21GMT
at com.ning.http.util.DateUtil.parseDate(DateUtil.java:176)
at com.ning.http.util.DateUtil.parseDate(DateUtil.java:104)

Although it works with 'Mon, 04 Aug 2014 01:08:21 GMT' (blank char before GMT), some servers do return this format.

Hi,
I'm a big fan of netty and as so I'm embracing every library that uses it. I like the ning "async-http-client"
I have tried to use NING http client (ning) in our "long run" testing environment and it failed with several issues:

1. After 2000 messages the ning stop working! It claims that I'm having to many connections when I'm having only one ("keep alive" one).

Reason in ning: After each message the function "markAsDoneAndCacheConnection" is called. This function increment the parameter "connectionPerHost.getAndIncrement()". This parameter count the number of messages and not the number of connections

See

http://github.com/martint/async-http-client/tree/master

If a user issues a GET for a url such as "http://foo.com" (no trailing '/'), async-http-client will silently create and execute the request with the request line "GET HTTP/1.1", when it should either complain that the user did not specify the path component of the url, or silently use "/".

it is legal to have many form params (in content body) or query params (in url) of the same name. The current impl uses a LinkedHashMap for these, so is broken,