GithubHelp home page GithubHelp logo

asynchttpclient / async-http-client Goto Github PK

View Code? Open in Web Editor NEW
6.2K 355.0 1.6K 18.64 MB

Asynchronous Http and WebSocket Client library for Java

License: Other

Java 100.00%
netty async java asynchttpclient ahc http-client

async-http-client's Introduction

Async Http Client

Build Maven Central

Follow @AsyncHttpClient on Twitter.

The AsyncHttpClient (AHC) library allows Java applications to easily execute HTTP requests and asynchronously process HTTP responses. The library also supports the WebSocket Protocol.

It's built on top of Netty. It's compiled with Java 11.

Installation

Binaries are deployed on Maven Central. Add a dependency on the main AsyncHttpClient artifact:

Maven:

<dependencies>
    <dependency>
        <groupId>org.asynchttpclient</groupId>
        <artifactId>async-http-client</artifactId>
        <version>3.0.0.Beta3</version>
    </dependency>
</dependencies>

Gradle:

dependencies {
    implementation 'org.asynchttpclient:async-http-client:3.0.0.Beta3'
}

Version

AHC doesn't use SEMVER, and won't.

  • MAJOR = huge refactoring
  • MINOR = new features and minor API changes, upgrading should require 1 hour of work to adapt sources
  • FIX = no API change, just bug fixes, only those are source and binary compatible with same minor version

Check CHANGES.md for migration path between versions.

Basics

Feel free to check the Javadoc or the code for more information.

Dsl

Import the Dsl helpers to use convenient methods to bootstrap components:

import static org.asynchttpclient.Dsl.*;

Client

import static org.asynchttpclient.Dsl.*;

AsyncHttpClient asyncHttpClient=asyncHttpClient();

AsyncHttpClient instances must be closed (call the close method) once you're done with them, typically when shutting down your application. If you don't, you'll experience threads hanging and resource leaks.

AsyncHttpClient instances are intended to be global resources that share the same lifecycle as the application. Typically, AHC will usually underperform if you create a new client for each request, as it will create new threads and connection pools for each. It's possible to create shared resources (EventLoop and Timer) beforehand and pass them to multiple client instances in the config. You'll then be responsible for closing those shared resources.

Configuration

Finally, you can also configure the AsyncHttpClient instance via its AsyncHttpClientConfig object:

import static org.asynchttpclient.Dsl.*;

AsyncHttpClient c=asyncHttpClient(config().setProxyServer(proxyServer("127.0.0.1",38080)));

HTTP

Sending Requests

Basics

AHC provides 2 APIs for defining requests: bound and unbound. AsyncHttpClient and Dsl` provide methods for standard HTTP methods (POST, PUT, etc) but you can also pass a custom one.

import org.asynchttpclient.*;

// bound
Future<Response> whenResponse=asyncHttpClient.prepareGet("http://www.example.com/").execute();

// unbound
        Request request=get("http://www.example.com/").build();
        Future<Response> whenResponse=asyncHttpClient.executeRequest(request);

Setting Request Body

Use the setBody method to add a body to the request.

This body can be of type:

  • java.io.File
  • byte[]
  • List<byte[]>
  • String
  • java.nio.ByteBuffer
  • java.io.InputStream
  • Publisher<io.netty.bufferByteBuf>
  • org.asynchttpclient.request.body.generator.BodyGenerator

BodyGenerator is a generic abstraction that let you create request bodies on the fly. Have a look at FeedableBodyGenerator if you're looking for a way to pass requests chunks on the fly.

Multipart

Use the addBodyPart method to add a multipart part to the request.

This part can be of type:

  • ByteArrayPart
  • FilePart
  • InputStreamPart
  • StringPart

Dealing with Responses

Blocking on the Future

execute methods return a java.util.concurrent.Future. You can simply block the calling thread to get the response.

Future<Response> whenResponse=asyncHttpClient.prepareGet("http://www.example.com/").execute();
        Response response=whenResponse.get();

This is useful for debugging but you'll most likely hurt performance or create bugs when running such code on production. The point of using a non blocking client is to NOT BLOCK the calling thread!

Setting callbacks on the ListenableFuture

execute methods actually return a org.asynchttpclient.ListenableFuture similar to Guava's. You can configure listeners to be notified of the Future's completion.

        ListenableFuture<Response> whenResponse = ???;
        Runnable callback = () - > {
            try {
               Response response = whenResponse.get();
               System.out.println(response);
            } catch (InterruptedException | ExecutionException e) {
               e.printStackTrace();
            }
        };

        java.util.concurrent.Executor executor = ???;
        whenResponse.addListener(() - > ??? , executor);

If the executor parameter is null, callback will be executed in the IO thread. You MUST NEVER PERFORM BLOCKING operations in there, typically sending another request and block on a future.

Using custom AsyncHandlers

execute methods can take an org.asynchttpclient.AsyncHandler to be notified on the different events, such as receiving the status, the headers and body chunks. When you don't specify one, AHC will use a org.asynchttpclient.AsyncCompletionHandler;

AsyncHandler methods can let you abort processing early (return AsyncHandler.State.ABORT) and can let you return a computation result from onCompleted that will be used as the Future's result. See AsyncCompletionHandler implementation as an example.

The below sample just capture the response status and skips processing the response body chunks.

Note that returning ABORT closes the underlying connection.

import static org.asynchttpclient.Dsl.*;

import org.asynchttpclient.*;
import io.netty.handler.codec.http.HttpHeaders;

Future<Integer> whenStatusCode = asyncHttpClient.prepareGet("http://www.example.com/")
        .execute(new AsyncHandler<Integer> () {
            private Integer status;
            
            @Override
            public State onStatusReceived(HttpResponseStatus responseStatus) throws Exception {
                status = responseStatus.getStatusCode();
                return State.ABORT;
            }
            
            @Override
            public State onHeadersReceived(HttpHeaders headers) throws Exception {
              return State.ABORT;
            }
            
            @Override
            public State onBodyPartReceived(HttpResponseBodyPart bodyPart) throws Exception {
                 return State.ABORT;
            }
        
            @Override
            public Integer onCompleted() throws Exception{
                return status;
            }
        
            @Override
            public void onThrowable(Throwable t) {
                t.printStackTrace();
            }
        });

        Integer statusCode = whenStatusCode.get();

Using Continuations

ListenableFuture has a toCompletableFuture method that returns a CompletableFuture. Beware that canceling this CompletableFuture won't properly cancel the ongoing request. There's a very good chance we'll return a CompletionStage instead in the next release.

CompletableFuture<Response> whenResponse=asyncHttpClient
        .prepareGet("http://www.example.com/")
        .execute()
        .toCompletableFuture()
        .exceptionally(t->{ /* Something wrong happened... */  })
        .thenApply(response->{ /*  Do something with the Response */ return resp;});
        whenResponse.join(); // wait for completion

You may get the complete maven project for this simple demo from org.asynchttpclient.example

WebSocket

Async Http Client also supports WebSocket. You need to pass a WebSocketUpgradeHandler where you would register a WebSocketListener.

WebSocket websocket=c.prepareGet("ws://demos.kaazing.com/echo")
        .execute(new WebSocketUpgradeHandler.Builder().addWebSocketListener(
        new WebSocketListener(){

            @Override
            public void onOpen(WebSocket websocket){
                websocket.sendTextFrame("...").sendTextFrame("...");
            }

            @Override
            public void onClose(WebSocket websocket) {
                // ...
            }

            @Override
            public void onTextFrame(String payload,boolean finalFragment,int rsv){
                 System.out.println(payload);
            }

            @Override
            public void onError(Throwable t){
                t.printStackTrace();
            }
        }).build()).get();

WebDAV

AsyncHttpClient has build in support for the WebDAV protocol. The API can be used the same way normal HTTP request are made:

        Request mkcolRequest=new RequestBuilder("MKCOL").setUrl("http://host:port/folder1").build();
        Response response=c.executeRequest(mkcolRequest).get();

or

        Request propFindRequest=new RequestBuilder("PROPFIND").setUrl("http://host:port").build();
        Response response=c.executeRequest(propFindRequest,new AsyncHandler() {
            // ...
        }).get();

More

You can find more information on Jean-François Arcand's blog. Jean-François is the original author of this library. Code is sometimes not up-to-date but gives a pretty good idea of advanced features.

User Group

Keep up to date on the library development by joining the Asynchronous HTTP Client discussion group

GitHub Discussions

Contributing

Of course, Pull Requests are welcome.

Here are the few rules we'd like you to respect if you do so:

  • Only edit the code related to the suggested change, so DON'T automatically format the classes you've edited.
  • Use IntelliJ default formatting rules.
  • Regarding licensing:
    • You must be the original author of the code you suggest.
    • You must give the copyright to "the AsyncHttpClient Project"

async-http-client's People

Contributors

asapin avatar basil3whitehouse avatar bentmann avatar bfg avatar brianm avatar coderunner avatar cowtowncoder avatar cstamas avatar dependabot[bot] avatar diagoras avatar doom369 avatar drmaas avatar gailh avatar gerdriesselmann avatar hyperxpro avatar jfarcand avatar jloomis avatar mpilquist avatar nabcos avatar neotyk avatar rlubke avatar simonetripodi avatar slandelle avatar spikhalskiy avatar stepancheg avatar sullis avatar taer avatar tomgranot avatar twz123 avatar wsargent avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

async-http-client's Issues

NPE with Grizzly

java.lang.NullPointerException
at com.ning.http.client.providers.grizzly.GrizzlyAsyncHttpProvider$AsyncHttpClientEventFilter.onInitialLineParsed(GrizzlyAsyncHttpProvider.java:1148)
at org.glassfish.grizzly.http.HttpClientFilter.decodeInitialLine(HttpClientFilter.java:336)
at org.glassfish.grizzly.http.HttpCodecFilter.decodeHttpPacket(HttpCodecFilter.java:688)
at org.glassfish.grizzly.http.HttpCodecFilter.handleRead(HttpCodecFilter.java:443)
at org.glassfish.grizzly.http.HttpClientFilter.handleRead(HttpClientFilter.java:157)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:119)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:265)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:200)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:134)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:112)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:78)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:814)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:111)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:115)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:55)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:135)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2012-05-14 11:16:20 org.glassfish.grizzly.filterchain.DefaultFilterChain execute
ATTENTION: Exception during FilterChain execution

Better default user agent string

Instead of NING/1.0, AsyncHttpClientConfig and AsyncHttpClientConfigBean should use a more appropriate user agent string, such as async-http-client or something.

Add support for SPDY protocol

Netty 3.3.x has support (donated by Twitter). We should investigate how that could be supported by AHC as well via the Upgrade interface

Following redirects, AHC uses the original request method

when POSTing to and URL that replies with a redirect code, instead of redirecting with GET, AHC continues using the POST method.

Trying with Firefox & LiveHttpHeaders, looks like the browser converts methods to GET...

Not sure what the specifications report

sometimes I call AsyncHttpClient.execute but no response on the handler

I've post this problem on https://issues.sonatype.org/browse/AHC-120. Since no comments there I want to ask for help here.
I'm working on Android platform and I try to use this http client. The detail about this problem is below. I wonder if it is only exposed on Android.

I do sth like:
m_asyncHttpClient.prepareGet(http_url).setPerRequestConfig(request_config) .setHeader("Connection", "Keep-Alive")
.setHeader("Accept-Language" , ...)
.execute(...)

with a AsyncHandler given as param.

If I do the above repeatly (several times) within a connected network situation, and a few times afterwards within a connected-less (no cable is connected) situation, then there's no callback function triggered.

I trace to the code and found that NettyAsyncHttpProvider's doConnect will create or get a future with a NettyConnectListener, whose operationComplete is called and "if (f.isSuccess())" is true, then "future.provider().writeRequest" is called.
in NettyAsyncHttpProvider's writeRequest,
"if (!channel.isOpen() || !channel.isConnected())" returns true, so it returns and no handler callback is triggered.

I use a ning of version 1.6.5. is it a known issue?

Support request compression

The following snippet fails to execute with AHC 1.7.3 and 1.7.4 (using default Netty provider):

  final AsyncHttpClientConfig.Builder configBuilder = new AsyncHttpClientConfig.Builder();
  configBuilder.setCompressionEnabled( true );
  configBuilder.setRequestCompressionLevel( 6 );
  final AsyncHttpClient client = new AsyncHttpClient( configBuilder.build() );

  try
  {
    final Response response = client.prepareGet( "http://www.google.com/" ).execute().get();
    System.out.println( response.getStatusCode() + " " + response.getStatusText() );
  }
  finally
  {
    client.close();
  }

If I comment out the line configBuilder.setRequestCompressionLevel( 6 ); all is fine and snippet above works.

Also, I'd like to have some explanation what this method is meant to do. By reading sources, it is used to set "compression level" on Netty's HttpContentEncoder (wanted to raise it from default 6 to 9). How are setRequestCompressionLevel() and setCompressionEnabled() corelated?

Exception thrown by snippet above is:

java.util.concurrent.ExecutionException:
java.lang.IllegalStateException: cannot send more responses than requests
       at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:297)
       at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.abort(NettyAsyncHttpProvider.java:1320)
       at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.exceptionCaught(NettyAsyncHttpProvider.java:1545)
       at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:117)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:777)
       at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:143)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:777)
       at org.jboss.netty.channel.SimpleChannelUpstreamHandler.exceptionCaught(SimpleChannelUpstreamHandler.java:143)
       at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:117)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:777)
       at org.jboss.netty.channel.SimpleChannelHandler.exceptionCaught(SimpleChannelHandler.java:163)
       at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:137)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:777)
       at org.jboss.netty.handler.codec.replay.ReplayingDecoder.exceptionCaught(ReplayingDecoder.java:456)
       at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:117)
       at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:72)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)
       at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:426)
       at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:47)
       at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:645)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:593)
       at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:770)
       at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:262)
       at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:119)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:585)
       at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:576)
       at org.jboss.netty.channel.Channels.write(Channels.java:605)
       at org.jboss.netty.channel.Channels.write(Channels.java:572)
       at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:245)
       at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.writeRequest(NettyAsyncHttpProvider.java:431)
       at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:82)
       at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:397)
       at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:388)
       at org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:332)
       at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:770)
       at org.jboss.netty.channel.socket.nio.NioWorker.processRegisterTaskQueue(NioWorker.java:250)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:192)
       at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
       at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.IllegalStateException: cannot send more responses than requests
       at org.jboss.netty.handler.codec.http.HttpContentEncoder.writeRequested(HttpContentEncoder.java:104)
       at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:262)
       at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:119)
       at org.jboss.netty.channel.Channels.write(Channels.java:605)
       at org.jboss.netty.channel.Channels.write(Channels.java:572)
       at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:245)
       at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.writeRequest(NettyAsyncHttpProvider.java:431)
       at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:82)
       at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:397)
       at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:388)
       at org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:332)
       at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:770)
       at org.jboss.netty.channel.socket.nio.NioWorker.processRegisterTaskQueue(NioWorker.java:250)
       at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:192)
       ... 3 more

callback InterruptedException in over 1000 request test.

java.lang.InterruptedException
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1199)
    at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:312)
    at java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:294)
    at com.huoli.comm.grabber.service.HttpGrabber.dealResp(HttpGrabber.java:472) (I try use :synchronized)
    at com.huoli.comm.grabber.httpclient.AsyncClient.dealResp(AsyncClient.java:339) (I try use :synchronized)
    at com.huoli.comm.grabber.httpclient.AsyncClient$AsyncResponseHandler.onThrowable(AsyncClient.java:673)  (I try use :synchronized)

    at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:300)
    at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:104)
    at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)
    at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:372)
    at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:334)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$1.operationComplete(NioClientSocketPipelineSink.java:147)
    at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)
    at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:372)
    at org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:316)
    at org.jboss.netty.channel.AbstractChannel$ChannelCloseFuture.setClosed(AbstractChannel.java:359)
    at org.jboss.netty.channel.AbstractChannel.setClosed(AbstractChannel.java:196)
    at org.jboss.netty.channel.socket.nio.NioSocketChannel.setClosed(NioSocketChannel.java:154)
    at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:583)
    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:91)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:742)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:60)
    at org.jboss.netty.handler.codec.http.HttpClientCodec.handleDownstream(HttpClientCodec.java:82)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:568)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:747)
    at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:114)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:568)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:563)
    at org.jboss.netty.channel.Channels.close(Channels.java:720)
    at org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:208)
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.finishChannel(NettyAsyncHttpProvider.java:1075)
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.closeChannel(NettyAsyncHttpProvider.java:1060)
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.abort(NettyAsyncHttpProvider.java:1311)
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.access$700(NettyAsyncHttpProvider.java:136)
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider$ReaperFuture.run(NettyAsyncHttpProvider.java:1803)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)

GrizzlyResponse.getResponseBodyAsBytes() implemented very inefficiently

Looking 'getResponseBodyAsBytes()' of GrizzlyResponse, code does:

public byte[] getResponseBodyAsBytes() throws IOException {
    return getResponseBody().getBytes(Charsets.DEFAULT_CHARACTER_ENCODING);
}

which is hideously expensive and backwards way of obtaining bytes. While 'responseBody' (grizzly Buffer) does not have a simple method to call, it should be simple to implement this properly; esp. looking at how 'getResponseBodyAsStream()' works.

Multiple requests in parallel with AsyncHttpClient

I need to connect to many servers in parallel with AsyncHttpClient.

What I think of is something like a collection of AsyncHttpClient.BoundRequestBuilder invoking in parallel. Imagine a HttpResult class as response.

Collection<AsyncHttpClient.BoundRequestBuilder> builders =  new LinkedList(...)
CompletionService<HttpResult> service  = new ExecutorCompletionService<HttpResult>();
List<Future<HttpResult>> futures  = new ArrayList<Future<HttpResult>>(builders.size());
try {
         for (Callable<HttpResult> b : builders)
             futures.add(service.submit(b));
         for (int i = 0; i < futures.size(); i++) {
             try {
                 HttpResult r = service.take().get();
                 if (r != null) {
                      ...
                 }
             } catch (ExecutionException ignore) {}
         }
     }
     finally {
         for (Future<HttpResult> f : futures)
             f.cancel(true);
     }

Right now, calling AsyncHttpClient.BoundRequestBuilder execute() now gives a single Future object. But I need to collect many Futures in the order as they return, so a take() method like the one in ExecutorCompletionService could process them.

How can I implement something like this with AsyncHttpClient? Any help is appreciated.

Thank for any hints!

Jörg

AsyncHttpProviderUtils.contentToString corrupts strings with multibyte encoding (like utf-8)

.contentToString is used in NettyResponse.getResponseBody(charset) to convert a chunked HTTP response into a string.

It first converts each chunk into a string with the given character set and then concatenates those strings.

This fails when the border between two chunks is in the middle of a multibyte utf-8 encoded character, in which case the character is split in two parts each of which are invalid utf-8 characters and get decoded into the unicode REPLACEMENT CHARACTER.

A correct implementation would first concatenate all the chunks and then create a string:

public final static String contentToString(Collection bodyParts, String charset) throws UnsupportedEncodingException {
return new String(contentToByte(bodyParts), charset);
}

We found this when using version 1.6.4 where the .contentToString method is part of com.ning.http.client.providers.netty.NettyResponse. In 1.7.0 this method has been moved to com.ning.http.util.AsyncHttpProviderUtils without change in implementation, so we are pretty sure that the bug still exists in 1.7.0.

Add connection timeout as a per request setting

Currently connection timeout is only settable on the pool through AsyncHttpClientConfig. Can connection timeout be added to PerRequestConfig so this value can be controlled at the request level?

NPE with Netty

12:40:00.296 [New I/O client worker #1-1] DEBUG c.n.h.c.p.n.NettyAsyncHttpProvider - Closing Channel [id: 0x01a62c31, /127.0.0.1:3865 => localhost/127.0.0.1:8080]
12:40:00.296 [New I/O client worker #1-1] DEBUG c.n.h.c.p.n.NettyAsyncHttpProvider - Channel Closed: [id: 0x01a62c31, /127.0.0.1:3865 :> localhost/127.0.0.1:8080] with attachment com.ning.http.client.providers.netty.NettyAsyncHttpProvider$DiscardEvent@1fe1feb
12:40:15.232 [New I/O client worker #1-2] DEBUG c.n.h.c.p.n.NettyAsyncHttpProvider - Unexpected I/O exception on channel [id: 0x01f6ba0f, /127.0.0.1:3870 => localhost/127.0.0.1:8080]
java.lang.NullPointerException: null
at org.jboss.netty.handler.codec.http.HttpClientCodec$Decoder.isContentAlwaysEmpty(HttpClientCodec.java:124) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.readHeaders(HttpMessageDecoder.java:450) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:191) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:108) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:96) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:465) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:438) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:72) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:343) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:274) ~[netty-3.3.1.Final.jar:na]
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:194) ~[netty-3.3.1.Final.jar:na]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [na:1.6.0_07]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.6.0_07]
at java.lang.Thread.run(Unknown Source) [na:1.6.0_07]

com.ning.http.util.SslUtils provoke NPE

[error] play - Waiting for a promise, but got an error: null
java.lang.NullPointerException: null
at java.util.Collections.addAll(Collections.java:3836) ~[na:1.7.0_04-ea]
at sun.security.ssl.AbstractTrustManagerWrapper.(SSLContextImpl.java:778)
~[na:1.7.0_04-ea]
at sun.security.ssl.SSLContextImpl.chooseTrustManager(SSLContextImpl.java:133)
~[na:1.7.0_04-ea]
at sun.security.ssl.SSLContextImpl.engineInit(SSLContextImpl.java:89)
~[na:1.7.0_04-ea]
at javax.net.ssl.SSLContext.init(SSLContext.java:283) ~[na:1.7.0_04-ea]
at com.ning.http.util.SslUtils.getLooseSSLContext(SslUtils.java:99)
~[async-http-client.jar:na]

problem is that in line 109 method getAcceptedIssuers() of
implantation of X509TrustManager return null, but in java api
(http://docs.oracle.com/javase/7/docs/api/javax/net/ssl/X509TrustManager.html)
wrote that this method must return not-null value. My suggestion is
use this:
return new java.security.cert.X509Certificate[0];
instead return null.

Should AsyncHttpClient implement AutoCloseable / Closeable?

Looking into try with resources, I've been wondering.

Should AsyncHttpClient implement AutoCloseable?

AutoCloseable interface only comes with java 1.7+, so that would break backwards compatibility for a small feature...

Should AsyncHttpClient implement Closeable?

Closeable is a subinterface of AutoCloseable and comes with java 1.5+, so compatibility should not be an issue. But this interface says:

void close() throws IOException

So we're breaking here the existing API (the current API doesn't throw any exception on close()).

  • People should subclass AsyncHttpClient and implement one of these if they feel the need to?
  • The "try with resource" statement doesn't play nicely with async?

Not able to configure my own ChannelPipeline

I want to override the HttpContentDecompressor which I would like to use it for decoding gzip file types. But since some response come in without the proper contentEncoding header, HttpContentDecompressor is ignored. Can we make it so that we can construct custom handlers to the ChannelPipeline?

plainBootstrap.setPipelineFactory(new ChannelPipelineFactory() {

        /* @Override */
        public ChannelPipeline getPipeline() throws Exception {
            ChannelPipeline pipeline = pipeline();

            pipeline.addLast(HTTP_HANDLER, new HttpClientCodec());

            if (config.getRequestCompressionLevel() > 0) {
                pipeline.addLast("deflater", new HttpContentCompressor(config.getRequestCompressionLevel()));
            }

            if (config.isCompressionEnabled()) {
                pipeline.addLast("inflater", new HttpContentDecompressor());
            }
            pipeline.addLast("chunkedWriter", new ChunkedWriteHandler());
            pipeline.addLast("httpProcessor", NettyAsyncHttpProvider.this);
            return pipeline;
        }
    });

Cannot use NettyAsyncHttpProvider

Hello.

I have the following sample code:

   AsyncHttpClient asyncHttpClient = new AsyncHttpClient();

    try {

        final Response response = asyncHttpClient.prepareHead(getFullUrl(filePath)).execute().get();

        return response.getStatusCode();

    } catch (Exception e) {

        return 0;
    }

And it gives the following log output:

DEBUG com.ning.http.client.providers.netty.NettyAsyncHttpProvider: Number of application's worker threads is 8
DEBUG com.ning.http.client.providers.netty.NettyAsyncHttpProvider:
Non cached request
DefaultHttpRequest(chunked: false)
HEAD /milton_server/webdav/b/a/hatiko.mkv HTTP/1.1
Host: localhost:8080
Connection: keep-alive
Accept: /
User-Agent: NING/1.0

using Channel
[id: 0x2f33cbcc, /127.0.0.1:52679]

DEBUG com.ning.http.client.providers.netty.NettyAsyncHttpProvider: Unexpected I/O exception on channel [id: 0x2f33cbcc, /127.0.0.1:52679 => localhost/127.0.0.1:8080]
java.nio.channels.ClosedChannelException
at org.jboss.netty.handler.stream.ChunkedWriteHandler.discard(ChunkedWriteHandler.java:168)
at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:120)
at org.jboss.netty.channel.Channels.write(Channels.java:712)
at org.jboss.netty.channel.Channels.write(Channels.java:679)
at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:245)
at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.writeRequest(NettyAsyncHttpProvider.java:432)
at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:82)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:399)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:390)
at org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:334)
at org.jboss.netty.channel.socket.nio.NioWorker$RegisterTask.run(NioWorker.java:187)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processRegisterTaskQueue(AbstractNioWorker.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:243)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

And after some time (during debugging session), this:

DEBUG com.ning.http.client.providers.netty.NettyAsyncHttpProvider: Unexpected I/O exception on channel [id: 0x2f33cbcc, /127.0.0.1:52679 :> localhost/127.0.0.1:8080]
java.lang.StackOverflowError
at sun.misc.Unsafe.compareAndSwapInt(Native Method)
at java.util.concurrent.atomic.AtomicBoolean.compareAndSet(Unknown Source)
at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:190)
at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:298)

And the last line is repeated several times.

The problem is, when I change provider to JDK everything works fine. But I cant use JDK provider because I wiil neen to upload really large files using the client and sun`s UrlConnetcion class is limited in this matter to approximetely 2GB.

Can somebody provide any help on this?

I am using Netty 3.4.3

Why is the WebSocket-Version hardcoded?

I use the AHC with new Tomcat 7.0.27 WebSocket support. I detect a version conflict.
AHC support only websocket protocol version "8" and Tomcat only support version *13" (RFC 6455)!

The websocket protocol version number is currently hardcoded.

com.ning.http.client.providers.netty.NettyAsyncHttpProvider

L-585
nettyRequest.addHeader("Sec-WebSocket-Version", "8");

it seems that netty 3.3.1 support also version 13.

deadlock in 1.7.0

Java-level deadlock has been detected

This means that some threads are blocked waiting to enter a synchronization block or
waiting to reenter a synchronization block after an Object.wait() call, where each thread
owns one monitor while trying to obtain another monitor already held by another thread.

Deadlock:


New I/O client worker #1-6 is waiting to lock com.ning.http.client.providers.netty.NettyAsyncHttpProvider$ReaperFuture@2bfea135 which is held by AsyncHttpClient-Reaper
AsyncHttpClient-Reaper is waiting to lock java.lang.Object@6109a3f which is held by New I/O client worker #1-6




Thread stacks


AsyncHttpClient-Reaper [BLOCKED; waiting to lock java.lang.Object@6109a3f]
org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:861)
org.jboss.netty.handler.ssl.SslHandler.closeOutboundAndChannel(SslHandler.java:1041)
org.jboss.netty.handler.ssl.SslHandler.handleDownstream(SslHandler.java:409)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:568)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:747)
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:60)
org.jboss.netty.handler.codec.http.HttpClientCodec.handleDownstream(HttpClientCodec.java:82)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:568)
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:747)
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:114)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:568)
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:563)
org.jboss.netty.channel.Channels.close(Channels.java:720)
org.jboss.netty.channel.AbstractChannel.close(AbstractChannel.java:208)
com.ning.http.client.providers.netty.NettyAsyncHttpProvider.finishChannel(NettyAsyncHttpProvider.java:1044)
com.ning.http.client.providers.netty.NettyAsyncHttpProvider.closeChannel(NettyAsyncHttpProvider.java:1029)
com.ning.http.client.providers.netty.NettyAsyncHttpProvider.abort(NettyAsyncHttpProvider.java:1280)
com.ning.http.client.providers.netty.NettyAsyncHttpProvider.access$700(NettyAsyncHttpProvider.java:136)
com.ning.http.client.providers.netty.NettyAsyncHttpProvider$ReaperFuture.run(NettyAsyncHttpProvider.java:1767)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)


New I/O client worker #1-6 [BLOCKED; waiting to lock com.ning.http.client.providers.netty.NettyAsyncHttpProvider$ReaperFuture@2bfea135]
com.ning.http.client.providers.netty.NettyAsyncHttpProvider$ReaperFuture.cancel(NettyAsyncHttpProvider.java)
com.ning.http.client.providers.netty.NettyResponseFuture.cancelReaper(NettyResponseFuture.java:196)
com.ning.http.client.providers.netty.NettyResponseFuture.setReaperFuture(NettyResponseFuture.java:352)
com.ning.http.client.providers.netty.NettyAsyncHttpProvider.writeRequest(NettyAsyncHttpProvider.java:495)
com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:82)
org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:381)
org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:367)
org.jboss.netty.channel.DefaultChannelFuture.setSuccess(DefaultChannelFuture.java:316)
org.jboss.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1016)
org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:889)
org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:605)
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:282)
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:216)
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:350)
org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:281)
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:201)
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)

getResponseBodyAsStream throws OOM

OOM is thrown when requesting a file size around 500 mb.
I thought the getResponseBodyAsStream supposed to be used for this exact case where we are requesting a big size data and use stream for it. It seems to me that we are actually writing out the entire data here.

Is there any other ways to get this data using a synchronous call?

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /home/y/logs/yjava_tomcat/java_pid29120.hprof ...
Heap dump file created [1674393227 bytes in 3.792 secs]
Exception in thread "ajp-bio-/127.0.0.1-8009-exec-1" java.lang.OutOfMemoryError: Java heap space
at org.jboss.netty.buffer.HeapChannelBuffer.(HeapChannelBuffer.java:42)
at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.(BigEndianHeapChannelBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:134)
at org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:69)
at org.jboss.netty.buffer.DynamicChannelBuffer.ensureWritableBytes(DynamicChannelBuffer.java:78)
at org.jboss.netty.buffer.DynamicChannelBuffer.writeBytes(DynamicChannelBuffer.java:227)
at org.jboss.netty.buffer.AbstractChannelBuffer.writeBytes(AbstractChannelBuffer.java:441)
at com.ning.http.client.providers.netty.NettyResponse.getResponseBodyAsStream(NettyResponse.java:106)

Netty concurrent.ExecutionException

FAILED: loginDuplicateUsernameXHRPollingTest
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: invalid version format: 6B
at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:297)
at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.abort(NettyAsyncHttpProvider.java:1315)
at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.exceptionCaught(NettyAsyncHttpProvider.java:1531)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:117)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:777)
at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:143)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:777)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.exceptionCaught(ReplayingDecoder.java:456)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:117)
at org.jboss.netty.handler.codec.http.HttpClientCodec.handleUpstream(HttpClientCodec.java:72)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:558)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)
at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:426)
at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:47)
at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:645)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:553)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:343)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:274)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:194)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Broken support for Digest authentication with query parameters

This is a follow up to a message I posted on the mailing list, https://groups.google.com/d/topic/asynchttpclient/kNCB8Mu7CHM/discussion. I am seeing an issue where the dialog between client and server for handling 401 digest authentication requests fail for all uris containing query parameters because, at some point in the dialog, the request parameters are duplicated invalidation the digest challenge. See my post for the debug logging output of the handshake.

I tried to trace the code through the netty provider. It looks like the request url was intact here But soon afterwards saw a request where the authentication request header contained the original uri but the actual request's query parameters were duplicated invalidating the auth header.

Note this is only an issue with uris containing query parameters. A uri without query parameters seemed to handle digest authentication fine.

Allow setting user in SpnegoEngine

Hello,
Using Gatling to load test a CAS / SPNEGO server, it appears that in SpnegoEngine, it is not possible to use other credentials than the connected user.
For a load test, it would be better to use different credentials (to avoid locking an account for example).
I think it is possible, in a JUnit we use this piece of code :

kinitArgs[0] = user;
kinitArgs[1] = password;
sun.security.krb5.internal.tools.Kinit.main(kinitArgs);
gssUserName = manager.createName(kinitArgs[0].substring(0, kinitArgs[0].indexOf("@")),
                            GSSName.NT_USER_NAME, krb5MechOid);

GSSCredential clientGssCreds = manager.createCredential(
                        gssUserName != null ? gssUserName.canonicalize(krb5MechOid) : null,
                        GSSCredential.INDEFINITE_LIFETIME, krb5MechOid, GSSCredential.INITIATE_ONLY);

clientGssCreds.add(gssUserName, GSSCredential.INDEFINITE_LIFETIME, GSSCredential.INDEFINITE_LIFETIME,
                        spnegoMechOid, GSSCredential.INITIATE_ONLY);

// create target server SPN
GSSName gssServerName = manager.createName(spn, GSSName.NT_USER_NAME);

GSSContext clientContext = manager.createContext(gssServerName.canonicalize(spnegoMechOid),
                        spnegoMechOid, clientGssCreds, GSSContext.DEFAULT_LIFETIME);

// optional enable GSS credential delegation
clientContext.requestCredDeleg(true);

spnegoToken = new byte[0];

// create a SPNEGO token for the target server
spnegoToken = clientContext.initSecContext(spnegoToken, 0, spnegoToken.length);

Do you think it would be possible to "improve" SpnegoEngine to allow setting the user ?
Thanks,
Fabien.

Cookie max-age is wrong

Salut,

AsyncHttpProviderUtils.parseCookie sets a -1 max-age for both expired regular cookies and session cookies (that don't have a max-age).

max-age should be an Integer (set to null for session cookies) instead of an int, so that one can tell expired and session cookies apart.

Cheers,

Stephane

ApacheAsyncHttpProvider should return method.getResponseBodyAsStream()

ApacheAsyncHttpProvider should return an InputStream from HttpMethodBase.getResponseBodyAsStream() instead of reading the full inputstream into an bytearray and returning it as part of asyncHandler.onBodyPartReceived. Doing this AsyncHttpProviderUtils.readFully on a large data will cause an OOM.

           if (state == AsyncHandler.STATE.CONTINUE) {
                InputStream is = method.getResponseBodyAsStream();
                if (is != null) {
                    Header h = method.getResponseHeader("Content-Encoding");
                    if (h != null) {
                        String contentEncoding = h.getValue();
                        boolean isGZipped = contentEncoding == null ? false : "gzip".equalsIgnoreCase(contentEncoding);
                        if (isGZipped) {
                            is = new GZIPInputStream(is);
                        }
                    }

                    int byteToRead = (int) method.getResponseContentLength();
                    InputStream stream = is;
                    if (byteToRead <= 0) {
                        int[] lengthWrapper = new int[1];
                        byte[] bytes = AsyncHttpProviderUtils.readFully(is, lengthWrapper);
                        stream = new ByteArrayInputStream(bytes, 0, lengthWrapper[0]);
                        byteToRead = lengthWrapper[0];
                    }

                    if (byteToRead > 0) {
                        int minBytes = Math.min(8192, byteToRead);
                        byte[] bytes = new byte[minBytes];
                        int leftBytes = minBytes < 8192 ? minBytes : byteToRead;
                        int read = 0;
                        while (leftBytes > -1) {

                            try {
                                read = stream.read(bytes);
                            } catch (IOException ex) {
                                logger.warn("Connection closed", ex);
                                read = -1;
                            }

                            if (read == -1) {
                                break;
                            }

                            future.touch();

                            byte[] b = new byte[read];
                            System.arraycopy(bytes, 0, b, 0, read);
                            leftBytes -= read;

                            asyncHandler.onBodyPartReceived(new ApacheResponseBodyPart(uri, b, ApacheAsyncHttpProvider.this, leftBytes > -1));

                        }
                    }
                }

SpnegoEngine thread safety

Hello,
Using gatling to load test a CAS with Spnego, I start to have NullPointerException at 20 concurrent users (all using my credentials).

The exception is :
java.io.IOException
at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.construct
(NettyAsyncHttpProvider.java:658)
Caused by: java.lang.NullPointerException
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:2
36)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:1
62)
at com.ning.http.client.providers.netty.spnego.SpnegoEngine.generateToke
n(SpnegoEngine.java:343)
at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.construct
(NettyAsyncHttpProvider.java:656)
... 18 more

I identified two attributes in SpnegoEngine that make it non thread safe (gatling use only one NettyAsyncHttpProvider) :
private GSSContext gssContext = null;
private byte[] token;

They can be defined as local variables in generateToken method : NPE disappears, and it solves the load problem with gatling.
Could you integrate this modification ?

As I can now load test our CAS Spnego server, this makes my issue (https://github.com/sonatype/async-http-client/issues/83) less important (I dont succeed by the way).

Thanks for gatling users,
Fabien.

authorization flow broken

For most use cases, especially when connecting to any api's, the current flow for authentication is unusable.

Netty and Grizzly adapters do not provide a way to get at the www-authenticate header or otherwise cache auth credentials. That's pretty much a requirement for any high volume traffic using auth.

If you do not have a realm set, netty will create one for you and use it. This makes no sense as its' guaranteed to fail.

In grizzly, if you do not have a realm set, grizzly will just abort. This is also bad because now you have to wait for the future to timeout. In most cases you want to immediately re-send the request with the authorization header, not wait on a timeout.

The correct long term solution IMO is as follows:

If the user doesn't set a realm, don't do anything additional for 401's.

Have a flag to cache the authorization, and an optional expiration on the cache.

A short term solution is to be able to disable 401 handling by the client. That way it's possible to still use AHC while it's auth support is being fixed.

[websocket] Support redirect

If the hanshake fail with a response like

T 127.0.0.1:8000 -> 127.0.0.1:49959 [AP]
HTTP/1.1 302 Found.
Date: Wed, 09 May 2012 18:24:59 GMT.
Location: http://localhost:8000/api/.
Content-Length: 0.

We need to redirect like HTTP is doing.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.