oliver-loeffler / niohttp

Building HTTP requests and responses using Java NIO/NIO2

NIOHttp

Reading and writing HTTP request and response objects from and to java.nio.channels .

⚠️ This project has exploratory character and is in a very early stage of development.

Goals

• Create a library/toolkit to write and read HTTP requests to and from java.nio.channels (Java's non-blocking IO)
• Explore java.util.concurrentjava.util.concurrent API
• Do not use external frameworks, integrate HTTP protocol and java.nio.channels using JDK 1.8+.

Boundary Conditions

• zero external dependencies beside standard JDK except for unit & integration testing and build automation

Idea

First one need to connect a SocketStream to a URL such as http://www.raumzeitfalle.de/.

 
 URL url = new URL("http://www.raumzeitfalle.de/");
 InetSocketAddress address = new InetSocketAddress(url.getHost(), url.getDefaultPort());
 SocketChannel socketChannel = SocketChannel.open(address);

The server won't respond unless a request is sent (for example a GET request).

  writeGetRequestTo(socketChannel);
  

Even with bad requests server will respond, however, this may take time and response time is different. Here the static HttpResponseReader.fromChannel(...) method provides a FutureTask< Void > which can be executed by an ExecutorService. To collect the result, a Consumer< HttpResponse > must be provided. (https://github.com/Oliver-Loeffler/NIOHttp/blob/master/src/main/java/net/raumzeitfalle/niohttp/playground/FutureDemo.java)

 FutureTask<Void> futureTask = HttpResponseReader
 				.fromChannel(socketChannel,r -> System.out.println(r.responseHeader());
 		
 ExecutorService executor = Executors.newFixedThreadPool(1);
 executor.submit(futureTask);

Furthermore HttpResponseReader.fromChannel(...) returns a Stream of HttpResponse objects (https://github.com/Oliver-Loeffler/NIOHttp/blob/master/src/main/java/net/raumzeitfalle/niohttp/playground/StreamDemo.java)

Stream<HttpResponse> responseStream = HttpResponseReader.fromChannel(socketChannel); 
responseStream.findFirst().ifPresent(consumer);

This would enable (given appropriate HttpResponse capabilities), interesting ways of working with HttpResponses.

HttpResponseReader.fromChannel(socketChannel)
	    .filter( r -> r.isNotBadRequest() )
	    .findFirst().ifPresent(consumer);
	    

The current implementation is not yet fully functional here, as the HttpResponseReader only reads exactly one message from a channel. The response reading process has to be extended for continuous reading so that HttpResponse streaming will work.

Resources

• HTTP Working Group: http://httpwg.org

The HTTP/1.1 Protocol

• Introduction for RFC7230: http://httpwg.org/specs/rfc7230.html#introduction
• Uniform Resource Identifier (URI) in HTTP: http://httpwg.org/specs/rfc7230.html#uri
• Message format: http://httpwg.org/specs/rfc7230.html#http.message
• Semantics and Content: http://httpwg.org/specs/rfc7231.html#top
• Request Methods http://httpwg.org/specs/rfc7231.html#methods
• Request Header fields: http://httpwg.org/specs/rfc7231.html#request.header.fields
• Response Status Codes: http://httpwg.org/specs/rfc7231.html#status.codes
• Response Header Fields: http://httpwg.org/specs/rfc7231.html#response.header.fields
• IANA Status code assignments: http://www.iana.org/assignments/http-status-codes/http-status-codes.xhtml

HTTP/2 Protocol

• Starting a HTTP/2 connection: http://httpwg.org/specs/rfc7540.html#starting
• The protocol: http://httpwg.org/specs/rfc7540.html
• HTTP/2 Message Exchanging: http://httpwg.org/specs/rfc7540.html#HTTPLayer

HTTP over TLS (transport layer security)

• https://tools.ietf.org/html/rfc2818#section-2

Java non-blocking IO (NIO and NIO2) and Networking

• Java API

• https://docs.oracle.com/javase/tutorial/

• Java NIO channels: http://docs.oracle.com/javase/8/docs/api/java/nio/channels/package-summary.html

• Guides & Tutorials, Inspiration

• NIO2: http://www.ibm.com/developerworks/java/library/j-nio2-1/index.html

• NIO2 Async FileChannel: http://www.baeldung.com/java-nio2-async-file-channel

• Jenkov: http://tutorials.jenkov.com/java-nio/index.html

• NIO Selectors: http://www.baeldung.com/java-nio-selector

• Apache HttpClient: http://www.baeldung.com/httpasyncclient-tutorial

• Javaworld on NIO/NIO2: http://www.javaworld.com/article/2078654/core-java/java-se-five-ways-to-maximize-java-nio-and-nio-2.html

• Java2s on non-blocking async IO: http://www.java2s.com/Tutorials/Java/Java_Network/0080__Java_Network_Asynchronous_Socket_Channels.htm

Other helpful references

• As RFC7230 uses the term ABNF, here the explanation: https://en.wikipedia.org/wiki/Augmented_Backus–Naur_form

niohttp's Issues

End of Response

I haven't tested it until know but it looks to me that currently the end of a response in the stream is found by searching for 2 line feeds. I think this is wrong since the payload of a response can contain several line feeds in the data. As far as I know each request / response has a content length field that defines the size of the content / payload (in bytes / characters?). I think this must be checked to find the end of a request / reponse, right?

Response Status Line parsing is error prone

With current implementation in HttpResponseReader in case of invalid messages there would be something like an IllegalArgumentError etc. Here it would be better to possibly return an Optional with a HttpResponse embedded when the response is valid. If response line is invalid, returning an empty Optional would be okay.

Consumers the can be called by:

HttpResponse.fromBytes(byte[] bytes).ifPresent(Consumer<HttpResponse> consumer)

Response status line is defined in http://httpwg.org/specs/rfc7230.html#status.line.

A valid response status line:

• has no preceding empty lines
• has no leading spaces
• follows: status-line = HTTP-version SP status-code SP reason-phrase CRLF
• with status being a 3 digit integer
• with reason-phrase being a string followed by a line terminator (carriage return, line feed)

Placing this into separate class / method would make it testable. Here similar approach would be possible for Request Line.

Header Field Parsing Exception

Messages w/o message body (payload) cause an Exception as the HttpResponse.fromBytes() method tries to access message lines which do not exist.

int lineIndex = 1;
	String line = responseLines[lineIndex];
	while (!line.isEmpty()) {
	    HeaderField messageField = HeaderFieldFactory.getInstance().fromString(line);
	    int separation = line.indexOf(SPACE) + 1;
	    responseBuilder.addMessageField(messageField, line.substring(separation, line.length()));
	    line = responseLines[lineIndex++];
	}

The last call of java line = responseLines[lineIndex++]; will cause an exception when lineIndex exceeds length or java responseLines.length .

Response Message Body Parsing

See: http://httpwg.org/specs/rfc7230.html#message.body

Currently message body is considered to be a String and a StingBuilder is used to process all the message body lines if there are some.
As the message body is not defined as a simple String, instead it's a byte sequence with different possible encodings and formats, a separate class might be needed to consider all eventualities in message body parsing.

Minimal criteria for successful message body (payload) parsing:

• define message body start criterion (http://httpwg.org/specs/rfc7230.html#http.message)
• according to RFC7230 section 3, message body is separated by an empty line (CRLF + CRLF) from message header
• message body presence is indicated by presence of a "Content-Length" or "Transfer-Encoding" header field (http://httpwg.org/specs/rfc7230.html#message.body)
• Not all HTTP messages contain a message body, certain messages types such as (1xx, 204, 304) must not contain a body, whereas all others have a message body optionally of 0 bytes length

Define a class / class structure to process or parse message body depending on header fields.

Header Field Parsing

Implement a generic header field type and corresponding parser following the guide lines as defined in http://httpwg.org/specs/rfc7230.html#field.parsing .

• field name and value are separated with a colon
• no whitespace is allowed between the header field-name and colon, clients should not face this;
• consider removal of bad whitespace
• Remove preceding and trailing optional whitespace from field values
• consider field values following the "obs-fold" (obsolete fold) rule
• replace "obs-fold" by space (SP)
• consider other encodings than standard US-ASCII, treat other octets in field content (obs‑text) as opaque data