GithubHelp home page GithubHelp logo

renatoathaydes / rawhttp Goto Github PK

View Code? Open in Web Editor NEW
189.0 9.0 28.0 3.87 MB

HTTP library to make it easy to deal with raw HTTP.

License: Apache License 2.0

Java 63.05% Kotlin 36.13% Shell 0.22% Groovy 0.15% Dockerfile 0.05% JavaScript 0.41%
http http-client java-library networking http-server

rawhttp's Introduction

RawHTTP

Module Name Latest Version Documentation Javadocs
rawhttp-core rawhttp-core RawHTTP Core javadoc
rawhttp-cli rawhttp-cli RawHTTP CLI javadoc
rawhttp-duplex rawhttp-duplex RawHTTP Duplex javadoc
rawhttp-cookies rawhttp-cookies RawHTTP Cookies javadoc
rawhttp-req-in-edit rawhttp-req-in-edit RawHTTP ReqInEdit (HTTP Tests) javadoc

Actions Status

A Java library to make it easy to deal with raw HTTP 1.1, as defined by RFC-7230, and most of HTTP 1.0 (RFC-1945).

For details about using RawHTTP and the motivation for this project, see the blog post I wrote about it!

For testing HTTP servers, check out the blog post I wrote about rawhttp-req-in-edit, which lets you write HTTP files to send requests and assert responses using JS scripts.

🌎 For more documentation, visit the website.

Introduction

HTTP is really simple in 99.9% of cases.

For example, the raw HTTP request you would make to fetch a resource from a web server looks like this:

The example below is taken from the HTTP 1.1 RFC 7230.

GET /hello.txt HTTP/1.1
User-Agent: curl/7.16.3 libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
Host: www.example.com
Accept-Language: en, mi

To send that request out to a HTTP server using RawHTTP, you can parse the Request and stream it out via a Socket.

Here's the whole code to do that:

RawHttp rawHttp = new RawHttp();

RawHttpRequest request = rawHttp.parseRequest(
    "GET /hello.txt HTTP/1.1\r\n" +
    "User-Agent: curl/7.16.3 libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3\r\n" +
    "Host: www.example.com\r\n" +
    "Accept-Language: en, mi");
Socket socket = new Socket("www.example.com", 80);
request.writeTo(socket.getOutputStream());

To read the response, it's just as easy:

RawHttpResponse<?> response = rawHttp.parseResponse(socket.getInputStream());

// call "eagerly()" in order to download the body
System.out.println(response.eagerly());

Which prints the complete response:

HTTP/1.1 404 Not Found
Accept-Ranges: bytes
Cache-Control: max-age=604800
Content-Type: text/html
Date: Mon, 04 Dec 2017 21:19:04 GMT
Expires: Mon, 11 Dec 2017 21:19:04 GMT
Last-Modified: Sat, 02 Dec 2017 02:10:22 GMT
Server: ECS (lga/1389)
Vary: Accept-Encoding
X-Cache: 404-HIT
Content-Length: 1270


<!doctype html>
...

A RawHttpResponse, just like a RawHttpRequest can be written to a File's, ServerSocket's or any other OutpuStream:

try (OutputStream out = Files.newOutputStream(responseFile.toPath())) {
    response.writeTo(out);
}

That simple!

Notice that just with the above, you have everything you need to send and receive HTTP messages.

To illustrate that, here is a simple implementation of a HTTP server that waits for a single request, then responds with a valid response:

RawHttp http = new RawHttp();
ServerSocket server = new ServerSocket(8083);

new Thread(() -> {
    try {
        Socket client = server.accept();
        RawHttpRequest request = http.parseRequest(client.getInputStream());

        if (request.getUri().getPath().equals("/saysomething")) {
            http.parseResponse("HTTP/1.1 200 OK\n" +
                    "Content-Type: text/plain\n" +
                    "Content-Length: 9\n" +
                    "\n" +
                    "something").writeTo(client.getOutputStream());
        } else {
            http.parseResponse("HTTP/1.1 404 Not Found\n" +
                    "Content-Type: text/plain\n" +
                    "Content-Length: 0\n" +
                    "\n").writeTo(client.getOutputStream());
        }
    } catch (IOException e) {
        e.printStackTrace();
    }
}).start();

HTTP client

Even though it's quite simple to implement your own HTTP client by using the RawHttp class to parse requests and responses (which can then be transmitted via Sockets), RawHTTP offers a simple HTTP client definition (and implementation) that makes it a little bit more convenient to consume HTTP APIs.

Here's the RawHttpClient interface:

public interface RawHttpClient<Response> {
    RawHttpResponse<Response> send(RawHttpRequest request) throws IOException;
}

The Response type parameter allows implementations to expose their own type for HTTP Responses, if needed.

In the core module, a simple implementation is provided: TcpRawHttpClient.

Example usage:

RawHttpClient<?> client = new TcpRawHttpClient();
EagerHttpResponse<?> response = client.send(request).eagerly();

Unless you want to take care of streaming the response body later, always call eagerly() as shown above to consume the full response body (allowing the connection to be re-used).

Other implementations are available in separate modules:

  • RawHttpComponentsClient - based on HttpComponents's HttpClient.

Requires the rawhttp:rawhttp-httpcomponents module.

You can use this if you need support for external specifications, such as cookies (RFC-6265), or basic-auth, for example.

Example usage:

// use a default instance of CloseableHttpClient
RawHttpClient<?> client = new RawHttpComponentsClient();

// or create and configure your own client, then pass it into the constructor
CloseableHttpClient httpClient = HttpClients.createDefault();
RawHttpClient<?> client = new RawHttpComponentsClient(httpClient);

HTTP server

RawHTTP also contains a package defining a few types that describe a simple HTTP server.

The main type is the interface RawHttpServer, which uses a Router to route HTTP requests, returning a HTTP response. Router is a functional interface (i.e. it can be implemented with a Java lambda), so implementing a full server is very simple.

A default implementation of TcpRawHttpServer is provided... it spans a Thread (but re-uses it when possible) for each connected client.

Here's an example, written in Kotlin, of how to use RawHttpServer:

val server = TcpRawHttpServer(8093)

server.start { req ->
    when (req.uri.path) {
        "/hello", "/" ->
            when (req.method) {
                "GET" ->
                    http.parseResponse("HTTP/1.1 200 OK\n" +
                            "Content-Type: text/plain"
                    ).withBody(StringBody("Hello RawHTTP!"))
                else ->
                    http.parseResponse("HTTP/1.1 405 Method Not Allowed\n" +
                            "Content-Type: text/plain"
                    ).withBody(StringBody("Sorry, can't handle this method"))
            }
        else ->
            http.parseResponse("HTTP/1.1 404 Not Found\n" +
                    "Content-Type: text/plain"
            ).withBody(StringBody("Content was not found"))
    }
}

Samples

Several samples showing how to use RawHTTP, including all examples in this page, can be found in the samples project.

Note: to run the samples, execute the tests with the -Prun-samples argument.

The rawhttp-duplex module has its own sample, a chat application.

rawhttp's People

Contributors

brian-mcnamara avatar f2xy avatar idanga avatar jmini avatar ophirorly avatar parry84 avatar renatoathaydes avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rawhttp's Issues

RequestLine#parseRequestLine(String statusLine) escapes already-escaped path

e.g.

GET /hello?field=encoded%20value HTTP/1.0

Creates a URI with a query value of "field=encoded%2520value". The "%" in the "%20" itself gets escaped to "%25".

Example test, with failure results: lukewpatterson@295f172

In my debugging it makes it all the way here before the URI class double-escapes it. I hacked some stuff up in another project with a URLDecoder to compensate but it isn't clean enough for a Pull Request.



P.S. Great library! It's pairing up nicely with some GoReplay-based testing I'm doing. Keep up the good work.

parseResponse results in "Illegal character in HTTP header name" exception

I've issued the following CURL statement:

curl -skLIXGET --http1.1 -m 30 -b curlcookies --url http://www.kijkwijzer.nl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.88 Safari/537.36"

I feed the returning output (which I have attached to this issue) as a string to the parseResponse (String) method of the RawHttp library. The parsing however ends up in an InvalidHttpResponse exception with the message "Illegal character in HTTP header name".

Is there something this library is not dealing with correctly or has kijkwijzer.nl simply misformed their response headers?

response-headers.txt

HTTP redirects not being followed correctly

The new RedirectingRawHttpClient does not properly follow redirects.

  • it doesn't change the host when needed.
  • it doesn't change the HTTP method to GET on 303.

The first problem is actually a bug in the also new method,

rawhttp.core.RawHttpRequest#withRequestLine

Which is in the core module.

Will make a bugfix release with the fixes.

Server response body doesn't match Content-Length, RawHTTP crashes

Currently when a response returns less data than specified in Content-Length, RawHTTP will crash

if (actuallyRead < 0) {
throw new IOException("InputStream provided " + offset + ", but " + bodyLength + " were expected");

In my case I'm not the owner of the server, I can't fix this flaw. So even though the server is technically incorrect, I still want to be able to handle it.

A possible solution would be to add a flag to indicate that I want to either consume Content-Length amount or less. If it is less than Content-Length then I want to pull as much as possible without crashing.

SSL error

ssl does not send data

java.net.SocketException: Connection reset

at rawhttp.core.HttpMetadataParser.parseStartLine(HttpMetadataParser.java:285)
at rawhttp.core.HttpMetadataParser.parseStatusLine(HttpMetadataParser.java:192)
at rawhttp.core.RawHttp.parseResponse(RawHttp.java:209)
at rawhttp.core.RawHttp.parseResponse(RawHttp.java:192)

Support application/x-www-form-urlencoded

Hello!

parseRequestLine func cannot parse parameters in body with Content-Type: application/x-www-form-urlencoded
Example:
POST https://httpbin.org/post
Content-Type: application/x-www-form-urlencoded
X-Content-Type-Options: nosniff

id=999&value=content

Do u have plans to support it?

Make the checking of host value as configurable

I am trying to convert a string to a http request using this library which has no host.
String - >

GET /storage/ua/example.psd/:applicationMetadata
content-type: application/json

The current code mandates the presence of host value which I dont have nor do I want.
Could this be made optional using some configuration ?

Fails to parse header when running ReqInEdit file

The following request:

POST {{baseUrl}}/dev/oauth/token
# Client Credentials flow POSTs a HTML form and receives JSON back
Content-Type: application/x-www-form-urlencoded

client_id=client&client_secret=Secret&grant_type=client_credentials

Is being sent without the header!

▶ rawhttp run client_credentials.http -e local -l
POST /dev/oauth/token HTTP/1.1
Host: localhost
Content-Length: 75

client_id=client&client_secret=Secret&grant_type=client_credentials
HTTP/1.1 400 Bad Request

Which breaks the test.

Tested on RawHTTP-CLI 1.5.1

Query parameter containing encoded &

The query parameter foo & bar is encoded as foo%20%26%20bar

Request:

GET /test?custom=foo%20%26%20bar

Try following java code:

String requestText = "GET /test?custom=foo%20%26%20bar HTTP/1.1\n" +
    "Host: localhost:8090\n" +
    "\n";
RawHttp http = new RawHttp();
RawHttpRequest httpRequest = http.parseRequest(requestText);
try (Socket socket = new Socket("localhost", 8090)) {
    httpRequest.writeTo(socket.getOutputStream());
    RawHttpResponse<?> httpResponse = http.parseResponse(socket.getInputStream());
    String responseText = httpResponse.eagerly()
        .toString();
    socket.close();

    System.out.println(responseText);
}

Expected:

Server-side, 1 query parameter:

  • name custom
  • value (decoded) foo & bar

Current:

2 parameters:

  • first:
    • name custom
    • value foo
  • second:
    • name bar
    • no value

I think that this URI rewrite (line 88) might be the root cause:

URI pathURI;
String path = uri.getPath();
if (path == null || path.trim().isEmpty()) {
path = "/";
}
try {
// only path and query are sent to the server
pathURI = new URI(null, null, null, -1, path, uri.getQuery(), null);
} catch (URISyntaxException e) {
throw new RuntimeException(e);
}
return method + " " + pathURI + " " + httpVersion;

Stacktrace

Thread [main] (Suspended (breakpoint at line 93 in RequestLine))	
	RequestLine.toString() line: 93	
	RequestLine.writeTo(OutputStream) line: 70	
	RawHttpRequest(HttpMessage).writeTo(OutputStream, int) line: 152	
	RawHttpRequest(HttpMessage).writeTo(OutputStream) line: 141	
	RawHttpMain.main(String[]) line: 28	

The pathURI on line 93 is wrong:

Is: "/test?custom=foo%20&%20bar"
Expected: "/test?custom=foo%20%26%20bar"


Feedback is welcomed.

I am happy to contribute a PR for this issue, but I would like to know the direction I should take.

Fix uri-encoded query string parameters handling (legal and illegal)

There are three distinct but related issues here:

  1. The following raw request will fail to parse correctly into an HTTP request (allowIllegalCharacters set to off):
GET /hello?encoded=%2F%2Fencoded%3Fa%3Db%26c%3Dd&json=%7B%22a%22%3A%20null%7D HTTP/1.1\r\n
Host: www.example.com\r\n\r\n

This is because the "withHost" method of RequestLine incorrectly handles the encoded '&' (%26),
and it will break the query string into %2F%2Fencoded%3Fa%3Db -- & -- %3Dd&json=%7B%22a%22%3A%20null%7D

(This will also affect parsing with allowIllegalCharacters on)

  1. The following raw request will fail as well, :
GET /hello/{illegal}?url=%2F%2Fencoded%3Fa%3Db%26c%3Dd HTTP/1.1\r\n
Host: www.example.com\r\n\r\n

This time with a subtle double-encoding of the already encoded query string.
I guess this is a product decision - to not re-encode legal uri encoded part.

And additional wrinkle - even if there is not double-encoding, there is still the encoded '&' issue.

Reading from GZipUncompressorOutputStream can hang if the gzipped data is corrupted.

If the gzip header is corrupted, anyone trying to read from it will get stuck on the PipedInputStream.awaitSpace() function
This is a sample stack trace for such a case:

First we get an exception on the corrupted header:
java.util.zip.ZipException: Not in GZIP format at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165) at java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) at java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) at rawhttp.core.body.encoding.GZipUncompressorOutputStream.lambda$startReader$0(GZipUncompressorOutputStream.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Then later, when trying to read the stream (via the LazyBodyReader) we get:
java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.io.PipedInputStream.awaitSpace(PipedInputStream.java:273) at java.io.PipedInputStream.receive(PipedInputStream.java:231) - locked <0x00000006d2ba5eb0> (a java.io.PipedInputStream) at java.io.PipedOutputStream.write(PipedOutputStream.java:149) at rawhttp.core.body.encoding.GZipUncompressorOutputStream.write(GZipUncompressorOutputStream.java:47) at java.io.FilterOutputStream.write(FilterOutputStream.java:97) at rawhttp.core.body.BodyConsumer$ChunkedBodyConsumer.lambda$consumeDataInto$2(BodyConsumer.java:102) at rawhttp.core.body.BodyConsumer$ChunkedBodyConsumer$$Lambda$46/1788247731.accept(Unknown Source) at rawhttp.core.body.ChunkedBodyParser.parseChunkedBody(ChunkedBodyParser.java:104) at rawhttp.core.body.BodyConsumer$ChunkedBodyConsumer.consumeDataInto(BodyConsumer.java:101) at rawhttp.core.body.BodyReader.writeDecodedTo(BodyReader.java:115) at rawhttp.core.body.BodyReader.decodeBody(BodyReader.java:200) at rawhttp.core.body.BodyReader.decodeBodyToString(BodyReader.java:212)

allowIllegalStartLineCharacters() ignored: InvalidHttpRequest{message='Illegal character in HTTP start line', lineNumber=1}

RawHttp rawHttp = new RawHttp(RawHttpOptions.newBuilder()
                .allowIllegalStartLineCharacters()
                .allowContentLengthMismatch()
                .allowComments()
                .allowIllegalConnectAuthority()
                .build());

I am trying to send some naughty requests via rawhttp, and its complaining about illegal start line characters even though I explicitly allowed it.

It appears that allowIllegalStartLineCharacters is ignored by HttpMetadataParser

private String parseStartLine(InputStream inputStream,
BiFunction<String, Integer, RuntimeException> createError,
boolean skipLeadingNewLine) throws IOException {
StringBuilder metadataBuilder = new StringBuilder();
final boolean allowNewLineWithoutReturn = options.allowNewLineWithoutReturn();
int b;
while ((b = inputStream.read()) >= 0) {
if (b == '\r') {
// expect new-line
int next = inputStream.read();
if (next < 0 || next == '\n') {
if (skipLeadingNewLine) {
skipLeadingNewLine = false;
continue;
}
break;
} else {
inputStream.close();
throw createError.apply("Illegal character after return", 1);
}
} else if (b == '\n') {
if (skipLeadingNewLine) {
skipLeadingNewLine = false;
continue;
}
if (!allowNewLineWithoutReturn) {
inputStream.close();
throw createError.apply("Illegal new-line character without preceding return", 1);
}
// unexpected, but let's accept new-line without returns
break;
} else {
char c = (char) b;
if (c == ' ' || FieldValues.isAllowedInVCHARs(c)) {
metadataBuilder.append(c);
} else {
throw createError.apply("Illegal character in HTTP start line", 1);
}
}
skipLeadingNewLine = false;
}
return metadataBuilder.toString();
}

Can a HTTP request be made over a SocketChannel

I'm investigating if I can use rawhttp to call devices over a UnixDomainSocketAddress. Problem is that these works with SocketChannel which has read and write, but not getInputStream and getOutputStream.

Would be great if something like this was possible...

try (var clientChannel = SocketChannel.open(UnixDomainSocketAddress.of(address))) {
   RawHttp rawHttp = new RawHttp();

   RawHttpRequest request = rawHttp.parseRequest(
                    "GET /hello.txt HTTP/1.1\r\n" +
                    "...");
   request.writeTo(clientChannel);
   
   RawHttpResponse<?> response = rawHttp.parseResponse(clientChannel);
   // Do something with response
}

rawhttp api ?

I am wondering if you have api for

  .setMethod(...)
  .setHost(...)
  .setHeaders(Map ....)


  then.

rawHttp.outputStream become rawHttpResult. Because i want to use it for Nonblocking IO methodology

Post request

Hi, Having trouble sending a simple rest request parsed from string.
while sending the following
` try{
Socket serverSocket = null;
RawHttp http = new RawHttp();
RawHttpRequest request = null;

            serverSocket = new Socket("localhost", 8081);

            request = http.parseRequest(
                    "POST /api/events/admin/newEvent HTTP/1.1\r\n" +
                            "Host: localhost\r\n" +
                            "User-Agent: ProxyServer\r\n" +
                            "Accept: application/json\r\n" +
                            "Content-Length: " + 85 + "\r\n" +
                            "\r\n" +
                            "{" +
                            "\"subType\": \"subType\"," +
                            "\"source\": \"proxy\"," +
                            "\"description\": \"Tunnel initiated\"," +
                            "\"ID\": 1000" +
                            "}").eagerly();
            //System.out.println(request.toString() + request.getBody());
            System.out.println("Body: "+request.getBody());
            RawHttpResponse<?> response = null;
            request.eagerly().writeTo(serverSocket.getOutputStream());
            response = http.parseResponse(serverSocket.getInputStream()).eagerly();
            serverSocket.close();
        }`

Getting the following output from sniffer :
`POST /api/events/admin/newEvent HTTP/1.1

Host: localhost
User-Agent: ProxyServer
Accept: application/json
Content-Length: 85

{"subType": "subType","source": "proxy","description": "Tunnel initiated","ID": 1000}`

Seems that an extra CRLF is inserted after the start line and after the headers.
What did I miss ?

Thanks,

Question: support for variables?

I am not sure if this is out of topic for the RawHTTP library, but there are several IDE supporting an Extended version of the HTTP RFC where you can defined environments and custom/system variables.


You can do stuff like this (this is a VSCode example):

### Rest client init:
@base_url = http://localhost:8090

### Send ping request:
GET {{base_url}}/ping
Accept: application/json

Where base_url can be defined in an other file so that you can run the request against different servers (in my eyes related to the question raised in #24).


The grammar implemented by IntelliJ is open-source and defined here: http-request-in-editor-spec. After having a quick look, it seems to me that you can do more with the VSCode extension.


I am looking for a way to test the *.http files we are writing and I am wondering if RawHTTP could be a fit or not.

At the end this can probably be implemented as a pre-processor (you take the *.http files that you edit in VSCode, replace the variables based on some context and use RawHTTP to send the real HTTP Request).

I am curious about your opinion on this topic. Thank you a lot in advance…

Redirect cycle on 302 POST

I'm running in to the issue that a POST returning a 302 redirects to the same URL resulting in a redirect cycle. I Believe the way POST -> 302 should be handled is to change the method to GET (but I could be wrong about this)

To fix this we might need to add

        if (statusCode == 302) {
            method = "GET";
        }

Somewhere here

if (statusCode == 303 && !request.getMethod().equals("HEAD")) {
method = "GET";
} else {
method = request.getMethod();
}

I'm not sure if you need to remove the content body from the request in this case.

Just a question

HI! It´s seem a great Java work on HTTPs requests.

I´m stucked on a very simple problem (i wish), trying to get image files by a kind of URL like this:

https://clubeceticismo.com.br/download/file.php?avatar=147_1649299329.jpg

I am using the code below, which usually works to download any file pointed by the url parameter to the specified pathname parameter. But, in this case, this php script (file.php) just returns an empty file.

I don't know why, but I suppose it´s because the method don´t identify the user-agent and the php script send files only to browsers connections.

Could you help me? Any tip?

    public static void downloadUrl2Pathname(
        final String url, 
        final String pathname
    ) throws MalformedURLException, FileNotFoundException, IOException {
        
        URL theFile = new URL(url);
        
        try (
                
            FileOutputStream fos = new FileOutputStream(pathname);

            ReadableByteChannel rbc = Channels.newChannel(theFile.openStream());
                
        ) {
            
            fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
        }           

    }//downloadUrl2Pathname() 

Thanks a lot in advance.

By the way... Do You speak Portuguese? Renato Athaydes sounds like brazilian name...

content-security-policy-report-only header very big.

hello,

Some webstite have a big value on content-security-policy-report-only
we got this Exception :
InvalidHttpResponse{message='Header value is too long', lineNumber=4}
How can we modify the maxHeaderValueLength on Rawhttp ?

Thanks.

Path segments with percents encoding

Similar to #30, I have the feeling that the library is decoding path segments prior to sending the request.


URL encoding with percents encoding in path segments seems to be tricky.
For example it is not clear if @ must be encoded or not in path-segments. E.g:`

http://localhost:1080/test/[email protected]/test
http://localhost:1080/test/someone%40example.com/test

According to this Stackoverflow question: URL encoding the character @ in query path keeping the @ not encoded is valid, but a lot of libs are replacing the @ with%40.

It seems to me that RawHttp is doing the contrary: it is decoding the symbol prior to sending it.

import java.net.Socket;

import rawhttp.core.RawHttp;
import rawhttp.core.RawHttpRequest;
import rawhttp.core.RawHttpResponse;

public class Snippet {
    public static void main(String[] args) throws Exception {
        String requestText = "GET /test/someone%40example.com/test HTTP/1.1\n" +
            "Host: localhost:1080\n" +
            "\n";
        RawHttp http = new RawHttp();
        RawHttpRequest httpRequest = http.parseRequest(requestText);
        try (Socket socket = new Socket("localhost", 1080)) {
            httpRequest.writeTo(socket.getOutputStream());
            RawHttpResponse<?> httpResponse = http.parseResponse(socket.getInputStream());
            String responseText = httpResponse.eagerly()
                .toString();
            socket.close();

            System.out.println(responseText);
        }
    }
}

Is arriving at the server side as /test/[email protected]/test


Can be tested with Mock-Server by adding 2 expectations:

PUT http://localhost:1080/expectation
{
  "httpRequest" : {
    "method" : "GET",
    "path" : "/test/[email protected]/test"
  },
  "httpResponse" : {
    "body" : "not_encoded"
  }
}

PUT http://localhost:1080/expectation
{
  "httpRequest" : {
    "method" : "GET",
    "path" : "/test/someone%40example.com/test"
  },
  "httpResponse" : {
    "body" : "is_encoded"
  }
}

When trying the Java Snippet, is_encoded is expected. Got not_encoded.


By comparison I have tried:

  1. Firefox: working as expected, the client is not doing any de-encoding before sending the request.

  2. rest-client for vs-code also working as expected. Those 2 requests are not the same:

### Rest client init:
@base_url = http://localhost:1080

### Send request:
GET {{base_url}}/test/[email protected]/test HTTP/1.1
### Rest client init:
@base_url = http://localhost:1080

### Send request:
GET {{base_url}}/test/someone%40example.com/test HTTP/1.1

Encoding issue in headers data

When headers data contains UTF8 characters, the value data returns an unreadable string
I have an HTTP request with UTF8 headers values
for example:
POST /app/v3.1.0/sample-api HTTP/1.1
HOST: api.example.com
Application-Id: B30F8BB0-3B37-11E7-8C40-000D3A196736
Connection: keep-alive
X-TEST: こんにちは
Content-Type: application/json

I parsed the request data using RawHttp.
When I read the header 'X-TEST' value, I get garbage string instead of the Japanese string

Code example:
RawHttp rawHttp = new RawHttp();
rawHttp.parseRequest(_requestData);
String headerValue = _httpRequest.getHeaders().get("X-TEST");

I also tried to provide InputStream to rawHttp.parseRequest and I get the same result.
I could not find any option to parse the headers as UTF8.

Thanks,
Sharon

response Transfer-Encoding: chunked error

error info
java.lang.IllegalStateException: Invalid chunk-size (too big, more than 4 hex-digits) at rawhttp.core.body.ChunkedBodyParser.readChunkSize(ChunkedBodyParser.java:215) at rawhttp.core.body.ChunkedBodyParser.parseChunkedBody(ChunkedBodyParser.java:97) at rawhttp.core.body.BodyConsumer$ChunkedBodyConsumer.consumeInto(BodyConsumer.java:93) null at rawhttp.core.body.BodyConsumer.consume(BodyConsumer.java:29) at rawhttp.core.body.EagerBodyReader.<init>(EagerBodyReader.java:35) at rawhttp.core.body.LazyBodyReader.eager(LazyBodyReader.java:68) at rawhttp.core.EagerHttpResponse.from(EagerHttpResponse.java:52) at rawhttp.core.RawHttpResponse.eagerly(RawHttpResponse.java:95) at rawhttp.core.RawHttpResponse.eagerly(RawHttpResponse.java:79)

Failed to parse request with InputStream (absolute-form + host header)

Hi,
While parsing a request with client.InputStream getting "Host specified both in Host header and in request line" exception. Request is sent with absolute-form path with identical host header.

Note that by RFC2616 in this case host header must be ignored.

I'm testing it with a proxy server, Proxy servers receive requests with absolute-form paths.

Error: not enough thanks to give.

After successfully performing a GET request with a body(!!!!1) required by the ugliest API I've ever used, thanks to this library, I got this error.

Seriously, this library saved my ass. Thanks. You can close this issue now.

Losing encoding in URI with encoded characters when Host given in a header

As demonstrated in #15 , the RequestLine#withHost method has a problem that it first decodes URI components (so that it can replace host), then puts them back together in a new URI via its constituents parts, which are now unencoded.

This process is destructive because once a component is decoded, it can't be encoded exactly anymore.

For example, take the following URI:

http://hello?and%26you&me

When the URI's parts are decoded, we have:

scheme = "http"
host = "hello"
query = "and&you&me"

This can never be encoded again back to the original value because once %26 is decoded into &, it's mixed up with the other & which are not supposed to be escaped when the query String is put back into a URI! But the information regarding which & is supposed to be encoded has been lost.

For this reason, when replacing the host of a URI, RawHTTP must be careful enough to not decode the constituent parts of the URI, avoiding this issue.

Timeout for RawHttp.parseResponse() and RawHttp.parseRequest

Hi,

You made a really great library. There is some small issue that stopps me at the moment from using it. I'm not sure if it is already possible or not but I could not figure out a way to set some sort of a timeout to parseResponse and parseRequest. The problem is that when the server sends a response that is not complete the parseRequest method hangs forever. The same problem occurres for the parseResponse.

It would be really great if there was a possibility to set a timeout to ensure the method does not block forever. Sometimes it can take a long time until the socket is closed and the method returns.

Support for Proxies

It would be really nice if Proxies http,https,socks with authentication would be supported.

Use RawHttp to implement a proxy

Hi,

I'm trying to use RawHttp to implement a simple http proxy but I encounter some problems. Here's what I try to do

    public static void main(String[] args) {
        RawHttpServer proxyServer = new TcpRawHttpServer(80);
        TcpRawHttpClient client = new TcpRawHttpClient();
        proxyServer.start(clientRequest -> {
            try {
                // Rewrite the request to point the end server
                clientRequest = clientRequest.withRequestLine(clientRequest.getStartLine().withHost("localhost:50005"));
                // Forward the request to the end server
                RawHttpResponse<Void> response = client.send(clientRequest);
                // Return the response as it is
                return Optional.of(response);
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
        });
    }

It seems to work great for GET requests but when I try a POST, things breaks. It appears that initial client gets a response with an empty body.
Here's a curl example on a /partenaires route that my end server can handle.

curl 'http://localhost/partenaires'   --data-raw 'email=iiuzhef%40izuef.com&school-name=ezfzef&postal-code=ozief&phone=fze'   --compressed -v
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 80 (#0)
> POST /partenaires HTTP/1.1
> Host: localhost
> User-Agent: curl/7.64.1
> Accept: */*
> Accept-Encoding: deflate, gzip
> Content-Length: 72
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 72 out of 72 bytes
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
* Closing connection 0

The POST request is forwarded and processed properly on the end server but the response is not sent to the original client (curl).
Is there a static state that makes RawHttpServer and TcpRawHttpClient not working together? Is there a special treatment of POST requests that could cause that? (again, it's working for GET requests)

Stopping server does not kill currently open client connnections

Code to try it out:

package my;

import rawhttp.core.RawHttp;
import rawhttp.core.client.TcpRawHttpClient;
import rawhttp.core.server.TcpRawHttpServer;

import java.io.IOException;
import java.net.Socket;
import java.util.Scanner;
import java.util.concurrent.atomic.AtomicBoolean;

import static java.util.Optional.ofNullable;

public class TestServer
{
    static final RawHttp http = new RawHttp();

    public static void main(String[] args) throws Throwable
    {
        var client = new TcpRawHttpClient(new TcpRawHttpClient.DefaultOptions()
        {
            @Override
            protected Socket createSocket(boolean useHttps, String host, int port) throws IOException
            {
                var s = super.createSocket(useHttps, host, port);
                s.setSoTimeout(2000);
                return s;
            }
        });

        var closeConnection = new AtomicBoolean(false);
        var clientRequest = http.parseRequest("GET localhost:8088/").eagerly();
        var server = new TcpRawHttpServer(8088);
        var scanner = new Scanner(System.in);

        try
        {
            while (true)
            {
                var line = scanner.nextLine();
                switch (line)
                {
                    case "start":
                        System.out.println("Starting server");
                        server.start((request -> {
                            try
                            {
                                return ofNullable(http.parseResponse("200 OK").eagerly(!closeConnection.get()));
                            }
                            catch (Exception e)
                            {
                                System.out.println("Error sending response: " + e);
                                server.stop();
                                throw new RuntimeException(e);
                            }
                        }));
                        break;
                    case "stop":
                        System.out.println("Stopping server");
                        server.stop();
                        break;
                    case "flip":
                        System.out.println("Flipping connection close responses");
                        closeConnection.set(!closeConnection.get());
                        break;
                    case "ping":
                        System.out.print("Ping --> ");
                        try
                        {
                            client.send(clientRequest);
                            System.out.println("OK");
                        }
                        catch (Exception e)
                        {
                            System.out.println("Error: " + e);
                        }
                        break;
                    case "exit":
                        return;
                    default:
                        System.out.println("Bad command");
                }
            }
        }
        finally
        {
            System.out.println("Shutdown");
            server.stop();
            client.close();
        }
    }

}

Example session:

ping

> Task :TestServer.main()
Ping --> Error: java.net.ConnectException: Connection refused
ping
Ping --> Error: java.net.ConnectException: Connection refused
start
Starting server
ping
Ping --> OK
ping
Ping --> OK
ping
Ping --> OK
stop
Stopping server
ping
Ping --> OK
ping
Ping --> OK
ping
Ping --> OK
ping
Ping --> OK
ping
Ping --> OK
flip
Flipping connection close responses
ping
Ping --> OK
ping
Ping --> OK
ping
Ping --> OK
ping
Ping --> OK
flip
Flipping connection close responses
ping
Ping --> OK
ping
Ping --> OK
exit
Shutdown

RedirectingRawHttpClient Redirect cycle detected.

Problem

I'm using the RedirectingRawHttpClient but it seems to get stuck in a redirect cycle with certain URLs.

Requesting google.com DOES work, but github.com, producthunt.com, news.ycombinator.com all fail

Version

2.5.1

Code

I'm using Clojure but I wrote the somewhat equivalent version in Java.

(->> "GET / HTTP/1.1\nHost: github.com\nAccept: */*"
     (.parseRequest (new RawHttp))
     (.send (new RedirectingRawHttpClient (TcpRawHttpClient.) 10)))
http = new RawHttp
rawClient = new RedirectingRawHttpClient(new TcpRawHttpClient, 10)
rawClient.send(http.parseRequest("GET / HTTP/1.1\nHost: github.com\nAccept: */*")).eagerly();

Error

1. Unhandled java.lang.IllegalStateException
   Redirect cycle detected. Visited locations: https://github.com/. Next
   location: https://github.com/

RedirectingRawHttpClient.java:   53  rawhttp.core.client.RedirectingRawHttpClient/send

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.