swift-server / http Goto Github PK

The current implementation doesn't honor the stop flag (the inout Bool) in HTTPBodyHandler. Need to fix that and write a test for it.

Determine the lifecycle of the HTTP server.

Refer to the discussions from #82 for history.

An initial idea discussed in that PR is:

• init: sets up the server only
• suspend: continues to accept work but doesn't process (socket would still be open, bound and listening, but not calling accept(). That will work until the kernel listen queue is full.)
• resume: resumes from suspend, starts the server if not already started.
• cancel: completely shutdown and destroy the server

@helje5 raised the point of:

For iOS, apps really need to close when going bg and completely reopen listen socks when going fg (unless you have special VoIP like blessing, in that mode the kernel will ensure the listen sockets stay alive - which may or may not be another thing we would want to support).

No instructions about installation are provided. I can't install it via Swift Package Manager.

The following code causes a crash at the specified line because parsedHTTPVersion is nil

try! server.start(port: port) { req, resp in
            DispatchQueue.main.async {
                resp.writeHeader(status: .ok)
                resp.done()
            }
            return .discardBody
}

It seems .discardBody causes resp.done() to be called, although writing a response after the request has been processed seems like a use case that should be supported.

I'm pulling comment from @helje5 from #28 over into this issue, because that issue is getting cluttered.

The worse aspect is that this calls the done-cb before the queued write has actually finished! This needs to be passed along the stream, that's the whole point.

My reason for not doing this was two-fold:

1. I was concerned about side-effects from passing the completion blocks into the Socket code, either in the form of memory leaks or in the form of deadlocks.
2. I don't see a case where it matters. The best this code can do is to hold onto the CB until after the socket write() completes. But, as the documentation for write(2) explicitly says "A successful return from write() does not make any guarantee that data has been committed," holding onto the CB that long doesn't provide any more reliability or information that if it was called where it is now.

There's no guarantee the connection won't be closed before the bytes that were queued will be written to the socket, but there's no guarantee they're going to make it to the far end of the TCP connection either.

So why does it matter whether the callback happens after the bytes are queued for write() or once write() had completed? The guarantee (or lack thereof) is the same either way. By Occam's Razor then, I think the simpler design is better.

I might well be missing something here. If so, I'd love to know what it is.

Thanks

I don't know where I can submit proposals for api changes so I opened this issue.

HTTPServer port

server port is determined in start(port:, handler:) method, while I think a server can start once with a one specified port. I could find a port property in HTTPServer class while with this design it have to be ports array. I propose relocating port argument from start() to class init() method.

HTTPResponse write methods

Personally I prefer something like response.header.write() instead of response.writeHeader(), response.body.write() instead of response.writeBody() and response.trailer.write() instead of response.writeTrailer(). Also we can keep room to extend header, body and trailer properties later.

HTTP version static variables

HTTP versions are limited, thus having HTTPVersion.v1_1 and HTTPVersion.v2_0 static vars might be fair.

A more complicated HTTPHeaders

HTTP headers are simply defined as a pair of strings. But for many of headers there have a set of defined values or a structured value. This implementation is prone to typo and human errors. So I propose to have some setters for frequent ones.

Implementation detail

// To set 'Range'
let range = 100... // half open or closed
headers.set(range: range) // result: 'Range: bytes 100-'

// To set 'Accept-Encoding'
headers.set(acceptEncodings: [.deflate, .gzip, .brotli, .identity])
// or
headers.set(acceptEncoding: .deflate, quality: 1.0)
headers.add(acceptEncoding: .gzip, quality: 0.8) // Note add instead of set
headers.add(acceptEncoding: .identity, quality: 0.5)
// values are defined in a enum/struct

// To set 'Cookie'
let cookie = HTTPCookie(properties: [.name : "name", .expires: Date(timeIntervalSinceNow: 3600)])
headers.set(cookie: cookie)
...
headers.add(cookie: cookie2)

// To set 'Accept-Language'
let locale = Locale.current
headers.set(acceptLanguages: [locale])
// also 'add(acceptLanguage:)' method to add progressively like Accept-Encoding

// To set 'Accept-Charset'
headers.set(acceptCharset: String.Encoding.utf8)

For Content-Type, we would define this struct in HTTPHeaders (should be extended with more types):

extension HTTPHeaders {
    struct MIMEType: RawRepresentable {
        public var rawValue: String
        public typealias RawValue = String
        
        public init(rawValue: String) {
            self.rawValue = rawValue
        }
        
        static let javascript = MIMEType(rawValue: "application/javascript")
        static let json = MIMEType(rawValue: "application/json")
        static let pdf = MIMEType(rawValue: "application/pdf")
        static let stream = MIMEType(rawValue: "application/octet-stream")
        static let zip = MIMEType(rawValue: "application/zip")
        
        // Texts
        static let css = MIMEType(rawValue: "text/css")
        static let html = MIMEType(rawValue: "text/html")
        static let plainText = MIMEType(rawValue: "text/plain")
        static let xml = MIMEType(rawValue: "text/xml")
        
        // Images
        static let gif = MIMEType(rawValue: "image/gif")
        static let jpeg = MIMEType(rawValue: "image/jpeg")
        static let png = MIMEType(rawValue: "image/png")
    }
}

And then we can set MIMEs this way:

headers.set(accept: .plainText)
headers.set(contentType: .json, encoding: .utf8)
// method converts 'NSStringEncoding' to IANA by 'CFStringConvertEncodingToIANACharSetName()' method

// Also similar approach for 'Pragma', 'TE' , etc...
headers.set(pragma: .noCache)
headers.set(transferEncodings: [.trailers, .deflate])

For some headers that accept date or integer or url, we implement methods accordingly:

// to set 'Content-Length'
headers.set(contentLength: 100)

// to set 'Date', 'If-Modified-Since', etc...
let fileModifiedDate = file.modifiedDate // or Date(timeIntervalSinceReferenceDate: 1000000)
headers.set(ifModifiedSince: fileModifiedDate)

// to set 'Referer' or 'Origin'
headers.set(referer: URL(string: "https://www.apple.com")!)

About Authorization header, we can have something like that, though I think it should be more refined and must be evaluated either we need them or not:

headers.set(authorizationType: .basic, user: "example", password: "pass")

Obviously, we must have some methods to fetch values. For example headers.contentLength will return a Int64? value, if it's defined by user, or headers.cookies will return a [HTTPCookie] array, headers.contentType will return a HTTPHeaders.MIMEType? optional, and so on and so forth.

I hope I can participate in implementing some parts if proposals are accepted

Best wish

From comments by @helje5 in #81

Port is insufficient. We need to allow an (or many!) arbitrary sockaddr_in (either via sockaddr_in or a wrapper). If only to allow IPv6 and such. af_local is very useful too.

With the last few swift-4.0 development snapshots, all but one of the XCTests with names that end with EndToEnd fail on Linux with timeouts. These are the tests that start a server on a random port and create a URLSession to send requests to it Those tests pass fine on Darwin (and on Linux under Swift 3.1.1).

If you comment all but one of the EndToEnd tests out, then the one you leave in (which ever one you pick) will pass. With two EndToEnd tests in, regardless of the order you choose, the first one will complete, and the second one will time out.

I'm guessing something isn't being cleaned up correctly, but I'm not sure what. Will update as I learn more.

It looks like the default branch has been deleted so the repository checks out empty.

It's easily fixed personally, but it would be nice if it could be fixed globally.
https://help.github.com/articles/error-remote-head-refers-to-nonexistent-ref-unable-to-checkout/

I started Swift 4 support (https://github.com/carlbrown/http/tree/swift4), but got blocked on #22. It doesn't look like that's going to get fixed, so I'll work around it, starting with #39

Once that gets merged, I'll push a new swift-tools-version = 4.0 Package.swift file and associated fixes to either develop or a swift4 specific branch, whichever we think makes the most sense.

The code uses POSIXError which I think is not an actual thing an implementation can create, right? It is just a namespace (at least on macOS, maybe that is different on Linux?).

Is that meant to be POSIXErrorCode?

https://github.com/swift-server/http/blob/develop/Sources/SwiftServerHttp/HTTPCommon.swift#L72

Note: This is reported as an Issue, because the error part of the API cannot be implemented. (My suggestion would be to not expose 'POSIX' at all but instead use an own HTTPError type which wraps the underlying socket error)

In theory with current implementation of HTTPStreamingParser it is possible to run out of the stack. This problem can occur if a long chain of callback receiving methods (i.e. writeBody) is called. To solve this completion callback should never be called before method returns.

Because of this problem it looks like two different APIs for sync/async should exist. Right now I see no solution to perform sync operations via async API.

See PR #20 for history.

Need to revisit this after #96 is complete and we have a perf baseline.

In #42 there was some discussion on the use of htons(). Moving that to this issue so that we can get that PR merged (which fixes port setting not being honoured).

Docs are exist in two incarnations: in the root of gh-pages branch and in docs directory of develop. Let's remove docs dir from develop branch. Why this should be done? Because it is very hard to read diffs of PRs containing docs update now.

In working on writing a framework on top of this one, I've discovered that I quite often want to send a payload that has a fixed length (e.g. a JSON Data produced by Codable), and the code paths to do that (e.g. in HTTPStreamingParser.swift when isChunked == false) are hard to use and poorly tested.

I need to fix that.

From @helje5:
#81 (comment)

An open question is, do we want to support multiple binds? I tend to say yes. This would modify the structure of the server a little.

(the other option is to use multiple servers if you want to listen on multiple addresses, but that would be kinda wasteful).

When explicitly declaring the Connection: close header on a client request, the server successfully completes the request, but fails to properly close the socket.

Looking deeper into this, it appears due to the below code block in HTTPStreamingParser in Done(). Here, self.parserConnector?.closeWriter() is called instead of self.parserConnector?.responseComplete(), which does not set self?.responseCompleted = true; thus, preventing PoCSocketConnectionListener.close() from passing the preconditions required to close the socket.

This only occurs on linux, which I think is due to URLSession replacing the explicit Connection: close header with a Connection: keep-alive header, but I'm not positive about that.

Code Block

public func done(completion: @escaping (Result) -> Void) {
    ...
    let closeAfter = {
            if self.clientRequestedKeepAlive {
                self.keepAliveUntil = Date(timeIntervalSinceNow: 
    StreamingParser.keepAliveTimeout).timeIntervalSinceReferenceDate
                self.parserConnector?.responseComplete()
            } else {
                self.parserConnector?.closeWriter()
            }
        }
    ...
}

   func close() {
        self.shouldShutdown = true
        
      // Always returns since responseComplete is false and no error has occurred
        if !self.responseCompleted && !self.errorOccurred {
            return
        }
        
        ...

    }

Below is a simple example. You'll notice in the terminal Closing idle socket (#) is continuously printed until the expectation timeout and the serverConnection.count is not decremented

 func testCloseConnections() {
        let expectation = self.expectation(description: "0 Open Connection")
        let server = HTTPServer()
        do {
            try server.start(port: 0, handler: OkHandler().handle)
            
            let session = URLSession(configuration: URLSessionConfiguration.default)
            let url1 = URL(string: "http://localhost:\(server.port)/echo")!
            var request = URLRequest(url: url1)
            request.httpMethod = "POST"
            request.setValue("close", forHTTPHeaderField: "Connection")
            
            let dataTask1 = session.dataTask(with: request) { (responseBody, rawResponse, error) in
                XCTAssertNil(error, "\(error!.localizedDescription)")
                DispatchQueue.main.asyncAfter(deadline: .now() + 25) {
                    XCTAssertEqual(server.connectionCount, 0)
                    expectation.fulfill()
                }
                
            }
            dataTask1.resume()
            
            self.waitForExpectations(timeout: 30) { (error) in
                if let error = error {
                    XCTFail("\(error)")
                }
            }
            server.stop()
        } catch {
            XCTFail("Error listening on port \(0): \(error). Use server.failed(callback:) to handle")
        }
    }

I'm pulling comment from @helje5 from #28 over into this issue, because that issue is getting cluttered.

I don't remember what we decided on here, IMO we should use DispatchData to pass buffers around as this supports iovecs (and AFAIK Data doesn't, right?).

A DispatchData doesn't require copying but supports custom deallocators, that is a key part of its design. (I'm not denying that there might be a better alternative, but DispatchData is much better than Data for this specific purpose (which presumably is the reason why it still exists ;-) )).

For what it's worth, I started with DispatchData in the APIs, and I wanted to keep it that way. The problem is that, at every turn, I found myself having to convert from Data/NSData to DispatchData and back. For example:

1. Part of this is because BlueSocket (that I really want to replace with something much simpler) explicitly reads/writes all socket data in NSData format.
2. Part of this is because of control messages. When you want to send "OK\r\n", that string easily converts to a Data, but getting it into a DispatchData is more of a pain in the ass, and when you have several things (like (String($0.count, radix: 16) + "\r\n").data(using: .utf8)) that you are sticking into the same packet, you end up doing that conversion over and over.
3. Part of it is because of response types. We found that the vast majority of the time, the response body that we wanted to send was either encoded as a String (Because the programmer typed it in to be turned into a response), or encoded as a Data (Because it's the contents of a file pulled off disk or it's the result of a JSONEncoder call). In both of those cases, it was easier not to have to do a bunch of Data<->DispatchData conversions before handing off to http.

So you tell me - What do you think the right answer is here?
[Just in case there's any confusion as to my tone here - I'm not being defensive - I honestly want to know. I'm not thrilled with the compromise I made, and I'd love for there to be an obvious or a consensus choice.]

In #96, @helje5 noted that the number of queues that we use in the HTTPServer should be lifted up into the Options

Just tracking that thought here.

Noticed that we don't seem to have an official style guide that we adhere to. I think it would be nice if we could pick one as we could encode at least parts of it into linting rules and ideally wouldn't have to look for examples in other Swift projects.

Based on comments from @helje5:
Don't do port: 0, should raise an error.
Do port: nil for kernel assigned, or a real port

I've been looking through example code for the HTTP library and the use of a associated value closure for the HTTPBodyProcessing.processBody case feels kind of usual. I've never seen such a pattern used in the Standard Library. And if this library's future is to be integrated into the Standard Library or shipped with Swift, I think it would be worth looking into keeping as close as possible to the API style of the Standard Library.

Any ideas?