GithubHelp home page GithubHelp logo

hisco / http2-client Goto Github PK

View Code? Open in Web Editor NEW
33.0 4.0 14.0 256 KB

Transparently make http request to both http1 / http2 server.

Home Page: https://www.npmjs.com/package/http2-client

License: MIT License

JavaScript 100.00%
http2 http2-client http request nodejs javascript

http2-client's Introduction

HTTP2 client

Greenkeeper badge

NPM Version Build Status Known Vulnerabilities

Drop-in replacement for Nodes http and https that transparently make http request to both http1 / http2 server. Currently, it's the only http2/https compatible API for clients.

Motivation

http2 in Node.JS works entirely differently, while in browsers the experience is the same. http2-client was created to enable http2 / http1.1 requests with the same interface as http1.1.

The reason is that many NPM modules cannot upgrade to use http2.0 as these are coupled into http1.1 interface. With http2-client it should be very straight forward.

Meaning you don't need to know which protocol the destination supports before making the request http2-client will chose the one that works.

If the Node.js version you are using is not supporting http2 http2-client will automatically fallback to http.

Features

Transparently supports all http protocol.

  • Http/1.1
  • Https/1.1
  • Http/2.0

In case of http1.1

  • Connection pool is managed as usual with an http agent.

In case of http2.0

  • Connection pool is managed by Http2 agent.
  • Requests to the same "origin" will use the same tcp connection (per request manager) - automatically.
  • All Http2 features are available except push.

Usage - Same interface

request()

const {request} = require('http2-client');
const h1Target = 'http://www.example.com/';
const h2Target = 'https://www.example.com/';
const req1 = request(h1Target, (res)=>{
    console.log(`
Url : ${h1Target}
Status : ${res.statusCode}
HttpVersion : ${res.httpVersion}
    `);
});
req1.end();

const req2 = request(h2Target, (res)=>{
    console.log(`
Url : ${h2Target}
Status : ${res.statusCode}
HttpVersion : ${res.httpVersion}
    `);
});
req2.end();

get()

const {get} = require('http2-client');
const h1Target = 'http://www.example.com/';
const h2Target = 'https://www.example.com/';
get(h1Target, (res)=>{
    console.log(`
Url : ${h1Target}
Status : ${res.statusCode}
HttpVersion : ${res.httpVersion}
    `);
});

get(h2Target, (res)=>{
    console.log(`
Url : ${h2Target}
Status : ${res.statusCode}
HttpVersion : ${res.httpVersion}
    `);
});

API

The module mimics the nodejs http module interface of ClientRequest, get() and request(). Same API as regular http/s modules. Different options will be used depending on the destination this method will get.

  • Http/1.1
  • Https/1.1
  • Http/2.0

HttpRequestManager

By default this module exports a default request method the will try to detect the currect protocol to use (http2/http1.1/https1.1). However, you can always create different request manager with your specfic defaults and seperated cache.

  • options <Object>
    • keepH2ConnectionFor <number> Time to keep http2 connection after used last time. Default: 1000ms.
    • keepH1IdentificationCacheFor <number> TTL time for identification results of http1.1. Default: 30000ms.
    • useHttp <boolean> Should enforce http socket.
    • useHttps <boolean> Should enforce https socket.
//Use the default
const {request} = require('http2-client');
//Make a request
const req = request(/*....*/);
req.end();

//Alternatively create a new request
const {HttpRequestManager} = require('http2-client');
const httpRequestManager = new HttpRequestManager();
//Make a request
const req = httpRequestManager.request(/*....*/);
req.end();

Http/1.1 - request(options[, callback]) | request(url [,options][, callback])

  • options <Object> | <string> | <URL>
    • protocol <string> Protocol to use. Default: 'http:'.
    • host <string> A domain name or IP address of the server to issue the request to. Default: 'localhost'.
    • hostname <string> Alias for host. To support url.parse(), hostname is preferred over host.
    • family <number> IP address family to use when resolving host and hostname. Valid values are 4 or 6.When unspecified, both IP v4 and v6 will be used.
    • port <number> Port of remote server. Default: 80.
    • localAddress <string> Local interface to bind for network connections.
    • socketPath <string> Unix Domain Socket (use one of host:port or socketPath).
    • method <string> A string specifying the HTTP request method. Default: 'GET'.
    • path <string> Request path. Should include query string if any. E.G. '/index.html?page=12'. An exception is thrown when the request path contains illegal characters. Currently, only spaces are rejected but that may change in the future. Default: '/'.
    • headers An object containing request headers.
    • auth <string> Basic authentication i.e. 'user:password' to compute an Authorization header.
    • agent <http.Agent> | <boolean> Controls Agent behavior. Possible values:
      • undefined (default): use http.globalAgent for this host and port.
      • Agent object: explicitly use the passed in Agent.
      • false: causes a new Agent with default values to be used.
    • createConnection A function that produces a socket/stream to use for the request when the agent option is not used. This can be used to avoid creating a custom Agent class just to override the default createConnection function. See agent.createConnection() for more details. Any Duplex stream is a valid return value.
    • timeout <number> : A number specifying the socket timeout in milliseconds. This will set the timeout before the socket is connected.
    • setHost <boolean>: Specifies whether or not to automatically add the Host header. Defaults to true.
    • callback <Function>
    • Returns: <ClientRequest>
    • All http protocols - get(options[, callback]) | get(url [,options][, callback])

      • Differences are per protocol as described in relevant request() and protocol.
      • Same interface as request() with the method always set to GET. Properties that are inherited from the prototype are ignored.
      • Since most requests are GET requests without bodies, Node.js provides this convenience method. The only difference between this method and http.request() is that it sets the method to GET and calls req.end() automatically

      Https/1.1 - request(options[, callback]) | request(url [,options][, callback])

      • options <Object> | <string> | <URL> Accepts all options from Http/1.1 , with some differences in default values and aditional tls options:
        • protocol Default: 'https:'
        • port Default: 443
        • agent Default: https.globalAgent
        • rejectUnauthorized <boolean> If not false, the server certificate is verified against the list of supplied CAs. An 'error' event is emitted if verification fails; err.code contains the OpenSSL error code. Default: true.
        • ALPNProtocols: <string[]> | <Buffer[]> | <Uint8Array[]> | <Buffer> | <Uint8Array> An array of strings, Buffers or Uint8Arrays, or a single Buffer or Uint8Array containing the supported ALPN protocols. Buffers should have the format [len][name][len][name]... e.g. 0x05hello0x05world, where the first byte is the length of the next protocol name. Passing an array is usually much simpler, e.g. ['hello', 'world'].
        • servername: <string> Server name for the SNI (Server Name Indication) TLS extension.
        • checkServerIdentity(servername, cert) A callback function to be used (instead of the builtin tls.checkServerIdentity() function) when checking the server's hostname (or the provided servername when explicitly set) against the certificate. This should return an if verification fails. The method should return undefined if the servername and cert are verified.
        • session <Buffer> A Buffer instance, containing TLS session.
        • minDHSize <number> Minimum size of the DH parameter in bits to accept a TLS connection. When a server offers a DH parameter with a size less than minDHSize, the TLS connection is destroyed and an error is thrown. Default: 1024.
        • secureContext: Optional TLS context object created with tls.createSecureContext(). If a secureContext is not provided, one will be created by passing the entire options object to tls.createSecureContext().
        • lookup: <Function> Custom lookup function. Default: dns.lookup().
      • callback <Function>
      • Returns: <ClientRequest>

      Https/2.0 - request(options[, callback]) | request(url [,options][, callback])

      • options <Object> | <string> | <URL> Accepts all options from Https/1.1
      • callback <Function>
      • Returns: <ClientRequest>

      How?

      http2-client implements 'Application-Layer Protocol Negotiation (ALPN)'. Which means it first creates TCP connection, after successful ALPN negotiation the supported protocol is known.

      If the supported protocol is http2.0 http2-client will re-use the same connection. After the http2.0 connection won't be used for keepH2ConnectionFor which defaults to 100 ms, it will be automatically closed.

      If the supported protocol is http1.x http2-client will only cache the identification result and not the actual socket for keepH1IdentificationCacheFor which defaults to 30000 ms. Any socket configuration is manged by the http agent. If none is defined the node globalAgent will be used.

      License

      MIT

http2-client's People

Contributors

greenkeeper[bot] avatar hisco avatar mikeralphson avatar pimterry avatar simon-neusoft avatar simonwoolf avatar stefan-guggisberg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

http2-client's Issues

TypeError: requestOptions.createConnection is not a function

Hello,

Shortly after using the library with an https agent several times..

inStream.emit('socket' , requestOptions.createConnection());
                                              ^
TypeError: requestOptions.createConnection is not a function
    at HttpsRequest.makeHttp2Request (node_modules/http2-client/lib/request.js:415:47)
    at HttpsRequest.makeRequest (node_modules/http2-client/lib/request.js:344:21)
    at HttpsRequest.onIdentification (node_modules/http2-client/lib/request.js:279:24)
    at Object.onceWrapper (events.js:273:13)
    at HttpsRequest.emit (events.js:187:15)
    at HttpsRequest.onIdentify (node_modules/http2-client/lib/request.js:380:14)
    at TLSSocket.onConnect (node_modules/http2-client/lib/request.js:448:7)
    at Object.onceWrapper (events.js:273:13)
    at TLSSocket.emit (events.js:182:13)
    at TLSSocket.onConnectSecure (_tls_wrap.js:1074:10)

Please ask if you need more information

Error [ERR_HTTP2_GOAWAY_SESSION]: New streams cannot be created after receiving a GOAWAY

Error [ERR_HTTP2_GOAWAY_SESSION]: New streams cannot be created after receiving a GOAWAY
at ClientHttp2Session.request (internal/http2/core.js:1361:13)
at HttpsRequest.makeHttp2Request (http2-client/lib/request.js:412:30)
at HttpsRequest.makeRequest (http2-client/lib/request.js:344:21)
at HttpsRequest.onMakeRequest (http2-client/lib/request.js:274:24)
at args.(anonymous function) (/usr/lib/node_modules/pm2/node_modules/event-loop-inspector/index.js:138:29)
at process._tickCallback (internal/process/next_tick.js:61:11)

Let me know if you need more information
Thanks

http2-client doesn't work in Node 12

As far as I can tell, this project doesn't work at all in node 12.

I'm testing with Node 12.18.3 (the current node LTS release), and all HTTP/2 requests are never successfully sent to the server.

Easy to reproduce: npm test for this project fails, with every http2 test timing out.

Any ideas why? Happy to help get this fixed, but pointers in the right direction would be very useful!

Performance degrade from https

Expected: http2-client should give a performance gain to https, as requests can multiplex.
Actual: https consistently outperform http2-client, often by a factor of >2.

$ node test-http-speed.js 
..........
https: 543.7 mean, 396 min, 900 max
http2: 1172.1 mean, 944 min, 1495 max
$ node test-http-speed.js 
..........
https: 609 mean, 401 min, 2029 max
http2: 1589.4 mean, 1027 min, 4226 max

test-http-speed.js:

const { get: httpsGet } = require('https')
const { get: http2Get } = require('http2-client')

main().catch(err => console.error(err))

async function main() {
  require('events').defaultMaxListeners = 50; // avoid MaxListenersExceededWarning

  let httpsResults = []
  let http2Results = []

  for (let i = 0; i < 10; i++) {
    let start = Date.now()
    await do50RequestsInParalell(httpsGet)
    httpsResults.push(Date.now() - start)

    start = Date.now()
    await do50RequestsInParalell(http2Get)
    http2Results.push(Date.now() - start)

    process.stdout.write('.')
  }

  console.log()
  console.log(`https: ${meanMaxMin(httpsResults)}`)
  console.log(`http2: ${meanMaxMin(http2Results)}`)
}

function do50RequestsInParalell(get) {
  const urls = [...Array(50).fill(0).map((_, i) => `https://nvdbcache.geodataonline.no/arcgis/rest/services/Trafikkportalen/GeocacheTrafikkJPG/MapServer/tile/10/369/${465+i}`)]

  const promisifiedGet = promisify(get)
  const promises = urls.map(url => promisifiedGet(url))
  return Promise.all(promises)
}

function promisify(get) {
  return function (href) {
    return new Promise((resolve, reject) => {
      const req = get(href)
      req.end()

      req.on('error', (error) => {
        reject(new Error(`Unable to get ${href}: ${error}`))
      })

      req.on('response', (response) => {
        if (response.statusCode !== 200) {
          return reject(new Error(`Got status code ${response.statusCode} for request ${href}.`))
        }
        let data = ''
        response.on('data', chunk => data += chunk)
        response.on('end', () => resolve(data))
      })
    })
  }
}

function meanMaxMin(times) {
  let sum = times.reduce((s, n) => s + n, 0)
  let mean = sum / times.length
  let min = Math.min(...times);
  let max = Math.max(...times);

  return `${mean} mean, ${min} min, ${max} max`
}

Cannot create custom HttpRequestManager

Hi,

Thanks for the library. I would like to make use of the HttpRequestManager to be able to reuse the connection. I'm following the exact code example:

const {HttpRequestManager} = require('http2-client');
const httpRequestManager = new HttpRequestManager();

and i'm getting this error:
TypeError: HttpRequestManager is not a constructor
I'm tried using nodejs12 and nodejs10. I use typescript and i'm compiling to ES6, but also tried ES2018. Did you export this class?

Example returns a 404

When I execute the usage example, I get a 404 from the HTTP URL.

Url : http://www.example.com/
Status : 404
HttpVersion : 1.1


Url : https://www.example.com/
Status : 200
HttpVersion : 2.0

I've verified that http://www.example.com/ actually works and is a valid URL.

curl -i -v "http://www.example.com/"
*   Trying 93.184.216.34...
* TCP_NODELAY set
* Connected to www.example.com (93.184.216.34) port 80 (#0)
> GET / HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.61.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Cache-Control: max-age=604800
Cache-Control: max-age=604800
< Content-Type: text/html
Content-Type: text/html
< Date: Tue, 21 Aug 2018 04:13:39 GMT
Date: Tue, 21 Aug 2018 04:13:39 GMT
< Etag: "1541025663"
Etag: "1541025663"
< Expires: Tue, 28 Aug 2018 04:13:39 GMT
Expires: Tue, 28 Aug 2018 04:13:39 GMT
< Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
< Server: ECS (lga/1318)
Server: ECS (lga/1318)
< Vary: Accept-Encoding
Vary: Accept-Encoding
< X-Cache: HIT
X-Cache: HIT
< Content-Length: 1270
Content-Length: 1270

HTTP/2 on HTTP/1.1 Tunnel

I apologize for making an issue if this turns out to be more of a question.

I'm trying to tunnel HTTP/2 over HTTP/1.1, but feeding an agent from the tunnel library to http2Client.request does not seem to do anything, and it seems that the agent gets ignored/discarded in HttpRequestManager.makeRequest. As an easy test, you can pass in a bad agent, and an HTTP/2 request still succeeds.

Is there some way presently to do that (HTTP/2 over HTTP/1.1)?

I am running version 1.3.3, by the way.

RangeError: Maximum call stack size exceeded

I'm using http2-client to parse sitemaps, some of them have about 50 nested sitemaps.

const fetch = (url, options, http) => {
	http ??= newhttp(options);
	return http
		.get(url)
		.then((response) =>
			response?.body ? xmlparser.parse(response.body) : Promise.reject("Empty response body!")
		);
};

const crawl = (url, options, http) =>
	fetch(url, options, http).then((data) =>
		haveUrlSet(data)
			? parseUrlSet(data)
			: haveSitemapIndex(data)
			? Promise.all(
					parseSitemapIndex(data).map((link) => limit(() => crawl(link, options, http)))
			  ).then(
					(list) => list.flat()
			  )
			: Promise.reject("Invalid sitemap")
	);

Sometimes i'm catching such an error:

RangeError: Maximum call stack size exceeded
at Arguments.values (<anonymous>)
at ClientRequest.genericStubber (/project/node_modules/http2-client/lib/request.js:128:26)
at ClientRequest.method (/project/node_modules/http2-client/lib/request.js:44:23)
at ClientRequest.genericStubber (/project/node_modules/http2-client/lib/request.js:128:26)
at ClientRequest.method (/project/node_modules/http2-client/lib/request.js:44:23)
at ClientRequest.genericStubber (/project/node_modules/http2-client/lib/request.js:128:26)
at ClientRequest.method (/project/node_modules/http2-client/lib/request.js:44:23)
at ClientRequest.genericStubber (/project/node_modules/http2-client/lib/request.js:128:26)
at ClientRequest.method (/project/node_modules/http2-client/lib/request.js:44:23)
at ClientRequest.genericStubber (/project/node_modules/http2-client/lib/request.js:128:26)
at ClientRequest.method (/project/node_modules/http2-client/lib/request.js:44:23)
at ClientRequest.genericStubber (/project/node_modules/http2-client/lib/request.js:128:26)
at ClientRequest.method (/project/node_modules/http2-client/lib/request.js:44:23)
at ClientRequest.genericStubber (/project/node_modules/http2-client/lib/request.js:128:26)
at ClientRequest.method (/project/node_modules/http2-client/lib/request.js:44:23)
at ClientRequest.genericStubber (/project/node_modules/http2-client/lib/request.js:128:26)

Uncaught TypeError: Cannot read property 'prototype' of undefined

When i use http2-client and got in the vue project.
i get an error as follows:

request.js:11 Uncaught TypeError: Cannot read property 'prototype' of undefined
    at addFunctions (request.js:11)
    at Object.<anonymous> (request.js:24)
    at Object../node_modules/http2-client/lib/request.js (request.js:568)
    at __webpack_require__ (bootstrap:723)
    at fn (bootstrap:100)
    at Object../node_modules/http2-client/lib/index.js (index.js:3)
    at __webpack_require__ (bootstrap:723)
    at fn (bootstrap:100)
    at Object../src/utils/request2.js (request2.js:8)
    at __webpack_require__ (bootstrap:723)

in the code :(request.js)

function addFunctions(container , obj){
  const proto = obj.prototype;   //error 
  Object.keys(proto).forEach((name)=>{
    if (container.indexOf(name)!=-1)
    return;
    if (name.indexOf('_')!=0 && typeof proto[name] == 'function'){
      container.push(name);
    }
  })
}

DC44D394-1589-4D43-A5C6-9093BCC8FC46

Typescript support

Hi! Thanks for the lib. Would be nice to have support for typescript.

Now I have to declare the module manually in types.d.ts:

declare module 'http2-client' {
    import { RequestOptions, IncomingMessage, ClientRequest } from 'http'

    let request: (options: RequestOptions | string | URL, callback?: (res: IncomingMessage) => void) => ClientRequest
}

Typescript support

I'm trying to use your package, but due to lack of Typescript definitions I am at an impasse. I tried to handwrite them myself, but the code is too difficult to reason about, no offense, just being unfamiliar with the codebase and lack of documentation or comments.

Could you kindly provide them? If not, could you annotate the return types of exports and function prototypes? I could then put them together. Thanks in advance.

Network errors are not properly propagated

Builtin http.request reports a network error by emitting an 'error' event on the returned http.ClientRequest:

const http = require('http');

const req = http.request('http://localhost:54321');
req.on('error', (err) => console.log(err));  // => Error: connect ECONNREFUSED 127.0.0.1:54321
req.end();

http2-client however doesn't emit the 'error' event in the same scenario:

const http = require('http2-client');

const req = http.request('http://localhost:54321');
req.on('error', (err) => console.log(err));  // => never get here...
req.end();  // => process exits

console output:

events.js:288
      throw er; // Unhandled 'error' event
      ^

Error: connect ECONNREFUSED 127.0.0.1:54321
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1137:16)
Emitted 'error' event on Socket instance at:
    at emitErrorNT (internal/streams/destroy.js:92:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 54321
}

Tests hang

When running the test suite (npm test) the tests complete successfully but the process doesn't terminate. I guess that's because some sockets are still open...

(I am on macos)

Calling any stubbed method after the request has been made results in a stack overflow.

Example of how to reproduce: set the timeout to something low and do req.on('timeout', () => req.abort()).

Lots of methods are stubbed, so they can be forwarded to the stream once the stream's actually been chosen. The problem comes if you call such a method after the stream has been chosen - for example, abort(), which forwards to close(). The stub calls this.genericStubber(), which checks for this[$stubs] -- which isn't present as it was set to null when the stream was take()n -- and defaults to just calling this[method]();. But that's just this.close(), which is the stub. So you get an infinite loop.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.