Comments (11)
Hey, @jcarlsonautomatiq. Thanks for reporting this.
Not sure what's going off here. The path
option in ClientRequest
must always be a string starting with /
. It appears that when using hpagent
, it's not the case. This may as well be one of the many real-world use cases that are tricky and have to be handled in getUrlByRequestOptions
.
We need to add a test for this.
from interceptors.
@jcarlsonautomatiq can you please share how you intend to mock those requests?
from interceptors.
Your original example also has a mistake: got
cannot accept that proxy agent as the value of agents.https
. Doing so produces both TypeScript and runtime errors.
TypeScript:
No overload matches this call.
The last overload gave the following error.
Property 'options' is missing in type 'HttpProxyAgent' but required in type 'Agent'.ts(2769)
https.d.ts(31, 9): 'options' is declared here.
Runtime:
RequestError: Protocol "https:" not supported. Expected "http:"
❯ Request._makeRequest ../node_modules/.pnpm/[email protected]/node_modules/got/dist/source/core/index.js:1191:19
❯ new ClientRequest ../node:_http_client:189:11
I can help with fixing this but I need an exact example that is complete and working (well, failing, in our case).
from interceptors.
This code is currently being migrated from raw working but untested JS to Typescript which was one of the reasons I was actually adding the unit tests. I wonder if there is an update in got as well we are on "got-cjs": "^12.0.1" and it doesn't cause a protocol error. I will see about stripping this down to remove all the company stuff.
As for testing it I was hoping I would be able to intercept the proxy url and then look at the original request to determine what to return (or just grab a ?url out of the call). A little hacky but it would let me use the proxy calls and ensure I didn't break anything in the migration. I will hack up a proper working snippet this morning.
from interceptors.
The issue with the path originates from here:
https://github.com/delvedor/hpagent/blob/96f45f1d40bfbdfd0fcc84cdba056be6e0fb8f4c/index.js#L23
I'm not sure what's the intention here by providing the following request options to create a request:
[
{
method: 'CONNECT',
host: 'fake.proxy',
port: '443',
path: 'fake.ping:80',
setHost: false,
headers: {
connection: 'keep-alive',
host: 'fake.ping:80',
'proxy-authorization': 'Basic cHJveHlVc2VybmFtZTpwcm94eVBhc3N3b3Jk'
},
agent: false,
timeout: 0
}
]
What's the intended end request URL here, I wonder?
I can assume it's https://fake.proxy/fake.ping:80
. Once I do that, we get the next error:
RequestError: 'CONNECT' HTTP method is unsupported.
And this one is far more interesting. See, the Fetch API specification forbids creating requests with CONNECT
method:
new Request('/resource', { method: 'CONNECT' })
This fails standalone.
from interceptors.
The code is using some long lived proxies from Nimble. I expect that might be some of the strange connection issue but that is definitely odd. And apologies I should have had the fully working example in place initially.
import { HttpsProxyAgent } from 'hpagent'; // version: <edit>. Updated to 1.2.0 and it still will repro.
import { got } from 'got-cjs'; // version: 12.5.4
const host = 'fake.proxy';
const username = "proxyUsername";
const password = "proxyPassword";
const port = '443';
let proxyUrl = `http://${username}:${password}@${host}:${port}`;
let proxy = new HttpsProxyAgent({
keepAlive: true,
keepAliveMsecs: 1000,
maxSockets: 256,
maxFreeSockets: 256,
scheduling: 'lifo',
proxy: proxyUrl,
});
// Then used in a got call like this:
const PING_URL = "https://fake.ping/pinging";
await got.get({
url: PING_URL,
throwHttpErrors: false, // No idea the code thought it should do this?
agent: { https: proxy},
https: { rejectUnauthorized: false },
});
from interceptors.
It seems like if the invalid URL error is fixed this code test will probably have to be moved to the browser version of the MSW lib and only tested e2e which should be fine. Plus eventually I will run into the same agent and type conflicts :)
I will look more at the hpagent code that it might need a note about not being used in Node directly. Or forcing a custom http agent when used with a keepAlive connection outside a browser. odd ad of Nodejs > 19 supposedly keepAlive works with native fetch but finding definitive info about CONNECT in the spec is surprisingly hard.
https://stackoverflow.com/questions/62500011/reuse-tcp-connection-with-node-fetch-in-node-js
from interceptors.
Okay, so learned a lot from the Node.js docs, Got, and hpagent
here. Sharing below.
The URL is invalid, or is it?
Turns out, when performing the CONNECT
method in ClientRequest
, the path
option is expected to point to the target host:
const options = {
port: 1337,
host: '127.0.0.1',
method: 'CONNECT',
path: 'www.google.com:80',
}
const req = http.request(options)
Excerpt from the example in Node.js docs
- We must handle this in Interceptors
Moreover, when the server receives that request, the req.url
already equals to www.google.com:80
. This is some Node.js server magic I presume, but Interceptors must do the same thing (req.method
remains CONNECT
).
Request flow during proxying
This is what happens to the request in a simplified proxy setup (following the same Node.js example):
- Request is constructed with
CONNECT
method and<target>
path
. - Server receives the
connect
event wherereq.method
isCONNECT
andreq.url
is<target>
. - Server creates a new connection to
req.url
. - Once the connection is established to the target host, server writes to the request the
200 Connection <target>
HTTP message (directrequest.write()
). - Server then writes the
head
from theconnect
event to the request also. - Request then emits the
connect
event with three arguments: res
,IncomingMessage
. This is the `200 Connection `` response written by the server to the request (the header of the connection HTTP message).socket
,net.Socket
, the socket of the new connection to the<target>
.head
,Buffer
, whateverhead
server has written in itsconnect
event.
In the connect
request callback, you can then issue new requests by writing directly onto the target socket
and receiving the response from the target via socket.on('data')
.
I assume that last part is handled by hpagent
here because it's rather low-level.
from interceptors.
Basically, we don't support this message exchange right now. This is a new feature request and I will prioritize it accordingly. Will push the WIP pull request I have right now with the failing test and some findings. Thanks for making this apparent, @jcarlsonautomatiq!
I'm not sure as of now how to properly support this. Since the connect
event on ClientRequest
exposes an extraneous Socket
, it means we have to support the socket-level messaging in Interceptors (see #399). This isn't that easy and I'd rather avoid it if we can.
from interceptors.
Thanks for putting all the information in here it makes it a lot easier to think it through! If I might recommend a possible test framework solution I think just about everyone would accept it might go like this (I certainly would):
- Tell people that proxy code with a keepAlive is currently out of scope in the docs which is completely reasonable.
- Allow for a setting called 'proxyBehavior': transparent(default)|error|etc which the interceptor can look for and then just ignore the proxy and use the normal matchers for the destination URL call and build the url that way with the eventual actual method call (GET|POST|ETC). The headers would still have the proxy information so you could test your proxy was 'used'.
- Allow for a setting called 'youCanImplementYourProxyIfYourAreInsane' which gives people the ability to define a 'proxy' function that can run before the interceptors and do any header or rejections as required.
from interceptors.
Checkpoint
- https://nodejs.org/docs/latest-v18.x/api/http.html#event-connect
- https://github.dev/delvedor/hpagent
from interceptors.
Related Issues (20)
- ReadableStream cannot be properly polyfilled in this engine HOT 5
- Error using BatchInterceptor with browserInterceptors in browser environment. HOT 7
- `Response.type` throws in miniflare-like environments HOT 1
- Support HTTP2 HOT 3
- Intercepted fetch results in the the process never exiting / hangs Mocha tests
- error listener get called twice on mocked error with sync handler HOT 2
- Use "urlToHttpOptions" from "node:url"
- Header keys for requests are being converted to lowercase
- ReferenceError: TextEncoder is not defined when running (jest) tests HOT 4
- Treat exceptions as 500 responses instead of a "Failed to fetch" error HOT 7
- XMLHttpRequestController creates bad URL when location exists but href not set HOT 4
- Support aborting intercepted requests HOT 6
- WebSocket: Outgoing client events have the wrong "currentTarget" HOT 2
- WebSocket: Decide how to handle actual server errors
- Flaky: `modules/http/response/readable-stream.test.ts`
- ClientRequestInterceptor incorrectly encodes basic authentication header HOT 15
- Abstract request emitting logic
- Await all "response" listeners to finish HOT 4
- Protocol "https:" not supported. Expected "http:" when using "proxy-agent" HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from interceptors.