koichik / node-tunnel Goto Github PK
View Code? Open in Web Editor NEWNode HTTP/HTTPS Agents for tunneling proxies
License: MIT License
Node HTTP/HTTPS Agents for tunneling proxies
License: MIT License
Does not honor the rejectUnauthorized: false option that is set inside https.get.
let agent = tunnel.httpsOverHttp({
proxy: {
host: 'localhost',
port: 3128
}
});
let options = {hostname: 'localhost', port: 8443, agent: agent, rejectUnauthorized: false}
https.get(options, (resp) => {
resp.statusCode.should.equal(200)
resp.on('data', console.log)
})
Error: self signed certificate
at TLSSocket.<anonymous> (_tls_wrap.js:1105:38)
at TLSSocket._finishInit (_tls_wrap.js:639:8)
at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:469:38)
Line 108 in 5fb2fb4
mergeOptions leaves the proxyOptions.headers behind and they don't get send to the proxy server.
An ugly example fix would be
Object.assign(connectOptions.headers, self.proxyOptions.headers)
on the following line
It seems the CONNECT request are sent as:
CONNECT <servername>:<port> HTTP/1.1
Host: <proxyhost>:<proxyport>
and it is rejected by some proxy implementations.
It should be
CONNECT <servername>:<port> HTTP/1.1
Host: <servername>:<port>
Using tunnel as https agent for Axios in a Kubernetes pod. Formerly was using the Javascript request
package, which worked just fine. Now migrating to Axios due to request
being deprecated.
If I don't use an https agent, I get an error about the certificate being expired, but with the https agent, I get the above error about connection refused. I'm guessing this is related to validating the certificate used in HTTPS. I can't track down whether this is a tunnel issue or an Axios issue. If anyone has any suggestions, I'd love to hear them.
For httpsOverHttp and httpsOverHttps tunnels, createSocket
method is replaced with createSecureSocket
which adds the actual socket as a member of the object returned from tls.connect
. Subsequent processing places listeners on this tls connection object and not on the underlying socket. When these listener callbacks invoke removeSocket, they end up providing the tls.connect object as an argument and not the actual underlying socket.
The removeSocket looks for a match of the provided tls.connect object to an element in the array in order to find the one to remove. However, since the actual underyling transport socket is what is placed in the sockets member array, no match is found, nothing is removed, and eventually the sockets array fills up to the maxSockets
length and subsequent requests will be queued permanently.
Recommending providing a removeSecureSocket
method
function removeSecureSocket (secureSocket) {
var self = this;
TunnelingAgent.prototype.removeSocket.call(self, secureSocket);
}
And then replacing the removeSocket
method for httpsOverHttp and httpsOverHttps instances with the secure socket version (essentially mirroring what is done for createSocket)
function httpsOverHttp(options) {
var agent = new TunnelingAgent(options);
agent.request = https.request;
agent.createSocket = createSecureSocket;
agent.removeSocket = removeSecureSocket;
return agent;
}
function httpsOverHttps(options) {
var agent = new TunnelingAgent(options);
agent.request = https.request;
agent.createSocket = createSecureSocket;
agent.removeSocket = removeSecureSocket;
return agent;
}
The IBM Product App Connect Enterprise has node-tunnel as an indirect dependency. We have a customer who is scanning docker images created as part of their CI/CD pipeline with the Twistlock security tool.
This is flagging the test keys in the test/keys directory as being secure credentials stored in the clear as described in the Embedded Secret Detection and Blocking section here:
https://www.twistlock.com/2017/04/17/introducing-twistlock-2-0/
We have informed our customer that we believe that removal of these files from the image will not adversely affect the function of the code and allow them to pass their security scans but wonder if the test data could be refactored to avoid the problem.
@koichik to be used by others a certain license is required for the project.
Thanks
tunnel.create("http", "http");
tunnel.create("http", "https");
tunnel.create("https", "https");
tunnel.create("https", "http");
For situations where we need to dynamically create agents.
I always get Request path contains unescaped characters.
while sending a request? I just use the default example
hi, localAddress not working, have you ever test this usage like my code?
it doesn't work on ubuntu 18.04
The users sometimes accidently put wrong data in proxy. Is it possible to catch these exceptions?
i tested timeout but not working
let agent = tunnel.httpsOverHttp({ proxy: { host: host, port: port, timeout: 3000 //not working // headers: request_headers } });
const url = "http://domain/";
https.request({
url: url,
agent: tunnel.auto(url)
});
This function would create reusable tunnels. proxy-from-env could be used for getting env vars.
Hi,
I get the following when installing:
npm WARN package.json [email protected] 'repositories' (plural) Not supported.
npm WARN package.json Please pick one as the 'repository' field
Is that intentional?
Thanks,
Martin
I am attempting to connect to aws s3 through a proxy and because of issues with authentication am trying to use node-tunnel.
I keep getting Networking Error: tunneling socket could not be established, cause=bind EADDRNOTAVAIL
these are my settings:
tunnelingAgent = tunnel.httpsOverHttp({
proxy: {
host: 'MY_PROXY_SERVER.com'
}
})
S3.config({
...
httpOptions: { agent: tunnelingAgent }
})
am I missing something?
Hi! I am trying to understand if when I use this agent it'll have keep alive enabled? Thanks
It would be great if node-tunnel
included http2
support.
Uncaught AssertionError: expected 2 to be 1
+ expected - actual
-2
+1
at Assertion.fail (node_modules/should/lib/assertion.js:91:17)
at Assertion.Object.defineProperty.value (node_modules/should/lib/assertion.js:163:19)
at Server.<anonymous> (test/https-over-https-error.js:253:28)
at net.js:1419:10
Hello @koichik ,
Thank you for maintaining tunnel
. I have been using it the past couple of months. However, I've started running into issues with Node v12. I tried digging into tunnel
/ gobal-tunnel-ng
codebase, but found that all the patches for legacy Node.js versions made it hard to comprehend the database. I have since written a HTTP/HTTPS agent for Node.js v12 and above – https://github.com/gajus/global-agent. Would you like to become a contributor?
Equally if you find global-agent
a useful alternative to tunnel
for new Node.js versions, I would appreciate if you mention it in README.md.
Thank you
In function onConnect you use assert.equal(head.length,0).
If assertion fails, it leads to uncaught process exception.
Please process this as an error so than exception is not beign thrown to nowhere.
This problem arises when I use request module and working with bad proxies.
The tests (and all the .PEM files) and others aren't needed for operation; please consider adding an .npmignore to filter out the files from the package 🥇
The Agent
internal APIs changed with nodejs 0.12, so addRequest
can be called with (req,options)
instead of (req,host,port,path)
. This change causes node-tunnel to break because it doesn't handle this case.
Is there any way to set keepAlive true in this lib?
I saw an older post that had some PRs open, but i didn't find if it's already enabled
It now defaults to Infinity
.
Hi, thanks for this module!
.httpsOverHttp
works, but only if I explicitly specify the port as 443. Otherwise it asks the proxy to CONNECT to port 80, my proxy does as requested, the SSL greeting goes to target server's port 80 and it replies with 400 Bad Request.
Could you please change the default port to 443 for the .httpsOver…
methods?
Update: Here's an example using example.net, they reply with 501 Not Implemented.
I'm using tunnel v0.0.5, Node.js v8.11.4, npm v5.6.0, Ubuntu 14.04.5 LTS trusty,
When I use an agent created by tunnel with proxy, if the proxy is not reachable, wrong exception raised.
Simple codes:
let a = tunnel.httpOverHttp({proxy: {host: 'nosuch.host.com', port: 8080}});
return got(anurl, {agent: a})
.then( res => {
if (res.statusCode === 200) return res.body;
else throw new Error('http status ' + res.statusCode + ', ' + res.body);
});
exception raised:
TypeError: options.request.emit is not a function
at ClientRequest.onError (/home/xxx/nodejs/proxylib/node_modules/tunnel/lib/tunnel.js:183:21)
at Object.onceWrapper (events.js:300:26)
at ClientRequest.emit (events.js:210:5)
at Socket.socketErrorListener (_http_client.js:406:9)
at Socket.emit (events.js:210:5)
at emitErrorNT (internal/streams/destroy.js:92:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
But EHOSTNOTFOUND or ENETWORK are expected error.
In TunnelingAgent.prototype.addRequest
a function is created
function onCloseOrRemove(err) {
self.removeSocket();
socket.removeListener('free', onFree);
socket.removeListener('close', onCloseOrRemove);
socket.removeListener('agentRemove', onCloseOrRemove);
}
I believe the socket
variable from closure needs to be provided as an argument
function onCloseOrRemove(err) {
//Provide argument to removeSocket
self.removeSocket(socket);
socket.removeListener('free', onFree);
socket.removeListener('close', onCloseOrRemove);
socket.removeListener('agentRemove', onCloseOrRemove);
}
npm http GET https://registry.npmjs.org/tunnel
npm http 200 https://registry.npmjs.org/tunnel
npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] 'repositories' (plural) Not supported.
npm WARN package.json Please pick one as the 'repository' field
i'd like to get this in to request, https://github.com/mikeal/request
whenever a proxy is enabled that is from one protocol to another we could enable this agent.
what do you think is the best way to get this in?
we could take the code in the way we do cookie code, or take it on as a dependency, the only caveat with taking it as a dependency is that it makes it a little harder for people to get fixes in from the request end but that might be manageable.
Maybe I'm not clear on the various functions offered by this module, but I've found it only works for me when I'm using httpOverHttp
. Whether or not I am connecting to a website that is http
or https
, and whether or not I'm using a proxy that is https
or not (see the https
column in https://free-proxy-list.net/) - regardless of which of these combinations I'm using, it only ever works with httpOverHttp
. Is this how it is meant to be or I'm doing something wrong?
How to deal with this error? it worked for several days, I didn't change any code, but this occurred today every time i createSocket.
Unhandled rejection Error: tunneling socket could not be established, cause=Parse Error
using global-tunnel to connect behind a proxy
when using https (bypassing options to http) throws following Exception
ps actual request was made by the twitter js sdk
Exception has occurred: TypeError
TypeError: Request path contains unescaped characters
at new ClientRequest (_http_client.js:52:11)
at TunnelingAgent.exports.request (http.js:31:10)
at TunnelingAgent.createSocket (/Users/d062254/Documents/sap git/srch-middleware/node_modules/tunnel/lib/tunnel.js:124:25)
at TunnelingAgent.createSecureSocket [as createSocket] (/Users/d062254/Documents/sap git/srch-middleware/node_modules/tunnel/lib/tunnel.js:196:41)
at TunnelingAgent.addRequest (/Users/d062254/Documents/sap git/srch-middleware/node_modules/tunnel/lib/tunnel.js:88:8)
at new ClientRequest (_http_client.js:159:16)
at Object.exports.request (http.js:31:10)
at Object.globalTunnel._defaultedAgentRequest (/Users/d062254/Documents/sap git/srch-middleware/node_modules/global-tunnel/index.js:211:38)
at Object.exports.request (https.js:199:15)
at Object.globalTunnel._defaultedAgentRequest (/Users/d062254/Documents/sap git/srch-middleware/node_modules/global-tunnel/index.js:211:38)
i fixed the issueby adding the following in /lib/tunnel.js
//... line 69 ...
TunnelingAgent.prototype.addRequest = function addRequest(req, host, port) {
var self = this;
var options;
if (typeof host === 'object')
{
options = host;
host = options.host;
port = options.port;
}
// end bugfix
if (self.sockets.length >= this.maxSockets) {
// We are over limit so we'll add it to the queue.
self.requests.push({host: host, port: port, request: req});
return;
}
usage of globalTunnel
import * as globalTunnel from "global-tunnel";
globalTunnel.initialize({
tunnel: 'both',
protocol: 'http:',
host: 'proxy.sam.corp',
port: 8080,
sockets: 50 // optional pool size for each http and https
});
actual request is being created by twitter-node-client
import * as TwitterLib from "twitter-node-client";
var twitterClient = new TwitterLib.Twitter({
consumerKey : "bla",
consumerSecret : "shh",
accessToken : "wa",
accessTokenSecret : "shshsh"
});
twitterClient.getSearch({'q': term, 'count': 10, 'result\_type':'popular'},
(err:Error)=> { reject(err); },
(tweets) => { resolve(tweets); });
Hey, I got the following an error.
tunneling socket could not be established, cause=Hostname/IP does not match certificate's altnames
Here my code :
const tunnellingAgent = tunnel.httpsOverHttps({
proxy: {
ca: [fs.readFileSync('ca.pem')],
host: "targetHost.com",
port: 3129,
proxyAuth: "user:pass",
}
})
const options = {
host: "api.ipify.org",
url: "https://api.ipify.org/?format=text",
method: "GET",
agent: tunnellingAgent,
checkServerIdentity: function (host, err) { }
}
R(options, (err, res) => {
console.log(err)
console.log(res)
})
The error seems similar to this issue request/request#1777 and #28
Add rejectUnauthorized: false
option solve the issue but the certificate file just do nothing in there.
request/request#1777 (comment)
So any other solution? Thanks
I am using express with this tunnel module ,i got a error ,i don't know why .
===api.js===
var http = require('http');
var tunnel = require('tunnel');
var tunnelingAgent = tunnel.httpOverHttp({
proxy : {
host : 'myproxy',
port : 74
}
});
exports.getWeather = function(){
http.request({
host : 'http://m.weather.com.cn/data/101010100.html',
port : 80,
agent: tunnelingAgent},function(res){
res.on('data',function(data){
console.log(data);
});
});
};
route/index.js
var api = require('api.js');
....
app.get('/api/weather',function(req,res){
api.getWeather();
});
then i got below error
Express server listening on port3000 mode production
events.js:72
throw er; // Unhandled 'error' event
^
Error: tunneling socket could not be established, cause=getaddrinfo ENOTFOUND
at ClientRequest.onError (c:\F\npm\N-blog-master\node_modules\tunnel\lib\tunnel.js:161:17)
at ClientRequest.g (events.js:175:14)
at ClientRequest.EventEmitter.emit (events.js:95:17)
at Socket.socketErrorListener (http.js:1528:9)
at Socket.EventEmitter.emit (events.js:95:17)
at net.js:830:16
at process._tickCallback (node.js:415:13)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.