follow-redirects / follow-redirects Goto Github PK
View Code? Open in Web Editor NEWNode.js module that automatically follows HTTP(S) redirects
Home Page: https://www.npmjs.com/package/follow-redirects
License: MIT License
Node.js module that automatically follows HTTP(S) redirects
Home Page: https://www.npmjs.com/package/follow-redirects
License: MIT License
Looking at the HTTP spec, I think it is entirely possible that the server will consume at least part of the request body before issuing a 307
redirect. When that happens, we are unable to redirect because whatever portion of the stream has been consumed is lost to us.
Perhaps we should buffer the request body when it's a stream (at least to a point).
Connections don't close on node v0.12.2 due to this line. The node server will eventually stop responding. In a production app I fixed it by commenting the line out.
Maybe putting a conditional for node version would be a better solution?
Thanks!
I think it is time for a 1.0.0 release.
It gives us the ability to publish updates in a semver
responsible way.
I have added a couple issues I want to close before then to the 1.0.0 milestone. Once those are done, I think we should do it.
It looks to me like if I e.g. do a file upload using this module (i.e. it won't be redirected) the entire file would be buffered in memory? If I have a file of several GB this could be a problem? Should we warn for this and possibly add a method + "content-length" check?
As far as I know, not all of the 300-399 http response codes needs to have a redirect. But looking at the code, I found you are redirecting everything in 300-399. Please look at the following url to have a reference:
Not taken into account set-cookie from the server
Your module don't keep options when he follows location
headers (usergagent, cookie ..) and agent (for keep alive connection) has gone one the first follow
exemple here :
http://pastebin.com/kxHViC2n
In the NodeJS framework, Iโd like to create a user agent (outside the browser) that also properly implements HTTP client caching. Problem with this library is that it handles redirects itself, and omits the cache in front of follow-redirects.
E.g., in pseudo javascript code:
var url = 'http://data.linkeddatafragments.org/bwv/classifications/10'
if (!cache.get(url)) {
http.get(url, function (...) {
var url2 = 'http://data.linkeddatafragments.org/bwv?subject=http%3A%2F%2Fdata.linkeddatafragments.org%2Fbwv%2Fclassifications%2F10';
if (!cache.get(url2)) {
http.get(url2);
}
});
}
This example will twice download the exact same page while the cached response should be given.
When the flag is set, the library will not automatically follow redirects, but will also allow for returning the new location URL, for which an external cache can then check what to do: resolve the promise, or again request it via follow-redirects.
Also allow a cache instance to be given to the follow-redirects library.
Inspired by express middleware design pattern, we could also document chaining HTTP request libraries, in which you can put caches behind follow-redirects without issues.
It seems the res.statusCode
always returns 200
even if I'm redirected. So is there any way I can be told I was redirected?
Thanks,
Elgs
Hello,
Could you please update the license to match either MIT or BSD?
Thanks,
-- Tito
I want to send send custom headers as I request a url.
Hey there,
I like your approach to plug in at a low level with http
and https
. I want to take the same approach for client http caching. I've spent a fair amount of time searching and didn't find any existing module, apart from cache-http which has a weird API and only look at If-Modified-Since
.
My question to you is are you aware of a library I may have missed and is there anything like connect
but for http clients?
Pretty much the same as Issue #27, HTTP 308 redirects should also use the same request method rather than changing the method to GET
.
Per RFC 7238:
308 Permanent Redirect
The 308 (Permanent Redirect) status code indicates that the target resource has been assigned a new permanent URI and any future references to this resource ought to use one of the enclosed URIs.
Clients with link editing capabilities ought to automatically re-link references to the effective request URI (Section 5.5 of [RFC7230]) to one or more of the new references sent by the server, where possible.
The server SHOULD generate a Location header field ([RFC7231], Section 7.1.2) in the response containing a preferred URI reference for the new permanent URI. The user agent MAY use the Location field value for automatic redirection. The server's response payload usually contains a short hypertext note with a hyperlink to the new URI(s).
A 308 response is cacheable by default; i.e., unless otherwise indicated by the method definition or explicit cache controls (see [RFC7234], Section 4.2.2).
Note: This status code is similar to 301 (Moved Permanently) ([RFC7231], Section 6.4.2), except that it does not allow changing the request method from POST to GET.
I'm getting the following error when running a test that uses nock.
TypeError: Cannot read property '_redirectable' of undefined
at eventHandlers.(anonymous function) (node_modules/follow-redirects/index.js:22:7)
at OverriddenClientRequest.RequestOverrider.req.once.req.on (node_modules/nock/lib/request_overrider.js:195:7)
at Writable.RedirectableRequest._performRequest (node_modules/follow-redirects/index.js:69:12)
at Writable.RedirectableRequest (node_modules/follow-redirects/index.js:46:7)
at Object.wrappedProtocol.request (node_modules/follow-redirects/index.js:202:10)
at Object.wrappedProtocol.get (node_modules/follow-redirects/index.js:207:33)
at getLatestUrl (lib/update.js:21:13)
...
The error doesn't happen when I run the program regularly. See this travis build for more context.
I have a fairly simple case where I'm building a tarball-stream on the fly and piping it to a request that's going through follow-redirects
.
It almost immediately crashes with this error:
Error: write after end
at ClientRequest.OutgoingMessage.write (_http_outgoing.js:439:15)
at Writable.RedirectableRequest._write (/home/espenh/webdev/sanity/cli/node_modules/follow-redirects/index.js:170:23)
at doWrite (_stream_writable.js:333:12)
at clearBuffer (_stream_writable.js:440:7)
at onwrite (_stream_writable.js:372:7)
at WritableState.onwrite (_stream_writable.js:89:5)
at CorkedRequest.finish (_stream_writable.js:547:7)
at afterWrite (_stream_writable.js:387:3)
at onwrite (_stream_writable.js:378:7)
at TLSSocket.WritableState.onwrite (_stream_writable.js:89:5)
I'll see if I can find time to write a reduced test case at some point, but don't have time right at this moment. Switching to regular HTTP/HTTPS for this request fixes the problem.
The following code will hang:
var req = http.request(opts, callback);
var stream = fs.createReadStream(filename);
stream.pipe(req);
Please note that the same code works fine with v0.0.7. Looks like the issue was introduced by #31. Removing the code that creates a request proxy resolves the problem.
I created a test that reproduces this issues. I will submit it as a PR.
Thank you!
I am making a fork of this project for use with IoTjs (which is like a light version of nodejs for embedded environments).
This fork may be used fo rcommercial purposes.
Going by the LICENSE file in the repo, its is perfectly acceptable for me to do so as long as I keep original LICENSE file right?
I spent half a day tracking this down. I'm not sure if this qualifies as a bug, but if you accept my suggested fix, I'll submit a PR.
The problem I was having was that one of my tests on my project was failing (timing out, specifically). Needless to say, there were many layers above follow-redirects
to comb through for the error. But I simply couldn't find anything. The problem I was seeing was that when my server returned a redirect, my client would hang. Hitting the server with a browser showed a proper Location header and status code. But the client (using axios
) just would not return. It was making the request and the server was sending back a response, but the promise in Axios was never resolved. I presumed that the redirect was being processed, but it never seemed to get a response.
So I rolled up my sleeves and tried to track down exactly where the process was breaking down. My original request was a POST
and follow-redirects
was converting that to a GET
. That by itself is what I expected, so that was no problem. I could change my original method to take a GET
instead and then the problem went away. So that fixed my problem (tests ran to completion), but it really bugged me that I didn't understand why this wasn't working and I didn't want to just leave it at that and just hope not to run into it again.
It clearly had something to do with redirects, so I turned on debugging output with DEBUG=follow-redirects
and took a look at what was going on. I could see follow-redirects
making the redirect request. I assumed, incorrectly, that perhaps when a POST
request was made, the POST
method was used on the redirect (that would have cause an error). But that wasn't the issue. In fact, if that were the case the request should have completed (which wasn't happening). This provides a subtle clue into what actually was the problem.
I compared the options
being passed in both the working and non-working cases. There were only two differences in those requests and they were related to headers. This is because my original request (which came through axios
had headers normally associated with a POST in them. Specifically Content-Type
and Content-Length
. When follow-redirects
switches the method to GET
in its 307 handling, it leaves those there. So the resulting redirected GET
request has a Content-Length
header in it. Since a GET
request has no body, this makes no sense.
I noticed that with the Chrome extension DHC, if I make a GET request with a Content-Length
header, it removes it automatically. But it appears what is happening is that follow-redirects
/http
are keeping that header in there and when it hits the Express server, the express server sits and waits until it gets all the bytes for the body. But, of course, there is no body.
What I did to fix this was to simply change this:
if (!(this._options.method in safeMethods)) {
this._options.method = 'GET';
}
...to this...
if (!(this._options.method in safeMethods)) {
this._options.method = 'GET';
// Remove a Content-Length header if one is present
delete this._options.headers["Content-Length"];
}
This seems perfectly reasonable to me and it is very much inline with what DHC does. But I wanted to first check here to see if a) this seems like a reasonable approach and b) whether you want me to create a PR for this.
I can't see what harm this would do, but you guys know more about the HTTP implications of this than I do. For that matter, there may be other headers to consider here as well (i.e., that would never apply to a GET
and therefore should be removed).
HEADs where being converted to GETs, meaning HEADing a relayed server results in the entire payload being dumped unceremoniously on the requester.
I also had an issue with being stuck in an endless relay loop.
my quick and dirty hack to make it work can be cut and pasted to the top of any jsnode file, (don't worry with npm or other bollocks, just have assert,debug and consume installed.
enjoy.
'use strict';
var assert = require('assert');
var debug = require('debug')('follow-redirects');
var consume = require('stream-consume');
// follow-redirects is taken and butchered from code obtained using
// "npm install follow-redirects"
// ie: https://www.npmjs.com/package/follow-redirects
// it had enough bugs in it to warrant fixing, but
// it's not worth doing
var followRedirects = (function (_nativeProtocols) {
var nativeProtocols = {};
var publicApi = {
maxRedirects: 5
};
for (var p in _nativeProtocols) {
/* istanbul ignore else */
if (_nativeProtocols.hasOwnProperty(p)) {
// http://www.ietf.org/rfc/rfc2396.txt - Section 3.1
assert(/^[A-Z][A-Z\+\-\.]*$/i.test(p), JSON.stringify(p) + ' is not a valid scheme name');
generateWrapper(p, _nativeProtocols[p]);
}
}
return publicApi;
// to redirect the result must have
// a statusCode between 300-399
// and a `Location` header
function isRedirect(res) {
return (res.statusCode >= 300 && res.statusCode <= 399 &&
'location' in res.headers);
}
function execute(options, callback) {
var req_method=options.method;
var fetchedUrls = [];
var clientRequest = cb();
// return a proxy to the request with separate event handling
var requestProxy = Object.create(clientRequest);
requestProxy._events = {};
requestProxy._eventsCount = 0;
if (callback) {
requestProxy.on('response', callback);
}
return requestProxy;
function cb(res) {
// skip the redirection logic on the first call.
if (res) {
var fetchedUrl = url.format(options);
fetchedUrls.unshift(fetchedUrl);
if (!isRedirect(res)) {
res.fetchedUrls = fetchedUrls;
requestProxy.emit('response', res);
return;
}
// we are going to follow the redirect, but in node 0.10 we must first attach a data listener
// to consume the stream and send the 'end' event
consume(res);
// need to use url.resolve() in case location is a relative URL
var redirectUrl = url.resolve(fetchedUrl, res.headers.location);
debug('redirecting to', redirectUrl);
options=url.parse(redirectUrl);
}
if (fetchedUrls.length > options.maxRedirects) {
var err = new Error('Max redirects exceeded.');
return forwardError(err);
}
options.nativeProtocol = nativeProtocols[options.protocol];
options.defaultRequest = defaultMakeRequest;
var req = (options.makeRequest || defaultMakeRequest)(options, cb, res);
req.on('error', forwardError);
return req;
}
function defaultMakeRequest(options, cb, res) {
if (res && res.statusCode !== 307) {
// This is a redirect, so use only GET methods, except for status 307,
// which must honor the previous request method.
options.method = 'GET';
}
if (req_method === 'HEAD')
options.method=req_method;
var req = options.nativeProtocol.request(options, cb);
if (res) {
// We leave the user to call `end` on the first request
req.end();
}
return req;
}
// bubble errors that occur on the redirect back up to the initiating client request
// object, otherwise they wind up killing the process.
function forwardError(err) {
requestProxy.emit('error', err);
}
}
function generateWrapper(scheme, nativeProtocol) {
var wrappedProtocol = scheme + ':';
var H = function () {};
H.prototype = nativeProtocols[wrappedProtocol] = nativeProtocol;
H = new H();
publicApi[scheme] = H;
H.request = function (options, callback) {
return execute(parseOptions(options, wrappedProtocol), callback);
};
// see https://github.com/joyent/node/blob/master/lib/http.js#L1623
H.get = function (options, callback) {
var req = execute(parseOptions(options, wrappedProtocol), callback);
req.end();
return req;
};
}
// returns a safe copy of options (or a parsed url object if options was a string).
// validates that the supplied callback is a function
function parseOptions(options, wrappedProtocol) {
if (typeof options === 'string') {
options = url.parse(options);
options.maxRedirects = publicApi.maxRedirects;
} else {
options.maxRedirects = publicApi.maxRedirects;
options.protocol = wrappedProtocol;
}
assert.equal(options.protocol, wrappedProtocol, 'protocol mismatch');
debug('options', options);
return options;
}
}) ({
http: require('http'),
https: require('https')
});
followRedirects.maxRedirects = 10;
var http = followRedirects.http,
https = followRedirects.https;
When trying to use this in Mashery's I/O Docs I see the following error:
TypeError: Object # has no method 'request'
at /home/warp/code/iodocs/node_modules/follow-redirects/index.js:60:40
at unsecuredCall (/home/warp/code/iodocs/app.js:568:23)
at processRequest (/home/warp/code/iodocs/app.js:481:9)
at callbacks (/home/warp/code/iodocs/node_modules/express/lib/router/index.js:272:11)
etc...
https://github.com/warpr/iodocs/blob/coverartarchive/app.js#L564
Do you have any idea if this Is a bug, or am I doing something wrong?
When no redirect happens, we would expect response.fetchedUrls[0] to contain the full URL for the request that was just executed. However, it only contains the protocol + the hostname.
Hi there,
I use node-rest-client module which uses your module in dependance.
When I wan't to make a HTTP request, it returns an error Object.assign is not a function
from your module (index.php lines 169 and 134).
My app is under Node 0.10.44 (I can't upgrade).
Please fix it asap (you can find a great polyfill on MDN or for NPM)
I think follow-redirects
default behavior should be to mimic the behavior of the http
and https
as used in the browser via browserify
. XMLHtppRequest
follows redirects transparently, and there is no way to disable that. Therefore the browserified versions of http
and https
automatically have a follow-redirects
like behavior, but the server side counterparts do not (obviously, there would be little point to this module otherwise). However, this behavior breaks one of the key goals of browserify
; the ability to write code that behaves identically on the client and server. This isn't the fault of browserify, due to the way XMLHttpRequest
behaves they can't possibly emulate the behavior of the native modules.
I am definitely not suggesting we turn follow-redirects
into a client side library, but instead help the server behave more like the client. This is backwards from the goal of most browserify modules, but as previously stated, it's just not possible to emulate the server in this case. Developers seeking to create a cross platform solution would use follow-redirects
on the server, and have browserify automatically swap it out with browserify-http
when creating the client bundle. Indeed, I have already detailed instructions for doing just that in the README.
We are already very close. For many scenarios the library will work as is. However, there are still a few points to be addressed.
300
level code. The spec notes many scenarios where typical browser behavior is actually not to spec. Rather than strictly conforming to the spec, we should follow whatever the most common behavior is.GET
on redirection, even if the original request was a POST
or otherwise. This is the correct response in most situations, but (I believe) not all.3XX
status code without a Location
header, we do not throw an error, but return the result to the registered callback. This all makes logical sense, but we should work to make our behavior the same as the client side modules.makeRequest
callback that allows us to change these behaviors on master. It's probably wise to keep a "legacy" version of that callback around that continues to mimic the behavior of the library as it has existed for years.http
and browserify-http
, etc). Plenty of cross-platorm options exist if you are willing to deal with a different API. The goal here should be an easy transitions for those using the core modules.browserify
implementations that are likely already bundled.Any chance you would consider changing the license to MIT or apache 2?
BSD requires attribution in any marketing done for a product build with this module.
Hi,
I was googling about an issue which i'm facing at my end about the Redirect URL and found your blog "http://syskall.com/how-to-follow-http-redirects-in-node-dot-js/" . Indeed it is an useful blog. However the problem at my end was
When I use the API end points manually with Postman (with interceptors turned ON) along with body and headers, I was able to get 302 redirects. However for my API Automation testing purpose I thought of using
request() // npm request
which is as follows. But always I get the response as 500. My objective of my test is to get the response.headers when I get the status code as 302 from response. Can you pls throw some light and provide me some direction as I got struck on this
request(
url : 'https://blah/blah'
client_id :'something.com.au'
redirect_uri :'http://something' // this is a dummy one as it is not required for mobile app
response_type :'code'
scope : 'ALLAPI'
Username : 'cxxx'
Password : 'ccxcxc'
headers : {
'Content-Type': 'application/x-www-form-urlencoded',
'followRedirect' : true
}
Thanks,
Brad
Module does not work with shortners which use meta refresh
http://gee.su/Ctv2n
it is only using _.extend
.
It could be replaced by merge. merge
is less than 1kB when minified (as opposed to 17
from underscore).
It might be better to just write an implementation ourselves though. merge
has a recursive option which we do not need.
An unexpected problem occurred during redirect url parsing:
https://github.com/olalonde/follow-redirects/blame/master/index.js#L145
My definitions as follows :
let options = {
host: 'proxy-host.com',
port: 8080,
method: 'GET',
path: 'http://download.image.com/test.jpg',
followRedirect: true,
maxRedirects: 3
};
This problem has led options.host
and options.port
failure after running redirect url passing. The new Host
and Port
in redirect url will be re-assigned to my options when I am using Host
and Port
as a proxy server.
@olalonde I've setup travis to automatically run our builds, and added badges for coverage, etc.
Can you head over to travis-ci.org and enable support (needs to be you).
Ignore the organization request from earlier, I thought that might have been a way to make it easier for me to maintain Travis stuff for you, but it seems I must be a full owner of the repo to set up travis on it. Other services (Coveralls, david-dm) allowed me to set up services just by being a collaborator, I think leaving it where it is will work fine and still require minimal drains on your time.
Would be useful to have a way to know if the url you passed redirected to another or not.
See #30 (comment)
Brings me to another problem BTW, that a call to abort() should probably abort everything in the entire chain.
I'm working on proxy for downloading files from Google Docs. Since the urls Google Drive API gives for non-gdrive filetypes (pdf, txt, jpeg) are redirects, I decided to use your library. So when you hit http://localhost:3000/oauth2_download?accessToken=x&url=my_url where x is my GDrive access_token, url is, say, https://docs.google.com/uc?id=0B_31K_MP92hUNjljYjIyMzgtZTBmNS00MGMwLWIxNmQtYjMyNDFiYjY0MTJl&export=download, the file is supposed to be downloaded. However, the browser sends me to the Google Docs sign-in page which (given that hitting https://docs.google.com/uc?id=0B_31K_MP92hUNjljYjIyMzgtZTBmNS00MGMwLWIxNmQtYjMyNDFiYjY0MTJl&export=download in browser downloads the file if you're logged into Google Docs) means the headers are not being sent correctly. Any idea how should I use 'follow-redirects' library to deal with this issue? Thank you in advance!
//var https = require('https');
var https = require('follow-redirects').https;
var app = require('express')();
var urllib = require('url');
// Downloads the file using OAuth2. Tested on Google Drive
app.get('/oauth2_download', function(req, res){
var url = req.param('url');
var accessToken = req.param('accessToken');
var headers = { "Authorization": "Bearer " + accessToken }
var searchname = urllib.parse(url).search;
var hostname = urllib.parse(url).hostname;
var pathname = urllib.parse(url).pathname;
if (url && accessToken) {
// downloading file with streaming
var options = {
hostname: hostname,
port: 443,
path: pathname + searchname,
method: 'GET',
headers: headers
};
https.request(options, function(response){
for (var key in response.headers) {
res.setHeader(key, response.headers[key]);
}
response.on('data', function(chunk) {
res.write(chunk);
});
response.on('end', function() {
res.end();
});
//console.log("Done downloading using OAuth2");
}).on('error', function(e) {
console.error(e);
}).end();
} else {
res.send(404);
res.end();
}
});
If I try to use a proxy and access URL that sends a single 302 redirect I'm ending up hitting the redirect limit.
The following code shows the problem.
node test.js <url of proxy> <url to test>
I've been using squid in a vm to test, if I access http://google.com it trys to redirect to http://www.google.com
GET http://google.com/ HTTP/1.1
Host: google.com
Connection: closeHTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Location: http://www.google.co.uk/?gfe_rd=cr&ei=Gnm5WL3dDqTS8Aec2IPwAg
Content-Length: 261
Date: Fri, 03 Mar 2017 14:09:30 GMT
X-Cache: MISS from ubuntu
X-Cache-Lookup: MISS from ubuntu:3128
Via: 1.1 ubuntu (squid/3.5.12)
Connection: close
var followRedirect = require('follow-redirects');
var http = followRedirect.http;
var urllib = require('url');
var proxy = "http://192.168.122.250:3128/";
var target = "http://google.com";
if (process.argv.length == 3) {
proxy = process.argv[2];
} else if (process.argv.length == 4) {
proxy = process.argv[2];
target = process.argv[3];
}
var options = urllib.parse(proxy);
var targetOpt = urllib.parse(target);
options.headers = {
"Host": targetOpt.host
}
options.pathname = target;
options.path = target;
console.log(options);
var req = http.request(options,function(res){
var body = "";
res.on('data',function(chunk){
body += chunk;
});
res.on('end',function(){
console.log("All Donei %d", res.statusCode);
console.log(body);
});
});
req.setTimeout(120000, function(){
console.log("timeout");
req.abort();
});
req.on('error', function(err){
console.log(err);
});
req.end();
As per RFC:
307 Temporary Redirect
The 307 (Temporary Redirect) status code indicates that the target resource resides temporarily under a different URI and the user agent MUST NOT change the request method if it performs an automatic redirection to that URI. Since the redirection can change over time, the client ought to continue using the original effective request URI for future requests.
The server SHOULD generate a Location header field in the response containing a URI reference for the different URI. The user agent MAY use the Location field value for automatic redirection. The server's response payload usually contains a short hypertext note with a hyperlink to the different URI(s).
Note: This status code is similar to 302 (Found), except that it does not allow changing the request method from POST to GET. This specification defines no equivalent counterpart for 301 (Moved Permanently) ([RFC7238], however, defines the status code 308 (Permanent Redirect) for this purpose).
Right now GET
is always being used.
function defaultMakeRequest(options, cb, res) {
if (res) {
// This is a redirect, so use only GET methods
options.method = 'GET';
}
var req = options.nativeProtocol.request(options, cb);
if (res) {
// We leave the user to call `end` on the first request
req.end();
}
return req;
}
I had some issues with redirects with axios that I'm now pretty sure are due to something in follow-redirects
. Axios is currently using 0.0.7, but I upgraded my local copy to use 0.2.0 and still seeing the issue. This does not occur with node 5.12.0, but it does happen with node 6.3.1.
The crux of it seems to be that I'm hitting an endpoint in my node app that is responding with a 301, but then I get a 401 from the resulting Location.
I set DEBUG=follow-redirects
but did not get any more useful info. I then added this line:
debug('making request with', options);
right here to log every outgoing request, and then I get the following output:
6:02:30 PM web.1 | Thu, 18 Aug 2016 01:02:30 GMT follow-redirects making request with { maxRedirects: 5,
6:02:30 PM web.1 | protocol: 'https:',
6:02:30 PM web.1 | hostname: 'postgres-api.heroku.com',
6:02:30 PM web.1 | port: null,
6:02:30 PM web.1 | path: '/client/v11/databases/dozing-patiently-6054/transfers',
6:02:30 PM web.1 | method: 'get',
6:02:30 PM web.1 | headers:
6:02:30 PM web.1 | { Accept: 'application/json, text/plain, */*',
6:02:30 PM web.1 | 'User-Agent': 'herokudata' },
6:02:30 PM web.1 | agent: undefined,
6:02:30 PM web.1 | auth: '[email protected]:<redacted>' }
6:02:31 PM web.1 | Thu, 18 Aug 2016 01:02:31 GMT follow-redirects redirecting to https://postgres-api.heroku.com/client/v11/apps/ac07e31a-710c-470e-831e-e97bfc88bcd3/transfers
6:02:31 PM web.1 | Thu, 18 Aug 2016 01:02:31 GMT follow-redirects making request with { maxRedirects: 5,
6:02:31 PM web.1 | protocol: 'https:',
6:02:31 PM web.1 | hostname: 'postgres-api.heroku.com',
6:02:31 PM web.1 | port: null,
6:02:31 PM web.1 | path: '/client/v11/apps/ac07e31a-710c-470e-831e-e97bfc88bcd3/transfers',
6:02:31 PM web.1 | method: 'get',
6:02:31 PM web.1 | headers:
6:02:31 PM web.1 | { Accept: 'application/json, text/plain, */*',
6:02:31 PM web.1 | 'User-Agent': 'herokudata' },
6:02:31 PM web.1 | agent: undefined,
6:02:31 PM web.1 | auth: null,
6:02:31 PM web.1 | nativeProtocol:
6:02:31 PM web.1 | { Server: { [Function: Server] super_: [Object] },
6:02:31 PM web.1 | createServer: [Function],
6:02:31 PM web.1 | globalAgent:
6:02:31 PM web.1 | Agent {
6:02:31 PM web.1 | domain: null,
6:02:31 PM web.1 | _events: [Object],
6:02:31 PM web.1 | _eventsCount: 1,
6:02:31 PM web.1 | _maxListeners: undefined,
6:02:31 PM web.1 | defaultPort: 443,
6:02:31 PM web.1 | protocol: 'https:',
6:02:31 PM web.1 | options: [Object],
6:02:31 PM web.1 | requests: {},
6:02:31 PM web.1 | sockets: [Object],
6:02:31 PM web.1 | freeSockets: {},
6:02:31 PM web.1 | keepAliveMsecs: 1000,
6:02:31 PM web.1 | keepAlive: false,
6:02:31 PM web.1 | maxSockets: Infinity,
6:02:31 PM web.1 | maxFreeSockets: 256,
6:02:31 PM web.1 | maxCachedSessions: 100,
6:02:31 PM web.1 | _sessionCache: [Object] },
6:02:31 PM web.1 | Agent: { [Function: Agent] super_: [Object] },
6:02:31 PM web.1 | request: [Function],
6:02:31 PM web.1 | get: [Function] },
6:02:31 PM web.1 | defaultRequest: [Function: defaultMakeRequest],
6:02:31 PM web.1 | slashes: true,
6:02:31 PM web.1 | host: 'postgres-api.heroku.com',
6:02:31 PM web.1 | hash: null,
6:02:31 PM web.1 | search: null,
6:02:31 PM web.1 | query: null,
6:02:31 PM web.1 | pathname: '/client/v11/apps/ac07e31a-710c-470e-831e-e97bfc88bcd3/transfers',
6:02:31 PM web.1 | href: 'https://postgres-api.heroku.com/client/v11/apps/ac07e31a-710c-470e-831e-e97bfc88bcd3/transfers' }
Note that the first request has the correct auth
field, but the subsequent request has auth: null
. Is this a node 6 breaking change? A bug in follow-redirects
? It looks like this change could be related, but as per the logs above, the host is not changing.
Got some problems on redirecting from a 302 POST request, basically the server refused to send back any response. After some investigation, I found it was caused by the Content-Length
header, which probably makes the server think it's an invalid GET request.
My current makeshift is deleting the Content-Length
field from this._options.headers
for all GET requests inside _processResponse
. Any better solution?
Someone sent me this email and I referred them to this issue:
Hi James
I have been using "follow-redirects" for making ReST calls and its very good. Thanks for the wonderful package. I am however stuck up when the site does a redirect. In that case, something very strange happens. Instead of getting a 307, for some very weird reason I keep getting 400 response.
Now I know this sounds very crazy as its the server which sends the response, and not the client, but whenever I use the normal "http" package, in my client code, instead of using the "follow-redirects", I get a 307 properly. Only for "follow-redirects" do I get a 400.
So is there any additional step that I need to do, to get this resolved.
I am trying below piece of code as you said.
var http = require('follow-redirects').http;
var https = require('follow-redirects').https;
var opts = {
host: "bitly.com",
path:"/UHfDGO",
port: 80};
http.get(opts, function (response) {
response.on('data', function (chunk) {
console.log(chunk);
});
}).on('error', function (err) {
console.error(err);
});
But when I execute it I get
vikas@vikas-pc:~/NodeJS Practice$ node trialnerror.js
<Buffer 55 73 65 72 2d 61 67 65 6e 74 3a 20 2a 0a 44 69 73 61 6c 6c 6f 77 3a 20 2f 6c 69 74 65 0a 44 69 73 61 6c 6c 6f 77 3a 20 2f 68 74 6d 6c 0a 0a 23 20 4e ... >
Instead, I need to get entire html page content
would be nice. Either add it to the response object or add a second argument to the callback function.
I would like to know what is wrong in this code.. I don't https.Request() gets called when I remove content-length.
setProperty: function(name, value) {
var deferred = q.defer(); // Take a deferral
var postData = JSON.stringify({name : value});
var options = {
protocol: 'https:',
host: 'developer-api.nest.com',
path: '/devices/thermostats/' + deviceId,
headers: {
'Authorization': 'Bearer ' + accessToken,
'Content-Type': 'application/json',
'Accept': 'application/json',
'Content-length': Buffer.byteLength(postData),
},
method: 'PUT',
};
var httpsRequest = https.request(options, (getRes) => {
console.log("Request sent");
var body = '';
getRes.setEncoding('utf8');
getRes.on('data', (data) => {
body += data;
});
getRes.on('end', () => {
if (getRes.statusCode != 200) {
deferred.reject("Error Code:" + getRes.statusCode);
} else {
var data = JSON.parse(body);
deferred.resolve(target_temperature_c);
}
});
getRes.on('error', (e) => {
deferred.reject(e);
});
});
httpsRequest.write(postData);
httpsRequest.end();
return deferred.promise; // return the promise
},
TypeError: Object function Object() { [native code] } has no method 'assign'
at Object.wrappedProtocol.request (/Users/james/Documents/D DRIVE/ES-SOURCE-CODE/TEST/ENHANCEMENT/Tagging/node_modules/follow-redirects/index.js:199:21)
When a request is done using the request
method, and when no redirect was made, the
response.responseUrl
will only contain the base url and not the full url with the path included.
var http = require('follow-redirects').http;
var URLParser = require('url');
// Does not work
var parsedUrl = URLParser.parse('http://graph.spitsgids.be/connections/?departureTime=2017-06-12T11%3A00');
var settings = { hostname: parsedUrl.hostname, port: parsedUrl.port, path: parsedUrl.path, method: 'GET'};
http.request(settings, function (response) {
console.log(response.responseUrl);
}).end();
// Works as it should:
http.get('http://graph.spitsgids.be/connections/?departureTime=2017-06-12T11%3A00', function (response) {
console.log(response.responseUrl);
}).on("error", function (err) {
console.error(err);
});
// Works as it should: (request to a URL that gets redirected)
var parsedUrl = URLParser.parse('http://graph.spitsgids.be/connections/?departureTime=2017-06-12T11%3A01');
var settings = { hostname: parsedUrl.hostname, port: parsedUrl.port, path: parsedUrl.path, method: 'GET'};
http.request(settings, function (response) {
console.log(response.responseUrl);
}).end();
Hi, I had a few questions:
If both agent
(nodejs http/https option) and agents
(follow-redirects option) are passed, then will the agent
option be used ever in the main or redirected requests?
It seems cookies set with a 302 redirect (i.e. having both location
and set-cookie
headers) are not passed over to the following URL? For example, this httpbin test URL: https://httpbin.org/cookies/set?k1=v1&k2=v2 should set two cookies, and do a 302 redirect to https://httpbin.org/cookies, which simply emits the cookie values, but it shows none.
Looks like the request timeout is applied per redirection, not as an overall measure, which is somewhat confusing. For example, if timeout is set to 2000 ms, and 5 redirections occurred taking a total of 5000 ms, then no error is thrown, although it should. I think the timeout value (just like the native http/s modules) should be applied to the whole request.
Cheers.
Using axios
, I discovered this is the underlying library that handles redirects, so ๐ .
The API I'm communicating with follows JSON API which describes that clients must add Content-Type: application/vnd.api+json
headers to their requests.
I have noticed though that when performing a redirect, this library will remove any header that starts with content-*
.
This results in my requests failing, with a 415 Unsupported Media Type
response (as per spec).
Just wondering what the intentions were behind this? Is there a workaround to "reinsert" headers such as Content-Type later?
Thanks!
I'm using node v0.10.12 and I'm quite new to this.
This little program doesn't end using follow-redirects but it completes using the native http module:
//var http = require('http');
var http = require('follow-redirects').http;
http.get('http://bit.ly/900913', function (res) {
var all = '';
res.on('data', function (chunk) {
all = all + chunk;
console.log(all.toString().length);
});
res.on('end', function (chunk) {
console.log(all.toString().length);
});
}).on('error', function (err) {
console.error(err);
});
Am I missing something obvious? Or maybe follow-redirects needs to be updated to node 0.10? Any hint would be appreciated. I could slowly take a stab at updating the module if that's indeed the problem.
I wanted to take a stab at #6 but I have to figure this out first.
According to this comment, we should handle the infinite loops here, in this library.
The description states that follow-redirects is a "drop-in replacement". However, the replacement is not complete, as the response
event is not correctly emitted.
In the original http
and https
implementations, the callback is a) optional and b) a shortcut for attaching a listener to the response
event. However, follow-redirects makes the callback mandatory and does not reuse the response
event. Any callbacks attached to response
will fire at the wrong time (at the first request).
I realize this is hard to implement correctly, given that EventEmitter
has no support for stopping event propagation. So if this cannot be fixed, it should probably be advertised that this module is not an exact replacement, but rather explain which parts it emulates and which parts it does not.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.