GithubHelp home page GithubHelp logo

follow-redirects / follow-redirects Goto Github PK

View Code? Open in Web Editor NEW
537.0 537.0 149.0 645 KB

Node.js module that automatically follows HTTP(S) redirects

Home Page: https://www.npmjs.com/package/follow-redirects

License: MIT License

JavaScript 100.00%
http https javascript nodejs redirect

follow-redirects's Introduction

Follow Redirects

Drop-in replacement for Node's http and https modules that automatically follows redirects.

npm version Build Status Coverage Status npm downloads Sponsor on GitHub

follow-redirects provides request and get methods that behave identically to those found on the native http and https modules, with the exception that they will seamlessly follow redirects.

const { http, https } = require('follow-redirects');

http.get('http://bit.ly/900913', response => {
  response.on('data', chunk => {
    console.log(chunk);
  });
}).on('error', err => {
  console.error(err);
});

You can inspect the final redirected URL through the responseUrl property on the response. If no redirection happened, responseUrl is the original request URL.

const request = https.request({
  host: 'bitly.com',
  path: '/UHfDGO',
}, response => {
  console.log(response.responseUrl);
  // 'http://duckduckgo.com/robots.txt'
});
request.end();

Options

Global options

Global options are set directly on the follow-redirects module:

const followRedirects = require('follow-redirects');
followRedirects.maxRedirects = 10;
followRedirects.maxBodyLength = 20 * 1024 * 1024; // 20 MB

The following global options are supported:

  • maxRedirects (default: 21) – sets the maximum number of allowed redirects; if exceeded, an error will be emitted.

  • maxBodyLength (default: 10MB) – sets the maximum size of the request body; if exceeded, an error will be emitted.

Per-request options

Per-request options are set by passing an options object:

const url = require('url');
const { http, https } = require('follow-redirects');

const options = url.parse('http://bit.ly/900913');
options.maxRedirects = 10;
options.beforeRedirect = (options, response, request) => {
  // Use this to adjust the request options upon redirecting,
  // to inspect the latest response headers,
  // or to cancel the request by throwing an error

  // response.headers = the redirect response headers
  // response.statusCode = the redirect response code (eg. 301, 307, etc.)

  // request.url = the requested URL that resulted in a redirect
  // request.headers = the headers in the request that resulted in a redirect
  // request.method = the method of the request that resulted in a redirect
  if (options.hostname === "example.com") {
    options.auth = "user:password";
  }
};
http.request(options);

In addition to the standard HTTP and HTTPS options, the following per-request options are supported:

  • followRedirects (default: true) – whether redirects should be followed.

  • maxRedirects (default: 21) – sets the maximum number of allowed redirects; if exceeded, an error will be emitted.

  • maxBodyLength (default: 10MB) – sets the maximum size of the request body; if exceeded, an error will be emitted.

  • beforeRedirect (default: undefined) – optionally change the request options on redirects, or abort the request by throwing an error.

  • agents (default: undefined) – sets the agent option per protocol, since HTTP and HTTPS use different agents. Example value: { http: new http.Agent(), https: new https.Agent() }

  • trackRedirects (default: false) – whether to store the redirected response details into the redirects array on the response object.

Advanced usage

By default, follow-redirects will use the Node.js default implementations of http and https. To enable features such as caching and/or intermediate request tracking, you might instead want to wrap follow-redirects around custom protocol implementations:

const { http, https } = require('follow-redirects').wrap({
  http: require('your-custom-http'),
  https: require('your-custom-https'),
});

Such custom protocols only need an implementation of the request method.

Browser Usage

Due to the way the browser works, the http and https browser equivalents perform redirects by default.

By requiring follow-redirects this way:

const http = require('follow-redirects/http');
const https = require('follow-redirects/https');

you can easily tell webpack and friends to replace follow-redirect by the built-in versions:

{
  "follow-redirects/http"  : "http",
  "follow-redirects/https" : "https"
}

Contributing

Pull Requests are always welcome. Please file an issue detailing your proposal before you invest your valuable time. Additional features and bug fixes should be accompanied by tests. You can run the test suite locally with a simple npm test command.

Debug Logging

follow-redirects uses the excellent debug for logging. To turn on logging set the environment variable DEBUG=follow-redirects for debug output from just this module. When running the test suite it is sometimes advantageous to set DEBUG=* to see output from the express server as well.

Authors

License

MIT License

follow-redirects's People

Contributors

af-mikecrowe avatar anthonygauthier avatar cavis avatar claasahl avatar commanderroot avatar davecardwell avatar daviddenton avatar digitalbrainjs avatar evanw avatar faust64 avatar flarna avatar fnando avatar jamestalmage avatar markherhold avatar mbargiel avatar modosc avatar motoenduroboy avatar ne-smalltown avatar nickuraltsev avatar olaf-k avatar olalonde avatar psmoros avatar realityking avatar rexxars avatar rubenverborgh avatar sashashura avatar shorrockin avatar smarusa avatar splitice avatar strugee avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

follow-redirects's Issues

Exceeding Max Redirects when using a proxy and hitting 302

If I try to use a proxy and access URL that sends a single 302 redirect I'm ending up hitting the redirect limit.

The following code shows the problem.

node test.js <url of proxy> <url to test>

I've been using squid in a vm to test, if I access http://google.com it trys to redirect to http://www.google.com

GET http://google.com/ HTTP/1.1
Host: google.com
Connection: close

HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Location: http://www.google.co.uk/?gfe_rd=cr&ei=Gnm5WL3dDqTS8Aec2IPwAg
Content-Length: 261
Date: Fri, 03 Mar 2017 14:09:30 GMT
X-Cache: MISS from ubuntu
X-Cache-Lookup: MISS from ubuntu:3128
Via: 1.1 ubuntu (squid/3.5.12)
Connection: close

var followRedirect = require('follow-redirects');
var http = followRedirect.http;
var urllib = require('url');

var proxy = "http://192.168.122.250:3128/";
var target = "http://google.com";
if (process.argv.length == 3) {
	proxy = process.argv[2];
} else if (process.argv.length == 4) {
	proxy = process.argv[2];
	target = process.argv[3];
}

var options = urllib.parse(proxy);
var targetOpt = urllib.parse(target);

options.headers = {
	"Host": targetOpt.host
}

options.pathname = target;
options.path = target;

console.log(options);

var req = http.request(options,function(res){
	var body = "";
	res.on('data',function(chunk){
		body += chunk;
	});

	res.on('end',function(){
		console.log("All Donei %d", res.statusCode);
		console.log(body);
	});
});

req.setTimeout(120000, function(){
	console.log("timeout");
	req.abort();
});

req.on('error', function(err){
	console.log(err);
});

req.end();

Callback and response event do not correspond to original

The description states that follow-redirects is a "drop-in replacement". However, the replacement is not complete, as the response event is not correctly emitted.

In the original http and https implementations, the callback is a) optional and b) a shortcut for attaching a listener to the response event. However, follow-redirects makes the callback mandatory and does not reuse the response event. Any callbacks attached to response will fire at the wrong time (at the first request).

I realize this is hard to implement correctly, given that EventEmitter has no support for stopping event propagation. So if this cannot be fixed, it should probably be advertised that this module is not an exact replacement, but rather explain which parts it emulates and which parts it does not.

I face the below issue while trying to do a https request

TypeError: Object function Object() { [native code] } has no method 'assign'
at Object.wrappedProtocol.request (/Users/james/Documents/D DRIVE/ES-SOURCE-CODE/TEST/ENHANCEMENT/Tagging/node_modules/follow-redirects/index.js:199:21)

Enable Travis Build

@olalonde I've setup travis to automatically run our builds, and added badges for coverage, etc.

Can you head over to travis-ci.org and enable support (needs to be you).

Ignore the organization request from earlier, I thought that might have been a way to make it easier for me to maintain Travis stuff for you, but it seems I must be a full owner of the repo to set up travis on it. Other services (Coveralls, david-dm) allowed me to set up services just by being a collaborator, I think leaving it where it is will work fine and still require minimal drains on your time.

Auth lost on redirect to same host in node 6.3.1

I had some issues with redirects with axios that I'm now pretty sure are due to something in follow-redirects. Axios is currently using 0.0.7, but I upgraded my local copy to use 0.2.0 and still seeing the issue. This does not occur with node 5.12.0, but it does happen with node 6.3.1.

The crux of it seems to be that I'm hitting an endpoint in my node app that is responding with a 301, but then I get a 401 from the resulting Location.

I set DEBUG=follow-redirects but did not get any more useful info. I then added this line:

debug('making request with', options); 

right here to log every outgoing request, and then I get the following output:

6:02:30 PM web.1  |  Thu, 18 Aug 2016 01:02:30 GMT follow-redirects making request with { maxRedirects: 5,
6:02:30 PM web.1  |    protocol: 'https:',
6:02:30 PM web.1  |    hostname: 'postgres-api.heroku.com',
6:02:30 PM web.1  |    port: null,
6:02:30 PM web.1  |    path: '/client/v11/databases/dozing-patiently-6054/transfers',
6:02:30 PM web.1  |    method: 'get',
6:02:30 PM web.1  |    headers:
6:02:30 PM web.1  |     { Accept: 'application/json, text/plain, */*',
6:02:30 PM web.1  |       'User-Agent': 'herokudata' },
6:02:30 PM web.1  |    agent: undefined,
6:02:30 PM web.1  |    auth: '[email protected]:<redacted>' }
6:02:31 PM web.1  |  Thu, 18 Aug 2016 01:02:31 GMT follow-redirects redirecting to https://postgres-api.heroku.com/client/v11/apps/ac07e31a-710c-470e-831e-e97bfc88bcd3/transfers
6:02:31 PM web.1  |  Thu, 18 Aug 2016 01:02:31 GMT follow-redirects making request with { maxRedirects: 5,
6:02:31 PM web.1  |    protocol: 'https:',
6:02:31 PM web.1  |    hostname: 'postgres-api.heroku.com',
6:02:31 PM web.1  |    port: null,
6:02:31 PM web.1  |    path: '/client/v11/apps/ac07e31a-710c-470e-831e-e97bfc88bcd3/transfers',
6:02:31 PM web.1  |    method: 'get',
6:02:31 PM web.1  |    headers:
6:02:31 PM web.1  |     { Accept: 'application/json, text/plain, */*',
6:02:31 PM web.1  |       'User-Agent': 'herokudata' },
6:02:31 PM web.1  |    agent: undefined,
6:02:31 PM web.1  |    auth: null,
6:02:31 PM web.1  |    nativeProtocol:
6:02:31 PM web.1  |     { Server: { [Function: Server] super_: [Object] },
6:02:31 PM web.1  |       createServer: [Function],
6:02:31 PM web.1  |       globalAgent:
6:02:31 PM web.1  |        Agent {
6:02:31 PM web.1  |          domain: null,
6:02:31 PM web.1  |          _events: [Object],
6:02:31 PM web.1  |          _eventsCount: 1,
6:02:31 PM web.1  |          _maxListeners: undefined,
6:02:31 PM web.1  |          defaultPort: 443,
6:02:31 PM web.1  |          protocol: 'https:',
6:02:31 PM web.1  |          options: [Object],
6:02:31 PM web.1  |          requests: {},
6:02:31 PM web.1  |          sockets: [Object],
6:02:31 PM web.1  |          freeSockets: {},
6:02:31 PM web.1  |          keepAliveMsecs: 1000,
6:02:31 PM web.1  |          keepAlive: false,
6:02:31 PM web.1  |          maxSockets: Infinity,
6:02:31 PM web.1  |          maxFreeSockets: 256,
6:02:31 PM web.1  |          maxCachedSessions: 100,
6:02:31 PM web.1  |          _sessionCache: [Object] },
6:02:31 PM web.1  |       Agent: { [Function: Agent] super_: [Object] },
6:02:31 PM web.1  |       request: [Function],
6:02:31 PM web.1  |       get: [Function] },
6:02:31 PM web.1  |    defaultRequest: [Function: defaultMakeRequest],
6:02:31 PM web.1  |    slashes: true,
6:02:31 PM web.1  |    host: 'postgres-api.heroku.com',
6:02:31 PM web.1  |    hash: null,
6:02:31 PM web.1  |    search: null,
6:02:31 PM web.1  |    query: null,
6:02:31 PM web.1  |    pathname: '/client/v11/apps/ac07e31a-710c-470e-831e-e97bfc88bcd3/transfers',
6:02:31 PM web.1  |    href: 'https://postgres-api.heroku.com/client/v11/apps/ac07e31a-710c-470e-831e-e97bfc88bcd3/transfers' }

Note that the first request has the correct auth field, but the subsequent request has auth: null. Is this a node 6 breaking change? A bug in follow-redirects? It looks like this change could be related, but as per the logs above, the host is not changing.

Drop 'Content-Length' after re-assigning method to GET?

Got some problems on redirecting from a 302 POST request, basically the server refused to send back any response. After some investigation, I found it was caused by the Content-Length header, which probably makes the server think it's an invalid GET request.

My current makeshift is deleting the Content-Length field from this._options.headers for all GET requests inside _processResponse. Any better solution?

Agent, cookies, timeout

Hi, I had a few questions:

  1. If both agent (nodejs http/https option) and agents (follow-redirects option) are passed, then will the agent option be used ever in the main or redirected requests?

  2. It seems cookies set with a 302 redirect (i.e. having both location and set-cookie headers) are not passed over to the following URL? For example, this httpbin test URL: https://httpbin.org/cookies/set?k1=v1&k2=v2 should set two cookies, and do a 302 redirect to https://httpbin.org/cookies, which simply emits the cookie values, but it shows none.

  3. Looks like the request timeout is applied per redirection, not as an overall measure, which is somewhat confusing. For example, if timeout is set to 2000 ms, and 5 redirections occurred taking a total of 5000 ms, then no error is thrown, although it should. I think the timeout value (just like the native http/s modules) should be applied to the whole request.

Cheers.

Design pattern for caching redirected responses

In the NodeJS framework, I’d like to create a user agent (outside the browser) that also properly implements HTTP client caching. Problem with this library is that it handles redirects itself, and omits the cache in front of follow-redirects.

E.g., in pseudo javascript code:

var url = 'http://data.linkeddatafragments.org/bwv/classifications/10'
if (!cache.get(url)) {
    http.get(url, function (...) {
        var url2 = 'http://data.linkeddatafragments.org/bwv?subject=http%3A%2F%2Fdata.linkeddatafragments.org%2Fbwv%2Fclassifications%2F10';
        if (!cache.get(url2)) {
             http.get(url2);
        }
    });
}

This example will twice download the exact same page while the cached response should be given.

Suggested solutions

1. A flag

When the flag is set, the library will not automatically follow redirects, but will also allow for returning the new location URL, for which an external cache can then check what to do: resolve the promise, or again request it via follow-redirects.

2. Also implement an (in-memory) caching system in follow-redirects

Also allow a cache instance to be given to the follow-redirects library.

3. Document how follow-redirects can be used as one part of the chain

Inspired by express middleware design pattern, we could also document chaining HTTP request libraries, in which you can put caches behind follow-redirects without issues.

Piping into request stream is broken

The following code will hang:

var req = http.request(opts, callback);
var stream = fs.createReadStream(filename);
stream.pipe(req);

Please note that the same code works fine with v0.0.7. Looks like the issue was introduced by #31. Removing the code that creates a request proxy resolves the problem.

I created a test that reproduces this issues. I will submit it as a PR.

Thank you!

responseUrl not correct when using "request" method

Description

When a request is done using the request method, and when no redirect was made, the
response.responseUrl will only contain the base url and not the full url with the path included.

Reproduce

var http = require('follow-redirects').http;
var URLParser = require('url');
// Does not work
 var parsedUrl = URLParser.parse('http://graph.spitsgids.be/connections/?departureTime=2017-06-12T11%3A00');
var settings = { hostname: parsedUrl.hostname, port: parsedUrl.port, path: parsedUrl.path, method: 'GET'};
http.request(settings, function (response) {
  console.log(response.responseUrl);
}).end();

// Works as it should:
http.get('http://graph.spitsgids.be/connections/?departureTime=2017-06-12T11%3A00', function (response) {
  console.log(response.responseUrl);
}).on("error", function (err) {
  console.error(err);
});

// Works as it should: (request to a URL that gets redirected)
 var parsedUrl = URLParser.parse('http://graph.spitsgids.be/connections/?departureTime=2017-06-12T11%3A01');
var settings = { hostname: parsedUrl.hostname, port: parsedUrl.port, path: parsedUrl.path, method: 'GET'};
http.request(settings, function (response) {
  console.log(response.responseUrl);
}).end();

Error observed while testing w/ nock

I'm getting the following error when running a test that uses nock.

TypeError: Cannot read property '_redirectable' of undefined
      at eventHandlers.(anonymous function) (node_modules/follow-redirects/index.js:22:7)
      at OverriddenClientRequest.RequestOverrider.req.once.req.on (node_modules/nock/lib/request_overrider.js:195:7)
      at Writable.RedirectableRequest._performRequest (node_modules/follow-redirects/index.js:69:12)
      at Writable.RedirectableRequest (node_modules/follow-redirects/index.js:46:7)
      at Object.wrappedProtocol.request (node_modules/follow-redirects/index.js:202:10)
      at Object.wrappedProtocol.get (node_modules/follow-redirects/index.js:207:33)
      at getLatestUrl (lib/update.js:21:13)
      ...

The error doesn't happen when I run the program regularly. See this travis build for more context.

Use same request method when following 307 status

As per RFC:

307 Temporary Redirect

The 307 (Temporary Redirect) status code indicates that the target resource resides temporarily under a different URI and the user agent MUST NOT change the request method if it performs an automatic redirection to that URI. Since the redirection can change over time, the client ought to continue using the original effective request URI for future requests.

The server SHOULD generate a Location header field in the response containing a URI reference for the different URI. The user agent MAY use the Location field value for automatic redirection. The server's response payload usually contains a short hypertext note with a hyperlink to the different URI(s).

Note: This status code is similar to 302 (Found), except that it does not allow changing the request method from POST to GET. This specification defines no equivalent counterpart for 301 (Moved Permanently) ([RFC7238], however, defines the status code 308 (Permanent Redirect) for this purpose).

Right now GET is always being used.

function defaultMakeRequest(options, cb, res) {
  if (res) {
    // This is a redirect, so use only GET methods
    options.method = 'GET';
  }

  var req = options.nativeProtocol.request(options, cb);

  if (res) {
    // We leave the user to call `end` on the first request
    req.end();
  }

  return req;
}

Buffer request bodies?

Looking at the HTTP spec, I think it is entirely possible that the server will consume at least part of the request body before issuing a 307 redirect. When that happens, we are unable to redirect because whatever portion of the stream has been consumed is lost to us.

Perhaps we should buffer the request body when it's a stream (at least to a point).

"Bug" when switching methods

I spent half a day tracking this down. I'm not sure if this qualifies as a bug, but if you accept my suggested fix, I'll submit a PR.

The problem I was having was that one of my tests on my project was failing (timing out, specifically). Needless to say, there were many layers above follow-redirects to comb through for the error. But I simply couldn't find anything. The problem I was seeing was that when my server returned a redirect, my client would hang. Hitting the server with a browser showed a proper Location header and status code. But the client (using axios) just would not return. It was making the request and the server was sending back a response, but the promise in Axios was never resolved. I presumed that the redirect was being processed, but it never seemed to get a response.

So I rolled up my sleeves and tried to track down exactly where the process was breaking down. My original request was a POST and follow-redirects was converting that to a GET. That by itself is what I expected, so that was no problem. I could change my original method to take a GET instead and then the problem went away. So that fixed my problem (tests ran to completion), but it really bugged me that I didn't understand why this wasn't working and I didn't want to just leave it at that and just hope not to run into it again.

It clearly had something to do with redirects, so I turned on debugging output with DEBUG=follow-redirects and took a look at what was going on. I could see follow-redirects making the redirect request. I assumed, incorrectly, that perhaps when a POST request was made, the POST method was used on the redirect (that would have cause an error). But that wasn't the issue. In fact, if that were the case the request should have completed (which wasn't happening). This provides a subtle clue into what actually was the problem.

I compared the options being passed in both the working and non-working cases. There were only two differences in those requests and they were related to headers. This is because my original request (which came through axios had headers normally associated with a POST in them. Specifically Content-Type and Content-Length. When follow-redirects switches the method to GET in its 307 handling, it leaves those there. So the resulting redirected GET request has a Content-Length header in it. Since a GET request has no body, this makes no sense.

I noticed that with the Chrome extension DHC, if I make a GET request with a Content-Length header, it removes it automatically. But it appears what is happening is that follow-redirects/http are keeping that header in there and when it hits the Express server, the express server sits and waits until it gets all the bytes for the body. But, of course, there is no body.

What I did to fix this was to simply change this:

	if (!(this._options.method in safeMethods)) {
		this._options.method = 'GET';
	}

...to this...

	if (!(this._options.method in safeMethods)) {
		this._options.method = 'GET';
                // Remove a Content-Length header if one is present
		delete this._options.headers["Content-Length"];
	}

This seems perfectly reasonable to me and it is very much inline with what DHC does. But I wanted to first check here to see if a) this seems like a reasonable approach and b) whether you want me to create a PR for this.

I can't see what harm this would do, but you guys know more about the HTTP implications of this than I do. For that matter, there may be other headers to consider here as well (i.e., that would never apply to a GET and therefore should be removed).

Redirect history

would be nice. Either add it to the response object or add a second argument to the callback function.

Content-Type header is dropped

Using axios, I discovered this is the underlying library that handles redirects, so 👋 .

The API I'm communicating with follows JSON API which describes that clients must add Content-Type: application/vnd.api+json headers to their requests.

I have noticed though that when performing a redirect, this library will remove any header that starts with content-*.

This results in my requests failing, with a 415 Unsupported Media Type response (as per spec).

Just wondering what the intentions were behind this? Is there a workaround to "reinsert" headers such as Content-Type later?

Thanks!

[Question] Query regarding Fork License

I am making a fork of this project for use with IoTjs (which is like a light version of nodejs for embedded environments).
This fork may be used fo rcommercial purposes.
Going by the LICENSE file in the repo, its is perfectly acceptable for me to do so as long as I keep original LICENSE file right?

https.request() do no get called when Content-length specified in options.

I would like to know what is wrong in this code.. I don't https.Request() gets called when I remove content-length.

setProperty: function(name, value) {
var deferred = q.defer(); // Take a deferral

    var postData = JSON.stringify({name : value});
    var options = {
        protocol: 'https:',
        host: 'developer-api.nest.com',
        path: '/devices/thermostats/' + deviceId,
        headers: {
            'Authorization': 'Bearer ' + accessToken,
            'Content-Type': 'application/json',
            'Accept': 'application/json',
            'Content-length': Buffer.byteLength(postData),
        },
        method: 'PUT',
    };

    var httpsRequest = https.request(options, (getRes) => {
        console.log("Request sent");
        var body = '';
        getRes.setEncoding('utf8');
        getRes.on('data', (data) => {
            body += data;
        });

        getRes.on('end', () => {
            if (getRes.statusCode != 200) {
                deferred.reject("Error Code:" + getRes.statusCode);
            } else {
                var data = JSON.parse(body);
                deferred.resolve(target_temperature_c);
            }
        });

        getRes.on('error', (e) => {
            deferred.reject(e);
        });
    });
    httpsRequest.write(postData);
    httpsRequest.end();

    return deferred.promise; // return the promise
},

1.0.0 release

@olalonde

I think it is time for a 1.0.0 release.

It gives us the ability to publish updates in a semver responsible way.

I have added a couple issues I want to close before then to the 1.0.0 milestone. Once those are done, I think we should do it.

Headers are not being passed when doing the redirects

I'm working on proxy for downloading files from Google Docs. Since the urls Google Drive API gives for non-gdrive filetypes (pdf, txt, jpeg) are redirects, I decided to use your library. So when you hit http://localhost:3000/oauth2_download?accessToken=x&url=my_url where x is my GDrive access_token, url is, say, https://docs.google.com/uc?id=0B_31K_MP92hUNjljYjIyMzgtZTBmNS00MGMwLWIxNmQtYjMyNDFiYjY0MTJl&export=download, the file is supposed to be downloaded. However, the browser sends me to the Google Docs sign-in page which (given that hitting https://docs.google.com/uc?id=0B_31K_MP92hUNjljYjIyMzgtZTBmNS00MGMwLWIxNmQtYjMyNDFiYjY0MTJl&export=download in browser downloads the file if you're logged into Google Docs) means the headers are not being sent correctly. Any idea how should I use 'follow-redirects' library to deal with this issue? Thank you in advance!

//var https = require('https');
var https = require('follow-redirects').https;
var app = require('express')();
var urllib = require('url');
// Downloads the file using OAuth2. Tested on Google Drive
app.get('/oauth2_download', function(req, res){
    var url = req.param('url');
    var accessToken = req.param('accessToken');
    var headers = { "Authorization": "Bearer " + accessToken }
    var searchname = urllib.parse(url).search;
    var hostname = urllib.parse(url).hostname;
    var pathname = urllib.parse(url).pathname;
    if (url && accessToken) {
        // downloading file with streaming
        var options = {
            hostname: hostname,
            port: 443,
            path: pathname + searchname,
            method: 'GET',
            headers: headers
        };
        https.request(options, function(response){
            for (var key in response.headers) {
                res.setHeader(key, response.headers[key]);
            }
            response.on('data', function(chunk) {
                res.write(chunk);
            });
            response.on('end', function() {
                res.end();
            });
            //console.log("Done downloading using OAuth2");
        }).on('error', function(e) {
            console.error(e);
        }).end();
    } else {
        res.send(404);
        res.end();
    }
});

Object #<Object> has no method 'request'

When trying to use this in Mashery's I/O Docs I see the following error:

TypeError: Object # has no method 'request'
at /home/warp/code/iodocs/node_modules/follow-redirects/index.js:60:40
at unsecuredCall (/home/warp/code/iodocs/app.js:568:23)
at processRequest (/home/warp/code/iodocs/app.js:481:9)
at callbacks (/home/warp/code/iodocs/node_modules/express/lib/router/index.js:272:11)

etc...

https://github.com/warpr/iodocs/blob/coverartarchive/app.js#L564

Do you have any idea if this Is a bug, or am I doing something wrong?

drop underscore as dependency

it is only using _.extend.

It could be replaced by merge. merge is less than 1kB when minified (as opposed to 17 from underscore).

It might be better to just write an implementation ourselves though. merge has a recursive option which we do not need.

project question

Hey there,

I like your approach to plug in at a low level with http and https. I want to take the same approach for client http caching. I've spent a fair amount of time searching and didn't find any existing module, apart from cache-http which has a weird API and only look at If-Modified-Since.

My question to you is are you aware of a library I may have missed and is there anything like connect but for http clients?

Use same request method when following 308 status

Pretty much the same as Issue #27, HTTP 308 redirects should also use the same request method rather than changing the method to GET.

Per RFC 7238:

308 Permanent Redirect

The 308 (Permanent Redirect) status code indicates that the target resource has been assigned a new permanent URI and any future references to this resource ought to use one of the enclosed URIs.

Clients with link editing capabilities ought to automatically re-link references to the effective request URI (Section 5.5 of [RFC7230]) to one or more of the new references sent by the server, where possible.

The server SHOULD generate a Location header field ([RFC7231], Section 7.1.2) in the response containing a preferred URI reference for the new permanent URI. The user agent MAY use the Location field value for automatic redirection. The server's response payload usually contains a short hypertext note with a hyperlink to the new URI(s).

A 308 response is cacheable by default; i.e., unless otherwise indicated by the method definition or explicit cache controls (see [RFC7234], Section 4.2.2).

Note: This status code is similar to 301 (Moved Permanently) ([RFC7231], Section 6.4.2), except that it does not allow changing the request method from POST to GET.

Create optimal cross-platform development experience via browserify.

I think follow-redirects default behavior should be to mimic the behavior of the http and https as used in the browser via browserify. XMLHtppRequest follows redirects transparently, and there is no way to disable that. Therefore the browserified versions of http and https automatically have a follow-redirects like behavior, but the server side counterparts do not (obviously, there would be little point to this module otherwise). However, this behavior breaks one of the key goals of browserify; the ability to write code that behaves identically on the client and server. This isn't the fault of browserify, due to the way XMLHttpRequest behaves they can't possibly emulate the behavior of the native modules.

I am definitely not suggesting we turn follow-redirects into a client side library, but instead help the server behave more like the client. This is backwards from the goal of most browserify modules, but as previously stated, it's just not possible to emulate the server in this case. Developers seeking to create a cross platform solution would use follow-redirects on the server, and have browserify automatically swap it out with browserify-http when creating the client bundle. Indeed, I have already detailed instructions for doing just that in the README.

We are already very close. For many scenarios the library will work as is. However, there are still a few points to be addressed.

  1. Issue #12 needs to be fixed. Specifically, our current logic for deciding when to redirect is pretty basic. It's not always OK to redirect for every 300 level code. The spec notes many scenarios where typical browser behavior is actually not to spec. Rather than strictly conforming to the spec, we should follow whatever the most common behavior is.
  2. Also related to #12, we currently always perform a GET on redirection, even if the original request was a POST or otherwise. This is the correct response in most situations, but (I believe) not all.
  3. Handling errors. Currently we emit an error on the returned client request object if we exceed the maximum number of redirects. We also forward any errors we encounter on subsequent redirected requests to the original one. If we encounter a 3XX status code without a Location header, we do not throw an error, but return the result to the registered callback. This all makes logical sense, but we should work to make our behavior the same as the client side modules.
  4. Make all this configurable. There is already an (optional and as yet undocumented) makeRequest callback that allows us to change these behaviors on master. It's probably wise to keep a "legacy" version of that callback around that continues to mimic the behavior of the library as it has existed for years.
  5. We should do all this while still maintaining API compatibility with the core modules (nodes native http and browserify-http, etc). Plenty of cross-platorm options exist if you are willing to deal with a different API. The goal here should be an easy transitions for those using the core modules.
  6. Be as lightweight on the client side as possible. Ideally, this means not existing at all on the client, and instead differing to the core browserify implementations that are likely already bundled.

Wrong HTTP status code?

Someone sent me this email and I referred them to this issue:

Hi James
I have been using "follow-redirects" for making ReST calls and its very good. Thanks for the wonderful package. I am however stuck up when the site does a redirect. In that case, something very strange happens. Instead of getting a 307, for some very weird reason I keep getting 400 response.

Now I know this sounds very crazy as its the server which sends the response, and not the client, but whenever I use the normal "http" package, in my client code, instead of using the "follow-redirects", I get a 307 properly. Only for "follow-redirects" do I get a 400.

So is there any additional step that I need to do, to get this resolved.

http.get never ends

I'm using node v0.10.12 and I'm quite new to this.

This little program doesn't end using follow-redirects but it completes using the native http module:

//var http = require('http');
var http = require('follow-redirects').http;

http.get('http://bit.ly/900913', function (res) {
  var all = '';
  res.on('data', function (chunk) {
    all = all + chunk;
    console.log(all.toString().length);
  });

  res.on('end', function (chunk) {
    console.log(all.toString().length);
  });
}).on('error', function (err) {
  console.error(err);
});

Am I missing something obvious? Or maybe follow-redirects needs to be updated to node 0.10? Any hint would be appreciated. I could slowly take a stab at updating the module if that's indeed the problem.

I wanted to take a stab at #6 but I have to figure this out first.

Limit upload size to avoid exceeding memory limits

It looks to me like if I e.g. do a file upload using this module (i.e. it won't be redirected) the entire file would be buffered in memory? If I have a file of several GB this could be a problem? Should we warn for this and possibly add a method + "content-length" check?

Object.assign is not a function

Hi there,

I use node-rest-client module which uses your module in dependance.
When I wan't to make a HTTP request, it returns an error Object.assign is not a function from your module (index.php lines 169 and 134).

My app is under Node 0.10.44 (I can't upgrade).

Please fix it asap (you can find a great polyfill on MDN or for NPM)

Host && Port will be overwritten after running url parsing

An unexpected problem occurred during redirect url parsing:
https://github.com/olalonde/follow-redirects/blame/master/index.js#L145

My definitions as follows :

let options = {
    host: 'proxy-host.com',
    port: 8080,
    method: 'GET',
    path: 'http://download.image.com/test.jpg',
    followRedirect: true,
    maxRedirects: 3
 };

This problem has led options.host and options.port failure after running redirect url passing. The new Host and Port in redirect url will be re-assigned to my options when I am using Host and Port as a proxy server.

License

Any chance you would consider changing the license to MIT or apache 2?

BSD requires attribution in any marketing done for a product build with this module.

HEADs converted to GETs on relayed requests

HEADs where being converted to GETs, meaning HEADing a relayed server results in the entire payload being dumped unceremoniously on the requester.

I also had an issue with being stuck in an endless relay loop.

my quick and dirty hack to make it work can be cut and pasted to the top of any jsnode file, (don't worry with npm or other bollocks, just have assert,debug and consume installed.

enjoy.

'use strict';

var assert = require('assert');
var debug = require('debug')('follow-redirects');
var consume = require('stream-consume');
// follow-redirects is taken and butchered from code obtained using
// "npm install follow-redirects"
// ie: https://www.npmjs.com/package/follow-redirects
// it had enough bugs in it to warrant fixing, but
// it's not worth doing
var followRedirects = (function (_nativeProtocols) {

  var nativeProtocols = {};

  var publicApi = {
    maxRedirects: 5
  };

  for (var p in _nativeProtocols) {
    /* istanbul ignore else */
    if (_nativeProtocols.hasOwnProperty(p)) {
      // http://www.ietf.org/rfc/rfc2396.txt - Section 3.1
      assert(/^[A-Z][A-Z\+\-\.]*$/i.test(p), JSON.stringify(p) + ' is not a valid scheme name');
      generateWrapper(p, _nativeProtocols[p]);
    }
  }

  return publicApi;

  // to redirect the result must have
  // a statusCode between 300-399
  // and a `Location` header
  function isRedirect(res) {
    return (res.statusCode >= 300 && res.statusCode <= 399 &&
    'location' in res.headers);
  }


  function execute(options, callback) {
    var req_method=options.method;
    var fetchedUrls = [];
    var clientRequest = cb();

    // return a proxy to the request with separate event handling
    var requestProxy = Object.create(clientRequest);
    requestProxy._events = {};
    requestProxy._eventsCount = 0;
    if (callback) {
      requestProxy.on('response', callback);
    }
    return requestProxy;

    function cb(res) {
      // skip the redirection logic on the first call.
      if (res) {
        var fetchedUrl = url.format(options);
        fetchedUrls.unshift(fetchedUrl);

        if (!isRedirect(res)) {
          res.fetchedUrls = fetchedUrls;
          requestProxy.emit('response', res);
          return;
        }

        // we are going to follow the redirect, but in node 0.10 we must first attach a data listener
        // to consume the stream and send the 'end' event
        consume(res);

        // need to use url.resolve() in case location is a relative URL
        var redirectUrl = url.resolve(fetchedUrl, res.headers.location);
        debug('redirecting to', redirectUrl);

        options=url.parse(redirectUrl);
      }

      if (fetchedUrls.length > options.maxRedirects) {
        var err = new Error('Max redirects exceeded.');
        return forwardError(err);
      }

      options.nativeProtocol = nativeProtocols[options.protocol];
      options.defaultRequest = defaultMakeRequest;

      var req = (options.makeRequest || defaultMakeRequest)(options, cb, res);
      req.on('error', forwardError);
      return req;
    }

    function defaultMakeRequest(options, cb, res) {
      if (res && res.statusCode !== 307) {
        // This is a redirect, so use only GET methods, except for status 307,
        // which must honor the previous request method.
        options.method = 'GET';
      }

      if (req_method === 'HEAD') 
        options.method=req_method;

      var req = options.nativeProtocol.request(options, cb);

      if (res) {
        // We leave the user to call `end` on the first request
        req.end();
      }

      return req;
    }

    // bubble errors that occur on the redirect back up to the initiating client request
    // object, otherwise they wind up killing the process.
    function forwardError(err) {
      requestProxy.emit('error', err);
    }
  }

  function generateWrapper(scheme, nativeProtocol) {
    var wrappedProtocol = scheme + ':';
    var H = function () {};
    H.prototype = nativeProtocols[wrappedProtocol] = nativeProtocol;
    H = new H();
    publicApi[scheme] = H;

    H.request = function (options, callback) {
      return execute(parseOptions(options, wrappedProtocol), callback);
    };

    // see https://github.com/joyent/node/blob/master/lib/http.js#L1623
    H.get = function (options, callback) {
      var req = execute(parseOptions(options, wrappedProtocol), callback);
      req.end();
      return req;
    };
  }

  // returns a safe copy of options (or a parsed url object if options was a string).
  // validates that the supplied callback is a function
  function parseOptions(options, wrappedProtocol) {
    if (typeof options === 'string') {
      options = url.parse(options);
      options.maxRedirects = publicApi.maxRedirects;
    } else {
      options.maxRedirects = publicApi.maxRedirects;
      options.protocol     = wrappedProtocol;
    }
    assert.equal(options.protocol, wrappedProtocol, 'protocol mismatch');

    debug('options', options);
    return options;
  }
}) ({
  http: require('http'),
  https: require('https')
});
followRedirects.maxRedirects = 10; 
var  http  = followRedirects.http,
     https = followRedirects.https;

Node v0.12.2 connections don't close

Connections don't close on node v0.12.2 due to this line. The node server will eventually stop responding. In a production app I fixed it by commenting the line out.

Maybe putting a conditional for node version would be a better solution?

Thanks!

How to get html page contents instead of <Buffer ...

I am trying below piece of code as you said.

var http = require('follow-redirects').http;
var https = require('follow-redirects').https;

var opts = {
		host: "bitly.com",
		path:"/UHfDGO", 
		port: 80};

http.get(opts, function (response) {
  response.on('data', function (chunk) {
    console.log(chunk);
  });
}).on('error', function (err) {
  console.error(err);
});

But when I execute it I get

vikas@vikas-pc:~/NodeJS Practice$ node trialnerror.js 
<Buffer 55 73 65 72 2d 61 67 65 6e 74 3a 20 2a 0a 44 69 73 61 6c 6c 6f 77 3a 20 2f 6c 69 74 65 0a 44 69 73 61 6c 6c 6f 77 3a 20 2f 68 74 6d 6c 0a 0a 23 20 4e ... >

Instead, I need to get entire html page content

Help required for Redirect URL to return response.headers when the status code is 302

Hi,
I was googling about an issue which i'm facing at my end about the Redirect URL and found your blog "http://syskall.com/how-to-follow-http-redirects-in-node-dot-js/" . Indeed it is an useful blog. However the problem at my end was

When I use the API end points manually with Postman (with interceptors turned ON) along with body and headers, I was able to get 302 redirects. However for my API Automation testing purpose I thought of using

request() // npm request

which is as follows. But always I get the response as 500. My objective of my test is to get the response.headers when I get the status code as 302 from response. Can you pls throw some light and provide me some direction as I got struck on this

request(
url : 'https://blah/blah'
client_id :'something.com.au'
redirect_uri :'http://something' // this is a dummy one as it is not required for mobile app
response_type :'code'
scope : 'ALLAPI'
Username : 'cxxx'
Password : 'ccxcxc'
headers : {
'Content-Type': 'application/x-www-form-urlencoded',
'followRedirect' : true
}
Thanks,
Brad

Write after end when piping

I have a fairly simple case where I'm building a tarball-stream on the fly and piping it to a request that's going through follow-redirects.

It almost immediately crashes with this error:

Error: write after end
    at ClientRequest.OutgoingMessage.write (_http_outgoing.js:439:15)
    at Writable.RedirectableRequest._write (/home/espenh/webdev/sanity/cli/node_modules/follow-redirects/index.js:170:23)
    at doWrite (_stream_writable.js:333:12)
    at clearBuffer (_stream_writable.js:440:7)
    at onwrite (_stream_writable.js:372:7)
    at WritableState.onwrite (_stream_writable.js:89:5)
    at CorkedRequest.finish (_stream_writable.js:547:7)
    at afterWrite (_stream_writable.js:387:3)
    at onwrite (_stream_writable.js:378:7)
    at TLSSocket.WritableState.onwrite (_stream_writable.js:89:5)

I'll see if I can find time to write a reduced test case at some point, but don't have time right at this moment. Switching to regular HTTP/HTTPS for this request fixes the problem.

Set-cookie

Not taken into account set-cookie from the server

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.