Comments (32)
@mostafadahab thanks for pointing out the error. I'm not understanding what you mean by "isRedirect" not working. Goreq isRedirect method is tested here: https://github.com/franela/goreq/blob/master/goreq_test.go#L605 and it seems to be doing what it's supposed to do. Can you please provide an example where it's not doing what it should do?.
One thing I see is that Goreq MaxRedirects is initialized with a zero value whereas the link that you sent says that Golang default http client supports up to 10 redirects. Is that what you're refering?
Thanks.
from goreq.
thank you very much for quick response !
in the past i build full http bot in python using this http://docs.python-requests.org and that was fast and helpful ,, but i want to change to golang for fast and less of ram usage .. when i build first function see whats happens :
this is the code and this is result :
it's follow 2 redirects i don't want it to follow redirects
and i'am try using Is MaxRedirects 1 or -1
and it's the same thing
this is not the all ! in the redirects all of the headers i was set it's gone ! :
no support for cookie jar and headers are gone ,, thanks agine 👍
from goreq.
@mostafadahab can you please paste some code here and the compelte URL you're trying to access so we can give it a shot?.
Thanks.
from goreq.
package main
import (
"fmt"
"./goreq"
)
func main() {
/*for {
res, err := goreq.Request{ Uri: "http://www.google.com" ,Timeout:60 * time.Second}.Do()
if err != nil {
continue
}
fmt.Print(res.Body.ToString())
break
}*/
amflogin("","")
/*req:
res, err := goreq.Request{ Uri: "http://www.google.com" ,Timeout:60 * time.Second}.Do()
if err != nil {
goto req
}
fmt.Print(res.StatusCode)*/
}
func amflogin(email,password string) (points string , err error){
req := goreq.Request{Method: "POST",MaxRedirects: -1,Proxy:"http://127.0.0.1:8888",Uri: "http://addmefast.com",ContentType:"application/x-www-form-urlencoded",Body: "[email protected]&password=abc123abc123&login_button=Login",Accept:"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",UserAgent:"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36"}
res,err := req.Do()
if err != nil {
fmt.Println(err.Error())
}
fmt.Println(res.Body.ToString())
return "test",nil
}
from goreq.
@mostafadahab I'm looking into is as I've noticed that something is strange with the site that you sent me. If you run the request against www.google.com (per your example) or some other site that returns 302 then by default Goreq won't follow redirects unless MaxRedirects is specified.
However, for the POST that you're sending I'm seeing that Golang is following the redirect either way. I'm looking into it and I'll answer back shortly
from goreq.
i think it's about this code in client.go file :
CheckRedirect specifies the policy for handling redirects.
42 // If CheckRedirect is not nil, the client calls it before
43 // following an HTTP redirect. The arguments req and via are
44 // the upcoming request and the requests made already, oldest
45 // first. If CheckRedirect returns an error, the Client's Get
46 // method returns both the previous Response and
47 // CheckRedirect's error (wrapped in a url.Error) instead of
48 // issuing the Request req.
49 //
50 // If CheckRedirect is nil, the Client uses its default policy,
51 // which is to stop after 10 consecutive requests.
52 CheckRedirect func(req *Request, via []*Request) error
from goreq.
@mostafadahab I've found the error and fixed it here: fa5798b. Amazingly seems like a bug in Golang http client that by default it doesn't follow redirects when Method is not explicitly set in .Do
function.
http://golang.org/src/pkg/net/http/client.go?s=5094:5155#L138 checks for Method explicitly and performs redirects based on that. As Goreq uses the .Do
method of http.Client and sets the http method depending on Request parameter, that's why MaxRedirects didn't work.
Expect this PR to be merged today once a collaborator reviews it.
from goreq.
Great Job !! 👍
but when redirect why it's remove headers i was set !! like UserAgent replaced with default golang User Agent and automatically add Referer Header ! see : http://golang.org/src/pkg/net/http/client.go#L310
from goreq.
@mostafadahab that has been fixed in this commit also: c7a1e8a can you please checkout the redirect_fix branch and give it a shot with your test scenario?
from goreq.
i have tested the new code same problem !
from goreq.
@mostafadahab this is the test i'm running
package main
import (
"fmt"
"github.com/franela/goreq"
)
func main() {
req := goreq.Request{
Method: "POST",
Uri: "http://addmefast.com",
ContentType: "application/x-www-form-urlencoded",
Body: "[email protected]&password=abc123abc123&login_button=Login",
Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
UserAgent: "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36",
}
res, _ := req.Do()
body, _ := res.Body.ToString()
fmt.Println(body, res.StatusCode, res.Header)
}
and this is the output i'm getting:
302 map[Date:[Tue, 02 Dec 2014 21:37:24 GMT] Content-Type:[text/html; charset=utf-8] Set-Cookie:[__cfduid=d00acd0cfb14fd6df7a193a1e9a3dd75b1417556243; expires=Wed, 02-Dec-15 21:37:23 GMT; path=/; domain=.addmefast.com; HttpOnly PHPSESSID=6pggm4lip9errvh631jup00191; path=/; domain=.addmefast.com SERVERID=adasd2cb8blsl234werasddfb85a13qw5; path=/] Server:[cloudflare-nginx] Connection:[keep-alive] Last-Modified:[Tue, 02 Dec 2014 21:37:24 GMT] Expires:[Thu, 19 Nov 1981 08:52:00 GMT] Cf-Ray:[192aaa5cdd9d097b-SCL] Cache-Control:[no-store, no-cache, must-revalidate, post-check=0, pre-check=0] Pragma:[no-cache] Location:[/free_points]]
As you can see redirect is not being handled (because MaxRedirects is not specified) and I'm getting the corresponding request headers.
What it's not working for you?
from goreq.
Very strange !! i discover the problem was in proxy !! when remove proxy no redirects follow but when adding proxy the redirect working !!
from goreq.
i have fix the error by rise this code
if r.Proxy != "" {
proxyUrl, err := url.Parse(r.Proxy)
if err != nil {
// proxy address is in a wrong format
return nil, &Error{Err: err}
}
if proxyTransport == nil {
proxyTransport = &http.Transport{Dial: defaultDialer.Dial, Proxy: http.ProxyURL(proxyUrl)}
proxyClient = &http.Client{Transport: proxyTransport}
} else {
proxyTransport.Proxy = http.ProxyURL(proxyUrl)
}
transport = proxyTransport
client = proxyClient
}
before this code
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
if len(via) > r.MaxRedirects {
redirectFailed = true
return errors.New("Error redirecting. MaxRedirects reached")
}
return nil
}
to become the full Do func :
func (r Request) Do() (*Response, error) {
var req *http.Request
var er error
var transport = defaultTransport
var client = defaultClient
var redirectFailed bool
r.Method = valueOrDefault(r.Method, "GET")
if r.Proxy != "" {
proxyUrl, err := url.Parse(r.Proxy)
if err != nil {
// proxy address is in a wrong format
return nil, &Error{Err: err}
}
if proxyTransport == nil {
proxyTransport = &http.Transport{Dial: defaultDialer.Dial, Proxy: http.ProxyURL(proxyUrl)}
proxyClient = &http.Client{Transport: proxyTransport}
} else {
proxyTransport.Proxy = http.ProxyURL(proxyUrl)
}
transport = proxyTransport
client = proxyClient
}
client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
if len(via) > r.MaxRedirects {
redirectFailed = true
return errors.New("Error redirecting. MaxRedirects reached")
}
return nil
}
if r.Insecure {
transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
} else if transport.TLSClientConfig != nil {
// the default TLS client (when transport.TLSClientConfig==nil) is
// already set to verify, so do nothing in that case
transport.TLSClientConfig.InsecureSkipVerify = false
}
b, e := prepareRequestBody(r.Body)
if e != nil {
// there was a problem marshaling the body
return nil, &Error{Err: e}
}
if r.QueryString != nil {
param, e := paramParse(r.QueryString)
if e != nil {
return nil, &Error{Err: e}
}
r.Uri = r.Uri + "?" + param
}
var bodyReader io.Reader
if b != nil && r.Compression != nil {
buffer := bytes.NewBuffer([]byte{})
readBuffer := bufio.NewReader(b)
writer, err := r.Compression.writer(buffer)
if err != nil {
return nil, &Error{Err: err}
}
_, e = readBuffer.WriteTo(writer)
writer.Close()
if e != nil {
return nil, &Error{Err: e}
}
bodyReader = buffer
} else {
bodyReader = b
}
req, er = http.NewRequest(r.Method, r.Uri, bodyReader)
if er != nil {
// we couldn't parse the URL.
return nil, &Error{Err: er}
}
// add headers to the request
req.Host = r.Host
req.Header.Add("User-Agent", r.UserAgent)
req.Header.Add("Content-Type", r.ContentType)
req.Header.Add("Accept", r.Accept)
if r.Compression != nil {
req.Header.Add("Content-Encoding", r.Compression.ContentEncoding)
req.Header.Add("Accept-Encoding", r.Compression.ContentEncoding)
}
if r.headers != nil {
for _, header := range r.headers {
req.Header.Add(header.name, header.value)
}
}
//use basic auth if required
if r.BasicAuthUsername != "" {
req.SetBasicAuth(r.BasicAuthUsername, r.BasicAuthPassword)
}
timeout := false
var timer *time.Timer
if r.Timeout > 0 {
timer = time.AfterFunc(r.Timeout, func() {
transport.CancelRequest(req)
timeout = true
})
}
res, err := client.Do(req)
if timer != nil {
timer.Stop()
}
if err != nil {
if !timeout {
switch err := err.(type) {
case *net.OpError:
timeout = err.Timeout()
case *url.Error:
if op, ok := err.Err.(*net.OpError); ok {
timeout = op.Timeout()
}
}
}
var response *Response
//If redirect fails we still want to return response data
if redirectFailed {
response = &Response{StatusCode: res.StatusCode, ContentLength: res.ContentLength, Header: res.Header, Body: &Body{reader: res.Body}}
}
return response, &Error{timeout: timeout, Err: err}
}
if r.Compression != nil && strings.Contains(res.Header.Get("Content-Encoding"), r.Compression.ContentEncoding) {
compressedReader, err := r.Compression.reader(res.Body)
if err != nil {
return nil, &Error{Err: err}
}
return &Response{StatusCode: res.StatusCode, ContentLength: res.ContentLength, Header: res.Header, Body: &Body{reader: res.Body, compressedReader: compressedReader}}, nil
} else {
return &Response{StatusCode: res.StatusCode, ContentLength: res.ContentLength, Header: res.Header, Body: &Body{reader: res.Body}}, nil
}
}
from goreq.
@mostafadahab I believe I know why it failed when setting a proxy. Even though raising the code worked for you I believe it's not the best solution. Let me code a test to reproduce the scenario and I'll fix the code in a couple of mins.
from goreq.
Do it man nice work ;) 👍 ,, but the issue "removing headers like UserAgent etc.." still there !
if that form default go client ??
from goreq.
@mostafadahab I've been looking for the headers issue and yes, it seems like it's something from the go client. There's an issue reported here: https://code.google.com/p/go/issues/detail?id=4800&q=redirect&colspec=ID%20Status%20Stars%20Release%20Owner%20Repo%20Summary.
However, I believe that we can do something on our end to implement it. Let me investigate a bit more and I'll get back to you shortly
from goreq.
You seem like be doing a great work !!
2 small things
- the cookie header is not unchangeable value it's changed by the server and the best way to implement this by using cookiejar : http://golang.org/pkg/net/http/cookiejar/
- how i can support this project to grow is there any why like donate or something like that
thanks in advance
from goreq.
@mostafadahab the proxy redirect issue is already fixed. I'll be adding the cookie and header redirect support later today.
Regarding to contributions, you're already doing a great job by using our library and spotting bugs which will make it better for some other people. We're doing this just to return something to the open source community.
from goreq.
@mostafadahab header forward when redirect has been added here: 3478220
Yo need to set the RedirectHedears flag to true
if you want headers to be copied on subsequent requests when redirecting. I'll be looking at the cookie feature later today.
from goreq.
@mostafadahab did this finally work for you?
from goreq.
@marcosnils thanks for update i was waiting for adding cookies support 🔨
from goreq.
any updates ??
from goreq.
@mostafadahab sorry, I've started the work but I got occupied with some other stuff and I couldn't complete it. This weekend I hope to dedicate it enough time to have it ready.
Contributions are very welcome though. You can make the modifications yourself and propose the changes so we can move faster.
from goreq.
@mostafadahab just wanted to let u know that cookie support has been included in #78 thanks to @bfontaine .
from goreq.
good job ! i tried the new version of goreq but still there issue in redirect with headers and i can't find RedirectHedears ! and the cookie jar not saving set-cookie and resend it in redirect requests
from goreq.
@mostafadahab can have you updated the library locally?. The RedirectHeaders flag can be set here:
https://github.com/franela/goreq/blob/master/goreq.go#L38
from goreq.
@mostafadahab regarding cookies on redirection it hasn't been done because of security concerns. It's not safe to automatically redirect cookies as the final URL might get sensitive cookies information. If you still want do send the cookies anyways, you can redefine the CheckRedirect function here: https://github.com/franela/goreq/blob/master/goreq.go#L276
from goreq.
@marcosnils RedirectHeaders works the problem was not having the last version
i say the cookie jar must attached for example if i'am make login to facebook.com and facebook redirect to facebook.com/home.php the must have cookies ! if not will log me out !
from goreq.
@mostafadahab makes sense. We can add a RedirectCookies flag into goreq.Request for that purpose.
I'll craft a change later today.
from goreq.
thanks alot you're awesome
i'm developing me skills in go but it's still weak
when i'm ready i will try to contribute
from goreq.
@mostafadahab speaking with @xetorthio I've realized that cookies are currently being forwarded within redirects. Cookies it's just another header in the request, so if you do the following request:
req := goreq.Request{
Method: "GET",
Uri: "http://somedomain.com/redirects",
MaxRedirects: 2,
RedirectHeaders: true,
}
req.AddCookie(&http.Cookie{Name: "Test", Value: "Value"})
res, _ := req.Do()
Then the Cookie
header will included in the redirect accordingly.
Per your facebook example seems like you want goreq
to handle the set-cookie
header and then send it back to the redirect url specified. That's not something that goreq
should do because the User-Agent (usually browsers in this case) make a lot of validations when handling cookies which we don't think it's competent to goreq
.
What I would recommend you to do is to make the login in two steps
- Make a request to login (without redirect) and obtain the cookies from the response header (set-cookie)
- Make further requests using the
goreq
AddCookie
option andRedirectHeaders: true
. If a redirect occurs thengoreq
will forward the specified cookie correctly to the destination URL.
from goreq.
or just assign global cookiejar to each request
@marcosnils thanks man 👍 😍
from goreq.
Related Issues (20)
- Request Query Strings are normalized to small letters HOT 3
- Support for Custom Transport? HOT 2
- Add a DoByClient method
- json not being passed to request HOT 1
- missing license HOT 1
- Concurrency issue: call .Do() in different goroutines replace CheckRedirect of singleton client HOT 1
- using goreq with appengine HOT 1
- when assign client cookiejar not attached
- How to send POST application/x-www-form-urlencoded (NOT JSON) HOT 7
- Individual debug flags for header and body?
- DNS resolve time out.
- how to post json to sever
- How to request https via proxy
- FromJsonTo() doesn't work
- Empty URI in response HOT 1
- Timeout unit tests fail
- unknown http.Transport field 'ProxyConnectHeader' in struct literal HOT 4
- can't call FromJsonTo after ToString() method has been called
- Possible for a tag/release of the "stable"
- 301 302 go func(){}() error
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goreq.