Flag to ignore errors and keep running (or -sr bug fix) about httpx

[intrd]

Hey @intrd,

Can you add some sample URLs where it gets failed, so we can make changes accordingly.

sure, this input breaks the httpx:

http://target.com:80/%22:269092299880021,%224456808911257%22:117599551732038,%224456808791254%22:219099868222532,%224456808631250%22:117241395101609,%224456808471246%22:379290365485680,%224456808271241%22:217348818399031,%224456808071236%22:512726018751670,%224456807991234%22:291913630927125,%224456807831230%22:291001267685617,%224456807791229%22:378181278930281%7D

maybe a good idea is to hash or randomize the url path and stores like this:

/target.com_80/32e152b6824592d175fc8cfda4237433e595292770a67c543c4992d30cb0f9f2

[ehsandeep]

@intrd This is fixed with #113

[ehsandeep]

Hey @intrd,

Can you add some sample URLs where it gets failed, so we can make changes accordingly.

[intrd]

Dirty/temporary fix @ httpx/cmd/httpx/httpx.go:

"encoding/hex"
"crypto/md5"

...

        // store responses in directory
        if scanopts.StoreResponse {
                splitted := strings.Split(domain, "/")
                path := ""
                if len(splitted) == 1 {
                        path = "/"
                }else{
                        path = splitted[1]
                }
                domain := splitted[0]
                hasher := md5.New()
                hasher.Write([]byte(path))
                out := hex.EncodeToString(hasher.Sum(nil))
                fullpath := "output/"+domain+"/"+out
                if _, err := os.Stat("output/"+domain); os.IsNotExist(err) { 
                        os.MkdirAll("output/"+domain, 0700)
                }
                err := ioutil.WriteFile(fullpath, []byte(resp.Raw), 0644)
                if err != nil {
                        gologger.Fatalf("Could not write response, at path to disc: %s", fullpath)
                }
        }

$ find output/

output/
output/target1.com
output/target1.com/a8f5f167f44f4964e6c998dee827110c
output/target2.com
output/target2.com/a660a4c512ce611d365624fe1bc6b5c6
output/target2.com/22bd4dae3f7b3f28e87813a4ccd13a12
output/help.target3.com
output/help.target3.com/2f8a6bf31f3bd67bd2d9720c58b19c9a