c4milo / gofetch Goto Github PK

View Code? Open in Web Editor NEW

27.0 4.0 2.0 10.26 MB

Go library for downloading content from internet, in parallel.

License: Mozilla Public License 2.0

Go 100.00%

parallel fetcher http gofetch etags downloader golang-library golang

gofetch's People

Contributors

Stargazers

Watchers

Forkers

junbinku devopsmi

gofetch's Issues

If server doesn't allow preflight HEAD request, do not parallelize.

This is to fix an issue with Github Release servers generating wrong signatures when redirecting HEAD requests to S3.

Add support for net.Context so we can cancel the download at any time.

Check file integrity if a hash is provided

Here is a start:

func verify(f *os.File, checksumType, checksum string) (string, error) {
    // Makes sure file cursor is positioned at the beginning
    _, err := f.Seek(0, 0)
    if err != nil {
        return "", err
    }

    var hasher hash.Hash
    switch checksumType {
    case "md5":
        hasher = md5.New()
    case "sha1":
        hasher = sha1.New()
    case "sha256":
        hasher = sha256.New()
    case "sha512":
        hasher = sha512.New()
    default:
        return "", fmt.Errorf("crypto algorithm no supported: %s", checksumType)
    }
    _, err = io.Copy(hasher, f)
    if err != nil {
        return "", err
    }

    result := fmt.Sprintf("%x", hasher.Sum(nil))

    if result != checksum {
        return result, fmt.Errorf("checksum does not match\n found: %s\n expected: %s", result, checksum)
    }

    return result, nil
}

Resume test has a data race

FYI

$ go test -race -run Resume
==================
WARNING: DATA RACE
Read by goroutine 6:
  github.com/c4milo/gofetch.TestResume()
      /home/caleb/p/go/src/github.com/c4milo/gofetch/gofetch_test.go:129 +0x8db
  testing.tRunner()
      /home/caleb/apps/go/src/testing/testing.go:456 +0xdc

Previous write by goroutine 8:
  github.com/c4milo/gofetch.TestResume.func2()
      /home/caleb/p/go/src/github.com/c4milo/gofetch/gofetch_test.go:116 +0x15d

Goroutine 6 (running) created at:
  testing.RunTests()
      /home/caleb/apps/go/src/testing/testing.go:561 +0xaa3
  testing.(*M).Run()
      /home/caleb/apps/go/src/testing/testing.go:494 +0xe4
  main.main()
      github.com/c4milo/gofetch/_test/_testmain.go:58 +0x20f

Goroutine 8 (finished) created at:
  github.com/c4milo/gofetch.TestResume()
      /home/caleb/p/go/src/github.com/c4milo/gofetch/gofetch_test.go:118 +0x70f
  testing.tRunner()
      /home/caleb/apps/go/src/testing/testing.go:456 +0xdc
==================
PASS
Found 1 data race(s)
exit status 66
FAIL    github.com/c4milo/gofetch       1.126s

Occasionally I also get an "malformed HTTP status code"

Unsolicited response received on idle HTTP channel starting with "\xd7"; err=<nil>

We are getting a weird error when running TestFetchWithoutContentLength

We should avoid fetching a file if it was already fetched and its ETag matches with the previous download. I think we can enable ETag support by default and expose a configuration option to disable it, forcing downloads.

c4milo / gofetch Goto Github PK

gofetch's People

Contributors

Stargazers

Watchers

Forkers

gofetch's Issues

If server doesn't allow preflight HEAD request, do not parallelize.

Add support for net.Context so we can cancel the download at any time.

Check file integrity if a hash is provided

Resume test has a data race

Occasionally I also get an "malformed HTTP status code"

Unsolicited response received on idle HTTP channel starting with "\xd7"; err=<nil>

Add ETag support

Add TravisCI

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

Jobs