nsdown / proxyinabox

:inbox_tray:Proxy in a box. 自动抓取、调度代理 IP。

Proxy-in-a-Box

Proxy-in-a-Box helps programmers quickly and easily develop powerful crawler services. one-script, easy-to-use: proxies in a box.

Usage:
  proxy-in-a-box [flags]

Flags:
  -c, --conf string   config file (default "./pb.yaml")
  -p, --ha string     http proxy server addr (default "127.0.0.1:8080")
  -h, --help          help for proxy-in-a-box
  -s, --sa string     https proxy server addr (default "127.0.0.1:8081")

Usage

1. get lastest Proxy-in-a-Box

   go get -u -v github.com/naiba/proxyinabox/cmd/proxy-in-a-box/...

2. enter the application directory

   cd $GOPATH/bin

3. write config file #Config
4. run it

   ./proxy-in-a-box

5. configured in your code

   HTTP proxy: `http://[IP]:8080`
   HTTPS proxy: `https://[IP]:8081`
   * Please set http header when requesting: "Naiba: lifelonglearning" ref:https://github.com/naiba/proxyinabox/blob/master/cmd/proxy-in-a-box/test_server.sh
   

   Set in the code, and then grab it, the Proxy-in-a-Box will automatically assign the proxy.

Config

# run in debug mode
debug: true
# mysql config
mysql:
  host: 127.0.0.1
  port: 3306
  user: root
  pass: 123456
  dbname: proxy
# system config
sys:
  name: Naiba
  # verify proxy's worker num
  proxy_verify_worker: 20
  # how many domains can request per ip in 30min
  domains_per_ip: 30
  # how many request can do per ip in 1s
  request_limit_per_ip: 10
  # verify interval of the proxy stored in the database
  verify_duration: 30

Benchmark

ab -H 'Naiba: lifelonglearning' -v4  -n100 -c10 -X 127.0.0.1:8080 http://api.ip.la/cn