henson / proxypool Goto Github PK
View Code? Open in Web Editor NEWGolang实现的IP代理池
License: Apache License 2.0
proxypool's People
Contributors
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
Stargazers
Watchers
Forkers
davidmr001 chennqqi baiweiyll piety68 cqspirit vus520 chinayin cngolang hunny-lh zhanglei zhuzhengyi strongc zz0412 chenyongze genjiluo growthring itachi123456789 sekift lufred gaowenbin czm1129 zhangjunfang leozvc transzero smartdriver001 xluohome a-long-long-markable-list chinarefers zdoop cnryb yocruzer freewh shejianmin sweetcard borays flybird119 emmafh xiaohuangniuu goribun gotoolkits xinghanggogogo simpdo chenxuelee liuxiaozhen solostart go-xx a596544022 cnoldtree guoyu07 solertis dariolourenco tx991020 koalacxr ouyangdage awesome-golang simlan sunminghong foxundermoon cancheung yang-f githubzwj zanjs jackerzhou s0x06 eudemon000 bozzg flyman-9527 jun7th victorruan justintung liasica dst1213 lubyruffy huanglei890714 chenjingxiong fiysid cwchiu yong-peng iheio th002 lzjian119 hookttg lorock ban11111 kaka10000 tgkplvfy wangjunbiao giddily mumugoah arryboom dulumao albert2lyu zerochl an-jun shellsec ghostlyman golangltd lishuntaono1 yc8332 aiicyproxypool's Issues
强烈建议贴出依赖库的版本号~~
运行后报错
按照步骤安装完成后，编译并运行后出现了如下错误，请问是什么原因？谢谢！
panic: runtime error: slice bounds out of range
goroutine 26 [running]: github.com/henson/ProxyPool/getter.IP181(0x0, 0x0, 0x0) C:/works/golang/src/github.com/henson/ProxyPool/getter/ip181.go:40 +0x636 main.run.func1(0xc042032480, 0xc0420b8270, 0x7bc560) C:/works/golang/src/github.com/henson/ProxyPool/main.go:66 +0x32 created by main.run C:/works/golang/src/github.com/henson/ProxyPool/main.go:71 +0x15c Web server running on port 2017 Web server running on portW e2b0 1s6e rver running on port 2015
我本地代码增加了一些特性，请问怎么给你提交代码
1）对代理可用性的验证
2）增加speed字段，验证代理的速度（毫秒）
3）调用API获取代理IP时自动过滤掉慢的代理（>=1s）
pkg/storage 里面缺少一个rand_darwin.go mac build报错
好像存在性能瓶颈
通过mongostat分析了下mongodb的连接数。发现只要是该程序在运行，mongodb连接数就加1，连接数一直在增加。直到mongodb连接数用完了
gomodule模式下报错
gomodule开启后
1.go mod init
2.go mod tidy
然后报错
github.com/go-xorm/[email protected]: parsing go.mod: unexpected module path "xorm.io/core"
我对代码做了些修改
89ip.go err处理有问题，有时会导致空指针。
if resp.StatusCode != 200 {
// clog.Warn(err.Error())
return
}
xicidl.go 请求有问题，我重写了。
详见 https://github.com/nlh1996/gopa/blob/master/proxy/xicidl.go
启动时报错，麻烦看一下
./proxypool conf/app.ini
2018/10/17 18:51:29 [TRACE] Log path:
2018/10/17 18:51:29 [TRACE] Log Mode: File (Info)
2018/10/17 18:51:29 [ INFO] ProxyPool
2018/10/17 18:51:29 Starting server 0.0.0.0:3000
2018/10/17 18:51:29 Chan: 0, IP: 0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x7f3f2a]
goroutine 21 [running]:
github.com/Aiicy/htmlquery.(*NodeNavigator).NodeType(0xc4202c8100, 0x7f260e858bf8)
/root/go/src/github.com/Aiicy/htmlquery/query.go:145 +0x2a
github.com/antchfx/xpath.axisPredicate.func1(0xb2c9c0, 0xc4202c8100, 0x4128c8)
/root/go/src/github.com/antchfx/xpath/build.go:45 +0x4f
github.com/antchfx/xpath.(*descendantQuery).Select.func1(0x8a6500, 0xc420163ec0)
/root/go/src/github.com/antchfx/xpath/query.go:236 +0x158
github.com/antchfx/xpath.(*descendantQuery).Select(0xc420163da0, 0xb231a0, 0xc4202c80c0, 0xc42002d400, 0x8000e851000)
/root/go/src/github.com/antchfx/xpath/query.go:263 +0x36
github.com/antchfx/xpath.(*filterQuery).Select(0xc4202c8060, 0xb231a0, 0xc4202c80c0, 0x881200, 0x0)
/root/go/src/github.com/antchfx/xpath/query.go:526 +0x4b
github.com/antchfx/xpath.(*descendantQuery).Select(0xc420163e00, 0xb231a0, 0xc4202c80c0, 0x412167, 0xc420163e90)
/root/go/src/github.com/antchfx/xpath/query.go:226 +0xd5
github.com/antchfx/xpath.(*filterQuery).Select(0xc4202c80a0, 0xb231a0, 0xc4202c80c0, 0xc4202c80c0, 0x20)
/root/go/src/github.com/antchfx/xpath/query.go:526 +0x4b
github.com/antchfx/xpath.(*descendantQuery).Select(0xc420163e60, 0xb231a0, 0xc4202c80c0, 0x0, 0xb28f01)
/root/go/src/github.com/antchfx/xpath/query.go:226 +0xd5
github.com/antchfx/xpath.(*descendantQuery).Select(0xc420163e90, 0xb231a0, 0xc4202c80c0, 0xc4202c80c0, 0xc420163e90)
/root/go/src/github.com/antchfx/xpath/query.go:226 +0xd5
github.com/antchfx/xpath.(*NodeIterator).MoveNext(0xc4202c80c0, 0xb2c9c0)
/root/go/src/github.com/antchfx/xpath/xpath.go:85 +0x49
github.com/Aiicy/htmlquery.Find(0x0, 0x9045ea, 0x39, 0xb241e0, 0xc420163bf0, 0x0, 0xc, 0xc0ad00)
/root/go/src/github.com/Aiicy/htmlquery/query.go:32 +0x112
github.com/henson/proxypool/getter.PLP(0xc, 0xc4201f8150, 0xc4201f2060)
/root/go/src/github.com/henson/proxypool/getter/plp.go:14 +0x6e
main.run.func1(0xc42010e1e0, 0xc4201f20e0, 0x916658)
/root/go/src/github.com/henson/proxypool/main.go:70 +0x2b
created by main.run
/root/go/src/github.com/henson/proxypool/main.go:69 +0xe0
IP重复过滤问题
查看代码 的时候，看到这段代码的CheckIP的代码：
这里其实是没有判断，数据表中是否有这个IP的话，无论有没有都是直接去更新数据表的记录。在配合第一张图的ProxyAdd，其实是没有判断是嘛。。如果有重复IP就不管了，就直接插入。。
麻烦看一下，这报错了，ubuntu编译后，第一次运行
root@cloud:~/gopath/src/github.com/henson/proxypool# ./proxypool
2019/04/27 23:34:38 [TRACE] Log path:
2019/04/27 23:34:38 [TRACE] Log Mode: File (Info)
2019/04/27 23:34:38 [ INFO] ProxyPool
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xc8 pc=0x81a450]
goroutine 1 [running]:
github.com/go-xorm/xorm.(*Session).Init(0xc0000fab00)
/root/gopath/src/github.com/go-xorm/xorm/session.go:94 +0x200
github.com/go-xorm/xorm.(*Engine).NewSession(...)
/root/gopath/src/github.com/go-xorm/xorm/engine.go:315
github.com/go-xorm/xorm.(*Engine).Where(0x0, 0x9fc1a0, 0xb84060, 0xc0001527f0, 0x1, 0x1, 0xc0001fff30)
/root/gopath/src/github.com/go-xorm/xorm/engine.go:592 +0x57
github.com/henson/proxypool/pkg/models.countIps(0x8)
/root/gopath/src/github.com/henson/proxypool/pkg/models/ip.go:39 +0x97
github.com/henson/proxypool/pkg/models.CountIPs(...)
/root/gopath/src/github.com/henson/proxypool/pkg/models/ip.go:45
main.main()
/root/gopath/src/github.com/henson/proxypool/main.go:45 +0xde
启动提示这个错误，是怎么回事啊
⚡ root@instance-69nkc38q ~ ./proxypool
2019/04/19 11:21:36 [TRACE] Log path:
2019/04/19 11:21:36 [TRACE] Log Mode: File (Info)
2019/04/19 11:21:36 [ INFO] ProxyPool
2019/04/19 11:21:37 Starting server 0.0.0.0:3000
2019/04/19 11:21:37 Chan: 0, IP: 810
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x818708]
goroutine 71 [running]:
github.com/henson/proxypool/getter.IP89(0x0, 0x0, 0x0)
D:/goworker/src/github.com/henson/proxypool/getter/89ip.go:27 +0x478
main.run.func1(0xc00060a7e0, 0xc000665330, 0x93bf18)
D:/goworker/src/github.com/henson/proxypool/main.go:72 +0x2b
created by main.run
D:/goworker/src/github.com/henson/proxypool/main.go:71 +0xee
mysql 数据表结构呢
请问可以根据速度筛选代理吗
类似于这样的
https://gimmeproxy.com/ 多维度筛选代理ip地址
build 报错
# github.com/henson/proxypool/pkg/storage
pkg/storage/filter.go:90:15: undefined: RandInt
pkg/storage/filter.go:103:15: undefined: RandInt
ip池里的代理好多都不能用
代理不能用的现象特别频繁
从代理池中拿出代理ip后，设置好代理，基本上10个里有9个不能用。是因为代理过期了么？
程序运行久了，chan数量保持再100以上不变
你好：
我将项目fork了一份，打包成docker跑在服务器上面，过几天查看日志，发现chan的数量保持在100以上，chan不消费了，程序不会自动爬取ip，需要重启：
请问这可能是什么情况导致的？
panic: runtime error: invalid memory address or nil pointer dereference
mysql的数据结构需要自己设置吗
有配置文件可以一键生成就好了
Zombie Process(phantomjs) 过多
运行了大约 24 小时左右，查看系统有 297 个 Zombie Process
➜ ~ phantomjs --version
2.1.1
➜ ~ ps -ef | grep defunct
root 502 20310 0 13:15 pts/7 00:00:00 [phantomjs] <defunct>
root 503 20310 0 13:15 pts/7 00:00:00 [phantomjs] <defunct>
root 504 20310 0 13:15 pts/7 00:00:00 [phantomjs] <defunct>
root 876 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 877 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 878 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 940 20310 0 05:45 pts/7 00:00:00 [phantomjs] <defunct>
root 941 20310 0 05:45 pts/7 00:00:00 [phantomjs] <defunct>
root 942 20310 0 05:45 pts/7 00:00:00 [phantomjs] <defunct>
root 1323 20310 0 13:25 pts/7 00:00:00 [phantomjs] <defunct>
root 1324 20310 0 13:25 pts/7 00:00:00 [phantomjs] <defunct>
root 1325 20310 0 13:25 pts/7 00:00:00 [phantomjs] <defunct>
root 1608 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 1609 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 1610 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 1691 20310 0 05:55 pts/7 00:00:00 [phantomjs] <defunct>
root 1692 20310 0 05:55 pts/7 00:00:00 [phantomjs] <defunct>
root 1693 20310 0 05:55 pts/7 00:00:00 [phantomjs] <defunct>
root 2363 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 2364 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 2365 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 2429 20310 0 06:05 pts/7 00:00:00 [phantomjs] <defunct>
root 2430 20310 0 06:05 pts/7 00:00:00 [phantomjs] <defunct>
root 2431 20310 0 06:05 pts/7 00:00:00 [phantomjs] <defunct>
root 3082 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 3083 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 3084 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 3147 20310 0 06:15 pts/7 00:00:00 [phantomjs] <defunct>
root 3148 20310 0 06:15 pts/7 00:00:00 [phantomjs] <defunct>
root 3149 20310 0 06:15 pts/7 00:00:00 [phantomjs] <defunct>
root 3430 20310 0 13:55 pts/7 00:00:00 [phantomjs] <defunct>
root 3431 20310 0 13:55 pts/7 00:00:00 [phantomjs] <defunct>
root 3432 20310 0 13:55 pts/7 00:00:00 [phantomjs] <defunct>
root 3796 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 3797 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 3798 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 3964 20310 0 06:25 pts/7 00:00:00 [phantomjs] <defunct>
root 3965 20310 0 06:25 pts/7 00:00:00 [phantomjs] <defunct>
root 3966 20310 0 06:25 pts/7 00:00:00 [phantomjs] <defunct>
root 4154 20310 0 14:05 pts/7 00:00:00 [phantomjs] <defunct>
root 4155 20310 0 14:05 pts/7 00:00:00 [phantomjs] <defunct>
root 4156 20310 0 14:05 pts/7 00:00:00 [phantomjs] <defunct>
root 4513 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 4514 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 4515 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 4681 20310 0 06:35 pts/7 00:00:00 [phantomjs] <defunct>
root 4682 20310 0 06:35 pts/7 00:00:00 [phantomjs] <defunct>
root 4683 20310 0 06:35 pts/7 00:00:00 [phantomjs] <defunct>
root 5227 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 5228 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 5229 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 5401 20310 0 06:45 pts/7 00:00:00 [phantomjs] <defunct>
root 5402 20310 0 06:45 pts/7 00:00:00 [phantomjs] <defunct>
root 5403 20310 0 06:45 pts/7 00:00:00 [phantomjs] <defunct>
root 5947 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 5948 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 5949 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 6119 20310 0 06:55 pts/7 00:00:00 [phantomjs] <defunct>
root 6120 20310 0 06:55 pts/7 00:00:00 [phantomjs] <defunct>
root 6121 20310 0 06:55 pts/7 00:00:00 [phantomjs] <defunct>
root 6198 20310 0 14:35 pts/7 00:00:00 [phantomjs] <defunct>
root 6199 20310 0 14:35 pts/7 00:00:00 [phantomjs] <defunct>
root 6200 20310 0 14:35 pts/7 00:00:00 [phantomjs] <defunct>
root 6661 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 6662 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 6663 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 6839 20310 0 07:05 pts/7 00:00:00 [phantomjs] <defunct>
root 6840 20310 0 07:05 pts/7 00:00:00 [phantomjs] <defunct>
root 6841 20310 0 07:05 pts/7 00:00:00 [phantomjs] <defunct>
root 6921 20310 0 14:45 pts/7 00:00:00 [phantomjs] <defunct>
root 6922 20310 0 14:45 pts/7 00:00:00 [phantomjs] <defunct>
root 6923 20310 0 14:45 pts/7 00:00:00 [phantomjs] <defunct>
root 7379 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 7380 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 7381 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 7558 20310 0 07:15 pts/7 00:00:00 [phantomjs] <defunct>
root 7559 20310 0 07:15 pts/7 00:00:00 [phantomjs] <defunct>
root 7560 20310 0 07:15 pts/7 00:00:00 [phantomjs] <defunct>
root 8101 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 8102 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 8103 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 8282 20310 0 07:25 pts/7 00:00:00 [phantomjs] <defunct>
root 8283 20310 0 07:25 pts/7 00:00:00 [phantomjs] <defunct>
root 8284 20310 0 07:25 pts/7 00:00:00 [phantomjs] <defunct>
root 8827 20310 0 00:05 pts/7 00:00:00 [phantomjs] <defunct>
root 8828 20310 0 00:05 pts/7 00:00:00 [phantomjs] <defunct>
root 8829 20310 0 00:05 pts/7 00:00:00 [phantomjs] <defunct>
root 8964 20310 0 15:15 pts/7 00:00:00 [phantomjs] <defunct>
root 8965 20310 0 15:15 pts/7 00:00:00 [phantomjs] <defunct>
root 8966 20310 0 15:15 pts/7 00:00:00 [phantomjs] <defunct>
root 9000 20310 0 07:35 pts/7 00:00:00 [phantomjs] <defunct>
root 9001 20310 0 07:35 pts/7 00:00:00 [phantomjs] <defunct>
root 9002 20310 0 07:35 pts/7 00:00:00 [phantomjs] <defunct>
root 9551 20310 0 00:15 pts/7 00:00:00 [phantomjs] <defunct>
root 9552 20310 0 00:15 pts/7 00:00:00 [phantomjs] <defunct>
root 9553 20310 0 00:15 pts/7 00:00:00 [phantomjs] <defunct>
root 9730 20310 0 07:45 pts/7 00:00:00 [phantomjs] <defunct>
root 9731 20310 0 07:45 pts/7 00:00:00 [phantomjs] <defunct>
root 9732 20310 0 07:45 pts/7 00:00:00 [phantomjs] <defunct>
root 9810 9531 0 15:26 pts/5 00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn defunct
root 10272 20310 0 00:25 pts/7 00:00:00 [phantomjs] <defunct>
root 10273 20310 0 00:25 pts/7 00:00:00 [phantomjs] <defunct>
root 10274 20310 0 00:25 pts/7 00:00:00 [phantomjs] <defunct>
root 10986 20310 0 00:35 pts/7 00:00:00 [phantomjs] <defunct>
root 10987 20310 0 00:35 pts/7 00:00:00 [phantomjs] <defunct>
root 10988 20310 0 00:35 pts/7 00:00:00 [phantomjs] <defunct>
root 11108 20310 0 08:05 pts/7 00:00:00 [phantomjs] <defunct>
root 11109 20310 0 08:05 pts/7 00:00:00 [phantomjs] <defunct>
root 11110 20310 0 08:05 pts/7 00:00:00 [phantomjs] <defunct>
root 11704 20310 0 00:45 pts/7 00:00:00 [phantomjs] <defunct>
root 11705 20310 0 00:45 pts/7 00:00:00 [phantomjs] <defunct>
root 11706 20310 0 00:45 pts/7 00:00:00 [phantomjs] <defunct>
root 11825 20310 0 08:15 pts/7 00:00:00 [phantomjs] <defunct>
root 11826 20310 0 08:15 pts/7 00:00:00 [phantomjs] <defunct>
root 11827 20310 0 08:15 pts/7 00:00:00 [phantomjs] <defunct>
root 12423 20310 0 00:55 pts/7 00:00:00 [phantomjs] <defunct>
root 12424 20310 0 00:55 pts/7 00:00:00 [phantomjs] <defunct>
root 12425 20310 0 00:55 pts/7 00:00:00 [phantomjs] <defunct>
root 12553 20310 0 08:25 pts/7 00:00:00 [phantomjs] <defunct>
root 12554 20310 0 08:25 pts/7 00:00:00 [phantomjs] <defunct>
root 12555 20310 0 08:25 pts/7 00:00:00 [phantomjs] <defunct>
root 13146 20310 0 01:05 pts/7 00:00:00 [phantomjs] <defunct>
root 13147 20310 0 01:05 pts/7 00:00:00 [phantomjs] <defunct>
root 13148 20310 0 01:05 pts/7 00:00:00 [phantomjs] <defunct>
root 13280 20310 0 08:35 pts/7 00:00:00 [phantomjs] <defunct>
root 13281 20310 0 08:35 pts/7 00:00:00 [phantomjs] <defunct>
root 13282 20310 0 08:35 pts/7 00:00:00 [phantomjs] <defunct>
root 13874 20310 0 01:15 pts/7 00:00:00 [phantomjs] <defunct>
root 13875 20310 0 01:15 pts/7 00:00:00 [phantomjs] <defunct>
root 13876 20310 0 01:15 pts/7 00:00:00 [phantomjs] <defunct>
root 14000 20310 0 08:45 pts/7 00:00:00 [phantomjs] <defunct>
root 14001 20310 0 08:45 pts/7 00:00:00 [phantomjs] <defunct>
root 14002 20310 0 08:45 pts/7 00:00:00 [phantomjs] <defunct>
root 14601 20310 0 01:25 pts/7 00:00:00 [phantomjs] <defunct>
root 14602 20310 0 01:25 pts/7 00:00:00 [phantomjs] <defunct>
root 14603 20310 0 01:25 pts/7 00:00:00 [phantomjs] <defunct>
root 15315 20310 0 01:35 pts/7 00:00:00 [phantomjs] <defunct>
root 15316 20310 0 01:35 pts/7 00:00:00 [phantomjs] <defunct>
root 15317 20310 0 01:35 pts/7 00:00:00 [phantomjs] <defunct>
root 15379 20310 0 09:05 pts/7 00:00:00 [phantomjs] <defunct>
root 15380 20310 0 09:05 pts/7 00:00:00 [phantomjs] <defunct>
root 15381 20310 0 09:05 pts/7 00:00:00 [phantomjs] <defunct>
root 16033 20310 0 01:45 pts/7 00:00:00 [phantomjs] <defunct>
root 16034 20310 0 01:45 pts/7 00:00:00 [phantomjs] <defunct>
root 16035 20310 0 01:45 pts/7 00:00:00 [phantomjs] <defunct>
root 16098 20310 0 09:15 pts/7 00:00:00 [phantomjs] <defunct>
root 16099 20310 0 09:15 pts/7 00:00:00 [phantomjs] <defunct>
root 16100 20310 0 09:15 pts/7 00:00:00 [phantomjs] <defunct>
root 16747 20310 0 01:55 pts/7 00:00:00 [phantomjs] <defunct>
root 16748 20310 0 01:55 pts/7 00:00:00 [phantomjs] <defunct>
root 16749 20310 0 01:55 pts/7 00:00:00 [phantomjs] <defunct>
root 17466 20310 0 02:05 pts/7 00:00:00 [phantomjs] <defunct>
root 17467 20310 0 02:05 pts/7 00:00:00 [phantomjs] <defunct>
root 17468 20310 0 02:05 pts/7 00:00:00 [phantomjs] <defunct>
root 17477 20310 0 09:35 pts/7 00:00:00 [phantomjs] <defunct>
root 17478 20310 0 09:35 pts/7 00:00:00 [phantomjs] <defunct>
root 17479 20310 0 09:35 pts/7 00:00:00 [phantomjs] <defunct>
root 18182 20310 0 02:15 pts/7 00:00:00 [phantomjs] <defunct>
root 18183 20310 0 02:15 pts/7 00:00:00 [phantomjs] <defunct>
root 18184 20310 0 02:15 pts/7 00:00:00 [phantomjs] <defunct>
root 18197 20310 0 09:45 pts/7 00:00:00 [phantomjs] <defunct>
root 18198 20310 0 09:45 pts/7 00:00:00 [phantomjs] <defunct>
root 18199 20310 0 09:45 pts/7 00:00:00 [phantomjs] <defunct>
root 18902 20310 0 02:25 pts/7 00:00:00 [phantomjs] <defunct>
root 18903 20310 0 02:25 pts/7 00:00:00 [phantomjs] <defunct>
root 18904 20310 0 02:25 pts/7 00:00:00 [phantomjs] <defunct>
root 18915 20310 0 09:55 pts/7 00:00:00 [phantomjs] <defunct>
root 18916 20310 0 09:55 pts/7 00:00:00 [phantomjs] <defunct>
root 18917 20310 0 09:55 pts/7 00:00:00 [phantomjs] <defunct>
root 19617 20310 0 02:35 pts/7 00:00:00 [phantomjs] <defunct>
root 19618 20310 0 02:35 pts/7 00:00:00 [phantomjs] <defunct>
root 19619 20310 0 02:35 pts/7 00:00:00 [phantomjs] <defunct>
root 20291 20310 0 10:15 pts/7 00:00:00 [phantomjs] <defunct>
root 20292 20310 0 10:15 pts/7 00:00:00 [phantomjs] <defunct>
root 20293 20310 0 10:15 pts/7 00:00:00 [phantomjs] <defunct>
root 20317 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 20318 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 20319 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 20356 20310 0 02:45 pts/7 00:00:00 [phantomjs] <defunct>
root 20357 20310 0 02:45 pts/7 00:00:00 [phantomjs] <defunct>
root 20358 20310 0 02:45 pts/7 00:00:00 [phantomjs] <defunct>
root 21024 20310 0 10:25 pts/7 00:00:00 [phantomjs] <defunct>
root 21025 20310 0 10:25 pts/7 00:00:00 [phantomjs] <defunct>
root 21026 20310 0 10:25 pts/7 00:00:00 [phantomjs] <defunct>
root 21048 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 21049 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 21050 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 21093 20310 0 02:55 pts/7 00:00:00 [phantomjs] <defunct>
root 21094 20310 0 02:55 pts/7 00:00:00 [phantomjs] <defunct>
root 21095 20310 0 02:55 pts/7 00:00:00 [phantomjs] <defunct>
root 21787 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 21788 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 21789 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 21813 20310 0 03:05 pts/7 00:00:00 [phantomjs] <defunct>
root 21814 20310 0 03:05 pts/7 00:00:00 [phantomjs] <defunct>
root 21815 20310 0 03:05 pts/7 00:00:00 [phantomjs] <defunct>
root 22410 20310 0 10:45 pts/7 00:00:00 [phantomjs] <defunct>
root 22411 20310 0 10:45 pts/7 00:00:00 [phantomjs] <defunct>
root 22412 20310 0 10:45 pts/7 00:00:00 [phantomjs] <defunct>
root 22514 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 22515 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 22516 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 22539 20310 0 03:15 pts/7 00:00:00 [phantomjs] <defunct>
root 22540 20310 0 03:15 pts/7 00:00:00 [phantomjs] <defunct>
root 22541 20310 0 03:15 pts/7 00:00:00 [phantomjs] <defunct>
root 23135 20310 0 10:55 pts/7 00:00:00 [phantomjs] <defunct>
root 23136 20310 0 10:55 pts/7 00:00:00 [phantomjs] <defunct>
root 23137 20310 0 10:55 pts/7 00:00:00 [phantomjs] <defunct>
root 23233 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 23234 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 23235 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 23267 20310 0 03:25 pts/7 00:00:00 [phantomjs] <defunct>
root 23268 20310 0 03:25 pts/7 00:00:00 [phantomjs] <defunct>
root 23269 20310 0 03:25 pts/7 00:00:00 [phantomjs] <defunct>
root 23969 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 23970 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 23971 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 23996 20310 0 03:35 pts/7 00:00:00 [phantomjs] <defunct>
root 23997 20310 0 03:35 pts/7 00:00:00 [phantomjs] <defunct>
root 23998 20310 0 03:35 pts/7 00:00:00 [phantomjs] <defunct>
root 24531 20310 0 11:15 pts/7 00:00:00 [phantomjs] <defunct>
root 24532 20310 0 11:15 pts/7 00:00:00 [phantomjs] <defunct>
root 24533 20310 0 11:15 pts/7 00:00:00 [phantomjs] <defunct>
root 24685 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 24686 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 24687 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 24718 20310 0 03:45 pts/7 00:00:00 [phantomjs] <defunct>
root 24719 20310 0 03:45 pts/7 00:00:00 [phantomjs] <defunct>
root 24720 20310 0 03:45 pts/7 00:00:00 [phantomjs] <defunct>
root 25261 20310 0 11:25 pts/7 00:00:00 [phantomjs] <defunct>
root 25262 20310 0 11:25 pts/7 00:00:00 [phantomjs] <defunct>
root 25263 20310 0 11:25 pts/7 00:00:00 [phantomjs] <defunct>
root 25407 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 25408 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 25409 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 25449 20310 0 03:55 pts/7 00:00:00 [phantomjs] <defunct>
root 25450 20310 0 03:55 pts/7 00:00:00 [phantomjs] <defunct>
root 25451 20310 0 03:55 pts/7 00:00:00 [phantomjs] <defunct>
root 26123 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 26124 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 26125 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 26174 20310 0 04:05 pts/7 00:00:00 [phantomjs] <defunct>
root 26175 20310 0 04:05 pts/7 00:00:00 [phantomjs] <defunct>
root 26176 20310 0 04:05 pts/7 00:00:00 [phantomjs] <defunct>
root 26643 20310 0 11:45 pts/7 00:00:00 [phantomjs] <defunct>
root 26644 20310 0 11:45 pts/7 00:00:00 [phantomjs] <defunct>
root 26645 20310 0 11:45 pts/7 00:00:00 [phantomjs] <defunct>
root 26843 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 26844 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 26845 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 26895 20310 0 04:15 pts/7 00:00:00 [phantomjs] <defunct>
root 26896 20310 0 04:15 pts/7 00:00:00 [phantomjs] <defunct>
root 26897 20310 0 04:15 pts/7 00:00:00 [phantomjs] <defunct>
root 27364 20310 0 11:55 pts/7 00:00:00 [phantomjs] <defunct>
root 27365 20310 0 11:55 pts/7 00:00:00 [phantomjs] <defunct>
root 27366 20310 0 11:55 pts/7 00:00:00 [phantomjs] <defunct>
root 27559 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 27560 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 27561 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 27620 20310 0 04:25 pts/7 00:00:00 [phantomjs] <defunct>
root 27621 20310 0 04:25 pts/7 00:00:00 [phantomjs] <defunct>
root 27622 20310 0 04:25 pts/7 00:00:00 [phantomjs] <defunct>
root 28276 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 28277 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 28278 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 28337 20310 0 04:35 pts/7 00:00:00 [phantomjs] <defunct>
root 28338 20310 0 04:35 pts/7 00:00:00 [phantomjs] <defunct>
root 28339 20310 0 04:35 pts/7 00:00:00 [phantomjs] <defunct>
root 28748 20310 0 12:15 pts/7 00:00:00 [phantomjs] <defunct>
root 28749 20310 0 12:15 pts/7 00:00:00 [phantomjs] <defunct>
root 28750 20310 0 12:15 pts/7 00:00:00 [phantomjs] <defunct>
root 29003 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 29004 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 29005 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 29057 20310 0 04:45 pts/7 00:00:00 [phantomjs] <defunct>
root 29058 20310 0 04:45 pts/7 00:00:00 [phantomjs] <defunct>
root 29059 20310 0 04:45 pts/7 00:00:00 [phantomjs] <defunct>
root 29474 20310 0 12:25 pts/7 00:00:00 [phantomjs] <defunct>
root 29475 20310 0 12:25 pts/7 00:00:00 [phantomjs] <defunct>
root 29476 20310 0 12:25 pts/7 00:00:00 [phantomjs] <defunct>
root 29731 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 29732 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 29733 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 29788 20310 0 04:55 pts/7 00:00:00 [phantomjs] <defunct>
root 29789 20310 0 04:55 pts/7 00:00:00 [phantomjs] <defunct>
root 29790 20310 0 04:55 pts/7 00:00:00 [phantomjs] <defunct>
root 30457 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 30458 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 30459 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 30514 20310 0 05:05 pts/7 00:00:00 [phantomjs] <defunct>
root 30515 20310 0 05:05 pts/7 00:00:00 [phantomjs] <defunct>
root 30516 20310 0 05:05 pts/7 00:00:00 [phantomjs] <defunct>
root 30863 20310 0 12:45 pts/7 00:00:00 [phantomjs] <defunct>
root 30864 20310 0 12:45 pts/7 00:00:00 [phantomjs] <defunct>
root 30865 20310 0 12:45 pts/7 00:00:00 [phantomjs] <defunct>
root 31180 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 31181 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 31182 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 31231 20310 0 05:15 pts/7 00:00:00 [phantomjs] <defunct>
root 31232 20310 0 05:15 pts/7 00:00:00 [phantomjs] <defunct>
root 31233 20310 0 05:15 pts/7 00:00:00 [phantomjs] <defunct>
root 31584 20310 0 12:55 pts/7 00:00:00 [phantomjs] <defunct>
root 31585 20310 0 12:55 pts/7 00:00:00 [phantomjs] <defunct>
root 31586 20310 0 12:55 pts/7 00:00:00 [phantomjs] <defunct>
root 31894 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 31895 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 31896 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 31954 20310 0 05:25 pts/7 00:00:00 [phantomjs] <defunct>
root 31955 20310 0 05:25 pts/7 00:00:00 [phantomjs] <defunct>
root 31956 20310 0 05:25 pts/7 00:00:00 [phantomjs] <defunct>
root 32616 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 32617 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 32618 20310 0 Jul09 pts/7 00:00:00 [phantomjs] <defunct>
root 32674 20310 0 05:35 pts/7 00:00:00 [phantomjs] <defunct>
root 32675 20310 0 05:35 pts/7 00:00:00 [phantomjs] <defunct>
root 32676 20310 0 05:35 pts/7 00:00:00 [phantomjs] <defunct>
新增代理网站之注册
上午有伙伴提到，可以实现注册机制。
刚好，最近也在玩Java的代理池，也想到这个问题，如果新增代理网站，如何对源代码改动最小，看了看设计模式，发现观察者模式还是比较契合，不知道对go会不会有帮助。
都用go了，直接把代理功能加上不更好么？
直接本地监听端口，为其他程序提供代理，类似于其他程序增加了squid服务
这样其他程序就不用写任何代理的逻辑，请求只需要proxypool分发/负载均衡，不更好么
data/ProxyPool.db这个文件在哪呢
; For "sqlite3" and "tidb", use absolute path when you start as service
PATH = data/ProxyPool.db
检查数据库中的ip的时候为什么不循环一直去检测，而是只检测一次？
Proxy plugin model
对每个代理，应该定义一个Interface，然后提供一个注册函数。这样只要别人实现了代理插件并进行注册，然后重新编译一下代码，就可以使用新的代理。这样方式，不用修改原包的代码，只需要添加一个导入语句即可，如：
import _ "github.com/someone/proxy-pool-plugin"没有 https 代理.
GET localhost:3000/v2/https
得到的结果是空 json 如下：
相应的，数据库中也无 https 代理 IP.
运行出现错误
你好，运行出现如下报错
C:\Users\AppData\Local\Programs\Python\Python36\python.exe F:/code/ProxyPool/run.py
Traceback (most recent call last):
File "F:/code/ProxyPool/run.py", line 1, in <module>
from proxypool.api import app
File "F:\code\ProxyPool\proxypool\api.py", line 1, in <module>
from flask import Flask,g
File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\flask\__init__.py", line 17, in <module>
from werkzeug.exceptions import abort
File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\werkzeug\__init__.py", line 151, in <module>
__import__('werkzeug.exceptions')
File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\werkzeug\exceptions.py", line 71, in <module>
from werkzeug.wrappers import Response
File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\werkzeug\wrappers.py", line 37, in <module>
from werkzeug.formparser import FormDataParser, default_stream_factory
File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\werkzeug\formparser.py", line 28, in <module>
from werkzeug.wsgi import make_line_iter, \
File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\werkzeug\wsgi.py", line 13, in <module>
import httplib
File "C:\Users\AppData\Local\Programs\Python\Python36\lib\site-packages\httplib.py", line 345
print "reply:", repr(line)
^
SyntaxError: invalid syntax
Process finished with exit code 1
open connections reached
2019/06/17 18:46:02 Unsolicited response received on idle HTTP channel starting with "HTTP/1.1 503 Too many open connections\r\nContent-Type: text/plain\r\nConnection: close\r\n\r\nMaximum number of open connections reached.\r\n"; err=
可以使用Redis作为数据库么
我留意到README里都是用mysql协议连接的，作者是没有加入对redis的支持么
go mod support
作者不打算加入Reids作为存储吗
作者不打算加入Reids作为存储吗
太多僵尸进程
跑两天新增300个僵尸进程，都是phantomjs的
Mac下执行无效
log内容如下：
2019/04/08 19:16:17 [ INFO] ProxyPool
2019/04/08 19:16:17 [FATAL] [...g/initial/initial.go:42 Database()] Fail to set test ORM engine: dial tcp 127.0.0.1:5432: connect: connection refused
2019/04/08 19:17:48 [ INFO] ProxyPool
2019/04/08 19:17:48 [FATAL] [...g/initial/initial.go:42 Database()] Fail to set test ORM engine: dial tcp 127.0.0.1:5432: connect: connection refused
phantomjs.exe
您好，关于这个内核放在什么路径呢
setting can add some default val？
配置使用ini，如果有一些默认值，零配置即可使用，是不是会更好用一点？可以考虑使用viper重构一下配置，支持一下环境变量覆盖配置，支持12因素，看起来很棒
安装提示undefined
只执行了go get 操作
跑了2个小时自己挂了
2017/05/10 10:57:55 All getters finished.
2017/05/10 10:59:28 Chan: 144, IP: 275
2017/05/10 11:01:28 Chan: 0, IP: 277
panic: runtime error: index out of range
goroutine 8554 [running]:
panic(0x6daf40, 0xc042006060)
C:/Go/src/runtime/panic.go:500 +0x1af
github.com/henson/ProxyPool/getter.IP66(0x0, 0x0, 0x0)
E:/golang/src/github.com/henson/ProxyPool/getter/66ip.go:20 +0x759
main.run.func1(0xc042172060, 0xc04222a4b0, 0x768160)
E:/golang/src/github.com/henson/ProxyPool/main.go:66 +0x32
created by main.run
E:/golang/src/github.com/henson/ProxyPool/main.go:71 +0x16b
exit status 2
新增代理网站-计划
http://www.feiyiproxy.com/?page_id=1457
已经可以采集代理IP
go get 时候报错 undefined: clog.CONSOLE
PS D:\WorkSpace\GO\proxy> go get -u github.com/henson/proxypool
github.com/henson/proxypool/pkg/setting
D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:61:11: undefined: clog.CONSOLE
D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:61:43: cannot use clog.ConsoleConfig literal (type clog.ConsoleConfig) as type clog.Initer in argument to clog.New
D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:65:14: cannot use 2 (type int) as type string in argument to clog.Fatal
D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:89:14: cannot use 2 (type int) as type string in argument to clog.Fatal
D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:98:14: cannot use 2 (type int) as type string in argument to clog.Fatal
D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:105:14: cannot use 2 (type int) as type string in argument to clog.Fatal
D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:137:27: undefined: clog.LEVEL
D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:148:15: cannot use 2 (type int) as type string in argument to clog.Fatal
D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:162:10: undefined: clog.MODE
D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:163:8: undefined: clog.CONSOLE
D:\DevEnv\GOPATH\src\github.com\henson\proxypool\pkg\setting\setting.go:163:8: too many errors
dockerfile not exists
是否可以写一个dockerfile，方便其他人本地部署呢？或者之后我提个pr？
已经正常启动，获取https代理时报错
2019/03/20 17:25:33 http: panic serving [::1]:14121: runtime error: invalid memory address or nil pointer dereference
goroutine 1023 [running]:
net/http.(*conn).serve.func1(0xc000056b40)
C:/Go/src/net/http/server.go:1769 +0x140
panic(0xb127a0, 0xa68160)
C:/Go/src/runtime/panic.go:522 +0x1c3
github.com/henson/proxypool/pkg/storage.ProxyFind(0xb9f032, 0x5, 0xba5f21)
D:/goworker/src/github.com/henson/proxypool/pkg/storage/filter.go:102 +0x5a
github.com/henson/proxypool/api.FindHandler(0xc7a400, 0xc0001e0540, 0xc0000e2800)
D:/goworker/src/github.com/henson/proxypool/api/api.go:41 +0x145
net/http.HandlerFunc.ServeHTTP(0xbd41a0, 0xc7a400, 0xc0001e0540, 0xc0000e2800)
C:/Go/src/net/http/server.go:1995 +0x4b
net/http.(*ServeMux).ServeHTTP(0xc0001b0100, 0xc7a400, 0xc0001e0540, 0xc0000e2800)
C:/Go/src/net/http/server.go:2375 +0x1dd
net/http.serverHandler.ServeHTTP(0xc000286000, 0xc7a400, 0xc0001e0540, 0xc0000e2800)
C:/Go/src/net/http/server.go:2774 +0xaf
net/http.(*conn).serve(0xc000056b40, 0xc7b8c0, 0xc0001b0fc0)
C:/Go/src/net/http/server.go:1878 +0x858
created by net/http.(*Server).Serve
C:/Go/src/net/http/server.go:2884 +0x2fb
go build 报错：pkg/storage/filter.go:90:15: undefined: RandInt
github.com/henson/proxypool/pkg/storage/filter.go # 122 添加缺省方法
//RandInt get the random numer in [min, max]
func RandInt(min, max int) int {
if min >= max || max == 0 {
return max
}
rand.Seed(time.Now().UnixNano())
//x := r.Intn(max-min) + min
x := rand.Intn(max-min) + min
//fmt.Println("RandInt: = ",x)
return x
}
有时候会闪退？
报错如下图：
parnurzeal/gorequest 这个包貌似不支持 https的代理吧
parnurzeal/gorequest 这个包貌似不支持 https的代理吧
run error
phantomjs 装了，环境看似没问题，就是起不起来，求教 -。-
➜ ProxyPool git:(master) ✗ ./ProxyPool
2017/10/23 12:55:07 Starting server :8080
2017/10/23 12:55:07 Chan: 0, IP: 35
2017/10/23 12:55:07 Before check, DB has: 35 records.
2017/10/23 12:55:07 IP66 done.
2017/10/23 12:55:07 Data5u done.
2017/10/23 12:55:07 IP181 done.
2017/10/23 12:55:08 PLP done.
panic: Post http://localhost:2015: dial tcp 127.0.0.1:2015: getsockopt: connection refused
goroutine 86 [running]:
panic(0x319340, 0xc4201afd40)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/nladuo/go-phantomjs-fetcher.(*Fetcher).GetWithOption(0xc4200dc270, 0x37bed8, 0x23, 0x38393c, 0x6c, 0x371cac, 0xc, 0xc42057c880, 0x6f15f, 0xc420036e30, ...)
/Users/xxx/Dev/go/src/github.com/nladuo/go-phantomjs-fetcher/fetcher.go:134 +0x436
github.com/nladuo/go-phantomjs-fetcher.(*Fetcher).GetWithJS(0xc4200dc270, 0x37bed8, 0x23, 0x38393c, 0x6c, 0x371cac, 0xc, 0x0, 0x0, 0x0)
/Users/xxx/Dev/go/src/github.com/nladuo/go-phantomjs-fetcher/fetcher.go:97 +0x80
github.com/henson/ProxyPool/getter.XDL(0x0, 0x0, 0x0)
/Users/xxx/Dev/go/src/github.com/henson/ProxyPool/getter/xdl.go:25 +0x1dc
main.run.func1(0xc42006e480, 0xc420158070, 0x3a5c50)
/Users/xxx/code/test/ProxyPool/main.go:66 +0x2b
created by main.run
/Users/xxx/code/test/ProxyPool/main.go:71 +0x14c
➜ ProxyPool git:(master) ✗ Error: Could not create web server listening on port 2016
Error: Could not create web server listening on port 2017
openapi support
是否可以添加openapi/swagger 的api文档，这样方便其他人generate其他语言的客户端？而不需要自己写适配接口
panic: runtime error: invalid memory address or nil pointer dereference
启动报错
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xc8 pc=0x7fdf60]
goroutine 19 [running]:
github.com/go-xorm/xorm.(*Session).Init(0xc0000ed080)
/home/chen/work/gopath/pkg/mod/github.com/go-xorm/[email protected]/session.go:94 +0x200
github.com/go-xorm/xorm.(*Engine).NewSession(...)
/home/chen/work/gopath/pkg/mod/github.com/go-xorm/[email protected]/engine.go:317
github.com/go-xorm/xorm.(*Engine).Where(0x0, 0xa025a0, 0xb8acd0, 0xc0001425d0, 0x1, 0x1, 0x0)
/home/chen/work/gopath/pkg/mod/github.com/go-xorm/[email protected]/engine.go:594 +0x57
github.com/henson/proxypool/pkg/models.countIps(0x0)
/home/chen/work/gopath/src/github.com/henson/proxypool/pkg/models/ip.go:39 +0x97
github.com/henson/proxypool/pkg/models.CountIPs(...)
/home/chen/work/gopath/src/github.com/henson/proxypool/pkg/models/ip.go:45
github.com/henson/proxypool/pkg/storage.CheckProxyDB()
/home/chen/work/gopath/src/github.com/henson/proxypool/pkg/storage/filter.go:60 +0x35
main.main.func2()
/home/chen/work/gopath/src/github.com/henson/proxypool/main.go:31 +0x20
created by main.main
/home/chen/work/gopath/src/github.com/henson/proxypool/main.go:30 +0x8a
memory error on windows
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x6815f9]
goroutine 21 [running]:
gopkg.in/mgo%2ev2.(*Session).Clone(0x0, 0xc042123f78)
C:/Users/xxx/go/src/gopkg.in/mgo.v2/session.go:1603 +0x29
github.com/henson/ProxyPool/storage.(*Storage).GetDBSession(0xc042123f38, 0xc042123fac)
C:/Users/xxx/go/src/github.com/henson/ProxyPool/storage/storage.go:30 +0x36
github.com/henson/ProxyPool/storage.(*Storage).Count(0xc042123f38, 0x0)
C:/Users/xxx/go/src/github.com/henson/ProxyPool/storage/storage.go:58 +0x53
github.com/henson/ProxyPool/storage.CheckProxyDB()
C:/Users/xxx/go/src/github.com/henson/ProxyPool/storage/filter.go:36 +0xd3
main.main.func2()
E:/git/ProxyPool/main.go:27 +0x27
created by main.main
E:/git/ProxyPool/main.go:26 +0x110
有报错
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x8997ca]
goroutine 148 [running]:
proxypool/vendor/github.com/Aiicy/htmlquery.(*NodeNavigator).NodeType(0xc42042afc0, 0x7fa97b7dc078)
/home/worker/go/src/proxypool/vendor/github.com/Aiicy/htmlquery/query.go:145 +0x2a
proxypool/vendor/github.com/antchfx/xpath.axisPredicate.func1(0xf4cc20, 0xc42042afc0, 0x41c4d8)
/home/worker/go/src/proxypool/vendor/github.com/antchfx/xpath/build.go:45 +0x4f
proxypool/vendor/github.com/antchfx/xpath.(*descendantQuery).Select.func1(0xa01320, 0xc4207c5f50)
/home/worker/go/src/proxypool/vendor/github.com/antchfx/xpath/query.go:236 +0x158
proxypool/vendor/github.com/antchfx/xpath.(*descendantQuery).Select(0xc4207c5e30, 0xf43900, 0xc42042af80, 0xc420035400, 0x8007b8f5000)
/home/worker/go/src/proxypool/vendor/github.com/antchfx/xpath/query.go:263 +0x36
proxypool/vendor/github.com/antchfx/xpath.(*filterQuery).Select(0xc42042af20, 0xf43900, 0xc42042af80, 0x9dbd80, 0x0)
/home/worker/go/src/proxypool/vendor/github.com/antchfx/xpath/query.go:526 +0x4b
proxypool/vendor/github.com/antchfx/xpath.(*descendantQuery).Select(0xc4207c5e90, 0xf43900, 0xc42042af80, 0x41bd77, 0xc4207c5f20)
/home/worker/go/src/proxypool/vendor/github.com/antchfx/xpath/query.go:226 +0xd5
proxypool/vendor/github.com/antchfx/xpath.(*filterQuery).Select(0xc42042af60, 0xf43900, 0xc42042af80, 0xc42042af80, 0x20)
/home/worker/go/src/proxypool/vendor/github.com/antchfx/xpath/query.go:526 +0x4b
proxypool/vendor/github.com/antchfx/xpath.(*descendantQuery).Select(0xc4207c5ef0, 0xf43900, 0xc42042af80, 0x0, 0xf49301)
pkg/storage 里面缺少一个rand_darwin.go mac
咱们这个检测代理是否可以用的功能有点问题
咱们获取的这个代理地址是可以用的
117.143.109.143:80
但是我用http代理请求数据无法请求
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.