liuroy / github_spider Goto Github PK
View Code? Open in Web Editor NEW使用代理调用github API爬去用户数据
github_spider's People
Contributors
Stargazers
Watchers
Forkers
wycg1984 kevien jelver veterun smallmeet jessen163 xiaosimao moonsea vianer warren288 11figure satchel9 brian-it cht2014 c0ntr01 mmczoo rerui laike23 cshanxiao xunux jn7163 awesome-archive ltc210111 zy0001 jiajie999 zhaochl eternal1025 lidapang michaelfeng87 zengyee lybwb liusheding smalllee6 xvhuanlin lxjerry s0x06 mengmeng7183 whitefirer yongxiongwei wutenghu huieric 15863004186sunchi xujun05 supermanqin yzsunlei qoiuy zxser lishixin7 ruo2012 bytesnomad goodluckwlx xzk-seu xyjsdo chenyanmo wdh5151 twlzyg yuanqingmei caoe zjt613 xsgchao ldy1691github honggaofan yihuneo hhy5277 kwb0716 konnomiya lunvs tianyichow 13jqq zhangweipu fym-yy linzy1004 yangruolalala dmdaguan xrosliang zhaoqin-100 agclkawauso ericqjj evayo yangxin1994 lohasle yuimaster qyfabc jialin72 huacaixuezhang beetrootwanggithub_spider's Issues
消费者执行不了
process_list = map(lambda x: Process(target=x.run), consumer_list)
map(lambda p: p.start(), process_list)
map(lambda p: p.join(), process_list)
无法执行，立马退出
mongo存储切换为celery异步任务
能否做成扫描CODE的，可以自己定义扫描条件
如题，，主要作用是可以用于各公司扫描自己公司敏感信息是否有泄露在github，，这样受众应该挺广的。
关于爬虫网络
你好，请问那个用户网为什么能保证无论从哪一个用户出发都能得到所有网络的用户呢？
你的路由没有绑定队列，发送不了消息到相应的队列中
你的路由没有绑定队列，发送不了消息到相应的队列中
redis查询访问的url切换为异步方式
没有代理，容易被屏蔽
publish.py
publish为空，感觉是半成品
优化代理管理方式
补充文档
rabbitmq unbound
# python queue/main.py
<unbound Queue user -> <unbound Exchange u''(direct)> -> >
<unbound Queue repo -> <unbound Exchange u''(direct)> -> >
<unbound Queue follower -> <unbound Exchange u''(direct)> -> >
<unbound Queue following -> <unbound Exchange u''(direct)> -> >
hi,执行main.py之后只显示前半部分，程序也跑不下去 请问是rabbitmq的绑定关系的问题吗？
测试环境：
- CentOS Linux release 7.2 + Python 2.7.5
- Ubuntu 14.04.5 LTS + Python 2.7.13
生产者消费者模式，多进程提高爬虫效率
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.