GithubHelp home page GithubHelp logo

checkgoogleip's Introduction

checkgoogleip

主要是用来检查哪些IP可以用在goagent上面

检查方法

  • 默认使用内置的ssl库连接到该IP,并使用cacert.pem来获取服务器证书,可以通过修改变量来支持pyOpenSSL库获取服务器证书,
  • 检查该 IP是否使用google.com的证书,如果不是该域名,则发送http请求,然后检查回应头部的Server是否为gws
  • 以响应时间排序,时间越少,排序就越前

注意

  • IP组的格式
    1.xxx.xxx.xxx.xxx-xx.xxx.xxx.xxx
    如218.253.0.80-218.253.0.90
    2.xxx.xxx.xxx.xxx/xx
    如218.253.0.80/24
    3.xxx.xxx.xxx.
    如218.253.0.
    4.xxx.xxx.xxx.xxx
    如218.253.0.80
    组与组之间可以用换行、'|'或','相隔开
  • connect超时时间可以看g_commtimeout变量,时间为5秒,握手超时请看g_handshaketimeout,时间为7秒
  • 默认会尝试使用gevent及内置的ssl库查询,好处:明显优化cpu及内存使用,最大线程数量限制了128条,可以支持pyOpenSSL库,但需要设置g_usegevent为0
  • 默认增加一些IP段检查,并且只随机检查700个IP
  • 默认会保存测试IP的结果到ip_tmperror.txt(失败)、ip_tmpno.txt(非gae可用IP),ip_tmpok.txt(gae可用IP),下次运行脚本时会预先读取,并会跳过ip_tmperror.txt与ip_tmpno.txt的查询,如果不想保留ip_tmperror.txt,则g_autodeltmperrorfile为1,如果不想保留ip_tmpno.txt,则需要设置g_autodeltmpnofile=1,如果程序正常运行结束,则会检查g_autodeltmperrorfile及g_autodeltmpnofile,并执行对应的操作,如果程序运行异常,需要关闭程序,则会保留这两个文件
  • 增加gogotest里面的部分IP列表,特别感谢该开发者
  • 程序在启动时会优先读取ip_tmpok.txt检查,如果不想优先检查,可以设置g_checklastgoogleipfirst为0
  • 程序在正常结束时会对ip_tmpok.txt的结果进行排序,当次没有检查到的IP时排在最后,第二列是时间值,以NA_开头(表示当次查询超时或没检查到),如果不想对文件排序,可以设置 g_needsorttmpokfile为0
  • 设置默认线程为60(g_maxthreads),默认最大IP延时1500毫秒(g_maxhandletimeout),默认检查可用IP数50(g_maxhandleipcnt)
  • 程序支持优先读取外部IP列表,文件名为googleip.txt,当这个文件存在时,默认不会读取脚本里面的IP列表,如果需要检查里面的IP列表,可以在新一行中填写@default,表示读取脚本预设IP列表
  • 检查SSL证书时,需要检查organizationName为google Inc,其他内容暂时被认为非google ip,如果有其他名称也可用,可以在g_organizationName中添加
  • 解决使用OpenSSL检查IP时,Ctrl-C无响应的情况
  • 命令增加IP参数,用来检查该IP的ssl证书的情况
python checkip.py xxx.xxx.xxx.xxx
use gevent to check  xxx.xxx.xxx.xxx
try connect to xxx.xxx.xxx.xxx
ssl subject:  ((('countryName', u'US'),), (('stateOrProvinceName', u'California'
),), (('localityName', u'Mountain View'),), (('organizationName', u'Google Inc')
,), (('commonName', u'google.com'),))
Exception KeyError: KeyError(41875536,) in <module 'threading' from 'D:\Python27
\lib\threading.pyc'> ignored
  • 增加IP黑名单配置,暂时把216开头的IP屏蔽
    g_blackiplist = ("216.",)

使用方法

windows

可以放在把checkip.py与checkip.bat放在goagent/local目录下面,由于cacert.pem是取自goagent/local目录里面的同名文件,因此不需要复制cacert.pem到该目录,然后执行checkip.bat
如果执行成功,会在该目录新建一个ip.txt文件,里面的IP列表就是可用列表,ip_tmperror.txt会保留查询失败的IP,ip_tmpok.txt会保留查询成功的IP

linux

把checkip.py与cacert.pem放在同一个目录,然后执行 python checkip.py 即可

由于开发时主要用python2.7版本,暂时不支持python3版本

更新说明

2014.08.13

  • 支持只保留小于最大延时的IP,支持只获取指定数量的IP列表

2014.06.27

  • 优化查询可用IP,支持随机查询IP

2014.06.13

  • 使用两个临时文件来记录程序运行过程中查询成功与失败的IP地址,方便下次使用
  • 支持gevent,但只能在关闭pyOpenSSL库时使用
  • 支持多进程方式,在pyOpenSSL库下默认使用,在gevent下禁用

2014.06.07

  • 优化线程队列的使用
  • 默认会尝试使用pyOpenSSL库,支持高并发量,并且有效地优化cpu及内存使用,程序在启动时cpu较高,几秒后会降下来
  • 设置默认线程的stack大小,减少内存消耗

2014.06.06

  • 使用队列来控制线程数量,默认改为256条线程,测试发现超过378条线程时,在windows中出现部分线程加载证书出错
  • g_ssldomain 使用元组类型,检查多种域名的证书

checkgoogleip's People

Contributors

gulucn avatar moonshawdo avatar ykg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

checkgoogleip's Issues

扫描出ip_tmpok.txt的格式问题

我扫出了一些IP后,过几天想用gogo tester测试一下,打开ip_tmpok.tx,复制扫描出的IP,再导入gogo tester的时候,除了第一个IP,其它的IP导入格式都是错误的。我用的是XP系统。请问有没有啥办法解决一下这个问题。我不是IT人士,能弄个方便的方案么?

Segmentation fault (core dumped)

c179dd6 后运行,开始connect一个ip段后出错:

segmentation falt (core dumped)

再次运行时出错:

[Thread-140]try connect to 1.179.249.147 
Exception in thread Thread-167:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "checkgoogleip-master/checkip.py", line 267, in run
    self.runJob()
  File "checkgoogleip-master/checkip.py", line 254, in runJob
    (ssldomain, costtime) = ssl_obj.getssldomain(self.getName(), ipaddr)
  File "checkgoogleip-master/checkip.py", line 168, in getssldomain
    c.do_handshake()
  File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1076, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 871, in _raise_ssl_error
    _raise_current_error()
  File "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 22, in exception_from_error_queue
    raise exceptionType(errors)
Error: [('memory buffer routines', 'BUF_MEM_grow', 'malloc failure')]

[Thread-34]ip: 1.179.248.95,CN: *.googlevideo.com 
[Thread-278]ip: 1.179.248.81,CN: *.googlevideo.com 
[Thread-34]try connect to 1.179.249.148 
[Thread-278]try connect to 1.179.249.149 
[Thread-152]ip: 1.179.248.57,CN: *.googlevideo.com 
[Thread-152]try connect to 1.179.249.150 
Segmentation fault (core dumped)

调低到128条线程依然如此,是不是内存泄漏了?
本机系统archlinux,内存1G,support pyopenssl

第一版256x3条线程依然能正常运行完,注释掉超级多的connect和timeout提示后,我也注意到了某些ip段会出现eof的握手错误,不过估计是受gfw干扰所致。

报[Errno 10055]是什么问题?

"[Thread-29]Catch IO Exception(64.18.0.28): [Errno 10055] , times:1156 ms"
开始一堆129 ip timeout似乎是没找到ok ip ,一会后就报这个了,是什么问题?xpsp3+goagent 3.2.3 谢谢!

想增加扫描结果数目

运行批命令,每次只扫出来50个结果,如果我想让它每次扫出300个,怎么样能办到?

大家的checkip正常?帮忙看看出了啥问题

界面提示如下:
C:\Program Files\GoAgent\local>"C:\Program Files\GoAgent\local\python27.exe" "C:\Program F i1es\GoAgen t\1oca1\c hec k i p.py"
[Thread-1 ]begin to get ratndotn ip
[MainThread]need create max threads count: 128
[Thread-1]load extra ip ok.line:1642.load default ip: 0
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Program Files\GoAgent\local\python27.zip\threading.py". line 810. in__bootstrap_inner
File "C:\Program Files\GoAgent\local\checkip.py". line 836. in run
File "C:\Program Files\GoAgent\local\checkip.py". line 749. in ramdomip
File "C:\Program Files\GoAgent\local\checkip.py". line 884. in splitip
UalueError: too many ualues to unpack

导入自定义的IP不能扫完

我把线程改成900,自定义IP0/16的6万5个IP扫完要8分钟左右,但是导入600多万个IP只要50分钟就扫完了,我已经把最大可用IP数量g_maxhandleipcnt改成1千万了还是没用,扫了50分钟就停住了窗口2分钟没动请问该怎么解决呢

含gws的域名也被放进ip_tmpok.txt里了

61.113.12.26 1031 gwsrv.ticstage.co.jp
108.59.50.82 1391 magik.trngws.dcs.in.gov
122.248.245.155 578 *.ciggws.net
122.248.247.89 1188 *.ciggws.net
160.96.2.63 1985 internet-stg.gwsx.gov.sg
160.96.2.67 1797 www.gwsx.gov.sg
175.41.131.13 531 *.ciggws.net
175.41.133.26 500 *.ciggws.net
175.41.140.160 750 *.ciggws.net
175.41.140.254 563 *.ciggws.net
175.41.146.34 781 *.ciggws.net
192.200.179.40 1063 www.gwsupplyco.com
203.116.15.206 484 secure1.cgwstage.com
203.116.82.37 563 vpngwsgems.mindef.gov.sg
212.113.20.244 2344 gentingws.acteol.co.uk
219.117.205.118 704 rdgws.iwink.jp

是否支持前置代理?

公司网络,需要设置代理才能连外网。可以设置前置代理,用这个工具吗?怎么设置?谢谢!

mac os failed

[4894][MainThread]support PyOpenSSL
[4894][MainThread]support multiprocess
[4894][MainThread]support fuzzy matching ssl domain
Traceback (most recent call last):
File "checkip.py", line 668, in
list_ping()
File "checkip.py", line 639, in list_ping
if checkqueue.qsize() > 0:
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 143, in qsize
return self._maxsize - self._sem._semlock._get_value()
NotImplementedError

找出来几十个IP地址,但是放到goagent里全都不能用

北京·联通。

找出来的ip(每次不太一样,但大部分差不多):

210.242.125.35|218.189.25.166|210.242.125.44|218.189.25.180|210.242.125.20|210.242.125.59|61.219.131.114|61.19.1.55|61.19.1.99|64.233.183.50|203.66.124.241|64.233.181.60|111.168.255.173|64.233.182.114|210.139.253.123|64.233.183.72|173.194.123.99|210.153.73.40|64.233.171.100|64.233.181.76|106.162.192.167|74.125.31.36|64.233.165.78|64.233.183.39|64.233.185.84|64.233.165.142|64.233.171.114|64.233.165.75|64.233.185.87|173.194.118.186|64.233.186.112|106.162.192.158|64.233.171.148|74.125.31.37|74.125.31.40|61.19.1.93|74.125.31.53|173.194.72.53|74.125.200.54|210.153.73.44|74.125.31.43|74.125.31.60|64.233.164.105|74.125.31.38

放到goagent里面,仍然全部create_ssl_connection的时候return SysCallError(-1, 'Unexpected EOF')。good_ipaddrs一直为0。有什么办法吗?GA 3.1.19

这是个什么错误?好像我下了最新的checkip之后,每次扫描完ip.txt只有 4 个ip。现在又完全扫描不了。

C:\WINDOWS\system32>"C:\Program Files\goagent-3.0\local\python27.exe" "C:\Program Files\goagent-3.0\local\checkip.py"
[MainThread]load last result,ok cnt:1482,ok file line:116,error cnt: 225204
[MainThread]need create max threads count: 60
[Thread-1]begin to get ramdom ip
[Thread-1]load last gae ip cnt: 115
[Thread-1]ramdom ip thread stopped.had check ip: 60,rest ip queue size: 66
[Thread-3]ip: 64.233.162.87,CN: *.orkut.com,O:Google Inc,svr: gws,t:2480ms,ok:0
[Thread-5]ip: 64.233.162.85,CN: *.orkut.com,O:Google Inc,svr: gws,t:2572ms,ok:0
[Thread-8]ip: 64.233.162.81,CN: sandbox.google.com,O:Google Inc,svr: gws,t:2574ms,ok:0
[Thread-6]ip: 64.233.162.82,CN: *.googlecode.com,O:Google Inc,svr: gws,t:3187ms,ok:0
[Thread-2]ip: 64.233.162.86,CN: *.orkut.com,O:Google Inc,svr: gws,t:3360ms,ok:0
[Thread-7]ip: 64.233.162.83,CN: mail.google.com,O:Google Inc,svr: gws,t:3637ms,ok:0
[MainThread]120 ip timeout
[MainThread]try to collect ssl result,check ip cnt: 126,times:51s
[MainThread]write to file C:\Program Files\goagent-3.0\local\ip.txt ok,count:0
[MainThread]sort C:\Program Files\goagent-3.0\local\ip_tmpok.txt file ok

检查SSL证书能否同时检查多个证书域名?

现在发现除了*.google.com/google.com的证书服务器可以用来做GAE之外,还发现其他另外的75个域名证书服务器也可以用于做GAE。总不能每个证书开一个进程来扫吧···
*.au.doubleclick.net
*.blogger.com
*.de.doubleclick.net
*.doubleclick.net
*.fr.doubleclick.net
*.g.doubleclick.net
*.gfsvc.com
*.google.ae
*.google.at
*.google.be
*.google.cn
*.google.ca
*.google.ch
*.google.co.in
*.google.co.il
*.google.co.jp
*.google.co.kr
*.google.co.ve
*.google.co.nz
*.google.co.za
*.google.com
*.google.com.ar
*.google.com.au
*.google.com.br
*.google.com.hk
*.google.com.ly
*.google.com.mx
*.google.com.my
*.google.com.pe
*.google.com.ph
*.google.com.pk
*.google.com.ru
*.google.com.sg
*.google.com.tr
*.google.com.tw
*.google.com.ua
*.google.com.vn
*.google.de
*.google.dk
*.google.es
*.google.fi
*.google.fr
*.google.it
*.google.nl
*.google.lt
*.google.lv
*.google.pl
*.google.pt
*.google.ro
*.google-analytics.com
*.googleadservices.com
*.googleapis.com
*.googlecode.com
*.googlegroups.com
*.googleusercontent.com
*.gstatic.com
*.jp.doubleclick.net
*.mail.google.com
*.orkut.com
*.uk.doubleclick.net
*.vp.video.l.google.com
accouts.google.com
adwords.google.com
adwords.google.sk
cert-test.sandbox.google.com
checkout.google.com
google.com
m.google.com
mail.google.com
sandbox.google.com
ssl.google-analytics.com
talk.google.com
tpc.googlesydicatio.com
upload.video.google.com
wifi.google.com
www.google.com
www.googleadservices.com

@moonshawdo

你好moonshawdo
能不能增加对于ipv6的扫描呢?

扫描多个文件时读取不到IP段

因为扫描IP库过大时会内存不够,于是就分割IP段为googleip-1.txt、googleip-2.txt、googleip-3.txt,可是扫描的时候,第一个文件扫描完,第二个开始就:ip queue size: 0,没法扫描,请问下是否有办法可以解决?

begin check googleip-2.txt
[Thread-52]begin to get ramdom ip
[MainThread]need create max threads count: 50
[Thread-52]load extra ip ok,line:1,load default ip: 0
[Thread-52]ramdom ip thread stopped.had check ip: 0,rest ip queue size: 0
[MainThread]try to collect ssl result,check ip cnt: 256,times:0s
[MainThread]write to file D:\Portable\GoAgent\checkiptools\ip.txt ok,count:0

begin check googleip-3.txt
[Thread-103]begin to get ramdom ip
[MainThread]need create max threads count: 50
[Thread-103]load extra ip ok,line:5,load default ip: 0
[Thread-103]ramdom ip thread stopped.had check ip: 0,rest ip queue size: 0
[MainThread]try to collect ssl result,check ip cnt: 256,times:0s
[MainThread]write to file D:\Portable\GoAgent\checkiptools\ip.txt ok,count:0

更改的代码从1060行开始:

if __name__ == '__main__':
    if len(sys.argv) > 1:
        checkip(sys.argv[1])
    else:
        files = os.listdir(g_filedir)
        i = 0
        for item in files:
            if "googleip-" in item:
                i = re.findall(r'([0-9]+)',item)[0]
                if os.path.exists("googleip-%s.txt" % i):
                    g_googleipfile = os.path.join(g_filedir,"googleip-%s.txt" % i)
                    print "\nbegin check googleip-%s.txt" % i
                    list_ping()
                    if not os.path.exists(g_filedir + "/tmp"): os.mkdir("tmp")
                    if os.path.exists("ip_tmperror.txt"): os.rename("ip_tmperror.txt", "ip_tmperror-%s.txt" % i)
                    if os.path.exists("ip_tmpno.txt"): os.rename("ip_tmpno.txt", "ip_tmpno-%s.txt" % i)
                    if os.path.exists("ip_tmpok.txt"): os.rename("ip_tmpok.txt", "ip_tmpok-%s.txt" % i)
                    if os.path.exists("ip_tmperror-%s.txt" % i): shutil.move("ip_tmperror-%s.txt" % i, "tmp/")
                    if os.path.exists("ip_tmpno-%s.txt" % i): shutil.move("ip_tmpno-%s.txt" % i, "tmp/")
                    if os.path.exists("ip_tmpok-%s.txt" % i): shutil.move("ip_tmpok-%s.txt" % i, "tmp/")
            elif item == 'googleip.txt' and i == 0:
                list_ping()

error: [Errno 10057] A request to send or receive data was disallowed because th e socket is not connected and (when sending on a datagram socket using a sendto call) no address was supplied

Exception in thread Thread-141:
Traceback (most recent call last):
  File "C:\Downloads\goagent-3.0\local\python27.zip\threading.py", line 810, in
__bootstrap_inner
  File "C:\Downloads\goagent-3.0\local\checkip.py", line 238, in run
  File "C:\Downloads\goagent-3.0\local\checkip.py", line 222, in runJob
  File "C:\Downloads\goagent-3.0\local\checkip.py", line 201, in getssldomain
  File "C:\Downloads\goagent-3.0\local\python27.zip\ssl.py", line 292, in shutdo
wn
  File "C:\Downloads\goagent-3.0\local\python27.zip\socket.py", line 224, in met
h
error: [Errno 10057] A request to send or receive data was disallowed because th
e socket is not connected and (when sending on a datagram socket using a sendto
call) no address was supplied

try to collect ssl result
write to file C:\Downloads\goagent-3.0\local\ip.txt ok,count:0
Press any key to continue . . .

OS: windows xp sp3
Goagent Version: goagent/goagent@e0c5b1e
Checkgoogleip Version: 1434c47

最近扫出来的IP提示ok但是并不能正常使用

扫描很快,一会儿就出来一堆可用的地址,放进去之后goagent也说是good ip,但是打开网页时提示:

The request method POST is inappropriate for the URL /_gh/. That’s all we know

是不是在判断ip是否可用时还要加一些条件?

一个修改版本的功能建议采用

https://code.google.com/p/goagent/issues/detail?id=19050
修改内容:
1、可设置参数提前,方便修改。
2、输出中文化。
3、增加 ip_tmpnot 区分非 google IP。
4、清空内置 IP 库,使用外挂 eip.txt 自定义 IP。
5、去除 IP 输出最后一个“|”。
使用方法:
下载 checkip.bat、checkip.py、eip.txt 放入 goagent 目录,运行 checkip.bat,结果输出到 ip.txt,请自行粘贴到配置文件。

IP 库放在一个文件中这个功能是否能够合并进来?
另外,能否自动整理ip_tmpok.txt,删除相同的IP,只保留最后一次的记录?

我的内存太小,无法完成大的 IP 地址段的扫描,可否提供减少内存占用的方法

问题与求助

大的 IP 地址段如 15.0.0.0-23.15.255.255 有 1.3 亿 IP 地址包含于其中,我的内存扫不到四分之一就满负荷,以至于无法完成此 IP 段的扫描,可否提供减少内存占用的方法?

我想到的解决方案

  • 方案 1

    拆分 IP 段,把大段 IP 地址拆分成 15.0.0.0-15.1.0.0, ..., 23.15.1.255-23.15.255.255 这样无数个小 IP 段并把它们写到一个文件内,每次只测其中一个 IP 段,测完后清空内存

  • 方案 2

    去掉 temp*.txt 内容留在内存中的功能, IP 地址按从小到大的顺序测

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.