misecurity / x-patrol

github泄露扫描系统

Github leaked patrol

Github leaked patrol为一款github泄露巡航工具：

1. 提供了WEB管理端，后台数据库支持SQLITE3、MYSQL和POSTGRES
2. 双引擎搜索，github code接口搜索全局github以及本地搜索例行监控的repos
3. 支持规则管理（github搜索规则及本地repos搜索规则）
4. 支持github token管理和用户管理
5. 支持在WEB中对扫描结果审核

使用方法

• 命令行参数如下：
  1. web指令表示启动web管理端
  2. scan指令表示只启动github搜索
  3. scan -m local，表示只启动本地代码搜索功能
  4. scan -m all，表示同时启动github代码搜索与本地Repos搜索功能

• 配置好conf/app.ini中的参数后使用WEB参数后启动WEB服务器。默认会监听到本地的8000端口，默认的管理员账户和密码分别为：xsec和[email protected]。

• 登录WEB管理端，录入github token、规则。

• 启动搜索功能：

• 审核结果
  1. github code搜索结果审核：
  2. 本地repos详细搜索结果审核：

更新记录

2018/5/25，修改了本地扫描的逻辑

1. 只扫描在后台添加的仓库了
2. 同时支持git远程地址和本地地址，格式如下：

x-patrol's Issues

Potential case-insensitive import collision

Due to GitHub handle change (to lowercase) for long term purpose, go get may fail fetching github.com/Unknwon/com.
Please consider take some time to update it to github.com/unknwon/com in the go.mod file.
I truly apology for the inconvenience and unintended troubles caused.

安装go

还是不行 我在服务器上安装 用源码安装

一运行就panic

./x-patrol_darwin_amd64
[0000] PANIC xsec patrol: Error 1045: Access denied for user 'xsec'@'localhost' (using password: YES)
panic: (*logrus.Entry) (0xb455e0,0xc00009c730)

goroutine 1 [running]:
github.com/sirupsen/logrus.Entry.log(0xc00009c0a0, 0xc00008ebd0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7f2300000000, ...)
/opt/apps/gopath/src/github.com/sirupsen/logrus/entry.go:126 +0x2ab
github.com/sirupsen/logrus.(*Entry).Panic(0xc00009c190, 0xc0000d3ec8, 0x1, 0x1)
/opt/apps/gopath/src/github.com/sirupsen/logrus/entry.go:194 +0x103
github.com/sirupsen/logrus.(*Entry).Panicln(0xc00009c190, 0xc0000d3f28, 0x1, 0x1)
/opt/apps/gopath/src/github.com/sirupsen/logrus/entry.go:289 +0xc6
x-patrol/models.init.0()
/opt/apps/gopath/src/x-patrol/models/models.go:70 +0x3f4

这段代码写的太迷幻了：）

func Search(rules []models.Rules) () {
	var wg sync.WaitGroup
	wg.Add(len(rules))
	client, token, err := GetGithubClient()
	if err == nil && token != "" {
		for _, rule := range rules {
			go func(rule models.Rules) {
				defer wg.Done()
//这个goroutine的意义在哪？？？？
			}(rule)

			SaveResult(client.SearchCode(rule.Pattern))
		}
		wg.Wait()
	}
}

求助如何解决，感谢

./x-patrol_darwin_amd64 scan
[0000] INFO xsec patrol: scan github code
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x15a5a9a]

goroutine 18 [running]:
x-patrol/util/githubsearch.(*Client).SearchCode(0xc000257d60, 0xc000274760, 0x16, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/lbb/go/src/x-patrol/util/githubsearch/gitclient.go:158 +0x32a
x-patrol/util/githubsearch.Search.func1(0xc000270fa0, 0xc000257d60, 0x17, 0xc000270f28, 0x6, 0xc000270f48, 0x5, 0xc000274760, 0x16, 0xc000270f70, ...)
/Users/lbb/go/src/x-patrol/util/githubsearch/search.go:67 +0x6f
created by x-patrol/util/githubsearch.Search
/Users/lbb/go/src/x-patrol/util/githubsearch/search.go:65 +0x168

task_tes.go单元测试运行报错

后来查到了原因，当执行 task_tes.go 里面的单元测试的时候，会改变当前程序正在运行的路径，当前的路径 是 task，就没办法找到 conf/app.ini 文件。

本地仓库搜索报错

Failed to git fetch repos
fatal:couldn‘t find remote ref master
fatal:the remote end hung up unexpectedly
localrepos error : exit status 128

import 方式

go 新手，最近把这个项目导入到goland 里面，发觉有一些引入，比如 models.go 里面

import (
	"x-patrol/settings"
	"x-patrol/logger"
)

IDE 提示无法引入，改为相对路径 ../settings 就没有问题，这样是正常的么，还是我哪里设置的不正确。

在Kali下go build main.go错误

build报错

401 Bad credentials

I try to scan after adding the token , but it come 401 Bad credentials.
The token is valid which can be use by curl -u username:token https://api.github.com/user

go run main.go web 报了一个错误log

go run main.go web 报了一个错误log，如下

[0000] INFO xsec patrol: Init rules, err:

Error: cannot load github.com/etsy/hound/codesearch/sparse

github.com/etsy/hound/codesearch/sparse

rename

github.com/hound-search/hound/codesearch/sparse

scan功能一直提示空指针，go语言不知道怎么修改，求大佬指点

运行无结果

root@kali:/usr/local/go/src/x-patrol# ./x-patrol_linux_amd64 scan -m all
[0000] INFO xsec patrol: scan github code and local repos
[0000] INFO xsec patrol: Complete the scan local repos, start to sleep 15m0s seconds
[0000] INFO xsec patrol: Complete the scan of Github, start to sleep 15m0s seconds
一直卡在这没反应了

功能咨询

大神好，
想问下大神，咱们的源码监控目前是否实现 增量扫描以及邮件通知功能，感谢大神回复。真心感谢大神无私的指点！

全局搜索关键字

例如 "example.com"+in:file+language:go这种关键字+描述符形式的全局搜索如何展开呢？目前看好像不支持这个特性

忽略仓库只能单次生效

忽略仓库只能单次生效
不能这次点了 忽略仓库后 永久生效

代码扫描不全

你好，我不会go，没法看具体的逻辑，搭建起来测试了一下，发现通过github.com网站，扫描我要的关键词，如“zhichengcredit.com”能搜过出好多结果，但是用这个工具扫描的时候，只能扫出几条数据来，见下面的截图

403 You have triggered an abuse detection mechanism. Please wait a few minutes before you try again

运行没有1分钟就这样
确保没有超过5000的限额
确保已经使用了token

添加github账户一直报502错误。这是什么原因啊

Github方法已不兼容

go build main.go 提示：

x-patrol/models

models/github.go:97:35: response.Remaining undefined (type *github.Response has no field or method Remaining)
models/github.go:98:31: response.Reset undefined (type *github.Response has no field or method Reset)
models/github.go:99:31: response.Limit undefined (type *github.Response has no field or method Limit)

logout点击后404

404 page not found

program can't run success

When I download this program:
cd x-patrol, then I execute "go build main.go"，I got the following errors:

main.go:29:2: cannot find package "github.com/urfave/cli" in any of:
	/usr/lib/golang/src/github.com/urfave/cli (from $GOROOT)
	/root/go/src/github.com/urfave/cli (from $GOPATH)
main.go:28:2: cannot find package "x-patrol/cmd" in any of:
	/usr/lib/golang/src/x-patrol/cmd (from $GOROOT)
	/root/go/src/x-patrol/cmd (from $GOPATH)

I think you maybe forget to write some packages that need to depend on, and the package path may be have some problem,
Please update the readme, describe the install&run part clearly, then more newbies can run you program easily.

方不方便给个表结构

如果有sqlite的表结构或者其他数据库的方便给一下么，如果不方便就算了，谢谢

请检查

[0000] PANIC xsec patrol: Error 1045: Access denied for user 'root'@'localhost' (using password: YES)
panic: (*logrus.Entry) (0xab5420,0xc420196050)

goroutine 1 [running]:
github.com/sirupsen/logrus.Entry.log(0xc420088280, 0xc420083320, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/yang/gocode/src/github.com/sirupsen/logrus/entry.go:112 +0x254
github.com/sirupsen/logrus.(*Entry).Panic(0xc420088370, 0xc420175bd0, 0x1, 0x1)
	/home/yang/gocode/src/github.com/sirupsen/logrus/entry.go:182 +0xaa
github.com/sirupsen/logrus.(*Entry).Panicln(0xc420088370, 0xc420175cc0, 0x1, 0x1)
	/home/yang/gocode/src/github.com/sirupsen/logrus/entry.go:277 +0xd5
x-patrol/models.init.0()
	/home/yang/gocode/src/x-patrol/models/models.go:70 +0x415
exit status 2

cannot load github.com/hound-search/hound/codesearch/sparse

部署于AWS海外节点

root@li:/root/go/src/x-patrol# ./build.sh
go: finding github.com/hound-search/hound/codesearch/sparse latest
go: finding github.com/hound-search/hound/codesearch latest
go get github.com/hound-search/hound/codesearch/sparse: no matching versions for query "latest"
go: finding github.com/go-macaron/cache latest
go: finding github.com/go-macaron/captcha latest
go: finding github.com/go-macaron/csrf latest
go: finding github.com/go-macaron/session latest
go: finding github.com/google/go-github/github latest
go: finding golang.org/x/oauth2 latest
go: finding github.com/hound-search/hound/codesearch/sparse latest
go: finding github.com/hound-search/hound/codesearch latest
build command-line-arguments: cannot load github.com/hound-search/hound/codesearch/sparse: cannot find module providing package github.com/hound-search/hound/codesearch/sparse

root@li:/root/go/src/x-patrol# go build main.go
go: finding github.com/hound-search/hound/codesearch/sparse latest
go: finding github.com/hound-search/hound/codesearch latest
build command-line-arguments: cannot load github.com/hound-search/hound/codesearch/sparse: cannot find module providing package github.com/hound-search/hound/codesearch/sparse

GOBIN=/usr/local/go/bin
GOROOT=/usr/local/go
GOPATH=/root/go
GO111MODULE=on

希望可以有一个搭建文档，和一个新手问题解决如果有那就太棒了

嗨，已忽略的仓库，下次scan还会被扫描出来吗

嗨，非常感谢作者能开源这个项目，我有个问题，已忽略的仓库，下次scan还会被扫描出来吗

默认密码不对

想运行下看看效果真是难

build报错，换vps build还是报错，用docker环境倒是build 成功了但是运行报错，不知道要开放几个端口，心累

我总结了个搭建文档

111

error while running go-get

Running go-get results in

root@bugdiscloseguy:~# go get github.com/MiSecurity/x-patrol package x-patrol/cmd: unrecognized import path "x-patrol/cmd" (import path does not begin with hostname)

panic: runtime error: invalid memory address or nil pointer dereference

go version go1.10.1 linux/amd64
./main scan
报错：

-bash: ./main: No such file or directory

ubuntu 14.04，使用 ./main web启动的时候报下面错误

使用token进行爬取的时候好像被封了

在用token进行爬取的时候，在测试的几次之后就爬不出东西来了，后来发现是timeout了。
请问这个问题是怎么处理的啊。
谢谢！