s045pd / darknet_chinesetrading

🚇暗网中文网监控爬虫(DEEPMIX)

Hey bro, good to see you.

• 🔭 I’m currently working on [some where]
• 🌱 I’m currently learning Fullstack and Pentest
• 💬 Ask me about anything here! I am happy to help
• 📫 How to reach me: [email protected]
• ⚡ Fun fact: type(...).mro()[-1] == object

⌨️ Things I code with

🤡 Funny code you will learn

darknet_chinesetrading's Issues

bash restart_celery.sh on README.md

I can't find this file-restart_celery.sh, is it that restart_task.sh?

request 的代理设置问题

self.proxy_url = 'socks5h://127.0.0.1:9050'  #设置tor的端口
        self.session.proxies = {    
            'https': self.proxy_url,
            'http': self.proxy_url
        }

这样子设置代理mac下不会成功，使用

self.proxies = {
            'https': self.proxy_url,
            'http': self.proxy_url
        }
session.get(f'http://{target.domain}',proxies=self.proxies)

tor代理成功

页面上url 无法打开

页面上的: http://whereme.top:3000/dashboard/snapshot/CHljrHbs1PfDdb9KRZSoa67e9h3Cjy0O
无法访问

连接不上

SOCKSHTTPConnectionPool .(host='deepmixbf6xqt3m7kagmurdt4v43f2h3doc23h7hrkjlroovyjsvseqd.onion', port=80):
连接不上啊

tor代理一直无法登陆

我自己搭建的tor服务器，注册没有问题，登陆一直都是提示：A. 您的输入有误, 请重新输入. B. 您的账户访问存在问题 或者 跳回切换页面
排错过程：账户在洋葱浏览器登陆正常，另外在chrome浏览器使用这个tor代理也是一样问题

能帮助一下吗？

暗网中文网的注册验证方式好像有所改变

暗网中文网的注册验证方式好像有所改变，导致自动注册自动登陆失败。无法进行爬取

联系作者

能否联系作者

docker、docker-compose版本是多少

docker、docker-compose版本是多少，我看docker-compose.yml里面的version是3.9。3.9对应哪个版本，官网都没给，盲目升docker-compose又报python2不支持，可以给个详细的安装教程吗？跪求。

请问各位大佬，这代码还能用吗？ 我现在部署一直报错

具体错误如下：
[ make_new_tor_id]: SOCKSHTTPConnectionPool(host='icanhazip.com', port=80): Max retries exceeded with url: / (Caused by ..

请问Linux上搭建和Mac搭建是一样的步骤吗

我用的ubuntu 19，brew安装的tor好像只能在Mac上启动？

无法登录，网站认证失败

是不是darknet把登录修改了导致无法登录

where is restart_tor.sh??

TG配置的问题

请问一下.env.default文件里的第二第三项是填写什么呢？

有个小问题

偶尔报这个，这破网站的缘故吧

你看怎么处理更好，我临时这么写处理了下

    def get_details(self, url, muti, name, page, index_str, referer_url):
        bs_data = ''
        while True:
            resp = self.session.get(
                url, headers={"Referer": referer_url, "Cookie": self.get_cookie_string()}
            )
            info(resp.headers.get("Set-Cookie", ""))
            resp.encoding = "utf8"
            if not self.check_if_need_relogin(resp):
                return

            bs_data = bs_4(resp.text, "lxml")
            if bs_data.select_one('title').text != '出错了.':
                break
            else:
                warning('重新获取一次')

Suggest to loosen the dependency on telepot

Dear developers,

Your project DarkNet_ChineseTrading requires "telepot==12.7" in its dependency. After analyzing the source code, we found that the following versions of telepot can also be suitable without affecting your project, i.e., telepot 12.6. Therefore, we suggest to loosen the dependency on telepot from "telepot==12.7" to "telepot>=12.6,<=12.7" to avoid any possible conflict for importing more packages or for downstream projects that may use ddos_script.

May I pull a request to further loosen the dependency on telepot?

By the way, could you please tell us whether such dependency analysis may be potentially helpful for maintaining dependencies easier during your development?

---

Details:

Your project (commit id: e377918) directly uses 5 APIs from package telepot.

telepot.api.set_proxy, telepot.__init__.Bot.sendLocation, telepot.__init__.Bot.sendMessage, telepot.__init__.Bot.sendPhoto, telepot.__init__.Bot.__init__

Beginning fromwhich, 30 functions are then indirectly called, including 23 telepot's internal APIs and 7 outsider APIs as follows:

[/s045pd/DarkNet_ChineseTrading]
+--telepot.api.set_proxy
|      +--urllib3.PoolManager
|      +--urllib3.make_headers
|      +--urllib3.ProxyManager
+--telepot.__init__.Bot.sendLocation
+--telepot.__init__.Bot.sendMessage
+--telepot.__init__.Bot.sendPhoto
+--telepot.__init__.Bot.__init__

Since all these functions have not been changed between any version for package "telepot" from [12.6] and 12.7. Therefore, we believe it is safe to loosen the corresponding dependency.

这个程序还能用嘛

看到搭建教程有点简单，三句代码搞定，请问作者，现在这个程序还能跑嘛？

老哥 能加一下微信或者QQ telegram吗?有些问题想问下

received exception "peek of closed file"

你好，我运行的时候有两个错误请帮忙指教一下。
我的运行环境是centos7里的pythoon3.8虚拟环境
tor因为之前搭梯子运行过，所以使用的端口是9050和9051，我把本程序里能搜到"915"的地方都改为了“905”。
第一个错误是我运行会报错：
Error while receiving a control message (SocketClosed): received exception "read of closed file"

第二个错误是我莫名其妙注册成功后，登录的时候验证码识别不对导致登录错误，但等待10秒后直接就退出了，没有循环登录。

辛苦帮忙，多谢。

run.py 报错

[2019-05-21 11:40:34,395][Cursor->get_last_domain]: list index out of range
[2019-05-21 11:40:34,396]New Tor ID
[2019-05-21 11:40:34,402][ make_new_tor_id]: no passphrase provided
[2019-05-21 11:40:34,402]Cache Timeout!
[2019-05-21 11:40:34,402]Domain: deepmix3m7iv2vcz.onion
[2019-05-21 11:40:34,402]Already Cleaned Session Cookies.
[2019-05-21 11:40:34,402]Fetch Main Page: http://deepmix3m7iv2vcz.onion
[2019-05-21 11:40:34,471]Refresh Checking
Unable to round-trip http request to upstream: address deepmix3m7iv2vcz.onion: no suitable address found
[2019-05-21 11:40:34,473][Parser->get_login_and_reg_payload]: 'NoneType' object has no attribute 'attrs'
[2019-05-21 11:40:34,473]'NoneType' object has no attribute 'attrs', retrying in 2 seconds...

跑了一段时间了,今天登陆出错了,我看了字段也没有更换

Auth Faild: 暗网交易市场 - 用户控制面板 - 登录 暗网交易市场 中文暗网担保交易 登录 提交的表单无效， 请再尝试一次。 用户名： 密码： 在这次登录中隐藏我的在线状态 注册...

JUST_UPDATE设置无效的一个问题

default.py 里面定义
just_update = _ _("JUST_UPDATE", True)
双下划线本身定义了一个转小写， _ _ = lambda x, y: _(x, str(y)).lower() == str(y).lower()
但是这样设置的话just_update = __("JUST_UPDATE", False)，False转小写false，false 再转bool值，还是为True了。no_porn_img、send_for_test、debug和limit_log可能都会有这个问题。
直接修改：
just_update = _("JUST_UPDATE", True)

你怎么不上telegram了

暗网交易论坛修改的登录策略

如题，登录页面需要三个下一步，才能进入登录页面。

最近貌似爬取不到数据了

是不是暗网又更新了，最近这几天爬取不到数据，还有个问题就是脚本运行一段时间会sleep然后就不启了

telegram send message failed

[2019-09-19 05:24:14,212]telegram_with_pic: ('Bad Request: chat not found', 400, {'ok': False, 'error_code': 400, 'description': 'Bad Request: chat not found'})

催更了，暗网更新了

加个WX:vvvvvvv888

老铁更换tor身份

self.SaveError("step2.html", step2resp)
step2 = jq(step2resp.text)
token = step2('input[name="form_token"]').attr("value")
creation_time = step2('input[name="creation_time"]').attr("value")
qa_answer = re.findall("请在右边框中输入： (.*?)：", step2resp.text)[0]

这里出现越界的错误,看保存的step2.html中显示需要重新打开tor浏览器,或是换个身份.我用的纯tor,请问如何更换tor的新身份

爬虫凉了吗

数据怎么只显示到12月21号的了

怎么实现不从情报开始爬取

我用的surge 抓取不到了

[2019-03-20 18:40:21,617] >>> INFO DarkNet_ChineseTradingNetwork: Start Regging
[2019-03-20 18:40:21,699] >>> INFO DarkNet_ChineseTradingNetwork: RegPage Confirmed. False
[2019-03-20 18:40:21,700] >>> INFO DarkNet_ChineseTradingNetwork: Got Token: None
[2019-03-20 18:40:21,700] >>> INFO DarkNet_ChineseTradingNetwork: Got Create_time: None
[2019-03-20 18:40:21,782] >>> INFO DarkNet_ChineseTradingNetwork: Html Log Saved to datas/step2.html
[2019-03-20 18:40:21,784] >>> WARNING retry.api: list index out of range, retrying in 2 seconds...

docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

root@goorm:/workspace/DarkNet_ChineseTradi/DarkNet_ChineseTrading(new_site_2019)# docker-compose build --pull && docker-compose --env-file .env.default up

`
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
return self.version(api_version=False)["ApiVersion"]
File "/usr/local/lib/python3.7/site-packages/docker/api/daemon.py", line 181, in version
return self._result(self._get(url), json=True)
File "/usr/local/lib/python3.7/site-packages/docker/utils/decorators.py", line 46, in inner
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 237, in _get
return self.get(url, **self._set_request_timeout(kwargs))
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 543, in get
return self.request('GET', url, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/docker-compose", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/compose/cli/main.py", line 81, in main
command_func()
File "/usr/local/lib/python3.7/site-packages/compose/cli/main.py", line 200, in perform_command
project = project_from_options('.', options)
File "/usr/local/lib/python3.7/site-packages/compose/cli/command.py", line 70, in project_from_options
enabled_profiles=get_profiles_from_options(options, environment)
File "/usr/local/lib/python3.7/site-packages/compose/cli/command.py", line 153, in get_project
verbose=verbose, version=api_version, context=context, environment=environment
File "/usr/local/lib/python3.7/site-packages/compose/cli/docker_client.py", line 43, in get_client
environment=environment, tls_version=get_tls_version(environment)
File "/usr/local/lib/python3.7/site-packages/compose/cli/docker_client.py", line 170, in docker_client
client = APIClient(use_ssh_client=not use_paramiko_ssh, **kwargs)
File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 197, in init
self._version = self._retrieve_server_version()
File "/usr/local/lib/python3.7/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
f'Error while fetching server API version: {e}'
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))`

请问这个是怎么回事呢

restart_task.sh

restart_task.sh 这个执行结果:
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

应该有点问题

老哥,ss已经可以翻墙,tor我也开开了,我在服务器是上出现了SOCKSHTTPConnectionPool链接失败

SOCKSHTTPConnectionPool(host='deepmixaasic2p6vm6f4d4g52e4ve6t37ejtti4holhhkdsmq3jsf3id.onion', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.contrib.socks.SOCKSConnection object at 0xb418bb0c>: Failed to establish a new connection: 0x01: General SOCKS server failure',)), retrying in 2 seconds...
我服务器是ubunt的,是不是我tor安装的有问题,我的tor的配置文件的目录有2个文件
torrc torsocks.conf这两个配置文件
我本地的mac是ok的

readme 错误

readme里说的config_dev.py 实际上是conf_dev.py
readme里说的config.py 实际上是conf.py

验证码大佬发下力

尝试不一样的二值化处理，甚至用jTessBoxEditorFX训练了模型，但识别率还是很低。

二值化后的字符

#!/usr/bin/env python
# -*- coding:utf-8 -*-
from PIL import Image
import pytesseract
import os

def verify(png_name):
    img = Image.open(png_name)
    width = img.size[0]
    height = img.size[1]

    #逆序取像素值TOOP
    data = {}
    for w in range(0, width):
        for h in range(0, height):
            rgb = img.getpixel((w, h))
            key = str(rgb[0]) + str(rgb[1]) + str(rgb[2])
            if data.get(key) is None:
                data[key] = 0
            data[key] += 1

    vals = list(data.values())
    vals.sort(reverse=True)

    keys = []
    #取除了背景色外的最多的TOP4颜色，即为4位验证码
    for k,v in data.items():
        if v == vals[1] or v == vals[2] or v == vals[3] or v == vals[4]:
            keys.append(k)
    # 黑白二值化
    for w in range(0, width):
        for h in range(0, height):

            rgb = img.getpixel((w, h))
            key = str(rgb[0]) + str(rgb[1]) + str(rgb[2])
            if key not in keys:
                img.putpixel((w, h), (255, 255, 255, 255))
            else:
                img.putpixel((w, h), (0, 0, 0, 255))

    #img.save('./11.png')
    t = pytesseract.image_to_string(img, lang='snum').replace(" ", "")
    #t = pytesseract.image_to_string(img, lang='all').replace(" ", "")
    print(t.strip())
    #ts = t.strip()
    #print(ts)
    #img.save('/tmp/c3/1%s.png' %ts)

又停止推送了吗

10月27之后没数据量

地址是不是挂了

deepmix4izfgaal2mkfpn3cbjxxcs6wyp3lcgp6ksjhtt75vn2gangqd.onion用tor浏览器也打不开(已连接成功)