opennaslab / kubespider Goto Github PK
View Code? Open in Web Editor NEWA global resource download orchestration system, build your home download center.
Home Page: https://kubespider.netlify.app/
License: Apache License 2.0
A global resource download orchestration system, build your home download center.
Home Page: https://kubespider.netlify.app/
License: Apache License 2.0
可否下载m3u8地址视频,需要批量下载那种
官方群晖流程报错如下
docker: Error response from daemon: Bind mount failed: '/volume2/kubespider/.config' does not exists.
环境:群晖 920
流程:参考官方的群晖部署流程,进入 bash install 环节就一直报这个错误,容器倒是都安装好了,可是报错后就没法看到 aria2 的链接密钥。
尝试:
Currently, the notification is unfriendly. Let's improve it.
Currently, the notification is unfriendly. Let's improve it.
使用的是docker run 收到安装,aira2使用的是以前就有的不是脚本自带的
配置如下:
aria2:
type: aria2_download_provider
enable: true
download_base_path: "/downloads/"
rpc_endpoint_host: http://ip
rpc_endpoint_port: '6800'
secret: pass
priority: 1
填域名能收到数据包。日志提示
2023-04-25 15:48:22,127-ERROR: 125.70.202.219 - - [25/Apr/2023 15:48:22] code 400, message Bad request version ('ºº\x13\x01\x13\x02\x13\x03À+À/À,À0̨̩À\x13À\x14\x00\x9c\x00\x9d\x00/\x005\x01\x00\x01\x93\x1a\x1a\x00\x00\x00\x00\x00\x14\x00\x12\x00\x00\x0f公网域名替换显示\x00\x17\x00\x00ÿ\x01\x00\x01\x00\x00')
2023-04-25 15:48:22,127-INFO: 125.70.202.219 - - [25/Apr/2023 15:48:22] "üq&<lhÂäüø¸û[øQdøÄiægýkÁ}! £VWns@ÏÜA4V:$õe໡wH8tÙ
y0
ººÀ+À/À,À0̨̩ÀÀ公网域名替换显示ÿ" HTTPStatus.BAD_REQUEST -
2023-04-25 15:48:22,142-ERROR: 125.70.202.219 - - [25/Apr/2023 15:48:22] code 400, message Bad request version ('JJ\x13\x01\x13\x02\x13\x03À+À/À,À0̨̩À\x13À\x14\x00\x9c\x00\x9d\x00/\x005\x01\x00\x01\x93úú\x00\x00\x00\x00\x00\x14\x00\x12\x00\x00\x0f公网域名替换显示\x00\x17\x00\x00ÿ\x01\x00\x01\x00\x00')
2023-04-25 15:48:22,143-INFO: 125.70.202.219 - - [25/Apr/2023 15:48:22] "üïÐsM#D`v/ûÆEB8ù Z=Ä_»ìîH AíçOͬûWæá
vÞöÓç3"¯^ JJÀ+À/À,À0̨̩ÀÀúú公网域名替换显示ÿ" HTTPStatus.BAD_REQUEST -
填IP地址的话倒是正常,公网IP,和内网IP都是正常的
023-04-25 15:51:38,820-INFO: 10.253.0.4 - - [25/Apr/2023 15:51:38] "GET /healthz HTTP/1.1" 200 -
@miRemid 有兴趣来搞搞这个吗
试了几次,姿势不对啊
Currently, the download trigger is only executed once, which may not be optimal. It would be better to attempt it at least three times.
Currently, the download trigger is only executed once, which may not be optimal. It would be better to attempt it at least three times.
我想尝试改善下二开体验,开发时支持代码热更新,结果发现当前指定的 python版本是3.1
而 watchdog 只支持,2.6,2.7 以及 3.4 以上的版本,原因似乎是 3.1 collections 模块中不存在的 MutableSet 属性。
我找了个 downloaders 的子项目,直接升到 3.8 去似乎没有问题
直接 docker-compose up 可以启动 /app 内的文件改变会热更新
配置怎么改 , 运行脚本如下提示:
Unable to find image 'cesign/aria2-pro:latest' locally
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.
[root@localhost kubespider]# ls
Invite link: https://t.me/+RR09tPuCvI9iNTc1
2023-06-20 22:59:42,941-ERROR: Exception on /api/v1/download [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2529, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/app/kubespider/core/webhook_server.py", line 22, in decorated
return func(*args, **kwargs)
File "/app/kubespider/core/webhook_server.py", line 92, in download_handler
err = download_trigger.kubespider_downloader.download_file(source, path, link_type)
File "/app/kubespider/core/download_trigger.py", line 88, in download_file
return self.handle_magnet_download(url, path, downloader_list, extra_param)
File "/app/kubespider/core/download_trigger.py", line 129, in handle_magnet_download
provider.load_config()
File "/app/kubespider/download_provider/xunlei_download_provider/provider.py", line 73, in load_config
with open(token_js_path, 'r', encoding='utf-8') as js_file:
FileNotFoundError: [Errno 2] No such file or directory: '/root/.config/dependencies/xunlei_download_provider/get_token.js'
感觉是创建 container 的时候没有映射对路径? 我是把 get_token.js
复制到了映射的目录里,然后改了一下配置就可以下载了
kubespider/core/config_handler.py
line: 143
The config_path is missing when determining kubespider.yaml
file exists
aria2 bt下载不如qbittorent好用
Support automatic downloading of updated videos by bloggers of Bilibili.
Support automatic downloading of updated videos by bloggers of Bilibili.
不支持Docker直接部署吗?
我非常喜欢这个项目,目前有没有用户交流群,很想参与进来,啊啊啊啊啊啊
feat: download YouTube channel video automatically
feat: download YouTube channel video automatically
The magic source provider has significant potential for development, and therefore we should allocate a space to store the configuration of every adapted website.
Where we should put this demo configuration? cc @qingchoulove
Send a notification to the user when a task is successfully added, or when a download is accepted from rss
现有模式,资源站(source_provider)依赖开发人员为每一个网站适配,成本较高。
是否有模式可以通过配置项支持有相同模式的资源站。
根据source_provider.provider.SourceProvider
定义,需要实现should_handle
和get_links
两个关键方法。
阅读MeijuttSourceProvider
代码,get_links
其实是通过BeautifulSoup
进行解析。
综上,是否可以通过BeautifulSoup
的select
支持的能力,通过用户配置xpath
或css selector
来支持不同资源站。
例如:
xpath: /html/body/div[2]/div[6]/div/div/div[3]/div[1]/ul/li
css selector: body > div.warp > div.o_cn2 > div > div > div.tabs-list.current-tab > div.down_list > ul > li > input
xpath: //*[@id="downlist"]/table/tbody/tr/td/a
css selector: #downlist > table > tbody > tr > td > a
可以通过chrome devtool进行预选则,然后根据实际情况进行调整。
大家讨论下配置项:
是否还有其他的?
@qingchoulove @xiaozhikang0916 @miRemid @ijwstl
我目前自己部署了几个qb和aria2容器,用来分别下载不同类别的资源,如qb-1下载pt站的,qb-2下载mikan的动画资源。
请问现在是否支持这样的指定下载器功能?
现在在pr合入之后,上传docker的workflow基本都是失败的:actions,并且每次都在dockerhub上覆盖旧的镜像tag,较难随时更新版本使用,考虑优化一下上传的workflow?
To make this project more international, we should provide an English version. This will encourage the participation of foreign developers.
doc: Provide the english version of all the docs
Xxx
RT
系统Unraid, docker-compose.yml参照教程有做修改
services:
kubespider:
image: cesign/kubespider:latest
depends_on:
- aria2
environment:
- PUID=1000
- PGID=100
- TZ=Asia/Shanghai
ports:
- 3080:3080
volumes:
- /mnt/user/Files/Config/kubespider/config:/app/.config
networks:
- kb
aria2:
container_name: aria2-qb
image: abcminiuser/docker-aria2-with-webui:latest-ng
network_mode: "bridge"
ports:
- 6800:6800
- 6880:80
volumes:
- /mnt/user/Files/Download/kubespider:/data
- /mnt/user/Files/Config/kubespider/aria2:/conf
environment:
- PGID=100
- PUID=1000
restart: unless-stopped
networks:
kb:
name: kb
浏览器插件Aria2 explorer已经安装,能够正常连接
在测试网址https://www.meijutt.tv/content/meiju28582.html 右键触发下载后,aria2没有显示正在下载的任务
docker logs kubespider日志如下:
2023-07-06 21:44:37,432-INFO: meijutt tv link is:https://www.meijutt.tv/content/meiju28582.html
2023-07-06 21:44:38,294-INFO: meijutt find magnet:?xt=urn:btih:96EC4CB3707382B9945D...
2023-07-06 21:44:38,294-INFO: meijutt find magnet:?xt=urn:btih:9B762A8C71AEFE413DC3...
2023-07-06 21:44:38,294-INFO: meijutt find magnet:?xt=urn:btih:3F4F1EFB8104B557B66F...
2023-07-06 21:44:38,295-INFO: Find new resource:meijutt_source_provider/magnet:?xt=urn:btih:96EC4CB3707382B9945D...
2023-07-06 21:44:38,299-INFO: filtering downloader by name None type None, result ['aria2']
2023-07-06 21:44:38,302-INFO: download link type magnet, with provider size 1
2023-07-06 21:44:38,302-INFO: Download mangent file with downloader(aria2)
2023-07-06 21:44:38,306-INFO: Start magnet download:magnet:?xt=urn:btih:96EC4CB3707382B9945DA60439DD0B1F19A187D3
2023-07-06 21:44:38,309-WARNING: Please ensure your aria2 server is ok:HTTPConnectionPool(host='127.0.0.1', port=6800): Max retries exceeded with url: /jsonrpc (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1496c7a8c430>: Failed to establish a new connection: [Errno 111] Connection refused'))
2023-07-06 21:44:38,309-WARNING: Download torrent file error:HTTPConnectionPool(host='127.0.0.1', port=6800): Max retries exceeded with url: /jsonrpc (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x1496c7a8c430>: Failed to establish a new connection: [Errno 111] Connection refused'))
环境信息:在宿主机上挂载了一个 NFS 文件夹,其目录为 /root/kubespider/nas,然后安装该工具,登录aira 的容器,可以正常创建文件和读取已有文件,担心和这个 NFS 挂载有关所以特意说明
下载时提示错误,容器输出的日志信息:
05/06 23:08:19 [NOTICE] Serialized session to '/config/aria2.session' successfully.
[#d8fb97 56KiB/56KiB(100%) CN:5 SD:1]
05/06 23:08:20 [NOTICE] Download complete: [MEMORY][METADATA]梅菲尔女巫.Mayfair.Witches.S01E01.The.Witching.Hour.1080p.H265-NEW字幕组.mp4
05/06 23:08:20 [NOTICE] Saved metadata as /downloads/57c7d3f11ac1c98cc9ef52a3ccf5091e14b97854.torrent.
05/06 23:08:20 [NOTICE] Serialized session to '/config/aria2.session' successfully.
05/06 23:08:20 [ERROR] CUID#86 - Exception caught while allocating file space.
Exception: [AbstractDiskWriter.cc:557] errNum=95 errorCode=17 fallocate failed. cause: Operation not supported
05/06 23:08:20 [ERROR] CUID#86 - Download not complete: /downloads/梅菲尔女巫.Mayfair.Witches.S01E01.The.Witching.Hour.1080p.H265-NEW字幕组.mp4
05/06 23:08:20 [NOTICE] Download GID#6d32f38fc9378fd2 not complete: /downloads/梅菲尔女巫.Mayfair.Witches.S01E01.The.Witching.Hour.1080p.H265-NEW字幕组.mp4
05/06 23:08:20 [WARRING] Number of files is zero, maybe a Magnet Link.
05/06 23:08:20 [INFO] Download task error, deleting files...
removed '/downloads/梅菲尔女巫.Mayfair.Witches.S01E01.The.Witching.Hour.1080p.H265-NEW字幕组.mp4.aria2'
removed '/downloads/梅菲尔女巫.Mayfair.Witches.S01E01.The.Witching.Hour.1080p.H265-NEW字幕组.mp4'
05/06 23:08:20 [INFO] Deleting .torrent file ...
removed '/downloads/57c7d3f11ac1c98cc9ef52a3ccf5091e14b97854.torrent'
05/06 23:08:20 [INFO] Deleting empty directory ...
用的就是示例中的资源
请问我应该如何排查,谢谢。
通过修改download_provider.yaml
aria2:
type: aria2_download_provider
enable: false
download_base_path: "/downloads/"
rpc_endpoint_host: http://127.0.0.1
rpc_endpoint_port: 6800
secret: kubespider
priority: 3
qbittorrent:
type: qbittorrent_download_provider
enable: true
download_base_path: "/downloads/"
http_endpoint_host: http://127.0.0.1
http_endpoint_port: 8080
username: admin
password: adminadmin
verify_webui_certificate: false
priority: 0
tags:
- kubespider
但是自己测试,还是不行,手动添加任务到qbittorrent可以。
然后访问
http://192.168.0.111:3080/api/v1/downloadproviders
回复:
{
"aria2": true,
"qbittorrent": false,
"transmission": false,
"xunlei": false,
"youget": false,
"yt-dlp": false
}
Automatically download the video pushed by telegram channel.
Automatically download the video pushed by telegram channel.
/assign @DDSRem
浏览器插件的方式依赖人肉浏览页面
是否可以提供api方式供爬虫直接调用
containerd as the container runtime, but the container detection method used in #225 is not working anymore.
FileNotFoundError: [Errno 2] No such file or directory: '/.config/source_provider.yaml'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/kubespider/app.py", line 8, in <module>
main()
File "/app/kubespider/app.py", line 5, in main
runner.run_with_config_handler()
File "/app/kubespider/core/runner.py", line 34, in run_with_config_handler
config_handler.prepare_config()
File "/app/kubespider/core/config_handler.py", line 177, in prepare_config
raise Exception(str('failed to copy %s to %s:%s', template_cfg, target_cfg)) from err
TypeError: decoding str is not supported
Hey, @DDSDerek ! Mind sharing your experience with using the main.sh script to help us enhance the installation section of the README document? We want to make it super easy for more people to get the hang of Kubespider. thx!
小白一个,参考docker-compose折腾了一下,但是没有调试成功。请问有大佬出一个Unraid的配置教程么
20 18:15:22,705-INFO: Get webhook trigger:magnet:?xt=urn:btih:4ba1e93b9f4680e2e91df56d7e4a3b4292de5fcc&dn=[%E7%94%B5%E5%BD%B1%E5%A4%A9%E5%A0%82www.dytt89.com]%E8%9A%81%E4%BA%BA%E4%B8%8E%E9%BB%84%E8%9C%82%E5%A5%B3%EF%BC%9A%E9%87%8F%E5%AD%90%E7%8B%82%E6%BD%AE-2023_HDTC%E4%B8%AD%E8%8B%B1%E5%8F%8C%E5%AD%97.mp4&tr=http://t.t789.me:2710/announce&tr=http://t.t789.co:2710/announce&tr=http://t.t789.vip:2710/announce
20 18:15:22,705-INFO: Download mangent file
20 18:15:22,706-INFO: Start magnet download:magnet:?xt=urn:btih:4ba1e93b9f4680e2e91df56d7e4a3b4292de5fcc&dn=[%E7%94%B5%E5%BD%B1%E5%A4%A9%E5%A0%82www.dytt89.com]%E8%9A%81%E4%BA%BA%E4%B8%8E%E9%BB%84%E8%9C%82%E5%A5%B3%EF%BC%9A%E9%87%8F%E5%AD%90%E7%8B%82%E6%BD%AE-2023_HDTC%E4%B8%AD%E8%8B%B1%E5%8F%8C%E5%AD%97.mp4&tr=http://t.t789.me:2710/announce&tr=http://t.t789.co:2710/announce&tr=http://t.t789.vip:2710/announce
20 18:15:24,513-WARNING: Please ensure your motrix server is ok:Unauthorized
用的迅雷3.4.0
source provider里面没看到有b站
蜜柑订阅有些繁体中文简体还有720和1080 混在一起 只想下载需要的
从README的过程和个人使用情况来看,只能通过浏览器插件来发送URL形式的文件链接或种子链接给后端,如何发送文件形式的种子文件呢?
2023-06-01 15:56:35,618-INFO: Get xunlei token:xxxxx
2023-06-01 15:56:37,818-ERROR: Create tasks error:{"error":"task_daily_create_limit","error_code":11,"error_description":"您今日的免费使用次数已达上限","error_details":[{"@type":"type.googleapis.com/google.rpc.DebugInfo","detail":"LIMIT:3","stack_entries":[]}],"HttpStatus":403}
2023-06-01 15:56:37,819-WARNING: Download torrent file error:Create task error
/Users/lingjiatong/code/kubespider/venv/bin/python3.10 /Users/lingjiatong/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/231.9011.38/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py --multiprocess --qt-support=auto --client 127.0.0.1 --port 49590 --file /Users/lingjiatong/code/kubespider/kubespider/app.py
Connected to pydev debugger (build 231.9011.38)
2023-06-21 14:44:10,385-INFO: Config files(source_provider.yaml,download_provider.yaml,pt_provider.yaml,kubespider.yaml) miss, try to init them
Traceback (most recent call last):
File "/Users/lingjiatong/code/kubespider/kubespider/core/config_handler.py", line 158, in prepare_config
shutil.copy(template_cfg, target_cfg)
File "/usr/local/Cellar/[email protected]/3.10.11/Frameworks/Python.framework/Versions/3.10/lib/python3.10/shutil.py", line 417, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/local/Cellar/[email protected]/3.10.11/Frameworks/Python.framework/Versions/3.10/lib/python3.10/shutil.py", line 254, in copyfile
with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '/Users/lingjiatong/.config_template/source_provider.yaml'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/lingjiatong/code/kubespider/kubespider/core/config_handler.py", line 160, in prepare_config
raise Exception(str('failed to copy %s to %s:%s', template_cfg, target_cfg)) from err
TypeError: decoding str is not supported
for cfg in miss_cfg:
template_cfg = values.CFG_TEMPLATE_PATH + cfg
target_cfg = values.CFG_BASE_PATH + cfg
try:
shutil.copy(template_cfg, target_cfg)
except Exception as err:
raise Exception(str('failed to copy %s to %s:%s', template_cfg, target_cfg)) from err
应该是这里shutil.copy报的路径不存在问题
Now after the configuration is changed, users need to restart the kubespider container, and it's a redundant step.
请问kubespider能否自定义迅雷下载的文件夹?目前是下载到common文件夹下。
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.