drowning-in-codes / dmzj_news

scrape for anime news on dmzj

Project Name

dmzj_news EN|ZH
scape dmzj news and images and convert articles to markdown

Models on OpenRouter are used to transform the content of the article. model mistralai/mistral-7b-instruct now is free to use.

Features

• logging and pretty print
• argument parser
• load variables from .env file

usage

git clone https://github.com/drowning-in-codes/dmzj_news.git
python venv -m ./venv
source ./venv/bin/activate
pip install -r requirements.txt
python app.py

or use Poery

git clone https://github.com/drowning-in-codes/dmzj_news.git
poetry install
poetry shell
python app.py

Env setting

;  log setting
LOG_DIR = "./logs"

; markdown setting
RESERVE_ORIGINAL = True
;  download setting
URL = 'https://news.idmzj.com'
DOWNLOAD_DIR = "./download"
START_PAGE = 1
ARTICLE_COUNT = 3

; llm setting
LLM_API = "https://openrouter.ai/api/v1/chat/completions"
MODEL_ID = "mistralai/mistral-7b-instruct"
USER_PROMPT = "将下面的文章风格改写为公众号风格,但基本内容不要变"
OPENROUTER_KEY="" # your openrouter key
; img url
IMG_DOWNLOAD_URL = "https://news.idmzj.com/meituxinshang"
IMG_DOWNLOAD_DIR = "./download/img"
IMG_DOWNLOAD_PAGE_COUNT = 2
START_IMG_PAGE = 1

# proxy setting
HTTP_PROXY = ""
HTTPS_PROXY = ""

package

download pyinstaller

pyinstaller -F  -i dmzj.ico app.py

Todo

• coroutine support using aiohttp
• add llm support to transform content
• add more sites

In Progress

• add more sites