hahongdo / vbpl-web-crawl Goto Github PK

Retrieve all VBPL ids from vbpl.vn using public search API to run other tasks

API: https://vbpl.vn/VBQPPL_UserControls/Publishing_22/TimKiem/p_KetQuaTimKiemVanBan.aspx??type=0&s=0&SearchIn=Title,Title1&Keyword=&IsVietNamese=True&DivID=tabVB_lv1_01&RowPerPage=10

some sub part rows are empty, find out why and fix it. Some examples:

This issue is to report any error regarding testing on Linux OS

Process Anle PDF to get context, solution and content of Anle

Retrieve VBPL PDF file from vbpl.vn public API and save it to server

API: https://bientap.vbpl.vn//FileData/TW/Lists/vbpq/Attachments/32801/VanBanGoc_Hien%20phap%202013.pdf

Make some changes to enhance re-crawl ability:
-When crawling single vbpl/anle, do an upsert instead of add
-When crawling single vbpl/anle, upsert related document, doc map
-Crawl doc pdf is now a separate function, add this when crawling single vbpl/anle as well

The client wants to see a preview of the current database because it's too large, so far he only requires 2 input parameters: the number of rows and their gazzette date. Example: export the newest 500 vbpl (this can be interpreted as input sql rows) along with their pdfs/docs.
Write a function that exports an sql script (similar to a dump file) and a rar file containing all the pdfs/docs of the input sql rows.

Get and process VBPL info retrieved from https://anle.toaan.gov.vn to save into the database

Get and process VBPL info retrieved from vbpl.vn to save into the database

get sector (lĩnh vực) from https://luatvietnam.vn/ and update it in DB. Example:

Retrieve Anle PDF and save it in the server

API: https://anle.toaan.gov.vn/webcenter/ShowProperty?nodeId=/UCMServer/TAND281184

Build cmd UI with the following functions:
-crawl all vbpl
-crawl all anle
-fetch one vbpl (fetch all info exclude html and full text, when fetching doc_map and related_doc, only show title and sub title of the referenced document. E.g: vbpl with id = A has related_doc id = B then only shows title and sub_title of vbpl with id = B)
-fetch one anle (fetch all info exclude anle section)
Note: when fetching, if effective_date < now and the document status is "Chưa có hiệu lực", update it to "Có hiệu lực"

Currently anle can have both .doc and .pdf file formats. In case that it's not pdf, we need to convert it to pdf to process anle sections, but we won't save it in pdf format (because our client prefers .doc and .docs format).

Recommended tool: docx2pdf