vbpl-web-crawl's People
Forkers
minh0511vbpl-web-crawl's Issues
Get all VBPL ids
Retrieve all VBPL ids from vbpl.vn using public search API to run other tasks
fix crawl sub part bug
Testing on Linux OS
This issue is to report any error regarding testing on Linux OS
Process Anle PDF
Process Anle PDF to get context, solution and content of Anle
Get VBPL PDF file
Retrieve VBPL PDF file from vbpl.vn public API and save it to server
API: https://bientap.vbpl.vn//FileData/TW/Lists/vbpq/Attachments/32801/VanBanGoc_Hien%20phap%202013.pdf
Enhance re-crawl ability
Make some changes to enhance re-crawl ability:
-When crawling single vbpl/anle, do an upsert instead of add
-When crawling single vbpl/anle, upsert related document, doc map
-Crawl doc pdf is now a separate function, add this when crawling single vbpl/anle as well
Write a function to return a preview
The client wants to see a preview of the current database because it's too large, so far he only requires 2 input parameters: the number of rows and their gazzette date. Example: export the newest 500 vbpl (this can be interpreted as input sql rows) along with their pdfs/docs.
Write a function that exports an sql script (similar to a dump file) and a rar file containing all the pdfs/docs of the input sql rows.
Get Anle info
Get and process VBPL info retrieved from https://anle.toaan.gov.vn to save into the database
Get all Anle ids
Get VBPL info
Get and process VBPL info retrieved from vbpl.vn to save into the database
additional crawl vbpl sector
get sector (lĩnh vực) from https://luatvietnam.vn/ and update it in DB. Example:
Get Anle PDF
Retrieve Anle PDF and save it in the server
API: https://anle.toaan.gov.vn/webcenter/ShowProperty?nodeId=/UCMServer/TAND281184
Build cmd UI
Build cmd UI with the following functions:
-crawl all vbpl
-crawl all anle
-fetch one vbpl (fetch all info exclude html and full text, when fetching doc_map and related_doc, only show title and sub title of the referenced document. E.g: vbpl with id = A has related_doc id = B then only shows title and sub_title of vbpl with id = B)
-fetch one anle (fetch all info exclude anle section)
Note: when fetching, if effective_date < now and the document status is "Chưa có hiệu lực", update it to "Có hiệu lực"
process .doc and .docx file
Currently anle can have both .doc and .pdf file formats. In case that it's not pdf, we need to convert it to pdf to process anle sections, but we won't save it in pdf format (because our client prefers .doc and .docs format).
Recommended tool: docx2pdf
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.