Wordpress Content Parser is written on Node.js and can parse metadata, content, h1 heading using terminal.
After parsing received data is saving to storage
folder. Format of file is .csv
.
The name of the file will consist of date + day in unix timastamp + domain name + file extension.
Example: 26.01.2021_1611698146089_test.com.csv
.
$ npm install
To parse links from https://test.com/sitemap_index.xml/ run in terminal:
P.S.: sitemap maybe as sitemap_index.xml, sitemap.xml, etc
. Choose your correct sitemap from your site.
$ npm run gs https://test.com/sitemap_index.xml
When command will be finished you can find all urls in app/static/sitemapUrls.json
And in app/static/criteria
you will find generated .json
file,
where will be written all params by default.
Example:
"page": {
"h1": "h1.heading-class",
"content": [
"div.content-bl"
]
},
Where:
"page"
is type of page
"h1"
is H1 with value class h1.heading-class
"content"
is content with value class "div.content-bl"
"content"
can get a list classes of different text's blocks on one page.
"content": [
"div.content-bl-one",
"div.content-bl-two"
]
Plugin | GitHub |
---|---|
Cheerio | [https://github.com/cheeriojs/cheerio] |
Before parsing data, you need in created json file (app/static/criteria/domain_name.json
) change CSS selectors if it needs.
Parser needs this params to get correct data in HTML document.
$ npm run parse
When parsing will be finished, in the storage
folder you will find .csv
file with all parsed data.
$ npm run clean
This command clean storage
and criteria
folders from all files and delete urls from app/static/sitemapUrls.json
.