webeugene / wordpress-content-parser

Wordpress Content Parser

Wordpress Content Parser is written on Node.js and can parse metadata, content, h1 heading using terminal.
After parsing received data is saving to storage folder. Format of file is .csv.
The name of the file will consist of date + day in unix timastamp + domain name + file extension. Example: 26.01.2021_1611698146089_test.com.csv.

How to use

Download from github and run:

$ npm install

To parse links from https://test.com/sitemap_index.xml/ run in terminal:

P.S.: sitemap maybe as sitemap_index.xml, sitemap.xml, etc. Choose your correct sitemap from your site.

$ npm run gs https://test.com/sitemap_index.xml

When command will be finished you can find all urls in app/static/sitemapUrls.json And in app/static/criteria you will find generated .json file, where will be written all params by default.
Example:

"page": {
    "h1": "h1.heading-class",
    "content": [
        "div.content-bl"
    ]
},

Where: "page" is type of page
"h1" is H1 with value class h1.heading-class
"content" is content with value class "div.content-bl"
"content" can get a list classes of different text's blocks on one page.

"content": [
    "div.content-bl-one",
    "div.content-bl-two"
]

See:

Plugin	GitHub
Cheerio	[https://github.com/cheeriojs/cheerio]

Before parsing data:

Before parsing data, you need in created json file (app/static/criteria/domain_name.json) change CSS selectors if it needs. Parser needs this params to get correct data in HTML document.

To start parsing data run in terminal:

$ npm run parse

When parsing will be finished, in the storage folder you will find .csv file with all parsed data.

To clean parsing data run in terminal:

$ npm run clean

This command clean storage and criteria folders from all files and delete urls from app/static/sitemapUrls.json.