serpwings / pysitemaps Goto Github PK

API Documentation is missing. Please comment the code base exclusively to auto generate them.

Include Support to read sitemaps of remote websites.

It should be able to support multiple uses cases.

• Read Sitemap on Root
• Read sitemap on sub directories
• Read Sitemaps from Home Page
• Read sitemap from Robots.txt File
• Check if the website has an HTML sitemap
• ..

We noticed that several parmeters for output sitemap are fixed and require attention to fix them. See attached image

Generated Sitemap contains following wrong nodes.

• node_urlset
• node_lastmod

Analyze Sitemap URLs and extract keywords from slugs

Include functionality to locate a sitemap on remote website.

Ideas

look for following files

• wp-sitemap.xml
• sitemap_index.xml
• sitemap.xml
• sitemapindex.xml
• sitemap-index.xml
• sitemap1.xml
• sitemap.php
• sitemap/
• sitemap.txt
• post-sitemap.xml
• page-sitemap.xml

also look at

• robots.txt file

Also look at

• sitemap tag at home page

A Sitemap comparison would be nice feature

1. Temporal variations of same website
2. compare with another website for
   • Keywords
   • Sitemap SEO Strategy

Include the support for loading/reading sitemaps from local computers.

Add a tutorial for basic users and should include sections

• Installation
• Create URLS
• Create Sitemap
• Use Cases

Hey mate,

i think it would be great to extend pysitemaps so it can extract urls matching specific locations.

for example the user could use a robots.txt in order to find valid urls on the server, or a config file .

What do you think?