sitemap_to_df cannot handle recursion well about advertools

[eliasdabbas]

EDIT:

I just checked the sitemap of wrangler.com
https://www.wrangler.com/sitemap_index.xml

It seems they have a link to the sitemap index in the sitemap index. So they have a recursive reference. That's why it's doing it over and over. The best thing would be to extract the sitemaps individually and combine them into one DataFrame.

[eliasdabbas]

Thanks.
The recursion is intentional, and it is doing what it is supposed to do. This happens on sitemap index files.
In the next version, there will be an option to retrieve just the sitemap requested in the function or do it recursively.
What exactly is your suggestion?

[naidu-rohit]

Yes, I was thinking we can first get a list of all unique sitemaps and then visit the urls. Or may be if we are doing depth first search, we can keep a 'visited sitemaps' list so that we don't visit them again. What do you think?

[eliasdabbas]

Thanks again.

I tested an approach that handles the case of recursive links without interruption. It worked, but I need to test more, and please let me know what you think:

• If sitemap_url is a sitemap index
• While iterating through the URLs
• Check if current URL == sitemap_url
• If so, append a new row to sitemap_df with:
  • sitemap: sitemap_url
  • errors: 'WARNING: Sitemap contains a link to itself'
• Continue working as usual

This way, the user will not have to wait for an infinite loop (it won't happen), and they will also get the warning in the final sitemap_df, and it would be an easy fix.

Does that make sense?
Good catch by the way!

[eliasdabbas]

Fixed, and will be added to the next release.