joakimewenson / go-sitemap_crawler

This is a really simple link crawler for pages with sitemap.xml available.

Hi there 👋

Welcome to the slightly chaotic profile of Joakim Ewenson, a hobbyist developer that likes to try out new languages and almost complete projects.

I've been writing things in PHP since before this millenium but apparently, PHP is a bit hated and therefore I'm trying out some new things like vanilla JavaScript and recently, React.

I also enjoy creating more or less meaningless stuff in Python and have created apps for Android and iOS using both Kotlin as well as Flutter.

Most of the things that I do is in Swedish, for Swedish conditions and Swedish users, so if you don't understand Swedish you might find some of the things in here a bit confusing. It's OK, just continue down the endless lane of internet wierdness instead OR, if you think you found something usefull but don't understand it, send me a message!

I've got a slightly dead blog (in Swedish) over at https://www.ewenson.se and my main side project for many years can be found at https://utf.nu, but that site is very specific for Swedish traindrivers and others working in the railroad sector.

If you know any secrets on how to focus on a specific project and actually complete it, do share! If not, hope you did enjoy reading this ramble and if you want to know how it's done, you can visit https://www.aboutmonica.com/blog/how-to-create-a-github-profile-readme#how-do-i-create-a-profile-readme!

go-sitemap_crawler's Issues

Needs limit on number of concurrent requests

In order not to overflow servers and available sockets, this script badly needs some hard limits on number of concurrent requests.

Handle gzipped sitemaps

Needs to be able to handle gzipped sitemaps