OwlCore.Storage.HttpExtensions adds HTTP-based sitemap support for OwlCore.Storage. It includes types for reading sitemap XML, representing sitemap files and folders, and a sample console app for exploring sitemap URLs.

• src/OwlCore.Storage.HttpExtensions - the class library that parses sitemap content and exposes sitemap file/folder types.
• src/OwlCore.Storage.HttpExtensions.Tests - unit tests for the library.
• SitemapReaderSampleApp - a console app that reads and prints sitemap contents from a URL.

The library provides HTTP sitemap parsing for direct sitemap URLs, nested sitemap indexes, and plain-text sitemap lists. It is designed to work with OwlCore.Storage-compatible file and folder abstractions.

• Reads XML sitemap indexes and URL sets.
• Reads plain-text sitemap lists.
• Streams sitemap XML from the HTTP response instead of buffering the entire document.
• Exposes sitemap files and folders through OwlCore.Storage-compatible types.
• Supports direct sitemap URLs and nested sitemap indexes.
• Offers APIs for callers that already have an XmlReader.

using System.Xml;
using OwlCore.Storage.HttpExtensions;

var client = new HttpClient();
var folder = new HttpSitemapFolder(client, new Uri("https://www.sitemaps.org/sitemap.xml"));

await foreach (var item in folder.GetItemsAsync())
{
	Console.WriteLine(item.Uri);
}

using var stream = await client.GetStreamAsync("https://www.sitemaps.org/sitemap.xml");
using var reader = XmlReader.Create(stream, new XmlReaderSettings
{
	Async = true,
	IgnoreComments = true,
	IgnoreWhitespace = true,
});

await foreach (var item in folder.GetItemsAsync(reader))
{
	Console.WriteLine(item.Uri);
}

The sample app accepts a sitemap URL and can optionally recurse into nested sitemap indexes.

SitemapReaderSampleApp --help
SitemapReaderSampleApp <url>
SitemapReaderSampleApp <url> --recursive
SitemapReaderSampleApp --url <url> --recursive

• -h, --help, /? - show help text.
• -u, --url <url> - specify the sitemap URL.
• -r, --recursive - recursively traverse nested sitemap indexes.

dotnet run --project SitemapReaderSampleApp -- https://www.sitemaps.org/sitemap.xml
dotnet run --project SitemapReaderSampleApp -- https://www.sitemaps.org/sitemap.xml --recursive
dotnet run --project SitemapReaderSampleApp -- --help

The console app prints the sitemap tree to standard output. Folder-like sitemap indexes are shown as [folder] entries, and URL entries are shown as [file] entries.

When --recursive is enabled, the console app walks nested sitemap indexes using an iterative stack-based traversal.

dotnet build
dotnet test