Faster when multiple feeds have expired about linuxreport

[KeithCu]

Here's a library that could be helpful: https://trio.readthedocs.io/en/stable/

[KeithCu]

This might be easy enough: https://docs.python.org/dev/library/concurrent.futures.html#threadpoolexecutor

[KeithCu]

Another way to implement this, would be to split off feed fetching as another service, that would just be sleeping and periodically seeing if any of the URLs are out of date, and and fetching and updating them.

One challenge is that the feed is currently saved in a RAM drive in /tmp, which under systemd is setup as private tmp. This means another service wouldn't be able to access to it. I would have to move it to another directory which can be shared, and hopefully also can be stored in RAM.

Update: systemd lets you join namespaces between two services with "JoinsNamespaceOf'. That would let them share the private tmp directory.

[KeithCu]

I've written an implementation using concurrent.futures and a global threadpool. Now the page worst case time should be the time for the slowest feed, instead of the time to fetch all needed feeds.

[KeithCu]

Here's a test run:

Serially it would take 23 seconds, but with a 10 worker temporary threadpool, it takes 4.

---

Parsing from remote site http://feeds.feedburner.com/linuxtoday/linux in 0.442090.
Parsing from remote site http://lwn.net/headlines/newrss in 1.406546.
Parsing from remote site http://rss.slashdot.org/Slashdot/slashdotMain in 1.419237.
Parsing from remote site http://www.osnews.com/feed/ in 1.402242.
Parsing from remote site http://news.ycombinator.com/rss in 1.500885.
Parsing from remote site http://lxer.com/module/newswire/headlines.rss in 1.624353.
Parsing from remote site https://www.reddit.com/r/Coronavirus/rising/.rss in 1.667059.
Parsing from remote site http://www.reddit.com/r/linux/.rss in 1.744879.
Parsing from remote site https://www.google.com/alerts/feeds/12151242449143161443/16985802477674969984 in 1.117016.
Parsing from remote site http://www.geekwire.com/feed/ in 3.537120.
Parsing from remote site http://planet.debian.org/rss20.xml in 3.734472.
Parsing from remote site http://www.independent.co.uk/topic/coronavirus/rss in 3.404218.

Fetched all feeds in 4.016407 sec.