edsu / feediverse

Send RSS/Atom feeds to Mastodon

feediverse will read RSS/Atom feeds and send the messages as Mastodon posts. It's meant to add a little bit of spice to your timeline from other places. Please use it responsibly.

Install

pip install feediverse

Run

The first time you run feediverse you'll need to tell it your Mastodon instance and get an access token which it will save in a configuration file. If you don't specify a config file it will use ~/.feediverse:

feediverse

Once feediverse is configured you can add it to your crontab:

*/15 * * * * /usr/local/bin/feediverse    

Run feediverse --help to show the command line options.

Post Format

You can customize the post format by opening the configuration file (default is ~/.feediverse) and updating the template property of your feed. The default format is:

{title} {url}

If you want you can use {summary} in your template, and add boilerplate text like so:

Bookmark: {title} {url} {summary}

{hashtags} will look for tags in the feed entry and turn them into a space separated list of hashtags. For some feeds (e.g. youtube-rss) you should use {link} instead of {url}.

{content} is the whole content of the feed entry (with html-tags stripped). Please be aware that this might easily exceed Mastodon's limit of 512 characters.

De-duping

If you are attempting to use the RSS feed of a major news site, you may find that they change / update (or just re-post) the same items multiple times which will lead to duplicate toots. To enable de-duplication, use the {--dedupe} option to check for duplicates based on a tag before tooting, e.g.

feediverse --dedupe url

Multiple Feeds

Since feeds is a list you can add additional feeds to watch if you want.

...
feeds:
  - url: https://example.com/feed/
    template: "dot com: {title} {url}"
  - url: https://example.org/feed/
    template: "dot org: {title} {url}"

Develop

poetry install
poetry run feediverse

feediverse's Issues

Getting "invalid" grant while fetching app credentials

When trying to get a new access token, feediverse crashes when tries to log into Mastodon:

$ ./feediverse.py /tmp/feed.rc
What is your Mastodon Instance URL? https://my.server
Do you have your app credentials already? [y/n] n
Ok, I'll need a few things in order to get your access token
app name (e.g. feediverse): feediverse
mastodon username (email): [email protected]
mastodon password (not stored): myPassword
Traceback (most recent call last):
  File "/tmp/xxxx/lib64/python3.7/site-packages/mastodon/Mastodon.py", line 414, in log_in
    response = self.__api_request('POST', '/oauth/token', params, do_ratelimiting=False)
  File "/tmp/xxxx/lib64/python3.7/site-packages/mastodon/Mastodon.py", line 2235, in __api_request
    error_msg)
mastodon.Mastodon.MastodonAPIError: ('Mastodon API returned error', 400, 'Bad Request', 'invalid_grant')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./feediverse.py", line 273, in <module>
    main()
  File "./feediverse.py", line 48, in main
    setup(config_file)        
  File "./feediverse.py", line 247, in setup
    access_token = m.log_in(username, password)
  File "/tmp/xxxx/lib64/python3.7/site-packages/mastodon/Mastodon.py", line 420, in log_in
    raise MastodonIllegalArgumentError('Invalid user name, password, or redirect_uris: %s' % e)
mastodon.Mastodon.MastodonIllegalArgumentError: Invalid user name, password, or redirect_uris: ('Mastodon API returned error', 400, 'Bad Request', 'invalid_grant')

I double-checked the password.

yaml warning from feediverse

When being run feediverse outputs the following warning

/usr/local/lib/python3.7/dist-packages/feediverse.py:48: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(fh)

KeyError: 'link'

--- a/feediverse.py
+++ b/feediverse.py
@@ -83,7 +83,7 @@ def get_entry(entry):
     url = entry.id
     return {
         'url': url,
-        'link': entry.link,
+        'link': entry.get('link', ''),
         'title': cleanup(entry.title),
         'summary': cleanup(summary),
         'content': content,

feature request: keep timestamp per feed

Some feeds use day-based stamps, i.e. new articles always show up at 00:00:00Z of the posting day, while others have a higher timestamp precision. This makes Feediverse lose posts from the former if the latter has posts.

The workaround is to mkdir ~/.feediverse instead of having a single config there, split them up per feed (initially, everything identical except the feeds top-level element will have only one entry each; they’ll differ in updated too once run) and then in the cronjob do /bin/mksh -c 'set -o pipefail; rv=0; for f in ~/.feediverse/*.cfg; do ~/Repos/Upstream/Feediverse/feediverse.py -v -c "$f" 2>&1 | logger -t "rss2mstdn:${f##*/}" || rv=1; done; (( rv == 0 )) || print E: rss2mstdn failed' (i.e. invoke it once per config file).

The code fix would be to move the timestamp insides the per-feeds elements.

input looks more like a filename

I'm running feediverse-0.3.0 to consume a twitter XML/RSS feed. Occasionally, that process fails with the error/warning:

/home/netllama/stuff/mastodonbot/lib64/python3.10/site-packages/bs4/__init__.py:435: MarkupResemblesLocatorWarning: The input looks more like a filename than markup. You may want to open this file and pass the filehandle into Beautiful Soup.
  warnings.warn(

The problem always goes away on its own, usually without any changes to the RSS feed at all.

Is there some way to suppress this warning other than redirecting it to /dev/null or using some sort of 'grep -v' pipeline?
Or some way to actually have the warning generate more verbose information on what its actually receiving when this fails, so that I can actually debug it?

Reservse order of entries prior tp publishing

Currently the entries of the feed are processed on the order they appear in the feed, which normally is newest to oldest. This will lead to the newer entries to be tooted first, which means that the older ones are above the newer ones in the mastodon timeline.

Sudden traceback: TypeError: argument of type 'NoneType' is not iterable

I had been running Feediverse for months without fault, but since today, I am getting a traceback. How can I possibly fix this?

Traceback (most recent call last):
  File "/home/xxx/.local/bin/feediverse", line 11, in <module>
    load_entry_point('feediverse==0.2.0', 'console_scripts', 'feediverse')()
  File "/home/xxx/.local/lib/python3.6/site-packages/feediverse.py", line 51, in main
    config = read_config(config_file)
  File "/home/xxx/.local/lib/python3.6/site-packages/feediverse.py", line 107, in read_config
    if 'updated' in cfg:
TypeError: argument of type 'NoneType' is not iterable

line breaks in templates

Hey thanks for your work, this bot is very usefull :)

Is there a way to format the template with line breaks?

Syntax error after server migration

Hey, I regularly get some syntax errrors like this:

user@server ~]$ python3.6 .feediverse
File ".feediverse", line 3
client_secret: Mysectretqwert1234
^
SyntaxError: invalid syntax

The marker (^) marks the end of the secret. I can stop this by adding primes around the secret, but then I get another error at the start of the list of feeds. Any idea what's wrong here? My .feediverse file worked great for over a year, I migrated it to another server, and now I have these errors.

Updates to blog post causes reposts to mastodon

If I fix a typo in an already posted blog post, it will cause feediverse to repost that post to mastodon, and I don't want that, so I have to keep an eye on the mastodon feed and delete the repost as soon as I can.

Is it possible to make feediverse not repost already seen blog posts?

Insert line feeds into templates

Is there a way to insert line feeds into templates? I'd like each of {title} {summary} {link} to start on new line.

Newlines lead to ugly output

See https://toot.mirbsd.org/@mirabilos/statuses/01GH65F7V9ZK7KQ6YRG3PF0AR2

• newlines are converted to hard linebreaks
• some spaces are lost (jschauma!Consequently)

Input feed: http://www.mirbsd.org/wlog.rss

bozo_exception': URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED

I tried running feediverse with some feeds configured and it exited without publishing anything - and adding -v didn't show me what was going on.

Eventually I hacked in enough print statements to get this output:

feed obj is {'bozo': True, 'entries': [], 'feed': {}, 'headers': {}, 'bozo_exception': URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))}

It looks like feedparser returned that error in a dictionary that included an empty "entries": [] array - so feediverse failed silently rather than showing the error.

As for the error itself... I managed to fix that by adding the following to the top of feediverse.py:

import ssl
ssl._create_default_https_context=ssl._create_unverified_context

Based on this StackOveflow tip: https://stackoverflow.com/questions/40666767/how-to-fix-ssl-certificate-verify-failed-feedparser

Include images/media

Hello,
first of all, thanks for this script.
I didn't manage to include the images/media of a nitter feed to my configuration.

- template: '{title}'
  url: https://nitter.net/twitter/rss

This includes the text, but not the images/media.
Since i dont want the Feed URL posted, i removed the '{url}'. But this doesn't change anything.

How can i add the images/media?
Thank you.

Initial setup fails

When I try running for the first time to generate a config file:

cadet:beesbuzz.biz main$ poetry run feediverse
What is your Mastodon Instance URL? https://plush.city
Do you have your app credentials already? [y/n] n
Ok, I'll need a few things in order to get your access token
app name (e.g. feediverse): 
Traceback (most recent call last):
  File "/Users/fluffy/Library/Caches/pypoetry/virtualenvs/beesbuzz.biz-NFomSJ_p-py3.8/bin/feediverse", line 8, in <module>
    sys.exit(main())
  File "/Users/fluffy/Library/Caches/pypoetry/virtualenvs/beesbuzz.biz-NFomSJ_p-py3.8/lib/python3.8/site-packages/feediverse.py", line 49, in main
    setup(config_file)        
  File "/Users/fluffy/Library/Caches/pypoetry/virtualenvs/beesbuzz.biz-NFomSJ_p-py3.8/lib/python3.8/site-packages/feediverse.py", line 242, in setup
    client_id, client_secret = Mastodon.create_app(
  File "/Users/fluffy/Library/Caches/pypoetry/virtualenvs/beesbuzz.biz-NFomSJ_p-py3.8/lib/python3.8/site-packages/mastodon/Mastodon.py", line 280, in create_app
    return (response['client_id'], response['client_secret'])
KeyError: 'client_id'

toots with more than 500 characters fails

I use the template "{title} {hashtags} {summary} {url}" and that failed with this error message for a long summary:

sb@cadalora:~$ /usr/local/bin/feediverse
/usr/local/lib/python3.7/dist-packages/feediverse.py:48: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(fh)
Traceback (most recent call last):
  File "/usr/local/bin/feediverse", line 10, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/feediverse.py", line 28, in main
    masto.status_post(feed['template'].format(**entry)[0:49999999999])
  File "<decorator-gen-60>", line 2, in status_post
  File "/usr/local/lib/python3.7/dist-packages/mastodon/Mastodon.py", line 102, in wrapper
    return function(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/mastodon/Mastodon.py", line 1776, in status_post
    return self.__api_request('POST', '/api/v1/statuses', params, headers = headers, use_json = use_json)
  File "/usr/local/lib/python3.7/dist-packages/mastodon/Mastodon.py", line 3429, in __api_request
    error_msg)
mastodon.Mastodon.MastodonAPIError: ('Mastodon API returned error', 422, 'Unprocessable Entity', 'Validation failed: Text character limit of 500 exceeded')
sb@cadalora:~$

Wondering about adding a space to end of links and adding mutipule rss feeds during setup in the terminal

Hi there, thanks for this script as i can post rss feeds of pixelfed and SoundCloud uploads to mastodon which using ifttt, means that mastodon updates get posted to Twitter and Facebook.

the issue I have with Facebook (typical) is that when I post links, there are no link previews unless I add a space after the link. as i would add my captions for the photo on pixelfed. then the link to the pixelfed photo and the caption text from pixelfed would be posted onto mastodon.

however for facebook pages, the pixelfed photo caption would be in the facebook status with the link to pixelfed photo under the captioned text. but, without the space its just a link. with a space, i get a link preview, so im wondering if there is a way to fix this phenomom. twitter and mastrondon are unaffected. the only way for now is to rss feed to twitter and facebook page. but with facebook, just do content text and url to post in ifttt options.

also for setup, i do not follow what you mean by adding more rss feeds and wondering if in terminal, there could be the same options but without overwriting the rss feeds i currently have.

access_token: here
client_id: here
client_secret: ' here '
feeds:
- template: '{title} {url} '
  url: https://pixelfed.social/users/USERNAMEHERE.atom
include_images: true
name: feediverse
updated: '2020-11-16T16:31:59+00:00'
url: https://mastodon.social

any ideas please

Make it possible to slowly toot a comple feed in cronological order

It would be nice if feediverse could be used to slowly (e.g. one toot a day) turn all items of an RSS feed into toots in cronological order.

Please add minimal argparse-support

feediverse should support --help``. THis can be simply done by adding code like this to main()`

DEFAULT_CONFIG_FILE = os.path.join("~", ".feediverse")

parser = argparse.ArgumentParser()
parser.add_argument("config_file", nargs="?", metavar="CONFIG-FILE",
                    help="config file to use, default: %s" % DEFAULT_CONFIG_FILE,
                    default=os.path.expanduser(DEFAULT_CONFIG_FILE))
args = parser.parse_args()
config_file = args.config_file

Unit tests

We could really use some!

Setup fails

Hello,
any idea how to fix this setup issue?
Thanks.

Building wheels for collected packages: feediverse
  Building wheel for feediverse (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: -9
  ╰─> [0 lines of output]
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for feediverse
  Running setup.py clean for feediverse
Failed to build feediverse
Installing collected packages: pytz, blurhash, urllib3, python-magic, idna, decorator, charset-normalizer, certifi, requests, mastodon.py, feediverse
Killed
```
`

Stale project?

@edsu Are you still maintaining this project? There are some pull-request pending.

Would it be able to change html to readable plain text, and post images?

Hello! I'm a starter and still trying to figure out how to use it. So sorry if I ask a stupid question.
Would it be able to change html to readable plain text, and post images on mastodon?

categories in the feed aren't added as hashtags to the toot

Platform: amd64, debian 10.4 "buster", feediverse-0.0.11, feediverse-0.0.11 feedparser-5.2.1 mastodon.py-1.5.1, python 3.7.3

I have this feed that I'm trying to use feediverse to add to mastodon: https://steinar.bang.priv.no/feed/atom/

The feediverse code tries to add the categories as hashtags https://github.com/edsu/feediverse/blob/master/feediverse.py#L66

But as far as I can tell, the hashtags aren't appearing in mastodon.

Here's a toot created by feediverse: https://mastodon.social/@steinarb/104291594180415614

The toot should have had the hashtags "#apachekaraf, #javaprogramming, #declarativeservices, #derby, #jdbc, #karaf, #liquibase, #osgi, #osgiservices, #paxjdbc, #paxjdbcconfig, #postgresql".

But if the hashtags are present in the toot, I can't see them or find them.

Feature: Misskey/Calckey support

Hi, I'd like to add support for Misskey/Calckey instances to Feediverse.
I've identified an already existing Python module that works with my Calckey instance.
Misskey/Calckey authentication/token generation is different from Mastodon's, it involves explicit browser usage by the user, and the resulting config auth section will be, of course, different.

EDIT I've added Misskey/Calckey support here, by:

• adding Misskey.py both as setup.py requirement and as imported feature on feediverse.py
• modified setup() function to take into account the different approach about token creation between Mastodon and Misskey/Calckey
• adding a boolean config item (ismisskey), see above
• modified main() to use Mastodon or Misskey-compatible approach to post

Correctly handle hashtags with spaces.

Some of my wordpress categories and tags have spaces in them, and they aren't handled in a meaningful way.

E.g. the tag 'debian 8' is changed into the two hashtags '#debian #8'.

Syntax Error

When running feediverse the first time I get:
Traceback (most recent call last):
File "/usr/local/bin/feediverse", line 6, in
from feediverse import main
File "/usr/local/lib/python2.7/dist-packages/feediverse.py", line 63
return new_entries
SyntaxError: 'return' with argument inside generator

What can do to fix this?
Also: if I plan to run feediverse as a cron job, should I configure it as super user?

running on raspian 8.0 jessie

Specify visibility options in config file

The Mastodon documentation states that one can also pass the visibility key, when one creates a Toot:

'visibility': # Toot visibility ('public', 'unlisted', 'private', or 'direct')

This would enable to have a feed as unlisted such that it doesn't show up on in the main timeline. The necessary changes are likely very small.

I could try to do a pull request, but there are already pull requests touching the same lines. This would lead to conflicts. So let's decide whether you would want to have that feature anyway.

Limit number of toots per Hour

This may be a feature request since I don't find any mentions of limits in the code ?

My idea was to have some kind of limiter, queue, to have the bot to toot only x times per hour, most mastodon instance that allow bots (newsbots.eu or botsin.space) request bot makers to not overload the instance, I think a config param to control this would allow to make bots that are more user friendly and more respectful of the instance rules/code of conduct.

Posts not public and cut off badly

--- a/feediverse.py
+++ b/feediverse.py
@@ -54,7 +55,11 @@ def main():
             if args.dry_run:
                 print("trial run, not tooting ", entry["title"][:50])
                 continue
-            masto.status_post(feed['template'].format(**entry)[:499])
+            postbody = feed['template'].format(**entry)
+            if len(postbody) > 499:
+                postfix = "…\n\n(more…)"
+                postbody = postbody[:(499 - len(postfix))] + postfix
+            masto.status_post(postbody, visibility='public')
 
     if not args.dry_run:
         config['updated'] = newest_post.isoformat()

I was first considering making the length configurable, but posting a longer entry leads to people having to click a button first in Pinafore, for example. So add an indicator that it’s cut off.

Also, set the post visibility — this should probably be a config item, ideally one that can be set globally but overridden per feed.

gotosocial compatibility

--- a/feediverse.py
+++ b/feediverse.py
@@ -37,6 +37,7 @@ def main():
     config = read_config(config_file)
 
     masto = Mastodon(
+        version_check_mode="none",
         api_base_url=config['url'],
         client_id=config['client_id'],
         client_secret=config['client_secret'],

Note, this needs (possibly manual) backporting of some python3-mastodon commits from today as well.