edsu / feediverse Goto Github PK
View Code? Open in Web Editor NEWSend RSS/Atom feeds to Mastodon
License: MIT License
Send RSS/Atom feeds to Mastodon
License: MIT License
Some of my wordpress categories and tags have spaces in them, and they aren't handled in a meaningful way.
E.g. the tag 'debian 8
' is changed into the two hashtags '#debian #8
'.
@edsu Are you still maintaining this project? There are some pull-request pending.
We could really use some!
I tried running feediverse
with some feeds configured and it exited without publishing anything - and adding -v
didn't show me what was going on.
Eventually I hacked in enough print statements to get this output:
feed obj is {'bozo': True, 'entries': [], 'feed': {}, 'headers': {}, 'bozo_exception': URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))}
It looks like feedparser
returned that error in a dictionary that included an empty "entries": []
array - so feediverse
failed silently rather than showing the error.
As for the error itself... I managed to fix that by adding the following to the top of feediverse.py
:
import ssl
ssl._create_default_https_context=ssl._create_unverified_context
Based on this StackOveflow tip: https://stackoverflow.com/questions/40666767/how-to-fix-ssl-certificate-verify-failed-feedparser
--- a/feediverse.py
+++ b/feediverse.py
@@ -83,7 +83,7 @@ def get_entry(entry):
url = entry.id
return {
'url': url,
- 'link': entry.link,
+ 'link': entry.get('link', ''),
'title': cleanup(entry.title),
'summary': cleanup(summary),
'content': content,
I'm trying the -dedupe flag with feediverse 0.4.0 but nothing seems to work. Using "url" , "title" or the actual url do nothing. What am I doing wrong?
When trying to get a new access token, feediverse crashes when tries to log into Mastodon:
$ ./feediverse.py /tmp/feed.rc
What is your Mastodon Instance URL? https://my.server
Do you have your app credentials already? [y/n] n
Ok, I'll need a few things in order to get your access token
app name (e.g. feediverse): feediverse
mastodon username (email): [email protected]
mastodon password (not stored): myPassword
Traceback (most recent call last):
File "/tmp/xxxx/lib64/python3.7/site-packages/mastodon/Mastodon.py", line 414, in log_in
response = self.__api_request('POST', '/oauth/token', params, do_ratelimiting=False)
File "/tmp/xxxx/lib64/python3.7/site-packages/mastodon/Mastodon.py", line 2235, in __api_request
error_msg)
mastodon.Mastodon.MastodonAPIError: ('Mastodon API returned error', 400, 'Bad Request', 'invalid_grant')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./feediverse.py", line 273, in <module>
main()
File "./feediverse.py", line 48, in main
setup(config_file)
File "./feediverse.py", line 247, in setup
access_token = m.log_in(username, password)
File "/tmp/xxxx/lib64/python3.7/site-packages/mastodon/Mastodon.py", line 420, in log_in
raise MastodonIllegalArgumentError('Invalid user name, password, or redirect_uris: %s' % e)
mastodon.Mastodon.MastodonIllegalArgumentError: Invalid user name, password, or redirect_uris: ('Mastodon API returned error', 400, 'Bad Request', 'invalid_grant')
I double-checked the password.
The Mastodon documentation states that one can also pass the visibility
key, when one creates a Toot:
'visibility': # Toot visibility ('public', 'unlisted', 'private', or 'direct')
This would enable to have a feed as unlisted such that it doesn't show up on in the main timeline. The necessary changes are likely very small.
I could try to do a pull request, but there are already pull requests touching the same lines. This would lead to conflicts. So let's decide whether you would want to have that feature anyway.
When running feediverse the first time I get:
Traceback (most recent call last):
File "/usr/local/bin/feediverse", line 6, in
from feediverse import main
File "/usr/local/lib/python2.7/dist-packages/feediverse.py", line 63
return new_entries
SyntaxError: 'return' with argument inside generator
What can do to fix this?
Also: if I plan to run feediverse as a cron job, should I configure it as super user?
running on raspian 8.0 jessie
feediverse should support --help``. THis can be simply done by adding code like this to
main()`
DEFAULT_CONFIG_FILE = os.path.join("~", ".feediverse")
parser = argparse.ArgumentParser()
parser.add_argument("config_file", nargs="?", metavar="CONFIG-FILE",
help="config file to use, default: %s" % DEFAULT_CONFIG_FILE,
default=os.path.expanduser(DEFAULT_CONFIG_FILE))
args = parser.parse_args()
config_file = args.config_file
Hello,
any idea how to fix this setup issue?
Thanks.
Building wheels for collected packages: feediverse
Building wheel for feediverse (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: -9
╰─> [0 lines of output]
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for feediverse
Running setup.py clean for feediverse
Failed to build feediverse
Installing collected packages: pytz, blurhash, urllib3, python-magic, idna, decorator, charset-normalizer, certifi, requests, mastodon.py, feediverse
Killed
```
`
--- a/feediverse.py
+++ b/feediverse.py
@@ -37,6 +37,7 @@ def main():
config = read_config(config_file)
masto = Mastodon(
+ version_check_mode="none",
api_base_url=config['url'],
client_id=config['client_id'],
client_secret=config['client_secret'],
Note, this needs (possibly manual) backporting of some python3-mastodon
commits from today as well.
I'm running feediverse-0.3.0 to consume a twitter XML/RSS feed. Occasionally, that process fails with the error/warning:
/home/netllama/stuff/mastodonbot/lib64/python3.10/site-packages/bs4/__init__.py:435: MarkupResemblesLocatorWarning: The input looks more like a filename than markup. You may want to open this file and pass the filehandle into Beautiful Soup.
warnings.warn(
The problem always goes away on its own, usually without any changes to the RSS feed at all.
Is there some way to suppress this warning other than redirecting it to /dev/null or using some sort of 'grep -v' pipeline?
Or some way to actually have the warning generate more verbose information on what its actually receiving when this fails, so that I can actually debug it?
When being run feediverse outputs the following warning
/usr/local/lib/python3.7/dist-packages/feediverse.py:48: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(fh)
Hello,
first of all, thanks for this script.
I didn't manage to include the images/media of a nitter feed to my configuration.
- template: '{title}'
url: https://nitter.net/twitter/rss
This includes the text, but not the images/media.
Since i dont want the Feed URL posted, i removed the '{url}'. But this doesn't change anything.
How can i add the images/media?
Thank you.
Hi there, thanks for this script as i can post rss feeds of pixelfed and SoundCloud uploads to mastodon which using ifttt, means that mastodon updates get posted to Twitter and Facebook.
the issue I have with Facebook (typical) is that when I post links, there are no link previews unless I add a space after the link. as i would add my captions for the photo on pixelfed. then the link to the pixelfed photo and the caption text from pixelfed would be posted onto mastodon.
however for facebook pages, the pixelfed photo caption would be in the facebook status with the link to pixelfed photo under the captioned text. but, without the space its just a link. with a space, i get a link preview, so im wondering if there is a way to fix this phenomom. twitter and mastrondon are unaffected. the only way for now is to rss feed to twitter and facebook page. but with facebook, just do content text and url to post in ifttt options.
also for setup, i do not follow what you mean by adding more rss feeds and wondering if in terminal, there could be the same options but without overwriting the rss feeds i currently have.
access_token: here
client_id: here
client_secret: ' here '
feeds:
- template: '{title} {url} '
url: https://pixelfed.social/users/USERNAMEHERE.atom
include_images: true
name: feediverse
updated: '2020-11-16T16:31:59+00:00'
url: https://mastodon.social
any ideas please
Platform: amd64, debian 10.4 "buster", feediverse-0.0.11, feediverse-0.0.11 feedparser-5.2.1 mastodon.py-1.5.1, python 3.7.3
I have this feed that I'm trying to use feediverse to add to mastodon: https://steinar.bang.priv.no/feed/atom/
The feediverse code tries to add the categories as hashtags https://github.com/edsu/feediverse/blob/master/feediverse.py#L66
But as far as I can tell, the hashtags aren't appearing in mastodon.
Here's a toot created by feediverse: https://mastodon.social/@steinarb/104291594180415614
The toot should have had the hashtags "#apachekaraf, #javaprogramming, #declarativeservices, #derby, #jdbc, #karaf, #liquibase, #osgi, #osgiservices, #paxjdbc, #paxjdbcconfig, #postgresql".
But if the hashtags are present in the toot, I can't see them or find them.
This may be a feature request since I don't find any mentions of limits in the code ?
My idea was to have some kind of limiter, queue, to have the bot to toot only x times per hour, most mastodon instance that allow bots (newsbots.eu or botsin.space) request bot makers to not overload the instance, I think a config param to control this would allow to make bots that are more user friendly and more respectful of the instance rules/code of conduct.
Hey thanks for your work, this bot is very usefull :)
Is there a way to format the template with line breaks?
When I try running for the first time to generate a config file:
cadet:beesbuzz.biz main$ poetry run feediverse
What is your Mastodon Instance URL? https://plush.city
Do you have your app credentials already? [y/n] n
Ok, I'll need a few things in order to get your access token
app name (e.g. feediverse):
Traceback (most recent call last):
File "/Users/fluffy/Library/Caches/pypoetry/virtualenvs/beesbuzz.biz-NFomSJ_p-py3.8/bin/feediverse", line 8, in <module>
sys.exit(main())
File "/Users/fluffy/Library/Caches/pypoetry/virtualenvs/beesbuzz.biz-NFomSJ_p-py3.8/lib/python3.8/site-packages/feediverse.py", line 49, in main
setup(config_file)
File "/Users/fluffy/Library/Caches/pypoetry/virtualenvs/beesbuzz.biz-NFomSJ_p-py3.8/lib/python3.8/site-packages/feediverse.py", line 242, in setup
client_id, client_secret = Mastodon.create_app(
File "/Users/fluffy/Library/Caches/pypoetry/virtualenvs/beesbuzz.biz-NFomSJ_p-py3.8/lib/python3.8/site-packages/mastodon/Mastodon.py", line 280, in create_app
return (response['client_id'], response['client_secret'])
KeyError: 'client_id'
Hi, I'd like to add support for Misskey/Calckey instances to Feediverse.
I've identified an already existing Python module that works with my Calckey instance.
Misskey/Calckey authentication/token generation is different from Mastodon's, it involves explicit browser usage by the user, and the resulting config auth section will be, of course, different.
EDIT I've added Misskey/Calckey support here, by:
If I fix a typo in an already posted blog post, it will cause feediverse to repost that post to mastodon, and I don't want that, so I have to keep an eye on the mastodon feed and delete the repost as soon as I can.
Is it possible to make feediverse not repost already seen blog posts?
Some feeds use day-based stamps, i.e. new articles always show up at 00:00:00Z
of the posting day, while others have a higher timestamp precision. This makes Feediverse lose posts from the former if the latter has posts.
The workaround is to mkdir ~/.feediverse
instead of having a single config there, split them up per feed (initially, everything identical except the feeds
top-level element will have only one entry each; they’ll differ in updated
too once run) and then in the cronjob do /bin/mksh -c 'set -o pipefail; rv=0; for f in ~/.feediverse/*.cfg; do ~/Repos/Upstream/Feediverse/feediverse.py -v -c "$f" 2>&1 | logger -t "rss2mstdn:${f##*/}" || rv=1; done; (( rv == 0 )) || print E: rss2mstdn failed'
(i.e. invoke it once per config file).
The code fix would be to move the timestamp insides the per-feeds
elements.
I had been running Feediverse for months without fault, but since today, I am getting a traceback. How can I possibly fix this?
Traceback (most recent call last):
File "/home/xxx/.local/bin/feediverse", line 11, in <module>
load_entry_point('feediverse==0.2.0', 'console_scripts', 'feediverse')()
File "/home/xxx/.local/lib/python3.6/site-packages/feediverse.py", line 51, in main
config = read_config(config_file)
File "/home/xxx/.local/lib/python3.6/site-packages/feediverse.py", line 107, in read_config
if 'updated' in cfg:
TypeError: argument of type 'NoneType' is not iterable
--- a/feediverse.py
+++ b/feediverse.py
@@ -54,7 +55,11 @@ def main():
if args.dry_run:
print("trial run, not tooting ", entry["title"][:50])
continue
- masto.status_post(feed['template'].format(**entry)[:499])
+ postbody = feed['template'].format(**entry)
+ if len(postbody) > 499:
+ postfix = "…\n\n(more…)"
+ postbody = postbody[:(499 - len(postfix))] + postfix
+ masto.status_post(postbody, visibility='public')
if not args.dry_run:
config['updated'] = newest_post.isoformat()
I was first considering making the length configurable, but posting a longer entry leads to people having to click a button first in Pinafore, for example. So add an indicator that it’s cut off.
Also, set the post visibility — this should probably be a config item, ideally one that can be set globally but overridden per feed.
Is there a way to insert line feeds into templates? I'd like each of {title} {summary} {link} to start on new line.
Currently the entries of the feed are processed on the order they appear in the feed, which normally is newest to oldest. This will lead to the newer entries to be tooted first, which means that the older ones are above the newer ones in the mastodon timeline.
It would be nice if feediverse could be used to slowly (e.g. one toot a day) turn all items of an RSS feed into toots in cronological order.
I use the template "{title} {hashtags} {summary} {url}" and that failed with this error message for a long summary:
sb@cadalora:~$ /usr/local/bin/feediverse
/usr/local/lib/python3.7/dist-packages/feediverse.py:48: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
config = yaml.load(fh)
Traceback (most recent call last):
File "/usr/local/bin/feediverse", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.7/dist-packages/feediverse.py", line 28, in main
masto.status_post(feed['template'].format(**entry)[0:49999999999])
File "<decorator-gen-60>", line 2, in status_post
File "/usr/local/lib/python3.7/dist-packages/mastodon/Mastodon.py", line 102, in wrapper
return function(self, *args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/mastodon/Mastodon.py", line 1776, in status_post
return self.__api_request('POST', '/api/v1/statuses', params, headers = headers, use_json = use_json)
File "/usr/local/lib/python3.7/dist-packages/mastodon/Mastodon.py", line 3429, in __api_request
error_msg)
mastodon.Mastodon.MastodonAPIError: ('Mastodon API returned error', 422, 'Unprocessable Entity', 'Validation failed: Text character limit of 500 exceeded')
sb@cadalora:~$
See https://toot.mirbsd.org/@mirabilos/statuses/01GH65F7V9ZK7KQ6YRG3PF0AR2
jschauma!Consequently
)Input feed: http://www.mirbsd.org/wlog.rss
Hello! I'm a starter and still trying to figure out how to use it. So sorry if I ask a stupid question.
Would it be able to change html to readable plain text, and post images on mastodon?
Hey, I regularly get some syntax errrors like this:
user@server ~]$ python3.6 .feediverse
File ".feediverse", line 3
client_secret: Mysectretqwert1234
^
SyntaxError: invalid syntax
The marker (^) marks the end of the secret. I can stop this by adding primes around the secret, but then I get another error at the start of the list of feeds. Any idea what's wrong here? My .feediverse file worked great for over a year, I migrated it to another server, and now I have these errors.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.