audiosutras / feed_bot Goto Github PK
View Code? Open in Web Editor NEWA discord bot for emulating an rss feed reader within your guild channels
Home Page: https://docs.chrisdixononcode.dev/feed_bot.html
License: MIT License
A discord bot for emulating an rss feed reader within your guild channels
Home Page: https://docs.chrisdixononcode.dev/feed_bot.html
License: MIT License
feed_bot-bot-prod-1 | 2024-02-10 05:15:50 ERROR discord.ext.tasks Unhandled exception in internal background task 'subreddit_task'.
feed_bot-bot-prod-1 | Traceback (most recent call last):
feed_bot-bot-prod-1 | File "/usr/local/lib/python3.12/site-packages/discord/ext/tasks/__init__.py", line 239, in _loop
feed_bot-bot-prod-1 | await self.coro(*args, **kwargs)
feed_bot-bot-prod-1 | File "/code/feed_bot/bot.py", line 98, in subreddit_task
feed_bot-bot-prod-1 | await self.post_subreddit(*args, **kwargs)
feed_bot-bot-prod-1 | File "/code/feed_bot/bot.py", line 146, in post_subreddit
feed_bot-bot-prod-1 | await channel.send(embeds=[embed])
feed_bot-bot-prod-1 | File "/usr/local/lib/python3.12/site-packages/discord/abc.py", line 1561, in send
feed_bot-bot-prod-1 | data = await state.http.send_message(channel.id, params=params)
feed_bot-bot-prod-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
feed_bot-bot-prod-1 | File "/usr/local/lib/python3.12/site-packages/discord/http.py", line 745, in request
feed_bot-bot-prod-1 | raise HTTPException(response, data)
feed_bot-bot-prod-1 | discord.errors.HTTPException: 400 Bad Request (error code: 50035): Invalid Form Body
feed_bot-bot-prod-1 | In embeds.0.url: Not a well formed URL.
This should be added as a test as part of issue #22
This PR remediates a ValueError
bug that is thrown when more than 10 embeds are tried to be sent to a channel. Now sending embeds in batches.
In asyncpraw you can fetch multiple subreddits at once. Let's retrieve a channel's subreddits and fetch new listings at one in order to cut down on api requests.
This ticket is to start consuming and posting website rss feeds to discord.
Some rss feed's selftext is an html string. Use beautifulsoup to parse selftext if html elements are detected.
Ex:
https://www.reddit.com/r/reticulum/
--> reticulum
r/reticulum/
--> reticulum
A couple of bugs need to be worked out of 0.2.6
docker image. Use 0.2.5
but limit the amount of rss feeds you are following.
Also looking to retire this project in favor another that utilizes webhooks and its own self-hostable server. More on that in the future.
For example:
discord.ext.commands.errors.CommandNotFound: Command "subredit" is not found
```
Let end users know where they can support the project.
Need to make sure only unique entries are stored in database.
See corbett report 'February Open Thread' for an example of a duplicated entry. inserted_entries
are being sent to channels after being inserted into the db. inserted_entries
should be unique.
To access database:
$ docker compose exec mongo mongosh mongodb://root:feed_bot@mongo:27017/
$ use feed_bot_db
$ db.rss.find({feed_url: 'https://corbettreport.com/feed', title: 'February Open Thread'})
Assign a unique rss feed colour for the source.
Create a dictionary for rss feed source colors, check domains against dictionary for assigning colours. Update dictionary when a new source is found
feed_bot-bot-1 | 2024-04-01 10:23:56 ERROR discord.ext.tasks Unhandled exception in internal background task 'rss_feeds_task'.
feed_bot-bot-1 | Traceback (most recent call last):
feed_bot-bot-1 | File "/usr/local/lib/python3.12/site-packages/discord/ext/tasks/__init__.py", line 239, in _loop
feed_bot-bot-1 | await self.coro(*args, **kwargs)
feed_bot-bot-1 | File "/code/feed_bot/bot.py", line 208, in rss_feeds_task
feed_bot-bot-1 | await self.update_all_rss_feeds(*args, **kwargs)
feed_bot-bot-1 | File "/code/feed_bot/bot.py", line 259, in update_all_rss_feeds
feed_bot-bot-1 | await self.channel_send(
feed_bot-bot-1 | File "/code/feed_bot/bot.py", line 119, in channel_send
feed_bot-bot-1 | await channel.send(*args, **kwargs)
feed_bot-bot-1 | File "/usr/local/lib/python3.12/site-packages/discord/abc.py", line 1545, in send
feed_bot-bot-1 | with handle_message_parameters(
feed_bot-bot-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^
feed_bot-bot-1 | File "/usr/local/lib/python3.12/site-packages/discord/http.py", line 171, in handle_message_parameters
feed_bot-bot-1 | raise ValueError('embeds has a maximum of 10 elements.')
Do more to add strong typing for a better understanding of the code for yourself and others. Add typing across all files where possible.
If strapped for time focus rss
workflow.
Add unit testings of REDDITRSS
and any logic that is not directly tied to discord.py
.
feed_bot-bot-1 | 2024-02-12 16:55:50 ERROR discord.ext.tasks Unhandled exception in internal background task 'subreddit_task'.
feed_bot-bot-1 | Traceback (most recent call last):
feed_bot-bot-1 | File "/usr/local/lib/python3.12/site-packages/discord/ext/tasks/__init__.py", line 239, in _loop
feed_bot-bot-1 | await self.coro(*args, **kwargs)
feed_bot-bot-1 | File "/code/feed_bot/bot.py", line 98, in subreddit_task
feed_bot-bot-1 | await self.post_subreddit(*args, **kwargs)
feed_bot-bot-1 | File "/code/feed_bot/bot.py", line 146, in post_subreddit
feed_bot-bot-1 | await channel.send(embeds=[embed])
feed_bot-bot-1 | File "/usr/local/lib/python3.12/site-packages/discord/abc.py", line 1561, in send
feed_bot-bot-1 | data = await state.http.send_message(channel.id, params=params)
feed_bot-bot-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
feed_bot-bot-1 | File "/usr/local/lib/python3.12/site-packages/discord/http.py", line 745, in request
feed_bot-bot-1 | raise HTTPException(response, data)
feed_bot-bot-1 | discord.errors.HTTPException: 400 Bad Request (error code: 50035): Invalid Form Body
feed_bot-bot-1 | In embeds.0.title: Must be 256 or fewer in length.
```
For an entry when a user adds an emoji (like a paper emoji) dm that user with the full content if an article. TBD"
Need to add a remove subreddit from channel feed command. Doing so removes any unsent posts for that subreddit in the particular channel as well.
async with ctx.typing():
# expensive operations
This is will make the bot show up as typing in discord
This error occurs when the bot attempts to send
a message to a channel that has been deleted.
List the subreddits that are available within a particular channel
Same error we thought we resolved in Issue #20
feed_bot-bot-prod-1 | File "/usr/local/lib/python3.12/site-packages/discord/ext/tasks/__init__.py", line 239, in _loop
feed_bot-bot-prod-1 | await self.coro(*args, **kwargs)
feed_bot-bot-prod-1 | File "/code/feed_bot/bot.py", line 98, in subreddit_task
feed_bot-bot-prod-1 | await self.post_subreddit(*args, **kwargs)
feed_bot-bot-prod-1 | File "/code/feed_bot/bot.py", line 146, in post_subreddit
feed_bot-bot-prod-1 | await channel.send(embeds=[embed])
feed_bot-bot-prod-1 | File "/usr/local/lib/python3.12/site-packages/discord/abc.py", line 1561, in send
feed_bot-bot-prod-1 | data = await state.http.send_message(channel.id, params=params)
feed_bot-bot-prod-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
feed_bot-bot-prod-1 | File "/usr/local/lib/python3.12/site-packages/discord/http.py", line 745, in request
feed_bot-bot-prod-1 | raise HTTPException(response, data)
feed_bot-bot-prod-1 | discord.errors.HTTPException: 400 Bad Request (error code: 50035): Invalid Form Body
feed_bot-bot-prod-1 | In embeds.0.url: Scheme "/r/trading212/comments/1amu4an/i_made_a_trading_212_dashboard_free_to_use/" is not supported. Scheme must be one of ('http', 'https'). Not a well formed URL.
channel_id
and subreddit
columns out of the database.get_channel_subreddit_dicts
For this task see:
get_channel_subreddit_dicts
update_or_insert_documents
definition within subreddit
groupNeed to fix update_or_insert_documents
.
find_one
and insert_one
While implementing RSS feeds it occurred to me that I'm only working RSS 2.0 feeds. Need to test and make sure that atom feeds are compatible with workflow.
Some RSS Feeds have Entry image urls such as this one.
Let's think about how to present this image if available.
The cavaet here is we need to make sure that no other channels are also subscribed to the feed_url
. If no other channel is subscribed delete feed_url
related entries.
We need better logging for rss feeds. There seems to be an issue where an error is occurring and our loop is stopping.
For example if you delete a channel while a background task is sending embeds to said channel an error is generated and the task stops. Make sure we handle exceptions and restart the task.
.subreddit start
should be refactored to automatically start when bot is up.
.subreddit rm
command.subrredit prune
commandHave earliest published added first to Feed Updates
and send to channel the latest published embed last
Create a couple environment variables for task.loop intervals and update documentation.
r/<subreddit>
and <subreddit>
for add definiton in cogadd
method in cog.py
Eventually would like to copy the rss workflow for reddit. Only associate a subreddit name with a channel. Get the channels that subscribed to that subreddit. Send new listings to channels and store the new listing in the db.
See workflow for rss for more information.
We need a script to preserve the db before killing the containers and repopulate the database after building the new image
User should be able to filter on an entry's dt_published
key.
Allow users to get a csv
file for their subscriptions.
.rss export
.subreddit export
Exporting feed_url
s as a comma seperated list will make importing again very easy
Relevant resources:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.