audiosutras / feed_bot Goto Github PK

feed_bot-bot-prod-1  | 2024-02-10 05:15:50 ERROR    discord.ext.tasks Unhandled exception in internal background task 'subreddit_task'.
feed_bot-bot-prod-1  | Traceback (most recent call last):
feed_bot-bot-prod-1  |   File "/usr/local/lib/python3.12/site-packages/discord/ext/tasks/__init__.py", line 239, in _loop
feed_bot-bot-prod-1  |     await self.coro(*args, **kwargs)
feed_bot-bot-prod-1  |   File "/code/feed_bot/bot.py", line 98, in subreddit_task
feed_bot-bot-prod-1  |     await self.post_subreddit(*args, **kwargs)
feed_bot-bot-prod-1  |   File "/code/feed_bot/bot.py", line 146, in post_subreddit
feed_bot-bot-prod-1  |     await channel.send(embeds=[embed])
feed_bot-bot-prod-1  |   File "/usr/local/lib/python3.12/site-packages/discord/abc.py", line 1561, in send
feed_bot-bot-prod-1  |     data = await state.http.send_message(channel.id, params=params)
feed_bot-bot-prod-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
feed_bot-bot-prod-1  |   File "/usr/local/lib/python3.12/site-packages/discord/http.py", line 745, in request
feed_bot-bot-prod-1  |     raise HTTPException(response, data)
feed_bot-bot-prod-1  | discord.errors.HTTPException: 400 Bad Request (error code: 50035): Invalid Form Body
feed_bot-bot-prod-1  | In embeds.0.url: Not a well formed URL.



This should be added as a test as part of issue #22

This PR remediates a ValueError bug that is thrown when more than 10 embeds are tried to be sent to a channel. Now sending embeds in batches.

In asyncpraw you can fetch multiple subreddits at once. Let's retrieve a channel's subreddits and fetch new listings at one in order to cut down on api requests.

This ticket is to start consuming and posting website rss feeds to discord.

Some rss feed's selftext is an html string. Use beautifulsoup to parse selftext if html elements are detected.

Ex:

• https://www.reddit.com/r/reticulum/ --> reticulum
• r/reticulum/ --> reticulum

A couple of bugs need to be worked out of 0.2.6 docker image. Use 0.2.5 but limit the amount of rss feeds you are following.

Also looking to retire this project in favor another that utilizes webhooks and its own self-hostable server. More on that in the future.

For example:

discord.ext.commands.errors.CommandNotFound: Command "subredit" is not found
```

Let end users know where they can support the project.

• Include a url and qr code to kofi
• amazon affiliate link

Need to make sure only unique entries are stored in database.

See corbett report 'February Open Thread' for an example of a duplicated entry. inserted_entries are being sent to channels after being inserted into the db. inserted_entries should be unique.

To access database:

$  docker compose exec mongo mongosh mongodb://root:feed_bot@mongo:27017/
$ use feed_bot_db
$ db.rss.find({feed_url: 'https://corbettreport.com/feed', title: 'February Open Thread'})



Assign a unique rss feed colour for the source.

• Reddit stays orange
• website stays blue
• substack set to green

Create a dictionary for rss feed source colors, check domains against dictionary for assigning colours. Update dictionary when a new source is found

Do more to add strong typing for a better understanding of the code for yourself and others. Add typing across all files where possible.

If strapped for time focus rss workflow.

Add unit testings of REDDITRSS and any logic that is not directly tied to discord.py.


For an entry when a user adds an emoji (like a paper emoji) dm that user with the full content if an article. TBD"


Need to add a remove subreddit from channel feed command. Doing so removes any unsent posts for that subreddit in the particular channel as well.

async with ctx.typing():
	# expensive operations

This is will make the bot show up as typing in discord

This error occurs when the bot attempts to send a message to a channel that has been deleted.

List the subreddits that are available within a particular channel

Same error we thought we resolved in Issue #20

feed_bot-bot-prod-1  |   File "/usr/local/lib/python3.12/site-packages/discord/ext/tasks/__init__.py", line 239, in _loop
feed_bot-bot-prod-1  |     await self.coro(*args, **kwargs)
feed_bot-bot-prod-1  |   File "/code/feed_bot/bot.py", line 98, in subreddit_task
feed_bot-bot-prod-1  |     await self.post_subreddit(*args, **kwargs)
feed_bot-bot-prod-1  |   File "/code/feed_bot/bot.py", line 146, in post_subreddit
feed_bot-bot-prod-1  |     await channel.send(embeds=[embed])
feed_bot-bot-prod-1  |   File "/usr/local/lib/python3.12/site-packages/discord/abc.py", line 1561, in send
feed_bot-bot-prod-1  |     data = await state.http.send_message(channel.id, params=params)
feed_bot-bot-prod-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
feed_bot-bot-prod-1  |   File "/usr/local/lib/python3.12/site-packages/discord/http.py", line 745, in request
feed_bot-bot-prod-1  |     raise HTTPException(response, data)
feed_bot-bot-prod-1  | discord.errors.HTTPException: 400 Bad Request (error code: 50035): Invalid Form Body
feed_bot-bot-prod-1  | In embeds.0.url: Scheme "/r/trading212/comments/1amu4an/i_made_a_trading_212_dashboard_free_to_use/" is not supported. Scheme must be one of ('http', 'https'). Not a well formed URL.

• We need to get unique together channel_id and subreddit columns out of the database.
• Then run get_channel_subreddit_dicts
• Then check if a document already exists in the database. If not insert this document

For this task see:

• reddit.py get_channel_subreddit_dicts
• cogs.py update_or_insert_documents definition within subreddit group

Need to fix update_or_insert_documents.

• move it to bot class
• break into two parts, find_one and insert_one

While implementing RSS feeds it occurred to me that I'm only working RSS 2.0 feeds. Need to test and make sure that atom feeds are compatible with workflow.

Some RSS Feeds have Entry image urls such as this one.

Let's think about how to present this image if available.

The cavaet here is we need to make sure that no other channels are also subscribed to the feed_url. If no other channel is subscribed delete feed_url related entries.

We need better logging for rss feeds. There seems to be an issue where an error is occurring and our loop is stopping.

For example if you delete a channel while a background task is sending embeds to said channel an error is generated and the task stops. Make sure we handle exceptions and restart the task.

.subreddit start should be refactored to automatically start when bot is up.

• Need .subreddit rm command
• Need .subrredit prune command

Have earliest published added first to Feed Updates and send to channel the latest published embed last

Create a couple environment variables for task.loop intervals and update documentation.

• LOOP_SECONDS
• LOOP_MINUTES
• LOOP_HOURS
  

• Except r/<subreddit> and <subreddit> for add definiton in cog
• Handle errors generated by API calls to reddit
• Handle space seperated input and comma seperated input for add method in cog.py

Eventually would like to copy the rss workflow for reddit. Only associate a subreddit name with a channel. Get the channels that subscribed to that subreddit. Send new listings to channels and store the new listing in the db.
See workflow for rss for more information. 

We need a script to preserve the db before killing the containers and repopulate the database after building the new image

User should be able to filter on an entry's dt_published key.

Allow users to get a csv file for their subscriptions.

• .rss export
• .subreddit export

Exporting feed_urls as a comma seperated list will make importing again very easy



Relevant resources:

• docker hub page for mongo
• using volumes with docker compose