thelovinator1 / discord-twitter-webhooks Goto Github PK

View Code? Open in Web Editor NEW

109.0 9.0 24.0 1.39 MB

🤖Stream tweets to Discord

License: GNU General Public License v3.0

Python 59.68% Dockerfile 1.17% CSS 0.11% HTML 39.03%

discord python webhook webhooks twitter tweepy discord-twitter-webhooks tweets bot twitter-api

discord-twitter-webhooks's People

Contributors

Stargazers

Watchers

discord-twitter-webhooks's Issues

This stream has been disconnected for operational reasons

Hello again! When a tweet have no text, this triggers an error and shut down the stream.

19:23:23 ERROR Received errors: [{'title': 'operational-disconnect', 'disconnect_type': 'OperationalDisconnect', 'detail': 'This stream has been disconnected for operational reasons.', 'type': 'https://api.twitter.com/2/problems/operational-disconnect'}]
19:23:23 ERROR No text found for tweet
19:23:23 ERROR Stream encountered an exception
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\pypoetry\Cache\virtualenvs\discord-twitter-webhooks-FHtO4lAG-py3.9\lib\site-packages\tweepy\streaming.py", line 91, in _connect
    self.on_data(line)
  File "C:\Users\Administrator\AppData\Local\pypoetry\Cache\virtualenvs\discord-twitter-webhooks-FHtO4lAG-py3.9\lib\site-packages\tweepy\streaming.py", line 928, in on_data
    self.on_response(
  File "C:\Users\Administrator\Desktop\Nintendo Italia\Nintendo Twitter\discord_twitter_webhooks\main.py", line 119, in on_response
    main(response)
  File "C:\Users\Administrator\Desktop\Nintendo Italia\Nintendo Twitter\discord_twitter_webhooks\main.py", line 61, in main
    if data.entities:
AttributeError: 'NoneType' object has no attribute 'entities'
19:23:23 INFO Stream disconnected

In the logs, show which feed a tweet matches

Related to #158, it would be good if the log output shows which feed a tweet is associated with. Currently, it appears that neither the group name or uuid, nor feed name or uuid, is included in log output when a tweet is posted, or if a tweet is rejected for not matching a whitelist (and presumably a blacklist).

Webhook status code 400

Settings used
update every 1 minute

What to send:

Output from docker logs -f discord-twitter-webhooks

2023-12-31 at 11:59:15 DEBUG Webhook URL: https://discord.com/api/webhooks/<redacted>
Webhook status code 400: {"embeds": ["0"]}
2023-12-31 at 11:59:19 ERROR Got 400 from <discord_webhook.webhook.DiscordWebhook object at 0x7f9890eb65d0>. Response: {"embeds": ["0"]}
INFO:     10.25.0.4:64753 - "GET / HTTP/1.1" 200 OK
INFO:     10.25.0.4:64753 - "GET /favicon.svg HTTP/1.1" 200 OK
INFO:     10.25.0.4:64881 - "GET /settings HTTP/1.1" 200 OK
INFO:     10.25.0.4:64881 - "GET /favicon.svg HTTP/1.1" 200 OK
INFO:     10.25.0.4:64889 - "GET / HTTP/1.1" 200 OK
INFO:     10.25.0.4:64889 - "GET /favicon.svg HTTP/1.1" 200 OK

I also don't get any embeds posted in my webhook, I've tried changing webhooks to a new one too

Add Keyword Filter for Tweets

In the previous version, we could filter for Tweets containing specific keywords in the search query. For example, the following query would only post Tweets to the webhook if the words "sale", "discount" or "deal" were in the Tweet body.

(sale OR discount OR deal) from:Steam

This functionality could be re-implemented in the rewrite by adding a text field containing the required keywords for a feed.

Nonetype error

Every few days or so, I keep getting this error:
discord-twitter-webhooks: An unhandled exception was raised while streaming. Shutting down
Exception: 'NoneType' object has no attribute 'startswith'

Add Toggle for Tweets Containing Media

In the previous version, we could filter to only Tweets with media attached using the has:media flag in the search query.

It would be great to see this functionality return as a togglable feed option!

Tweets matching two feeds, one with whitelist and one without, are posted to neither.

Since it seems nitter.net has gotten the RSS back up and running (for now!), I got a bunch of tweets posted in my Discord server today. From reading the issues on nitter's repo, I knew it was nearly ready, and in preparation I added a new feed. It does not seem to have worked, though other feeds did, and I'm trying to troubleshoot.

I have one feed set as a 'catch-all' to post all tweets and replies from NHC_Atlantic to one webhook, and another feed set for tweets and replies from NHC_Atlantic with Franklin whitelisted going to a different webhook.

The following tweet should have matched both, but instead it matches neither. Since other tweets from NHC_Atlantic went through, I guess it's something with the second feed with whitelist causing both feeds to fail?

Log using blockquote instead of codeblock as it's all on one line, this should be easier to read:

2023-08-21 at 14:51:50 INFO Webhook posted for https://nitter.lovinator.space/NHC_Atlantic/status/1693634581017026628#m
2023-08-21 at 14:51:50 INFO Skipping entry Entry(id='https://nitter.lovinator.space/NHC_Atlantic/status/1693634581017026628#m', updated=None, title='Tropical Storm #Franklin Advisory 4: Franklin Slowing Down Over the Caribbean Sea. Heavy Rainfall Expected For Puerto Rico and Hispaniola. http://hurricanes.gov', link='https://nitter.lovinator.space/NHC_Atlantic/status/1693634581017026628#m', author='@NHC_Atlantic', published=datetime.datetime(2023, 8, 21, 14, 42, 4, tzinfo=datetime.timezone.utc), summary='
Tropical Storm #Franklin Advisory 4: Franklin Slowing Down Over the Caribbean Sea. Heavy Rainfall Expected For Puerto Rico and Hispaniola. hurricanes.gov
', content=(), enclosures=(), read=False, read_modified=None, important=None, important_modified=None, added=datetime.datetime(2023, 8, 21, 14, 51, 45, 465354, tzinfo=datetime.timezone.utc), added_by='feed', last_updated=datetime.datetime(2023, 8, 21, 14, 51, 45, 465354, tzinfo=datetime.timezone.utc), original_feed_url='https://nitter.lovinator.space/NHC_Atlantic/rss', feed=Feed(url='https://nitter.lovinator.space/NHC_Atlantic/rss', updated=None, title='National Hurricane Center / @NHC_Atlantic', link='https://nitter.lovinator.space/NHC_Atlantic', author=None, subtitle='Twitter feed for: @NHC_Atlantic. Generated by nitter.lovinator.space', version='rss20', user_title=None, added=datetime.datetime(2023, 7, 19, 17, 19, 0, 70563, tzinfo=datetime.timezone.utc), last_updated=datetime.datetime(2023, 8, 21, 14, 51, 45, 465354, tzinfo=datetime.timezone.utc), last_exception=None, updates_enabled=True)) as it is not whitelisted

The 'Skipping entry' line is repeated again exactly, suggesting it was compared to both feeds.

Reality check, here's a screenshot of my 'catch-all' feed:

and the feed with 'Franklin' whitelisted:

They are identical with the exception of the webhook address, and one has a whitelist item while the other does not.

Only 1 Tweet Posted per Feed Fetch

I noticed a decrease in Tweets sent to my feed webhooks recently, so I used a throwaway account to test my theory. It appears that, upon finding a new, valid Tweet in the feed, the oldest timestamp is set to that Tweet's creation - meaning any further new, valid Tweets made since the last fetch will not be posted.

Using a test account, tweet 3 separate Tweets: "1", "2", "3"
Observe that only Tweet "3" (the latest) will be posted to the webhook

In logs, I see missed Tweets as Skipping entry ... as it is older than the oldest tweet we have despite never being posted.

Typo in the Usernames Field Description of a Feed

In the Web UI, the description beneath the Usernames text field says "webhook" instead of "username."

Bot Disabled / Old Tweets Spamming Webhook

When this app receives a bad gateway response from the configured Nitter instance, it seems to consider all Tweets, regardless of age, as new and sends them to the Feed Webhook. Note there is a lot of log spam and I'm not 100% confident in the cause.

2023-07-21 at 18:07:00 INFO  Skipping entry Entry(id='https://nitter.lovinator.space/SHGames/status/1453410867656146944#m', updated=None, title="RT by @AdamIscove: It's been an honor to work with legendary composer @BearMcCreary on the score for #Vanguard\n\nHere's a look behind the scenes of how it all came together 👀", link='https://nitter.lovinator.space/SHGames/status/1453410867656146944#m', author='@SHGames', published=datetime.datetime(2021, 10, 27, 17, 18, 48, tzinfo=datetime.timezone.utc), summary='<p>It\'s been an honor to work with legendary composer <a href="https://nitter.lovinator.space/bearmccreary" title="Bear McCreary 🐻🎶">@BearMcCreary</a> on the score for <a href="https://nitter.lovinator.space/search?q=%23Vanguard">#Vanguard</a><br />\n<br />\nHere\'s a look behind the scenes of how it all came together 👀</p>\n<img src="https://nitter.lovinator.space/pic/media%2FFCuM7LEWEAAjcjQ.jpg" />', content=(), enclosures=(), read=False, read_modified=None, important=None, important_modified=None, added=datetime.datetime(2023, 7, 21, 17, 45, 29, 579605, tzinfo=datetime.timezone.utc), added_by='feed', last_updated=datetime.datetime(2023, 7, 21, 17, 45, 29, 579605, tzinfo=datetime.timezone.utc), original_feed_url='https://nitter.lovinator.space/AdamIscove/rss', feed=Feed(url='https://nitter.lovinator.space/AdamIscove/rss', updated=None, title='Adam Iscove / @AdamIscove', link='https://nitter.lovinator.space/AdamIscove', author=None, subtitle='Twitter feed for: @AdamIscove. Generated by nitter.lovinator.space', version='rss20', user_title=None, added=datetime.datetime(2023, 7, 21, 4, 0, 25, 836343, tzinfo=datetime.timezone.utc), last_updated=datetime.datetime(2023, 7, 21, 17, 45, 29, 579605, tzinfo=datetime.timezone.utc), last_exception=ExceptionInfo(type_name='requests.exceptions.HTTPError', value_str='502 Server Error: Bad Gateway for url: https://nitter.lovinator.space/AdamIscove/rss', traceback_str='Traceback (most recent call last):\n  File "/home/botuser/.local/lib/python3.11/site-packages/reader/_retrievers.py", line 176, in _caching_get\n    response.raise_for_status()\n  File "/home/botuser/.local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status\n    raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://nitter.lovinator.space/AdamIscove/rss\n'), updates_enabled=True)) as it is a retweet

Won't be able to enter the UI after setting up Docker

Hey @TheLovinator1, I added you on Discord Can you help me set that up?
For some reason after Installing your Docker from Docker Hub, I'm not being able to enter the web UI via localhost:8000

My username in Discord ID - 272760338259378176
or just username _ofir

t.co links remain regardless of setting; embedded images are cropped

I've tried the DISABLE_REMOVE_TCO_LINKS setting both ways (because it's not super clear which setting does what) but it still provides the t.co link.

Additionally, is it possible to present the attached image in an uncropped format? Discord does apparently support that, because Tweetshift does it:

Cheers

Fix code scanning alert - Clear-text logging of sensitive information

Tracking issue for:

https://github.com/TheLovinator1/discord-twitter-webhooks/security/code-scanning/2

Custom Nitter Instance Not Working

Nitter, and many of its instances (including nitter.lovinator.space) are currently down due to Twitter changes (zedeus/nitter#983). There are a handful of instances currently using a rewrite branch that are working, but setting the Nitter Instance in this app doesn't appear to function. In the logs, I see the settings are applied, but the app doesn't seem to ever fetch Tweets from the new instance. Restarting the app after applying the setting does not resolve the issue.

2023-08-23 05:09:02.479 | INFO     | discord_twitter_webhooks.reader_settings:get_data_location:36 - Data will be stored in /home/botuser/.local/share/discord_twitter_webhooks
2023-08-23 05:09:02.479 | DEBUG    | discord_twitter_webhooks.reader_settings:get_reader:68 - Data directory is owned by botuser:botuser
INFO:     Started server process [1]
INFO:     Waiting for application startup.
2023-08-23 at 05:09:02 INFO  I will check for new tweets every 1 minutes
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     X:52346 - "GET / HTTP/1.1" 200 OK
INFO:     X:52346 - "GET /settings HTTP/1.1" 200 OK
2023-08-23 at 05:28:13 DEBUG Saved application settings: ApplicationSettings(nitter_instance='https://nitter.cz', deepl_auth_key='', piped_instance='https://piped.video', teddit_instance='https://teddit.net', delay=1)
INFO:     X:52389 - "POST /settings HTTP/1.1" 200 OK

Tweets are not posted

I've tried with a few different accounts/webhooks and nothing gets posted.

user@server# docker run --env-file=/home/TwitterBot/.env 91657908f2fb
08:17:32 INFO Rule 0: RULE="from:GranblueRdmAcct" will get send to https://discord.com/api/webhooks/1073179489057648691/bleh
08:17:35 INFO Rule 1638454858926764038 added to Twitter.com
08:17:35 INFO Stream connected

if I turn on debug mode I can see it create the rule and Twitter accept it, but when I tweet, nothing happens.

Here's the relevant parts from the config:

WEBHOOK_URL=https://discord.com/api/webhooks/1088027533385138176/TEqzA5JzwhfxNu4dIpp17iHPB-CO9LZQrb_Ir6do_SL81guldkf60PFo6u4NVYWZ-zl9
RULE="from:GranblueRdmAcct"
ERROR_WEBHOOK=https://discord.com/api/webhooks/1088027533385138176/TEqzA5JzwhfxNu4dIpp17iHPB-CO9LZQrb_Ir6do_SL81guldkf60PFo6u4NVYWZ-zl9
SEND_ERRORS=True
LOG_LEVEL=DEBUG
TWITTER_IMAGE_COLLAGE_API=https://twitter.lovinator.space/add

everything else is default. I have elevated access from having a developer account from years ago.

(I created a brand new empty server for that webhook)

Fix code scanning alert - Module is imported with 'import' and 'import from'

Tracking issue for:

https://github.com/TheLovinator1/discord-twitter-webhooks/security/code-scanning/4

Embed Link Directly to Twitter, not Nitter

Opening an Issue for tracking this request as I recall seeing a commit commenting it as a TODO.

Either per-feed or globally, it would be great to have a toggle to use native Twitter links in Embeds, rather than Nitter links.

Retweets Are Truncated at 140 Characters

It appears the text of Retweeted Tweets will be truncated to 140 characters. This does not occur for regular Tweets, only Retweets.

Example

Retweet: https://twitter.com/InfinityWard/status/1644093247948484609
Original Tweet: https://twitter.com/COD_Esports/status/1644067889174007828

Embed Timestamp is Not Accurate to Tweet Publish Time

When using Embeds for Tweets, the timestamp will be populated with the time that the Tweet is sent to the Webhook, as opposed to when the Tweet was published, as reported by Twitter/Nitter. In the Nitter RSS feed, this value can be found in the pubDate key.

Keywords

Any chance keywords would work?

Send Tweets in Chronological Order

Currently, Tweets are sent to the Webhook in order of newest to oldest. This can be confusing when there are related Tweets, such as a thread with multiple Tweets, or a Tweet in response to an earlier Tweet.

UX: Add Feed Button

When editing an existing feed, the Add Feed button should change to Save Feed to better communicate that a new feed will not be created - but instead, the existing feed will be updated.
After clicking the Add Feed button, set its state to disabled to better communicate that the app is working. The page load indicator of the browser is not obvious enough and it's not easy to tell if the app has gone unresponsive while it's working in the background.

Reply Tweets Missing From Feed

I've noticed that replies are no longer being sent to the webhook despite my feed being configured to do so. Replies are not included in the base RSS feed but were instead moved to the with_replies RSS feed. I can't seem to find log entires of the reply tweet being skipped, or the with_replies RSS feed being fetched, so I can only assume this is the cause.

Example: https://nitter.lovinator.space/elonmusk/rss vs https://nitter.lovinator.space/elonmusk/with_replies/rss

IndexError: list index out of range

Getting this error when I run the bot

Job "sched_func (trigger: interval[0:15:00], next run at: 2023-07-25 06:53:34 UTC)" raised an exception
Traceback (most recent call last):
  File "C:\Users\MediaCenter\AppData\Local\pypoetry\Cache\virtualenvs\discord-twitter-webhooks-RQC6o9-p-py3.11\Lib\site-packages\apscheduler\executors\base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\MediaCenter\Documents\discord_twitter\discord-twitter-webhooks\discord_twitter_webhooks\main.py", line 399, in sched_func
    send_to_discord(reader)
  File "C:\Users\MediaCenter\Documents\discord_twitter\discord-twitter-webhooks\discord_twitter_webhooks\send_to_discord.py", line 271, in send_to_discord
    if entry.published < the_oldest_tweet[-1].published:
                         ~~~~~~~~~~~~~~~~^^^^
IndexError: list index out of range

env sample

From my reading there should be a sample ENV file but I'm not able to find it.

Dependency Dashboard

This issue provides visibility into Renovate updates and their statuses. Learn more

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Update github/codeql-action action to v2

Check this box to trigger a request for Renovate to run again on this repository

Bug: Can't use multiple webhooks in a single rule

With the latest image, I would get this error when trying to turn up the stack:

dtw  | 2023-04-01 at 18:46:14 ERROR 404: Not Found
dtw  | 404 Not Found - I can't connect to https://discord.com/api/webhooks/(blah blah),https://discord.com/api/webhooks/(another blah blah), please check if it is a valid webhook.

This rule, sending the same notification to two different webhooks, had worked in the previous version.

To check if the webhook URL was valid, I created a separate webhook and rule combo using the second webhook URL (with a slightly different rule, as it can't run the same exact rule in a different webhook). This time it works, so it seems it may not be parsing the comma correctly in WEBHOOK_URLx=https://url1,https://url2,...

Tweets Are Truncated at 280 Characters

The text of all Tweets appears to be limited to 280 characters. However, Twitter Blue allows for up to 10,000 characters while Discord Embed descriptions allow for up to 4,096 characters. Ideally, this project should truncate at the embed description limit.

Example

https://twitter.com/charlieINTEL/status/1646556919752949761

Twitter Restricts Filtered Stream API to Enterprise

Although it has slowly rolled out, as my access was only cut off in the last 24 hours, Twitter now restricts access to the Filtered Stream API that this project relies on to the Enterprise tier. Today, my Docker instance of this project could not authenticate...

Traceback (most recent call last):
  File "/discord_twitter_webhooks/main.py", line 244, in <module>
    start()
  File "/discord_twitter_webhooks/main.py", line 200, in start
    delete_old_rules(stream=stream)
  File "/discord_twitter_webhooks/rules.py", line 23, in delete_old_rules
    old_rules = stream.get_rules()
                ^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/streaming.py", line 480, in get_rules
    return self._make_request(
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/client.py", line 129, in _make_request
    response = self.request(method, route, params=request_params,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/client.py", line 100, in request
    raise Forbidden(response)
tweepy.errors.Forbidden: 403 Forbidden
When authenticating requests to the Twitter API v2 endpoints, you must use keys and tokens from a Twitter developer App that is attached to a Project. You can create a project via the developer portal.

I see there is a rewrite branch of this project that appears to remove the API dependency and instead uses RSS feeds. This appears to be the best solution moving forward. Is this still in active development? Could a Docker image be provided for us to use?

Regardless, thanks for your work, @TheLovinator1

REQUEST: Custom mention role per feed or "everyone"

I managed to add "everyone" in the tweet message, but it affects all feeds, It would be great to add an option to mention a role for each feed.

Recommend adding clarity to envars

Many of the variables in the .env file are ambiguous because of several layers of negation in the verbiage, which leads to confusion and frustration about what the settings will accomplish.

# Don't convert special characters
# Disabled: &gt; and &lt;
# Enabled: > and <
DISABLE_UNESCAPE_TEXT=True

Setting to true could mean disabling the conversion of special characters. Or, it could mean the opposite. The juxtaposition of affirmational and negational words makes it unclear. Something clearer would be:

# Controls the conversion of special characters
# True: > and < are converted to &gt; and &lt;
# False: > and < are left as is
ESCAPE_TEXT=True

The first line states exactly what the feature does without a negating "Don't".
The description of the setting values have the positive setting first,
they use the same variables the user will use (true/false rather than 'Enabled' and 'Disabled'), and
show exactly what the result of the setting will be.

Finally, the variable itself is changed to remove the DISABLE_, and other negational words or prefixes ('UN'), even if that requires reversing the effect, because it's another unnecessary and ambiguous layer of negation. Rather than unescape, it should just be escape.

Another example, REPLACE_SUBREDDIT= makes more sense than DISABLE_REPLACE_SUBREDDIT=, especially because setting this particular feature to DISABLE_REPLACE_SUBREDDIT=True seems to actually enable the replacement of /r/whatever with a clickable link. That's the opposite of what the variable name itself suggests! Really the variable should be something like CREATE_SUBREDDIT_LINK, where setting to true leaves no doubt about what will happen: a subreddit link will be created.

Feature: support for other webhook clients

I want to use this project on other applications supporting webhook like slack. Can we support customizing the webhook url?

Some code to help you

Hey, I saw the TODO that said add support for usernames, so I made this.

IDs = ['MohitVarikuti', 'TarzanTrades', 'CNBC'] 
for index, item in enumerate(IDs):
    IDs[index] = api.get_user(item).id_str

This converts names to IDs

Image Gallery Embed Detached from Tweet Embed

Seeing this cosmetic issue on the latest version. I believe the URL (and title?) of each embed needs to match for the image gallery to merge, allowing for one seamless embed - as you see when pasting the Tweet link into Discord as a user.

Tweet: https://twitter.com/BKTOOR_/status/1683976829189730305

Stream encountered HTTP Error: 406

2019-10-23 13:19:35,288 ERROR Error: 406
2019-10-23 13:19:41,477 ERROR Error: 406
2019-10-23 13:19:52,752 ERROR Error: 406
2019-10-23 13:20:14,164 ERROR Error: 406
2019-10-23 13:20:55,472 ERROR Error: 406
2019-10-23 13:22:20,760 ERROR Error: 406
2019-10-23 13:25:02,640 ERROR Error: 406

Embed Customization Options Are Ignored

Following the recent .env rewrite (370b306), the following options are not properly set in their respective get_settings.py functions. The returned values will always be "", and I'm not sure truncation is needed for URLs?

EMBED_FOOTER_TEXT
EMBED_FOOTER_ICON
EMBED_IMAGE
EMBED_THUMBNAIL

Additionally, I noticed logging for EMBED_FOOTER_TEXT and EMBED_FOOTER_ICON is malformed as it does not make sense.

INFO  Twitter set to Twitter
INFO  https://i.imgur.com/6BAmj0I.png set to https://i.imgur.com/6BAmj0I.png

Lastly, I'd like to suggest defaults for two of these options, based on what the Discord client uses for Twitter link embeds.

EMBED_FOOTER_TEXT = Twitter
EMBED_FOOTER_ICON = https://abs.twimg.com/icons/apple-touch-icon-192x192.png

Embed Adjustments to Match Native Twitter Embeds

In an attempt to match the native Tweet Embed that Discord generates, the following changes are needed for the current Embeds:

Add display name to the Embed Author: Display Name (@username)
Set Embed Footer Text: Twitter
Set Embed Footer Icon to the Twitter logo: https://abs.twimg.com/icons/apple-touch-icon-192x192.png

authenticating requests to the Twitter API v2 endpoints

I've config my .env

WEBHOOK_URL=https://discord.com/api/webhooks/foo/bar
BEARER_TOKEN=AAAAAAAAAAAAAAAAAAAAA
RULE="(from:mytest_account)"

ERROR_WEBHOOK=https://discord.com/api/webhooks/foo/bar
SEND_ERRORS=True
LOG_LEVEL=DEBUG

and config docker-compose to

version: "3"

services:
  discord-twitter-webhooks:
    image: thelovinator/discord-twitter-webhooks
    env_file:
      - .env
    container_name: discord-twitter-webhooks
    environment:
      - WEBHOOK_URL=${WEBHOOK_URL}
      - BEARER_TOKEN=${BEARER_TOKEN}
      - RULE=${RULE}
      - ERROR_WEBHOOK=${ERROR_WEBHOOK}
      - SEND_ERRORS=${SEND_ERRORS}
      - LOG_LEVEL=${LOG_LEVEL}
    stop_grace_period: 30s
    restart: unless-stopped

and when I docker-compose up I got error they want to authen using access token and secret instead of bearer token
or anything missing in my docker-compose environment config ?

discord-twitter-webhooks    | 2023-03-23 at 10:13:05 DEBUG Failed to get a valid value for 'DISABLE_REMOVE_COPYRIGHT_SYMBOLS' which is set to ''. Defaulting to 'False'.
discord-twitter-webhooks    | Traceback (most recent call last):
discord-twitter-webhooks    |   File "/discord_twitter_webhooks/main.py", line 221, in <module>
discord-twitter-webhooks    |     start()
discord-twitter-webhooks    |   File "/discord_twitter_webhooks/main.py", line 217, in start
discord-twitter-webhooks    |     asyncio.run(start_bot())
discord-twitter-webhooks    |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
discord-twitter-webhooks    |     return runner.run(main)
discord-twitter-webhooks    |            ^^^^^^^^^^^^^^^^
discord-twitter-webhooks    |   File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
discord-twitter-webhooks    |     return self._loop.run_until_complete(task)
discord-twitter-webhooks    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
discord-twitter-webhooks    |   File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
discord-twitter-webhooks    |     return future.result()
discord-twitter-webhooks    |            ^^^^^^^^^^^^^^^
discord-twitter-webhooks    |   File "/discord_twitter_webhooks/main.py", line 170, in start_bot
discord-twitter-webhooks    |     await delete_old_rules(stream=stream)
discord-twitter-webhooks    |   File "/discord_twitter_webhooks/rules.py", line 23, in delete_old_rules
discord-twitter-webhooks    |     old_rules = await stream.get_rules()
discord-twitter-webhooks    |                 ^^^^^^^^^^^^^^^^^^^^^^^^
discord-twitter-webhooks    |   File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/asynchronous/streaming.py", line 720, in get_rules
discord-twitter-webhooks    |     return await self._make_request(
discord-twitter-webhooks    |            ^^^^^^^^^^^^^^^^^^^^^^^^^
discord-twitter-webhooks    |   File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/asynchronous/client.py", line 147, in _make_request
discord-twitter-webhooks    |     response = await self.request(method, route, params=request_params,
discord-twitter-webhooks    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
discord-twitter-webhooks    |   File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/asynchronous/client.py", line 118, in request
discord-twitter-webhooks    |     raise Forbidden(response, response_json=response_json)
discord-twitter-webhooks    | tweepy.errors.Forbidden: 403 Forbidden
discord-twitter-webhooks    | When authenticating requests to the Twitter API v2 endpoints, you must use keys and tokens from a Twitter developer App that is attached to a Project. You can create a project via the developer portal.
discord-twitter-webhooks exited with code 1

Cleanup WEBHOOK_URLs in Environment Variables

In addition to the recent .env improvements (370b306), perhaps RULE2-RULE5 could default to WEBHOOK_URL if WEBHOOK_URL2-WEBHOOK_URL5 are not set? This change would avoid the need to have the same webhook listed multiple times in config.

Example:

WEBHOOK_URL="https://discord.com/api/webhooks/1234567890/XXXXXXXXXX"
WEBHOOK_URL2="https://discord.com/api/webhooks/1234567890/XXXXXXXXXX"
WEBHOOK_URL3="https://discord.com/api/webhooks/1234567890/XXXXXXXXXX"
WEBHOOK_URL4="https://discord.com/api/webhooks/1234567890/XXXXXXXXXX"
WEBHOOK_URL5="https://discord.com/api/webhooks/1234567890/XXXXXXXXXX"
RULE="a rule at character limit"
RULE2="character limit forces a new rule"
RULE3="but we want all tweets to 1 webhook"
RULE4="so the webhook is listed 5 times"
RULE5="when it could just be 1 :)"

WEBHOOK_URL="https://discord.com/api/webhooks/1234567890/XXXXXXXXXX"
RULE="only 1 webhook is listed"
RULE2="but all rules will send tweets to it"
RULE3="this makes for"
RULE4="more maintainable config"
RULE5="thanks"

Related: #79

First Username Mention in Reply Tweets is Not Properly Linked

With USERNAME_LINK set to true (default), the first occurrence of a username mention in a reply Tweet will not be properly formatted as a markdown link, as expected. See the following examples...

https://twitter.com/SHGames/status/1644076930319392772

https://twitter.com/InfinityWard/status/1644093882911670274

Fix code scanning alert - Clear-text logging of sensitive information

Tracking issue for:

https://github.com/TheLovinator1/discord-twitter-webhooks/security/code-scanning/1

AttributeError: 'NoneType' object has no attribute 'text'

Hi Joakim!
I got a new error, something which happen randomly for some reason. Here is the log.

17:12:16 ERROR Stream encountered an exception
Traceback (most recent call last):
  File "C:\Users\Administrator\AppData\Local\pypoetry\Cache\virtualenvs\discord-twitter-webhooks-KyRCDMAP-py3.9\lib\site-packages\tweepy\streaming.py", line 91, in _connect
    self.on_data(line)
  File "C:\Users\Administrator\AppData\Local\pypoetry\Cache\virtualenvs\discord-twitter-webhooks-KyRCDMAP-py3.9\lib\site-packages\tweepy\streaming.py", line 928, in on_data
    self.on_response(
  File "C:\Users\Administrator\Desktop\Nintendo Italia\Twitter Nintendo\discord_twitter_webhooks\main.py", line 114, in on_response
    main(response)
  File "C:\Users\Administrator\Desktop\Nintendo Italia\Twitter Nintendo\discord_twitter_webhooks\main.py", line 48, in main
    text = data.text
AttributeError: 'NoneType' object has no attribute 'text'
17:12:16 INFO Stream disconnected

After i got this message, i simply restart the stream with poetry, but this happen like every couple of days.

Watchtower Updated Image - Internal Server Error, KeyError: only_send_if_media

I'm running Watchtower to monitor for Docker image updates. After updating from image 4cf490a83362 to 15400a125ce6, I see an Internal Server Error upon attempting to access the Web UI.

Related #118

INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     REDACTED - "GET / HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/botuser/.local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 428, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/botuser/.local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/botuser/.local/lib/python3.11/site-packages/fastapi/applications.py", line 289, in __call__
    await super().__call__(scope, receive, send)
  File "/home/botuser/.local/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/botuser/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/home/botuser/.local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/home/botuser/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/home/botuser/.local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/home/botuser/.local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/home/botuser/.local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/home/botuser/.local/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/home/botuser/.local/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/home/botuser/.local/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/home/botuser/.local/lib/python3.11/site-packages/fastapi/routing.py", line 273, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/botuser/.local/lib/python3.11/site-packages/fastapi/routing.py", line 190, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/discord_twitter_webhooks/main.py", line 56, in index
    list_of_groups = [get_group(reader, group) for group in groups]
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/discord_twitter_webhooks/main.py", line 56, in <listcomp>
    list_of_groups = [get_group(reader, group) for group in groups]
                      ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/discord_twitter_webhooks/_dataclasses.py", line 89, in get_group
    only_send_if_media=group["only_send_if_media"],
                       ~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'only_send_if_media'
Job "sched_func (trigger: interval[0:15:00], next run at: 2023-07-16 20:40:36 UTC)" raised an exception
Traceback (most recent call last):
  File "/home/botuser/.local/lib/python3.11/site-packages/apscheduler/executors/base.py", line 125, in run_job
    retval = job.func(*job.args, **job.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/discord_twitter_webhooks/main.py", line 389, in sched_func
    send_to_discord(reader)
  File "/app/discord_twitter_webhooks/send_to_discord.py", line 241, in send_to_discord
    group = get_group(reader, str(_group))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/discord_twitter_webhooks/_dataclasses.py", line 89, in get_group
    only_send_if_media=group["only_send_if_media"],
                       ~~~~~^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'only_send_if_media'

Suggest a translation feature

While wandering around Discord, I came across this bot that translates Twitter like this
but the creator of that bot is not releasing the code, so
I tried to implement something like this, but the
Twitter embedded messages sent to this discord-twitter-webhooks to the
I realized that it's not possible for another bot to take a Twitter embedded message sent to this discord-twitter-webhooks as a separate output and send a translated message.
....
In the end, to build the bot I wanted, I had to use the code you wrote to make it a bot and add the translation function.
But I'm a newbie user who uses chat gpt to make discord bot, so I'm not confident to do that well.

I want my discord members to be able to see the translated messages and use them more comfortably... Is it difficult?
The translation APIs I thought of using are Google Translate and Papago Translate.

I apologize if this is an inconvenient suggestion

But I haven't found any other code that is as complete a solution for Twitter streams as yours.
Anyway, thanks for the great code!

Quote Tweets Always Display Nitter Links

Regardless of the "Link to Nitter instead of Twitter" feed setting toggle, Quote Tweets will always display in the Tweet body (embed description) as a Nitter link, not a direct Twitter link,

Tweet: https://twitter.com/TheGhostOfHope/status/1684045346664599553

Related: #124 #123

Discord Canary Webhook URLs Invalid for ERROR_WEBHOOK

The ERROR_WEBHOOK environment variable seems to invalidate webhook URLs copied from the Discord Canary client. This could be fixed by removing canary. from the URL, but seeing as a warning is not thrown for canary URLs on WEBHOOK_URL, this seems to be a bug.

I'm not sure if this prevents errors from being sent or not as I've never received an error on this webhook 👀

WARNING Your ERROR_WEBHOOK does not start with https://discord.com/api/webhooks/, is this correct? It is currently set to 'https://canary.discord.com/api/webhooks/XXXXXXXXXX/XXXXXXXXXX'

Project Uses Unsupported Tweet URL Format

Hi there, I've recently migrated from another project - I appreciate the active development!

One issue I've run into is that the Discord and/or Twitter client on Android doesn't support the /i/web/status/ Tweet URLs that this project uses. When attempting to open these links (via embed author or message content), I'm kicked back to Discord (see video below.) I'm unsure if this issue is present on iOS.

Question is: Can a config variable be added to utilize a working Tweet URL format?

Examples:

Current (Non-Standard?, Does Not Work): https://twitter.com/i/web/status/1637860827133796370
Proposed (Standard, Works): https://twitter.com/charlieINTEL/status/1637860827133796370
Alternative (Non-Standard, Works): https://twitter.com/userId/status/1637860827133796370

Video:

screen-20230320-1232102.mp4

Revenge of the t.cos and UTMs

The t.co shortened URLs are back—sometimes:

In the first tweet (original) the t.co link appears, followed by the unobfuscated link. As you can see in the original tweet, only one link was in the tweet. Curiously, the unobfuscated link includes the ?t=... parameter which does not appear to be included in the original link or used by the volcano discovery website. I'm not sure where that came from.

In the second tweet (original) both of the links supplied in the tweet failed to convert from the t.co link to the original URL.

CONVERT_TCO_LINKS envar was not explicitly defined in the .env.

Lastly, here's the third tweet in the series:

In this tweet (original), there's no t.co link at all, but all the UTM parameters are present. In this case, the REMOVE_UTM=True envar was explicitly defined in the .env.

Running the latest image.

Fatal error when processing linked url for embed when site uses unverified certificates

The application crashes when it tries to access a site without a valid signed ssl certtificate

(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)')))

Full log below (this build was from ~3 days ago)

2023-03-24 at 17:26:23 ERROR Got an error: discord-twitter-webhooks: An unhandled exception was raised while streaming. Shutting down
Exception: HTTPSConnectionPool(host='pages.stern.nyu.edu', port=443): Max retries exceeded with url: /~pschnabl/research/DSS_SVB.pdf (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)')))
Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
                ^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1075, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1346, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pages.stern.nyu.edu', port=443): Max retries exceeded with url: /~pschnabl/research/DSS_SVB.pdf (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/asynchronous/streaming.py", line 84, in _connect
    await self.on_data(line)
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/asynchronous/streaming.py", line 842, in on_data
    await self.on_response(
  File "/discord_twitter_webhooks/main.py", line 161, in on_response
    main(response)
  File "/discord_twitter_webhooks/main.py", line 73, in main
    twitter_card_image = get.meta_image(entities)
                         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/discord_twitter_webhooks/get.py", line 66, in meta_image
    response: requests.Response = requests.get(url_list[0], timeout=5)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/requests/adapters.py", line 563, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='pages.stern.nyu.edu', port=443): Max retries exceeded with url: /~pschnabl/research/DSS_SVB.pdf (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/asynchronous/streaming.py", line 135, in _connect
    await self.on_exception(e)
  File "/discord_twitter_webhooks/main.py", line 156, in on_exception
    sys.exit(error_msg)
SystemExit: discord-twitter-webhooks: An unhandled exception was raised while streaming. Shutting down
Exception: HTTPSConnectionPool(host='pages.stern.nyu.edu', port=443): Max retries exceeded with url: /~pschnabl/research/DSS_SVB.pdf (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:992)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/discord_twitter_webhooks/main.py", line 225, in <module>
    start()
  File "/discord_twitter_webhooks/main.py", line 221, in start
    asyncio.run(start_bot())
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/discord_twitter_webhooks/main.py", line 188, in start_bot
    await stream.filter(
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/asynchronous/streaming.py", line 514, in _connect
    await super()._connect(method, url, headers=headers, **kwargs)
  File "/opt/pysetup/.venv/lib/python3.11/site-packages/tweepy/asynchronous/streaming.py", line 137, in _connect
    await self.session.close()

Add an Additional Embed for Quote Tweets

Perhaps this would be a global or per-feed toggle, but I would like to see the option to include Quoted Tweets as a separate, additional Embed object when posting Tweets to a Webhook.

Currently, a Quoted Tweet will simply appear as a URL at the end of the Embed description. This option would replace that URL with a new Embed, below the original Embed, containing the details of the Quoted Tweet.

thelovinator1 / discord-twitter-webhooks Goto Github PK

discord-twitter-webhooks's People

Contributors

Stargazers

Watchers

Forkers

discord-twitter-webhooks's Issues

Reality check, here's a screenshot of my 'catch-all' feed:

and the feed with 'Franklin' whitelisted:

Open

Recommend Projects

Recommend Topics

Recommend Org

Jobs