Comments (3)
Thank you for the details. We actually could repeat similar issue with Twitter users and timelines. It is likely because of Fetch module that we currently use does not re-use cookies between redirects by the default. We have used our jars elsewhere in the code, but not for Twitter (due to followHTTPredirect
bypass used).
The issue is resolved in #502. Also, we now have disabled Twitter thumbnail by default to avoid issues in the future. To enable, please configure twitter: {thumbnail: true, ...}
in your providerOptions
config.
from iframely.
The plugin is trying to get a thumbnail image if it sees that there is one in the tweet:
iframely/plugins/domains/twitter.com/twitter.status.js
Lines 70 to 74 in c38710d
Around when Twitter announced their monetization attempt for their developer APIs couple months ago, they also added stricter bot restrictions, hence you probably get blocked. The options we set when we fetch the URL are actually there to prevent the plugin from failing on bot rejections.
When you remove options.followHTTPRedirect
, the second option kicks in (exposeStatusCode
), and basically you get a 3xx
redirect code instead. You probably don't get a picture either.
The proper fix would be to disable that fetch altogether via a feature flag. But before we do, it would be interesting to understand what Twitter gives you. Could you perhaps try to cURL that URL and follow the redirection chain manually and see what happens?
from iframely.
Here is what I get when I fetch the pic.twitter URL:
* Preparing request to https://pic.twitter.com/3SGK87Xz2j
* Current time is 2023-05-11T07:16:30.317Z
* Enable automatic URL encoding
* Using default HTTP version
* Disable SSL validation
* Enable cookie sending with jar of 16 cookies
* Trying 104.244.42.131:443...
* Connected to pic.twitter.com (104.244.42.131) port 443 (#10)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; CN=*.twitter.com
* start date: Oct 1 00:00:00 2022 GMT
* expire date: Oct 1 23:59:59 2023 GMT
* issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fbf5211dc00)
> GET /3SGK87Xz2j HTTP/2
> Host: pic.twitter.com
> user-agent: insomnia/2023.1.0
> accept: */*
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 301
< date: Thu, 11 May 2023 07:16:30 GMT
< perf: 7626143928
< vary: Origin
< server: tsa_f
< expires: Thu, 11 May 2023 07:21:30 GMT
< location: https://twitter.com/RockoPeppe/status/582323285825736704/photo/1
* skipped cookie with bad tailmatch domain: t.co
< set-cookie: muc=39a24e40-5d91-4fcc-8e81-f8e911f565ee; Max-Age=34214400; Expires=Mon, 10 Jun 2024 07:16:30 GMT; Domain=t.co; Secure; SameSite=None
< set-cookie: guest_id=v1%3A168378939055769957; Max-Age=34214400; Expires=Mon, 10 Jun 2024 07:16:30 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None
< cache-control: private,max-age=300
< content-length: 0
< x-transaction-id: 926e5921c18b3c41
< strict-transport-security: max-age=631138519
< x-response-time: 108
< x-connection-hash: b65db43e615e4f76b1d0b6a728f7e0e41b8f668dcd243d8bc59518453e796b05
* Connection #10 to host pic.twitter.com left intact
* Issue another request to this URL: 'https://twitter.com/RockoPeppe/status/582323285825736704/photo/1'
* Trying 104.244.42.193:443...
* Connected to twitter.com (104.244.42.193) port 443 (#11)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; CN=twitter.com
* start date: Dec 25 00:00:00 2022 GMT
* expire date: Dec 25 23:59:59 2023 GMT
* issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fbf5211dc00)
> GET /RockoPeppe/status/582323285825736704/photo/1 HTTP/2
> Host: twitter.com
> user-agent: insomnia/2023.1.0
> cookie: guest_id=v1%3A168378939055769957
> accept: */*
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200
< date: Thu, 11 May 2023 07:16:30 GMT
< perf: 7626143928
< expiry: Tue, 31 Mar 1981 05:00:00 GMT
< pragma: no-cache
< server: tsa_f
< set-cookie: ct0=; Max-Age=-1683789389; Expires=Thu, 01 Jan 1970 00:00:01 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=Lax
< content-type: text/html; charset=utf-8
< x-powered-by: Express
< cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
< last-modified: Thu, 11 May 2023 07:16:30 GMT
< x-frame-options: DENY
< x-transaction-id: c711532431cb9e9b
< x-xss-protection: 0
< x-content-type-options: nosniff
< content-security-policy: connect-src 'self' blob: https://*.pscp.tv https://*.video.pscp.tv https://*.twimg.com https://api.twitter.com https://api-stream.twitter.com https://ads-api.twitter.com https://aa.twitter.com https://caps.twitter.com https://pay.twitter.com https://sentry.io https://ton.twitter.com https://twitter.com https://upload.twitter.com https://www.google-analytics.com https://accounts.google.com/gsi/status https://accounts.google.com/gsi/log https://app.link https://api2.branch.io https://bnc.lt https://checkoutshopper-live.adyen.com wss://*.pscp.tv https://vmap.snappytv.com https://vmapstage.snappytv.com https://vmaprel.snappytv.com https://vmap.grabyo.com https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net https://media.riffsy.com https://*.giphy.com https://media.tenor.com https://c.tenor.com ; default-src 'self'; form-action 'self' https://twitter.com https://*.twitter.com; font-src 'self' https://*.twimg.com; frame-src 'self' https://twitter.com https://mobile.twitter.com https://pay.twitter.com https://cards-frame.twitter.com https://accounts.google.com/ https://client-api.arkoselabs.com/ https://iframe.arkoselabs.com/ https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/; img-src 'self' blob: data: https://*.cdn.twitter.com https://ton.twitter.com https://*.twimg.com https://analytics.twitter.com https://cm.g.doubleclick.net https://www.google-analytics.com https://maps.googleapis.com https://www.periscope.tv https://www.pscp.tv https://media.riffsy.com https://*.giphy.com https://media.tenor.com https://c.tenor.com https://*.pscp.tv https://*.periscope.tv https://prod-periscope-profile.s3-us-west-2.amazonaws.com https://platform-lookaside.fbsbx.com https://scontent.xx.fbcdn.net https://scontent-sea1-1.xx.fbcdn.net https://*.googleusercontent.com; manifest-src 'self'; media-src 'self' blob: https://twitter.com https://*.twimg.com https://*.vine.co https://*.pscp.tv https://*.video.pscp.tv https://dhdsnappytv-vh.akamaihd.net https://pdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://mdhdsnappytv-vh.akamaihd.net https://mpdhdsnappytv-vh.akamaihd.net https://mmdhdsnappytv-vh.akamaihd.net https://dwo3ckksxlb0v.cloudfront.net; object-src 'none'; script-src 'self' 'unsafe-inline' https://*.twimg.com https://recaptcha.net/recaptcha/ https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ https://client-api.arkoselabs.com/ https://www.google-analytics.com https://twitter.com https://app.link https://accounts.google.com/gsi/client https://appleid.cdn-apple.com/appleauth/static/jsapi/appleid/1/en_US/appleid.auth.js 'nonce-Y2I5ZTA0NWItZjVkOC00ODBjLWIyNTItOWMwMDVlN2I1N2Ew'; style-src 'self' 'unsafe-inline' https://accounts.google.com/gsi/style https://*.twimg.com; worker-src 'self' blob:; report-uri https://twitter.com/i/csp_report?a=O5RXE%3D%3D%3D&ro=false
< strict-transport-security: max-age=631138519
< cross-origin-opener-policy: same-origin-allow-popups
< cross-origin-embedder-policy: unsafe-none
< x-response-time: 118
< x-connection-hash: 061e7c5cf8e2dfb8ef30f41b88b4636218828b118415f7c422ae1341433a9460
* Received 874 B chunk
* Received 15.7 KB chunk
* Received 2.2 KB chunk
* Received 16 KB chunk
* Received 9 B chunk
* Received 16 KB chunk
* Received 9 B chunk
* Received 16 KB chunk
* Received 16 KB chunk
* Received 18 B chunk
* Received 16 KB chunk
* Received 16 KB chunk
* Received 16 KB chunk
* Received 16 KB chunk
* Received 42 B chunk
* Received 16 KB chunk
* Received 3.8 KB chunk
* Connection #11 to host twitter.com left intact
I only get one 301 and then a 200.
I checked again without commenting options.followHTTPRedirect = true;
and it seems it fails after the first redirect. It doesn't get endless redirect before failing (ie: the maximum redirect reached from the error message).
from iframely.
Related Issues (20)
- Spotify Video podcast do not have the right aspect ratio HOT 1
- Missing attributes for Spotify embeds HOT 2
- Batch querying by content IDs does not work HOT 1
- Facebook public posts redirect to facebook.com/login HOT 4
- Intermittent issues with Google share links HOT 4
- Inspect Network Calls HOT 1
- AWS cli for pushing build image
- New Tumblr embeds HOT 2
- Issue with a long html page without javascripts. HOT 2
- remove racially loaded terms from the website HOT 1
- Core plugins tests are failing in Node 18 HOT 5
- Better aspect ratio for Loom embeds HOT 1
- Tweet with media are failing in self hosted version HOT 1
- Tweet with media are failing in self hosted version HOT 1
- Self-hosted, instagram embed, redirect loop. HOT 9
- cannot parse medium article HOT 1
- Release Schedule HOT 2
- Allow setting configuration file path via environment variable HOT 1
- Figma app link missing in self-hosted version HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from iframely.