Comments (8)
Maybe cloudflare is detecting the headless Chrome?
from wrp.
The particular website (fanfiction.net) is actually known for being very aggressive about it (or maybe Cloudflare is particularly aggressive about that website? who knows). All I know is that I see 30-50 "checking the security of the connection" pages on an average day (and some days - at every page load), using a regular browser.
I don't think hiding the headless Chrome better will fix it. There has to be a way to pass this check.
from wrp.
Headless Chrome is often used for data scraping. Websites like Cloudflare block it more aggressively.
What exactly happens when you go to the page? Is there an "I'm not a robot" checkbox?
from wrp.
yeah crap, we need to come up with some solution for this
from wrp.
@TheTechRobo Yes, there is. It does the whole song & dance, and then redirects. Assuming the check works, the "browser" would need to be able to navigate, and/or send screen updates or something.
from wrp.
do you have any handy examples of pages that fail this?
from wrp.
Anything on fanfiction.net should qualify. They pursue verification so aggressively, that I have to go through the verification almost every other chapter on my desktop. It happens most often when switching chapters using the prev/next buttons.
That was actually the website I was hoping to use it with, as previously mentioned.
from wrp.
🤷 I have released 4.6.2; it has some basic anti crawler detection bits, also allows setting user agent with a flag; this seems to help the most;
I have tried ./wrp -ua="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
and it seems better
from wrp.
Related Issues (20)
- Help? HOT 2
- display IP address of WRP host HOT 1
- add context retry
- Discord
- Access Denied to website
- WRP unable to load document on Dreamcast web browser
- Google Sign-In not Working HOT 4
- Ad Blocker? HOT 1
- Starting with custom ports, but still default to regular ones
- How to load cookies from file
- No longer working on FreeBSD HOT 2
- AWS App Runner
- dyld: Symbol not found: _clock_gettime HOT 4
- update full page capture routine HOT 2
- Can't build HOT 1
- circumvent headless browser detection HOT 2
- Text boxes do not work HOT 2
- Add CLI argument to override Browser executable Path HOT 1
- add context timeout
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wrp.