Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️♂️ when scraping the web?
Home Page: https://niespodd.github.io/browser-fingerprinting/
I found a new robot detection site that may help:
https://www.browserscan.net/bot-detection
Curious to know if any open-source stealth browsers exist?
FWIW: Firefox
Uncaught TypeError: window.performance.memory is undefined
o https://niespodd.github.io/browser-fingerprinting/assets/index.e859646e.js:1
webgl is disabled
THREE.WebGLRenderer: Error creating WebGL context. [vendor.0d07671f.js:55:476104](https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js)
Error: Error creating WebGL context.
Vz https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:55
e https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:64
rW https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:64
rW https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:64
dl https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:27
kc https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:27
unstable_runWithPriority https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:18
Uo https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:27
Tc https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:27
pc https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:27
Wo https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:27
unstable_runWithPriority https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:18
Uo https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:27
Wo https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:27
Vo https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:27
lc https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:27
Es https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:27
e https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:64
l https://niespodd.github.io/browser-fingerprinting/assets/vendor.0d07671f.js:64
This claim about Bright Data is false and misleading. Why are you being so antagonistic?
Hi, you've listed a bunch of "antidetect browsers", but Dolphin Anty is missing, it's currently quite popular. It's free for 10 browser profiles or less, so you can test the latest and greatest version for free. What you think about it?
just a question where does this stand on the DMCA?
I'm fairly sure website owners are the copyright holder of their content and are perfectly within their rights to prevent anyone including software from seeing their content, so would this be classed as a circumnavigation of digital security and if so is that illegal under the DMCA? the US law is Title 17, Chapter 12, section 1201.
If I'm wrong please inform me because this seems like a powerful tool but then could its use be a crime?
Just wanted to drop by and say thanks. It's good to be aware of those techniques. It's insanely complex to not get detected.
What kind of scraping setup do you suggest?
I am currently going with something like this, what do you think?
/**
* This test uses the real Google Chrome browser and not a precompiled puppeteer binary.
*
* Furthermore, we start the browser manually and not with puppeteer.
*/
const puppeteer = require('puppeteer-core');
const exec = require('child_process').exec;
const fs = require('fs');
// change this when necessary
const GOOGLE_CHROME_BINARY = '/usr/bin/google-chrome-stable';
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
function execute(command, callback){
exec(command, function(error, stdout, stderr){ callback(stdout); });
}
/**
* Poll browser.log periodically until we see the wsEndpoint
* that we use to connect to the browser.
*/
async function getWsEndpoint() {
let wsEndointFile = './browser.log';
for (let i = 1; i <= 10; i++) {
await sleep(500);
if (fs.existsSync(wsEndointFile)) {
let logContents = fs.readFileSync(wsEndointFile).toString();
var regex = /DevTools listening on (.*)/gi;
let match = regex.exec(logContents);
if (match) {
return match[1];
}
}
}
console.log('Could not get wsEndpoint');
process.exit(0);
}
(async () => {
// start browser
const command = GOOGLE_CHROME_BINARY + ' --remote-debugging-port=9222 --no-first-run --no-default-browser-check 2> browser.log &';
execute(command, (stdout) => {
console.log(stdout);
});
// now connect to the browser
// we do not start the brwoser with puppeteer,
// because we want to influence the startup process
// as little as possible
const browser = await puppeteer.connect({
browserWSEndpoint: await getWsEndpoint(),
defaultViewport: null,
});
const page = await browser.newPage();
await page.goto('https://google.com');
await sleep(1000);
await page.screenshot({path: "bot.png", fullPage: true});
await page.close();
await browser.close();
})();Hi @niespodd ,
Great summary on browser-fp!
would you please share more details on the anti-anti-bot solution Specialized bot software that targets the unique detection surface of the target website ?
looking forward to your comments, thank you!
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.