oxylabs / selenium-proxy-integration-python Goto Github PK

I got TypeError:

WebDriver.init() got multiple values for argument 'options'

Hi, I have created a package named botasaurus-proxy-authentication, which enables SSL support for proxies requiring authentication.

For instance, when using an authenticated proxy with a tool like seleniumwire to scrape a Cloudflare-protected website such as G2.com, a non-SSL connection typically results in being blocked.

To illustrate, run this code:

First, install the required packages:

python -m pip install selenium_wire chromedriver_autoinstaller

Then, execute this Python script:

from seleniumwire import webdriver
from chromedriver_autoinstaller import install

# Define the proxy
proxy_options = {
    'proxy': {
        'http': 'http://username:password@proxy-provider-domain:port', # Replace with your proxy
        'https': 'http://username:password@proxy-provider-domain:port', # Replace with your proxy
    }
}

# Install and set up the driver
driver_path = install()
driver = webdriver.Chrome(driver_path, seleniumwire_options=proxy_options)

# Navigate to the desired URL
link = 'https://www.g2.com/products/github/reviews'
driver.get("https://www.google.com/")
driver.execute_script(f'window.location.href = "{link}"')

# Wait for user input
input("Press Enter to exit...")

# Clean up
driver.quit()

You'll likely be blocked by Cloudflare:

First, install the required packages:

python -m pip install botasaurus-proxy-authentication

However, using botasaurus_proxy_authentication with proxies circumvents this problem. Notice the difference by running the following code:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from chromedriver_autoinstaller import install
from botasaurus_proxy_authentication import add_proxy_options

# Define the proxy settings
proxy = 'http://username:password@proxy-provider-domain:port'  # Replace with your proxy

# Set Chrome options
chrome_options = Options()
add_proxy_options(chrome_options, proxy)

# Install and set up the driver
driver_path = install()
driver = webdriver.Chrome(driver_path, options=chrome_options)

# Navigate to the desired URL
link = 'https://www.g2.com/products/github/reviews'
driver.get("https://www.google.com/")
driver.execute_script(f'window.location.href = "{link}"')

# Wait for user input
input("Press Enter to exit...")

# Clean up
driver.quit()

Result:

I suggest using botasaurus_proxy_authentication for its SSL support for authenticated proxies, improving the success rate of scraping Cloudflare-protected websites and thus increasing revenue for Oxylabs.
Also, Thanks Oxylabs for your Great Work in Proxy.
Good Luck to the Team.

python version 3.10
MacOS m1 14.1.2 (23B92)

selenium-wire==5.1.0
webdriver-manager==4.0.1

Traceback (most recent call last): File "/Documents/projects/selenium/main.py", line 4, in <module> from seleniumwire import webdriver File "/Documents/projects/selenium/.venv/lib/python3.10/site-packages/seleniumwire/webdriver.py", line 28, in <module> from seleniumwire import backend, utils File "/Documents/projects/selenium/.venv/lib/python3.10/site-packages/seleniumwire/backend.py", line 4, in <module> from seleniumwire.server import MitmProxy File "/Documents/projects/selenium/.venv/lib/python3.10/site-packages/seleniumwire/server.py", line 5, in <module> from seleniumwire.handler import InterceptRequestHandler File "/Documents/projects/selenium/.venv/lib/python3.10/site-packages/seleniumwire/handler.py", line 5, in <module> from seleniumwire import har File "/Documents/projects/selenium/.venv/lib/python3.10/site-packages/seleniumwire/har.py", line 11, in <module> from seleniumwire.thirdparty.mitmproxy import connections File "/Documents/projects/selenium/.venv/lib/python3.10/site-packages/seleniumwire/thirdparty/mitmproxy/connections.py", line 10, in <module> from seleniumwire.thirdparty.mitmproxy.net import tls, tcp File "/Documents/projects/selenium/.venv/lib/python3.10/site-packages/seleniumwire/thirdparty/mitmproxy/net/tls.py", line 15, in <module> import seleniumwire.thirdparty.mitmproxy.options File "Documents/projects/selenium/.venv/lib/python3.10/site-packages/seleniumwire/thirdparty/mitmproxy/options.py", line 5, in <module> from seleniumwire.thirdparty.mitmproxy import optmanager File "/Documents/projects/selenium/.venv/lib/python3.10/site-packages/seleniumwire/thirdparty/mitmproxy/optmanager.py", line 9, in <module> import blinker._saferef ModuleNotFoundError: No module named 'blinker._saferef'