Extends Selenium WebDriver classes to include the request function from the Requests library, while doing all the needed cookie and request headers handling.
License: MIT License
The risk I took was calculated, but man, am I bad at math.
If you appreciate my work and want to say thanks, feel free to buy me a coffee.
Hey,
Is there any way I can control HTTP headers when making requests just like the requests library?
Btw, great job on this, thank you!
Greetings:
I am attempting to use the selenium-requests library and pass query string parameters.
I have something like:
self.driver.get(self.url_get)
that works fine to get the page, but if I add:
data = {
"param1": "data1",
"param2": 'data2"
}
self.driver.get(self.url_get, data)
There is no documentation on the main page:
https://pypi.org/project/selenium-requests
about passing query string params, how is this accomplished with 'selenium-requests'?
Also same question about passing header data like:
headers = {
<header-data-key-value-pairs-here>
}
How do I pass query string params with Python GET
Error:
'get() takes 2 positional arguments, but 3 were given'
'self' is the first argument, the URL is the 2nd positional argument, I am trying to pass 'data' as the third argument, and am getting the above error.
Using:
from seleniumrequests import Chrome
Does anyone have a proven, working code snippet from a working example?
Thank You!
This doesn't seem to be a regression in the code, but rather something that only occurs sometimes under certain conditions. From what I can tell, it's an issue with the ChromeDriver itself: When the page that is used to retrieve the default WebDriver headers is served by Selenium Requests and then immediately closed by the JavaScript snippet, the code execution sometimes freezes in the subsequent callback where the current URL is requested, after switching to the already closed window handle in the find_window_handle method. This shouldn't happen; instead a NoSuchWindowException should be raised immediately by the WebDriver which the module would and can handle.
I'm not sure how to fix this yet, and adding ChromeDriver-specific code or artificial wait times seems like a rather bad solution. I reported this bug to the ChromeDriver bug tracker. If you want it to be fixed in a proper manner, maybe star the issue and thus highlight its importance.
So when I requests an URL through
response = webdriver.request('GET', 'https://www.google.com/')
I have the webdriver loading the site, but once it has loaded it, the tab closes by itself. Is there a way to sit on the tab? I can't find so...
i use instaPy library for instagram, but i want to update profile picture, give me ideal when using your lib?
Hello when im trying to run this code:
from seleniumrequests.request import RequestMixin from selenium import webdriver class SpecialDriver(webdriver, RequestMixin): pass custom_webdriver = SpecialDriver() response = custom_webdriver.request('GET', 'https://www.google.com/') print(response)
I get error: module.init() takes at most 2 arguments (3 given)
You know, the property, or method is very useful to get contents from inner requests of website.
Selenium-wire has them, I suggest to add.
hi
after tryng
class MyCustomWebDriver(RequestsSessionMixin, CustomWebDriver):
pass
i got this error
(TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases)
i am using customdriver https://github.com/ultrafunkamsterdam/undetected-chromedriver
this is my code
from seleniumrequests.request import RequestsSessionMixin
import undetected_chromedriver.v2 as CustomWebDriver
class MyCustomWebDriver(RequestsSessionMixin, CustomWebDriver):
pass
sloboz = MyCustomWebDriver()
response = sloboz.request('GET', 'https://www.google.com/')
print(response)
Hello,
I'm in Python3 , I try on Windows and Debian
I have a trouble, I want to laun request using seleniumrequest with chrome.
But, when I send request, it's work 1/3 times.
when it doesn't work , Chrome open another tab, with url : 127.0.0.1:xxxxx ( random port ) ; and I can't do anything.
I must kill python.
Even with timeout option in my request, it's the same.
It's the same trouble with every request ( post or get )
For example, a very simple request.
from seleniumrequests import Chrome
webdriver=Chrome()
page=webdriver.request('GET','https://www.google.com/')
Sometimes it's work, other time I still block on the tab , and do Nothing…
Have you ever see it ?
Thanks
Hello,
I tried the following code with Python 3.5 on Debian Testing:
from seleniumrequests import PhantomJS
driver = PhantomJS()
driver.set_window_size(1024, 768)
request = driver.request('GET', 'http://example.com/')
print(request.text)
driver.save_screenshot("example.png")
The result seeps to be that the requests module makes the request as usual, instead of usind webdriver. The webdriver returns no content.
cj = cookielib.CookieJar()
p = Firefox()
page = p.request('GET', 'https://www.google.com', headers=headers, cookies=cj)
returns:
File "/Library/Python/2.7/site-packages/seleniumrequests/request.py", line 215, in request
cookies.update(kwargs['cookies'])
While
s = requests.Session()
s.post(url, data=parameters, headers=headers, cookies=cj)
works just fine.
Does anyone know why this error occurs?
I'm trying to make post requests with formData and the script happens to hang for long time.
Selenium-Requests sometime fall into a infinite loop . On a windows machine while requesting http://shopeverythingbaby.com/7am-enfant-polar-warmmuffs-bordeaux.html . It just open a normal second (auto close) tab but it didn't close it automatically . But on linux machine using ubuntu 16.04 it works fine .
Below are the logs from Scrapy
08/15/2018 18:08:04 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:52622/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "platformName": "any", "goog:chromeOptions":
{"extensions": [], "args": ["--disable-infobars", "--start-maximized"]}
}}, "desiredCapabilities": {"browserName": "chrome", "version": "", "platform": "ANY", "goog:chromeOptions":
{"extensions": [], "args": ["--disable-infobars", "--start-maximized"]}
}}
08/15/2018 18:08:04 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): 127.0.0.1:52622
08/15/2018 18:08:07 [urllib3.connectionpool] DEBUG: http://127.0.0.1:52622 "POST /session HTTP/1.1" 200 867
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://127.0.0.1:52622/session/eaddf464138563424cbec27fc53a37d2/window_handles {"sessionId": "eaddf464138563424cbec27fc53a37d2"}
08/15/2018 18:08:07 [urllib3.connectionpool] DEBUG: http://127.0.0.1:52622 "GET /session/eaddf464138563424cbec27fc53a37d2/window_handles HTTP/1.1" 200 113
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://127.0.0.1:52622/session/eaddf464138563424cbec27fc53a37d2/window_handle {"sessionId": "eaddf464138563424cbec27fc53a37d2"}
08/15/2018 18:08:07 [urllib3.connectionpool] DEBUG: http://127.0.0.1:52622 "GET /session/eaddf464138563424cbec27fc53a37d2/window_handle HTTP/1.1" 200 111
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://127.0.0.1:52622/session/eaddf464138563424cbec27fc53a37d2/execute {"script": "window.open('http://127.0.0.1:52716/');", "args": [], "sessionId": "eaddf464138563424cbec27fc53a37d2"}
08/15/2018 18:08:07 [urllib3.connectionpool] DEBUG: http://127.0.0.1:52622 "POST /session/eaddf464138563424cbec27fc53a37d2/execute HTTP/1.1" 200 72
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://127.0.0.1:52622/session/eaddf464138563424cbec27fc53a37d2/window_handle {"sessionId": "eaddf464138563424cbec27fc53a37d2"}
08/15/2018 18:08:07 [urllib3.connectionpool] DEBUG: http://127.0.0.1:52622 "GET /session/eaddf464138563424cbec27fc53a37d2/window_handle HTTP/1.1" 200 111
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://127.0.0.1:52622/session/eaddf464138563424cbec27fc53a37d2/window_handles {"sessionId": "eaddf464138563424cbec27fc53a37d2"}
08/15/2018 18:08:07 [urllib3.connectionpool] DEBUG: http://127.0.0.1:52622 "GET /session/eaddf464138563424cbec27fc53a37d2/window_handles HTTP/1.1" 200 113
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://127.0.0.1:52622/session/eaddf464138563424cbec27fc53a37d2/url {"sessionId": "eaddf464138563424cbec27fc53a37d2"}
08/15/2018 18:08:07 [urllib3.connectionpool] DEBUG: http://127.0.0.1:52622 "GET /session/eaddf464138563424cbec27fc53a37d2/url HTTP/1.1" 200 76
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://127.0.0.1:52622/session/eaddf464138563424cbec27fc53a37d2/window_handle {"sessionId": "eaddf464138563424cbec27fc53a37d2"}
08/15/2018 18:08:07 [urllib3.connectionpool] DEBUG: http://127.0.0.1:52622 "GET /session/eaddf464138563424cbec27fc53a37d2/window_handle HTTP/1.1" 200 111
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://127.0.0.1:52622/session/eaddf464138563424cbec27fc53a37d2/window_handle {"sessionId": "eaddf464138563424cbec27fc53a37d2"}
08/15/2018 18:08:07 [urllib3.connectionpool] DEBUG: http://127.0.0.1:52622 "GET /session/eaddf464138563424cbec27fc53a37d2/window_handle HTTP/1.1" 200 111
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://127.0.0.1:52622/session/eaddf464138563424cbec27fc53a37d2/url {"sessionId": "eaddf464138563424cbec27fc53a37d2"}
08/15/2018 18:08:07 [urllib3.connectionpool] DEBUG: http://127.0.0.1:52622 "GET /session/eaddf464138563424cbec27fc53a37d2/url HTTP/1.1" 200 76
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
08/15/2018 18:08:07 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://127.0.0.1:52622/session/eaddf464138563424cbec27fc53a37d2/window_handles {"sessionId": "eaddf464138563424cbec27fc53a37d2"} ....
My code is below,
from seleniumrequests import Firefox
webdriver = Chrome()
response = webdriver.request('GET', 'https://www.google.com/')
print response
but when I run this ：
Traceback (most recent call last):
File "E:\MACD\test.py", line 56, in
webdriver = Chrome()
NameError: name 'Chrome' is not defined
How I can fix it?
How can I use sharepy to make a connection within selenium using selenium-requests?
Should the sharepy connection mechanism be put inside a MyCustomWebDriver, use it to make the connection, and then use (regular) selenium for all of the responses (which would be request objects)?
Or should the 'main' connection be sharepy, and let sharepy get a requests.models.Response and then somehow pass that to selenium using selenium-requests?
Example stack trace:
from seleniumrequests import Chrome
File "/Users/jeff/repos/mint-amazon-tagger/dev_venv/lib/python3.8/site-packages/seleniumrequests/__init__.py", line 8, in <module>
from selenium.webdriver import Opera as _Opera
ImportError: cannot import name 'Opera' from 'selenium.webdriver' (/Users/jeff/repos/mint-amazon-tagger/dev_venv/lib/python3.8/site-packages/selenium/webdriver/__init__.py)
Commit that removed Opera from selenium is here - 10 days ago.
I guess this module simply passes headers and cookies to requests whereas cloudflare v2 protection needs you to decrypt a javascript cipher in order to verify that you're human. Since requests can't render DOM it's failing.
For example:
response = webdriver.request('GET', 'http://stats.nba.com/game/#!/0021500004/')
will only return the contents of http://stats.nba.com/
Using Chrome driver.
Hi , your library is great ! But can i used it with proxy ? Is there some example how it can be used ? im using chromedriver and tried to add chrome_options , driver works OK but when im trying to make a request call im getting pop up tab with ERR_ACCESS_DENIED .
Thanks in advance.
Regards
Aleksandr
First of all: thanks for releasing this! Really appreciate the hard work.
Instead of using a headers global variable, why don't we pass it as a keyword argument on the request method?
Hi, I was trying to use PhantomJs, however I'm keep on getting this error.
response = webdriver.request('GET', 'https://www.google.com/')
TypeError: request() missing 1 required positional argument: 'url'
driver.request('GET', url) always hangs. Ctrl+C yields the following stack trace:
PS C:\cygwin64\home\james\code\py\gva-data> python main.py
Traceback (most recent call last):
File "main.py", line 42, in <module>
main()
File "main.py", line 11, in main
query_id = driver.find_element_by_name('query[query_id]').get_property('value')
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 487, in find_element_by_name
return self.find_element(by=By.NAME, value=name)
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 955, in find_element
'value': value})['value']
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: [name="query[query_id]"]
PS C:\cygwin64\home\james\code\py\gva-data> python main.py
Traceback (most recent call last):
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\socket.py", line 586, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 46, in <module>
File "main.py", line 13, in main
print('GET', 'http://www.gunviolencearchive.org/query')
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\site-packages\seleniumrequests\request.py", line 226, in request
self.add_cookie(cookie_dict)
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 872, in add_cookie
self.execute(Command.ADD_COOKIE, {'cookie': cookie_dict})
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 310, in execute
response = self.command_executor.execute(driver_command, params)
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 466, in execute
return self._request(command_info[0], url, body=data)
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 528, in _request
resp = opener.open(request, timeout=self._timeout)
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 526, in open
response = self._open(req, data)
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 544, in _open
'_open', req)
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 1346, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\urllib\request.py", line 1321, in do_open
r = h.getresponse()
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 1331, in getresponse
response.begin()
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 297, in begin
version, status, reason = self._read_status()
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\http\client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\james\AppData\Local\Programs\Python\Python36-32\lib\socket.py", line 586, in readinto
return self._sock.recv_into(b)
KeyboardInterrupt
Thanks for this encapsulation, it's been extremely helpful. I encountered a problem when using the syntax driver.request('POST', dl_url, stream = True): the returned response object does not produce any content despite receiving a 202 success and despite the headers containing every sign of success ({'Content-Disposition': 'attachment; filename=Report.csv', 'Transfer-Encoding': 'chunked' ...).
stream is a necessary option for POST downloads of chunk-encoded content in the requests library. Based on my understanding of your code, line 211 in request.py seems to leave out the stream option entirely, thus hard-coding in stream=False, as this is the default. Is that reading correct?
My project (and users) are seeing a new issue which started with the release of chrome & chromedriver v106.
Minimal reproduction case (full code here) - pseudocode:
webdriver = Chrome(options=ChromeOptions(),
executable_path=<path to chromedriver>)
webdriver.get('https://www.google.com')
# This fails spuriously:
response = webdriver.request('GET', 'https://www.google.com/images/branding/googlelogo/2x/googlelogo_light_color_272x92dp.png')
response.raise_for_status()
Could you help me triage what's going on? At first glance it seems like the window spawning/window switching isn't working correctly anymore.
I was seeing this on both selenium-requests v2.0.0 and v1.4.1.
Win11:
DevTools listening on ws://127.0.0.1:59212/devtools/browser/fe19f00b-78d2-4317-b61c-47387f710856
[17404:4820:1001/224935.886:ERROR:interface_endpoint_client.cc(689)] Message 0 rejected by interface blink.mojom.WidgetHost
Traceback (most recent call last):
File "C:\Python310\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Python310\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "C:\Users\jeffp\repos\mint-amazon-tagger\mintamazontagger\repro_mac_issue.py", line 32, in <module>
main()
File "C:\Users\jeffp\repos\mint-amazon-tagger\mintamazontagger\repro_mac_issue.py", line 21, in main
response = webdriver.request(
File "C:\Users\jeffp\repos\mint-amazon-tagger\dev_venv\lib\site-packages\seleniumrequests\request.py", line 159, in request
self.requests_session.headers = get_webdriver_request_headers(self, proxy_host=self.__proxy_host)
File "C:\Users\jeffp\repos\mint-amazon-tagger\dev_venv\lib\site-packages\seleniumrequests\request.py", line 76, in get_webdriver_request_headers
webdriver.switch_to.window(original_window_handle)
File "C:\Users\jeffp\repos\mint-amazon-tagger\dev_venv\lib\site-packages\selenium\webdriver\remote\switch_to.py", line 134, in window
self._w3c_window(window_name)
File "C:\Users\jeffp\repos\mint-amazon-tagger\dev_venv\lib\site-packages\selenium\webdriver\remote\switch_to.py", line 142, in _w3c_window
send_handle(window_name)
File "C:\Users\jeffp\repos\mint-amazon-tagger\dev_venv\lib\site-packages\selenium\webdriver\remote\switch_to.py", line 138, in send_handle
self._driver.execute(Command.SWITCH_TO_WINDOW, {'handle': h})
File "C:\Users\jeffp\repos\mint-amazon-tagger\dev_venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "C:\Users\jeffp\repos\mint-amazon-tagger\dev_venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidArgumentException: Message: invalid argument: No target with given id found
(Session info: chrome=106.0.5249.91)
Stacktrace:
Backtrace: Can provide upon request - not useful
Mac:
Traceback (most recent call last):
File "/usr/local/Cellar/[email protected]/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/Cellar/[email protected]/3.8.3_2/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/jeff/repos/mint-amazon-tagger/mintamazontagger/repro_selenium_issue.py", line 32, in <module>
main()
File "/Users/jeff/repos/mint-amazon-tagger/mintamazontagger/repro_selenium_issue.py", line 21, in main
response = webdriver.request(
File "/Users/jeff/repos/mint-amazon-tagger/dev_venv/lib/python3.8/site-packages/seleniumrequests/request.py", line 162, in request
self.requests_session.headers = get_webdriver_request_headers(
File "/Users/jeff/repos/mint-amazon-tagger/dev_venv/lib/python3.8/site-packages/seleniumrequests/request.py", line 79, in get_webdriver_request_headers
webdriver.switch_to.window(original_window_handle)
File "/Users/jeff/repos/mint-amazon-tagger/dev_venv/lib/python3.8/site-packages/selenium/webdriver/remote/switch_to.py", line 134, in window
self._w3c_window(window_name)
File "/Users/jeff/repos/mint-amazon-tagger/dev_venv/lib/python3.8/site-packages/selenium/webdriver/remote/switch_to.py", line 142, in _w3c_window
send_handle(window_name)
File "/Users/jeff/repos/mint-amazon-tagger/dev_venv/lib/python3.8/site-packages/selenium/webdriver/remote/switch_to.py", line 138, in send_handle
self._driver.execute(Command.SWITCH_TO_WINDOW, {'handle': h})
File "/Users/jeff/repos/mint-amazon-tagger/dev_venv/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "/Users/jeff/repos/mint-amazon-tagger/dev_venv/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: disconnected: received Inspector.detached event
(Session info: chrome=106.0.5249.61)
Ubuntu (both errors):
Traceback (most recent call last):
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/bin/mint-amazon-tagger-repro_mac_issue", line 8, in <module>
sys.exit(main())
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/mintamazontagger/repro_mac_issue.py", line 21, in main
response = webdriver.request(
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/seleniumrequests/request.py", line 159, in request
self.requests_session.headers = get_webdriver_request_headers(self, proxy_host=self.__proxy_host)
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/seleniumrequests/request.py", line 76, in get_webdriver_request_headers
webdriver.switch_to.window(original_window_handle)
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/selenium/webdriver/remote/switch_to.py", line 134, in window
self._w3c_window(window_name)
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/selenium/webdriver/remote/switch_to.py", line 142, in _w3c_window
send_handle(window_name)
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/selenium/webdriver/remote/switch_to.py", line 138, in send_handle
self._driver.execute(Command.SWITCH_TO_WINDOW, {'handle': h})
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: disconnected: received Inspector.detached event
(Session info: chrome=106.0.5249.61)
Stacktrace:
#0 0x5624511412c3 <unknown>
...useless stack...
#34 0x7f65e85b3b43 <unknown>
Traceback (most recent call last):
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/bin/mint-amazon-tagger-repro_mac_issue", line 8, in <module>
sys.exit(main())
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/mintamazontagger/repro_mac_issue.py", line 21, in main
response = webdriver.request(
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/seleniumrequests/request.py", line 159, in request
self.requests_session.headers = get_webdriver_request_headers(self, proxy_host=self.__proxy_host)
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/seleniumrequests/request.py", line 76, in get_webdriver_request_headers
webdriver.switch_to.window(original_window_handle)
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/selenium/webdriver/remote/switch_to.py", line 134, in window
self._w3c_window(window_name)
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/selenium/webdriver/remote/switch_to.py", line 142, in _w3c_window
send_handle(window_name)
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/selenium/webdriver/remote/switch_to.py", line 138, in send_handle
self._driver.execute(Command.SWITCH_TO_WINDOW, {'handle': h})
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py", line 429, in execute
self.error_handler.check_response(response)
File "/home/jeff/repos/mint-amazon-tagger/pypi_test_venv/lib/python3.10/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: disconnected: received Inspector.detached event
(failed to check if window was closed: no such frame: Session with given id not found.)
(Session info: chrome=106.0.5249.61)
linux x86_64
I tried using an older version of phantomjs (1.9.2 and 1.9.8) from another issue but the site seems to reject the cookies from that attempt and redirects. I switched to phantomjs 2.1.1 and the request seems to get the response I want but never returns because it times out while trying to update the phantomjs cookies. Here is the code that it seems to fail on:
# Some WebDrivers (PhantomJS) take some time until the new window
# handle has loaded and cookies can be set
start = time.time()
while page_load_timeout < 0 or time.time() - start <= page_load_timeout:
try:
self.add_cookie(cookie_dict)
break
except WebDriverException:
pass
else:
raise TimeoutException('page took too long to load')
This is lines 221 - 231 of request.py of the latest version available for install through pip. On line 211 I have:
response = self._seleniumrequests_session.request(method, url, **kwargs)
With phantomjs 2.1.1 the response.text looks good but with phantomjs 1.9.8 it seems to redirect me to the start page which leads me to expect the cookies don't look right. Please let me know if you need more info. I can provide a sample.
Is it possible to set desired_capabilities and service_args without editing requests.py?
Thanks
Step to reproduce:
def get_digital_pass(context):
c = context
print("="*10)
response = c.browser.request('POST', "mainwebsiteaddr"+'/authentication/challenge',
data={"user.document.id": c.id})
print("challenge hd", response.headers)
print("challenge ct", response.content)
But from server side my request and response looks as anonimus.
This is bug, or may be i do some wrong, please, help me
ADD: Also, may be important, that the site is protected from CSRF
when I executed a request with turned off windows firewall, it worked.
However, when I turned on the firewall, the selenium-request module didn't work.
I think it because the port for request are banned by firewall.
Could you fix it?
I really appreciate this valuable package
Although there is a tagged commit for 1.3.1, it's not marked as the latest release nor was setup.py updated to have version=1.3.1. Due to this, installing this package from pip will get the outdated 1.3 version.
I have set up an SSH tunnel with SOCKS by doing:
ssh -4 -f -N -D 9991 remotehost
When I try to use seleniumrequests to fetch a site through the SOCKS proxy (see below), the driver opens a tab to localhost:<seemingly random port>.
from seleniumrequests import Chrome
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--proxy-server=socks5://localhost:9991')
ch = Chrome(options=options)
ch.request('GET', 'http://google.com')
It does work, however, when I use Selenium driver's get method and, interestingly, the code above also works if the SOCKS sever is in localhost (e.g., ssh -4 -f -N -D 9991 localhost).
Description:
Any GET request attempted with this library ends in the titular error. This includes the (slightly modified) code given as an example in the documentation:
from seleniumrequests import Chrome
webdriver = Chrome()
response = webdriver.request('GET', 'https://www.google.com/')
Environment:
Python 3.8, PyCharm, Windows10
STR:
UnicodeDecodeError exception is raisedCurrent result:
Traceback (most recent call last):
File "C:/JetBrains/projects/PyCharm/xxxxx/test.py", line 61, in <module>
response = webdriver.request('GET', 'https://www.google.com/')
File "C:\JetBrains\projects\PyCharm\xxxxx\venv\lib\site-packages\seleniumrequests\request.py", line 141, in request
self._seleniumrequests_session.headers = get_webdriver_request_headers(self)
File "C:\JetBrains\projects\PyCharm\xxxxx\venv\lib\site-packages\seleniumrequests\request.py", line 60, in get_webdriver_request_headers
server = BaseHTTPServer.HTTPServer(('', port), HTTPRequestHandler)
File "C:\Program Files\Python38\lib\socketserver.py", line 452, in __init__
self.server_bind()
File "C:\Program Files\Python38\lib\http\server.py", line 139, in server_bind
self.server_name = socket.getfqdn(host)
File "C:\Program Files\Python38\lib\socket.py", line 756, in getfqdn
hostname, aliases, ipaddrs = gethostbyaddr(name)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 3: invalid start byte
ImportError: cannot import name 'ChromiumEdge' from 'selenium.webdriver' when trying to run:
from seleniumrequests import Chrome
I have a selenium test that i have been using that goes to a list of pages and checks the http response code using the selenium-Requests module.
I have been running these tests fine when using the firefox webdriver but when i switch to using PhantomJS i don't get a response back and the tests runs indefinitely (Unless i use the page_load_timeout and then it get a timeout exception)
Not sure what i am doing wrong. See below for code that does work (Using Firefox weddriver)
import unittest
from seleniumrequests import Firefox
class flyout_test(unittest.TestCase):
def setUp(self):
self.wd = Firefox()
self.wd.maximize_window()
self.wd.implicitly_wait(10)
def test_page(self):
Page = self.wd.request("GET", "http://google.com")
print (Page)
def tearDown(self):
self.wd.quit()
if __name__ == '__main__':
unittest.main()
If i change the code to this i dont get back a response:
import unittest
from seleniumrequests import PhantomJS
class flyout_test(unittest.TestCase):
def setUp(self):
self.wd = PhantomJS()
self.wd.maximize_window()
self.wd.implicitly_wait(10)
def test_page(self):
Page = self.wd.request("GET", "http://google.com")
print (Page)
def tearDown(self):
self.wd.quit()
if __name__ == '__main__':
unittest.main()
Any help would be much appricated!
Hey,
i've downloaded the 1.3.1 version for some testing. Additionally am I using this with the https://github.com/ultrafunkamsterdam/undetected-chromedriver driver.
I did some light debugging, and found out that when doing my POST request, the in the title mentioned command doesnt finish. Anyone having an idea how to fix this?
hey, i use this tools, code like this:
self.driver = Chrome(...)
response = self.driver.request('get', url)
self.driver.exeucte("some js to scroll")
i test on headless and without headless, seem like page load and closed quickly, my screen shot is a write page, is it driver.request() return response and close page automatically ?
There seems to be a bug where selenium-requests will hang trying to acquire the mutex from the web server.
The root cause of this is that the popup blocker inhibits the window.open JavaScript code.
This is at least a problem when trying to combine it with undetected-chromedriver, see ultrafunkamsterdam/undetected-chromedriver#380, but I would suspect that it can also trigger in other cases.
To fix it, one approach is to do:
opts = ChromeOptions()
opts.add_argument('--disable-popup-blocking')
driver = MyCustomWebDriver(options=opts)
It's possible that there are other approaches too, such as selectively toggling it while still blocking other popups, or using another method to open a tab - someone suggests driver.tab_new(url) in the issue I linked above.
λ py
Python 3.7.3 (default, Jul 25 2020, 13:03:44)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from seleniumrequests import Firefox
>>> webdriver = Firefox()
>>> response = webdriver.request('GET', 'https://www.google.com/')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/owo/.local/lib/python3.7/site-packages/seleniumrequests/request.py", line 166, in request
window_handle = find_window_handle(self, condition)
File "/home/owo/.local/lib/python3.7/site-packages/seleniumrequests/request.py", line 118, in find_window_handle
webdriver.switch_to.window(original_window_handle)
File "/home/owo/.local/lib/python3.7/site-packages/selenium/webdriver/remote/switch_to.py", line 112, in window
self._w3c_window(window_name)
File "/home/owo/.local/lib/python3.7/site-packages/selenium/webdriver/remote/switch_to.py", line 123, in _w3c_window
send_handle(window_name)
File "/home/owo/.local/lib/python3.7/site-packages/selenium/webdriver/remote/switch_to.py", line 119, in send_handle
self._driver.execute(Command.SWITCH_TO_WINDOW, {'handle': h})
File "/home/owo/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/owo/.local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidArgumentException: Message: Expected "handle" to be a string, got [object Undefined] undefined
>>>
Hi there,
due to a change in pypandoc (convert is deprecated) selenium-request does not install.
Can this be fixed?
Furthermore, is there an update planned?
Thanks in advance, dornech
This is the current bug I am facing:
Logging in and navigating through a certain site works through the normal Selenium API. But as soon as I want to do a request through your module, the website returns a 403: Invalid API key. I suspect requests is not sending all the headers needed (Authorization header maybe?). Any ideas why and if so, is there a fix?
Hi,
I was wondering if there are any plans to remove PhantomJS from this package, the newest versions of Selenium do not include it and so this package is not compatible with selenium 4+ (as you say this package is only tested on selenium 3.0.1).
Here is an issue on SeleniumHQ github discussing removal of it: SeleniumHQ/selenium#5295
and in the current version you get an error on import as PhantomJs is no longer part of selenium.webdriver
from selenium.webdriver import Firefox, Chrome, Ie, Opera, Safari, PhantomJS, Android, Remote
E ImportError: cannot import name 'PhantomJS' from 'selenium.webdriver'
I am happy to try and create a PR that removes it if that is helpful.
Thanks
Nick
pytest-selenium provides a "selenium" fixture. I don't see how this would work with seleniumrequests.
ImportError: cannot import name 'Android' from 'selenium.webdriver'
Sample Code:
from seleniumrequests import Chrome
webdriver = Chrome()
response = webdriver.request('GET', 'https://www.google.com/')
print(response)
Hi 😄
Thanks for writing this library! It's a neat solution, and it made doing some cookie based auth a breeze.
Recently (yesterday? I just noticed it today, but I see there was a release yesterday) I started running into the following error in an automated test pipeline that utilizes selenium & selenium-requests.
from seleniumrequests.request import RequestMixin
ImportError: cannot import name 'RequestMixin' from 'seleniumrequests.request'
The project being run in the pipeline was using selenium-requests = "^1.3" to ensure compatibility, but I've since had to lock it to selenium-requests = "1.3" to resolve the import problem.
I'd love to be able to rely on the minor version again, but having breaking changes come in a PATCH version makes me nervous.
Going forward, would it be possible to bump the major version if there is going to be a significant change like dropping support for a version of Python / something incompatible?
Thank you!! 🙇
I'm able to open login.live.com successfully using the browser. I'm also able to get the response code through urllib. But when I try to use this package to get the response code it times out. It might because this website forces us to use https protocol.
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/seleniumrequests/request.py", line 141, in request self._seleniumrequests_session.headers = get_webdriver_request_headers(self) File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/seleniumrequests/request.py", line 83, in get_webdriver_request_headers del headers_['host'] TypeError: 'NoneType' object does not support item deletion
in the source code:
if webdriver.current_window_handle != original_window_handle:
webdriver.switch_to.window(original_window_handle)
global headers
headers_ = headers
headers = None
# Remove the host header, which will simply contain the localhost address
# of the HTTPRequestHandler instance
del headers_['host']
return headers_
why headers = None and global headers ,but last del headers_
I want to know how to set the required parameter like "chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--headless')"
Enviroment:
Windows 11
Chrome 107.0.5304.88
Python 3.9
I tried execute this code:
def print_hi(name):
options = Options()
options.headless = False
webdriver = Chrome(ChromeDriverManager().install(), options=options)
webdriver.get("https://www.python.org")
print(webdriver.title)
response = webdriver.request('GET', "python.org")
print(response)
if __name__ == '__main__':
print_hi('PyCharm')
I got error:
selenium.common.exceptions.NoSuchWindowException: Message: no such window: No target with given id found
(Session info: chrome=107.0.5304.88)
Stacktrace:
Backtrace:
Ordinal0 [0x0087ACD3+2075859]
Ordinal0 [0x0080EE61+1633889]
Ordinal0 [0x0070B7BD+571325]
Ordinal0 [0x006FE9CF+518607]
Ordinal0 [0x006FD2A3+512675]
Ordinal0 [0x006FD6D8+513752]
Ordinal0 [0x006FD668+513640]
Ordinal0 [0x006F289F+469151]
Ordinal0 [0x006F1FB0+466864]
Ordinal0 [0x006F23CC+467916]
Ordinal0 [0x007613D9+922585]
Ordinal0 [0x007572E3+881379]
Ordinal0 [0x0072FEFD+720637]
Ordinal0 [0x00730F3F+724799]
GetHandleVerifier [0x00B2EED2+2769538]
GetHandleVerifier [0x00B20D95+2711877]
GetHandleVerifier [0x0090A03A+521194]
GetHandleVerifier [0x00908DA0+516432]
Ordinal0 [0x0081682C+1665068]
Ordinal0 [0x0081B128+1683752]
Ordinal0 [0x0081B215+1683989]
Ordinal0 [0x00826484+1729668]
BaseThreadInitThunk [0x76DEFA29+25]
RtlGetAppContainerNamedObjectPath [0x77147BBE+286]
RtlGetAppContainerNamedObjectPath [0x77147B8E+238]
Process finished with exit code 1
I also tried headless mode and got other error
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(10054, 'An existing connection was forcibly closed by the remote host', None, 10054, None))
When (get_webdriver_request_headers webdriver.execute_script("window.open('http://127.0.0.1:%d/');" % port) ) is executed。it may be blocked.
about cookie
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.