Investigate Selenium 3 compatibility about tor-browser-selenium

[gunesacar]

Fixed in #88 (comment)

[gunesacar]

@fowlslegs, in case it helps with your internal scheduling, we need to wait until the release of the ESR 52 based Tor Browsers to land Selenium 3 support.

Tor Browser 6.X versions are based on ESR 45, which is not supported by geckodriver.

The expected date for ESR52 based Tor Browser is June 13: https://lists.torproject.org/pipermail/tbb-dev/2017-February/000472.html.

[mjuarezm]

For the record, I merged the tbgeckodriver branch onto tor-browser-crawler and tested it with the latest TBB (7.0.2). I am able to run the crawler successfully in our setting (haven't tested edge cases). However, it doesn't work for newer versions of geckodriver or/and Selenium.

[gunesacar]

Here's a WIP branch: https://github.com/gunesacar/tor-browser-selenium/tree/tbgeckodriver

• Almost everything works fine for TBB versions 6+ with Selenium 3.0.0b2.
• There's a minor issue is driver.set_context, which is noted here:
  gunesacar@b105ce2#diff-7a7914417f2bac1e28f9c83336bce811R39
• 5.5.5, will not work with Selenium 3. We may need to drop support.
• CI tests pass for version 6 TBBs, python 3.4 and 3.5 errors are due to Selenium's own bug, which should be fixed in 3.0.0b3.

[gunesacar]

Import errors in Python 3.4 and 3.5 fixed with the release of Selenium 3.0.0.b3.
https://travis-ci.org/gunesacar/tor-browser-selenium/builds/160031352

[psivesely]

👍 for landing Selenium 3 support.

[gunesacar]

Thanks @mjuarezm !
I guess the next step would be to check which tests fail with that branch and address them.

[gunesacar]

Here's the PR for the selenium 3 compatibility: #88

[tadatitam]

Getting this error message on torbrowser-linux64-7.0.8. Tested this with selenium 3.6, tbselenium 0.3.2 on Ubuntu 16.04 (Xenial64).

    self.driver = TorBrowserDriver(os.environ['TBB_PATH'])
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/tbselenium/tbdriver.py", line 57, in __init__
    log_path=tbb_logfile_path)
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 144, in __init__
    self.service.start()
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/common/service.py", line 81, in start
    os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'geckodriver' executable needs to be in PATH. 

The same code was working perfectly for torbrowser-linux64-7.0.6.

[gunesacar]

@tadatitam you need to have geckodriver executable (version 0.17.0) in your system's path starting from 0.3.2. It's required by Firefox actually.

You can download it from the following address:
https://github.com/mozilla/geckodriver/releases/download/v0.17.0/geckodriver-v0.17.0-linux64.tar.gz"

You can also check the relevant MDN documents:
https://developer.mozilla.org/en-US/docs/Mozilla/QA/Marionette/WebDriver#Setting_up_the_geckodriver_executable

I will open an issue to document that requirement. Sorry about that.

[tadatitam]

@gunesacar Ah, I see. I was getting away on Firefox by specifying executable_path='path/to/geckodriver'.

Thanks to your suggestion, I was able to get Tor to launch by adding the location of geckodriver to PATH: i.e. I can see the window showing the Tor circuit setup, and then a Firefox window opens up. However, Tor fails to open any page, and throws this new error:

    self.driver = TorBrowserDriver(os.environ['TBB_PATH'])
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/tbselenium/tbdriver.py", line 57, in __init__
    log_path=tbb_logfile_path)
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 154, in __init__
    keep_alive=True)
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 151, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 240, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 308, in execute
    self.error_handler.check_response(response)
  File "/home/ubunutu16/.local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: connection refused

I'm going to try and figure out why this is happening and get back with my findings. Thanks for helping out. :)

[gunesacar]

This is a generic error when the geckodriver doesn't respond for one reason or other. To debug this interface you may listen to loopback interface with Wireshark. But, this is not guaranteed to give you anything.

I'd check the geckodriver version, you need to have version 0.17.0.

If you are using custom parameters for the initialization, feel free to run your code by me.

I just checked to make sure, the below code seems to work with tbselenium 0.3.2:
https://gist.github.com/gunesacar/1b48b71d8619885a0dcb95bb59da592c

[tadatitam]

Thank you thank you. There was a mismatch in the geckodriver version. I was using 0.19.0. :)