Forums

How to make Selenium work ?

Hi,

Can we use Selenium in Pythonanywhere ? I discovered that selenium is pre-installed.

I executed the following list of commands in python and I get an error:-

>>> from selenium import webdriver
>>> from selenium.common.exceptions import NoSuchElementException
>>> from selenium.webdriver.common.keys import Keys                                                                                                                    
>>> import time
>>> 
>>> browser = webdriver.Firefox()                                                                                                                                      
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 61, in __init__
    self.binary, timeout),
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__
    self.binary.launch_browser(self.profile)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 51, in launch_browser
    self._wait_until_connectable()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 90, in _wait_until_connectable
    self._get_firefox_output())
selenium.common.exceptions.WebDriverException: Message: "The browser appears to have exited before we could connect.     The output was: \n(process:23691): GLib-CRITICAL * 
*: g_slice_set_config: assertion `sys_page_size == 0' failed\nError: no display specified\n"

Kindly let me know how to make selenium work in Pythonanywhere ?

Thanks, San

PythonAnywhere does not have a default GUI screen for the Selenium browser to run in. You need to create one first. Add this to your code before you create your browser.

from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()

Sometimes the browser creation fails for no good reason (at least we haven't been able to work out why) and you just need to try again. We use code like this to ensure that we get a working browser:

for retry in range(3):
    try:
        browser = webdriver.Firefox()
        break
    except:
        time.sleep(3)

Also, be sure to clean up after you're done with:

browser.quit()
display.stop()

Thanks Glenn !

Really really do make sure you use the browser.quit() and display.stop(). I would put them in a finally block at the end of your tests. It's quite easy to break your account by leaving too many processes lying around.

In case anyone else comes across this old thread -- the Ubuntu xvfb package now comes with a handy command-line utility which will let you run any command with a virtual display.

So if you have:

python my_selenium_tests.py

rather than needing to use pyvirtualdisplay in your python code, you can just run:

xvfb-run python my_selenium_tests.py
Hello Guys -

I am trying to run Selenium on a FireFox browser and I have followed the instructions here. Everytime I have:

from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()

I get an import error pyvirtualdisplay model does not exist --- I tried running xvfb-run python my_selenium_tests.py from the command line and that didn't work either.

Any suggestions`

HI

May I replace

xvfb-run python my_selenium_tests.py

with

xvfb-run python34 my_selenium_tests.py

to run my script in python 3.4?

ES

yes. But it's python3.4, not python34.

Then how can driver.quit() and display.stop() when the app is dead suddenly without calling the method having code to quit()and stop () ?

It's one of my problem I have, because living driver and living is consuming my account cpu time

strong text

bonjour je suis en python 1999

I have several old versions of python .

It is just a question of time to solve your problem , make time more precise in your program.

import date or import datetime or import aclock

use pselect from C and make an extension with it!

SELECT(2) Linux Programmer's Manual SELECT(2)

NAME select, pselect, FD_CLR, FD_ISSET, FD_SET, FD_ZERO - synchronous I/O multiplexing

SYNOPSIS / According to POSIX.1-2001 / #include <sys/select.h>

   /* According to earlier standards */
   #include <sys/time.h>
   #include <sys/types.h>
   #include <unistd.h>

   int select(int nfds, fd_set *readfds, fd_set *writefds,

https://www.youtube.com/watch?v=dBmMKXFmILk

what is an fd_set ???????????????????????????????????????????????????????,

fd : file descriptor

               0 -> in
               1 -> out
               2 -> errors

python is written in C.

java is written in C also!

Peut -on recompiler son user domain sur ce site ????

@LeeKyungMoon if you want that fine grained control, you will have to use pyvirtualdisplay instead of xvfb-run. Also, you can go to the processes section under the consoles page to get all your processes, and kill the xvfb ones.

Everytime I try to connect from Selenium a get this error. I mean it is a web page error like. But i cant find any page. It loads and the webdriver appears to work but i cant find any element because this appears instead.

<!DOCTYPE html [ <!ENTITY % htmlDTD PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "DTD/xhtml1-strict.dtd"> %htmlDTD; <!ENTITY % netErrorDTD SYSTEM "chrome://global/locale/netError.dtd"> %netErrorDTD;

<!ENTITY loadError.label "Problem loading page"> <!ENTITY retry.label "Try Again">

<!-- Specific error messages -->

<!ENTITY connectionFailure.title "Unable to connect"> <!ENTITY connectionFailure.longDesc "&sharedLongDesc;">

<!ENTITY deniedPortAccess.title "This address is restricted"> <!ENTITY deniedPortAccess.longDesc "">

<!ENTITY dnsNotFound.title "Server not found"> <!ENTITY dnsNotFound.longDesc " <ul> <li>Check the address for typing errors such as <strong>ww</strong>.example.com instead of <strong>www</strong>.example.com</li> <li>If you are unable to load any pages, check your computer's network connection.</li> <li>If your computer or network is protected by a firewall or proxy, make sure that &brandShortName; is permitted to access the Web.</li> </ul> ">

<!ENTITY fileNotFound.title "File not found"> <!ENTITY fileNotFound.longDesc " <ul> <li>Check the file name for capitalization or other typing errors.</li> <li>Check to see if the file was moved, renamed or deleted.</li> </ul> ">

<!ENTITY generic.title "Oops."> <!ENTITY generic.longDesc " <p>&brandShortName; can't load this page for some reason.</p> ">

<!ENTITY malformedURI.title "The address isn't valid"> <!ENTITY malformedURI.longDesc " <ul> <li>Web addresses are usually written like <strong>http://www.example.com/</strong></li> <li>Make sure that you're using forward slashes (i.e. <strong>/</strong>).</li> </ul> ">

<!ENTITY netInterrupt.title "The connection was interrupted"> <!ENTITY netInterrupt.longDesc "&sharedLongDesc;">

<!ENTITY notCached.title "Document Expired"> <!ENTITY notCached.longDesc "<p>The requested document is not available in &brandShortName;'s cache.</p><ul><li>As a security precaution, &brandShortName; does not automatically re-request sensitive documents.</li><li>Click Try Again to re-request the document from the website.</li></ul>">

<!ENTITY netOffline.title "Offline mode"> <!ENTITY netOffline.longDesc2 " <ul> <li>Press "Try Again" to switch to online mode and reload the page.</li> </ul> ">

<!ENTITY contentEncodingError.title "Content Encoding Error"> <!ENTITY contentEncodingError.longDesc " <ul> <li>Please contact the website owners to inform them of this problem.</li> </ul> ">

<!ENTITY unsafeContentType.title "Unsafe File Type"> <!ENTITY unsafeContentType.longDesc " <ul> <li>Please contact the website owners to inform them of this problem.</li> </ul> ">

<!ENTITY netReset.title "The connection was reset"> <!ENTITY netReset.longDesc "&sharedLongDesc;">

<!ENTITY netTimeout.title "The connection has timed out"> <!ENTITY netTimeout.longDesc "&sharedLongDesc;">

<!ENTITY protocolNotFound.title "The address wasn't understood"> <!ENTITY protocolNotFound.longDesc " <ul> <li>You might need to install other software to open this address.</li> </ul> ">

<!ENTITY proxyConnectFailure.title "The proxy server is refusing connections"> <!ENTITY proxyConnectFailure.longDesc ">

<ul> <li>Check the proxy settings to make sure that they are correct.</li> <li>Contact your network administrator to make sure the proxy server is working.</li> </ul> ">

<!ENTITY proxyResolveFailure.title "Unable to find the proxy server"> <!ENTITY proxyResolveFailure.longDesc " <ul> <li>Check the proxy settings to make sure that they are correct.</li> <li>Check to make sure your computer has a working network connection.</li> <li>If your computer or network is protected by a firewall or proxy, make sure that &brandShortName; is permitted to access the Web.</li> </ul> ">

<!ENTITY redirectLoop.title "The page isn't redirecting properly"> <!ENTITY redirectLoop.longDesc " <ul> <li>This problem can sometimes be caused by disabling or refusing to accept cookies.</li> </ul> ">

<!ENTITY unknownSocketType.title "Unexpected response from server"> <!ENTITY unknownSocketType.longDesc " <ul> <li>Check to make sure your system has the Personal Security Manager installed.</li> <li>This might be due to a non-standard configuration on the server.</li> </ul> ">

<!ENTITY nssFailure2.title "Secure Connection Failed"> <!ENTITY nssFailure2.longDesc " <ul> <li>The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.</li> <li>Please contact the website owners to inform them of this problem. Alternatively, use the command found in the help menu to report this broken site.</li> </ul> ">

<!ENTITY nssBadCert.title "Secure Connection Failed"> <!ENTITY nssBadCert.longDesc2 " <ul> <li>This could be a problem with the server's configuration, or it could be someone trying to impersonate the server.</li> <li>If you have connected to this server successfully in the past, the error may be temporary, and you can try again later.</li> </ul> ">

<!ENTITY sharedLongDesc " <ul> <li>The site could be temporarily unavailable or too busy. Try again in a few moments.</li> <li>If you are unable to load any pages, check your computer's network connection.</li> <li>If your computer or network is protected by a firewall or proxy, make sure that &brandShortName; is permitted to access the Web.</li> </ul> ">

<!ENTITY malwareBlocked.title "Suspected Attack Site!"> <!ENTITY malwareBlocked.longDesc " <p>Attack sites try to install programs that steal private information, use your computer to attack others, or damage your system.</p> <p>Website owners who believe their site has been reported as an attack site in error may <a href='http://www.stopbadware.org/home/reviewinfo' >request a review</a>.</p> ">

<!ENTITY phishingBlocked.title "Suspected Web Forgery!"> <!ENTITY phishingBlocked.longDesc " <p>Entering any personal information on this page may result in identity theft or other fraud.</p> <p>These types of web forgeries are used in scams known as phishing attacks, in which fraudulent web pages and emails are used to imitate sources you may trust.</p> ">

<!ENTITY cspFrameAncestorBlocked.title "Blocked by Content Security Policy"> <!ENTITY cspFrameAncestorBlocked.longDesc "<p>&brandShortName; prevented this page from loading in this way because the page has a content security policy that disallows it.</p>">

<!ENTITY corruptedContentError.title "Corrupted Content Error"> <!ENTITY corruptedContentError.longDesc "<p>The page you are trying to view cannot be shown because an error in the data transmission was detected.</p><ul><li>Please contact the website owners to inform them of this problem.</li></ul>">

<!ENTITY securityOverride.linkText "Or you can add an exception…"> <!ENTITY securityOverride.getMeOutOfHereButton "Get me out of here!"> <!ENTITY securityOverride.exceptionButtonLabel "Add Exception…">

<!-- LOCALIZATION NOTE (securityOverride.warningContent) - Do not translate the contents of the <button> tags. It uses strings already defined above. The button is included here (instead of netError.xhtml) because it exposes functionality specific to firefox. -->

<!ENTITY securityOverride.warningContent " <p>You should not add an exception if you are using an internet connection that you do not trust completely or if you are not used to seeing a warning for this server.</p>

<button id='getMeOutOfHereButton'>&securityOverride.getMeOutOfHereButton;</button> <button id='exceptionDialogButton'>&securityOverride.exceptionButtonLabel;</button> ">

<!ENTITY remoteXUL.title "Remote XUL"> <!ENTITY remoteXUL.longDesc "<p><ul><li>Please contact the website owners to inform them of this problem.</li></ul></p>">

<!ENTITY % globalDTD SYSTEM "chrome://global/locale/global.dtd"> %globalDTD; ]>

It says "Web addresses are usually written like <strong>http://www.example.com/</strong></li> <li>Make sure that you're using forward slashes". Maybe you're not giving selenium the correct url?

It seems like there may be some dns lookup problem on the us-east coast amzn servers. We are investigating now.

I already solved it. It was a restricted connexion problem. thnks

I tried your following code in python3.4:

from selenium import webdriver

from pyvirtualdisplay import Display

import time

display = Display(visible=0, size=(800, 600))

display.start()

for retry in range(3):

try:
    browser = webdriver.Firefox()
    break
except:
    time.sleep(3)

browser.get('http://www.google.com')

print(browser.title)

browser.quit()

display.stop()

.....and I got:

NameError: name 'browser' is not defined

That code was part of an example - it's not something you can just copy and paste and have it work. In your case, what is happening is that the try..except around the browser creation is eating all the exceptions, so you have no way of knowing whether the browser variable has been assigned. Try printing out the exception you're getting to see what is going on.

Another usual exception when trying to run selenium:

selenium.common.exceptions.WebDriverException: Message: 'geckodriver' executable needs to be in PATH.

That's probably because you're using a newer version of selenium that won't work on PythonAnywhere. We know that 2.53.6 works. So I would suggest using that. Also I notice that your code does not properly clean up the browser and the display. See this help page for the right way to do it.

@glenn are you saying that the selenium version you have preinstalled (3.0.2) is not old enough?

It does appear to be that way. That is an error in our image. There isn't much we can do about it in the short term, though.

I cannot get my tests with selenium to work.

I have selenium version 2.53.6 and PyVirtualDisplay version 0.2.1 installed in my virtualenv. I am setting up my tests like this... (edited for brevity and science)

from pyvirtualdisplay import Display
from selenium import webdriver
import unittest

with Display():

    class TestyTestTestathon(unittest.TestCase):

        def setUp(self):
            self.browser = webdriver.Firefox()

        def tearDown(self):
            self.browser.quit()

        def test_testy_testaroony_wagga_wagga_nub_nub(self):
            # testy stuff asserts and suchlike gubbins....

I get:

"The browser appears to have exited "
selenium.common.exceptions.WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.

Any help appreciated. Thanks :)

An update. If I run my test with :

 xvfb-run -a python manage.py test pa_functional_tests

From the test: :::python self.browser.get('http://127.0.0.1:8000/accounts/login/') self.assertIn('Sign In', self.browser.title)

I get:

    self.assertIn('Sign In', self.browser.title)
    AssertionError: 'Sign In' not found in '400 The plain HTTP request was sent to HTTPS port'

That with Display() is not doign what you think it is. Essentially, it is opening the display for the time it takes to define your class. By the time the code inside your class is executed, the display is gone, so firefox cannot start.

In your second case, the display is provided by running xvfb, but then you're sending a request that is probably not going to where you think it is. Port 8000 on a console server is not a site that you can make requests to.

Of course, thanks - stupid mistake by me. 8) How on earth do I connect to a django test version of my site served with the temporary database from a console then?

If I run my test with :

 xvfb-run -a python manage.py test pa_functional_tests

From the test:

self.browser.get('https://127.0.0.1/accounts/login/')
self.assertIn('Sign In', self.browser.title)

It appears to setup the server - it tells me...

Creating test database for alias 'default'...
System check identified no issues (0 silenced).

But I get:

    self.assertIn('Sign In', self.browser.title)
    AssertionError: 'Sign In' not found in '404: Not Found'

You cannot run a Django web app in a PythonAnywhere console. That's why we have web apps. Run your Django site in a web app and connect to it through it's domain name.

Thanks. That solves it. Much appreciated. :)

Sorry I have another difficulty relating to this. My tests work fine if I disable the password security on the web app with the url:

self.browser.get('https://nverse.pythonanywhere.com/accounts/login/')

but I do not want to expose my testing app to the internet even for a short time. I tried using the url:

self.browser.get('https://user:password@nverse.pythonanywhere.com/accounts/login/')

With password enabled in the app, but it does not work. It appears to hang and when I kill it I get:

Traceback (most recent call last):
  File "/home/nverse/nvtest3/nvlive1/pa_functional_tests/tests.py", line 24, in test_login_account
self.browser.get('https://username:password@nverse.pythonanywhere.com/accounts/login/')
  File "/home/nverse/.virtualenvs/nvtest/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 248, in get
self.execute(Command.GET, {'url': url})
  File "/home/nverse/.virtualenvs/nvtest/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 234, in execute
response = self.command_executor.execute(driver_command, params)
  File "/home/nverse/.virtualenvs/nvtest/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 401, in execute
return self._request(command_info[0], url, body=data)
  File "/home/nverse/.virtualenvs/nvtest/lib/python3.6/site-packages/selenium/webdriver/remote/remote_connection.py", line 433, in _request
resp = self._conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse
response.begin()
  File "/usr/lib/python3.6/http/client.py", line 297, in begin
version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 266, in _read_status
raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

What would be the best way to run a functional test on my web app with selenium and the web app password security enabled? Thanks for your time.

Edit*

I also tried using:

    self.browser.get('https://nverse.pythonanywhere.com/accounts/login/')
    alert = self.browser.switch_to_alert()
    alert.send_keys('username')
    alert.send_keys(Keys.TAB)
    alert.send_keys('password')
    alert.send_keys(Keys.ENTER)

But this also does not work. I am pretty lost here. Thanks.

It appears that selenium does not support http basic auth (the method that the password feature uses) very well. I found a bunch of attempts online, but none of them seem like they'd work very well.

Thanks for trying. I'll post here if I find a solution. :)

*edit - For now I have put Login Required mixins and decorators in all my views and disabled password security on the web app until I find a fix. Then I put a login script in the def setUp(self): part of my unittest.TestCase Class so that from then on that selenium test has access. It is a workaround that lets me run functional tests and is probably also good security at this stage anyway.

OK -- glad you have a solution!

I am facing the same issue. In my case I cannot disable the authentication, is there another way?

I don't think so. The problem is that Selenium can't access a site that uses HTTP basic authentication.

url = 'https://solarmoviex.to/watch/pickings.7w5k8/pz1l59' display = Display(visible=0, size=(800, 600)) display.start() for retry in range(3): try: browser = webdriver.Firefox() break except: time.sleep(3) browser.get(url) html_soup = BeautifulSoup(browser.page_source, "html.parser") browser.quit() display.stop()

This is not working, does anyone have any ideas as to what the issue might be? html_soup remains empty. Thanks in advance.

I actually get the following error:

raise RemoteDisconnected("Remote end closed connection without"#012http.client.RemoteDisconnected: Remote end closed connection without response

I tried the code in shell and it works perfectly fine. How can I replicate the shell code in my views.py?

It looks like you're leaving browser and virtualdisplay processes lying around because you're not closing them correctly. I have killed them for you.

There are some caveats and advice about using selenium on PythonAnywhere here: http://help.pythonanywhere.com/pages/selenium/