Web2py Automate Unittesting, Doctesting and UI testing (selenium)

A script on a web2py slice got me started on automated testing my Web2py apps. It did take some time to get all the concepts lined up in my head, but now I'm getting somewhere. The script looks for unittests in a special folder inside your app. Each file should contain a separate testcase as  a class based on unittest.TestCase and executes the test by adding each of the found classes to a testsuite. matclab enhanced the script to include Doctests. The script also runs all doctests in the controller files.


I first adapted the script testRunner.py to include UI testing using the Selenium webdriver module and made some more changes.
When testing you want to use a separate database to prevent polluting the production database. The original suggestion was to add a few lines to the model file.

# create a test database by copying the original db
test_db = DAL('sqlite://testing.sqlite')
for tablename in db.tables:  # Copy tables!
    table_copy = [copy(f) for f in db[tablename]]
    test_db.define_table(tablename, *table_copy)

db=test_db

I moved this code to the testrunner script to make the testing less intrusive for the application code. And I added code to use a fixture file which fill the database so that we have a fixed state to start with for each test.

Integrating the UI tests in the script proved to be a little hard. Running a script in web2py, in this case the testRunner script with the -M parameter makes the code of the web2py model files available to the script and indirectly to the unit and doctests. But running a UI test means simulating a browser-interaction, and that can only be another web2py instance.

Normally this new instance of our app will use the standard(production) database, but that not what we want. We need to signal, one way or another that this instance should use the testing db. I solved it by using an environment var WEB2PY_TESTING_DB. When set, the database definitions in the model file switches  to the testing db and fills it using the fixture file. The last step is not strictly necessary because testRunner.py already initialized the db, but we want to make sure the fixture is fresh.

    if os.getenv('WEB2PY_USE_DB_TESTING')
        db = DAL('sqlite://testing.sqlite')

    if os.getenv('WEB2PY_USE_DB_TESTING') and not

       os.getenv('WEB2PY_DB_TESTING_FILLED'):

        # use fixture file only once during lifetime of the app
        import yaml
        for table in db.tables:
        # Make sure to cascade, or this will fail
        # for tables that have FK references.
        db[table].truncate("CASCADE")
        data = yaml.load(open('applications/%s/private/testfixture.yaml'%current.request.application))
        for table_name, rows in data.items():
            for r in rows:
                db[table_name].insert(**r)
        os.environ['WEB2PY_DB_TESTING_FILLED']='TRUE'

Excerpt from the fixture file, see the pyyaml and yaml documentation for details.


activity:
 - name: 1. No workshops
   nr_prefs: 0
   nr_sessions: 0
   amount: 100
   w_start: !!timestamp '2021-01-02 10:00:00'
   w_end: !!timestamp '2021-01-02 17:00:00'
   w_register: !!timestamp '2021-01-02 10:00:00'
 - name: 2. No workshops, no prefs, only options
   nr_prefs: 0
   nr_sessions: 1
   amount: 200
   w_start: !!timestamp '2021-01-02 10:00:00'
   w_end: !!timestamp '2021-01-02 17:00:00'
   w_register: !!timestamp '2021-01-02 10:00:00'
workshop:
 - name: act1, ws1
   activity: 1
   number: 1
 - name: act1, ws2
   activity: 1
   number: 2

But wait, when we use sqlite, the database file is already in use in our testing environment, so we get a database locked error. One way of solving this is to close the database connection in our script after running the doc- and unittests like this.

...
unittest.TextTestRunner(verbosity=2).run(suite) # run the doctests and unittests
db._adapter.connection.close()
unittest.TextTestRunner(verbosity=2).run(ui_suite) # run the ui tests
...

Preparation

1. Put the script in the root of your web2py instance
2. Add the helper to the gluon\contrib folder
3. Inside your application folder, create a folder tests with subfolders controller and model
4. If you want to use UI testing  install selenium using pip install selenium, see http://ncdegroot.blogspot.com/2011/06/using-selenium-2-with-python-and-web2py.html and create an userinterface folder inside tests 

Unittest

An incomplete example. For an introduction see http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-1-unittest.html



""" Unittest for controller x
"""
import unittest
import cPickle as pickle
from gluon import * 
from gluon.contrib.test_helpers import form_postvars
from gluon import current
import components




class AanmeldingController(unittest.TestCase):
    """ test CRUD for contact, participant using the classes from module components"""
    def __init__(self, p):
        global auth, session, request
        unittest.TestCase.__init__(self, p)
        self.session = pickle.dumps(current.session)
        current.request.application = 'dibsa'
        current.request.controller = 'default'
        self.request = pickle.dumps(current.request)
        self.part1=Storage(firstname="Nico",
                           prefix="de",
                           lastname="Groot",
                           option1=2,
                           )
        self.partpref=Storage(firstname="Nico",
                           prefix="de",
                           lastname="Groot",
                           pref0=1,
                           )
        self.assertTrue(current.app.db._uri, 'sqlite://testing.sqlite')
        self.resetDB()


    def setUp(self):
        global response, session, request, auth
        current.session = pickle.loads(self.session)
        current.request = pickle.loads(self.request)
        #preconditions
        
    def resetDB(self):
        """"Start with a fixture, read from yaml file, first empty db table"""
        import yaml


        for table in db.tables:
            # Make sure to cascade, or this will fail 
            # for tables that have FK references.
            db[table].truncate("CASCADE")    
        data = yaml.load(open('applications/%s/private/testfixture.yaml'%current.request.application))


        for table_name, rows in data.items():
            for r in rows:
                db[table_name].insert(
                    **r
                )        
        




    def testContactAutopay(self):
        """ create and save contact, autopay
        """
        setCurrentActivityId(2)
        self._contact_autopay(True)


        
    def _contact_autopay(self, autopay=True):
        """ create and save contact, autopay, not a participant
            returns contactid
        """
        # Register a user in the db
        cp=components.newClient()
        contact=Storage({"firstname":"Nico",
                 "prefix":"de",
                 "lastname":"Groot",
                 "account":"123"})
        cp.form.vars=contact
        cp.manualsave(autopay=autopay) # save in db.person and in db.p2a
        # save ok?
        found=db((db.person.id==cp.id)).select(db.person.ALL)[0]
        self.assertTrue(cp.id==found.id) # id ok
        # all other fields
        for key in contact.keys():        
            #print key, contact[key]
            self.assertTrue(found[key]==contact[key]) # field ok
        found=db( (db.person2activity.person==cp.id)&
                  (db.person2activity.autopay==autopay)&
                  (db.person2activity.pclass==current.app.constants.CONTACT)&
                  (db.person2activity.activity==current.app.activity.id)
                ).select(db.person2activity.person)[0].person 
        self.assertTrue(cp.id==found.id)
        return cp.id
    
        

Doctests

You can add doctests to your controllers as described in the web2py book. You can already run your doctests using the admin interface or from the commandline. Nothing new here, except that a separate testing database is used with the fixture file. An example: 

>>> current.app.db._uri  # make sure it IS the testing db'sqlite://testing.sqlite'>>> current.app.activity.name'1. No workshops'>>> current.app.activity.amount100.0>>> # case 1: create contact, save to db, check its there and delete it (and check if it is deleted)>>> cp=components.newClient()>>> cp.form.vars={"firstname":"nico","account":"123"}>>> cp.manualsave(autopay=False)>>> cp.id==db((db.person2activity.person==cp.id)&(db.person2activity.autopay==False)&(db.person2activity.activity==current.app.activity.id)).select(db.person2activity.person)[0].personTrue

UI tests

Another incomplete example to get you started.

""" Uses Selenium2/webdriver to test the UI of DIBSA
    Version 0.9: 
    contain tests for aanmelding/registration pages
"""
import unittest, time, re
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver import ActionChains
from util import Browser




class Aanmelding(unittest.TestCase):
    def setUp(self):
        """test using Chrome"""
        self.verificationErrors = []
        self.browser = webdriver.Chrome()
        self.action_chains = ActionChains(self.browser)


    def _fill_name(self):
        self.browser.find_element_by_name("company").send_keys("TST")
        self.browser.find_element_by_name("firstname").send_keys("N.C.")
        self.browser.find_element_by_name("prefix").send_keys("de")
        self.browser.find_element_by_name("lastname").send_keys("Groot")
        el = self.browser.find_element_by_id("person_gender")
        el.send_keys("M")    
    def test_xxx(self):

        self._fill_name()
        el = self.browser.find_element_by_id("pref0")
        self.assertEqual(el.text,'option1option2',"prefs not ok: %s"%el.text)

Running the tests

The script is run on the command line as follows

python web2py.py -S dibsa -M -R testRunner.py

The parameter after -S specifies the application name, -M makes sure the model files are used, and -R specifies our script.

Documentation
http://www.web2pyslices.com/slices/take_slice/67
http://www.web2pyslices.com/slices/take_slice/142 (needs an update)
Virtualenv slides

Using Selenium 2 with Python and web2py

I've used Selenium some time ago to create automated tests for a few Web2py applications I've written. On FLOSS weekly I heard of the new version which no longer needs the separate Java server I used in Selenium RC. I've got no problems using the Java server, in my testsuite I test if the server is available and then start it with

selserver = subprocess.Popen(["java", '-jar', r"C:\selenium-remote-control-1.0.1\selenium-server-1.0.1\selenium-server.jar"])

 and then connect to it. Works fine, but less is more...

Installing Selenium in Python

I installed Selenium using pip install selenium. I'm using Python 2.7.1. There where two syntax errors while installing in webdriver\mouse.py.  and keyboard.py I guess this needs repairing at some point...

But first let's see if it works. In Python I can now

 from selenium import webdriver
 driver = webdriver.Chrome()

This gives me an exception, apparently I need an executable, to replace the former selenium server, but now a specialized browserspecific one. You can find it at http://code.google.com/p/selenium/downloads/list I downloaded chromedriver_win32_12.0.727.0.zip since I'm on Windows 7, unzipped it and placed the executable in a folder and made it reachable with my PATH by changing this environment variable.

Now my code - after starting a server in the background - opens a Chrome browser instance, which is good news. Let's see if we can control it by sending the browser to my app url at 127.0.0.1:8000/dibsa. This is the Web2py application I want to test. And let's say we want to login, and the menu item, the link to login is 'Login'

from selenium import webdrive
driver = webdriver.Chrome()
driver.get("http://127.0.0.1:8000/dibsa")
link=driver.find_element_by_partial_link_text("Login")
link.click()
driver.quit()

This code, in the opened browser now redirects to the login page. We can now continue to fill in the inputs, and then clicking the submit button. I could have used other methods like find_element_by_xpath() to select the link.
When ready you can close the browser with the close() method, but in that case the chromedriver binary stays in memory. If you use quit() it ends. If you first call close() and then quit() the selenium module crashes.

Conclusion
Selenium 2 seems to work fine, at least with Chrome. The server-executables are an improvement. No configuring needed, no fiddling with profiles and no java. Don't forget to end the chromedriver process by calling the quit() method.