Test Automation with Python

Python code just like in any other language requires testing. Unittest is a python framework dedicated for it. It has origins in Junit in terms of code structure and behavior. In this article I will try to illuminate a little bit the topic of testing in Python and provide some good practices.

Structure

Before we start let’s discuss a little bit the correct (or rather most preferable) project structure including tests. In order to have clean and effective code, master test folder shall be at the same level as master code folder, like:

sample_
|   ??? core.py
project:
??? jlabs
|   ??? __init__.py
|   ??? functions.py
??? tests
|   ??? __init__.py
|   ??? test_core.py
|   ??? test_functions.py
??? README.rst
??? setup.py

Of course, test folder is preferable when our test suite is bigger than one file – otherwise simple test.py is sufficient. From the other hand keeping 2-digit number of tests in one file is not a good idea too so I suggest creating test folder and dividing test logically in separate files.

First test

As mentioned at the beginning of this article unittest is a build-in python library so we do not have to take any actions before we start creating any test. Every test class in unittest has to inherit from unittest.TestCase class. Every method in this class with a name starting with ‘test’ will be treated as method representing test – this is a naming convention informing runner what to execute. Let’s write our first test with basic assertion:

import unittest


class TestFunctions(unittest.TestCase):
    def test_basic_01(self):
        self.assertEqual(20 + 20, 40)
        self.assertEqual(20 + 20, 41, '20+20 is not 41')

In second assertion I added a custom message that will appear in case of failure (in my test always).

Ran 1 test in 0.003s

FAILED (failures=1)

20+20 is not 41
41 != 40

Expected :40 
Actual   :41

Those are all the basics required to create a test for your code. Of course, it is a trivial example – real scenarios would actually test some code which need to be imported into test class and its functionality will be checked:

import unittest
from jlabs.core import Person


class TestCore(unittest.TestCase):
    def test_core_01(self):
        me = Person('me')

        self.assertTrue(me.is_handsome, 'I am not handsome')

How to run

Now we know how to write a code for testing and in this chapter we will get to know how to execute it. Of course, there is more than one way to do it. First one is running it from command line with:

python -m unittest discover

It will run all the tests files that are modules – in our case TestFunctions and TestCore and will produce execution report:

FF
======================================================================
FAIL: test_basic_01 (tests.test_basic.TestBasic)
----------------------------------------------------------------------
Traceback (most recent call last ):
  File "C:\dev\sample_project\tests\test_functions.py", line 7, in test_basic_01
    self.assertEqual(20 + 20, 41, '20+20 is not 41')
AssertionError: 20+20 is not 41

======================================================================
FAIL: test_core_01 (tests.test_core.TestCore)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\dev\sample_project\tests\test_core.py", line 9, in test_core_01
    self.assertTrue(me.is_handsome, 'I am not handsome')
AssertionError: I am not handsome

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=2)

We can narrow down the tests we want to execute by adding pattern to the command:

python -m unittest discover -p *_core.py

What will execute tests from modules matching given pattern. We can also add start directory option (what make sense when you have subfolders in tests directory):

python -m unittest discover –s tests/subfolder

Another way to launch tests with a better level of control is to create a custom script (run_tests.py) to do it where we can specify exactly what shall be executed:

from unittest import TestLoader, TestSuite, TextTestRunner
from tests.test_functions import TestBasic

modules_to_run = [TestBasic]

suites = list(TestLoader().loadTestsFromTestCase(t) for t in modules_to_run)
TextTestRunner().run(TestSuite(suites))

than we can execute it simply by:

python run_tests.py

Generated report will be similar. The biggest advantage of the second option besides full control on tests to be run is possibility to create custom test runner with different output. It is very convenient when we need report in e.g. HTML format or other fitting some continuous integration tool (like TeamCity).

Assertions

We know how to write tests and how to execute them. Next step will be to get to know what can we verify within the test and how. To check specific functionality we need assertions – an expression surrounding some testable logic what basically simplifies to checking if output is true. Of course, there is possibility to use only one type of assertion in our test code but it is not elegant and recommended solution. Unittest provides us many different types of assertions:

• comparable – group of assertions that can evaluate 2 values relative to each other (are they equal, is one greater than another, and so on)

def test_basic_02(self):
    self.assertEqual(20 + 20, 40)
    self.assertGreater(20 + 20, 39)
    self.assertLessEqual(20 + 20, 40)
    self.assertTrue(20 + 20 == 40)
    self.assertIsNotNone(20)

• iterable – assertions operating on sequences

def test_basic_03(self):
    self.assertEqual([1, 2, 3], [1, 2, 3])
    self.assertIn(2, [1, 2, 3])
    self.assertDictEqual({1: 'a', 2: 'b'}, {2: 'b', 1: 'a'})
    self.assertDictContainsSubset({1: 'a'}, {1: 'a', 2: 'b'})

• exceptions – assertions dealing with exceptions

def test_basic_04(self):
    with self.assertRaises(Exception):
        raise Exception('ERROR')

    with self.assertRaisesRegexp(Exception, 'ERR.+'):
        raise Exception('ERROR')

• instantional – comparing objects and checking if they are instances of class

def test_basic_05(self):
    self.assertIsInstance('str', str)
    self.assertIs(None, None)

• failure – failing test on purpose when specific situation happen

def test_basic_06(self):
    a = 5
    if a < 6:
        self.fail('a < 6')

Organizing your code

When we have everything covered in terms of functionality – all assertions are in place we can start thinking about our code transparency and effectiveness. It is a good practice not to write long tests but rather short ones covering little piece of logic, because every assertion failure causes tests being stopped and not executed further (so everything after failed assertion will be skipped). When we have such granularity, it is very probable we could have code duplications and to avoid that we can use pre- and post-processing blocks called setUp() and tearDown(). These methods define actions that will be taken before and after each test. If setup fails test will not be executed and will end with failure status. If setup succeed teardown will be executed no matter what the test’s status is. Also whole TestSuite (module with tests defined) can have its setUpClass() and tearDownClass() methods defined – these will be executed before first test within TestSuite and after last one and mast be annotated as @classmethod. Below there is a simple example of these methods usage:

import unittest
from jlabs.core import PersonDbFactory


class TestCore(unittest.TestCase):
    factory = None

    @classmethod
    def setUpClass(cls):
        cls.factory = PersonDbFactory()
        cls.factory.connect()

    def setUp(self):
        self.me = self.factory.get_person('me')

    def tearDown(self):
        self.me.destroy()

    def test_core_handsome(self):
        self.assertFalse(self.me.is_handsome, 'I am not handsome')

    def test_core_smart(self):
        self.assertTrue(self.me.is_smart, 'I am not smart')

Running this suite will give us output like:

Ran 2 tests in 0.002s

OK
Connecting to DB
Gathering person from DB
Destroying person object
Gathering person from DB
Destroying person object
Disconnecting from DB

This demonstrates setup() and tearDown() are executed before every test while setUpClass() and tearDownClass() are executed once.

There are also 3 useful annotations allowing us to control test execution and skipping tests.

• @skip – just skipping the test
• @skipIf – adding condition when this test shall not be executed
• @skipUnless – adding condition when this test shall not be skipped

Summary

We reached the end of the article. I hope after lecture one can successfully write and execute tests of its code. Please keep in mind that unittest is a very universal library and it is not dedicated for tests on unit level only. It provides the test template and what is the subject of test depends on us.