Selenium testing with a Django application - locally and in CI

Updated - check out the end of this entry to see how I solved chrome driver instability!

It's great to have unit tests in any piece of software, but for web applications they aren't enough to ensure funcitonality. To fully ensure things work, you need to actually open each page in a web browser and use it, which is of course something that's not even remotely doable by hand. Selenium allows you to automate the process of using your web application in a real browser, and it has a fantastic Python library that's easy to integrate into a Django project.

With a good suite of Selenium tests, you could have full confidence that your application fully works without manually using any part of it yourself. Join me as I discuss adding Selenium tests to a Django project that are ran both locally and in a CI environment on a headless server.

→ Setup

Setting up what I needed so that I could actually write and run tests that used Selenium involved a few steps:

1. I wanted to run tests using both Chrome and Firefox drivers, so both the "chromedriver" and "geckodriver" command-line tools are needed. Your OS package manager may have one or more of these available, so do check.
2. Install the selenium pip package (pip3 install --user selenium, or however you prefer to do that.)
3. I already had working installs of Chromium and Firefox, but those are both needed in case they aren't already set up

→ Get busy testing

From here, you can start writing tests - I recommend looking into the StaticLiveServerTestCase class (your tests will ultimately subclass this.) I wanted a setup where each module that has tests would simply define a class of tests, and that class could subclass some more generic ones that did the boilerplate stuff for Selenium. This has the added benefit of not instantiating the browser/driver for each module (if you have more than one); instead each browser is instantiated once and all tests are ran at once. I set up a base class with each test I wanted to run, and then subclassed that in two other classes where I selected the web driver for each browser:

class BaseSelenium:
    """
    This class implements some boilerplate/helpful methods for writing tests
    that use Selenium.  Handles settings up and tearing down as required by the
    varisou Selenium drivers (but specifically Firefox and Chrome.)
    """
    @classmethod
    def setUpClass(cls):
        super().setUpClass()
        # General setup and/or common things can go here

    @classmethod
    def tearDownClass(cls):
        cls.driver.quit()
        super().tearDownClass()

    # Any common helper methods can go here...

And then in each module you can create a class for module-specific tests:

class ModuleSeleniumTests:
    def setUp(self):
        super().setUp()
        self.f = RequestFactory()

    # Tests go here...

Finally, in the project-level tests the action happens:

... imports ...
from selenium.webdriver.chrome.webdriver import WebDriver as ChromeDriver
from selenium.webdriver.firefox.webdriver import WebDriver as FFDriver
... other imports ...

class ProjectFirefoxTests(BaseSelenium, ModuleSeleniumTests, OtherModuleSeleniumTests, MoreModuleSeleniumTests, StaticLiveServerTestCase):
    @classmethod
    def setUpClass(cls):
        super().setUpClass()
        cls.driver = FFDriver()

class ProjectChromeTests(BaseSelenium, ModuleSeleniumTests, OtherModuleSeleniumTests, MoreModuleSeleniumTests, StaticLiveServerTestCase):
    @classmethod
    def setUpClass(cls):
        super().setUpClass()
        cls.driver = ChromeDriver()

Now, executing your top-level project's tests will also fire up Selenium tests with Chrome and Firefox.

→ Okay, cool! But...

As I got to this point, I thought OK this is nice but it isn't worth too much if I can't run it in CI along with my other tests. I don't think I've written much about it before, but I use Buildbot for my continuous integration needs. I like it a lot more than other common software that fits the same function for a lot of reasons, maybe I'll go into those sometime. Anyways I wanted to be sure I could run this via builds on my Buildbot instance without too much crazyness, so I got to work.

→ Running selenium tests via Buildbot

As it turns out, not a whole lot is required to get things going! I already have a working Buildbot configuration that's driven by Ansible, I just needed to add or change a few things. Specifically:

• Added ansible tasks to handle installing the packages that were required from the package manager, as well as one to download (and another to verify - I like doing this in two steps) the geckodriver executable.
• Created a runit service for Xvfb, which provides the "headless" X server I need for opening web browsers to run the tests.
• Added sudoers rules that allow my build user to enable, disable, start, stop, and issue other management commands related to the xvfb service.
• With the above points handled, I then added build steps for enabling and starting the xvfb service before the tests run, and then another to stop and disable it when that step completes.

The end result of all this is that my build user can start and stop the xvfb service with sudo but no password, and it can use this service to run X11 applications like web browsers. I did need to add a wrapper for starting the xvfb service because it could either already be enabled or take a few seconds to be ready. This wrapper exited nicely if the service was already enabled, and ran sudo sv check xvfb in a while loop until it exits ok or hits a specific number of checks (in which case it exits nonzero.) I suppose if my build machine was really fast this might not be needed, but even on my cheapo build VM it usually only needs about three seconds.

→ [build successful]

So there you have it! Zero to selenium with just a bit of effort, and it was simple and intuitive to write reusable bits that make writing these tests just a bit less of a chore. I can't say the setup has been without issues - it was for a while then something happened and now in CI my chrome driver will (seemingly) randomly not connect to the browser. I've done various things to try and track this down with no success, but aside from that point my experiment with Selenium has been a success and I now use it on all Django projects.

→ Update! Chromedriver stability...

As mentioned above, I had been seeing an issue where the chrome driver would fail to initialize and cause tests to fail with a timeout. After a bit of research, I discovered this is actually a known issue that can be worked around in a relatively simple albeit sort of hideious way. To that end, I created a small wrapper for starting up the chrome driver:

def start_driver_wrapper(cls, DriverClass):
    while not hasattr(cls, "driver"):
        try:
            cls.driver = DriverClass()
        except ConnectionResetError:
            pass

The driver init code above would then be modified like so:

class ProjectChromeTests(BaseSelenium, ModuleSeleniumTests, OtherModuleSeleniumTests, MoreModuleSeleniumTests, StaticLiveServerTestCase):
    @classmethod
    def setUpClass(cls):
        super().setUpClass()
        start_driver_wrapper(cls, ChromeDriver)

With the above snippet in place, chrome driver will attempt to connect until it does. In practice I've not seen it fail more than twice in a row very often, so we're not adding a whole lot of extra time to the build.

In the event of a legitimate timeout situation (we are thrown ConnectionResetError for another, presumably serious reason) the above snippet would end up in an infinite loop -- a situation that would be rectified in CI/Buildbot with timeouts there. Generally speaking, I've never had issues like this but it's good to be aware at least.

Source link for this page