Adding Test Code Coverage Analysis to a Python Project's Setup Command

I recently integrated unit test code coverage analysis (using coverage) as a setuptools command extension into the DendroPy phylogenetic computing library, and thought that I would share how this was done.

Providing the Command Extension

The first step is to provide the command functionality in a class that derives from “setuptools.Command” in a separate module of the package, which, in my case, was called “coverage_analysis.py”, located in the “test/support” subdirectory of the DendroPy package

#! /usr/bin/env python

import unittest
import shutil
import sys
from dendropy.test import get_test_suite
from dendropy.test.support import pathmap

class CoverageAnalysis(Command):
    """
    Code coverage analysis command.
    """

    description = "run test coverage analysis"
    user_options = [
        ('erase', None, "remove all existing coverage results"),
        ('branch', 'b', 'measure branch coverage in addition to statement coverage'),
        ('test-module=', 't', "explicitly specify a module to test (e.g. 'dendropy.test.test_containers')"),
        ('no-annotate', None, "do not create annotated source code files"),
        ('no-html', None, "do not create HTML report files"),
    ]

    def initialize_options(self):
        """
        Initialize options to default values.
        """
        self.test_module = None
        self.branch = False
        self.erase = False
        self.no_annotate = False
        self.no_html = False
        self.omit_prefixes = ['dendropy/test']

    def finalize_options(self):
        pass

    def run(self):
        """
        Main command implementation.
        """

        if self.erase:
            try:
                shutil.rmtree(pathmap.TESTS_COVERAGE_DIR)
            except:
                pass
        else:
            if self.test_module is None:
                test_suite = get_test_suite()
            else:
                test_suite = get_test_suite([self.test_module])
            runner = unittest.TextTestRunner()
            cov = coverage.coverage(branch=self.branch)
            cov.start()
            runner.run(test_suite)
            cov.stop()
            if not self.no_annotate:
                cov.annotate(omit_prefixes=self.omit_prefixes,
                        directory=pathmap.TESTS_COVERAGE_SOURCE_DIR)
            if not self.no_html:
                cov.html_report(omit_prefixes=self.omit_prefixes,
                        directory=pathmap.TESTS_COVERAGE_REPORT_DIR)
            cov.report(omit_prefixes=self.omit_prefixes)

Things to note with this:

  • The “description” will be printed as part of the “setuptools” help.
  • The “user_options” should be a list of tuples that define the options supported by the command extension. Each tuple consists of three elements, with the first element being the long command option (i.e., prefixed with a double dash when invoked), the second element being the short command option (i.e., prefixed with a single dash when invoked), and the third element is the help string. If the first element is suffixed with an equals sign, “=", then it takes a string value as an argument, otherwise it functions as a boolean switch/flag type option.
  • The “initialize_options” method should define an attribute for every option described in “user_options”, setting them to their default values. Note how hyphens in the option name are mapped to underscores in the attribute name. Also note that we define an additional attribute, “omit_prefixes”, which is a list of path prefixes that we want to exclude from the tests: in this case, I only want the main library code to be analyzed for coverage in tests, and thus exclude the test code.
  • The “run” method implements the actual command execution. If the “erase” option is set, then the results of previous tests are cleared. Otherwise, the tests are run with coverage analysis.
  • get_test_suite()” returns a unittest test suite to be run (consult the unittest documentation for more details), while “pathmap.TESTS_COVERAGE_SOURCE_DIR” is the custom output directory for the tests, defined elsewhere in the DendroPy code base.
  • As noted, the “omit_prefixes” argument is used to exclude parts of the code from coverage analysis, and here I exclude the test code, so as to focus just on the main library code.

Extending “setup” to Include the Command Extension

With the above code in place, we then need to map it to an entry point of the setuptoolssetup“ command, which is usually in the “setup.py” of the package root, using the keyword argument “entry_points“:

setup(name='DendroPy',
      # (etc)
      long_description=open('README.txt').read(),
      entry_points = ,
      # (etc)
      )

As you can see, the entry points are given by a string consisting of the command name, followed by an equals sign, followed by the fully-qualified module name providing the command, followed by a colon, followed by the name of the class that implements the command.

Installing and Running the Command

After this, you will need to run a setuptools to create an “egg-info” which includes the entry point for the command extension:

$ python setup.py install_egg_info

Alternatively, you can simply re-install the package:

$ python setup.py develop

Once this is done, you can invoke the test code coverage analysis by:

$ python setup.py coverage

Robustness for Production Code

If you clone the DendroPy Git repository, you will note that the actual code is quite a bit different from the examples given here. The actual code wraps up a lot of the above in “try..except“ blocks that catches unsuccessful imports of various dependencies, has conditionals that exclude the command extension of these dependencies are not available, and provides lots of message and logging feedback reporting on the status of the command availability:

#! /usr/bin/env python

###############################################################################
##  DendroPy Phylogenetic Computing Library.
##
##  Copyright 2009 Jeet Sukumaran and Mark T. Holder.
##
##  This program is free software; you can redistribute it and/or modify
##  it under the terms of the GNU General Public License as published by
##  the Free Software Foundation; either version 3 of the License, or
##  (at your option) any later version.
##
##  This program is distributed in the hope that it will be useful,
##  but WITHOUT ANY WARRANTY; without even the implied warranty of
##  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
##  GNU General Public License for more details.
##
##  You should have received a copy of the GNU General Public License along
##  with this program. If not, see .
##
###############################################################################

"""
Support for coverage analysis.
"""

import unittest
import shutil
import sys
from optparse import OptionParser
from dendropy.utility import messaging
_LOG = messaging.get_logger(__name__)

DENDROPY_COVERAGE_ANALYSIS_AVAILABLE = False
try:
    from setuptools import Command
except ImportError:
    _LOG.warn("setuptools.Command could not be imported: setuptools extensions not available")
else:
    try:
        import coverage
    except ImportError:
        _LOG.warn("coverage could not be imported: test coverage analysis not available")
    else:
        _LOG.info("coverage imported successfully: test coverage analysis available")
        DENDROPY_COVERAGE_ANALYSIS_AVAILABLE = True

        from dendropy.test import get_test_suite
        from dendropy.test.support import pathmap

        class CoverageAnalysis(Command):
            """
            Code coverage analysis command.
            """

            description = "run test coverage analysis"
            user_options = [
                ('erase', None, "remove all existing coverage results"),
                ('branch', 'b', 'measure branch coverage in addition to statement coverage'),
                ('test-module=', 't', "explicitly specify a module to test (e.g. 'dendropy.test.test_containers')"),
                ('no-annotate', None, "do not create annotated source code files"),
                ('no-html', None, "do not create HTML report files"),
            ]

            def initialize_options(self):
                """
                Initialize options to default values.
                """
                self.test_module = None
                self.branch = False
                self.erase = False
                self.no_annotate = False
                self.no_html = False
                self.omit_prefixes = ['dendropy/test']

            def finalize_options(self):
                pass

            def run(self):
                """
                Main command implementation.
                """

                if self.erase:
                    _LOG.warn("removing coverage results directory: '%s'" % pathmap.TESTS_COVERAGE_DIR)
                    try:
                        shutil.rmtree(pathmap.TESTS_COVERAGE_DIR)
                    except:
                        pass
                else:
                    _LOG.info("running coverage analysis ...")
                    if self.test_module is None:
                        test_suite = get_test_suite()
                    else:
                        test_suite = get_test_suite([self.test_module])
                    runner = unittest.TextTestRunner()
                    cov = coverage.coverage(branch=self.branch)
                    cov.start()
                    runner.run(test_suite)
                    cov.stop()
                    if not self.no_annotate:
                        cov.annotate(omit_prefixes=self.omit_prefixes,
                                directory=pathmap.TESTS_COVERAGE_SOURCE_DIR)
                    if not self.no_html:
                        cov.html_report(omit_prefixes=self.omit_prefixes,
                                directory=pathmap.TESTS_COVERAGE_REPORT_DIR)
                    cov.report(omit_prefixes=self.omit_prefixes)

While the setuptools hook looks something like this:

# ... lots ...
# ... of ...
# ... stuff ...

###############################################################################
# setuptools/distuils command extensions

ENTRY_POINTS =
try:
    from setuptools import Command
except ImportError:
    sys.stderr.write("setuptools.Command could not be imported: setuptools extensions not available\n")
else:
    sys.stderr.write("setuptools command extensions are available\n")
    command_hook = "distutils.commands"
    ENTRY_POINTS[command_hook] = []

    ###########################################################################
    # coverage
    from dendropy.test.support import coverage_analysis
    if coverage_analysis.DENDROPY_COVERAGE_ANALYSIS_AVAILABLE:
        sys.stderr.write("coverage analysis available ('python setup.py coverage')\n")
        ENTRY_POINTS[command_hook].append("coverage = dendropy.test.support.coverage_analysis:CoverageAnalysis")
    else:
        sys.stderr.write("coverage analysis not available\n")

# ... more ...
# ... stuff ...

setup(name='DendroPy',
     # ... setup configuration ...
      entry_points = ENTRY_POINTS,
     # ... more setup configuration ...
      )
Share