Writing Faster ModuleStore Tests
Update: SharedModuleStoreTestCase has evolved since this was written. Please see the docstring for updated usage info.
Most developers who have added features to edx-platform are familiar with
ModuleStoreTestCase. If your tests exercise anything relating to courseware
content (even it’s just creating an empty course), inheriting from this class
will ensure that data gets cleaned up properly between individual tests. This is
extremely valuable, but can also be wasteful in many situations. During last
week’s hackathon, I created a faster alternative
SharedModuleStoreTestCase only does
ModuleStore cleanup at the
tearDownClass() level. It’s meant to be employed
in situations where one or a small handful of courses can be initialized up
front, and then shared in a read-only manner across many tests. This usage
pattern is commonly found in LMS tests, which often simply recreate the same
course over and over again in their
To get an idea of the effect it could have, I switched over a few test modules as part of my hackathon work. These are only rough figures, as they are based on a relatively small number of Jenkins test runs. That being said, the results are promising:
So how do you convert your own tests?
Making the Switch
Most classes that inherit from
ModuleStoreTestCase start something like this:
from student.tests.factories import CourseEnrollmentFactory, UserFactory from xmodule.modulestore.tests.django_utils import ModuleStoreTestCase from xmodule.modulestore.tests.factories import CourseFactory, ItemFactory class MyContentModifyingTestCase(ModuleStoreTestCase): def setUp(self): super(MyContentModifyingTestCase, self).setUp() self.course = CourseFactory.create() self.user = UserFactory.create() CourseEnrollmentFactory.create(user=self.user, course_id=self.course.id)
If you are modifying
self.course in your individual test functions, then
this is perfect, and you should continue to use
if you’re just setting up the course once and treating it as read-only in your
tests, you can now do this instead:
from student.tests.factories import CourseEnrollmentFactory, UserFactory from xmodule.modulestore.tests.django_utils import SharedModuleStoreTestCase from xmodule.modulestore.tests.factories import CourseFactory, ItemFactory class MySharedModuleStoreTestCase(SharedModuleStoreTestCase): @classmethod def setUpClass(cls): """Any ModuleStore course/content operations can go here.""" super(MySharedModuleStoreTestCase, cls).setUpClass() cls.course = CourseFactory.create() def setUp(self): """Django ORM operations still need to go in setUp() for now.""" super(MySharedModuleStoreTestCase, self).setUp() self.user = UserFactory.create() CourseEnrollmentFactory.create(user=self.user, course_id=self.course.id)
It’s important that Django ORM operations remain in
setUp(). Any models that
you create in
setUpClass() must be manually deleted in your
SharedModuleStoreTestCase will not properly clean them up. Even if
you’re careful, you’re still likely to break other tests in the system in
unpredictable ways because they make bad assumptions about sequences and what
IDs will be created when they set up their data. This can be extremely tedious
When we upgrade to Django 1.8, you’ll be able to use
to safely do class-level initialization of Django models with automatic cleanup.
Please wait for that upgrade and place model manipulations in
setUp() for now,
even if it is a bit slower.
Which Tests Should I Convert?
The easiest place to hunt for test optimization targets is the Jenkins test build report. Click on “Duration” to sort by that column.
We primarily want to target expensive tests that either create complex course
data (e.g. CCX) or have simple course data but many, many tests (e.g.
discussions). Creating even the simplest course takes about 250-300ms or so,
which really adds up when using tools like
that effectively multiply the number of tests in a class.
Test data creation and cleanup can be an expensive, and the
ModuleStore is a
prime example of that. I hope that
SharedModuleStoreTestCase can be a useful
tool for bringing down test execution times. But beyond that, I hope that
understanding why it works will allow us to design faster test suites in general.