Support for retrying intermittently failing tests

[jelmer]

Migrated from Launchpad: https://bugs.launchpad.net/bugs/1515933

Original Reporter: Jonathan Lange
Date Created: 2015-11-13 09:47:37.318847+00:00
Status on Launchpad: Triaged
Importance: Wishlist

---

testtools should provide a way to retry intermittently failing tests, ala the https://pypi.python.org/pypi/flaky decorator that works with py.test and nose.

Ideally, we would support the @flaky decorator provided by the flaky package.

Essentially, users could specify a maximum number of attempts, N, and a minimum number of successes, M. The test would be retried up N times until there were M successes. If there are M successes, the test counts as passing. If there are not, then the test counts as failed.

Either way, we would attach the errors to the test as details, and also provide some other sort of annotation to make it clear that the test was flaky.

I think the right way to implement this is a RunTest object that wraps another arbitrary RunTest object, so that the flaky retrying logic would work with both Twisted & regular tests. This might require clarifying the interfaces provided by RunTest implementations.