[Discuss] writing unit tests for scientific software

C. Titus Brown ctbrown at ucdavis.edu
Mon Jul 17 11:11:38 CDT 2017


Hi Terri,

I think lots of people in the scientific Python community write their
own algorithms and test them.  But it's hard to give generic advice here,
I think, because it's so dependent on your algorithm.

Here's my try / our approach that has worked well for us over the last
decade or so.

* first, write automated "smoke" tests that check to see if your code is
  basically running/working.  They should be as dumb and robust as possible.
  (e.g. the equivalent of "check that 2+2 = 4").

  These are by far the most important in my experience, in that they deliver
  the most value for the least effort.

* set up CI on those tests.

* check code coverage of your code base, and try to get it to 30-40%
  by testing the basic code paths.

* write a series of basic tests for edge cases (divide by zero, boundary
  conditions, that kind of thing), trying to cover another 10-20%.

* as your code base matures and complexifies, write tests for new functionality
  and try to cover old functionality as well.  Here code coverage is your
  friend in terms of targeting effort.

* whenever you discover a bug, write a test against that bug before fixing it.
  That way your most error prone bits will get more coverage adaptively.
  I call this "stupidity driven testing."

Lather, rinse, repeat.

tl; dr? smoke tests, code coverage analysis, test against buggy code.

best,
--titus

On Mon, Jul 17, 2017 at 11:50:59AM -0400, Terri Yu wrote:
> Thanks everyone, those are interesting resources for testing in general.
> 
> I'm using Python's unittest framework and everything is already set up.
> The specific problem I need help with is what tests to write, in order to
> test numerical floating point output from algorithms.  Given the responses
> I've gotten, it seems like not many people write their own algorithms
> and/or test them.
> 
> Terri
> 
> On Sun, Jul 16, 2017 at 5:50 PM, Jeremy Gray <jeremycgray at gmail.com> wrote:
> 
> > Hi Terri,
> >
> >
> > It might also be worth checking out the workshop from this years pycon
> > from Eria ma:
> > Best Testing Practices for Data Science, on yotube here -
> > https://www.youtube.com/watch?v=yACtdj1_IxE
> >
> > The github repo is here:
> > https://github.com/ericmjl/data-testing-tutorial
> >
> > Cheers,
> > Jeremy
> >
> > On Fri, Jul 14, 2017 at 5:21 PM, Olav Vahtras <olav.vahtras at gmail.com>
> > wrote:
> >
> >> Dear Terri
> >>
> >> In addition I can recommend the following resource:
> >>
> >> pythontesting.net has a podcast series on testing and more, check out
> >> the new book on pytest by the site maintainer Brian Okken
> >>
> >> Regards
> >> Olav
> >>
> >>
> >>
> >> Olav
> >> > 14 juli 2017 kl. 21:36 skrev Ashwin Srinath <atrikut at g.clemson.edu>:
> >> >
> >> > If you're using Python, numpy.testing has the tools you'll need:
> >> >
> >> > https://docs.scipy.org/doc/numpy/reference/routines.testing.html
> >> >
> >> > There's also pandas.testing for testing code that uses Pandas.
> >> >
> >> > Thanks,
> >> > Ashwin
> >> >
> >> >> On Fri, Jul 14, 2017 at 3:27 PM, Terri Yu <terri at terriyu.info> wrote:
> >> >> Hi everyone,
> >> >>
> >> >> Are there any resources that explain how to write unit tests for
> >> scientific
> >> >> software?  I'm writing some software that processes audio signals and
> >> there
> >> >> are many parameters.  I'm wondering what's the best way to test
> >> floating
> >> >> point numeric results.
> >> >>
> >> >> Do I need to test every single parameter?  How can I verify accuracy of
> >> >> numeric results... use a different language / library?  I would like
> >> to do a
> >> >> good job of testing, but I also don't want to write a bunch of
> >> semi-useless
> >> >> tests that take a long time to run.
> >> >>
> >> >> I would appreciate any thoughts you have.
> >> >>
> >> >> Thank you,
> >> >>
> >> >> Terri
> >> >>
> >> >> _______________________________________________
> >> >> Discuss mailing list
> >> >> Discuss at lists.software-carpentry.org
> >> >> http://lists.software-carpentry.org/listinfo/discuss
> >> > _______________________________________________
> >> > Discuss mailing list
> >> > Discuss at lists.software-carpentry.org
> >> > http://lists.software-carpentry.org/listinfo/discuss
> >> _______________________________________________
> >> Discuss mailing list
> >> Discuss at lists.software-carpentry.org
> >> http://lists.software-carpentry.org/listinfo/discuss
> >>
> >
> >

> _______________________________________________
> Discuss mailing list
> Discuss at lists.software-carpentry.org
> http://lists.software-carpentry.org/listinfo/discuss

-- 
C. Titus Brown, ctbrown at ucdavis.edu


More information about the Discuss mailing list