On writing test-cases and testsuites.

Wednesday, 22 October 2014

Last night I mostly patched my local copy of less to build and link against the PCRE regular expression library.

I've wanted to do that for a while, and reading Raymond Chen's blog post last night made me try it out.

The patch was small and pretty neat, and I'm familiar with GNU less having patched it in the past. But it doesn't contain tests.

Test cases are hard. Many programs, such as less, are used interactively which makes writing a scaffold hard. Other programs suffer from a similar fate - I'm not sure how you'd even test a web browser such as Firefox these days - mangleme would catch some things, eventually, but the interactive stuff? No clue.

In the past MySQL had a free set of test cases, but my memory is that Oracle locked them up. SQLite is famous for its decent test coverage. But off the top of my head I can't think of other things.

As a topical example there don't seem to be decent test-cases for either bash or openssl. If it compiles it works, more or less.

I did start writing some HTTP-server test cases a while back, but that was just to automate security attacks. e.g. Firing requests like:

GET /../../../etc/passwd HTTP/1.0
GET //....//....//....//etc/passwd HTTP/1.0

(It's amazing how many toy HTTP server components included in projects and products don't have decent HTTP-servers.)

I could imagine that being vaguely useful, especially because it is testing the protocol-handling rather than a project-specific codebase.

Anyway, I'm thinking writing test cases for things is good, but struggling to think of a decent place to start. The project has to be:

  • Non-interactive.
  • Open source.
  • Widely used - to make it a useful contribution.
  • Not written in some fancy language.
  • Open to receiving submissions.

Comments welcome; but better yet why not think about the test-coverage of any of your own packages and projects...?



Comments On This Entry

[gravitar] Alexandre Rossi

Submitted at 10:25:24 on 22 october 2014

Writing test cases for interactive programs is possible and useful : one needs to test the backend of interactions. Unfortunately, this often involves much refactoring if logic and ui are merged into the same routines.

I think that one good way to contribute unit tests it to submit a single test case together with the associated bug fix, and integrate all this with the build system.

[gravitar] Diggory

Submitted at 10:43:01 on 22 october 2014

Even non-interactive programs can be hard to test properly. For example, a project I am working on: https://code.google.com/p/openmalaria/wiki/Start

This is a simulator, to make predictions — but are the predictions right? Some components I can test individually, but the overall model only via "black box" tests, which tell me no more than whether the result is the same as it was last time I ran that test.

[gravitar] mirabilos

Submitted at 11:41:31 on 22 october 2014

I have lots of testcases for mksh, decent ones too, but the last dozens of bugs are all only caught by actual use, nothing proactively tested, sadly (releases usually have 0 known test failures).

[author] Steve Kemp

Submitted at 14:14:51 on 22 october 2014

I'm still struggling to pick a project that I think I can add tests to from scratch - choosing a test-suite, format, etc, is non-trivial especially if you need to persuade upstream to use/maintain it.

But yeah, point taken mirabilos, at the point you ship you're almost certainly going to be bug-free. I guess the challenge is adding test-cases as bugs are reported, so you can be sure you've fixed them.

[gravitar] Anonymous Coward

Submitted at 15:08:44 on 22 october 2014

In general, I'd say that "infrastructure" projects might be suitable. By "infrastructure" I mean the software that everyone uses (even if it's only behind the scenes) and that's expected to just work, e.g. software that's shipped in a basic desktop or server install. On the programming side, I'd count tools like a compiler, a linker or a runtime environment as infrastructure projects.

These projects might have some kind of test infrastructure but I'd wager to say that there's usually room for improvement in test coverage.

I'm sure that there are also a lot of infrastructure projects that don't have tests and would love to have somebody writing them.

[gravitar] Guillem Jover

Submitted at 15:45:47 on 22 october 2014

If you are interested in adding more unit and functional tests, I offer dpkg. :)

It has unit testsuites in the dpkg git repo, for perl code under test/ and scripts/t/, and another one for C code under lib/dpkg/test/. There's also functional testsuites for some dpkg programs under src/t/ and utils/t/. The test coverage is at https://dpkg.alioth.debian.org/coverage/.

In addition there's a functional testsuite for dpkg itself at https://anonscm.debian.org/cgit/dpkg/dpkg-tests.git.

I think it qualifies for your requirements, not sure though if you might want a project that has currently no testsuite. In any case patches welcome!

[gravitar] Stewart Smith

Submitted at 10:11:00 on 23 october 2014

The MySQL test suite is still there and still being expanded. It is Oracle policy to not publish things that may be security things - i.e. crashing bugs and security bugs don't get public test cases anymore.

Yeah, it sucks and it's the wrong way of doing things... however MySQL is such a small part of Oracle they're not going to get that policy changed any time soon.

That all being said, the MySQL test suite is larger than it has ever been before, so it certainly hasn't gone backwards.

[gravitar] bertagaz

Submitted at 10:44:56 on 26 october 2014

For interactive testing, one can use sikuli to simulate user actions, like clicking on buttons or filling a form. It's in Java, but you can find various bindings out there.

There's even a project using it with selenium and firefox.

The Tails project uses it in its tests already. It's a bit hackish but does work.


Comments are closed on posts which are more than ten days old.

Recent Posts

Recent Tags