DiagnosticsWithTheCode

The practice of keeping easily disabled detailed diagnostic (or logging) information with the code, so that you can see what it is doing purely by enabling more diagnostics. And when you can't, you provide more.

It needs to be easily disabled because all that detailed text is too much information most of the time. You need to be able to turn it on in a focussed way when you know there is a problem with a certain aspect of the code. A whole family of tools exist to do this: they're called log4x where x is the usual shortcut for the language. (log4j, log4py, log4cpp are the main ones I've met). As usual, the java version is the original.

This is more or less essential with a text-based testing approach such as that used by TextTest. It can provide extremely effective debugging information by taking differences of detailed diagnostic text with the previous, bug-free version. Otherwise, you run into the objection usually raised by those who believe you need unit tests as well as AcceptanceTests: that an AcceptanceTest failure is too coarse-grained to be useful in debugging and a unit test must be created. An AcceptanceTest failure should at least tell you enough to know which diagnostics to enable, and then you have all the failure information you need.

-- GeoffBache

How does this compare with TestFirstDebugging? (aka writing tests before fixing bugs)? If you look at the source of a project like Axion (http://axion.tigris.org) they've been very explicit about writing a specific unit test for each bug and as such they get free regression testing and in effect a history of their debugging sessions. If all you have are relatively course-grained acceptance tests you lose these benefits.

Even assuming the diagnostics in the code provide enough information for debugging purposes we still need some way to get the 'executable bug history' feature of unit tests. Do your bug-tracking practices end up having to be enhanced as a consequence? -- AdewaleOshineye

We regard the purpose of the AcceptanceTests being to indicate that the bugs exist. The diagnostics then provide the detail needed to fix the bug. We do TestFirstDebugging? in this sense, our bug-fix procedure is something like

1. Get a failing AcceptanceTest
2. Run with all available and relevant diagnostics
3. If insufficient to fix the bug, write more diagnostics
4. Fix the bug
5. Check in any diagnostics written in step 3 along with the bug fix
6. Make the AcceptanceTest faster to run, if possible, now we know what the bug is, and check it in too

You are right that afterwards we cannot necessarily easily track which diagnostics were written with which bug. But the future purpose of the diagnostics is to detail the code they are written with, not to "prove" anything about the code. The AcceptanceTest does that.

Does that answer your question?

-- GeoffBache

Yes, thanks. Your answer clarified things for me. I do wonder though if your diagnostics might not benefit from being assertions as well as logging messages. Of course then we're left with the question of what to do if an assertion fails.-- AdewaleOshineye

Now I don't follow. The diagnostics describe the code behaviour in general terms, logging things like "The number of widgets is 377". But I cannot assert that the answer is 377, because this is real code, not a single unit test. I have hundreds of AcceptanceTests and the answer is different in each one. But I can see very precisely what went wrong by taking a diff of this diagnostic with my previous, correct version, assuming I have one (as I do 90% of the time when I'm developing and break something) -- GeoffBache