WhatCurrentlyWorksForPhilDawes

I'm constantly seeking SilverBullets?. This means that my opinion on what rocks and what sucks swing wildly from one week to the next. Rather than pollute the wiki with pages like TestingInIsolationRocks? and TestingInIsolationIsABigPileOShite?, I'll use this page to sandbox my thoughts and ideas before inflicting them on others. Kind of like a quarantine area for my drivel.

(Context: I'm currently investigating using XP techniques in FreeSoftware? projects. I'm working on the Wiki.BicycleRepairMan project in my spare time)

What currently works?

Small black box tests seem to work for me at the moment. I've started following the unit-verses-functional tests stuff on the c2 wiki, and there doesn't seem to be any conclusions.

• Black box tests encourage me to test what is important

...As long as important is defined by 'externally visible'. Seems to be about right

• Black box tests allow me to refactor without breaking tests

• DoTheSimplestThingThatSatisfiesTheTests appears to work

What currently doesn't work?

• Testing implementation doesn't work

...Because I can't refactor without breaking tests. In doing this I end up re-writing tests a lot, which isn't fun (one of my primary criteria for this is to have fun). This seems to imply that small functional tests work better that unit tests. hmmm...

• Testing in isolation doesn't seem to work

The problem seems to be that if you test everything in isolation, nothing asserts that the whole system meets your requirements. In addition, if you change or break something in one component/class/whatever, the tests wont tell you that it breaks another dependent component, or that the system doesn't work as a whole.

Even if you write integration tests, unless these tests confirm that everything works in the whole system (i.e. everything checked by the xpunittests is also checked in the context of the whole system) then you are basing these on an assumption that a limited set of 'hangs together' tests will do.

The most obvious StrawMan? solution to this is to have unit tests that can also be executed in the context of the whole system. --PhilDawes

Yes, but are those still unit tests? In lots of applications, setting up any particular state in the system is complicated enough to be effectively a functional test.

I think you're right - this is functional testing. The intention is to factor out the logical test; (e.g. withdrawls on a negative balance result in failure) from the state and implementation at various levels (e.g. at top level 'withdrawls on a negative balance results in 'not enough funds to make a withdraw' message', at credit check component 'checkCredit method returns false if amount is greater than funds in account' etc.. etc..

Actually on attempting to do this, there isn't much commonality to factor out between the tests, since as you stated, the setup-state part of the test is the majority of it, and doing so for both tests is radically different.

³host³³date³November 19, 2001³agent³Mozilla/5.0 (X11; U; Linux i686; en-US; rv:0.9.2.1) Gecko/20010901³WhatCurrentlyWorksForPhilDawes