My experience is that XP sits very well with developing algorithms. My approach is to chop the algorithm up into steps that are simple enough for me to figure out test cases on the back of an envelope. When each of the steps is tested you can have a simple test case that tests end to end. This way you don't have to figure out lots of complicated boundary conditions (which make for difficult to read and brittle tests).
The caveats with this approach are (I would guess) efficiency and floating point considerations: you may need to optimise your algorithms in ways which make them harder to test and if you have precision problems this form of testing won't pick it up.
MathematicalProgrammingAndXP