My experience with TDD is that it is very helpful from a design perspective because it gets you to think about the code before you write it. How is the API going to work? What should I name this method? As an industry we use to dedicate too much time to this (RUP, etc). Now, we don't think about it enough.
As far as actually improving code quality... not so much. For the money, I would much rather see an hour of functional test development rather than an hour of unit test development (and I put TDD in the unit test category). I think functional tests provide more value and our industry has put way too much emphasis on unit tests.
100% unit test coverage says nothing about code quality. 100% functional coverage does. Functional tests also tend to be less brittle – which is a nice bonus.
I tried it as an experiment. With 100% coverage, TDD everything.
Code still fails in production. That is because unit tests detract from running functional/integration tests. Database, connections, hardware all mocked and each little cog is the right size. Put the cogs together into a gearbox and it shits the bed.
The one thing TDD helped was to find exact code breakage easier. Functional tests give higher level (less precise) error, maybe "something failed in the backend" then one has to look at logs to see what happens and such. But I'll take that over a false sense of security.
So am definitely not against unit tests, I am all for having a mix of both.
Another way to look at it, unit tests are a bit like drugs. It is easy to get addicted, create 10000 tests and a little micro-burst of pleasure hormone every time you see the dot on the screen for a passing test, it just feels good. But it is important to realize that is what it is (maybe I am writing all those tests just as a therapeutic exercise...), as it might not translate immediately to a ship-able working product.
I test your way, but if you're not aware of it you might want to watch this talk called Integration Tests are a Scam: http://www.infoq.com/presentations/integration-tests-scam It may explain why the "gearbox shit the bed" and how to fix that problem.
Thank you, that was interesting! A lot of good points. I agree with some of them and don't agree with other parts of it.
First, let's start with this basic principle -- "customers don't care if I have 1M unit tests or 0 unit tests". They just don't. They will complain and not like my product if it keeps crashing or is showing other pathological behavior. First and foremost it is important to ship, and to ship working code as far as customers are concerned. Taken from that side, I want to ensure that when the system is setup in customer's configuration (and we do know it, in our line of work), then we want to stimulate the system as close as possible to what a customer would do. If that doesn't break and is behavior properly. We gain immediately a large degree of confidence because it is closer to "doesn't break when customer uses" use case.
Unit tests also promise that, and they would also promise that "if I have unit-tested my code well, there is a higher chance that the code will not break when it gets into customers' hands". However notice that this is further from being true than the first one.
The problem with integration tests as discussed in the talk it they take a long time to run and give imprecise results. When something is broken I might not be able to quickly zoom in on it. I can't experiment as often as I would like because it take too long to discover precise mistakes. And I agree with those, but, I still maintain it is better to have integration tests because once you get into a stable steady state and all of the sudden one commit breaks the tests it usually is pretty clear what broke, in the worst case just revert the commit or examine in detail until it works. That helps with the "I don't know what broke, something did" part. Usually that something is the one commit where tests started failing.
Yeah, it would be better to also have complete 100% unit test coverage as well and zoom in immediately on the problem. No doubt that 100% unit test coverage also costs a lot of time and money and I posit it is often un-necessary.
Another criticism, perhaps I misunderstood the presenter, it seems to me that the talk focuses more on machine-local, object oriented code. Today the systems are more and more distributed. In a pure, local, single threaded, object oriented code (say a 3D modeler or Photoshop).The behavior or objects is more predictable and it is easier to write what he calls "focused tests". Now introduce threads and well that starts to become hairy. Add in multiple machines (a distributed system) and it gets even more complicated. There is a whole slew of strange problems. Often involving consistency, timeouts, networks failure, partitions, hardware peculiarities. Un-expected input that is generated maybe because a front end service is broken and back-end write might never expect that case and crashes. Unit tests will catch basic errors but it will mask those harder to find those kind of errors.
One more criticism is psychological and it is the false sense of security. And I have seen this happen from those that advocate unit testing everything 100%, when their code breaks, they simply don't believe their code could break because it has 100% coverage and they "just ran it and it passed". Well in production it crashes. And customers don't pay for passing tests they pay for a working product. Anyway I am repeating myself at this point and this is longer rant, so, sorry about that.
Whenever I tried to learn TDD, it only seemed relevant for back-end testing (eg REST APIs). But is there anyone left who is strictly a back-end guy? Most programmers now bill themselves as full-stack app developers, and probably spend at least half their time working on the front-end/UI.
I hadn't heard of functional testing before. From a stackoverflow answer:
> Functional Tests are written from the user's perspective. They ensure that the system is functioning as users are expecting it to.
This makes much more sense to me than TDD. TDD is writing functions which test other functions. Are the users writing code, and passing arguments to function calls? If not, then time spent on TDD isn't adding much value for the users.
So what does it mean to write functional tests? I'd argue that writing modular React components is a form of functional testing. Devcards[1] is similar, and looks to me exactly like what would be considered functional testing. These two workflows feel much more natural than testing UIs methods like automated selenium browser interactions to generate screenshots.
> Whenever I tried to learn TDD, it only seemed relevant for back-end testing (eg REST APIs). But is there anyone left who is strictly a back-end guy? Most programmers now bill themselves as full-stack app developers, and probably spend at least half their time working on the front-end/UI.
yes?
> So what does it mean to write functional tests? I'd argue that writing modular React components is a form of functional testing. Devcards[1] is similar, and looks to me exactly like what would be considered functional testing. These two workflows feel much more natural than testing UIs methods like automated selenium browser interactions to generate screenshots.
As to code quality, the closest I can come to agreeing with you is that I think some people do unit tests ritualistically, not thoughtfully, and those suites can be relatively low in value. That brittleness you describe is often a sign of that. I rarely see that from people doing actual TDD, though.
Where I part ways, though is the notion that an hour of activity X substitutes for an hour of activity Y. For any given fear you have about your code failing, you should pick the testing approach that yields the most risk reduction per unit of effort. I don't think we can externally judge the right choice for a given team and code base. As long as it's reasonably easy for people to write tests at a variety of levels (unit, integration, functional, end-to-end), and as long as they're committed to continuous improvement, I'm inclined to trust that they'll figure out the right mix for their project.
I share your perspective on this. I have also found that writing tests forces you to write testable code (goes without saying). Often times, if code is impossible to test and requires to you to mock and inject a bunch of different things just to write the test then you're design is probably no good. So, while this necessarily mean you have to do TDD, it's interesting to see that testable code and good design kind of line up.
> it's interesting to see that testable code and good design kind of line up
They can line up, but I'm not sure there's a causation. There are many different ways to write tests the wrong way. It's taken a lot of trail and error for me to get better at it (I can't even say I'm "great" yet). I'd have given up long ago if I wasn't a passionate developer. A less passionate developer would think the whole concept is BS and give up long ago if he took my path.
One of the first mistakes I made was using a mocking framework to test my implementation. The test read like, "first call this method, then call this, then call this". Needless to say it was worthless. I was just writing my code twice.
Then you can use mocks inside-out and end up writing code you don't even need to call, etc.
I'm sure there's many ways to do TDD right, but from in my personal experience, there's more ways to do it wrong.
I don't know about your code quality assertion. My experience has been that it produces code right off the bat that is equivalent to code that has been through 1 or 2 rounds of functional testing. I ignore code coverage, though. It's not a good metric.
As far as actually improving code quality... not so much. For the money, I would much rather see an hour of functional test development rather than an hour of unit test development (and I put TDD in the unit test category). I think functional tests provide more value and our industry has put way too much emphasis on unit tests.
100% unit test coverage says nothing about code quality. 100% functional coverage does. Functional tests also tend to be less brittle – which is a nice bonus.