UI Automation tools snake oil?

Michael Feathers posted a thoughtful piece describing the general problems of UI testing tools and the industry in general.  In general, I’d agree here.  Automation tool vendors, as with almost every tool vendor out there, are eager to solve perceived software development problems.  Unfortunately, these tools usually only address symptoms of larger problems.

With UI testing tools, this is no different.  When we started out authoring UI tests for a large MVC application, we had a few large goals in mind.  The tests needed to be:

• Stable
• Understandable
• Maintainable
• Valuable

To achieve these goals, we needed to do exactly what we do when we write code using TDD, design for testability.  I won’t go into everything we did, there’s already a whole screencast on that.  However, Michael does bring up some key points:

• UI tests should test the UI layer
• Architect your system so that you can test all of your business logic separate from the UI layer

That’s what we’ve done, and it’s been very successful.  We have UI tests, around 500 of them, and we’re not staying up until all hours of the night keeping them working.  We also have subcutaneous tests, that work in a layer right below the UI layer, that allow us to send a UI message into our system and check what comes out on the other side, whether it’s business rule violations or entities modified.

All of our testability enhancements came from design for testability.  We use strongly-typed views.  We don’t have any business/domain logic in controllers, only UI controller logic.  Separating your architecture into layers isn’t for portability, it’s for separation of concerns.  Once we started designing for testability in the large, and not just the unit level, we were able to maximize the value of automated tests, both at the UI, integration and unit level.

If anything, this is another example of the need for a developer to work on their spidey sense for vendor product demos.  We rail on MS for drag-and-drop demos, but it’s every tool vendor that employs this technique.  Just like every other product sold in the world, it’s up to the buyer to be critical of the true value a tool can bring.