Continuous Integration Tip of the Day

We wasted quite a bit of time not following this rule today:

If the CI build fails, roll back the previous revision.

The build failed, and from the exception we assumed it was an environmental issue.  Well, a few commits later, and the build kept failing.  We couldn’t tell if it was the original commit or later commits that made the build keep failing.  We tried reverting individual revisions, but no go.

Eventually, the only thing that worked was reverting back to the last successful build.  Thus wasting quite a bit of everyone’s time.  Lots of excuses on why the build should be green when it wasn’t, but a green build is a green build.  A red build is a red build.

This is one of those lessons it seems like we have to learn every few months.  Hopefully, this bad taste will stick in our mouths for longer next time.