If you’re not currently saving your old builds, you should start now. Saving old builds can be very useful for a few reasons. First, it makes it quicker to start debugging problems that only shows up sin an old build. Second, it is insurance against the possibility of not being able to recreate your build environment correctly.

For hosted environments, where you control the deployment of the software, just keeping a couple of old builds around is probably enough. Being able to roll back if something goes horribly wrong can be a life-saver. For installed software, it’s important to keep all old builds that are still available in the wild or that customers might legitimately want for some reason. Practically, this works out to be all builds that you’ve released in some form to customers. During development, it can also be useful to have a few recent builds available that have only been released to the QA team to easily compare versions.

As a simple first step, you can simply store old builds on a file share. It’s not too much more work to store them in a version control system. Make sure to use a VCS that can handle large binary files easily - SVN is a good choice currently - so that you don’t have to worry about performance issues in the future.

Finally, when storing old builds, make sure to keep each configuration of the build. When a customer reports a problem in an old release, having a debug build of the code with symbols can rapidly decrease the time it takes to respond to their problem.

An official build machine is an important tool for building and releasing software. If you don’t have a machine dedicated to that purpose, you have to worry about where each build and release comes from. Is the environment the same? Are all the dependencies the same? Which developer’s box do builds typically come from? What if he or she is out on vacation? Can you emulate that box enough to produce an equivalent build and release?

The answers to these questions may vary from product to product and company to company. The simplest solution, however, is to have a dedicated build machine that has a known environment, has all of the correct dependencies set up, and can be used to cut official builds whenever necessary.

This build machine (or set of machines, for larger products and longer builds) will be where official builds are created for release to internally, to beta users, and for customers. It can also be the machine that handles continuous integration or continuous deployment responsibilities. As such, it will not only be building the software, but running sets of tests, once it is built. Getting this machine up and running, and maintaining it as necessary, are the primary responsibilities of the a build and release manager.

It’s important that you have a single build script to build your product - every configuration of your product. Having a separate script for debug vs release, or x86 vs x64 architecture, is only going to cause problems. If the actual built files need to be different for different deployment target, then these targets also become build configurations, such as hosted vs installed. And the build script need to be able to build each of these configurations.

The easiest way to do this is to have the build script take command line parameters that specify which configurations to build (or to build all of them). The command line parameters should be short, or have short versions, to make it easier to run the script frequently.

Rather than specify that the script should build one configuration or another, it should be allowed to build multiple configurations. This requires that each configuration build to a different location. So you may have a build directory next to or within your source directory, that has a subdirectory for each configuration built. The simplest example of this is Visual Studio’s bin\Debug and bin\Release directories for it’s default projects.

Obviously, you shouldn’t need to enter command line parameters. The defaults should be the configuration(s) that developers will use most often. It may be helpful to make it easy for developers to easily configure their defaults in a settings file, if different developers have different needs.

Sped up our build today by detecting service stop rather than waiting five minutes (twice) in hopes it had happened

You may ask this question when you notice that every time you build, you also deploy. What’s the point in keeping the scripts separate if they will always be run together? Before listing all of the reasons I can think of, let me tell you about a bug we recently had in our (currently combined) build and deployment script.

The installer for our licensed FogBugz product is created by a single script that builds both the FogBugz binaries, as well as the installer program that includes all of the binaries in a zipped up cab file. The script first copies all of the files that will be needed in the cab to a specific directory, then builds them, then adds everything in the directory to the cab file, and proceeds to build the installer. A developer recently noticed that certain files were not being included in the cab file that should have been. Closer investigation revealed that the step to create those files failed because the tool necessary to build them wasn’t one of the files copied into the directory to be zipped into the cab.

Now obviously, a lot had to go wrong for this to happen. The build didn’t correctly report the failure to build those files. The build tool required was part of the code repository instead of being placed in a common build tools repository. The build tool required was called via a path relative to the code, rather than relative to the build script. But the issue I’m looking at today is that we mixed up steps that were meant for the deployment with steps meant for the build.

In this case, the deployment script was the script that took built FogBugz files and created an installer with them (deploying to a setup program, as opposed to deploying to the web). Because the steps were mixed together instead of strictly separated, the necessary build tool was not at the expected location, and the build failed (silently, unfortunately).

There are obviously less architectural changes we could make that would have helped us identify and fix this issue more quickly - things like reporting all build errors, or calling the build tool with a path relative to the build script, not the code. But separating the build and deployment scripts would have prevented the bug from occurring in the first place.

Keeping the scripts separate has the following advantages:

1. Bugs caused by mixing deployment steps into the build script go away.
2. A single build can be used for multiple deployments (to an installer, to a web site, on a dev machine)
3. Developers can use the build script on their dev machine, instead of doing the build manually or having a separate script that must be kept in sync.
4. The development team can take ownership of the build scripts, without having to worry about the details of deploying, which can be left to system administrators and release managers.