What is build and release management?

Can you build your product with one command?

Where do developers keep their checked-out source code?

How do I eliminate absolute paths from my code and build scripts?

Do your builds have other hard dependencies?

You keep talking about hard dependencies. Are there any soft dependencies?

How are build and deploy scripts different?

What should you include in machine setup scripts for dev and build machines?

Can you deploy your product with one command?

Are your build scripts crusty and old?

Why should my build and deployment scripts be separate?

Can you build all possible configurations using command line parameters?

Do you have an official build machine or set of build machines?

Do you save your old builds?

Do you store built binaries in your VCS?

Do you have different build scripts for different configurations of your build?

Do you have different build scripts for official builds vs. dev builds?

More to come!

Related Posts

Deploying FogBugz 8 and Kiln 2

FogBugz and Kiln Build and Release Report Card

“Have you been flossing regularly?” - Your Dentist

Don’t you hate it when you go to the dentist and, while the dentist is digging around in your mouth, he asks if you’ve been flossing?

“Huh-mhqhg,” you respond, hoping that all the fingers in your mouth made your negative answer sound like a “Yes, sir” on the way out. But no; dentists (and their assistants) have an uncanny ability to understand the most garbled language correctly. Either that, or they’re just really good at reading the guilty look in your eyes.

And then the kicker comes:

“Yeah, I could tell by the profuse bleeding that happens when I start jabbing your gums with this sharp metal instrument of torture.” Ok, so the dentist doesn’t actually say that, but hey, if he can hear what he wants in my garbled mumblings, I should be able to hear his words the way I want to, right?

Anyway, let’s assume he’s probably right, and that regular flossing would actually eliminate the need for blood transfusions after each visit to replace all that was lost. From the scattered times in my life when I’ve been able to keep up flossing for a week or two, I know that after the first few days the bleeding does go down significantly. So maybe the dentist has a point, and not just at the end of those instruments of torture.

But how do you go from a lifetime of flossing for a week or two after each dentist visit and then forgetting about it completely until the next one, to actually making it something that you do each day without thinking much about it?

The trick is building a habit. No, it’s not easy. Yes, it will take time. If you need help, I recommend following these basic steps. There is no easy way to build a good habit, but you can make it easier. Choose a trigger, make it public, keep it simple, build anticipation, take your time, report to others.

I went through these steps over the last month or so, and now I’m flossing daily, even under the dental wire that is the last, permanent reminder that I had braces as a kid. Yeah, I was even more nerdy back then.

Anyway, I’m writing up this post because my next dentist visit isn’t coming for a few months. But if I tell my small corner of the internet how that visit will go it will help me stick to my habit until then.

So now, the next time my dentist asks, “Have you been flossing regularly?”, I’ll have a better answer for him:

“Huh-mhqhg”

Ok, you might not be able to tell the difference, but somehow the dentist will see the gleam in my eye, and know it is not the glint of guilt he saw at my last visit.

In Drowning in a C of Interviews, Benjamin points out that he doesn’t see much value in doing interviews exclusively in C. Additionally, he believes that “we’re making candidates nervous about how well they know C, and focusing too much on how well they understand C’s semantics, but not really getting any meaningful indicators back in return.”

I have a different take on our interviews. Though I haven’t done as many interviews here at The Creek as Benjamin, throughout the last intern hiring seasons I saw a sum total of one candidate that actually suffered in an interview from not feeling comfortable with C. While I agree with Ben that allowing the candidate to code in the language they are most comfortable with won’t really change how many candidates get the “hire” decision, I do think there are some benefits to interviewing in C that we might lose.

It will help to step back and consider what we are looking for at Fog Creek. Our listing for a software developer position has this to say about what we’re looking for:

Right now we use C#, JavaScript, XHTML, CSS and even Wasabi (an in-house .NET language) to develop FogBugz. Kiln uses C#, JavaScript and Python, Copilot uses C, C++ and Objective C, and we have some legacy code written in VBScript. Tomorrow we may be using something completely new.

Whatever technologies, languages, or development environments you’ve been using, we expect you have mastered them in depth, and we expect that you will be able to master any technology, language, or development environment that we need in the future.

So we’re really looking for two things. First, do they code enough to have mastery in the languages they are most comfortable with. And second, can they use an understanding of broader principles to quickly learn new languages and technologies.

If we always let candidates code in their language of choice, we’ll be testing more for first quality. As such, we’ll want to dive deep into the way things are implemented, different ways of using the language to tackle the problem presented, and we’ll expect that the code is both syntactically and semantically correct. We would only get indirect hints as to the second quality - if the candidate can handle the problem itself then it means they can learn quickly on a small scale, and it most likely means they have a good grasp of the principles required to quickly get up to speed on a new language. But that’s not a guarantee.

If we did all our interviews in C, then, for most candidates we’ll be testing for the second quality - their ability to get up to speed in a language they don’t use on a regular basis quickly. They may be familiar with C, or may not, but they know we interview in C, so they can prepare for it. I specifically request that candidates code in C or C++, and while it is generally easy to see who is comfortable with it, it’s also fairly easy to tell who picked it up recently and didn’t have any problems with the question. I get excited about these candidates, because I know they can pick up new technologies quickly. I get (slightly less) excited about the ones who are obviously comfortable with C and do well on the actual question.

I can think of two interesting third alternatives to the “all in C” vs “pick your language” debate. One would be to work for a good mix of interviews in a language the candidate chooses with interviews in C, the “desert island language”. That would allow candidates to show off mastery of their tools of choice as well as the ability to work well outside of those tools. And since Ben stopped doing his interviews in C, I guess I’ll have to keep doing mine in C. Especially since his blog post will probably convince a couple other guys here to give up the C requirement. And besides, it makes life easier for me.

Another option would be to use a relatively obscure language, or one we make up (Wasabi, anyone?), in place of C for part of our interviews. We could test for mastery of the tools they use by having them code up the solution to a problem their own language of choice. Then we give them a different language, possibly with a fundamentally different paradigm (OO vs functional vs procedural, etc.), and have them code it up again. At this point, they understand the problem and solution fairly well, so they’ll be more fully demonstrating their ability to learn a new language and write code in it.

I’m not totally happy with either of these options, but they can certainly add to the discussion we’re having at Fog Creek about how to find and interview developers who want to help all developers make better software.

I hope you don’t have one script or set of scripts for dev builds and a separate script for official builds. For teams that have made the good step of setting up a dedicated build machine, this form of build script decay is probably more common than having different scripts for different configurations.

For example, you start getting close to shipping an actual product to actual customers, and so you setup a dedicated build machine. In getting build scripts to work you copy the build scripts the developers have been using, but you copy them, instead of just using them directly. This is because there are all kind of changes you need to make now that your code is being compiled and deployed to production, whether that is a web server, or wrapped up in a client installer, or copied to an embedded device. Once you get everything working, you then forget to refactor and move things back to a single script. In the best case, your official build script just calls the development build script. In the worst case, they both do a ton of similar, or exactly the same, things. But now you have to manage both scripts.

Even in the best case, where the official build script calls the development build script, there are probably improvements you can make. There may be steps in the official script that would be useful and/or necessary for developers to run also. Maybe not for every build, but probably for some builds. Taking some of what was done for the official build script might also improve developers life’s by making it easier for them to have multiple installations on their development box with less overhead work on their part.

Of course, in the worst case, with tons of duplicated script code, you’re going to have issues when developers fix the development build script but forget to fix the official one. And vice-versa. But if you fix the scripts by combining them, and using appropriate command line parameters for any differences that absolutely need to exist, you get developers to help you maintain the official build scripts for free. By combining them you also encourage separation of deployment and build scripts, which will further improve your process.

One of the most natural ways that build scripts decay is when you create create separate scripts for different configurations of your build. This typically does not happen when configuration just means “debug” or “release”. That’s usually handled quite well with compiler and/or linker flags, or just using command line parameters or other ways of modifying run-time behavior. I’m referring instead to different configurations that you actually ship to customers. Here at Fog Creek we have hosted FogBugz and Kiln as well as licensed FogBugz and Kiln. Other products might have both “lite” and “pro” editions. Or maybe you have a separate configuration of your product for each customer. Or you’ve got an internal app that needs to be a little different at each of the company offices.

The obvious problem with having separate build scripts for each configuration is that you then have to remember to change multiple scripts when something common to all of them changes. And the common stuff will change more than the differences will. When your scripts are used to build every configuration, then you know going into a change that you need to consider each configuration. For example, it may very well be that you’re making a change that will only affect “pro” users, but then you’ll remember that you need to properly disable the feature in the “lite” edition.

Like most cases of repeating yourself, it’s fine to do it once, to understand the differences, for example when you first add a new configuration. But then you should refactor, eliminating the repetition by using common build scripting technology. Whether you’re using msbuild, make, ant, maven, or whatever, it’s quite straightforward to have a single script deal with multiple different configurations of the build, even when fairly major differences arise between them. Often, the largest differences between configurations are not build-related but deployment-related. So if you’ve separated your build and deployment scripts you’re already halfway there.