Pretty sweet comments about Kiln: http://www.offroadcode.com/2010/7/13/switching-our-version-control-system-from-svn-to-kiln-part-1.aspx

With all of the great advice about productivity in the world, why aren’t we all more productive? Why haven’t we figured out the right system for managing our time, our lives, our actions, or whatever, so that we can stop reading more self-help books? Or at least stop feeling like we should read more self-help books. Why do we jump at new planning systems, then jump back to old planning systems, only to give up on anything for a while before we realize we need something to keep us from forgetting stuff? More importantly, why didn’t the last system we tried, which seemed so promising, fall flat on its face? And why is it that your current planning system will fail - you know it will eventually, all the other ones did - leaving you looking once more for the silver bullet to end your inability to get things done while feeling calm and at peace?

You are the problem

The answer is simple, but hard to accept. You are the problem. You are the common factor through all of the failures. You are the one who couldn’t keep up the practices of each system; the weekly planning, the regular review of next actions, processing your inbox to zero, or remembering to copy tasks from one planner page to the next. And when you think about it you have to admit that for almost any practice espoused by some particular system or style of planning, if you were doing that one thing consistently you’d be better off now than if you weren’t.

I’ve often asked myself the questions above and often avoided the answer I just gave. It’s far more satisfying to just assume that the fault is in the system. That’s easy enough to do by claiming that it’s too complicated, or it’s not worth the effort, or it doesn’t cover enough unique situations, or it’s not a complete solution to your problems, or it tries to solve every problem, even ones you don’t have. While any or all of these might very well be true, I’m going to claim that that is not the real problem. If you’re still looking for the right planner, the right tool, the right technique, the right practice, it’s because you haven’t accepted personal responsibility for your own past failures. You’re blaming them on the tools, techniques, and practices you were using at the time.

Different Questions

Once you do take responsibility, truly owning up to the challenge of managing your own time and life, you’ll start to wonder what you could have done differently with the tools, practices, and techniques, rather than trying to find different tools, practices and techniques. The questions you’ll ask will be different. Why didn’t I do a weekly review each week? Why were my next actions lists overwhelming to me? Why didn’t my weekly compass translate into a good week? Why did my inbox zero devolve to inbox ten, then inbox 100? Why did a tickler file seem so useful in theory but fall flat on its face when I tried to use it?

Notice that these question don’t contain the assumption that the problem was with the tools, practices, or techniques. They contain the assumption that you could have done something differently to make things work out. Once you and I accept and internalize that assumption, we change the way we approach the problem. The answers to those questions look different than our past excuses for failure. Why didn’t I do a weekly review each week? Because I didn’t make it a priority and underestimated how much effort it would take to keep doing it regularly. Why were my next actions lists overwhelming? Because I spent more time making lists than actually doing anything on them. Why didn’t my weekly compass translate into a good week? Because I wrote it up on Sunday and never looked at it again through the week. Why did the tickler file fall flat when I tried to use it? Because I could never get into the habit of looking at it on a daily basis.

Habits

The root of each of these answers takes us back to developing good habits. The key to most productivity gains is not the actions that any given author or system espouses, but that you make those actions into habits. It’s not doing a weekly review, but the habit of doing a weekly review. It’s not next action lists, but the habit of keeping next action lists up to date and reviewing and doing the actions on them regularly. It’s not having a weekly compass but the habit of making one each week and using it to guide your actions each day. Its not getting to inbox zero, but the habit of getting to inbox zero on a regular basis. It’s not keeping a tickler file, but the habit of checking it daily.

Now I have to apologize. It’s not completely your fault that none of these planning systems or tools worked out for you. Part of the problem is that the proponents of Inbox Zero, GTD, 7 Habits, etc. don’t always focus on the development of habit as the key to success. They may pay lip service to it. They may, as 7 Habits does, even spend a fair amount of time talking about the importance of developing habits. But they don’t offer a step-by-step approach to creating new habits in your life, or eliminating the bad ones. And they rarely call out which specific habits you should develop and in what order. The 7 Habits mentioned in Stephen Coveys book also present the challenge of being quite abstract, and other systems sometimes espouse equally abstract principles that make it hard to develop actual habits which translate the principle into daily actions that improve your life.

Take responsibility

Remember, though you may not be totally to blame, it’s still your responsibility to do these things, not some author who lives across the country. You’re the one who needs to recognize that any given tool or practice isn’t going to improve your life unless you develop the habit of using it regularly and correctly. You’re the one who has to do the hard work to develop those habits. You’re the one who has to pick yourself up and keep trying after minor or major failures. And blaming the tool or practice when you haven’t done any of those things is a cop out.

I know. I used it as an excuse for too long. Even after I started developing new habits using the ideas at 6changes.com, it’s still taken months to realize how the power of good habits takes away my excuses. It’s humbling but it’s also very empowering.

One of the responses to my first post on the 6 Changes method of developing habits was from someone who admired how I had combined a few different things that I wanted to get done each day into a single habit that I did each night before going to bed. I took that to heart and tried to do the same again in March and April by bringing together some bad habits I wanted to overcome with my desire to start blogging regularly to create a single habit to work on. I started off well, setting aside time each day to write, timeboxing my internet surfing, etc. But because the individual steps weren’t really incremental improvements toward a single habit, I didn’t benefit from the slow but steady accumulation of conditioned responses. And because of that, some of the changes haven’t stuck at all. Others I manage to do sometimes, but I cannot say that they’ve become habits I’ll keep my whole life.

Basically, I fell into the classic beginners trap. Some initial success made me think I was more capable than I really was. So I tackled something even harder, only to have my true abilities and limitations made glaringly clear. The experience has taught me the value of some of the points Leo makes at 6Changes.com. I understood what he was saying my first time through the material, but I didn’t value them enough to make them a part of my efforts to develop new habits.

Pick One Thing

First, pick one thing to work on. That should be obvious from what I just said, but I want to emphasize this. If your “one thing” requires a blog post to explain how each incremental step is related to your overall habit, you’ve probably got more than one thing. If you find yourself explaining to a spouse or friend how this week’s “incremental step” fits into the overall goal, even if it doesn’t really build on last week’s, you’ve probably got more than one thing. If, after three weeks, you’ve got three separate triggers for the habit you’re developing, you’ve probably got more than one thing.

One thing means that there is one trigger for one simple activity that can be explained in a sentence of ten words or less, preferably without commas in the sentence. Yes, when you pick a habit like this, your weekly incremental steps will seem amazingly insignificant. That’s the point! To develop a habit that will last you need to do the same things over and over and over again.

Build Incrementally

Second, incrementally build the habit from the beginning to the end. My nightly planning session has largely stuck, despite being a few different activities, because I do them all at the same time. One thing I believe I did get right with that habit is that I built it up from start to finish. I didn’t begin with the things I felt would be most beneficial: planning my day, or writing in my journal, or going through my inbox. I began with simply organizing my desk area, putting things where they went. It was kind of hard at the time to start with something so simple, but by slowly building on what had gone before each week, it gave me a set of conditioned responses that are almost hard not to do each night.

If, instead, I had started with the things I most wanted done, and then added other steps before those, I would have broken that set of conditioned responses, possibly each week during the two months I focused on developing that habit. I never would have built up the conditioning that makes it effortless to keep practicing my habit.

Do The Core Quickly

Third, don’t leave the most important steps of your habit to the end. Just as you shouldn’t push the most important incremental steps for developing your habit to the first weeks if that will break up your conditioned responses, so you also shouldn’t leave them to the last weeks. However, the challenge here is a different one: you just won’t have as much time to solidify the habit if you add it at the end of your two months, before you move on to the next habit. The steps I added to my nightly planning in the last two weeks of February are still the weakest part of that habit. Partly, that’s due to the fact that I was trying to do more than one thing. But it’s also because I only did those steps for a week or two before focusing my mind and efforts on my next habit. This leads me to believe the best use for the last couple of weeks of habit development is reinforcement of earlier steps. It may be appropriate to divvy up your habit into 6 incremental steps and then for the last two or three weeks focus on those aspects of your habit that are the weakest.

Physical Actions

Fourth, physical actions are important. The habits and steps that involve physical action, even if it’s just typing on my keyboard, stick better than purely mental, emotional, or spiritual work. The physical actions of my nightly planning habit - sitting down, processing my inbox, typing on my keyboard, and kneeling in prayer - are easy and habitual. I have no problem doing them each day, and just doing them is a great benefit. But I know that my habit would be much more beneficial if I could really change and improve the mental and emotional aspects of the habit - the analysis of the past day, truly examining and consciously choosing what to do with my inbox items, as well as the plan for the next days activities, deeply communicating with God in prayer without being distracted by other thoughts, etc. Those changes are much harder to make. By their very nature, these mental activities cannot be both habitual and consciously chosen. If they are habitual, they lose their effectiveness.

However, if I could somehow tie them more closely to physical actions I suspect I could improve them more easily. I’d be interested in others thoughts on how to work on these aspects of developing habits.

Checkpoint

Steve Pavlina has said that you can do just about anything for a month. And he’s used that maxim to get him through some interesting changes in habit. But he also reserves the right to drop something if it’s not working out for him. I like this idea, and I feel that it would be a good addition to my current attempts to develop habits over the course of two months. I can take a step back after the first month and consider how the habit is helping and whether I want to keep it up. For most habits, especially the ones I’m working on initially, I suspect that the answers will be yes, but giving myself that kind of leeway can make it easier to start something more risky, something I’m not sure I want to commit to as a lifelong habit, even if I later decide to.

Exercise

So lets apply this learning to my next habit: exercise. My exercise of choice has always been running, and I recently got some Vibram Five Fingers to see if they would help me with some foot pain I’ve had off and on over the last few years. Overall, I think just wearing them on my commute is helping a little. It’s time to start running in them, and see how that goes. One of the things that probably contributed to past injuries was trying to run each day. So, although I’d like to work on my habit daily, I know I shouldn’t be running each day. And besides, I’d also like to develop some strength, by doing simple strength training (i.e. no weights, just stuff like situps and pushups). I can set aside 20-30 minutes for this each day, and possibly a little more on Saturday’s, though it will be hard to make Saturday’s workout follow a standard trigger.

For the pushups I’ll follow the program at http://hundredpushups.com, to develop a habit of doing them regularly. I’ll do the pushups on Monday, Wednesday, and Friday. On Tuesday, Thursday, and Saturday I’ll be running, initially very short distances. The pushups program by it’s nature is gradual, so I’m not worried about that. The running program I’ll follow will also be simple, and very gradual since I’m getting used to the Vibram Five Fingers:

1. Dress to run, then shower
2. Dress to run, Run a block and back, then shower
3. Do a full 1/4 mile warmup
4. Run 1/2 mile
5. Run 1/2 mile, sprint all out for block or two
6. Run 3/4 mile, sprint all out in the middle
7. Run a full mile, sprint all out in the middle
8. Reinforcement
9. Reinforcement

Finally, I’ll want to do the checkpoint at one month just to decide how the running is going with the VFF’s and also because I’ve had some minor pain in one of my arms, and may need to stop the pushups. Overall, I’ m excited to get started on this one.

Well, if converting the trunk was a necessary appetizer, and converting branches was the meat and potatoes of Subversion to Mercurial conversion, then surely converting the tags is the dessert. We now understand all of the basics of converting the trunk and branches of a Subversion repository to Mercurial. All we need to do is pull over the tags that label specific revisions. Before we dive into dessert, let’s admire it a little by examining how tags are stored in both Subversion and Mercurial.

What Are Tags?

First, in Subversion, because revisions are stored as a set of pointers to the files that make up a revision, then a tag is just a set of pointers to the files at specific states in their history. This is done be making a copy of the file pointers we want in the tag at another location in the repository directory structure. O course, because it uses this generic way of creating tags, then tags might not only be tags. They could be branches, if further changes were made to the copied files. Additionally, a tag need not include all of the files in the trunk or in a branch.

Mercurial, on the other hand, records a tag as just a name associated with a revision id. That revision id specifies the entire state of the repository at that time, so by definition a tag includes all files in a repository. Additionally, tags in Mercurial are versioned, as any other file is. This means that your repository won’t know about tags created in another repository unless you’ve pulled the changes that create those tags, even though you may have the revisions specified by the tags.

Generically

As you can imagine, these fundamental differences make converting tags a very ambiguous process. Not any more ambiguous than converting branches, of course, but not any less either. The generic part of the algorithm that hg convert uses for converting tags, or the work done no matter what the type of the source destination, is fairly straightforward. The converter asks the source converter object for the tags, as a simple dictionary from the tag name to the revision id. The source revision ids are then mapped to destination revision ids, as long as that revision wasn’t skipped in the conversion process, e.g. by a file mapping that excluded it. The destination converter object then records the tags in the destination repository. The converter then updates the revision map, so that future conversions parent new children to the revision that creates the tags, rather than its parent. This avoids the problem of branching to create the tags. It also means that all tags are created after all other revisions have been converted into the new repository, which may seem odd. Unfortunately, it’s the least ambiguous way to convert the tags.

Finding the tags

Of course, this high-level description abstracts away the real work - getting the tags from the Subversion repository and putting them into the Mercurial repository. So let’s dive into the Subversion side first. Because of the way tags are recorded in Subversion repositories, as well as all of the non-tag things you can do with and to the files there, this is a rather tricky process, that can easily miss tags if the Subversion repository is in any way unconventional. Additionally, the algorithm may change if someone comes up with better heuristics for determining the tags in a Subversion repository. This comment before the algorithm for getting tags sums things up well:

# svn tags are just a convention, project branches left in a
# 'tags' directory. There is no other relationship than
# ancestry, which is expensive to discover and makes them hard
# to update incrementally.  Worse, past revisions may be
# referenced by tags far away in the future, requiring a deep
# history traversal on every calculation.  Current code
# performs a single backward traversal, tracking moves within
# the tags directory (tag renaming) and recording a new tag
# everytime a project is copied from outside the tags
# directory. It also lists deleted tags, this behaviour may
# change in the future.

The Subversion converter object goes through the svn log from the latest revision to the revision specified as the start revision in the converter config section (defaults to 0). From each log entry, it finds changes that are copies from one location to another, and sorts those copies from more specific to more general. Then it looks to see if the most generic copy is actually copying the tags directory itself. If so, we make sure that as we continue backward through the log we start looking in the old tags directory, rather than where it was moved to. Then, for each copy we check to see if the file(s) copied are copied into the tags directory.

After that check, we then look through all the copies seen so far, to see if the current copy is just a rename of an existing tag. If so, we update our record of the tag. The subversion converter object then checks each of the files added in the revision against the pending tags. If any file added creates a tag that tags files from different branches of the repository—i.e. files from the trunk as well as files from a branch—then it is discarded, since it cannot be represented in Mercurial.

Finally the source converter object goes through each pending tag, and determines the name. If the tag is a rename of another tag, it leaves it in the pending list, and continues to the next tag. Next it gets the revision id for the tag, by looking at the source revision from which the tag was created. Finally, if it hasn’t yet been added to the official tags list, it is.

Tagging the new repository

The work to put these tags into the new repository is much simpler. But first, one limitation of the tags conversion code is that all converted tags must be placed in a single cloned branch. This limitation exists because tags are not converted in line with other revisions, so the challenge of ensuring that changes to the .hgtags file are in all the right cloned branch repos is a tricky one. Each cloned branch could determine which tags apply to it, but then each cloned branch would have a different set of changes to the .hgtags file at its tip after the conversion, all of which would have to be merged when doing merges between these cloned branches.

So first, the destination converter object determines in which repository to place the updated tags. This is necessary if the –clonebranches option was specified on the command line, otherwise there is only one repository to put them in. It then loads up the old tags from the .hgtags file, and creates the full list of entries to be placed in that file from the tags dictionary retrieved from Subversion. If no new tags have been created, then it returns. If some have, then it saves the full list of entries to the file and commits the changes to the repository.

Summing Up

Now we’ve gone over all the basics of converting a repository from Subversion to Mercurial: the trunk, the branches, and the tags. But we’ve just barely touched on the many different tricks of converting repositories, and cleaning them up after the fact. Tools like svnadmin dump with dumpfilters, the mq extension to Mercurial, hg histedit, the hgsubversion extension, not to mention the possibilities with going through another VCS in the process, such as Git, all offer possibilities worth exploring when you run into issues with conversion. Though I don’t have concrete plans for writing about each of these, I will occasionally share tips and tricks as I learn about them. In the meantime, happy coding!

After reading about how the hg convert extension can convert the trunk of a Subversion repository to Mercurial, you’re probably thinking: “But we have more than just a single line of development! We branch our code! We merge it! We tie it into knots! It’s like a great monster!” Of course it is. You wouldn’t be good developers if it weren’t. If a simple trunk is like a single snake, unbroken from head to tail, then any actively developed repository is more like the Hydra, with more heads then you can count, and poisonous breath besides. Even worse, the Hydra is immortal, and so cannot be killed - sounds like a legacy codebase to me! Heracles couldn’t kill the Hydra, but he could bring it under submission, and put it to his own uses.

So lets look at how hg convert can take the branches in a Subversion repository and bring them into a Mercurial repository, thus taming the beast.

Finding the Hydra’s Heads

Getting the head revision when we were just dealing with the trunk was easy - just find the latest revision under the trunk’s path. But now we’re dealing with the Hydra - we need to keep track of many heads, one for each branch. The source converter object (remember that one? It’s Subversion specific) does this work. After getting the head of the trunk, it then lists the contents of the branches directory. This directory is either detected as a child of the source url passed into the hg convert command, or it is specified in the convert configuration section. Typically that’s done on the command line using the –config parameter. So the source converter object lists the children of the branches directory. For each child, it checks if it’s a directory, and finds the latest revision in that directory. As long as the latest revision wasn’t the one that created it (i.e. it’s a branch with no changes in it), then it adds that latest revision to the set of heads.

As with the trunk, we then need to follow the parents of each head back to track down all of the revisions that need to be included in the convert. And as with the trunk, if hg convert has been run before against the same source, then we only track back till we find changes that have already been converted into the destination repository.

Sorting Things Out

The biggest challenge in fighting the Hydra isn’t the poisonous breath - Heracles overcame that with a simple cloth over his mouth and nose. No, it’s that when you cut off one of it’s heads, it grows back two more. So one of the first things you need to know is whether it’s heads will grow back in parallel or one after the other. When converting Subversion repositories to Mercurial, this concept corresponds to the sort order.

Now that we’re going to be converting multiple branches, the order in which we sort the changes to be imported is important. The hg convert extension offers three types of sorts: branchsort, datesort, and sourcesort. We can ignore sourcesort, because that only applies when importing from a Mercurial repository. The default for Subversion repositories is branchsort. This means that when importing from Subversion, the algorithm essentially sorts them as a depth first search. It imports one branch all the way to its head revision, then goes back and imports the next branch. In other words, the Hydra grows back one head, then grows a second. This is in contrast to datesort, which imports each revision in date order, or rather, the hydra’s heads grow in parallel. Datesort is more intuitive, and leads to repositories that are organized the way we expect them to be, with development going on in the trunk and in branches at the same time. Branchsort, however, actually creates smaller Mercurial repositories, because the diffs to files tend to be much smaller when they’re organized by branch, rather than intermingled.

Given that disk space is cheap, you will be happiest if you specify datesort, and only rerun hg convert with branchsort if you experience any problems with the resulting repository size.

Hacking at the Hydra

Heracles defeated the Hydra by cutting off it’s heads, then having his nephew cauterize the wounds so that new heads would not grow back. Finally he placed it’s one remaining immortal head under a heavy rock, trapping it.

The trick to defeating the Hydra is to limit the number of heads you’re dealing with. Unfortunately, normal conversions from Subversion to Mercurial will often leave more heads than expected. Remember, this is the Hydra - we should expect more heads than expected. In this case, it is because the convert extension cannot recognize Subversion merges . Doing so is a tricky problem because Subversion merges are so flexible. So after the conversion, merges don’t produce a nice revision with two parents. Rather, it looks like any other revision with one parent, while the other parent is left dangling. Because Subversion branches are often closed when they are merged back into trunk development, this means that we’re leaving an extra head in the converted repository that need not be there. Even if the Subversion branch was used further after the merge, we’ve still lost an important piece of history by not recording the revision as a merge in the new repository. Fortunately, the convert extension does provide a manual workaround to this limitation: the splicemap. Like Heracles firebrand wielding nephew, the splicemap can safely eliminate the Hydra’s heads. The splicemap does this by allowing you to specify the parents of any given revision.

As you might imagine, you can easily shoot yourself in the foot with this ability. You could rearrange revisions in any number of ways, creating an odd tree, or switching revisions around in ways that significantly increase the size of the converted repository. But rather than hacking up and grafting together a totally new Hydra, let’s just use it to get rid of a few of the hydra’s heads. It’s most obvious benefit is to specify the two parents of any merge operation, in most cases eliminating one head from the converted repository. It can also be used to bring together two disparate lines of development, which may occasionally be useful, e.g. when you realize that two separate repositories should really be combined into one.

The hg convert extensions implements the splicemap using a simple lookup of revision based on the ids specified. Then it replaces the parents on a commit that has been retrieved from the source converter object, before having the destination converter object put the commit into the destination repository.

One trick to using the splicemap is understanding the revision format used in the splicemap file. For subversion repositories, it is important to get this right, or it will be as if the splicemap hadn’t even been specified. A subversion repository has its revision in the splicemap formatted like so: svn:/path/to/module@branching in Mercurial guide.

If you decide to just create named branches in the destination repository, the source converter object records the branch that a given revision is on, and the destination converter object creates a commit with that named branch. Nothing too spectacular here.

Creating cloned branches, where there is a separate Mercurial repository for each Subversion branch, takes a little more work. For you, the little more work is just to specify –clonebranches on the command line. For the converter, it needs to make sure that each revision goes into the right repository. First, when copying each revision from the source to the destination, the converter object finds the branches that a revisions parents are on. It then tells the destination converter object which branch the child revision is on, as well as the branches that the parent revisions are on. The destination convert object first sets the correct repository to commit the revision to. If it doesn’t exist yet (i.e. the revision is the first in this branch), then the destination repository is created. Then it needs to make sure that the destination repository has all of the revisions leading up to the one being copied. So it pulls all of the appropriate revisions in from each of the parents branches. Finally it is ready to commit the child revision to the appropriate branch repository.

Cleaning up the mess

Fighting the Hydra can be messy. Here are some things we can do to clean up once we’re done. One thing to note when doing Subversion to Mercurial conversions is that you’ll want to eliminate empty revisions. Typically, these occur because the subversion revision either only changes subversion properties (and not files), or because it only creates the directory at the root of the trunk or one of the branches. They can also occur if a filemap is specified. But if it isn’t, then the hg convert extension doesn’t try to eliminate empty revisions. So the rule when doing conversions should be to always specify  a filemap file, even if you just leave it empty. This will make sure that the hg convert extension still tries to eliminate empty revisions.

You may also want to eliminate branches from the history, preserving only the merge commit. The easiest way to do this is to not use the splicemap to merge the branch into development, and then strip that branch from the mercurial repository after the conversion is done. The trunk will still have the proper changes from the revision that performed the merge, but all of the history of how that revision was created in the branch will be gone.

Victorious

Hopefully, that’s enough information to both understand how conversion of branches works, as well as to successfully convert the unique repositories you’ve got on hand. Once you’re done you’ll still have to deal with the immortal Hydra, i.e. all your existing code. But hopefully you’ve made things more manageable along the way, making it easier to leverage all the goodness in that code. Like Heracles, it should now be possible to go forth and conquer other monsters using the Hydra’s venomous poison.