Test & Release Control

“Our contractor just gave us a merge request for X, Y and Z. We need to release feature X and Z. Feature Y failed testing and it is holding up the rest from being released!”

Have you been in this situation before?  You are responsible for a release that has several features, and one of them did not pass UAT or other testing. The business wants to release the other features because fixing the one will take too long, or it has been aborted completely. You may be using a version control system (VCS). Even so, you may have found that trying to pluck the feature out for release made a big mess.

Leading to this issue, it is all too common to make a single branch in version control for a ‘sprint’ or a single project or SOW.  If one feature or bug fix fails, the whole thing is held up.  Even worse, if this ‘atomic change’ (as in bomb) was merged to master/trunk the whole train would be derailed. That is, nothing would be able to move forward until the culprit is fixed.  This is a great way to put yourself at the mercy of a contractor.

Changing the way things are developed can be challenging, however the rewards go beyond customer satisfaction. Getting a handle on this situation can bring personal satisfaction back to your job. Take a look at the diagram and I will introduce you to

The ‘Feature Branch’ Method

version control drawing

Chances are you may already be doing this. If you are, it may be good to reflect on the benefits.

The basic idea is to give each feature, task or ticket its own branch.  This allows for concise testing and uncluttered code audits.  This also gives you a measure of control over when to release the feature. It also adds the bonus of ignoring the feature if need be. You can also be free to experiment and prototype.

To keep the diagram simple, I left out some possible complexities.  The easiest way to circumvent VCS complexity is to architect the code to isolate change.  With such an architecture you may use multiple repositories. Multiple repositories moves some complexity to deployment, but if you are using CI/CD the automation could be made to addresses it.

Using Your VCS

Most modern version control systems can handle this method quite well if they handle merging well. Why is merging so important? Why not use a VCS that is able to remove features from code? Because only a programmer can decide what code belongs to a feature. They need to specify this to any person not acquainted with coding or any tools being used. Once they have done this, most modern VCS are able to address the rest of the issue.

All verson control systems require coders to act in a certain way to get the most out of them.  There are a few general guidelines that will help keep you from basic common issues related to verson control systems. The Feature Branch method outlined here work best if you operate under a couple of specific philosophies, or guidelines.


The first thing I will mention should be thought of as a rule and not a guideline: Only a programmer should merge.  Even if GitHub says you can merge without conflict, the outcome isn’t always so clean. If the person doing the merge is not familiar and responsible for the code, there will be issues.  This means your IT or QA person, however skillful at coding, should not be clicking the button.  Also, the person merging should be testing before they push the merge. This may seem menial and obvious, but you would be surprised if you think so.

Also, you will notice the term “other activity” on the continuity line in the diagram. This is there to remind you to always act as though there have been other changes to the branch you are merging into. This will make your activity and results consistent so you can focus on the code.

The Mainline Model

The first guideline that will lend to your sanity is to keep a dedicated branch for continuity.  This is sometimes referred to as The Mainline Model. This makes it possible for all kinds of changes to be performed and pushed to the central repository without throwing stumbling blocks for the rest of the team. Managing continuity is a hallmark of enterprise systems built to last.

Gelatin Model

Next use the Gelatin Model (sometimes referred to as the Tofu Scale).  If you think of your favorite wiggly desert molded into a mound of glistening goodness. If you poak or shake it, the top giggles more than the bottom.  It could be said that the top is less stable. From here on out we will build on this base assumption : top to bottom = risky to stable.

To build on that, I will mention the idea of ‘merge down, copy up’. This is really useful when branching from branches which is common in branching models other than Mainline.  This practice emerged from the difficulty of backporting changes using merging. I will save this discussion for another day, but if you are curious, I encourage you to investigate further.

In the diagram notice the Branching above the mainline. This is considered less stable as well as out of continuity.  To make it stable and part of continuity, it must be merged into the mainline.  Further, to stabilize for production, you can not freeze the mainline because it is needed to stabilize other branches. Once ready for release branch or tag.

This last step is important and you should already be doing it.  When it comes to bugs and hotfixes, if you kept track of the code you released, you can make a branch (if it isn’t already), make the fix, test and deploy. Then you can worry about getting it back into continuity. Fixing the issue further down the timeline may require different changes to fix the issue, and you don’t want new features holding up hotfixes.

bonus practices

There are two bonus practices that can make life more sane. First, consider git squash.  Squashing your changes compresses them all into one changeset so the person merging can see exactly what is going to change.

The second is to take advantage of pull requests.  This adds a little process flow to your lifecycle, and a checkpoint to control the merges. This is also recorded in the history.  If you are not using Git, your VCS may have similar functionality by different terminology. I encourage you to leverage it.


Depending on how you are testing now, this may add testing points. The truth is that there are no shortcuts when it comes to testing. The baseline for testing when using a ‘Feature Branch’ approach is: test the branch and then test continuity after it is merged and before it is pushed/committed.  This also means that the merges to mainline are very controlled, tested and tagged.

“Breathe. It’s just software, we’re not saving babies here.” – SCOTT HANSELMAN

When you first start this process, it will seem like more testing and merging, more work. However, this will result in less fire-drills, less nagging customers and more bandwidth to focus on what matters. You probably not saving babies, so the quantity of testing is up to you.

Also, the more changes that are made to continuity since a branch is made, the more difficult it will be to merge back into continuity. That is, a branch or pull request can go “stale”. If your code does not lend itself to compartmentalized code changes, letting a branch get too far out of date can cause headaches when it comes time to merge. The person managing the branch should get a feel for when they should be synchronizing with continuity (that is from mainline). Some VCS offer other options of dealing with this. However, it would be best to keep changes small and iterative so they don’t hang for long periods. This is better for reporting purposes and more agile, but accept that in reality there will be troublesome features. On a long-living branch synchronizing from mainline may need to be done on a regular basis.

The sad truth is some VCS simply do not handle merging well.  You can get better tools for fixing merge conflicts and diffs, but ultimately if you are on one of these systems you may need to change. If you haven’t noticed, the industry standard is Git. It is free as in beer and speech, so your only excuse is not learning it.


If your features are each on their own branch, they can be included in releases when they are ready. This can let you release more often to defuse go-fever. This also exposes testing points to prevent sketchy code from holding up releases. These may also help keep contractors in check.

Feature branching makes me happy. It is the best way I have found to manage collaboration on code. It took some effort to get into the habit of branching and merging often. However, the benefits include a smoother flowing deployment cadence, cleaner code in production and better customer satisfaction.


Test & Release Control

RIA – Flex and Java

Reading articles around I have come to notice that certain buzz words that have exact definitions seem to be obscured by those who use those terms inaccurately, usually for a marketing twist.  When I say “come to notice” I really mean that they have gotten irritating to me.  I can’t say I am perfect at using all words exactly according to their meaning. However, there are a few major buzz words floating around out there that I would like to consider.  The two that stick out most to me are Rich Internet Applications (RIA) and Representational State Transfer (REST).  Of coarse this isn’t the first time people have missuesed buzz words, or justified missuse through over simplification.  Do you remember Ajax?  Are you old enough to remember DHTML?

The term REST is one that I misunderstood for a while, and thus used improperly.  The mistake I made is trusting where I learned about the term without doing the research myself.  I attended several taks about REST and I mistook those speakers for athorities on the subject.  Eventyally I came to appreciate that the term was coined in 2000 by Rou Fielding who has a much richer history with the World Wide Web than anyone else who spoke about the subject.  I have to thank a mainstay of PHP, Ben Ramsey, for being the REST evangelest that he is and pointing that out.  If you are interested in learning more the Febuary edition of php|architect contains a great article “Grokking the REST Architectural Style” that is a great overview of REST.

Somehting else I would like to share is that architectural terms are never constrained to a vendor.  REST does not mean PHP or Javascript and RIA does not mean FLEX or Java.  This is just like Coke is not the only cola and Klenx is not the only tissue.

RIA – Flex and Java