Test & Release Control

“Our contractor just gave us a merge request for X, Y and Z. We need to release feature X and Z. Feature Y failed testing and it is holding up the rest from being released!”

Have you been in this situation before?  You are responsible for a release that has several features, and one of them did not pass UAT or other testing. The business wants to release the other features because fixing the one will take too long, or it has been aborted completely. You may be using a version control system (VCS). Even so, you may have found that trying to pluck the feature out for release made a big mess.

Leading to this issue, it is all too common to make a single branch in version control for a ‘sprint’ or a single project or SOW.  If one feature or bug fix fails, the whole thing is held up.  Even worse, if this ‘atomic change’ (as in bomb) was merged to master/trunk the whole train would be derailed. That is, nothing would be able to move forward until the culprit is fixed.  This is a great way to put yourself at the mercy of a contractor.

Changing the way things are developed can be challenging, however the rewards go beyond customer satisfaction. Getting a handle on this situation can bring personal satisfaction back to your job. Take a look at the diagram and I will introduce you to

The ‘Feature Branch’ Method

version control drawing

Chances are you may already be doing this. If you are, it may be good to reflect on the benefits.

The basic idea is to give each feature, task or ticket its own branch.  This allows for concise testing and uncluttered code audits.  This also gives you a measure of control over when to release the feature. It also adds the bonus of ignoring the feature if need be. You can also be free to experiment and prototype.

To keep the diagram simple, I left out some possible complexities.  The easiest way to circumvent VCS complexity is to architect the code to isolate change.  With such an architecture you may use multiple repositories. Multiple repositories moves some complexity to deployment, but if you are using CI/CD the automation could be made to addresses it.

Using Your VCS

Most modern version control systems can handle this method quite well if they handle merging well. Why is merging so important? Why not use a VCS that is able to remove features from code? Because only a programmer can decide what code belongs to a feature. They need to specify this to any person not acquainted with coding or any tools being used. Once they have done this, most modern VCS are able to address the rest of the issue.

All verson control systems require coders to act in a certain way to get the most out of them.  There are a few general guidelines that will help keep you from basic common issues related to verson control systems. The Feature Branch method outlined here work best if you operate under a couple of specific philosophies, or guidelines.


The first thing I will mention should be thought of as a rule and not a guideline: Only a programmer should merge.  Even if GitHub says you can merge without conflict, the outcome isn’t always so clean. If the person doing the merge is not familiar and responsible for the code, there will be issues.  This means your IT or QA person, however skillful at coding, should not be clicking the button.  Also, the person merging should be testing before they push the merge. This may seem menial and obvious, but you would be surprised if you think so.

Also, you will notice the term “other activity” on the continuity line in the diagram. This is there to remind you to always act as though there have been other changes to the branch you are merging into. This will make your activity and results consistent so you can focus on the code.

The Mainline Model

The first guideline that will lend to your sanity is to keep a dedicated branch for continuity.  This is sometimes referred to as The Mainline Model. This makes it possible for all kinds of changes to be performed and pushed to the central repository without throwing stumbling blocks for the rest of the team. Managing continuity is a hallmark of enterprise systems built to last.

Gelatin Model

Next use the Gelatin Model (sometimes referred to as the Tofu Scale).  If you think of your favorite wiggly desert molded into a mound of glistening goodness. If you poak or shake it, the top giggles more than the bottom.  It could be said that the top is less stable. From here on out we will build on this base assumption : top to bottom = risky to stable.

To build on that, I will mention the idea of ‘merge down, copy up’. This is really useful when branching from branches which is common in branching models other than Mainline.  This practice emerged from the difficulty of backporting changes using merging. I will save this discussion for another day, but if you are curious, I encourage you to investigate further.

In the diagram notice the Branching above the mainline. This is considered less stable as well as out of continuity.  To make it stable and part of continuity, it must be merged into the mainline.  Further, to stabilize for production, you can not freeze the mainline because it is needed to stabilize other branches. Once ready for release branch or tag.

This last step is important and you should already be doing it.  When it comes to bugs and hotfixes, if you kept track of the code you released, you can make a branch (if it isn’t already), make the fix, test and deploy. Then you can worry about getting it back into continuity. Fixing the issue further down the timeline may require different changes to fix the issue, and you don’t want new features holding up hotfixes.

bonus practices

There are two bonus practices that can make life more sane. First, consider git squash.  Squashing your changes compresses them all into one changeset so the person merging can see exactly what is going to change.

The second is to take advantage of pull requests.  This adds a little process flow to your lifecycle, and a checkpoint to control the merges. This is also recorded in the history.  If you are not using Git, your VCS may have similar functionality by different terminology. I encourage you to leverage it.


Depending on how you are testing now, this may add testing points. The truth is that there are no shortcuts when it comes to testing. The baseline for testing when using a ‘Feature Branch’ approach is: test the branch and then test continuity after it is merged and before it is pushed/committed.  This also means that the merges to mainline are very controlled, tested and tagged.

“Breathe. It’s just software, we’re not saving babies here.” – SCOTT HANSELMAN

When you first start this process, it will seem like more testing and merging, more work. However, this will result in less fire-drills, less nagging customers and more bandwidth to focus on what matters. You probably not saving babies, so the quantity of testing is up to you.

Also, the more changes that are made to continuity since a branch is made, the more difficult it will be to merge back into continuity. That is, a branch or pull request can go “stale”. If your code does not lend itself to compartmentalized code changes, letting a branch get too far out of date can cause headaches when it comes time to merge. The person managing the branch should get a feel for when they should be synchronizing with continuity (that is from mainline). Some VCS offer other options of dealing with this. However, it would be best to keep changes small and iterative so they don’t hang for long periods. This is better for reporting purposes and more agile, but accept that in reality there will be troublesome features. On a long-living branch synchronizing from mainline may need to be done on a regular basis.

The sad truth is some VCS simply do not handle merging well.  You can get better tools for fixing merge conflicts and diffs, but ultimately if you are on one of these systems you may need to change. If you haven’t noticed, the industry standard is Git. It is free as in beer and speech, so your only excuse is not learning it.


If your features are each on their own branch, they can be included in releases when they are ready. This can let you release more often to defuse go-fever. This also exposes testing points to prevent sketchy code from holding up releases. These may also help keep contractors in check.

Feature branching makes me happy. It is the best way I have found to manage collaboration on code. It took some effort to get into the habit of branching and merging often. However, the benefits include a smoother flowing deployment cadence, cleaner code in production and better customer satisfaction.


Test & Release Control

Chain of Application Responsibility: a way of keeping sane

Imagine a legacy application that processes orders.  The application was grown from a small LOB utility.  If you asked the original developer he would be surprised that it is still in use.  The application is made up of nearly 50,000 lines of code with little organization, and no standard has been defined for interactions with other systems.  It was a straight forward application at inception after all. But this means there is no defined place to start a task.  A programmer is reduced to keyword searches in hopes of finding a fringe piece of functionality. That, in some rare case, contains descriptive naming or on the off chance it has comments that match the vocabulary that one might ascribe to the item that needs fixed.

Imagine if you had to find and fix a tax calculation.  What if you were told that someone thought that there may be three or more places where this is done and as you talk to people it quickly becomes apparent that no one really knows where it is happening.  There is some code that sends an email confirmation to the customer that uses a calculation. Also there is some code that puts the order in the database as well as other code that actually charges the customer.  Each one may do its own calculation and they all need to match, or the customer might lose confidence and shop somewhere else.

Addressing the situation requires starting at the most fundamental levels and enhancing up the stack from code to the system level.

The Code

In an architectural void duplicate functionality is free to run rampant.  Analysis can shed light on which system is best suited to ensure an operation is carried out in a safe manner.  However, being unclear about where to expect certain types of operations in the first place stifles analysis.  This hampers problem resolution  and increases technical debt.  Analysis should be repeated periodically to ensure clarity, performance and maintainability.

On the other hand, when the code has a specific purpose, that is to deal with the entirety of a function or a business subject, its functionality can located in a single module or subset of modules.  By assigning responsibility to a module it would make it clear what code belonged and which did not, significantly reducing the amount of code within that module.  Further it would certainly make it easier to find code to enhance. Even in a system loosely organized this way would make it much easier to support.

The System

Rarely is a system grown in a void without communicating with other systems and exchanging data and functionality.  When this is the case it is prudent to analyze each system to assign appropriate responsibility and reduce redundant functionality as much as it makes sense.  When landing new functionality it would also be important to first scrutinize which, if any, existing systems should own that responsibility.


Analysing responsibility at the module and system level not only reduces time investment, it magnifies transparency to the business when enhancement activity must take place.

Doing this takes planning. The tradeoff being that an organized system has more flexibility. Modularising By responsibility could provide a way of assembling chains of application functionality that is not brittle.  This architectural style could also be made to scale horizontally as well if each of the modules were able to be distributed and operations were made asynchronously.

Ultimately placing responsibility in clear meaningful buckets and segregating the code will help keep this programmer sane in the most complicated systems.  Is this how you do it now?  How do you ensure there is no duplication in your system?

Chain of Application Responsibility: a way of keeping sane

RIA – Flex and Java

Reading articles around I have come to notice that certain buzz words that have exact definitions seem to be obscured by those who use those terms inaccurately, usually for a marketing twist.  When I say “come to notice” I really mean that they have gotten irritating to me.  I can’t say I am perfect at using all words exactly according to their meaning. However, there are a few major buzz words floating around out there that I would like to consider.  The two that stick out most to me are Rich Internet Applications (RIA) and Representational State Transfer (REST).  Of coarse this isn’t the first time people have missuesed buzz words, or justified missuse through over simplification.  Do you remember Ajax?  Are you old enough to remember DHTML?

The term REST is one that I misunderstood for a while, and thus used improperly.  The mistake I made is trusting where I learned about the term without doing the research myself.  I attended several taks about REST and I mistook those speakers for athorities on the subject.  Eventyally I came to appreciate that the term was coined in 2000 by Rou Fielding who has a much richer history with the World Wide Web than anyone else who spoke about the subject.  I have to thank a mainstay of PHP, Ben Ramsey, for being the REST evangelest that he is and pointing that out.  If you are interested in learning more the Febuary edition of php|architect contains a great article “Grokking the REST Architectural Style” that is a great overview of REST.

Somehting else I would like to share is that architectural terms are never constrained to a vendor.  REST does not mean PHP or Javascript and RIA does not mean FLEX or Java.  This is just like Coke is not the only cola and Klenx is not the only tissue.

RIA – Flex and Java