Feature Toggles from a Continuous Delivery Perspective

So what is really a feature toggle?

Feature toggles as a software development concept let you pursue parallel, concurrent feature development as an alternative to branching for parallel development (also called feature branches). Feature toggles are sometimes called feature flags, feature switches, feature flippers or conditional features. Feature toggles let you continuously integrate features while they’re under development. You can use feature toggles to hide, disable or enable individual features during runtime.

As with all software development techniques, you would use feature toggles in conjunction with version control. Using feature toggles by themselves doesn’t imply eliminating all branches from a comprehensive version control plan. One distinction of feature toggles is that all changes are checked into the main branch (mainline) instead of a development branch.

Feature toggle is basically an “if” statement in your code that is part of your standard code flow. If the toggle is “on” (the “if” statement == true) then the code is executed, and if the toggle is off then the code is not executed.
Every new feature you add to your system has to be wrapped in a feature toggle. This way developers can check-in unfinished code, as long as it compiles, that will never get executed until you change the toggle to “on”. If you design your code correctly you will see that in most cases you will only have ONE spot in your code for a specific feature toggle “if” statement.

Feature toggles do not have to be Booleans. You can have feature toggles that are Enumerations, Strings, integers or any other value, as long as it fits your flow. For instance let’s say you want to migrate from one database to another. During the migration process the feature toggle can have 4 values:

  1. read from the old database;
  2. read from old database and fallback to new database;
  3. read from new database and fallback to old database;
  4. use new database only.

Internal Testing

A feature is disabled or hidden from all users until you start feature development. During development, you can enable the feature for development and unit testing, and disable it for all other users. Quality Assurance (QA) testers can also enable features they want to test. Until the feature is complete and fully tested according to the Definition of Done (DoD) and passes quality tests, it will be hidden or disabled in the released software. You can dynamically change the value of a feature toggle without creating and deploying a new build.

Cleaning feature toggles

Feature Toggles should not exist forever in your code. After a feature is opened and you decide there is no need to change the toggle back to inactive, developers should go back to the code and remove the “if” statement from the code, cleaning it from unnecessary if statements and unused toggles. A Feature toggle manager can help you with that since it shows you all the features and how long they are active. Also you can do a quick search in the code to find the places feature toggles are used.

Using feature toggles for quick feedback

There is another great advantage to Feature toggles. Not only do they use to check-in unfinished code, they also used for getting fast feedback.

Once a developer thinks a feature is finished, or even half-finished you can use the feature toggle to expose the new feature for internal testing or for getting quick feedback from product managers. For instance you can have in your “if” statement that checks if the user who is logged-in has certain privileges; or works at your company, or even equals to a specific user id, then the new feature is open and operational. This way product managers can experience the new feature on production, on a live system very soon, even if it is not completed, and give the developer quick feedback about the product. While the product manager sees and experience the new feature, other users are not affected by this, because for them the feature toggle is “closed”.

feature toggle

Advantages and Problems of Feature Flags

Still, there are advantages to developers working this way, making merge problems go away, and eliminating the costs of maintaining and supporting long-lived branches. And carefully using feature flags can help you to reduce deployment risk through canary releases or other incremental release strategies, where you make the new code active for only some users or customers, or only on some systems, and closely check before releasing progressively to the rest of the user base – and turn off the new code if you run into problems. All of this makes it easier to get new code out faster for testing and feedback.

But using feature flags creates new problems of its own.

The plumbing and scaffolding logic to support branching in code becomes a nasty form of technical debt, from the moment each feature switch is introduced. Feature flags make the code more fragile and brittle, harder to test, harder to understand and maintain, harder to support, and less secure.

Feature Flags need to be Short Lived

Abhishek Tiwari does a good job of explaining feature toggles and how they should be used. He makes it clear that they should only be a temporary deployment/release management tool, and describes a disciplined lifecycle that all feature toggles need to follow, from when they are created by development, then turned on by operations, updated if any problems or feedback come up, and finally retired and removed when no longer needed.

Feature toggles require a robust engineering process, solid technical design and a mature toggle life-cycle management. Without these 3 key considerations, use of feature toggles can be counter-productive. Remember the main purpose of toggles is to perform release with minimum risk, once release is complete toggles need to be removed.

Feature Flags are Technical Debt – as soon as you add them

Like other sources of technical debt, feature flags are cheap and easy to add in the short term. But the longer that they are left in the code, the more that they will end up costing you.

Release toggles are supposed to make it easier and safer to push code out. You can push code out only to a limited number of users to start, reducing the impact of problems, or dark launch features incrementally, carefully assessing added performance costs as you turn on some of the logic behind the scenes, or run functions in parallel. And you can roll-back quickly by turning off features or optional behaviour if something goes wrong or if the system comes under too much load.

But as you add options, it can get harder to support and debug the system, keeping track of which flags are in which state in production and test can make it harder to understand and duplicate problems.

And there are dangers in releasing code that is not completely implemented, especially if you are following branching by abstraction and checking in work-in-progress code protected by a feature flag. If the scaffolding code isn’t implemented correctly you could accidentally expose some of this code at run-time with unpredictable results.

…visible or not, you are still deploying code into production that you know for a fact to be buggy, untested, incomplete and quite possibly incompatible with your live data. Your if statements and configuration settings are themselves code which is subject to bugs – and furthermore can only be tested in production. They are also a lot of effort to maintain, making it all too easy to fat-finger something. Accidental exposure is a massive risk that could all too easily result in security vulnerabilities, data corruption or loss of trade secrets. Your features may not be as isolated from each other as you thought you were, and you may end up deploying bugs to your production environment” – James McKay

The support dangers of using – or misusing – feature flags was illustrated by a recent high-profile business failure at a major financial institution. The team used feature flags to contain operational risk when they introduced a new application feature. Unfortunately, they re-purposed a flag which was used by old code (code left in the system even though it hadn’t been used in years).

Due to some operational mistakes in deployment, not all of the servers were successfully updated with the new code, and when the flag was turned on, old code and new code started to run on different computers at the same time doing completely different things with wildly inconsistent and, ultimately business-ending results. By the time that the team figured out what was going wrong, the company had lost millions of $.

As more flags get added, testing of the application becomes harder and more expensive, and can lead to an explosion of combinations: If a is on and b is off and c is on and d is off then… what is supposed to happen? Fowler says that you only need to test the combinations which should reasonably be expected to happen in production, but this demands that everyone involved clearly understand what options could and should be used together – as more flags get added, this gets harder to understand and verify.

And other testing needs to be done to make sure that switches can be turned on and off safely at run-time, and that features are completely and safely encapsulated by the flag settings and that behaviour doesn’t leak out by accident (especially if you are branching in code and releasing work-in-progress code). You also need to test to make sure that the structural changes to introduce the feature toggle do not introduce any regressions, all adding to testing costs and risks.

More feature flags also make it harder to understand how and where to make fixes or changes, especially when you are dealing with long-lived flags and nested options.

And using feature switches can make the system less secure, especially if you are hiding access to features in the UI. Adding a feature can make the attack surface of the application bigger, and hiding features at the UI level (for dark launching) won’t hide these features from bad guys.

Use Feature Flags with Caution

Feature flags are a convenient and flexible way to manage code, and can help you to get changes and fixes out to production more quickly. But if you are going to use flags, do so responsibly:

  • Minimize your use of feature flags for release management, and make the implementation as simple as possible. Martin Fowler explains that it is important to minimize conditional logic to the UI and to entry points in the system. He also emphasises that:

    Release toggles are a useful technique and lots of teams use them. However they should be your last choice when you’re dealing with putting features into production.

    Your first choice should be to break the feature down so you can safely introduce parts of the feature into the product. The advantages of doing this are the same ones as any strategy based on small, frequent releases. You reduce the risk of things going wrong and you get valuable feedback on how users actually use the feature that will improve the enhancements you make later.

  • Review flags often, make sure that you know which flags are on and which are supposed to be on and when features are going to be removed. Create dashboards (so that everyone can easily see the configuration) and health checks – run-time assertions – to make sure that important flags are on or off as appropriate.
  • Once a feature is part of mainline, be ruthless about getting it out of the code base as soon as it isn’t used or needed any more. This means carefully cleaning up the feature flags and all of the code involved, and testing again to make sure that you didn’t break anything when you did this. Don’t leave code in the mainline just in case you might need it again some day. You can always go back and retrieve it from version control if you need to.
  • Recognize and account for the costs of using feature flags, especially long-lived business logic branching in code.

Feature toggles start off simple and easy. They provide you with new options to get changes out faster, and can help reduce the risk of deployment in the short term. But the costs and risks of relying on them too much can add up, especially over the longer term.