Chef Cookbook Patterns

Cookbook Patterns

 

Role Cookbooks

When you start writing your first roles, you tend to define them as a run-list with a collection of attributes. Mostly because that’s the documentation about, but it’s not necessary a good practice, depending very much of your business needs.

Roles in chef are un-versioned.

As soon as you start sharing a single chef server and your roles between multiple environments, you might find yourself in the situation where in order to update the role X’s run-list (the order of cookbooks or a cookbook version) when deploying to environment Y, you’ll soon notice that updating the role will propagate the changes to all the environments, and it’s not really what you want.

You can try to suffix the cookbook version and dynamically fetch the cookbook version for that specific environment, but it turns out to be really ugly and hard to managed. This type of versioning is error prone, requires maintenance and if something goes bad, it’s pretty hard to determine where the bug is, especially if your working with many environments.

But cookbooks are versioned…

Cookbook are the only versioned Chef resources, and they can include other resources which is exactly what we need. By listing the run-list as a succession of include_recipe we can create our own run-list inside a cookbook.

Long story short.. Role Cookbooks!

You can create a cookbook with the same name as the roles, specifying the entire run-list. Let’s take a simple example for a web role:

roles/web.json

cookbooks/web-recipe/recipes/default.rb

cookbooks/web-recipe/metadata.rb

Using this approach still allows us to have a single role web, shared across the environments. However, the cookbook associated with this role can have as many versions as we need, controlling the evolution of the role and the promotion of the role between different environments. And all this by changing the role’s cookbook constrained in the target environment.

Wrapper Cookbooks

Either you’re using community cookbooks, or you’re trying to create a generic cookbook to be included by one of your application’s cookbooks, there is a pretty nice pattern to use here, pretty similar to a proxy. Instead of adding attributes specific for a specific application, you can leave it generic and add an application specific wrapper that includes your generic cookbook. Set all the required attributes in your wrapper cookbook to override the generic cookbook defaults.

Let’s take for example the tomcat cookbook. Instead of forking the community cookbook and override the attributes, making it difficult to keep it up to date with the community repo, create a separate wrapper recipe that set the required attributes and then included the recipe from the original cookbook. This resulted in a layout that looked like the following:

cookbooks/my-service/metadata.rb

 

cookbooks/my-service/recipes/default.rb

Some simple rules for setting attributes need to be followed:

  • Wrapper cookbooks should only ever set attributes using the ‘override’ precedence.
  • Cookbooks should set attributes using the ‘default’ precedence if a wrapper cookbook is allowed to override the attribute.
  • Cookbooks may set attributes using the ‘override’ precedence if they are publishing attribute data for other cookbooks to use but do not expect the other cookbooks to override the attribute data.

By following this rules, we can ensure that we’re also versioning the attributes for a specific recipes, giving us the ability to roll out the attribute changes in any of the development/staging/production environments

Abstract Cookbook

The Abstract Cookbook Pattern consists of two different types of cookbooks.

  • Abstract Cookbook – Contains a resource that defines a set of general actions such as initializing, backing up, and restoring a database, as well as a set of recipes that use these actions. (e.g. db) The default.rb recipe in abstract cookbooks sets up the provider and all provider-specific inputs and installs the client. The install_server.rb recipe sets all generic server inputs, includes the default.rb recipe and installs the server.
  • Concrete Cookbook – Contains Chef providers that perform the actions for a specific variety of abstract cookbooks. (e.g. db_mysql, db_postgres) and a setup_server.rb recipe that sets up server inputs and hard codes the provider name and version of the server.

screen_Concrete-Abstract_v1

Depending on your requirements, the db concrete recipe will include db_mysql or db_postgres, but every system with either MySQL or Postgres installed will include just the concrete db recipe.

Default Cookbook

Each cookbook in RightScale’s repository contains a default.rb recipe, which installs and configures any prerequisite utilities, configuration variables, etc. that are required in order to successfully execute any of the cookbooks resources (e.g. recipes). Before a recipe from a cookbook can be successfully executed, the cookbook’s default.rb recipe should be run first to resolve any cookbook dependencies.

You should use the Default Cookbook Pattern when:

  • your cookbook depends on packages, configurations, or LWRP to be present that are core to your cookbook and/or that other cookbooks might also depend on.
  • your cookbook provides packages, configurations, or Chef resources that other cookbooks may depend on.

Example: You need to use s3_file from the aws cookbook in multiple application recipes. Any time you call s3_file, you need to make sure that the aws gem is also installed in the Chef’s gem list. So every of these cookbooks must include the following:

Instead doing so, which will allow every cookbook developer to define it’s own version of this gem and could actually interfere with other cookbooks’ functionality, we can take use of the Default Cookbook pattern to move this code in the cookbooks/aws/recipes/default.rb file (let’s disregard that this is also a community cookbook, and pretend we’ve write one of our own)

Now, we can just add depends “aws” for every of our cookbook’s metadata that is going to use s3_file and by controlling when we check/install the aws gem we just have to make sure that our application cookbook is running after the aws cookbook default recipe  is executed

 

Cookbook Anti-Patterns

Forking community cookbooks

When finding a community cookbook that you need to use, the first thing that comes into your mind is to fork it and have it locally managed, so you can keep it up-t0-date from time to time. The changes appear to be very small at first, allowing monthly syncs without any conflicts, but as you’re starting to push the cookbook more and more into your business logic, having application specific attributes and flows injected directly in the fork, you’ll soon realise that resolving the conflicts takes more and more time.

Instead of modifying the community cookbook, you can leave it as it is, just to have it’s version freezed and create a wrapper cookbook that handles the application specific attributes or business logic. This way, you’re changes are kept totally separated making it cleaner and easier to debug. Also there are no conflicts and the upgrade is usually done seamless.

P.S. If you’re forking the community cookbooks in order to make some changes and share them back to the community, forking the cookbooks is totally acceptable, as long as your branches are short-lived and you integrate them as soon as the pull request is accepted.

Role attributes

Roles should not have business specific attributes embedded in the json. This is mostly because the role should define an abstract behaviour of your chef-run. Instead, if you need to set some role-specific attributes, by using the Wrapper Cookbooks pattern and setting the application specific variables from there.

 

Closing thoughts

In order to construct great cookbooks, there are some rules you need to follow:

  • Cookbooks must be usable through single inclusion in a run list. That means judicious use of include_recipe and cookbook dependencies.
  • Cookbooks must not require attributes to be set outside of the cookbook itself to function.

Roles are really useful, and they’re a great asset if used correctly. The only thing to do here is to keep them as simple as possible. If you need to share them between environments, Role Cookbooks comes very handy.In an ideal world where you have CI Integration and you actively take use of promotions, following this pattern will allow you to promote also the roles.

If you need to add business logic to a community cookbook, while using Wrapper Cookbooks patter, the chef-rewind gem does an awesome job for monkey-patching the community cookbooks with your code.