Chef Should Load Cookbooks In Dependency Order

Chef holds a philosophy that you should be able to reason about order. Some objects are not being loaded in a consistent order such as attributes in CHEF-2903, and we should fix that. However, we differed philosophically from other projects that use topological sorts from the start because of order. As an increasing number of cookbooks are added, ensuring there is only one solution to the topographical sort is difficult and expensive for the user. The alternative is that the load order has many solutions and thus is not consistent across sequential runs.

Order matters, but we should not give up order to add order.

Some history exists on the mailing list, such as here.

One option would be loading cookbook segments in the order of the expanded run list first. This would give the user the ability to ensure a definition or LWRP was loaded before using it on another cookbook. However, there could be situations where you're adding a noop recipe to the runlist for this purpose.

I think we could walk cookbook.metadata.dependencies in Chef::RunContext#foreach_cookbook_load_segment. I've thought about this and played around a little and I'm not sure. This doesn't work yet, but I've got to run for the day: https://gist.github.com/5332542a89c6ca14b825

A number of people in IRC have run into just this issue. I opened this ticket in response to someone trying to do the exact scenario illustrated above. They were writing a cookbook, and wanted to write a provider class that would subclass a provider class from another cookbook. But there was no way for them to ensure that the base class provider would be loaded first, even if they marked the second cookbook as a dependency of the first.

The only way to reason about order correctly is by means of declared dependencies. B declares a dependency on A means A gets loaded first and then B gets loaded. These are the orders that cookbooks *care about*. If there were any other orders that cookbooks cared about, then it would be their responsibility to mark it in their metadatas. Dependencies (and topological sort), is the only way for cookbooks to ask for (and receive) any constraints on the load order. Without this, there is no way for cookbooks to ask for or receive any constraints on their own load orders. Without this, there is no way for a cookbook to be guaranteed that it is loaded after another cookbook; therefore there is no way for a cookbook to depend on another cookbook. Thus this violates half of the premise of depends. (The other half is that, by the time recipes are running, the whole DAG is downloaded and loaded.)

The cookbook load order is probably not deterministic right now anyway. Any total order that currently exists is mere happenstance (there are some non-happenstance constraints to that total order, but the total order isn't guaranteed). Nothing in chef-client specifically sorts the cookbooks in any particular way. So far as I can tell, sorting, if any, is done at the server in computing the DAG of cookbooks to deliver to the client based on the run-list and env constraints.

Doing this in foreach_cookbook_load_segment will not work correctly because, for example, libraries in one cookbook may need to reference lightweight providers in another, but the libraries would be loaded before the lightweight providers they depend on, crashing the chef-client run with a NameError: uninitialized constant.

Topological sort is really easy to do: Ruby 1.8 ships with a stdlib tsort. All you need is that first example on the page. Indeed, Librarian-Chef uses that technique almost exactly, with a slight variation so as not to pollute the global class Hash.

I agree we need to work out some issues load order of cookbook segments. Along with related ordering issues, we'll talk more about engineering a solution in the short term at Opscode.

The Chef server does not calculate a DAG. The issue we hold with a DAG is that order is not guaranteed. Unless you have a fixed path between all the cookbooks, as you increase the number of cookbooks the number of possible solutions increases and so does the likelihood that one Chef run will be in a different order than the one preceding it.

Consider an oversimplified example, that we have three cookbooks, A, B, and C. Cookbook B depends on A, while A and C have no dependencies. There are two possible solutions:

A, B, C
C, A, B

Now lets state that cookbook B has a side-affect that causes cookbook C to fail. However, this only happens in one of the solutions, where B is run before C. Sometimes your run will succeed, other times it will fail.

If there were any other orders that cookbooks cared about, then it would be their responsibility to mark it in their metadatas.

Consider another tool where instead of just cookbooks having dependencies, each individual resource has dependencies. The apache2 cookbook has ~57 resources [1] (although you're not going to use all of them). It isn't hard to get into the thousands of resources, as we've done in the past. In this situation, it is likely that the topological sort will produce a solution that will cause a resource to fail because of ordering, and consequently runs will occasionally fail. Additionally, that failure will be difficult to reproduce because you can't reason about the order without a visualization of the DAG, and even then it is hard. It is easy to say, "it is your responsibility to declare all of your dependencies," but this doesn't solve the users problem.

Because many of the cookbooks in an expanded run list won't have dependencies at all, or insufficient dependencies to form a directed path, each run is likely to be different.

We currently handle recipe dependencies by 1) requiring the other cookbook as a dependency in the metadata so we have it and 2) using include_recipe to ensure that the recipe has already been run, and if it hasn't, to run it.

I would expect that we could use the same solution with cookbooks in the cookbook loader. We could consider loading all the segments for a dependent cookbook, but sooner or later you'll find an edge case that any reasonably simple solution fails for.

I'd like to see a set of cookbooks where a library requires an LWRP from another cookbook. I suspect this is possibly an edge case that could be solved by fixing the parameters of the cookbooks to fit the [future] loader, rather than vice versa.

[1]

for r in `grep resource_name ~/src/chef/chef/lib/chef/resource/* | awk -F: '{ print $3}'` ; do grep "^$r \"\| $r \"" ~/devel/opscode-cookbooks/apache2/recipes/* ; done | awk -F: '{print $2}' | sed -e "s/^\s*//" | sort | uniq | wc

There is a key distinction to be made between, on the one hand, running recipes and the ordering of resources and, on the other hand, loading cookbooks. The question of ordering, and the question of determinism of the ordering (ie, the question of partial vs total ordering), can have one answer in the one case and another answer in the other case. In the case of recipes, there is a clear total order; in the case of resources in the resources-collection, there is a clear total order. But in the case of cookbooks, there is no clear total order, but we have an extremely clear, indisputable partial order which I think should be enforced. Although some cookbooks may not actually declare all their dependencies, that's a bug and it can be reported and fixed.

Without libraries and lightweight resources and providers, there wouldn't really be an issue.

But these things are not recipe DSL. They are much more like gems than they are like recipes. Either a library needs to be able to Kernel#require another cookbook's library or lightweight resource or povider, or the cookbooks loader needs to load cookbooks in dependency order.

The following is an example of how a library file might require a resource in another cookbook:

require_lwrp_resource "other-cookbook::zany-resource"

class MyCoolSubclassResource < Chef::Resource::ZanyResource
  def initialize(*)
    super
    @action ||= [:zaniness]
  end
end

In this example, the dependencies are not declarative, in the metadata, but are computed at runtime by actually running the files and responding to require calls. This will achieve the same purpose, but I do not think it is necessarily as clean. But I think this is also a necessary feature. Indeed, within one cookbook, one library file may actually depend on another library file - see fnichol/chef-rvm. If that library file could write require_library "chef_rvm_shell_helpers" at the top, the silliness would be solved.

I think the approach I took with CHEF-2903 works for LWRPs and libraries as well. Basically, you load cookbooks in run_list order, but load dependencies first. By applying dependency loading recursively, you get the feature you want, namely, you can reuse code from a cookbook as long as you specify a dependency in the metadata.

We'll probably need to also add the equivalent of include_attribute for other filetypes just in case.

Patch here: https://github.com/opscode/chef/pull/508/files

• https://github.com/opscode/chef/pull/508
• https://github.com/opscode/chef/commit/713e23b85ef27ebb775dfa1386f040b197cbc18e

Above patch addresses the original intent of this ticket, "load cookbooks in dependency order".

The order for cookbook compilation is now:

• Libraries
• Attributes
• LWRPs
• Resource Definitions
• Recipes
  (this is the same as before)

For each type of file, the cookbooks are ordered by:

• Iterate over the run list
• Recurse through dependencies, placing them ahead of the cookbook that depends on them
• If a cookbook is already in the cookbook order list, ignore it (removes duplicates and prevents loops from circular deps)

For each cookbook, files are sorted and then loaded in sort order (except attributes files, which load default.rb first).

See also the code and comments in run_context/cookbook_compiler.

Note that the above patch does not provide a means to force one file to load before another (e.g., an equivalent of include_attribute for other types). There will be a follow up ticket for this.

This was released in Chef 11.