The cookbook shadowing feature should either be officially deprecated or well documented with a commitment to support it indefinitely

the resolution here is that cookbook shadowing is going away for real. We may implement a different behavior in its place, such as allowing people to maintain different versions of their cookbooks in different directories, but that is TBD. For now, we need to add deprecation warnings when the shadowing behavior is used.

Will Chef::Config[:cookbook_path] become a string, and throw a deprecation warning if it is an array? Or only if it is an array with a length > 1?

https://github.com/opscode/chef/commit/f1b7f2e965670fd3e50e035fc509972fedb4d1e5

Added deprecation warnings when the same cookbook exists in multiple locations.

Having multiple cookbook paths is not deprecated, and I see no reason that it should be. Only the magical file-wise set union stuff is deprecated.

As for what the new behavior will be in the future, that is TBD. I personally like allowing different versions of cookbooks in different cookbook paths, as this works well with non-git version control systems, or for users who don't want to use git branches to keep their versions separate.

Reopening for fix version change.

0.10.2 was a security release, 0.10.4 is the next maintenance release.

Bulk edit: this ticket should have been previously merged for 0.10.2, but will be released in 0.10.4

Just to provide my two cents: If this functionality were to be deprecated, it would really put a cramp in how I've begun keeping my chef-repo. I really hope that Chef stays non-opinionated in this respect, as I would be frustrated to be forced into a way of working that someone else has deemed "the right way". Having this approach for minor tweaks to established cookbooks (particularly while waiting on pull requests that may or may not be accepted), makes for a much maintainable and new-member-grokable chef-repo, IMHO.

References (in case it helps to have a use-case)
https://github.com/FLOSolutions/fga-chef-repo#readme
github.com/patcon/freight-cooking#readme

https://github.com/patcon/freight-cooking#readme

Just let me know if I should open a new issue to properly address this concern. I know this is a faux pas, but if it makes any difference, I rarely rarely reopen closed issues.

One technique I have used is to make a new cookbook in site-cookbooks to override specific resources in the original cookbook.

This example illustrates how to override the original my.cnf.erb template in in the mysql cookbook with your own custom template.

Create a new cookbook mysql-customizations in site-cookbooks with a single recipe as follows.

site-cookbooks/mysql-customizations/metadata.rb

depends "mysql"

site-cookbooks/mysql-customizations/recipes/default.rb

include_recipe "mysql"

resources(:template => "/etc/my.cnf").instance_exec do
  cookbook "mysql-customization"
end

site-cookbooks/mysql-customizations/templates/default/my.cnf.erb

<%
  # custom custom custom
  # your customized version of my.cnf.erb ...
  # custom custom custom
%>

And then, in your role/run-list, instead of using recipe[mysql], use recipe[mysql-customizations].

How is this going to work with knife solo/chef solo. Those just copy up the files in a deploy step, isn't this totally going to break that?

Jay Feldblum Your approach appears to still rely on the compositing that appears will be removed. When I do the same in the cookbooks/ folder, my template doesn't take precedence - it just results in overwriting the original template each and every run (firing service restarts, etc. which is undesirable).

That's a really clever solution Jay Feldblum, and I'm really appreciative that you shared, as it might be a viable alternative for me, rather than hanging back on a pre-deprecated version of Chef.

But in the bigger picture (directed at the Opscode folks), I'm very much in disagreement that deprecating this is the fair thing to do. The claim is that it's an extra layer of indirection, but it seems like it's the one thing that newcomers to chef can really understand very clearly. I'm trying to share the project and encourage adoption and contribution among people who are JUST meeting Chef and sometimes Ruby. If someone wants to use my tool and it doesn't perform exactly what they want to do the options are currently:

1) explain cookbook merge behavior and ask them to drop in a file.
2) if avoiding merge, teach them how vendor branches work (but I'm not even using this in my project, so irrelevant)
3) if avoiding merge, teach them everything about librarian, then ask them to fork the external repo, and point the Cheffile to their new forked repo.
4) Teach them how to do Jay's suggestion. This includes teaching them all the places a role/recipe might be defined: vagrantfile, cookbooks, roles – so that they can find the thing they need to switch from "mysql" to "mysql-customization". Nevermind that they will have NO IDEA how what they just did affected things.

This is a huge step backward in terms of potential user-friendliness of projects built on chef (when the maintainer doesn't want to force casual users to learn chef).

Please please please reconsider this deprecation. It will seriously harm the ability we have to create projects that ease newcomers into Chef (and at their own leisure, since they shouldn't NEED to know Chef to make simple changes)

I'm probably the only person that tries to reads all the tickets (there are a lot!) and I'm too far behind to engage this topic right now. I recommend you make a mailing list post to get more eyes (and link back here).

I wanted to add a pointer to http://devopsanywhere.blogspot.ca/2012/11/how-to-write-reusable-chef-cookbooks.html which contains a similar (but not identical) approach to Jay Feldblum's idea above. However it's not using the knife cookbook site install feature; I'm wondering if a hybrid approach could be feasible.

I'm sensitive to people like Patrick Connolly picking an approach today only to have it deprecated later; are the Opscode people (Bryan McLellan [Opscode] etc.) firmly behind knife cookbook site install for the indefinite future?

I personally think cookbook shadowing, the feature where cookbook segments (files) in the override directory replace those in the regular cookbook directory is only clear in one direction. It is simple to understand that you can modify a single file of a cookbook by placing a file with the same name in the override directory, but unless your changes are very limited it becomes confusing to tell what is going to happen when you run a cookbook.

If you've override 3 of 8 files in the overrides directory, you're asking to keep a lot in your head without tooling to help. 'knife cookbook site install' uses vendor branches so you can utilize existing git tooling easily, e.g. 'git diff chef-vendor-apache2 master – cookbooks/apache2' I completely grant that at this point you've got to learn more tools, but you're also doing something more complicated.

But lets say you're using shadowing because you want to be able to seamlessly download a new version of an upstream cookbook and continue to override it with your changes. What changed upstream? How do you know that the attributes you're setting in your override file haven't changed names? Point being, you still need to look over the new cookbook, it doesn't just slide in for free (usually).

Why not just edit the files in the cookbook in your local copy and use diff to compare a new download of the upstream cookbook with your cookbook and merge in the updates? It does seem that you're more likely to be looking at your cookbook trying to read it more often than merging it with a new release from upstream anyway.

I will say that we're behind 'knife cookbook site install' as the preferred path for making changes to community cookbooks for your own use (contributing upstream is usually a different story), but we're not in a particular hurry to remove cookbook shadowing. Notably we've been planning on doing it for quite a long time.

A quick comment about "being opinionated": by considering all paths in which a cookbook appears to be the primary plus overlays, we're making things a lot harder for everyone using a version control system that tracks branches via separate directories, which is all of them except git. For example, if you're using svn, and periodically forking off "stable-X" branches for prod while continuing to develop on trunk, you can't have both branches in your cookbook path (for uploading hotfixes to the stable/prod branch while primarily developing on trunk for, e.g., preprod), because knife will just merge them into a probably broken cookbook. So the shadowing/overlay feature as it exists now is making it more difficult for non-git users to have a convenient workflow.

In addition to that, while the overlay system is simple enough to understand conceptually once explained, it's not what most users expect to happen when they have the same cookbook in multiple places in their cookbook_path. Again, the common result here is that users upload a broken cookbook and then see weird behavior from chef-client, which is tracked down to "wrong" files being used, which is then tracked down to "wrong" files on the server, and if the user is lucky enough to punch the right search terms into google, they'll then learn that the root cause is the cookbook overlay feature.

Even people who are intentionally using the overlay feature often have difficulty troubleshooting errors when an unexpected version of a file is unintentionally overlaid in a cookbook--I have heard this from several really smart admins who've been using Chef since 0.6. What makes it difficult to debug is that there's no cookbook on disk that matches what's uploaded to the server, and further (before the deprecation warnings were added), no indication of which files in the composite cookbook came from where. The extra layer of indirection is easy to reason about when things are going well, but for debugging, it increases difficulty quite a bit.

I'll also point out that nothing stops anyone from building overlay-compiler tools on their own once it's removed from knife cookbook upload (you can copypasta the existing code to get started), and if this is the workflow that works best for you (vs. cookbook site install's vendor branches, patch-cookbooks, librarian, berkshelf, etc.) you can keep using it via a knife plugin or other tool. FWIW, most of the problems that caused us to decide to remove it go away as soon as it's an optional and opt-in thing, especially if the compiled version of the cookbook sticks around locally so you can see it in compiled form.

For what its worth, I was able to use a method similar to the above post by Jay Feldblum to override both the service and default config of haproxy and didn't see duplicate restarts or deprecation warnings(in version Chef: 10.16.2).

I found this posting and tried it out: https://gist.github.com/fujin/1713157

So to be 100% clear, we're using the standard haproxy cookbook: https://github.com/cookbooks/haproxy
And our code is simply the following in OURCOOKBOOK-haproxy/recipes/default.rb, which uses our own template under OURCOOKBOOK-haproxy/templates/default/, and our cookbook depends on haproxy:

include_recipe "haproxy"

# Override the service start protocol from init to upstart
begin
    t = resources("service[haproxy]")
    t.provider Chef::Provider::Service::Upstart
    rescue Chef::Exceptions::ResourceNotFound
    Chef::Log.warn "could not find service haproxy"
end

# Override the main service template with our own
begin
    t = resources("template[/etc/haproxy/haproxy.cfg]")
    t.source "haproxy.cfg.erb"
    t.cookbook "OURCOOKBOOK-haproxy"
    rescue Chef::Exceptions::ResourceNotFound
    Chef::Log.warn "could not find template /etc/haproxy/haproxy.cfg to modify"
end

One additional note. We do not use site-cookbooks. We use librarian chef to store cookbooks in one directory and the rest of ours are in the default cookbooks directory.