TL;DR: I hate the CHEF-3694 warning, so I made a cookbook to get rid of it. YMMV.
Resource cloning in Chef is a bit of a minefield. They have a ticket known as CHEF-3694 saying that the feature should be removed, and indicating that it will be by the time Chef 12.0.0 comes out. However, a lot of their Opscode-developed community cookbooks use (abuse?) resource cloning. The result is that you get tons of warnings about resource cloning that look like this:
1 2 3
Where I come from, it’s considered an error to have a warning in your output. Ignorable warnings bury important ones. So…for better or worse, I embarked upon a journey to see what I could do to use resources correctly and avoid these warnings…
What is resource cloning and why are you warning me about it?
The discussion about this this issue is an interesting read. You should be able to, for example, declare a service resource in one spot of your recipe, and later start it. You should also be able to have multiple recipes be able to install the same package resource and have it be idempotent, without having to worry about coordinating between cookbooks. That’s Chef’s job. To support this, Chef uses a technique they call resource cloning, which spews out warning messages because they plan to get rid of it. Proponents of the warning messages argue that if your cookbook relies on resource cloning, then you are doing something incorrectly and you have bigger problems. However, there are popular community cookbooks that won’t work without it.
Here’s an example of stock
iptables cookbooks causing this problem:
I really wouldn’t want
iptables to have to coordinate between each other in order to avoid this warning. Perhaps there is a way to re-order them? Not that I could figure out…at least not without either making dangerous assumptions or editing stock community cookbook code.
Even so…there are cookbooks that have this problem by themselves without the help of other cookbooks. For example, one of the most popular cookbooks,
Can we just remove resource cloning?
Since it was well-argued that cookbooks shouldn’t rely on resource cloning, and that it would be removed in a future version of Chef, I decided to replace it myself with resource duplication. Resource cloning and its associated warning messages are handled in a method called
Chef::Resource::load_prior_resources, so I just monkey-patched out that method to allow the duplicate resource without copying over any of the existing resources’s attributes, using a bit of code like this:
1 2 3 4 5 6 7 8
NOPE! That doesn’t work. While this works for some of my cookbooks and certain resources, the community
apache2 recipes clearly rely on the soon-to-be-deprecated resource cloning behavior. These recipes define the
service[apache2] resource a number of times to do things like enable/start/restart after config changes. Without resource cloning, the
apache2::logrotate recipe, for example, fails to process a restart of the apache2 service because it didn’t inherit the necessary attributes that needed to be cloned from the original service definition, giving errors like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
Then how about reusing the existing resource?
I think resource reuse is probably the intention in 99% of the use cases. Some commenters on this discussion have suggested making all their recipes look up the resource in the resources collection first, and using the existing one if possible, otherwise handling the not-found exception and creating the new resource. Not a bad suggestion…but there’s no way I’m going to modify every community cookbook to do that.
As an experiment, I tried simply overriding the
service DSL method (which is actually implemented in
method_missing) to test this theory, with some monkey-patching like this:
1 2 3 4 5 6 7 8 9 10 11 12 13
That’s close, but it doesn’t quite work. The most noticeable failure with this is that only the last
action will be run. So for example, say you have something like this:
1 2 3 4 5 6 7 8 9
Normally, this creates two
service[apache2] resources, each copying its configuration from the previous definition, and overriding the action(s). When executed, you’d end up with both actions being executed (but with a bunch of warnings that you’re using the dreaded resource cloning).
With the reuse technique above, the problem is that, in this simple example, the
action: start overwrites the
action: enable. In the end, you have your service started…but
chkconfig shows that it was never enabled. This can obviously be much worse in more complex scenarios.
The Workaround: resource merging
My workaround for this takes advantage of internal knowledge of how the
action DSL method works…and it only applies to that one method. We’re in dark magic territory, so I am sure this could potentially break somebody’s cookbooks.
Building on the resource reuse attempt above, I made it so that instead of letting the
action of a resource stomp over the pre-existing resource’s action, it would merge the actions together. In the over-simplified version, this looks like replacing the single
instance_eval line from above with code like this:
1 2 3 4 5 6
Putting it together
There are a few details I glossed over, such as managing the
:nothing action, different default actions for different types of resources, actions that are Arrays vs Symbols, etc. My final solution was to extend
Chef::DSL::Recipe with a
reusable_resource method that could be used by specific resource DSL overrides as much or as little as you want. Here’s what that looks like:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
With that, if you only wanted to override the default behavior for
service resources, you could monkey-patch those in like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Now all those warnings are gone. My complete initial install works great without complaint. So do my subsequent re-runs.
I’ve packaged this all up as a cookbook that has nothing but a library applying these monkey-patches. You can grab it from GitHub and put it at the front of your run_list with
This technique will probably fail in scenarios where you want to have multiple resources with the same name that have differing attributes other than
action. For example, two different
bash resources in two different places, with two different
command scripts, with the same name. Either resource cloning or resource duplication would work…but resource merging the way I’ve implemented it is going to crash and burn. Of course you can simply name these resources differently, but given that resources share a global namespace, there’s always a risk unless you make sure to prefix your resource names with something uniquely yours.
This is why I factored this technique into a
reusable_resource DSL method. You can use it directly in custom cookbooks if you want. You can override specific types of resources as I have shown in the example, only touching
service. Or, you can override those with additional logic to only do in in narrower cases (e.g., only if there is no block given). That’s up to you. Your Mileage May Vary.
I welcome any and all discussion on this. Especially from someone who knows the internals of Chef much more deeply than I do, who can tell me if I’m getting myself into too much trouble here.
I’m hoping that some day there is a proper mechanism for resource reuse, when that is what is intended, or perhaps some way to detect if two resources internals are the same and make a smart decision about whether to reuse or duplicate. Maybe a real resource merging solution could happen, where the entire blocks are chained and executed? Or perhaps we’ll see some resource namespace solution (though that would not have solved any of the issues I’ve had).