Initial Chef run crashes with Segmentation fault in provider/package/yum.rb

Where is your Ruby coming from? This is likely a Ruby bug. I would recommend first using the 'chef-full' template to use the Omnibus Client chef installation, and second to review CHEF-2413 for possible alternative Ruby rpms.

https://github.com/opscode/chef/blob/master/chef/lib/chef/knife/bootstrap/chef-full.erb

Hi Bryan -

My Ruby is being installed using the bootstrap file which I displayed in the Environment field above. After reviewing that, does your advice still hold?

Jordan

Hi Bryan -

What is most interesting is that this issue never manifested before today. And then I am hitting it 100% of the time.

Jordan

Based on review, it would appear that this is related to CHEF-2413.

This just started happening to me too. I noticed that it's only happening to newly deployed nodes, existing nodes are fine.

I thought it might have been a Chef version issue, since some of my currently running nodes are still on 0.10.2. I updated them to 0.10.8 and could not repro the problem.

This is on Amazon Linux, ami-1b814f72. Both newly deployed and existing instances are running Ruby 1.8.7p357.

It looks like Amazon may have updated the Ruby in their repos. I install ruby-devel on boot, which would have caused a new Ruby to be installed as well if ruby-devel required it.

On existing machine:

[ec2-user@domU-12-31-38-04-65-F2 ~]$ yum list installed | grep ruby
ruby.x86_64 1.8.7.357-1.10.amzn1 @amzn-updates
ruby-devel.x86_64 1.8.7.357-1.10.amzn1 @amzn-updates
ruby-irb.noarch 1.8.7.357-1.10.amzn1 @amzn-updates
ruby-libs.x86_64 1.8.7.357-1.10.amzn1 @amzn-updates
ruby-rdoc.noarch 1.8.7.357-1.10.amzn1 @amzn-updates
rubygems.noarch 1.3.7-1.7.amzn1 @amzn-main

On newly deployed machine:

[ec2-user@ip-10-28-140-3 ~]$ yum list installed | grep ruby
ruby.x86_64 1.8.7.357-1.16.amzn1 @amzn-main
ruby-devel.x86_64 1.8.7.357-1.16.amzn1 @amzn-main
ruby-irb.noarch 1.8.7.357-1.16.amzn1 @amzn-main
ruby-libs.x86_64 1.8.7.357-1.16.amzn1 @amzn-main
ruby-rdoc.noarch 1.8.7.357-1.16.amzn1 @amzn-main
rubygems.noarch 1.3.7-1.7.amzn1 @amzn-main

I think I've identified the root cause. Amazon just released a new version of their Amazon Linux distribution, and they've designed the update process such that older versions of the distro will by default install updated packages for the new version. e.g., 2011.09 will use packages for 2012.03.

It appears that there must be some dependency that does not get updated when updating the ruby package. I experience the segfault if I choose only to install ruby-devel on a 2011.09 image. However, if I install ruby-devel and execute a yum update for all packages, which effectively upgrades a 2011.09 to a 2012.03, the segfault behavior disappears.

Amazon has documentation here on how to change the behavior of yum so that it does not pull packages for later releases: http://aws.amazon.com/amazon-linux-ami/faqs/#lock

I've reported this issue to Amazon: https://forums.aws.amazon.com/thread.jspa?threadID=90714

It looks like I was wrong about a yum update for all packages solving this. It doesn't – I must have been confused about which terminal session I was looking at.

A fresh 2012.03 instance will exhibit this problem, as well as a 2011.09 that has installed the 2012.03 Ruby.

The only work-around right now is using cloud-config to restrict the instance from installing 2012.03 packages.

Old Rubies are hard to keep from segfaulting. I highly recommend using the chef-full template and the Omnibus client.

karmi reported on IRC that Amazon fixed their broken ruby package. Can someone here please confirm so we can close this ticket?

https://forums.aws.amazon.com/thread.jspa?threadID=90714#jive-message-332311

Confirmed, I was able to install Chef and successfully use a Package resource to install a package on ami-f565ba9c, with both Ruby 1.8.7 and 1.9.3.

Not sure if you want to test all AMIs (32/64, EBS/instance store, HPC/non-HPC) before closing.

Thanks, I'll consider that confirmed. Glad that Amazon stepped up and fixed their Ruby.

Yes, Amazon fixed things, but I believe the right thing to do here, now that Amazon has made Ruby 1.9.3 available on Amazon Linux, is to upgrade to it.

Below is a bootstrap file I put together to address this one:

bash -c '
<%= "export http_proxy=\"#{knife_config[:bootstrap_proxy]}\"" if knife_config[:bootstrap_proxy] -%>

if [ ! -f /usr/bin/chef-client ]; then
  yum install -y ruby19 ruby19-devel rubygems19 rubygem19-rdoc rubygem19-rake rubygem19-bigdecimal\
    gcc gcc-c++ libxml2-devel libxslt-devel automake autoconf make
  ln -sf /usr/bin/ruby1.9 /usr/bin/ruby
  ln -sf /usr/bin/gem1.9 /usr/bin/gem
  ln -sf /usr/bin/irb1.9 /usr/bin/irb
  ln -sf /usr/bin/rdoc1.9 /usr/bin/rdoc
  ln -sf /usr/bin/rake1.9 /usr/bin/rake
fi

gem update --system
gem update
gem install bigdecimal
gem install rdoc
gem install rake
gem install yaml --no-rdoc --no-ri --verbose
gem install ohai --no-rdoc --no-ri -v 0.6.10 --verbose
gem install chef --no-rdoc --no-ri -v 0.10.8 --verbose <%= bootstrap_version_string %>
gem install multi_json -v '1.0.3' --no-rdoc --no-ri --verbose
gem install aws-sdk --no-rdoc --no-ri --verbose
gem install fog --no-rdoc --no-ri --verbose

mkdir -p /etc/chef

(
cat <<'EOP'
<%= validation_key %>
EOP
) > /tmp/validation.pem
awk NF /tmp/validation.pem > /etc/chef/validation.pem
rm /tmp/validation.pem

(
cat <<'EOP'
<%= config_content %>
EOP
) > /etc/chef/client.rb

(
cat <<'EOP'
<%= { "run_list" => @run_list }.to_json %>
EOP
) > /etc/chef/first-boot.json

<%= start_chef %>'

This will install Ruby 1.9.3 and then configure it to be the default Ruby on the system. I used this approach vs. the Omnibus installer, because I need to be able do sidewise installs and the Omnibus installer does not yet have a clean way to do this.

If you have extreme paranoia, you may want to change the 'gem install' to use 'gem1.9 install'. If you have no need for Ruby 1.8.7, you also might want to consider performing a 'yum remove ruby ruby-devel'

We install the bigdecimal gem, because aws-sdk depends on it, but that dependency is not called out in the gemspec. Believe this is due to how bigdecimal was handled in Ruby 1.8.7.

The above is based in large part on the AWS Forum Discussion on Ruby and then my working through crap on multiple fronts.