File Resource Checksums Files Indiscriminately

Details

  • Type: Bug Bug
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: 11.4.0-client
  • Fix Version/s: 0.9.6
  • Component/s: Chef Client
  • Labels:
    None

Description

Given

file "/tmp/x" do
  owner "root"
  group "root"
  mode 0640
  action :create
end

where /tmp/x is a 2GB file already owned by root:root and mode 0640 - the entire thing will be read into memory to calculate the checksum despite it not being required. Found this was adding 200 seconds to my chef runs on one particular server with a large file.

The solution to me is to only checksum when necessary in the remote_file and template providers.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Matthew Kent added a comment -

Looking for review @ http://github.com/mdkent/chef/tree/CHEF-1260

Only wrinkle in stopping the checksum by default was the set_content/compare_content that rely on it, chose to generate it on demand.

My other solution was to simply remove the 'content' attribute from the file provider/resource completely as a scan of the code and opscode cookbooks shows it's unused. It's also not mentioned in the wiki which points people to template and remote_file for managing file contents. But I'm guessing it was added with something particular in mind.

Show
Matthew Kent added a comment - Looking for review @ http://github.com/mdkent/chef/tree/CHEF-1260 Only wrinkle in stopping the checksum by default was the set_content/compare_content that rely on it, chose to generate it on demand. My other solution was to simply remove the 'content' attribute from the file provider/resource completely as a scan of the code and opscode cookbooks shows it's unused. It's also not mentioned in the wiki which points people to template and remote_file for managing file contents. But I'm guessing it was added with something particular in mind.
Hide
Permalink
Daniel DeLeo added a comment -

This seems reasonable, but I'll want to check if any of the subclasses call compare_content more than once. The other option is to memoize the answer.

Show
Daniel DeLeo added a comment - This seems reasonable, but I'll want to check if any of the subclasses call compare_content more than once. The other option is to memoize the answer.
Hide
Permalink
Matthew Kent added a comment -

I couldn't see anything calling compare_content outside the file provider itself.

Show
Matthew Kent added a comment - I couldn't see anything calling compare_content outside the file provider itself.
Hide
Permalink
Daniel DeLeo added a comment -

Thanks for the patch, Matt, everyone loves speed

  • Dan DeLeo
Show
Daniel DeLeo added a comment - Thanks for the patch, Matt, everyone loves speed Dan DeLeo
Hide
Permalink
Daniel DeLeo added a comment - - edited

need to also patch the template provider to checksum in load_current_resource

oops, spoke too soon, it is there.. but the template provider tests are failing

Show
Daniel DeLeo added a comment - - edited need to also patch the template provider to checksum in load_current_resource oops, spoke too soon, it is there.. but the template provider tests are failing
Hide
Permalink
Daniel DeLeo added a comment -

I believe the problems that caused me to reopen this ticket were fixed.

Show
Daniel DeLeo added a comment - I believe the problems that caused me to reopen this ticket were fixed.
Hide
Permalink
Sean Horn added a comment -

Created a 4GB file with 

dd if=/dev/zero of=/home/vagrant/bojambo bs=1024k count=4000

Wrote a recipe like this

file "/home/vagrant/bojambo"

The file resource took 50+ seconds to execute.

Show
Sean Horn added a comment - Created a 4GB file with dd if =/dev/zero of=/home/vagrant/bojambo bs=1024k count=4000 Wrote a recipe like this file "/home/vagrant/bojambo" The file resource took 50+ seconds to execute.
Hide
Permalink
Sean Horn added a comment -

http://help2.opscode.com/admin.php?pg=request&reqid=14404

Show
Sean Horn added a comment - http://help2.opscode.com/admin.php?pg=request&reqid=14404

People

  • Assignee:
    Daniel DeLeo
    Reporter:
    Matthew Kent
Vote (0)
Watch (1)

Dates

  • Created:
    Updated: