chef-solr gets "in `reopen': closed stream (IOError)" on start

Details

  • Type: Bug Bug
  • Status: Open Open
  • Priority: Unknown Unknown
  • Resolution: Unresolved
  • Affects Version/s: 10.12.0
  • Fix Version/s: 10.12.0
  • Component/s: Chef Server
  • Labels:
    None
  • Environment:

    Amazon EC2 medium instance

Description

I got the following stack trace when I tried to run chef-solr.

/home/chef/vendor/bundle/ruby/1.9.1/gems/chef-solr-10.12.0/lib/chef/solr/application/solr.rb:268:in `reopen': closed stream (IOError)
	from /home/chef/vendor/bundle/ruby/1.9.1/gems/chef-solr-10.12.0/lib/chef/solr/application/solr.rb:268:in `close_and_reopen_log_file'
	from /home/chef/vendor/bundle/ruby/1.9.1/gems/chef-solr-10.12.0/lib/chef/solr/application/solr.rb:258:in `block in run_application'
	from /home/chef/vendor/bundle/ruby/1.9.1/gems/chef-solr-10.12.0/lib/chef/solr/application/solr.rb:249:in `chdir'
	from /home/chef/vendor/bundle/ruby/1.9.1/gems/chef-solr-10.12.0/lib/chef/solr/application/solr.rb:249:in `run_application'
	from /home/chef/vendor/bundle/ruby/1.9.1/gems/chef-10.12.0/lib/chef/application.rb:70:in `run'
	from /home/chef/vendor/bundle/ruby/1.9.1/gems/chef-solr-10.12.0/bin/chef-solr:26:in `<top (required)>'
	from /home/chef/vendor/bundle/ruby/1.9.1/bin/chef-solr:23:in `load'
	from /home/chef/vendor/bundle/ruby/1.9.1/bin/chef-solr:23:in `<main>'

The error is raised when I run the following command. (I know it's non-standard and not reproducible, but bear with me.)

RBENV_ROOT=/usr/local/lib/rbenv sudo -E -u chef -- rbenv exec bundle exec chef-solr -c /usr/local/etc/chef/solr.rb -L /usr/local/var/log/chef/solr.log

The important part is that I ran chef-solr on a TTY and not as a daemon, which is reminiscent of a previous ticket. After some digging, I found that STDOUT was closed by this line of code (1), and the subsequent line triggers the error (2). This led me to these lines (#close triggers #method_missing) and finally these lines. Notice that all of the conditions are satisfied, and hence an STDOUT Logger is appended to the Logger list, which eventually brings us back to the stack trace from (1) followed by (2).

I've developed a quick and dirty solution here, which is to modify the STDOUT Logger's metaclass to have a no-op #close method. Another hack is to pipe the result of chef-solr invocation to a file, thus failing the conditional and not creating an STDOUT logger in the first place.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Bryan McLellan [Opscode] added a comment -

Thank you for taking the time to help make Chef better! We require a Contributor License Agreement (CLA) from contributors to help protect all Chef users. It is easy to complete one online. Instructions to do so are on our wiki, which is linked below. Once your account is updated, you will have access to 'resolve' tickets that indicates to the code review team that your contribution is ready for review.

http://wiki.opscode.com/display/chef/How+to+Contribute

Show
Bryan McLellan [Opscode] added a comment - Thank you for taking the time to help make Chef better! We require a Contributor License Agreement (CLA) from contributors to help protect all Chef users. It is easy to complete one online. Instructions to do so are on our wiki, which is linked below. Once your account is updated, you will have access to 'resolve' tickets that indicates to the code review team that your contribution is ready for review. http://wiki.opscode.com/display/chef/How+to+Contribute
Hide
Permalink
Roy Liu added a comment -

Hi, I signed the CLA and got a confirmation email. What's the next step? FYI, I also have another ticket in the pipeline.

Show
Roy Liu added a comment - Hi, I signed the CLA and got a confirmation email. What's the next step? FYI, I also have another ticket in the pipeline.
Hide
Permalink
Bryan McLellan [Opscode] added a comment -

Great. You have new permissions on JIRA now. Click 'resolve issue' and set the resolution to fixed. Then we'll review it in our regular code review meeting (tomorrow) and give you any feedback.

Show
Bryan McLellan [Opscode] added a comment - Great. You have new permissions on JIRA now. Click 'resolve issue' and set the resolution to fixed. Then we'll review it in our regular code review meeting (tomorrow) and give you any feedback.
Hide
Permalink
Bryan McLellan [Opscode] added a comment -

https://github.com/carsomyr/chef/commit/ef33be551be7d2e3642878a7f2abf6bdfce241ee

Show
Bryan McLellan [Opscode] added a comment - https://github.com/carsomyr/chef/commit/ef33be551be7d2e3642878a7f2abf6bdfce241ee
Hide
Permalink
Bryan McLellan [Opscode] added a comment -

We've discussed this and would like a solution that's a little less quick and dirty. There are a decent solutions we've thought of, none are ideal.

1) have chef_solr override configure_logging and include everything but that if block.

2) move the if black to chef/application/client and chef/application/solo

3) move the if block into a mixin in chef/mixin and use #2

Show
Bryan McLellan [Opscode] added a comment - We've discussed this and would like a solution that's a little less quick and dirty. There are a decent solutions we've thought of, none are ideal. 1) have chef_solr override configure_logging and include everything but that if block. 2) move the if black to chef/application/client and chef/application/solo 3) move the if block into a mixin in chef/mixin and use #2
Hide
Permalink
Roy Liu added a comment -

After thinking about it, the monkey patch to the metaclass may actually be the cleanest solution. My concern with the three alternate solutions presented is that they are effectively saying that the logging semantics among chef-{solo, client} and chef-solr are different. While chef-solr does spend a brief moment of its lifecycle in Ruby before exec'ing Java (hence the dance with STDOUT and STDERR reassignment), I believe it should get uniform treatment.

As for the monkey patch, I don't see much controversy in it. After all, STDOUT is a shared resource, and closing a shared resource is a bad idea. Also notice that it's acting only on the metaclass and affects only stdout_logger.

If you don't agree with the above assessment, I can implement solution #3.

Show
Roy Liu added a comment - After thinking about it, the monkey patch to the metaclass may actually be the cleanest solution. My concern with the three alternate solutions presented is that they are effectively saying that the logging semantics among chef-{solo, client} and chef-solr are different. While chef-solr does spend a brief moment of its lifecycle in Ruby before exec'ing Java (hence the dance with STDOUT and STDERR reassignment), I believe it should get uniform treatment. As for the monkey patch, I don't see much controversy in it. After all, STDOUT is a shared resource, and closing a shared resource is a bad idea. Also notice that it's acting only on the metaclass and affects only stdout_logger . If you don't agree with the above assessment, I can implement solution #3.
Hide
Permalink
Roy Liu added a comment -

Have you guys decided on a path to the fix? Like I said, I am ambivalent on any of the three suggested solutions. Just tell me which one is preferred, and I'll try to implement it.

Show
Roy Liu added a comment - Have you guys decided on a path to the fix? Like I said, I am ambivalent on any of the three suggested solutions. Just tell me which one is preferred, and I'll try to implement it.

People

  • Assignee:
    Unassigned
    Reporter:
    Roy Liu
Vote (0)
Watch (0)

Dates

  • Created:
    Updated: