Solr is going into persistent vegetative state (happens for the second time).

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Not A Bug
  • Affects Version/s: 0.8.10
  • Fix Version/s: None
  • Component/s: Chef Server
  • Labels:
    None

Description

After about couple days of chef server running the solr goes into vegetanle state.

Webui does not return node status (node list comes up fine) :
"Could not list status", "You appear to have no nodes - try connecting one, or creating or editing a client"

the chef-server-webui returns 500 error:
2010-04-26_21:52:34.99068 ERROR: 500 "Internal Server Error"
2010-04-26_21:52:34.99072 /usr/lib/ruby/1.8/net/http.rb:2097:in `error!'
2010-04-26_21:52:34.99073 /var/lib/gems/1.8/gems/chef-0.8.10/lib/chef/rest.rb:29
6:in `run_request'
2010-04-26_21:52:34.99074 /var/lib/gems/1.8/gems/chef-0.8.10/lib/chef/rest.rb:10
6:in `get_rest'
2010-04-26_21:52:34.99075 /var/lib/gems/1.8/gems/chef-0.8.10/lib/chef/search/que
ry.rb:37:in `search'
2010-04-26_21:52:34.99076 /var/lib/gems/1.8/gems/chef-0.8.10/lib/chef/node.rb:35
5:in `list'
2010-04-26_21:52:34.99077 /var/lib/gems/1.8/gems/chef-server-webui-0.8.10/app/controllers/status.rb:29:in `index'
2010-04-26_21:52:34.99078 /var/lib/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/controller/abstract_controller.rb:315:in `send'
2010-04-26_21:52:34.99078 /var/lib/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/controller/abstract_controller.rb:315:in `_call_action'
2010-04-26_21:52:34.99079 /var/lib/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/controller/abstract_controller.rb:289:in `_dispatch'
2010-04-26_21:52:34.99080 /var/lib/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/controller/merb_controller.rb:252:in `_dispatch'
2010-04-26_21:52:34.99081 /var/lib/gems/1.8/gems/merb-core-1.0.15/lib/merb-core/dispatch/dispatcher.rb:102:in `dispatch_action'

the last message solr reports before going vegetable:

2010-04-25_18:46:25.04817 Apr 25, 2010 11:46:25 AM org.apache.solr.search.SolrIn
dexSearcher warm
2010-04-25_18:46:25.04822 INFO: autowarming Searcher@1427bca2 main from Searcher
@c278b6c main
2010-04-25_18:46:25.04823 fieldValueCache{lookups=0,hits=0,hitratio=0.00,i nserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0, cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2010-04-25_18:46:25.05905 Apr 25, 2010 11:46:25 AM org.apache.solr.search.SolrIn
dexSearcher warm
2010-04-25_18:46:25.05909 INFO: autowarming result for Searcher@1427bca2 main
2010-04-25_18:46:25.05910 fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}0}
2010-04-25_18:46:25.05911 Apr 25, 2010 11:46:25 AM org.apache.solr.search.SolrIn
dexSearcher warm
2010-04-25_18:46:25.05912 INFO: autowarming Searcher@1427bca2 main from Searcher@c278b6c main
2010-04-25_18:46:25.05913 filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=2,warmupTime=3,cumulative_lookups=114,cumulative_hits=112,cumulative_hitratio=0.98,cumulative_inserts=2,cumulative_evictions=0}
2010-04-25_18:46:25.06243 Apr 25, 2010 11:46:25 AM org.apache.solr.search.SolrIndexSearcher warm
2010-04-25_18:46:25.06246 INFO: autowarming result for Searcher@1427bca2 main
2010-04-25_18:46:25.06247 filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=2,warmupTime=2,cumulative_lookups=114,cumulative_hits=112,cumulative_hitratio=0.98,cumulative_inserts=2,cumulative_evictions=0}
2010-04-25_18:46:25.06248 Apr 25, 2010 11:46:25 AM org.apache.solr.search.SolrIndexSearcher warm
2010-04-25_18:46:25.06249 INFO: autowarming Searcher@1427bca2 main from Searcher@c278b6c main
2010-04-25_18:46:25.06250 queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=4,evictions=0,size=4,warmupTime=1,cumulative_lookups=218,cumulative_hits=214,cumulative_hitratio=0.98,cumulative_inserts=44,cumulative_evictions=0}
2010-04-25_18:46:25.06251 Apr 25, 2010 11:46:25 AM org.apache.solr.search.SolrIndexSearcher warm
2010-04-25_18:46:25.06252 INFO: autowarming result for Searcher@1427bca2 main
2010-04-25_18:46:25.06253 queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=4,evictions=0,size=4,warmupTime=1,cumulative_lookups=218,cumulative_hits=214,cumulative_hitratio=0.98,cumulative_inserts=44,cumulative_evictions=0}
2010-04-25_18:46:25.06254 Apr 25, 2010 11:46:25 AM org.apache.solr.search.SolrIndexSearcher warm
2010-04-25_18:46:25.06256 INFO: autowarming Searcher@1427bca2 main from Searcher
@c278b6c main
2010-04-25_18:46:25.06257 documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=8482,cumulative_hits=4536,cumulative_hitratio=0.53,cumulative_inserts=3946,cumulative_evictions=0}
2010-04-25_18:46:25.06258 Apr 25, 2010 11:46:25 AM org.apache.solr.search.SolrIndexSearcher warm
2010-04-25_18:46:25.06259 INFO: autowarming result for Searcher@1427bca2 main
2010-04-25_18:46:25.06260 documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=8482,cumulative_hits=4536,cumulative_hitratio=0.53,cumulative_inserts=3946,cumulative_evictions=0}
2010-04-25_18:46:25.06654 Apr 25, 2010 11:46:25 AM org.apache.solr.core.SolrCore registerSearcher
2010-04-25_18:46:25.06659 INFO: [] Registered new searcher Searcher@1427bca2 main
2010-04-25_18:46:25.07021 Apr 25, 2010 11:46:25 AM org.apache.solr.search.SolrIndexSearcher close
2010-04-25_18:46:25.07022 INFO: Closing Searcher@c278b6c main
2010-04-25_18:46:25.07023 fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
2010-04-25_18:46:25.07024 filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=2,warmupTime=3,cumulative_lookups=114,cumulative_hits=112,cumulative_hitratio=0.98,cumulative_inserts=2,cumulative_evictions=0}
2010-04-25_18:46:25.07029 queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=4,evictions=0,size=4,warmupTime=1,cumulative_lookups=218,cumulative_hits=214,cumulative_hitratio=0.98,cumulative_inserts=44,cumulative_evictions=0}
2010-04-25_18:46:25.07030 documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=8482,cumulative_hits=4536,cumulative_hitratio=0.53,cumulative_inserts=3946,cumulative_evictions=0}
2010-04-25_18:46:25.07045 Apr 25, 2010 11:46:25 AM org.apache.solr.update.DirectUpdateHandler2 commit
2010-04-25_18:46:25.07046 INFO: end_commit_flush

knife node list works fine,
knife search howevere fails:
knife search node ":"
/usr/lib/ruby/1.8/net/http.rb:2097:in `error!': 500 "Internal Server Error" (Net::HTTPFatalError)
from /var/lib/gems/1.8/gems/chef-0.8.10/lib/chef/rest.rb:296:in `run_request'
from /var/lib/gems/1.8/gems/chef-0.8.10/lib/chef/rest.rb:106:in `get_rest'
from /var/lib/gems/1.8/gems/chef-0.8.10/lib/chef/search/query.rb:37:in `search'
from /var/lib/gems/1.8/gems/chef-0.8.10/lib/chef/knife/search.rb:67:in `run'
from /var/lib/gems/1.8/gems/chef-0.8.10/lib/chef/application/knife.rb:110:in `run'
from /var/lib/gems/1.8/gems/chef-0.8.10/bin/knife:26
from /var/lib/gems/1.8/bin/knife:19:in `load'
from /var/lib/gems/1.8/bin/knife:19

chef indexer says:
2010-04-26_21:52:26.84559 [Mon, 26 Apr 2010 14:48:04 -0700] INFO: Indexing node 0535fc00-c8ec-46f7-a178-3e014d4602f4 from chef status error Connection refused - connect(2)
2010-04-26_21:52:26.84560 [Mon, 26 Apr 2010 14:50:10 -0700] INFO: Indexing node ffc560d1-de6e-4d94-89a3-4ed6a47dc15f from chef status error Connection refused - connect(2)
2010-04-26_21:52:26.84561 [Mon, 26 Apr 2010 14:50:10 -0700] INFO: Indexing node ffc560d1-de6e-4d94-89a3-4ed6a47dc15f from chef status error Connection refused - connect(2)
2010-04-26_21:52:26.84562 [Mon, 26 Apr 2010 14:50:10 -0700] INFO: Indexing node ffc560d1-de6e-4d94-89a3-4ed6a47dc15f from chef status error Connection refused - connect(2)

Everything is working fine after solr restart....

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Eugene A added a comment -

Dupe of http://tickets.opscode.com/browse/CHEF-1143, you can close original ticket.

Show
Eugene A added a comment - Dupe of http://tickets.opscode.com/browse/CHEF-1143 , you can close original ticket.
Hide
Permalink
Josef Dietl added a comment -

I've had an issue with very similar symptoms these last days.
What fixed it for me was this chat: http://community.opscode.com/chat/chef/2012-07-16 (search for "404" to find the relevant conversation between spoon and Adz).

In a nutshell:

"if you're running an old version of the chef-client recipe, or running it without the chef-server recipe, it will change the ownership of /var/log/chef and /var/run/chef"

What worked for me - at least for the past half hour or so - was to temporarily remove the chef-client recipe, ensure the above two directories and all files in them are owned by chef:chef and then re-start the entire VM. (Accordig to Adz, restarting solr should be sufficient, but I could afford to go for the full restart.)

Hope this helps, keep up the good work!

Show
Josef Dietl added a comment - I've had an issue with very similar symptoms these last days. What fixed it for me was this chat: http://community.opscode.com/chat/chef/2012-07-16 (search for "404" to find the relevant conversation between spoon and Adz). In a nutshell: "if you're running an old version of the chef-client recipe, or running it without the chef-server recipe, it will change the ownership of /var/log/chef and /var/run/chef " What worked for me - at least for the past half hour or so - was to temporarily remove the chef-client recipe, ensure the above two directories and all files in them are owned by chef:chef and then re-start the entire VM. (Accordig to Adz, restarting solr should be sufficient, but I could afford to go for the full restart.) Hope this helps, keep up the good work!
Hide
Permalink
Daniel DeLeo added a comment -

Looks like this was figured out and it's a problem with the chef client recipe when run on the server.

Show
Daniel DeLeo added a comment - Looks like this was figured out and it's a problem with the chef client recipe when run on the server.

People

  • Assignee:
    Unassigned
    Reporter:
    Eugene A
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: