Chef processes freezes randomly on FreeBSD

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Critical Critical
  • Resolution: Cannot Reproduce
  • Affects Version/s: 0.8.8
  • Fix Version/s: None
  • Component/s: Chef Client, Chef Server
  • Labels:
    None
  • Environment:

    FreeBSD 8, Chef 0.8.8

Description

This problem occured with chef-solr-indexer and chef-client.
The ruby process randomly freezes (still in PS, activity in logs stopped).

When I run truss -p <pid> (truss = strace-like for freebsd), the execution resumes.

Here is a lot with chef-client : http://pastie.org/888829

I think this is what happens : http://ph7spot.com/musings/system-timer
Ruby 1.8 use Green threads, and green threads are not guaranteed to be scheduled. So the process freeze. When I run truss, the FreeBSD kernel changes the state of the process, and this wakes up the thread.

Currently i am forced to run truss on all chef process using cron to avoid this problem.

Activity

Ascending order - Click to sort in descending order
Renaud Chaput made changes -
Field Original Value New Value
Component/s Chef Client [ 10000 ]
Component/s Chef Server [ 10002 ]
Hide
Permalink
Renaud Chaput added a comment -

Added a log.

Show
Renaud Chaput added a comment - Added a log.
Renaud Chaput made changes -
Description This problem occured with chef-solr-indexer and chef-client.
The ruby process randomly freezes (still in PS, activity in logs stopped).

When I run truss -p <pid> (truss = strace-like for freebsd), the execution resumes.

I think this is what happens : http://ph7spot.com/musings/system-timer
Ruby 1.8 use Green threads, and green threads are not guaranteed to be scheduled. So the process freeze. When I run truss, the FreeBSD kernel changes the state of the process, and this wakes up the thread.

Currently i am forced to run truss on all chef process using cron to avoid this problem. 		This problem occured with chef-solr-indexer and chef-client.
The ruby process randomly freezes (still in PS, activity in logs stopped).

When I run truss -p <pid> (truss = strace-like for freebsd), the execution resumes.

Here is a lot with chef-client : http://pastie.org/888829

I think this is what happens : http://ph7spot.com/musings/system-timer
Ruby 1.8 use Green threads, and green threads are not guaranteed to be scheduled. So the process freeze. When I run truss, the FreeBSD kernel changes the state of the process, and this wakes up the thread.

Currently i am forced to run truss on all chef process using cron to avoid this problem.
Adam Jacob made changes -
Triage Status [Triaged]
Hide
Permalink
Renaud Chaput added a comment -

I am running Chef 0.8.10 from rubygems (without any additional patches) on a ruby with pthread disabled, and it has not freezed for 3 days.

I will see if this is only luck or it really fixes the problem, and try to report the bug to FreeBSD & Ruby.

Show
Renaud Chaput added a comment - I am running Chef 0.8.10 from rubygems (without any additional patches) on a ruby with pthread disabled, and it has not freezed for 3 days. I will see if this is only luck or it really fixes the problem, and try to report the bug to FreeBSD & Ruby.
Hide
Permalink
Renaud Chaput added a comment -

After 10 days, still working. So this seems to be related to pthread under FreeBSD, will investigate.

Show
Renaud Chaput added a comment - After 10 days, still working. So this seems to be related to pthread under FreeBSD, will investigate.
Hide
Permalink
Jeffrey Hulten added a comment -

Does not appear to still be an issue as the reporter said it was working in 2010 and has not added information since.

Show
Jeffrey Hulten added a comment - Does not appear to still be an issue as the reporter said it was working in 2010 and has not added information since.
Jeffrey Hulten made changes -
Status Open [ 1 ] Closed [ 6 ]
Resolution Cannot Reproduce [ 5 ]
Bryan McLellan [Opscode] made changes -
Workflow jira [ 11576 ] New OSS [ 19164 ]
Bryan McLellan [Opscode] made changes -
Workflow New OSS [ 19164 ] OSS: Product Management [ 27073 ]
Transition Time In Source Status Execution Times Last Executer Last Execution Date
Open Open Closed Closed
940d 13h 23m 1 Jeffrey Hulten 24/Oct/12 9:48 PM

People

  • Assignee:
    Unassigned
    Reporter:
    Renaud Chaput
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: