IO.select hangs forever reading device files on older linux kernels

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: 0.5.4
  • Fix Version/s: 0.5.6, 0.6.4
  • Component/s: Block Device
  • Labels:
    None
  • Environment:

    Linux localhost.localdomain 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux, though other kernels of similar vintage are probably similarly affected

Description

The block device plugin has code that evaluates to this:

File.read("/sys/block/md0/size")

This uses a select() call internally:

[root@localhost ~]# strace -p 8000
Process 8000 attached - interrupt to quit
select(4, [3], [], [], NULL <unfinished ...>


lsof -p 8000
COMMAND  PID USER   FD   TYPE DEVICE     SIZE    NODE NAME
...
ruby    8000 root    3r   REG    0,0     4096    6104 /sys/block/md0/size

but the call to select() hangs forever (even though it should not):

irb(main):001:0> f=File.open("/sys/block/md0/size")
=> #<File:/sys/block/md0/size>
irb(main):002:0> IO.select([f], nil, nil, nil)
# hangs forever

So this plugin should be modified to use read_nonblock, which does not hang.

Issue Links

relates to

Bug - A problem which impairs or prevents the functions of the product. OHAI-366 Fix for older linux kernels and blocking read of /proc/* files in virtualization plugin

  • Unknown - Issue untriaged, priority unassigned or questioned
  • Open - The issue is open and ready for the assignee to start work on it.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Daniel DeLeo added a comment -

See also: http://gist.github.com/441278 for code to repro the root bug outside of ruby.

Show
Daniel DeLeo added a comment - See also: http://gist.github.com/441278 for code to repro the root bug outside of ruby.
Hide
Permalink
Ian Meyer added a comment -

From the linux kernel 2.6.30 changelog: 

commit 1af3557abdef34ee036a6de4cb79e24468544b8d
Author: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Date:   Thu Apr 9 13:53:22 2009 +0900

    sysfs: sysfs poll keep the poll rule of regular file.
    
    Currently, following test programs don't finished.
    
    % ruby -e '
    Thread.new { sleep }
    File.read("/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies")
    '
    
    strace expose the reason.
    
    ...
    open("/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies", O_RDONLY|O_LARGEFILE) = 3
    ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbf9fa6b8) = -1 ENOTTY (Inappropriate ioctl for device)
    fstat64(3, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
    _llseek(3, 0, [0], SEEK_CUR)            = 0
    select(4, [3], NULL, NULL, NULL)        = 1 (in [3])
    read(3, "1400000 1300000 1200000 1100000 1"..., 4096) = 62
    select(4, [3], NULL, NULL, NULL
    
    
    Because Ruby (the scripting language) VM assume select system-call
    against regular file don't block.  it because SUSv3 says "Regular files
    shall always poll TRUE for reading and writing".  see
    http://www.opengroup.org/onlinepubs/009695399/functions/poll.html it
    seems valid assumption.
    
    But sysfs_poll() don't keep this rule although sysfs file can read and
    write always.
    
    This patch restore proper poll behavior to sysfs.
    /sys/block/md*/md/sync_action polling application and another sysfs
    updating sensitive application still can use POLLERR and POLLPRI.
    
    Cc: Neil Brown <neilb@suse.de>
    Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
    Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
Show
Ian Meyer added a comment - From the linux kernel 2.6.30 changelog: commit 1af3557abdef34ee036a6de4cb79e24468544b8d Author: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> Date: Thu Apr 9 13:53:22 2009 +0900 sysfs: sysfs poll keep the poll rule of regular file. Currently, following test programs don't finished. % ruby -e ' Thread . new { sleep } File.read( "/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies" ) ' strace expose the reason. ... open( "/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies" , O_RDONLY|O_LARGEFILE) = 3 ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbf9fa6b8) = -1 ENOTTY (Inappropriate ioctl for device) fstat64(3, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0 _llseek(3, 0, [0], SEEK_CUR) = 0 select(4, [3], NULL, NULL, NULL) = 1 (in [3]) read(3, "1400000 1300000 1200000 1100000 1" ..., 4096) = 62 select(4, [3], NULL, NULL, NULL Because Ruby (the scripting language) VM assume select system-call against regular file don't block. it because SUSv3 says "Regular files shall always poll TRUE for reading and writing". see http: //www.opengroup.org/onlinepubs/009695399/functions/poll.html it seems valid assumption. But sysfs_poll() don't keep this rule although sysfs file can read and write always. This patch restore proper poll behavior to sysfs. /sys/block/md*/md/sync_action polling application and another sysfs updating sensitive application still can use POLLERR and POLLPRI. Cc: Neil Brown <neilb@suse.de> Signed-off-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>
Hide
Permalink
Daniel DeLeo added a comment -

patch verified on CentOS 5.4 and 5.3, merged and pushed.

Thanks to everyone who helped on this!

Show
Daniel DeLeo added a comment - patch verified on CentOS 5.4 and 5.3, merged and pushed. Thanks to everyone who helped on this!
Hide
Permalink
Ian Meyer added a comment -

Filed bug with redhat: https://bugzilla.redhat.com/show_bug.cgi?id=604887

Show
Ian Meyer added a comment - Filed bug with redhat: https://bugzilla.redhat.com/show_bug.cgi?id=604887
Hide
Permalink
Daniel DeLeo added a comment -

Imeyer filed a bug report upstream: https://bugzilla.redhat.com/show_bug.cgi?id=604887

Show
Daniel DeLeo added a comment - Imeyer filed a bug report upstream: https://bugzilla.redhat.com/show_bug.cgi?id=604887
Hide
Permalink
Daniel DeLeo added a comment -

This issue has regressed. The offending code is at the bottom of the linux/filesystem command, where it reads /proc/mounts.

FWIW, the original fix for this issue should be applicable to this regression, see commit 518d56a6cb7d021b47ed3d691ecf7fba7f74a6a7

Show
Daniel DeLeo added a comment - This issue has regressed. The offending code is at the bottom of the linux/filesystem command, where it reads /proc/mounts. FWIW, the original fix for this issue should be applicable to this regression, see commit 518d56a6cb7d021b47ed3d691ecf7fba7f74a6a7
Hide
Permalink
Bryan McLellan [Opscode] added a comment -

I had a system that was reproducing this and switching the /proc/mount read to non_blocking solved it. I've merged this to opscode/master.

Show
Bryan McLellan [Opscode] added a comment - I had a system that was reproducing this and switching the /proc/mount read to non_blocking solved it. I've merged this to opscode/master.

People

  • Assignee:
    Daniel DeLeo
    Reporter:
    Daniel DeLeo
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: