winrm commands need to exit cleanly or they leave the WINRM http service in a bad state

All well and good that we have a bad hostname, but that shouldn't kill all the other connections that are in the middle of executing, sometimes leaving their remote windows service in a bad state.

chris@chefserver:~/chef$ prodwinrm os:windows interactive -V -V
DEBUG: Using configuration from /home/chris/chef/.chef/knife.rb
DEBUG: Signing the request as production-admin
DEBUG: Sending HTTP Request via GET to chefserver.local:4000/search/node
DEBUG: Adding 192.168.128.156
DEBUG: :session => :init
DEBUG: Adding 192.168.129.126
DEBUG: Adding 192.168.129.152
DEBUG: Adding 192.168.129.148
DEBUG: Adding 192.168.129.129
DEBUG: Adding 192.168.128.124
DEBUG: Adding 192.168.129.124
DEBUG: Adding 192.168.128.123
DEBUG: Adding 192.168.128.127
DEBUG: Adding 192.168.129.125
DEBUG: Adding 192.168.128.155
DEBUG: Adding 192.168.129.153
DEBUG: Adding 192.168.128.4
DEBUG: Adding 192.168.129.46
DEBUG: Adding 192.168.128.153
DEBUG: Adding fe80::6891:8766:39f6:3364
DEBUG: Adding 192.168.129.149
DEBUG: Adding 192.168.129.122
DEBUG: Adding 192.168.129.8
DEBUG: Adding 192.168.129.156
DEBUG: Adding 192.168.128.154
DEBUG: Adding 192.168.129.128
DEBUG: Adding 192.168.128.125
DEBUG: Adding 192.168.129.4
DEBUG: Adding 192.168.128.9
DEBUG: Adding 192.168.129.155
DEBUG: Adding 192.168.129.9
DEBUG: Adding 192.168.129.127
DEBUG: Adding 192.168.128.122
DEBUG: Adding 192.168.129.10
DEBUG: Adding 192.168.129.123
DEBUG: Adding fe80::6cc5:35b3:2990:ab5b
DEBUG: Adding 192.168.128.149
DEBUG: Adding 192.168.128.128
DEBUG: Adding 192.168.128.126
DEBUG: Adding 192.168.128.152
DEBUG: Adding 192.168.128.129
DEBUG: Adding 192.168.128.10
DEBUG: Adding 192.168.129.154
DEBUG: Adding 192.168.128.148
Connected to 192.168.128.156, 192.168.129.126, 192.168.129.152, 192.168.129.148, 192.168.129.129, 192.168.128.124, 192.168.129.124, 192.168.128.123, 192.168.12
8.127, 192.168.129.125, 192.168.128.155, 192.168.129.153, 192.168.128.4, 192.168.129.46, 192.168.128.153, fe80::6891:8766:39f6:3364, 192.168.129.149, 192.168.1
29.122, 192.168.129.8, 192.168.129.156, 192.168.128.154, 192.168.129.128, 192.168.128.125, 192.168.129.4, 192.168.128.9, 192.168.129.155, 192.168.129.9, 192.16
8.129.127, 192.168.128.122, 192.168.129.10, 192.168.129.123, fe80::6cc5:35b3:2990:ab5b, 192.168.128.149, 192.168.128.128, 192.168.128.126, 192.168.128.152, 192
.168.128.129, 192.168.128.10, 192.168.129.154 and 192.168.128.148

To run a command on a list of servers, do:
on SERVER1 SERVER2 SERVER3; COMMAND
Example: on latte foamy; echo foobar

To exit interactive mode, use 'quit!'

knife-winrm> hostname
DEBUG: :relay_to_servers => hostname
DEBUG: :relayed => 192.168.128.156
DEBUG: :relayed => 192.168.129.126
DEBUG: :relayed => 192.168.129.152
DEBUG: :relayed => 192.168.129.148
DEBUG: :relayed => 192.168.129.129
DEBUG: :relayed => 192.168.128.124
DEBUG: :relayed => 192.168.129.124
DEBUG: :relayed => 192.168.128.123
DEBUG: :relayed => 192.168.128.127
DEBUG: :relayed => 192.168.129.125
DEBUG: :relayed => 192.168.128.155
DEBUG: :relayed => 192.168.129.153
DEBUG: :relayed => 192.168.128.4
DEBUG: :relayed => 192.168.129.46
DEBUG: :relayed => 192.168.128.153
DEBUG: :relayed => fe80::6891:8766:39f6:3364
DEBUG: :relayed => 192.168.129.149
DEBUG: :relayed => 192.168.129.122
DEBUG: :relayed => 192.168.129.8
DEBUG: :relayed => 192.168.129.156
DEBUG: :relayed => 192.168.128.154
DEBUG: :relayed => 192.168.129.128
DEBUG: :relayed => 192.168.128.125
DEBUG: :relayed => 192.168.129.4
DEBUG: :relayed => 192.168.128.9
DEBUG: :relayed => 192.168.129.155
DEBUG: :relayed => 192.168.129.9
DEBUG: :relayed => 192.168.129.127
DEBUG: :relayed => 192.168.128.122
DEBUG: :relayed => 192.168.129.10
DEBUG: :relayed => 192.168.129.123
DEBUG: :relayed => fe80::6cc5:35b3:2990:ab5b
DEBUG: :relayed => 192.168.128.149
DEBUG: :relayed => 192.168.128.128
DEBUG: :relayed => 192.168.128.126
DEBUG: :relayed => 192.168.128.152
DEBUG: :relayed => 192.168.128.129
DEBUG: :relayed => 192.168.128.10
DEBUG: 192.168.129.122 => :run_command
DEBUG: :relayed => 192.168.128.148
DEBUG: 192.168.129.8 => :run_command
DEBUG: 192.168.129.126 => :run_command
DEBUG: 192.168.129.148 => :run_command
DEBUG: 192.168.129.129 => :run_command
DEBUG: 192.168.128.124 => :run_command
DEBUG: 192.168.129.124 => :run_command
DEBUG: 192.168.128.123 => :run_command
DEBUG: 192.168.128.127 => :run_command
DEBUG: 192.168.129.125 => :run_command
DEBUG: 192.168.128.155 => :run_command
DEBUG: 192.168.129.153 => :run_command
DEBUG: 192.168.128.4 => :run_command
DEBUG: 192.168.129.46 => :run_command
DEBUG: 192.168.129.152 => :run_command
DEBUG: fe80::6891:8766:39f6:3364 => :run_command
/usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/uri/generic.rb:199:in `initialize': the scheme http does not accept registry part: fe80::6891:8766:39f6:33
64:5985 (or bad hostname?) (URI::InvalidURIError)
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/uri/http.rb:78:in `initialize'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/uri/common.rb:177:in `new'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/uri/common.rb:177:in `parse'
from /usr/local/rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/uri/common.rb:628:in `parse'
from /home/chris/chef/.bundle/vendor/ruby/1.9.1/gems/winrm-1.0.5/lib/winrm/http/transport.rb:12:in `initialize'
from /home/chris/chef/.bundle/vendor/ruby/1.9.1/gems/winrm-1.0.5/lib/winrm/http/transport.rb:49:in `initialize'
from /home/chris/chef/.bundle/vendor/ruby/1.9.1/gems/winrm-1.0.5/lib/winrm/winrm_service.rb:29:in `new'
from /home/chris/chef/.bundle/vendor/ruby/1.9.1/gems/winrm-1.0.5/lib/winrm/winrm_service.rb:29:in `initialize'
from /home/chris/chef/.bundle/vendor/ruby/1.9.1/gems/em-winrm-0.5.2/lib/em-winrm/server.rb:79:in `new'
from /home/chris/chef/.bundle/vendor/ruby/1.9.1/gems/em-winrm-0.5.2/lib/em-winrm/server.rb:79:in `client'
from /home/chris/chef/.bundle/vendor/ruby/1.9.1/gems/em-winrm-0.5.2/lib/em-winrm/server.rb:48:in `block (2 levels) in run_command'
from /home/chris/chef/.bundle/vendor/ruby/1.9.1/gems/eventmachine-1.0.0.beta.3/lib/eventmachine.rb:1009:in `call'
from /home/chris/chef/.bundle/vendor/ruby/1.9.1/gems/eventmachine-1.0.0.beta.3/lib/eventmachine.rb:1009:in `block in spawn_threadpool'
chris@chefserver:~/chef$

Maybe we should wrap https://github.com/schisamo/em-winrm/blob/master/lib/em-winrm/server.rb#L48 in a rescue?

Another command that can kill:

knife winrm 'os:windows' 'net user Administrator NEWPASS'

It's understandable that subsequent http requests won't authenticate, but the first one that responds with an error causes the entire app to crash, leaving the ones who are in the middle of the http transaction in a bad state.

I thought about going a bit deeper. I think that a user may want to handle these exceptions in a specific way. I propose that we add an exception channel to to the em-winrm and then allow knife to subscribe to this channel.

https://github.com/pmorton/em-winrm/commit/aa6be927280540bb5c3cd23659bd2c8145dd84df

Thoughts

I'm wondering if we need to trap the TERM signal in the knife winrm subcommand. We could then do graceful cleanup/exit in em-winrm. This could allow us to en all transactions.

I'll need to do some testing but I think calling unbind on the em-winrm session may be the ticket.

For history sake...

[11:41:00 AM PDT] Paul Morton: http://tickets.opscode.com/browse/KNIFE_WINDOWS-23?focusedCommentId=22850#comment-22850
[11:41:11 AM PDT] Paul Morton: Thoughts on the potential fix?
[12:04:00 PM PDT] Seth Chisamore: added a comment. I think a change in 'knife winrm' is all that is needed
[12:04:23 PM PDT] Seth Chisamore: we need to trap the TERM signal (ie that is the signal on a cntrl-C)
[12:04:37 PM PDT] Paul Morton: Ah. Yes
[12:04:38 PM PDT] Seth Chisamore: and then we can call unbind on the em-winrm session
[12:04:58 PM PDT] Paul Morton: Did you put that in commit comments?
[12:05:01 PM PDT] Seth Chisamore: not fully sure how to wire that all in yet though
[12:05:08 PM PDT] Seth Chisamore: I put it on the ticket
[12:05:19 PM PDT] Paul Morton: Thanks
[12:06:29 PM PDT] Paul Morton: The exception channel will default to the current behavior. Meaning that it will re-throw the exception if you did not subscribe to the channel. The additional fix for knife would be to subscribe to the channel and handle the exception
[12:06:46 PM PDT] Seth Chisamore: we are already doing some TRAPs in the base knife class..so not sure how that will play https://github.com/opscode/chef/blob/master/chef/lib/chef/application/knife.rb#L129-137
[12:07:09 PM PDT] Seth Chisamore: yeah that sounds reasonable
[12:07:21 PM PDT] Seth Chisamore: feel free to try the fix in knife winrm
[12:07:29 PM PDT] Paul Morton: I will put together a commit for knife and a commit for em-winrm to address the exception channel.
[12:07:31 PM PDT] Seth Chisamore: would be nice if hippie hacker could test a fix
[12:07:37 PM PDT] Seth Chisamore: since he can recreate it easily
[12:07:42 PM PDT] Paul Morton: I have the exception channel working with knife winrm
[12:07:52 PM PDT] Paul Morton: I will chat with hippiehacker
[12:07:53 PM PDT] Seth Chisamore: yeah thinking about the TRAP bit
[12:08:06 PM PDT] Seth Chisamore: ie...exiting gracefully from an em-winrm session
[12:08:35 PM PDT] Paul Morton: So the trapping CTRL^c Is that new functionality? Or is that something that looks broken in my implementation?
[12:08:57 PM PDT] Seth Chisamore: yeah you need to trap TERM
[12:09:01 PM PDT] Seth Chisamore: ie like this https://github.com/opscode/chef/blob/master/chef/lib/chef/application/knife.rb#L129-137
[12:09:12 PM PDT] Seth Chisamore: or INT forgot which one it is
[12:09:16 PM PDT] Seth Chisamore: really want to trap both
[12:09:32 PM PDT] Seth Chisamore: http://www.ruby-doc.org/core-1.9.3/Signal.html
[12:09:38 PM PDT] Paul Morton: KK. Thanks for taking a look