Bootstrap fails due to FATAL: Net::HTTPServerException: 403 "Forbidden

Details

  • Type: Bug Bug
  • Status: Open Open
  • Priority: Critical Critical
  • Resolution: Unresolved
  • Affects Version/s: 0.10.4
  • Fix Version/s: None
  • Component/s: Knife
  • Environment:

    Chef server on 64 bit ubuntu 10.04 and client vm is on Amazon Linux x86_64 x86_64 x86_64 GNU/Linux

    This happens on VMware instances also

Description

We get this exception while bootstrapping some times. We dont know why it works on most installations and doesnt work on some.

INFO: *** Chef 0.10.4 ***
[Tue, 29 Nov 2011 01:47:42 +0000] INFO: Client key /etc/chef/client.pem is not present - registering
[Tue, 29 Nov 2011 01:47:44 +0000] INFO: HTTP Request Returned 500 Internal Server Error: Connection refused - connect(2)
[Tue, 29 Nov 2011 01:47:44 +0000] ERROR: Server returned error for http://localhost:4000/clients, retrying 1/5 in 4s
[Tue, 29 Nov 2011 01:47:48 +0000] INFO: HTTP Request Returned 409 Conflict: Client already exists
[Tue, 29 Nov 2011 01:47:48 +0000] INFO: HTTP Request Returned 403 Forbidden: You are not allowed to take this action.
[Tue, 29 Nov 2011 01:47:48 +0000] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[Tue, 29 Nov 2011 01:47:48 +0000] FATAL: Net::HTTPServerException: 403 "Forbidden

We deleted the client and node already didnt have an entry. While deleting the client it gave following error

but after we did knife client list we didnt see the client

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Ritesh added a comment -

This is the error which we got while deleting. But also observed that the client gets deleted even after throwing this error.

sudo knife client delete ip-10-204-30-83.ec2.internal (on Chef server)

Do you really want to delete ip-10-204-30-83.ec2.internal? (Y/N) y
ERROR: Server returned error for http://chef.adam.hp.com:4000/clients/ip-10-204-30-83.ec2.internal, retrying 1/5 in 4 s
ERROR: the object you are looking for could not be found
Response: Cannot load client ip-10-204-30-83.internal

Uname –a (on ec2)

Linux ip-10-204-30-83 2.6.35.14-97.44.amzn1.x86_64 #1 SMP Mon Oct 24 16:03:08 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

Show
Ritesh added a comment - This is the error which we got while deleting. But also observed that the client gets deleted even after throwing this error. sudo knife client delete ip-10-204-30-83.ec2.internal (on Chef server) Do you really want to delete ip-10-204-30-83.ec2.internal? (Y/N) y ERROR: Server returned error for http://chef.adam.hp.com:4000/clients/ip-10-204-30-83.ec2.internal , retrying 1/5 in 4 s ERROR: the object you are looking for could not be found Response: Cannot load client ip-10-204-30-83.internal Uname –a (on ec2) Linux ip-10-204-30-83 2.6.35.14-97.44.amzn1.x86_64 #1 SMP Mon Oct 24 16:03:08 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux
Hide
Permalink
Sakchai Siripanyawuth added a comment -

I am having the same issue

Show
Sakchai Siripanyawuth added a comment - I am having the same issue
Hide
Permalink
Udomsak Chundang added a comment -

How to reproduce bug? VMware with Ubuntu 10.04 install some of machine ( 5 box ) 1 so pass but other so fail.

Show
Udomsak Chundang added a comment - How to reproduce bug? VMware with Ubuntu 10.04 install some of machine ( 5 box ) 1 so pass but other so fail.
Hide
Permalink
Ritesh added a comment -

if i remember this issue correctly .. if you change the hostname of the chef server you might be able to reproduce this ....
we were not able to determine why this happened

Show
Ritesh added a comment - if i remember this issue correctly .. if you change the hostname of the chef server you might be able to reproduce this .... we were not able to determine why this happened
Hide
Permalink
Christophe Arguel added a comment -

It seems it is due to rabbitmq.
Whenever the hostname change, rabbitmq creates a new mnesia database on server startup.

Therefore, it is necessary to to reconfigure RabbitMQ with the right vhost, username/password, permissions.

Have a look to http://lists.opscode.com/sympa/arc/chef/2012-01/msg00016.html and https://github.com/opscode/cookbooks/blob/master/chef-server/recipes/rabbitmq.rb#L56-69.

Hope it helps.

Show
Christophe Arguel added a comment - It seems it is due to rabbitmq. Whenever the hostname change, rabbitmq creates a new mnesia database on server startup. Therefore, it is necessary to to reconfigure RabbitMQ with the right vhost, username/password, permissions. Have a look to http://lists.opscode.com/sympa/arc/chef/2012-01/msg00016.html and https://github.com/opscode/cookbooks/blob/master/chef-server/recipes/rabbitmq.rb#L56-69 . Hope it helps.

People

  • Assignee:
    Unassigned
    Reporter:
    Ritesh
Vote (1)
Watch (2)

Dates

  • Created:
    Updated: