fe:ff:ff... arp entry not a reliable way of detecting ec2

Ohai detects ec2 by looking for an arp entry with a mac address of fe:ff:ff:ff:ff:ff. However, if the server has not received any traffic recently (which is the case with some dev/test servers we have), then this arp entry disappears from the cache.

This doesn't show up when running ohai manually, as you are connected to the server via ssh, keeping the arp entry alive. It only showed up for me by having chef-client running as a daemon, and on some hosts, the ec2 information would disappear when running knife node list. Those nodes also showed nothing when I ran 'knife node show --long nodename | grep arp'.

To verify that the arp entry was really disappearing, I ran the following in a screen session:

while true; do date; arp; echo =====; sleep 300; done > ohai_bug.log

and after leaving the machine alone for a while, the arp command showed no output. The arp entry would reappear on each chef-client run, due to network activity caused by chef itself, but not before ohai had run the network plugin.

The effect of this is that there is intermittently no ec2/cloud entries for the nodes, and no public_ipv4 address available for the node, which is breaking our load balancer configurations that use chef to generate the load balancer configuration.

I am unsure as to what a good fix would be, except to perhaps ping the default route before running the network plugin, or changing the method of detecting ec2.