beware of solr maxFieldLength option ! It will stop indexing nodes with a lot of attributes

I've got the same problem - my solr server is on debian, and only after this post AND editing file /var/lib/chef/solr/conf/solrconfig.xml and changing i tag mainIndex value maxFieldLength to 100000 (from default 10000) I was able to find ANY windows machine at all!

Completly agree that some warning would be great

+1 on this hit it on my upgrade from 0.9 -> 0.10.2

+1, this mucked up a bunch of stuff on my nagios server that uses search for everything. Bumped up maxFieldLength and ran 'knife index rebuild' to fix.

Note added to http://wiki.opscode.com/display/ChefCN/Search, needs to be amended when this is addressed.

Hello,
Hit the same issue. Search stopped working or was working abnormally. The awk script recommended above showed more than 54000 results. Followed Matthew's suggestion .Bumped up the maxFieldLength to 100000 and then did a 'knife index rebuild'. I have a follow up question as to what is the optimal value that we should set.

Thanks

Added "Fields missing from certain nodes" section mentioned by Matthew Kent above to the Chef Wiki - as he added it to ChefCN (the chinese translation of Chef) and not to the primary Wiki - http://wiki.opscode.com/display/chef/Search. As mentioned, the page will need to be amended when the bug is addressed.

Oh haha, not sure how I ended up there, thanks Tom

Here's some example data from our heavier configs:

server-1 (290 ips) - 17476 fields, data + fields = 647k
server-2 (complicated cluster config) - 13000 fields, data + fields = 385k
server-3 (complicated cluster config) - 15268 fields, data + fields = 471k

Looking at http://wiki.apache.org/solr/SolrPerformanceFactors#Factors_affecting_memory_usage I don't think fields+data are in danger of not fitting in memory any time soon as most of the data points are so small. Would it be safe to change this value to 100k or even 250k?

Didn't hear any negative feedback on the mailing list and given the success of users here I propose raising the limit in https://github.com/mdkent/chef/commits/CHEF-2346

This builds on CHEF-2553

2 more instances of this issue observed on irc

still, what about issuing a warning when the limit has been reached ?

I agree that a warning in the logs is also reasonable, provided there's a straightforward way to do so.
I'll follow up with Seth, et al.

1) This patch does not resolve the problem for users with an existing installation. The change to SCHEMA_VERSION appears to trigger chef-solr to not start due to the differing schema version but wouldn't direct the user with what to do next.

Running chef-solr-installer would work, but then would require a reindex. This is a pretty heavy list of caveats and we're reluctant to do so in a point release.

https://github.com/opscode/chef/blob/485ef3c254179ae2b36ed8aedd44c340a1037a95/chef-solr/lib/chef/solr/application/solr.rb#L176

If we can't find a better solution, perhaps this for 0.11.

2)

A good start would be modifying the chef-solr startup script to detect a low maxfieldlength and issue a warning.

3)

We do agree with unpacking the tarballs as suggested in CHEF-2553 to allow better access to these files. We need to evaluate the ramifications of that a little further though.

Totally agree with the concerns raised with solving this one properly but I don't have the time to implement them myself. Tossing it back to unassigned.

So what exactly are the steps for resolving this manually when using Debian packages? I've increased my maxFieldLength to 1000000 (yes, 1 million), restarted chef-solr and ran "knife index rebuild" to no avail. Running chef-solr-installer actually breaks chef-solr when using the Debian packages, so I ended up having to do a "aptitude reinstall chef-solr" to fix that. I also tried manually deleting and recreating the index files in solr data directory. Nothing has helped. I'm also not sure what is meant by "run chef-expander in debug mode." I have added "-l debug" to CHEF_EXPANDER_ARGS in /etc/default/chef-expander, but I'm not seeing anything in the expander logs at all. Any idea what I'm doing wrong?

For the record, here's the problem I'm dealing with:

$ knife search node 'fqdn:appserver30.example.com' -a chef_environment
1 items found

chef_environment: production
id: appserver30.example.com

$ knife search node 'chef_environment:production AND fqdn:appserver30.example.com' -i
0 items found

Michael, your issue doesn't seem to be related to this. First of all, if chef-expander isn't logging anything, then you need to fix the logging configuration. This could be a packaging bug, or maybe a chef-expander bug. You should see one line of output per item indexed when the log level is info.

Once that's sorted, change the log level of expander to debug. This can be done on the fly via chef-expanderctl log-level debug. This will dump the documents that chef-expander is sending to Solr. Grep that for chef_environment. If nothing is in there, it could be a bug.

If you find any bugs when doing the above, please file them.

Thanks.

Michael/Daniel,

I'm having the same issue.

I'm using the opscode repo:

cat /etc/apt/sources.list.d/opscode.list
deb http://apt.opscode.com/ lucid-0.10 main

expander is running and logging to /var/log/chef/expander.log

root@chefserver:~# ps ax | grep expander.log | grep -v grep
25986 ? S 0:03 /usr/bin/ruby1.8 /usr/bin/chef-expander -d -c /etc/chef/solr.rb -P /var/run/chef/expander.pid -L /var/log/chef/expander.log -n 1 -i 1

the loglevel is set to debug

root@chefserver:~# chef-expanderctl log-level debug
Host PID GUID Log Level
chefserver.local 25987 49cc44ef-4721-4532-bcb9-6e6c3bf4c3cb debug

And the logfile is still zero bytes.

root@chefserver:~# ls -la /var/log/chef/expander.log
rw-rr- 1 chef chef 0 2011-11-09 11:40 /var/log/chef/expander.log

Has anyone filed a bug for expander putting nothing in the logfile when debug is turned on?

It's output is in /var/chef/solor.log for some reason

FYI, I created and resolved a new ticket (CHEF-2927) to a log a warning during the chef-solr startup for this.

I have seen this exact issue on my KVM host servers (with tons of virtual network devices for all the guest VMs). I noticed it because a search (:node, "hostname:*") didn't return my KVM host servers, while a search (:node, "fqdn:*") did.

I was able to work around the issue in my recipes by changing my searches, and then I got the hostname search working by:

Increasing maxFieldLength 10-fold to 100000 in solrconfig.xml, restarting, and then rebuilding indexes (knife index rebuild). Note that maxFieldLength is in this file in 2 places and should be changed in both locations.

It is a bit scary that this can happen, silently, when searches are relied so much for critical dynamically built config files (monitoring and backups being examples).

It is also a bit unclear to me why hostname is affected, if this has to do more with the number of network interfaces on some systems. Ideally, even if the network interfaces can't be handled properly, beyond some number, it would ideal if the other data was preserved. It would also be useful to know when a search might be returning incomplete data for some reason like this.

To make this stand out a bit... You need to change the solrconfig.xml in:

/var/lib/chef/solr/conf

NOT the one in /etc/solr/conf if you installed from Ubuntu packages.

On newer versions of chef-server it's located at /var/lib/chef/solr/home/conf/solrconfig.xml.
This fixed me as well when performing searches on roles. I would see intermittent results.

Sometimes a knife search node 'role:nova-db AND role:leader' would return results, other times it would not.
Updating maxFieldLength 10-fold (in both sections) as described by Sean (above) fixed the problem after restarting chef-solr, chef-expander, and executing 'knife index rebuild'.

Has anyone really been able to get this to work properly? Anytime i raise maxFieldLength above 10k and reindex my api nodes hit 100% cpu and stop responding.

@chris gallimore and others –

If you increase the maxFieldLength, consider decreasing the number of chef-expander vnodes. I hit major CPU load when I dialed maxFieldLength up to 200200; it went away when I reduced the VNODE count to 32.

10-stable:

4666962e4269911198381890e2ed05cff2274395
15dd28bb65bbe0476a2b54d8dac4ed27a0f404e3