Recommendations
Free tools: wait for the new version of htdig due out anytime. Swish-E would be an alternative. If non-technical answer needed, look for server software incorporating search facility
Commercial: very dependent on what your requirements are. We chose Ultraseek from Infoseek to index 190+ servers on site.