We occasionally see major spikes in traffic due to robots and spiders on the site. While the traffic is a good thing, it is lower priority. Unfortunately, there is no way to tell apache to work slower on some requests. If we were a smaller site, our Varnish configuration would be sufficient since all of the pages could be cached for anonymous access. Unfortunately, we have a database of 20,000+ nodes that are publicly accessible, and spiders want to see all of them. When the wrong factors line up, it equates to a minor DOS attack.
Here are some of our rejected options: