Anything Forums

Slurp has been insane lately, I actually had to block it off the site. I was hoping for it to be temporary but does not look like it. I should actually make a robots.txt file and add it in, and hope it follows it, but meanwhile I can just monitor it's crazyness.




Archived topic from Iceteks, old topic ID:3315, old post ID:26891

What? I'm lost.

Archived topic from Iceteks, old topic ID:3315, old post ID:26893

It's web robots that crawl the site, but thing is, slurp will crawl multiple pages per second, and stay for hours, so it eats up lot of bandwidth and resources on the server. Slurp is used by yahoo, so we're not loosing much by blocking it.

Archived topic from Iceteks, old topic ID:3315, old post ID:26895