Crazy web robots

Red Squirrel · Post by **Red Squirrel** » Fri May 13, 2005 8:22 pm

Slurp has been insane lately, I actually had to block it off the site. I was hoping for it to be temporary but does not look like it. I should actually make a robots.txt file and add it in, and hope it follows it, but meanwhile I can just monitor it's crazyness.

Archived topic from Iceteks, old topic ID:3315, old post ID:26891

Pyr-O-Rgasm · Post by **Pyr-O-Rgasm** » Fri May 13, 2005 8:42 pm

What? I'm lost.

Archived topic from Iceteks, old topic ID:3315, old post ID:26893

Red Squirrel · Post by **Red Squirrel** » Fri May 13, 2005 8:45 pm

It's web robots that crawl the site, but thing is, slurp will crawl multiple pages per second, and stay for hours, so it eats up lot of bandwidth and resources on the server. Slurp is used by yahoo, so we're not loosing much by blocking it.

Archived topic from Iceteks, old topic ID:3315, old post ID:26895