14.02.13

IRLbot and the 3600 second crawl delay

A frequently-seen entry in robots.txt files is an instruction to Texas A&M University's IRLbot to limit its crawl speed to 1 page per hour.

The history behind this can be found on a 2006 forum posting on Webmasterworld.

One of the principles behind the design of Octocular was that it must never generate excessive traffic and it must follow the rules. (If you're an irate webmaster or sysadmin visiting because Octo just bombed your servers with a bazillion requests, then... oops. Sorry. That wasn't meant to happen. Truth is, Octocular probably doesn't have the bandwidth or processing power to DOS even a small website, so maybe it was someone else impersonating Octo?)

Octocular's original implementation of a robots.txt parser understood the crawl delay command, but ignored which bot it was aimed at and treated the first-encountered crawl delay command as though it were directed at Octocular. The thinking behind this was that if a website required 1 request per 5 seconds from someone, it must require it from everyone because that must be roughly the number of requests the server could handle if an average number of visitors made requests at a speed of 0.2 per second.

The result of all this is that Octo briefly ended up with a 1 hour crawl delay on some websites - before a workaround was implemented.