What does Octocular do?

Octocular primarily provides spidering functions to a new type of search engine which is still in development.

However, it also serves a number of other functions. If you've seen Octocular's user agent in your web server's logs and its search patterns show an interest in your Javascript files, then that's probably because it's scanning your site for viruses and malware. Octocular provides the malware scanning facilities associated with the Fair Pay Northeast Jobs List. At the present time this is the most likely cause of Octocular's interest in your site.

Some of Octocular's software is also used by the free web page change notification website, Changewatching.

Octo is a complicated but flexible beast and has been (and is) used for a wide variety of private research purposes, including somewhat insecure (but entertainingly unusual) cryptography and linguistic investigation. Octocular's main database (the 'search core') borrows ideas from Markov chains, which I independently re-invented, influenced (20 years ago) by reading the source code of the Amiga NIALL software (Non-Intelligent AMOS Language Learner). The search core currently houses a body of work greater than the largest publicly available corpus of written English AI training data.

As is generally the case with anything to do with programming, linguistic Markov chains are actually much simpler than Wikipedia's convoluted explanation would have you believe.

Link to this text

IRLbot and the 3600 second crawl delay

A frequently-seen entry in robots.txt files is an instruction to Texas A&M University's IRLbot to limit its crawl speed to 1 page per hour.

The history behind this can be found on a 2006 forum posting on Webmasterworld.

One of the principles behind the design of Octocular was that it must never generate excessive traffic and it must follow the rules. (If you're an irate webmaster or sysadmin visiting because Octo just bombed your servers with a bazillion requests, then... oops. Sorry. That wasn't meant to happen. Truth is, Octocular probably doesn't have the bandwidth or processing power to DOS even a small website, so maybe it was someone else impersonating Octo?)

Octocular's original implementation of a robots.txt parser understood the crawl delay command, but ignored which bot it was aimed at and treated the first-encountered crawl delay command as though it were directed at Octocular. The thinking behind this was that if a website required 1 request per 5 seconds from someone, it must require it from everyone because that must be roughly the number of requests the server could handle if an average number of visitors made requests at a speed of 0.2 per second.

The result of all this is that Octo briefly ended up with a 1 hour crawl delay on some websites - before a workaround was implemented.

Link to this text