
Back before there was Google, the big new search engine out there was AltaVista. In an effort to show off its power, the AltaVista team from Digital decided to crawl and index the entire web, which was a new concept at the time. There were many who didn’t like the idea of a “robot” program accessing every page on their web sites because it would cause more load time to their web servers and increase bandwidth costs for them. To address their growing concerns, in 1996 the Robots Exclusion Standard was created.
