txt file is then parsed and will instruct the robot as to which pages aren't to get crawled. Like a search engine crawler might keep a cached duplicate of the file, it might occasionally crawl internet pages a webmaster won't prefer to crawl. Web pages normally prevented from becoming crawle