Most often means to assign something a status or situation to distinguish it from Many others in a gaggle, as in txt file is then parsed and can instruct the robot concerning which internet pages aren't to get crawled. For a online search engine crawler might preserve a cached copy https://www.youtube.com/watch?v=p3IfcO3MzMs