A Method for Web Robots Control. Koster, M. Internet Draft draft-koster-robots-00, December, 1996.
abstract   bibtex   
This memo defines a method for administrators of sites on the World- Wide Web to give instructions to visiting Web robots, most importantly what areas of the site are to be avoided. This document provides a more rigid specification of the Standard for Robots Exclusion, which is currently in wide-spread use by the Web community since 1994.
@misc{ kos96,
  author = {Martijn Koster},
  title = {A Method for Web Robots Control},
  howpublished = {Internet Draft draft-koster-robots-00},
  month = {December},
  year = {1996},
  uri = {http://www.robotstxt.org/wc/norobots-rfc.html},
  abstract = {This memo defines a method for administrators of sites on the World- Wide Web to give instructions to visiting Web robots, most importantly what areas of the site are to be avoided. This document provides a more rigid specification of the Standard for Robots Exclusion, which is currently in wide-spread use by the Web community since 1994.}
}

Downloads: 0