User-agent: citeseerxbot
Disallow: /photos
Disallow: /download/docs
Disallow: /download/frequentis
Disallow: /download/phd
Disallow: /download/div


User-agent: *
Disallow: /