Robots.txt tells hackers the places you don't want them to look
Found on The Register on Tuesday, 19 May 2015
Melbourne penetration tester Thiebauld Weksteen is warning system administrators that robots.txt files can give attackers valuable information on potential targets by giving them clues about directories their owners are trying to protect.
Admins would be best excluding assets based on general terms and not through absolute references.
Very old news. This has been a common approach since the robots.txt exists. Admins should just use the access control options every webserver provides. If you don't want a resource to be accessible, put a password on it and/or restrict access based on the client IP address; or even better, don't put sensitive information on a public server.