Robot access control
Robots observe one file called robots.txt at root level of web server
Access is controlled for all users agents and for specific user agents
There must always be a disallow field even if it is empty (robot has access to all files), so
User-agent: * Disallow: / Will block access of all robots to all files
However if you don’t want the files read, don’t leave them on the server...