Robot exclusion protocols
Server-wide method (/robots.txt)
- Directives for the whole server must be under the top level /robots.txt.
META element method (per page).
- Directives are inserted per page with the META element. Directives allow for indexing (or not) and parsing for links (or not).