User-agent: * Allow: / Disallow: /media/cache Disallow: /leave* Disallow: /create/ Disallow: /redir* Disallow: /login* Disallow: /report/ # Allow the Grapeshot crawler full access User-agent: grapeshot Disallow: