# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file User-agent: * Crawl-delay: 17 Disallow: /posts/new Disallow: /posts/edit Disallow: /posts/delete Disallow: /comments/new Disallow: /comments/edit Disallow: /comments/delete