Remove redundant entries from robots.txt.
allow no crawling
Disallowed diff views in robots.txt concerns #440
urls with parameters excluded
rails skeleton