Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Robots.txt doesn't restrict anything. It's just a request of what should and shouldn't be scraped by search engine spiders.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: