Forums

My project is under construction. I have a notice on the my first page saying so. I have also placed a robots.txt in my app directory stopping all spiders.

One spider that doesn't pay attention to robots.txt is www.baidu.com. This is a Chinese web services company that doesn't seem to care about the robots.txt standard.

I am not sure that I have the robots.txt file in the correct place. Can someone take a look? Also, is there someway PA can block these folks until they get the message?

Thanks

Could you give us a link to the robots.txt? Use the "Send feedback" link at the top so that there's no public record of the URL on the forums here, as I'm sure that wouldn't make things better...

Also -- I think Baidu should obey robots.txt, they're pretty big players -- order of Google-sized for the Chinese-speaking world. This page has some hints on making sure that they don't index your site.