Hello,
I am trying to setup my robots.txt on my django site.
I added this to my views,
31 def robots(request):
32 data = {}
33 return render(request, 'name_app/robots.txt', data)
I added this to my urls
url(r'^robots.txt$',port_main_app.views.robots),
However when I navigate to the file, mydomain.com/robots.txt
the robots.txt file is displayed on a single line like so User-agent: * Disallow: /
I believe this will confuse the crawlers and prevent the file from being followed.
How do I fix this so that the robots.txt file is displayed properly like so,
User-agent: * Disallow: /
Thanks in advance!