💡本文介绍了在Django中动态生成robots.txt文件的方法，包括使用HttpResponse直接返回内容、作为模板服务、自定义视图函数以及通过NGINX配置。这些方法各有优劣，适用于不同的场景需求，帮助开发者有效管理网站爬虫访问权限。

🔧使用HttpResponse直接返回内容是最简单的方法，通过定义字符串变量提供robots.txt内容，并在目标URL被访问时以text/plain类型返回。这种方法易于理解和实现，适合小型项目或简单需求。

📄将robots.txt作为模板服务可以通过创建模板文件并使用TemplateView来动态生成。这种方法更灵活，可以结合Django的模板引擎动态插入变量，适合需要根据不同环境或用户生成不同robots.txt内容的场景。

🔐自定义视图函数可以提供更复杂的逻辑处理，例如根据请求参数动态生成不同的robots.txt内容。这种方法适合需要精细控制robots.txt生成逻辑的复杂项目，但实现起来相对复杂。

🚀通过NGINX配置生成robots.txt是一种更高效的方法，特别适用于已经使用NGINX作为反向代理或静态文件服务的项目。这种方法可以充分利用NGINX的性能优势，但需要额外的配置步骤。

There are various TXT files used for different purposes like informing crawler robots, domain name verification or identity verification. One of them is a robots.txt file. Sooner or later you will need to serve a robots.txt file with Django dynamically, in order to tell web crawler robots which parts of your website is open to search engines. Let's see how we can do it.

1. With HttpResponse in urlconf

This is simple and easy to understand. We define a string variable that provides the content of the TXT file and when the target URL accessed we serve that content in response with the content type text/plain.

from django.urls import pathfrom django.http import HttpResponseADS_TXT = 'Publisher ID: e98a0dce21'ROBOTS_TXT_LINES = [  'User-agent: Googlebot',  'Disallow: /nogooglebot/',  '\n',  'User-agent: *',  'Allow: /',]ROBOTS_TXT = '\n'.join(ROBOTS_TXT_LINES)urlpatterns = [  path('ads.txt', lambda r: HttpResponse(ADS_TXT, content_type="text/plain")),  path('robots.txt', lambda r: HttpResponse(ROBOTS_TXT, content_type="text/plain")),]

2. As a template

With this method, we'll create a robots.txt file and serve it as a template.

User-agent: GooglebotDisallow: /nogooglebot/User-agent: *Allow: /

from django.urls import pathfrom django.views.generic.base import TemplateViewurlpatterns = [  path(    'robots.txt',    TemplateView.as_view(template_name="robots.txt", content_type="text/plain")),]

3. With a custom view function

We can use function-based or class based views.

from django.http import HttpResponsefrom django.views.decorators.http import require_GET@require_GET  # only allow GET requestsdef ads_txt(request):    content = 'Publisher ID: e98a0dce21 \n'    return HttpResponse(content, content_type="text/plain")

from django.urls import pathfrom main.views import ads_txturlpatterns = [  path('robots.txt', ads_txt),]

4. Via NGINX

As we will serve static files via NGINX (or something similar), this method looks more accurate to prefer. However, each of these methods in this post has advantages and disadvantages. For instance, while you can write tests in other methods, you can't with this one in Django.

Just add a location /robots.txt ... line to your NGINX configuration:

upstream server_http {    server backend:8000;}server {    listen 80;    # ...    location / {        proxy_pass http://server_http;        # ...    }    location  /robots.txt {      alias  /static/robots.txt;    }    # ...}