torproject/lego
1
0
Fork 0
mirror of https://github.com/torproject/lego.git synced 2024-11-26 19:20:22 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Simplify robots.txt

Browse Source 
The robots.txt spec isn't as strict as I thought, so we don't need to worry
about whitespace or capitalization
This commit is contained in:
kez 2022-11-07 10:17:06 -08:00
parent ae268f2d69
commit 4235296de5
1 changed files with 6 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

18
templates/robots.txt
View File

@ -1,20 +1,14 @@
{%- macro capitalize_first(text) -%}
{{- text[0].upper() -}}{{- text[1:] -}}
{%- endmacro -%}
User-agent: *
Disallow: /javascript
Disallow: /scss
{#- the '\n' is because we need a newline,
but we need to strip trailing and leading whitespace as well.
fragile, be careful #}
{% if bag('robots_txt') %}
{%- for user_agent, directives in bag('robots_txt').items() -%}
{{ '\n' }}User-agent: {{ user_agent }}
{%- for directive, values in directives.items() -%}
{%- for value in values -%}
{{ '\n' }}{{ capitalize_first(directive) }}: {{ value -}}
{% endfor -%}
{% for user_agent, directives in bag('robots_txt').items() %}
User-agent: {{ user_agent }}
{% for directive, values in directives.items() %}
{% for value in values %}
{{ directive }}: {{ value }}
{% endfor %}
{% endfor %}
{% endfor %}
{% endif %}