gecko-dev/webtools/lxr/robots.txt

mirror of https://github.com/mozilla/gecko-dev.git synced 2024-10-31 22:25:30 +00:00
User-agent: *
Allow: /index.html
Disallow: /
attempt to dissuade robots from indexing the entire lxr.mozilla.org site 1999-06-15 06:34:45 +00:00			`User-agent: *`
If the application is set up at the server root (e.g. bugzilla.mozilla.org) then this robots.txt disallows robots from scanning the site except for the top level file. 2000-07-28 21:28:41 +00:00			`Allow: /index.html`
attempt to dissuade robots from indexing the entire lxr.mozilla.org site 1999-06-15 06:34:45 +00:00			`Disallow: /`