Skip to content

Commit a000038

Browse files
committed
src/robots.txt.ejs: remove blank line comments
Remove comment about blank lines not being allowed. This may have been true at one point, but recent verification with Google's own robotstxt tool shows no difference in parsing with or without the blank lines. Also, the RFC for the Robots Exclusion Protocol does not mention such a limitation. See: https://github.com/google/robotstxt See: https://www.rfc-editor.org/rfc/rfc9309.html
1 parent 082924c commit a000038

1 file changed

Lines changed: 2 additions & 3 deletions

File tree

src/robots.txt.ejs

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@ Sitemap: <%= origin %>/sitemap_index.html
55

66
##########################
77
# Default Access Group
8-
# (NOTE: blank lines are not allowable in a group record)
98
##########################
109
User-agent: *
1110
# Disable access to Discovery search and filters; admin pages; processes; submission; workspace; workflow & profile page
@@ -26,11 +25,11 @@ Disallow: /communities/*?spc.page
2625
# Optionally uncomment the following line ONLY if sitemaps are working
2726
# and you have verified that your site is being indexed correctly.
2827
# Disallow: /browse/*
29-
#
28+
3029
# If you have configured DSpace (Solr-based) Statistics to be publicly
3130
# accessible, then you may not want this content to be indexed
3231
# Disallow: /statistics
33-
#
32+
3433
# You also may wish to disallow access to the following paths, in order
3534
# to stop web spiders from accessing user-based content
3635
# Disallow: /contact

0 commit comments

Comments
 (0)