Skip to content

Handing empty robots.txt string#18

Open
tcz wants to merge 1 commit into
chrisakroyd:masterfrom
tcz:master
Open

Handing empty robots.txt string#18
tcz wants to merge 1 commit into
chrisakroyd:masterfrom
tcz:master

Conversation

@tcz

@tcz tcz commented Sep 5, 2024

Copy link
Copy Markdown

I got bugs when the robots.txt was empty on the server, possibly due to it missing altogether.

TypeError: string.match is not a function or its return value is not iterable
  at robustSplit (src/parser.js:23:50)
  at parser (src/parser.js:62:13)
  at Context.<anonymous> (test/parser/can-parse-test-files.js:72:25)
  at process.processImmediate (node:internal/timers:478:21)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant