Avoid detecting encoding comments in documentation text (ruby#1727)

RDoc currently reads file content with
[`RDoc::Encoding.read_file`](https://github.com/ruby/rdoc/blob/master/lib/rdoc/encoding.rb#L526-L613)
during
[`RDoc::RDoc#parse_file`](https://github.com/ruby/rdoc/blob/master/lib/rdoc/rdoc.rb#L2033-L2049),
before parser selection. The current header regexp starts with `^` and
permits arbitrary comment prose before `coding:`, so documentation text
can be interpreted as an encoding directive before the selected parser
handles the file ([current
regexp](https://github.com/ruby/rdoc/blob/master/lib/rdoc/encoding.rb#L487-L507)).

Ruby documents `^` as the beginning of a line and `\A` as the beginning
of a string, and documents magic comments as top-level directives with
plain `encoding` / `coding` syntax or Emacs-style `-*- ... -*-` syntax
([Regexp
anchors](https://docs.ruby-lang.org/en/3.4/Regexp.html#class-Regexp-label-Boundary+Anchors),
[magic
comments](https://docs.ruby-lang.org/en/master/syntax/comments_rdoc.html#label-Magic+Comments)).
This PR updates RDoc header detection to stay at the file header and
limits the `coding:` branch to those magic-comment forms ([updated
regexp](https://github.com/st0012/rdoc/blob/codex/avoid-encoding-comment-doc-text/lib/rdoc/encoding.rb#L10-L20)).

The regression test covers documentation text that mentions `#encoding:
Returns`, `Encoding::US_ASCII.name`, and a later prose `# coding:` line
so those examples are treated as documentation text instead of encoding
declarations ([test
fixture](https://github.com/st0012/rdoc/blob/codex/avoid-encoding-comment-doc-text/test/rdoc/rdoc_encoding_test.rb#L158-L164)).

Author: st0012

lib/rdoc/encoding.rb

@@ -7,13 +7,13 @@
 
 module RDoc::Encoding
 
-  HEADER_REGEXP = /^
+  HEADER_REGEXP = /\A
     (?:
-      \A\#!.*\n
+      \#!.*\n
       |
       ^\#\s+frozen[-_]string[-_]literal[=:].+\n
       |
-      ^\#[^\n]+\b(?:en)?coding[=:]\s*(?<name>[^\s;]+).*\n
+      ^\#\s*(?:-\*-\s*(?:[^;\n]*;\s*)*)?(?:en)?coding[=:]\s*(?<name>[^:\s;]+).*\n
       |
       <\?xml[^?]*encoding=(?<quote>["'])(?<name>.*?)\k<quote>.*\n
     )+

test/rdoc/rdoc_encoding_test.rb

@@ -155,6 +155,15 @@ def test_class_detect_encoding
     assert_equal Encoding::UTF_8, encoding
   end
 
+  def test_class_detect_encoding_ignores_documentation_comments
+    document = "# This document describes a method named #encoding: Returns an object.\n" \
+               "#     Encoding::US_ASCII.name # => \"US-ASCII\"\n" \
+               "# This is a documentation paragraph.\n" \
+               "# coding: UTF-8 is example text, not a magic comment.\n"
+
+    assert_nil RDoc::Encoding.detect_encoding document
+  end
+
   def test_class_set_encoding_bad
     s = ""
     encoding = RDoc::Encoding.detect_encoding s