Replace attribute_manager with a new rdoc-inline-format parser (#1559)

## RDoc::Markup::AttributeManager
RDoc parses block-level structure in `RDoc::Markup::Parser` and parses
inline styling in `RDoc::Markup::AttributeManager`.
`RDoc::Markup::AttributeManager` is a string-replacing/macro based
parser.
It's very complicated, and its mechanism are the root cause of many
bugs. We need to remove and replace it.

It converts to a flow/stream of string and styling changing operation.
Unfortunately, inline styles are structured data, output format (HTML)
is also a structured data. There's an architecture mismatch.

Tidylink `{label}[url]` should be a syntax rule, but it is currently
handled in a
regexp-based macro called `regexp_handling`.
Of course parsing nested structure such as styled tidylink label
frequently fails. It's uncontrollable.

## Solution

- Eliminate `RDoc::Markup::AttributeManager`
- Create a parser that generates structured data
- Traverse structured data to generate output instead of string
replacing
- Use controllable regexp-handling macro: only apply to text nodes

## New inline styling syntax

### Tokens

- Word pairs `+word+` `*word*` `_word_` `` `word` `` and so on
- Standalone tags `<br>`, `<tt>code_text</tt>`, `<code>code_text</code>`
- Open and close tags `<i>` `</i>`, `<b>` `</b>`, `<em>` `</em>`, `<s>`
`</s>`, `<del>` `</del>`
- Tidy link opening `{` and closing `}[url_part]`
- Simplified tidylink `word[url_part]`
- Text nodes

### Regexp handling macro

Matching with CROSSREF, RDOCREF, HYPERLINK regexp should be applied only
to text nodes after parsing phase.
Parsed tree modication instead of text-node gsub is also another option
to implement this.

### Error recovery

RDoc format doesn't have syntax error. We need to make the behavior as
similar before.

- Closing tags and braces will invalidate unclosed tags, and unclosed
tags are treated as plain text
  - `<a><b><c></a>` will be `<a>&lt;b&gt;&lt;c&gt;</a>`
  - `{<a><b>}[url]` will be a tidylink with label `&lt;a&gt;&lt;b&gt;`
- Unmatched closing tags and braces will be treated as plain text
  - `<a></b>}</a>` will be `<a>&lt;/b&gt;}</a>`
- Tidylink inside tidylink will invalidate outside tidylinks
- `{{inner}[url]}[<b>]</b>` will be `"{" + inner_tidylink + "}[" +
bold("]")`

### Simplified tidylink

RDoc was converting ``a*_`+<b>c[foo]`` to ``<a
href="foo">a*_`+&lt;b&gt;c</a>``. This is terrible, it can't coexist
with other syntaxes like `*word*` `_word_` `+word+`.
We should restrict characters and recommend `{label}[url]`.
In this pull request, only `Alphanumeric[url]` (should start with
alphabet) is supported .

Author: tompng

lib/rdoc/cross_reference.rb

@@ -1,7 +1,5 @@
 # frozen_string_literal: true
 
-require_relative 'markup/attribute_manager' # for PROTECT_ATTR
-
 ##
 # RDoc::CrossReference is a reusable way to create cross references for names.
 
@@ -33,7 +31,7 @@ class RDoc::CrossReference
   # See CLASS_REGEXP_STR
 
   METHOD_REGEXP_STR = /(
-    (?!\d)[\w#{RDoc::Markup::AttributeManager::PROTECT_ATTR}]+[!?=]?|
+    (?!\d)[\w]+[!?=]?|
     %|=(?:==?|~)|![=~]|\[\]=?|<(?:<|=>?)?|>[>=]?|[-+!]@?|\*\*?|[\/%\`|&^~]
   )#{METHOD_ARGS_REGEXP_STR}/.source.delete("\n ").freeze
 

lib/rdoc/markup.rb

@@ -79,7 +79,7 @@
 #
 #   class WikiHtml < RDoc::Markup::ToHtml
 #     def handle_regexp_WIKIWORD(target)
-#       "<font color=red>" + target.text + "</font>"
+#       "<font color=red>" + target + "</font>"
 #     end
 #   end
 #
@@ -110,10 +110,10 @@
 
 class RDoc::Markup
 
-  ##
-  # An AttributeManager which handles inline markup.
+  # Array of regexp handling pattern and its name. A regexp handling
+  # sequence is something like a WikiWord
 
-  attr_reader :attribute_manager
+  attr_reader :regexp_handlings
 
   ##
   # Parses +str+ into an RDoc::Markup::Document.
@@ -148,27 +148,11 @@ def self.parse(str)
   # structure (paragraphs, lists, and so on).  Invoke an event handler as we
   # identify significant chunks.
 
-  def initialize(attribute_manager = nil)
-    @attribute_manager = attribute_manager || RDoc::Markup::AttributeManager.new
+  def initialize
+    @regexp_handlings = []
     @output = nil
   end
 
-  ##
-  # Add to the sequences used to add formatting to an individual word (such
-  # as *bold*).  Matching entries will generate attributes that the output
-  # formatters can recognize by their +name+.
-
-  def add_word_pair(start, stop, name)
-    @attribute_manager.add_word_pair(start, stop, name)
-  end
-
-  ##
-  # Add to the sequences recognized as general markup.
-
-  def add_html(tag, name)
-    @attribute_manager.add_html(tag, name)
-  end
-
   ##
   # Add to other inline sequences.  For example, we could add WikiWords using
   # something like:
@@ -178,7 +162,7 @@ def add_html(tag, name)
   # Each wiki word will be presented to the output formatter.
 
   def add_regexp_handling(pattern, name)
-    @attribute_manager.add_regexp_handling(pattern, name)
+    @regexp_handlings << [pattern, name]
   end
 
   ##
@@ -197,15 +181,9 @@ def convert(input, formatter)
   end
 
   autoload :Parser,                "#{__dir__}/markup/parser"
+  autoload :InlineParser,          "#{__dir__}/markup/inline_parser"
   autoload :PreProcess,            "#{__dir__}/markup/pre_process"
 
-  # Inline markup classes
-  autoload :AttrChanger,           "#{__dir__}/markup/attr_changer"
-  autoload :AttrSpan,              "#{__dir__}/markup/attr_span"
-  autoload :Attributes,            "#{__dir__}/markup/attributes"
-  autoload :AttributeManager,      "#{__dir__}/markup/attribute_manager"
-  autoload :RegexpHandling,        "#{__dir__}/markup/regexp_handling"
-
   # RDoc::Markup AST
   autoload :BlankLine,             "#{__dir__}/markup/blank_line"
   autoload :BlockQuote,            "#{__dir__}/markup/block_quote"