Adding YARD document support to RDoc

[okuramasafumi]

Background

I implemented a plugin system for RDoc in #1321 and YARD parsing plugin is implemented.
In the discussion with @st0012, @kou and @vinistock we decided that we should implement YARD parsing feature as a standalone feature without plugin.
The note exists in #1257 (comment)

Steps

There are some steps to have effective YARD parsing feature in RDoc.

• Adding basic framework to support YARD style document (RDoc::Yard class or similar)
• Adding support of YARD tags that already works with current RDoc such as @yield and @private
  • @yield
  • @private
• Adding support of YARD tags that needs simple modification of RDoc such @deprecated
  • Adding :deprecated directive
  • Adding @deprecated
• Adding features such as type support to utilize information from YARD
  • @return
  • @param
  • others (will be edited later)
• Adding support of YARD directives and macros
  • @!attribute
  • others (will be edited later)

I will implement these step by step.

Note

The idea of plugin system is not completely abandoned but postponed.

[kou]

@st0012 Sorry. I forgot to confirm whether YARD style tag feature should be orthogonal to markups such as RDoc format and Markdown or not.

If YARD style tag feature is orthogonal to markups, we may implement it in RDoc::Markup::PreProcess, RDoc::Parser::* or somewhere.
If YARD style tag feature is NOT orthogonal to markups, we will implement it in RDoc::{Markdown,Markup,RD,TomDoc}.

(I'm not familiar with RDoc code base yet but I feel that YARD style tag feature is orthogonal to markups. It seems that YARD also uses orthogonal approach.)

@okuramasafumi How about using check list markup for the Steps section for easy to track?

* [ ] Adding support of YARD tags that already works with current RDoc such as `@yield` and `@private`
* [ ] Adding support of YARD tags that needs simple modification of RDoc such `@deprecated`
* [ ] Adding features such as type support to utilize information from YARD
* [ ] Adding support of YARD directives and macros

[okuramasafumi]

@kou At first I thought I can use sub-issues feature, but I couldn't. So I agree, I'll converted them into a list.
And I was going to implement YARD parsing feature in RDoc::Yard class using RDoc::Markup::PreProcess just like current RDoc::TomDoc class since YARD is similar to TomDoc from RDoc perspective (another documentation format).
But if we want to add just tag support to RDoc without specifying markup: yard directive, we might want to find another approach, maybe without RDoc::Yard class.

[okuramasafumi]

@st0012 This is a friendly ping about the approach to implement YARD parsing, whether we should implement is as a standalone documentation format or as a tag support along other formats such as rdoc format. I'd love to hear opinion.

[st0012]

I'll give it a look this weekend, sorry

[vinistock]

I'd like to give my perspective on this one. I see YARD comments falling into two categories: directives (or "attributes") and type annotations.

Directives

These are all of the things that are not related to types, like @deprecated or @private. I'm 100% in favour of adding these because standardizing these directives as part of RDoc gives guarantees to the entire tooling ecosystem - allowing all tools to build richer functionality on top.

For example, the Ruby LSP will be able to index that information and then we can show that a method is deprecated or private in the editor. We can also allow go to definition if the new substitute method is link to. Something like this

class Foo
  # @deprecated because of a, b and c. Use {Foo#baz} instead
  #                                         ^ going to definition here can take you to baz
  def bar; end

  def baz; end
end

Type annotations

For the @param, @return and all other type annotations, I'm not so convinced that we should standardize on the approach YARD took. All static analysis tooling in the Ruby ecosystem seem to be moving towards RBS in comments for types. Anecdotally, we have received great feedback so far about the RBS support for Sorbet.

So I wonder if we shouldn't instead standardize that type annotations must be made using RBS. While I understand that a lot of projects are already annotated with YARD, making this move towards standardizing the type annotations will help the entire tooling ecosystem to have guarantees about indexing type information. They can all speak a single language - the frontend of annotations is the same.

So I would vote that RDoc should lead the way and standardize like this:

class Foo
  # @deprecated because of a, b and c. Use {Foo#baz} instead
  #: (Integer) -> String
  def foo(a); end

  #: (Integer, String) -> String
  def baz(a, b); end

  # @private
  #: -> void
  def bar; end
end

Finally, if people agree with this direction, then I think all markups should support these features.

[kou]

@soutaro Do you have any opinion for the type annotation parts in #1344 (comment) ?

[st0012]

Hey sorry for the delay. I agree with what Vini said. Let's not introduce tags like @param or @return for now.
Though if possible I think directives in RDoc markup should still keep their current syntax (:directive:).

[soutaro]

+1 to Vini that supporting YARD type syntax may not make a lot sense.

For doc-style rbs-inline annotatios, like @rbs name: String, it will be great if RDoc can directly support that syntax.

I know some Ruby devs, like Shopify folks, prefers #: syntax. So adding @param syntax without type annotation should help those people.

Or combining YARD tags with RBS type syntax would be another possible option.

# Example using doc-style rbs inline annotations with comments
#
# @rbs name: String -- name of something
# @rbs return: bool -- returns true if name is empty
def foo(name)
  puts name
  name.empty?
end

# Example using untyped YARD style tags and `#:` style RBS type annotation
#
# @param name name of something
# @return true if name is empty
#: (String name) -> bool
def bar(name)
  puts name
  name.empty?
end

# Example using YARD style tags with RBS types
#
# @param name (String | Array[String]) name of something or an array
# @return (bool) true if name is empty
def bar(name)
  puts name
  name.empty?
end

[okuramasafumi]

Thank you everyone, what I can see is that we want RDoc to support rbs-inline style type syntax and not to support YARD style (correct me if I'm wrong).
Then the point is that there are lots of YARD-documented gems around, so we need a way to convert it into rbs-inline style so that we can use RDoc to both support types and generate documents based on type information.

This might be out of scope of RDoc, but if there is a tool to convert YARD types into RBS, it might be possible to utilize it.

[okuramasafumi]

My primary concern is the existing documents written with YARD param and return. If we don't support YARD type syntax, those type information need to be converted to RBS manually to get displayed in document. We cannot say we just don't support them since those parameters and return information are so important, obviously.
So, I believe we need to provide a way to transit from YARD to RDoc smoothly. Auto conversion might work, but it's not always the case that maintainers update existing documents.

[st0012]

We can still display them as plain comments.
And IMO this decision and the implementation are something we can change later too. So we don’t need to be blocked by it.

[okuramasafumi]

@st0012

We can still display them as plain comments.

Yes, but it's a "downgrade" for YARD users.

And IMO this decision and the implementation are something we can change later too. So we don’t need to be blocked by it.

True, I'm going to implement a few YARD tags mentioned above and see how it goes :)