Settle implementation plan

Author: st0012

docs/plans/2026-01-21-dsl-definitions-design.md

@@ -0,0 +1,391 @@
+# DSL Definitions Design
+
+**Issue**: <https://github.com/Shopify/rubydex/issues/497>
+
+## Problem
+
+Many Ruby DSLs fit a common pattern: a method call that produces multiple declarations. This includes:
+
+- Built-in DSLs: `attr_accessor`, `Struct.new`, `Class.new`
+- Framework DSLs: Rails' `belongs_to`, `has_many`, ActiveSupport's `class_methods`
+- Testing DSLs: RSpec's `describe`, `context`, `it`
+
+Currently, rubydex handles some of these as special cases in the indexer. However, this approach has a fundamental limitation: during indexing, we cannot guarantee that `Class.new` is actually `::Class` and not `MyImplementation::Class` because resolution hasn't run yet.
+
+## Solution Overview
+
+Capture DSL method calls as `DslDefinition` during indexing, then process them after resolution when we know the fully qualified receiver. This enables:
+
+1. Correct identification of DSLs (we know `Class` resolves to `::Class`)
+2. A unified architecture for both built-in and plugin-provided DSL handling
+
+## Architecture
+
+### Phase 1: Indexing
+
+The `Graph` maintains an allowlist of DSL target method names:
+
+```rust
+// In Graph
+internal_dsl_targets: Vec<StringId>,  // e.g., ["new", "attr_accessor"]
+// Future: external_dsl_targets for plugins
+```
+
+**Two definition types are created during indexing:**
+
+#### DslDefinition
+
+Captures the DSL method call itself:
+
+```rust
+struct DslDefinition {
+    receiver_name: NameId,              // e.g., Name(Class, parent_scope: "Foo")
+    method_name: StringId,              // e.g., "new"
+    arguments: DslArgumentList,         // Captured arguments
+    uri_id: UriId,
+    offset: Offset,
+    lexical_nesting_id: Option<DefinitionId>,
+    members: Vec<DefinitionId>,         // Owned block contents
+}
+```
+
+The `DslDefinition` is pushed onto the nesting stack when visiting its block, so any definitions inside (methods, nested DSL calls, etc.) become members of the `DslDefinition` rather than the lexical parent.
+
+#### ConstantToDslDefinition
+
+When a DSL call is assigned to a constant (e.g., `Bar = Class.new`), we create a `ConstantToDslDefinition` that links them:
+
+```rust
+struct ConstantToDslDefinition {
+    constant_name: NameId,              // e.g., Name(Bar, parent_scope: "Foo")
+    dsl_definition_id: DefinitionId,    // points to the DslDefinition
+    uri_id: UriId,
+    offset: Offset,
+    lexical_nesting_id: Option<DefinitionId>,
+    comments: Vec<Comment>,
+    flags: DefinitionFlags,
+}
+```
+
+**Indexing flow in `visit_constant_write_node`:**
+
+1. Call `index_dsl_definition_target(value)` - similar to existing `index_constant_alias_target`
+2. If it returns a `DslDefinition`, call `add_constant_to_dsl_definition(constant_name, dsl_def_id)`
+3. This creates `ConstantToDslDefinition` linking the constant to the DSL
+
+For non-constant cases (`var = Class.new` or just `Class.new`), only `DslDefinition` is created (detected in `visit_call_node`).
+
+### Phase 2: Resolution
+
+Standard resolution runs, which resolves `receiver_name` to a fully qualified name (e.g., `Class` in scope `Foo` → `::Class`).
+
+### Phase 3: DSL Processing (New Phase)
+
+After resolution, process all `ConstantToDslDefinition`s:
+
+1. Look up the referenced `DslDefinition`
+2. Get the resolved receiver name
+3. Dispatch to handler based on resolved receiver + method name:
+
+```text
+resolved receiver = ::Class, method = "new"  → ClassNewHandler
+resolved receiver = ::Module, method = "new" → ModuleNewHandler
+// Future: ::Struct, plugin-registered handlers
+```
+
+**Handler outcomes:**
+
+- **Handler matches (e.g., `::Class.new`)**: Transform `ConstantToDslDefinition` into `DynamicClassDefinition` with the constant name. Creates `ClassDeclaration`.
+- **No handler matches (e.g., `SomeClass.new`)**: Transform `ConstantToDslDefinition` into regular `ConstantDefinition`. We don't know what `SomeClass.new` returns.
+
+Each handler also decides what to do with owned members (reassign, discard, etc.).
+
+### Phase 4: Incremental Resolution
+
+Run resolution again for any new definitions created in Phase 3. This should be fast since the set is small.
+
+## DslDefinition Details
+
+### Argument Representation
+
+Arguments are captured in a structured format to save downstream consumers from parsing:
+
+```rust
+enum DslArgument {
+    Symbol(StringId),
+    String(StringId),
+    Boolean(bool),
+    Integer(i64),
+    Constant(NameId),
+    Block { offset: Offset },  // Location for AST revisit if needed
+}
+
+struct DslArgumentList {
+    positional: Vec<DslArgument>,
+    keyword: HashMap<StringId, DslArgument>,
+    block: Option<Offset>,
+}
+```
+
+### Block Ownership
+
+All DSL calls automatically own their block contents. This simplifies indexing logic—the handler decides what to do with owned definitions during processing:
+
+- Reassign them to a new owner
+- Discard them
+- Keep them under the DSL
+- Create new definitions based on them
+
+### Lifecycle
+
+`DslDefinition` is kept after processing. It shares a declaration with any generated definitions, enabling tooling features like "hover over `belongs_to` → see generated methods".
+
+## Example: Class.new
+
+```ruby
+class Foo
+  Bar = Class.new do
+    def baz; end
+  end
+end
+```
+
+**Indexing produces:**
+
+- `ClassDefinition(Foo)`
+- `DslDefinition(receiver: Name(Class), method: "new", members: [MethodDefinition(baz)])`
+- `ConstantToDslDefinition(constant: Bar, dsl: DslDefinition)` - links constant to DSL
+- `MethodDefinition(baz)` with lexical nesting pointing to DslDefinition
+
+**Resolution:**
+
+- `Name(Class)` in `DslDefinition` resolves to `::Class`
+
+**DSL Processing:**
+
+- Process `ConstantToDslDefinition(Bar)`
+- Look up its `DslDefinition` - receiver resolved to `::Class`, method is `new`
+- `ClassNewHandler` matches
+- Transforms `ConstantToDslDefinition` into `DynamicClassDefinition(Foo::Bar)`
+- Reassigns `MethodDefinition(baz)` from DslDefinition to DynamicClassDefinition
+- Creates `ClassDeclaration(Foo::Bar)` and `MethodDeclaration(Foo::Bar#baz)`
+
+**Result:**
+
+- `Declaration(Foo::Bar)` has two definitions:
+  - `DslDefinition(Class.new)` - the original DSL call (kept for tooling)
+  - `DynamicClassDefinition` - the class structure with members
+- `Declaration(Foo::Bar#baz)` has one definition: `MethodDefinition(baz)`
+
+**Anonymous case (`var = Class.new` or just `Class.new`):**
+
+- Only `DslDefinition` created (no `ConstantToDslDefinition`)
+- DSL Processing creates `DynamicClassDefinition` with `name_id: None`
+- No declaration created
+
+**Unknown receiver case (`Foo = SomeClass.new`):**
+
+- `DslDefinition(receiver: SomeClass, method: "new")`
+- `ConstantToDslDefinition(constant: Foo, dsl: DslDefinition)`
+- After resolution, `SomeClass` resolves to `::SomeClass` (not `::Class`)
+- No handler matches
+- `ConstantToDslDefinition` becomes regular `ConstantDefinition(Foo)`
+
+## Example: RSpec (Future Plugin)
+
+```ruby
+describe Foo do
+  context "when valid" do
+    it "works" do; end
+    def helper; end
+  end
+end
+```
+
+**Indexing produces (with `describe`, `context`, `it` as targets):**
+
+- `DslDefinition(describe Foo)` owns:
+  - `DslDefinition(context "when valid")` owns:
+    - `DslDefinition(it "works")`
+    - `MethodDefinition(helper)`
+
+**DSL Processing (via RSpec plugin):**
+
+- Plugin handler decides how to represent these—perhaps as test group declarations with the helper method scoped appropriately.
+
+## New Types
+
+### Definitions
+
+```rust
+// Captures DSL method call (e.g., Class.new, Module.new)
+struct DslDefinition {
+    receiver_name: NameId,
+    method_name: StringId,
+    arguments: DslArgumentList,
+    uri_id: UriId,
+    offset: Offset,
+    lexical_nesting_id: Option<DefinitionId>,
+    members: Vec<DefinitionId>,
+}
+
+// Links constant assignment to DSL call (e.g., Bar = Class.new)
+struct ConstantToDslDefinition {
+    constant_name: NameId,
+    dsl_definition_id: DefinitionId,
+    uri_id: UriId,
+    offset: Offset,
+    lexical_nesting_id: Option<DefinitionId>,
+    comments: Vec<Comment>,
+    flags: DefinitionFlags,
+}
+
+// Created in Phase 3 for Class.new blocks
+struct DynamicClassDefinition {
+    name_id: Option<NameId>,  // None for anonymous classes
+    uri_id: UriId,
+    offset: Offset,
+    members: Vec<DefinitionId>,  // Consistent with ClassDefinition
+    superclass_ref: Option<ReferenceId>,
+    mixins: Vec<Mixin>,
+    lexical_nesting_id: Option<DefinitionId>,
+    comments: Vec<Comment>,
+    flags: DefinitionFlags,
+}
+
+// Created in Phase 3 for Module.new blocks
+struct DynamicModuleDefinition {
+    name_id: Option<NameId>,  // None for anonymous modules
+    uri_id: UriId,
+    offset: Offset,
+    members: Vec<DefinitionId>,  // Consistent with ModuleDefinition
+    mixins: Vec<Mixin>,
+    lexical_nesting_id: Option<DefinitionId>,
+    comments: Vec<Comment>,
+    flags: DefinitionFlags,
+}
+```
+
+### Declarations
+
+No new declaration types needed. Dynamic definitions create standard declarations:
+
+- `DynamicClassDefinition` → `ClassDeclaration`
+- `DynamicModuleDefinition` → `ModuleDeclaration`
+
+### DSL Handler Trait
+
+```rust
+trait DslHandler {
+    /// Returns true if this handler should process the given DSL call
+    fn handles(&self, resolved_receiver: &str, method_name: &str) -> bool;
+
+    /// Process the DSL and create new definitions/declarations
+    fn process(&self, graph: &mut Graph, dsl: &DslDefinition);
+}
+
+struct DslHandlerRegistry {
+    handlers: Vec<Box<dyn DslHandler>>,
+}
+```
+
+### DefinitionId Generation
+
+`DslDefinition` generates its ID using receiver_name and method_name:
+
+```rust
+impl DslDefinition {
+    pub fn id(&self) -> DefinitionId {
+        DefinitionId::from(&format!(
+            "{}{}{}{}",
+            *self.uri_id,
+            self.offset.start(),
+            *self.receiver_name,
+            *self.method_name
+        ))
+    }
+}
+```
+
+## Implementation Scope
+
+### Phase 1 (This Issue)
+
+- Add `DslDefinition` struct
+- Add `ConstantToDslDefinition` struct
+- Add `DynamicClassDefinition` and `DynamicModuleDefinition` structs
+- Add `internal_dsl_targets` to Graph
+- Add `index_dsl_definition_target` helper in indexer (similar to `index_constant_alias_target`)
+- Add `add_constant_to_dsl_definition` helper in indexer
+- Implement `Nesting::Dsl` variant for nesting stack
+- Add DSL processing phase with `DslHandler` trait
+- Implement `Class.new` handler
+- Implement `Module.new` handler
+
+### Future Work
+
+- `Struct.new` handler (complex: inherits from Struct, generates accessor methods)
+- Plugin API for external DSL targets
+- RSpec/Minitest support via plugins
+
+## Open Questions
+
+1. **Memory impact**: One-to-many relationship between definitions and declarations. Monitor memory usage during implementation.
+
+## Design Decisions Made
+
+1. **Constant assignment detection**: Use `ConstantToDslDefinition` to link constants to DSL calls. Similar pattern to `index_constant_alias_target` in existing code. Detected in `visit_constant_write_node` via new `index_dsl_definition_target` helper.
+
+2. **Anonymous classes**: `DynamicClassDefinition` uses `name_id: Option<NameId>`. Named classes get `Some(name)` from `ConstantToDslDefinition`, anonymous classes get `None` and no declaration is created.
+
+3. **Declaration membership**: For `Bar = Class.new`, `Declaration(Foo::Bar)` contains two definitions: `DslDefinition` and `DynamicClassDefinition`. This enables tooling to trace generated code back to the DSL call.
+
+4. **Unknown receiver fallback**: If `ConstantToDslDefinition` references a DSL whose receiver doesn't match any handler (e.g., `Foo = SomeClass.new`), it becomes a regular `ConstantDefinition`.
+
+5. **lexical_nesting_id preservation**: When members are reassigned from `DslDefinition` to `DynamicClassDefinition`, their `lexical_nesting_id` is NOT updated. It preserves the lexical structure as written in source code.
+
+6. **Phase 4 resolution**: Simply call `resolve_all` again. The set of new definitions is small, so full resolution is acceptable.
+
+7. **Edge cases not handled**: Conditional assignments (`Foo = cond ? Class.new : Module.new`), method chaining (`Class.new.freeze`), and late assignment of anonymous classes (`klass = Class.new; Foo = klass`) are out of scope.
+
+8. **Phase 3 location**: New method `process_dsl_definitions()` called after `resolve_all()`. Caller does: `resolver.resolve_all(); resolver.process_dsl_definitions(); resolver.resolve_all();`
+
+9. **Receiver resolution**: During indexing, create a `ConstantReference` for `DslDefinition.receiver_name`. It gets resolved naturally in Phase 2. Phase 3 looks it up via `graph.names().get(&receiver_name)`.
+
+10. **DynamicClassDefinition ID**: Use same location as source `DslDefinition` with DSL name: `format!("{}{}Class.new", uri_id, offset)`. Similarly `Module.new` for `DynamicModuleDefinition`.
+
+## Test Cases Needed
+
+1. **Unknown receiver fallback**:
+
+   ```ruby
+   class MyClass
+     def self.new; end
+   end
+   Foo = MyClass.new
+   ```
+
+   `ConstantToDslDefinition(Foo)` should become `ConstantDefinition(Foo)` since `MyClass.new` doesn't match any handler.
+
+2. **Inheritance from dynamic class**:
+
+   ```ruby
+   Foo = Class.new do
+   end
+   class Bar < Foo; end
+   ```
+
+   After phase 3+4, `Bar` should have superclass pointing to `Foo`, which has a `DynamicClassDefinition`.
+
+3. **DSL inside DSL block**:
+
+   ```ruby
+   Foo = Class.new do
+     Bar = Class.new do
+       def inner; end
+     end
+     attr_accessor :name
+   end
+   ```
+
+   Verify `Bar` is correctly handled as a nested constant (`Foo::Bar` with `inner` method). And `Foo` has `name` and `name=` methods.