Skip to content

Add basic parallel indexing logic#36

Closed
vinistock wants to merge 1 commit into
mainfrom
07-10-add_bindings_with_parallel_indexing
Closed

Add basic parallel indexing logic#36
vinistock wants to merge 1 commit into
mainfrom
07-10-add_bindings_with_parallel_indexing

Conversation

@vinistock

@vinistock vinistock commented Jul 10, 2025

Copy link
Copy Markdown
Member

Add example Rust to Ruby VM bindings to the repository, so that people can use this while working on their prototypes. The keyword here is example.

This PR demonstrates how we can share an Arc of Repository with a Ruby object and how we can return a boxed value to produce a new Ruby Declaration object. That doesn't mean we'll actually need this exactly as is. Until we have settled on the data structures, we have no way of knowing exactly what we'll need. This is useful nonetheless so that people can test their prototypes with Ruby scripts and see if they actually do what they want.

The PR includes:

  • Parallel indexing
  • A basic visitor that captures classes and modules (notice that the name part is incorrect, it doesn't handle nesting yet)
  • The ability to add or get declarations from Ruby

I didn't stress over adding a bunch of tests since this code is a temporary example and will eventually be substituted by the real implementation.

The most important part of this PR, and what I would love for people to focus on, is the documentation of the c_interface file. Do you agree with everything in there? Is it clear enough for you to feel empowered to add more bindings to the Ruby VM? How can we ensure that the details are as easy to follow as possible?

Unfortunate note

Windows CI will be broken until ruby/prism#3594 is merged (provided it actually fixes the issue, which I'm only between 60-70% sure).

@vinistock vinistock self-assigned this Jul 10, 2025

Copy link
Copy Markdown
Member Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@vinistock vinistock added the enhancement New feature or request label Jul 10, 2025 — with Graphite App
@vinistock
vinistock marked this pull request as ready for review July 10, 2025 20:54
Comment thread cbindgen.toml
Comment on lines +11 to +12
[enum]
rename_variants = "ScreamingSnakeCase"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enums in C are usually all in caps, so I added this to make sure cbindgen will transform them for us.

Comment thread ext/index/index.c
#include <string.h>
#include "repository.h"

// Store the class references

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

C extensions are usually organized the same way as CRuby: each file contains everything related to a specific class/module and the file name is the same as the class/module name.

I moved everything to repository.c.

Comment thread lib/index/declaration.rb
Comment on lines +4 to +6
# Represents the base declaration class for all indexed items. This class is partially defined in C to connect with
# the Rust layer
class Declaration

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not strictly necessary, but a good example of how we can mix C and Ruby for defining the API. In reality, we only need C to access the Rust side of things. Simple things that aren't performance critical can be defined in Ruby.

Comment thread src/lib.rs
@ChallaHalla

Copy link
Copy Markdown
Contributor

@vinistock For ruby/prism#3594, how are we able to verify that the change successfully fixes the issue for Windows?

@vinistock

Copy link
Copy Markdown
Member Author

We can point to the GitHub repo and give it a try.

Comment thread src/ruby_indexer.rs
}
}

pub fn index(&mut self, file_path: String) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should return a Result<(), std::io::Error> in case we can't read the file?

Comment thread ext/index/repository.c

void* repository;
TypedData_Get_Struct(self, void*, &repository_type, repository);
idx_index_all_c(repository, converted_file_paths, length);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the indexing panics, is there a change we're leaking converted_file_paths?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should also raise an exception back to Ruby is the indexing failed? For example if we can't open one of the file in the given paths?

Comment thread src/c_interface.rs
Box::into_raw(Box::new(repository)) as *mut CRepository
/// Converts an array of C strings (const **char) to a Vec<String>
unsafe fn convert_double_pointer_to_vec(data: &mut &mut c_char, len: size_t) -> Result<Vec<String>, Utf8Error> {
unsafe {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need the unsafe block if the function itself is already marked unsafe?

@vinistock

Copy link
Copy Markdown
Member Author

Closing in favour of #54

@vinistock vinistock closed this Aug 5, 2025
@vinistock
vinistock deleted the 07-10-add_bindings_with_parallel_indexing branch August 5, 2025 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants