Migrate the workspace indexer to ArkFile and make it a query (#1266)

Branched from #1264
Progress towards #1212

With this PR the Ark indexer that currently powers legacy workspace
symbols, completions, and diagnostics is migrated to sources from the
Oak DB, instead of walking the disk and watching.

Benefits:

- Single source of truth
- Less memory consumption
- Reactivity to changes for workspace files that are not opened in the
editor

This last property is quite nice when changing git branches or editing
files with an agent. The changes used to be ignored until the file was
opened, but now the IDE reacts instantly.


https://github.com/user-attachments/assets/0b99be22-54a8-4522-b233-164b02be979f


- Building a file index is now a Salsa query over `oak_db::File`. Making
it a query per file is especially nice during initial workspace scans
where Salsa cancellations don't invalidate finished work per file, they
remain cached across edits without having to finish building the entire
index. This replaces the old global mutable index.

- New `workspace_files()` helper that collects, as the name says, all
workspace files. Unlike `all_files()`, library roots are excluded. This
is what `indexer::find()` and `map()` use to walk over files.

- Dropped the combined indexer + diagnostics queue. Now diagnostics
query the index, so the ordering we used to enforce by hand (finish
indexing, then run diagnostics) just falls out of the pull model. I've
kept diagnostics batching and deduping as these are still useful.

- Now that the indexer is a query, it's fully lazy. That's a good
property in principle but can lead to user-visible delays when
requesting workspace symbols in an empty workspace on startup. If a file
is opened, diagnostics will query the index and warm it up, but if none
are opened, nothing drives that.

To fix this issue, we now warm up the index manually after the initial
workspace scan has settled. This mirrors how Rust-Analyser warms up its
own general-purpose queries after the language server reaches
"quiescence" (after the initial scan, after a workspace refresh caused
by changing Cargo.toml, etc). There is no such mechanism in ty, it's
fully on-demand. I opted to go the r-a way as it's closer to what we had
before.

- Diagnostics triggering is now simplified by looking at
`salsa::plumbing::current_revision()` from the main loop. Rust-Analyser
uses it too, ty doesn't (they have a different pull model for
diagnostics that I didn't investigate much yet, our push model is closer
to R-A's).

- The indexer was the only consumer of the rename events. We now watch
files purely through `DidChangeWatchedFiles`, which is consistent which
is also what ty and r-a do.
 
The one event we might want to subscribe later on is `WillRename`. This
would be useful to e.g. update `source()` paths when a file is renamed.

Author: lionel-

crates/ark/src/lsp/backend.rs

@@ -136,9 +136,6 @@ pub(crate) enum LspNotification {
     DidChangeTextDocument(DidChangeTextDocumentParams),
     DidSaveTextDocument(DidSaveTextDocumentParams),
     DidCloseTextDocument(DidCloseTextDocumentParams),
-    DidCreateFiles(CreateFilesParams),
-    DidDeleteFiles(DeleteFilesParams),
-    DidRenameFiles(RenameFilesParams),
 }
 
 #[derive(Debug)]
@@ -297,18 +294,6 @@ impl LanguageServer for Backend {
         self.notify(LspNotification::DidChangeWatchedFiles(params));
     }
 
-    async fn did_create_files(&self, params: CreateFilesParams) {
-        self.notify(LspNotification::DidCreateFiles(params));
-    }
-
-    async fn did_delete_files(&self, params: DeleteFilesParams) {
-        self.notify(LspNotification::DidDeleteFiles(params));
-    }
-
-    async fn did_rename_files(&self, params: RenameFilesParams) {
-        self.notify(LspNotification::DidRenameFiles(params));
-    }
-
     async fn symbol(
         &self,
         params: WorkspaceSymbolParams,

crates/ark/src/lsp/completions/sources/composite/call.rs

@@ -21,6 +21,7 @@ use crate::lsp::completions::sources::utils::call_node_position_type;
 use crate::lsp::completions::sources::utils::set_sort_text_by_first_appearance;
 use crate::lsp::completions::sources::utils::CallNodePositionType;
 use crate::lsp::completions::sources::CompletionSource;
+use crate::lsp::db::ArkDb;
 use crate::lsp::document_context::DocumentContext;
 use crate::lsp::indexer;
 use crate::lsp::traits::node::NodeExt;
@@ -93,7 +94,7 @@ fn completions_from_call(
         },
     };
 
-    completions_from_arguments(document_context, callee, object)
+    completions_from_arguments(&context.state.db, document_context, callee, object)
 }
 
 fn get_first_argument(context: &DocumentContext, node: &Node) -> anyhow::Result<Option<RObject>> {
@@ -156,6 +157,7 @@ fn get_first_argument(context: &DocumentContext, node: &Node) -> anyhow::Result<
 }
 
 fn completions_from_arguments(
+    db: &dyn ArkDb,
     context: &DocumentContext,
     callable: &str,
     object: RObject,
@@ -168,7 +170,7 @@ fn completions_from_arguments(
         return Ok(Some(completions));
     }
 
-    if let Some(completions) = completions_from_workspace_arguments(context, callable)? {
+    if let Some(completions) = completions_from_workspace_arguments(db, callable, context)? {
         return Ok(Some(completions));
     }
 
@@ -234,14 +236,15 @@ fn completions_from_session_arguments(
 }
 
 fn completions_from_workspace_arguments(
-    context: &DocumentContext,
+    db: &dyn ArkDb,
     callable: &str,
+    context: &DocumentContext,
 ) -> anyhow::Result<Option<Vec<CompletionItem>>> {
     log::trace!("completions_from_workspace_arguments({callable:?})");
 
     // Try to find the `callable` in the workspace and use its arguments
     // if we can
-    let Some((_path, entry)) = indexer::find(callable) else {
+    let Some(entry) = indexer::find(db, callable) else {
         // Didn't find any workspace object with this name
         return Ok(None);
     };

crates/ark/src/lsp/completions/sources/composite/workspace.rs

@@ -69,7 +69,7 @@ fn completions_from_workspace(
     let token = token.as_str();
 
     // get entries from the index
-    indexer::map(|uri, symbol, entry| {
+    indexer::map(&state.db, |uri, symbol, entry| {
         if !symbol.fuzzy_matches(token) {
             return;
         }
@@ -112,7 +112,7 @@ fn completions_from_workspace(
                 let value = format!(
                     "Defined in `{}` on line {}.",
                     path,
-                    entry.range.start.line + 1
+                    entry.range.start.row + 1
                 );
                 let markup = MarkupContent {
                     kind: MarkupKind::Markdown,

crates/ark/src/lsp/diagnostics.rs

@@ -171,7 +171,7 @@ pub(crate) fn generate_diagnostics(
     context.document_symbols.push(HashMap::new());
 
     // Add the current workspace symbols.
-    indexer::map(|_uri, _symbol, entry| match &entry.data {
+    indexer::map(db, |_uri, _symbol, entry| match &entry.data {
         indexer::IndexEntryData::Function { name, arguments: _ } => {
             context.workspace_symbols.insert(name.to_string());
         },