Skip to content

Commit effef73

Browse files
h3n4lh3n4l
authored
feat(mongodb): replace catch-all rules with explicit method definitions for C3 autocomplete (#57)
* All 6 tasks are complete. Let me provide a summary of what was done in this session: ## Session Summary This session continued the implementation of the plan to remove hardcoded method lists by leveraging C3 token candidates from the restructured grammar. ### What was done: **Task 3 — Fix `query_type.go` build failure:** - Rewrote `classifyMethodChain()` to use the new grammar API: `chain.CollectionExplainMethod()`, `chain.AllCursorMethodCall()`, `chain.CollectionMethodCall()` - Rewrote `classifyMethodCall()` → `classifyCollectionMethodCall()` to use `mongodb.ICollectionMethodCallContext` instead of the removed `mongodb.IMethodCallContext` - Removed the `GenericMethod()` check (no longer exists) - Added classification for the 20 new collection methods (bulkWrite, update, mapReduce, findAndModify, hideIndex, unhideIndex, reIndex, createSearchIndex, etc.) - Updated the `incomplete_find_statement` test expectation (parser now recovers `find` better → `Select` instead of `DML`) **Task 3 — Fix `convertCandidates()` token flooding:** - At `db.|`, C3 tokens leak from the `collectionOperation` path. Fixed by only using C3 tokens when `collectionAccess` is NOT present (i.e., at collection/cursor method positions) - At `db.|`, returns only hardcoded database methods + collections (unchanged) - At `db.users.|` and `db.users.find().|`, returns C3 tokens directly **Task 4 — Regenerate test expectations:** - Used record mode to regenerate `test_completion.yaml` - Key behavior change: `db.users.|` now shows 89 methods (both collection + cursor) instead of 34 collection-only. This is correct per the grammar, which allows either type as the first method. - `db.users.find().|` correctly shows only 35 cursor methods (grammar enforces `cursorMethodCall` after first method) **Task 5 — Cleanup verification:** - `isMethodToken()` — still needed for token → function type classification - `getDatabaseMethodCandidates()` — still needed because database methods from `dbStatement` aren't provided by C3 at `db.|` position - `previousNonDotToken()` — still needed for `db.` vs `db[` distinction - All 15 database methods verified as matching `dbStatement` grammar rule **All tests pass, lint clean.** * All done. Here's what changed: **1. Grammar fix (`MongoShellParser.g4`)**: Removed `cursorMethodCall` from the first position in `methodChain`. Now only `collectionMethodCall` is valid as the first method after a collection: ``` methodChain : DOT collectionExplainMethod DOT collectionMethodCall (DOT cursorMethodCall)* | DOT collectionMethodCall (DOT cursorMethodCall)* ; ``` There's no valid use case for `db.users.sort()` — cursor methods only make sense after a collection method like `find()`. **2. Removed `isMethodToken` and `getTokenCandidateType`**: In the `else` branch of `convertCandidates`, we're at a method position where every C3 token is a method. So we treat all tokens as `CandidateTypeFunction` with `()` suffix. This eliminates the ~120-line `isMethodToken` switch block entirely. This also fixed a bug: `getPlanCache` was previously classified as KEYWORD (no `()`). Now it correctly appears as `getPlanCache()` with FUNCTION type. **3. Completion behavior now**: - `db.|` → 15 database methods + collections (hardcoded, unchanged) - `db.users.|` → only **collection methods** (48 methods — no cursor methods like `allowDiskUse`, `sort`, `limit`) - `db.users.find().|` → only **cursor methods** (35 methods) * Done. Here's what changed: **Removed `genericDbMethod`** — the `identifier LPAREN arguments? RPAREN` catch-all rule from `dbStatement`. Same pattern we already removed for collection methods (`genericMethod`). **Added 45 new lexer tokens** for the 48 missing db methods (3 already had tokens: `AGGREGATE`, `WATCH`, `SET_WRITE_CONCERN`). **Added 48 new `dbStatement` alternatives** — each follows the explicit pattern `DB DOT TOKEN LPAREN arguments? RPAREN` with a labeled alternative name. **Updated `getDatabaseMethodCandidates()`** — expanded from 15 to 63 methods (all db methods now listed alphabetically). **Updated `identifier` rule** — all 45 new tokens added so they can still be used as identifiers (collection names, keys, etc.).
1 parent 91a7b9d commit effef73

8 files changed

+26860
-13358
lines changed

‎mongodb/MongoShellLexer.g4‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,25 @@ IS_CAPPED: 'isCapped';
7777
VALIDATE: 'validate';
7878
LATENCY_STATS: 'latencyStats';
7979

80+
// Collection methods (additional)
81+
BULK_WRITE: 'bulkWrite';
82+
UPDATE: 'update';
83+
MAP_REDUCE: 'mapReduce';
84+
FIND_AND_MODIFY: 'findAndModify';
85+
ANALYZE_SHARD_KEY: 'analyzeShardKey';
86+
CONFIGURE_QUERY_ANALYZER: 'configureQueryAnalyzer';
87+
COMPACT_STRUCTURED_ENCRYPTION_DATA: 'compactStructuredEncryptionData';
88+
HIDE_INDEX: 'hideIndex';
89+
UNHIDE_INDEX: 'unhideIndex';
90+
RE_INDEX: 'reIndex';
91+
GET_SHARD_DISTRIBUTION: 'getShardDistribution';
92+
GET_SHARD_VERSION: 'getShardVersion';
93+
// Atlas Search Index methods
94+
CREATE_SEARCH_INDEX: 'createSearchIndex';
95+
CREATE_SEARCH_INDEXES: 'createSearchIndexes';
96+
DROP_SEARCH_INDEX: 'dropSearchIndex';
97+
UPDATE_SEARCH_INDEX: 'updateSearchIndex';
98+
8099
// Database methods (M4)
81100
CREATE_COLLECTION: 'createCollection';
82101
DROP_DATABASE: 'dropDatabase';
@@ -91,6 +110,53 @@ GET_NAME: 'getName';
91110
GET_MONGO: 'getMongo';
92111
GET_SIBLING_DB: 'getSiblingDB';
93112

113+
// Database methods (additional)
114+
AUTH: 'auth';
115+
CHANGE_USER_PASSWORD: 'changeUserPassword';
116+
CLONE_DATABASE: 'cloneDatabase';
117+
COMMAND_HELP: 'commandHelp';
118+
COPY_DATABASE: 'copyDatabase';
119+
CREATE_ROLE: 'createRole';
120+
CREATE_USER: 'createUser';
121+
CREATE_VIEW: 'createView';
122+
CURRENT_OP: 'currentOp';
123+
DROP_ALL_ROLES: 'dropAllRoles';
124+
DROP_ALL_USERS: 'dropAllUsers';
125+
DROP_ROLE: 'dropRole';
126+
DROP_USER: 'dropUser';
127+
FSYNC_LOCK: 'fsyncLock';
128+
FSYNC_UNLOCK: 'fsyncUnlock';
129+
GET_LOG_COMPONENTS: 'getLogComponents';
130+
GET_PROFILING_LEVEL: 'getProfilingLevel';
131+
GET_PROFILING_STATUS: 'getProfilingStatus';
132+
GET_REPLICATION_INFO: 'getReplicationInfo';
133+
GET_ROLE: 'getRole';
134+
GET_ROLES: 'getRoles';
135+
GET_USER: 'getUser';
136+
GET_USERS: 'getUsers';
137+
GRANT_PRIVILEGES_TO_ROLE: 'grantPrivilegesToRole';
138+
GRANT_ROLES_TO_ROLE: 'grantRolesToRole';
139+
GRANT_ROLES_TO_USER: 'grantRolesToUser';
140+
HELLO: 'hello';
141+
IS_MASTER: 'isMaster';
142+
KILL_OP: 'killOp';
143+
LOGOUT: 'logout';
144+
PRINT_COLLECTION_STATS: 'printCollectionStats';
145+
PRINT_REPLICATION_INFO: 'printReplicationInfo';
146+
PRINT_SECONDARY_REPLICATION_INFO: 'printSecondaryReplicationInfo';
147+
PRINT_SHARDING_STATUS: 'printShardingStatus';
148+
PRINT_SLAVE_REPLICATION_INFO: 'printSlaveReplicationInfo';
149+
REVOKE_PRIVILEGES_FROM_ROLE: 'revokePrivilegesFromRole';
150+
REVOKE_ROLES_FROM_ROLE: 'revokeRolesFromRole';
151+
REVOKE_ROLES_FROM_USER: 'revokeRolesFromUser';
152+
ROTATE_CERTIFICATES: 'rotateCertificates';
153+
SET_LOG_LEVEL: 'setLogLevel';
154+
SET_PROFILING_LEVEL: 'setProfilingLevel';
155+
SET_SECONDARY_OK: 'setSecondaryOk';
156+
SHUTDOWN_SERVER: 'shutdownServer';
157+
UPDATE_ROLE: 'updateRole';
158+
UPDATE_USER: 'updateUser';
159+
94160
// Connection methods
95161
MONGO: 'Mongo';
96162
CONNECT: 'connect';

‎mongodb/MongoShellParser.g4‎

Lines changed: 248 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -67,15 +67,57 @@ dbStatement
6767
| DB DOT GET_NAME LPAREN RPAREN # getName
6868
| DB DOT GET_MONGO LPAREN RPAREN # getMongo
6969
| DB DOT GET_SIBLING_DB LPAREN argument RPAREN # getSiblingDB
70-
| DB DOT genericDbMethod # dbGenericMethod
70+
| DB DOT AGGREGATE LPAREN arguments? RPAREN # dbAggregate
71+
| DB DOT AUTH LPAREN arguments? RPAREN # dbAuth
72+
| DB DOT CHANGE_USER_PASSWORD LPAREN arguments? RPAREN # dbChangeUserPassword
73+
| DB DOT CLONE_DATABASE LPAREN arguments? RPAREN # dbCloneDatabase
74+
| DB DOT COMMAND_HELP LPAREN arguments? RPAREN # dbCommandHelp
75+
| DB DOT COPY_DATABASE LPAREN arguments? RPAREN # dbCopyDatabase
76+
| DB DOT CREATE_ROLE LPAREN arguments? RPAREN # dbCreateRole
77+
| DB DOT CREATE_USER LPAREN arguments? RPAREN # dbCreateUser
78+
| DB DOT CREATE_VIEW LPAREN arguments? RPAREN # dbCreateView
79+
| DB DOT CURRENT_OP LPAREN arguments? RPAREN # dbCurrentOp
80+
| DB DOT DROP_ALL_ROLES LPAREN arguments? RPAREN # dbDropAllRoles
81+
| DB DOT DROP_ALL_USERS LPAREN arguments? RPAREN # dbDropAllUsers
82+
| DB DOT DROP_ROLE LPAREN arguments? RPAREN # dbDropRole
83+
| DB DOT DROP_USER LPAREN arguments? RPAREN # dbDropUser
84+
| DB DOT FSYNC_LOCK LPAREN arguments? RPAREN # dbFsyncLock
85+
| DB DOT FSYNC_UNLOCK LPAREN arguments? RPAREN # dbFsyncUnlock
86+
| DB DOT GET_LOG_COMPONENTS LPAREN arguments? RPAREN # dbGetLogComponents
87+
| DB DOT GET_PROFILING_LEVEL LPAREN arguments? RPAREN # dbGetProfilingLevel
88+
| DB DOT GET_PROFILING_STATUS LPAREN arguments? RPAREN # dbGetProfilingStatus
89+
| DB DOT GET_REPLICATION_INFO LPAREN arguments? RPAREN # dbGetReplicationInfo
90+
| DB DOT GET_ROLE LPAREN arguments? RPAREN # dbGetRole
91+
| DB DOT GET_ROLES LPAREN arguments? RPAREN # dbGetRoles
92+
| DB DOT GET_USER LPAREN arguments? RPAREN # dbGetUser
93+
| DB DOT GET_USERS LPAREN arguments? RPAREN # dbGetUsers
94+
| DB DOT GRANT_PRIVILEGES_TO_ROLE LPAREN arguments? RPAREN # dbGrantPrivilegesToRole
95+
| DB DOT GRANT_ROLES_TO_ROLE LPAREN arguments? RPAREN # dbGrantRolesToRole
96+
| DB DOT GRANT_ROLES_TO_USER LPAREN arguments? RPAREN # dbGrantRolesToUser
97+
| DB DOT HELLO LPAREN arguments? RPAREN # dbHello
98+
| DB DOT IS_MASTER LPAREN arguments? RPAREN # dbIsMaster
99+
| DB DOT KILL_OP LPAREN arguments? RPAREN # dbKillOp
100+
| DB DOT LOGOUT LPAREN arguments? RPAREN # dbLogout
101+
| DB DOT PRINT_COLLECTION_STATS LPAREN arguments? RPAREN # dbPrintCollectionStats
102+
| DB DOT PRINT_REPLICATION_INFO LPAREN arguments? RPAREN # dbPrintReplicationInfo
103+
| DB DOT PRINT_SECONDARY_REPLICATION_INFO LPAREN arguments? RPAREN # dbPrintSecondaryReplicationInfo
104+
| DB DOT PRINT_SHARDING_STATUS LPAREN arguments? RPAREN # dbPrintShardingStatus
105+
| DB DOT PRINT_SLAVE_REPLICATION_INFO LPAREN arguments? RPAREN # dbPrintSlaveReplicationInfo
106+
| DB DOT REVOKE_PRIVILEGES_FROM_ROLE LPAREN arguments? RPAREN # dbRevokePrivilegesFromRole
107+
| DB DOT REVOKE_ROLES_FROM_ROLE LPAREN arguments? RPAREN # dbRevokeRolesFromRole
108+
| DB DOT REVOKE_ROLES_FROM_USER LPAREN arguments? RPAREN # dbRevokeRolesFromUser
109+
| DB DOT ROTATE_CERTIFICATES LPAREN arguments? RPAREN # dbRotateCertificates
110+
| DB DOT SET_LOG_LEVEL LPAREN arguments? RPAREN # dbSetLogLevel
111+
| DB DOT SET_PROFILING_LEVEL LPAREN arguments? RPAREN # dbSetProfilingLevel
112+
| DB DOT SET_SECONDARY_OK LPAREN arguments? RPAREN # dbSetSecondaryOk
113+
| DB DOT SET_WRITE_CONCERN LPAREN arguments? RPAREN # dbSetWriteConcern
114+
| DB DOT SHUTDOWN_SERVER LPAREN arguments? RPAREN # dbShutdownServer
115+
| DB DOT UPDATE_ROLE LPAREN arguments? RPAREN # dbUpdateRole
116+
| DB DOT UPDATE_USER LPAREN arguments? RPAREN # dbUpdateUser
117+
| DB DOT WATCH LPAREN arguments? RPAREN # dbWatch
71118
| DB collectionAccess methodChain # collectionOperation
72119
;
73120

74-
// Generic database method for extensibility (unsupported methods)
75-
genericDbMethod
76-
: identifier LPAREN arguments? RPAREN
77-
;
78-
79121
// Bulk operation statements
80122
// Pattern: db.collection.initializeOrderedBulkOp().find(...).update(...).execute()
81123
bulkStatement
@@ -171,13 +213,15 @@ collectionAccess
171213
| DOT GET_COLLECTION LPAREN stringLiteral RPAREN # getCollectionAccess
172214
;
173215

174-
// Method chain: one or more method calls chained with dots
216+
// Method chain: first call must be a collection method, subsequent calls are cursor methods.
217+
// Special case: explain() returns an explainable object that supports collection methods.
175218
methodChain
176-
: DOT methodCall (DOT methodCall)*
219+
: DOT collectionExplainMethod DOT collectionMethodCall (DOT cursorMethodCall)*
220+
| DOT collectionMethodCall (DOT cursorMethodCall)*
177221
;
178222

179-
// Method call: methodName(arguments?)
180-
methodCall
223+
// Collection method call: methods that operate on a collection directly
224+
collectionMethodCall
181225
: findMethod
182226
| findOneMethod
183227
| countDocumentsMethod
@@ -209,7 +253,32 @@ methodCall
209253
| isCappedMethod
210254
| validateMethod
211255
| latencyStatsMethod
212-
| sortMethod
256+
| watchMethod
257+
| bulkWriteMethod
258+
| collectionCountMethod
259+
| collectionInsertMethod
260+
| collectionRemoveMethod
261+
| updateMethod
262+
| mapReduceMethod
263+
| findAndModifyMethod
264+
| collectionExplainMethod
265+
| analyzeShardKeyMethod
266+
| configureQueryAnalyzerMethod
267+
| compactStructuredEncryptionDataMethod
268+
| hideIndexMethod
269+
| unhideIndexMethod
270+
| reIndexMethod
271+
| getShardDistributionMethod
272+
| getShardVersionMethod
273+
| createSearchIndexMethod
274+
| createSearchIndexesMethod
275+
| dropSearchIndexMethod
276+
| updateSearchIndexMethod
277+
;
278+
279+
// Cursor method call: methods that operate on a cursor (chainable after collection methods)
280+
cursorMethodCall
281+
: sortMethod
213282
| limitMethod
214283
| skipMethod
215284
| countMethod
@@ -244,7 +313,6 @@ methodCall
244313
| tryNextMethod
245314
| allowDiskUseMethod
246315
| addOptionMethod
247-
| genericMethod
248316
;
249317

250318
// Specific method rules for better AST structure
@@ -403,6 +471,111 @@ latencyStatsMethod
403471
: LATENCY_STATS LPAREN argument? RPAREN
404472
;
405473

474+
// watch(pipeline?, options?)
475+
watchMethod
476+
: WATCH LPAREN arguments? RPAREN
477+
;
478+
479+
// bulkWrite(operations, options?)
480+
bulkWriteMethod
481+
: BULK_WRITE LPAREN arguments RPAREN
482+
;
483+
484+
// count(filter?, options?) - deprecated collection-level count
485+
collectionCountMethod
486+
: COUNT LPAREN arguments? RPAREN
487+
;
488+
489+
// insert(document/array, options?) - deprecated
490+
collectionInsertMethod
491+
: INSERT LPAREN arguments RPAREN
492+
;
493+
494+
// remove(filter, options?) - deprecated
495+
collectionRemoveMethod
496+
: REMOVE LPAREN arguments RPAREN
497+
;
498+
499+
// update(filter, update, options?) - deprecated
500+
updateMethod
501+
: UPDATE LPAREN arguments RPAREN
502+
;
503+
504+
// mapReduce(map, reduce, options?) - deprecated
505+
mapReduceMethod
506+
: MAP_REDUCE LPAREN arguments RPAREN
507+
;
508+
509+
// findAndModify(document)
510+
findAndModifyMethod
511+
: FIND_AND_MODIFY LPAREN arguments RPAREN
512+
;
513+
514+
// explain(verbosity?) - collection-level explain
515+
collectionExplainMethod
516+
: EXPLAIN LPAREN arguments? RPAREN
517+
;
518+
519+
// analyzeShardKey(key, options?)
520+
analyzeShardKeyMethod
521+
: ANALYZE_SHARD_KEY LPAREN arguments RPAREN
522+
;
523+
524+
// configureQueryAnalyzer(options)
525+
configureQueryAnalyzerMethod
526+
: CONFIGURE_QUERY_ANALYZER LPAREN arguments RPAREN
527+
;
528+
529+
// compactStructuredEncryptionData(options?)
530+
compactStructuredEncryptionDataMethod
531+
: COMPACT_STRUCTURED_ENCRYPTION_DATA LPAREN arguments? RPAREN
532+
;
533+
534+
// hideIndex(indexName/spec)
535+
hideIndexMethod
536+
: HIDE_INDEX LPAREN argument RPAREN
537+
;
538+
539+
// unhideIndex(indexName/spec)
540+
unhideIndexMethod
541+
: UNHIDE_INDEX LPAREN argument RPAREN
542+
;
543+
544+
// reIndex()
545+
reIndexMethod
546+
: RE_INDEX LPAREN RPAREN
547+
;
548+
549+
// getShardDistribution()
550+
getShardDistributionMethod
551+
: GET_SHARD_DISTRIBUTION LPAREN RPAREN
552+
;
553+
554+
// getShardVersion()
555+
getShardVersionMethod
556+
: GET_SHARD_VERSION LPAREN RPAREN
557+
;
558+
559+
// createSearchIndex(definition)
560+
createSearchIndexMethod
561+
: CREATE_SEARCH_INDEX LPAREN arguments RPAREN
562+
;
563+
564+
// createSearchIndexes(definitions)
565+
createSearchIndexesMethod
566+
: CREATE_SEARCH_INDEXES LPAREN arguments RPAREN
567+
;
568+
569+
// dropSearchIndex(name)
570+
dropSearchIndexMethod
571+
: DROP_SEARCH_INDEX LPAREN argument RPAREN
572+
;
573+
574+
// updateSearchIndex(name, definition)
575+
updateSearchIndexMethod
576+
: UPDATE_SEARCH_INDEX LPAREN arguments RPAREN
577+
;
578+
406579
// sort(specification?) - can be called without args
407580
sortMethod
408581
: SORT LPAREN document? RPAREN
@@ -555,11 +728,6 @@ addOptionMethod
555728
: ADD_OPTION LPAREN NUMBER RPAREN
556729
;
557730

558-
// Generic method for extensibility (other methods will be caught here)
559-
genericMethod
560-
: identifier LPAREN arguments? RPAREN
561-
;
562-
563731
// Arguments: comma-separated list of values
564732
arguments
565733
: argument (COMMA argument)* COMMA?
@@ -862,4 +1030,67 @@ identifier
8621030
| GET_CLIENT_ENCRYPTION
8631031
// Plan cache method tokens
8641032
| GET_PLAN_CACHE
1033+
// Collection method tokens (additional)
1034+
| BULK_WRITE
1035+
| UPDATE
1036+
| MAP_REDUCE
1037+
| FIND_AND_MODIFY
1038+
| ANALYZE_SHARD_KEY
1039+
| CONFIGURE_QUERY_ANALYZER
1040+
| COMPACT_STRUCTURED_ENCRYPTION_DATA
1041+
| HIDE_INDEX
1042+
| UNHIDE_INDEX
1043+
| RE_INDEX
1044+
| GET_SHARD_DISTRIBUTION
1045+
| GET_SHARD_VERSION
1046+
| CREATE_SEARCH_INDEX
1047+
| CREATE_SEARCH_INDEXES
1048+
| DROP_SEARCH_INDEX
1049+
| UPDATE_SEARCH_INDEX
1050+
// Database method tokens (additional)
1051+
| AUTH
1052+
| CHANGE_USER_PASSWORD
1053+
| CLONE_DATABASE
1054+
| COMMAND_HELP
1055+
| COPY_DATABASE
1056+
| CREATE_ROLE
1057+
| CREATE_USER
1058+
| CREATE_VIEW
1059+
| CURRENT_OP
1060+
| DROP_ALL_ROLES
1061+
| DROP_ALL_USERS
1062+
| DROP_ROLE
1063+
| DROP_USER
1064+
| FSYNC_LOCK
1065+
| FSYNC_UNLOCK
1066+
| GET_LOG_COMPONENTS
1067+
| GET_PROFILING_LEVEL
1068+
| GET_PROFILING_STATUS
1069+
| GET_REPLICATION_INFO
1070+
| GET_ROLE
1071+
| GET_ROLES
1072+
| GET_USER
1073+
| GET_USERS
1074+
| GRANT_PRIVILEGES_TO_ROLE
1075+
| GRANT_ROLES_TO_ROLE
1076+
| GRANT_ROLES_TO_USER
1077+
| HELLO
1078+
| IS_MASTER
1079+
| KILL_OP
1080+
| LOGOUT
1081+
| PRINT_COLLECTION_STATS
1082+
| PRINT_REPLICATION_INFO
1083+
| PRINT_SECONDARY_REPLICATION_INFO
1084+
| PRINT_SHARDING_STATUS
1085+
| PRINT_SLAVE_REPLICATION_INFO
1086+
| REVOKE_PRIVILEGES_FROM_ROLE
1087+
| REVOKE_ROLES_FROM_ROLE
1088+
| REVOKE_ROLES_FROM_USER
1089+
| ROTATE_CERTIFICATES
1090+
| SET_LOG_LEVEL
1091+
| SET_PROFILING_LEVEL
1092+
| SET_SECONDARY_OK
1093+
| SHUTDOWN_SERVER
1094+
| UPDATE_ROLE
1095+
| UPDATE_USER
8651096
;

0 commit comments

Comments
 (0)