Skip to content

feat: Support Gemini via Vertex AI#2030

Merged
dividedmind merged 7 commits intomainfrom
feat/support-vertex-ai
Oct 22, 2024
Merged

feat: Support Gemini via Vertex AI#2030
dividedmind merged 7 commits intomainfrom
feat/support-vertex-ai

Conversation

@dividedmind
Copy link
Copy Markdown
Collaborator

@dividedmind dividedmind commented Oct 3, 2024

Note to access the API you have to create a service account at https://console.cloud.google.com/apis/credentials — after creation you'll get a JSON file with credentials. This JSON has to be put into GOOGLE_WEB_CREDENTIALS environment variable.

Example configuration:

GOOGLE_WEB_CREDENTIALS='{   "type": "service_account",   "project_id": "gen-lang-client-0551265112",   "private_key_id": "1fd731b64e9543d54e2921d7cfbe40c5f6f2845a",   "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",   "client_email": "divide-s-navie-experiments@gen-lang-client-0551265112.iam.gserviceaccount.com",   "client_id": "112105442388554579010",   "auth_uri": "https://accounts.google.com/o/oauth2/auth",   "token_uri": "https://oauth2.googleapis.com/token",   "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",   "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/divide-s-navie-experiments%40gen-lang-client-0551265112.iam.gserviceaccount.com",   "universe_domain": "googleapis.com" }'
APPMAP_NAVIE_MODEL=gemini-1.5-pro-002

@dividedmind dividedmind self-assigned this Oct 3, 2024
@kgilpin kgilpin marked this pull request as draft October 3, 2024 12:16
kgilpin
kgilpin reviewed Oct 3, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@kgilpin kgilpin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! There are some build errors so I flipped this to Draft, temporarily.

dustinbyrne
dustinbyrne reviewed Oct 3, 2024
View reviewed changes
Comment thread packages/navie/package.json
@dustinbyrne dustinbyrne marked this pull request as ready for review October 4, 2024 14:10
@kgilpin
Copy link
Copy Markdown
Contributor

kgilpin commented Oct 6, 2024

Tests still failing.

@dividedmind dividedmind force-pushed the feat/support-vertex-ai branch from f8a16c3 to aa92192 Compare October 13, 2024 11:09
@dividedmind
Copy link
Copy Markdown
Collaborator Author

dividedmind commented Oct 13, 2024

I forked the library that was ESM-only and added a CommonJS build (kriasoft/web-auth-library#32) — hopefully this will make it work without fuss.

@kgilpin
Copy link
Copy Markdown
Contributor

kgilpin commented Oct 13, 2024

Thanks @dividedmind

Do you have thoughts on the error we were seeing in the benchmark? Would really like to get a clean run though with Gemini.

@dividedmind dividedmind force-pushed the feat/support-vertex-ai branch from aa92192 to a31dfaa Compare October 13, 2024 11:38
@dividedmind
Copy link
Copy Markdown
Collaborator Author

dividedmind commented Oct 13, 2024

Thanks @dividedmind

Do you have thoughts on the error we were seeing in the benchmark? Would really like to get a clean run though with Gemini.

You mean the recitation problem? I'm looking at it next.

@dividedmind dividedmind force-pushed the feat/support-vertex-ai branch 2 times, most recently from 99e51f5 to f798f3e Compare October 18, 2024 15:08
dustinbyrne
dustinbyrne approved these changes Oct 18, 2024
View reviewed changes
Comment thread packages/navie/src/services/google-vertexai-completion-service.ts
Comment on lines +112 to +119
if (attempt < maxAttempts - 1 && tokens.length === 0) {
const nextAttempt = CompletionRetryDelay * 2 ** attempt;
warn(`Received ${JSON.stringify(cause)}, retrying in ${nextAttempt}ms`);
await new Promise<void>((resolve) => {
setTimeout(resolve, nextAttempt);
});
continue;
}
Copy link
Copy Markdown
Contributor

@dustinbyrne dustinbyrne Oct 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this doesn't attempt to prune input tokens

Copy link
Copy Markdown
Collaborator Author

@dividedmind dividedmind Oct 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I figured it's unlikely to be necessary with 2M token input limit...

Comment thread packages/navie/src/services/vector-terms-service.ts
Comment on lines +98 to +100
context: z.string(),
instructions: z.string(),
terms: z.array(z.string()),
Copy link
Copy Markdown
Contributor

@dustinbyrne dustinbyrne Oct 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description of these properties could move out of the prompt and in to the schema

Copy link
Copy Markdown
Collaborator Author

@dividedmind dividedmind Oct 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, although I tried to change it as little as possible as not to break anything.

@dividedmind dividedmind force-pushed the feat/support-vertex-ai branch from f798f3e to f1e6372 Compare October 21, 2024 18:47
@dividedmind dividedmind merged commit f37e4a5 into main Oct 22, 2024
@dividedmind dividedmind deleted the feat/support-vertex-ai branch October 22, 2024 12:17
@appland-release
Copy link
Copy Markdown
Contributor

appland-release commented Oct 22, 2024

🎉 This PR is included in version @appland/navie-v1.34.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kgilpin kgilpin Awaiting requested review from kgilpin

1 more reviewer

@dustinbyrne dustinbyrne dustinbyrne approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@dividedmind dividedmind

Labels

released

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dividedmind @kgilpin @appland-release @dustinbyrne