🚀 Serverless LiteLLM Vertex AI Proxy on Google Cloud Run

A lightweight, serverless proxy packaged for Google Cloud Run that emulates the standard OpenAI API schema. It dynamically translates LLM requests into Google Cloud Vertex AI regional endpoints using the latest Gemini 2.5 and Gemini 3 / 3.1 Preview models.

This allows developer tools like Cursor IDE, VS Code Continue, or any custom code assistant to securely leverage high-performance, data-resident Vertex AI Gemini endpoints via standard public HTTPS connections.

---

💎 Cloud Native Features

• Zero-Key Google Authentication: Eliminates the risk of downloading and managing service-account-key.json files. The service leverages Cloud Run's attached runtime Service Account via native Application Default Credentials (ADC).
• Serverless Scalability: Scales automatically from zero to handle intensive multi-file codebase indexing or code refactoring, and scales down to zero when idle, resulting in near-zero operational costs.
• Dynamic Port Injection: Automatically binds to Cloud Run's dynamically injected runtime $PORT environment variable.
• IDE Compatibility Engine (drop_params): Intercepts and strips out unsupported OpenAI-specific parameters (e.g. presence_penalty, logit_bias) automatically sent by IDEs that cause direct Vertex AI calls to crash.
• End-to-End Secure Gateway: Publicly exposes an authenticated endpoint requiring an custom LITELLM_MASTER_KEY as a Bearer token, securing your service on the open internet.

---

🚀 1-Click Deployment

We have provided an all-in-one bash automation script deploy_cloud_run.sh that handles the entire Google Cloud pipeline from your local machine without needing Docker installed locally (it builds directly in the cloud via Cloud Build!).

Execution Steps:

1. Open your terminal in this workspace.
2. Make sure you are logged in and your active project is set:

   gcloud auth login
   gcloud config set project YOUR_GCP_PROJECT_ID

3. Run the deployment manager:

   ./deploy_cloud_run.sh

4. Enter your target deployment region (e.g., us-central1, europe-west3) and your desired Proxy Master Key (or hit Enter to let the script generate an ultra-secure key automatically).

The script will enable Google APIs, provision an Artifact Registry repository, compile the image via Google Cloud Build, deploy to Cloud Run, and return your public service URL.

---

🔐 Mandatory IAM Permissions (Execute After Deploy)

To allow Cloud Run to communicate with the Vertex AI APIs without hardcoded JSON files, you MUST grant the Vertex AI User role to the Cloud Run service account.

Run this snippet in your terminal right after deployment finishes (replace YOUR_GCP_PROJECT_ID and PROJECT_NUMBER accordingly):

# Extract the runtime account or use the default Compute Service Account:
gcloud projects add-iam-policy-binding YOUR_GCP_PROJECT_ID \
    --member="serviceAccount:YOUR_PROJECT_NUMBER-compute@developer.gserviceaccount.com" \
    --role="roles/aiplatform.user"

Note: The exact gcloud command containing your project's custom numbers will be automatically generated and printed for you at the end of the ./deploy_cloud_run.sh script!

---

🛠️ IDE Configuration Guide

1. Cursor IDE Setup

1. Open Cursor Settings (Ctrl + , or click the gear icon in the top right corner).
2. Navigate to Models.
3. Find OpenAI API / OpenAI Resource Override.
4. API Key: Paste your secure LITELLM_MASTER_KEY (generated by the deployment script).
5. Override Base URL: Enter your public Cloud Run service URL followed by /v1:

   https://litellm-vertex-proxy-xxxxxxx.run.app/v1
   

6. Enable Target Models: Click + Add Model and add your desired Gemini entries:
   • gemini-2.5-pro (Recommended for high-intelligence tasks)
   • gemini-2.5-flash (Recommended for autocompletion)
   • gemini-3-flash-preview
   • gemini-3.1-pro-preview
7. Disable "Auto" mode in cursor and select one of the models you just created
8. Disable all the other models (unless you want to use them)
9. Start coding! Select these from any chat prompt or inline code generation window.

---

2. VS Code Continue Extension Setup

Add this model block to your ~/.continue/config.json file under the "models" array:

{
  "models": [
    {
      "title": "Gemini 2.5 Pro (Cloud Run)",
      "provider": "openai",
      "model": "gemini-2.5-pro",
      "apiBase": "https://YOUR_CLOUD_RUN_URL.run.app/v1",
      "apiKey": "YOUR_LITELLM_MASTER_KEY"
    },
    {
      "title": "Gemini 3.1 Pro Preview (Cloud Run)",
      "provider": "openai",
      "model": "gemini-3.1-pro-preview",
      "apiBase": "https://YOUR_CLOUD_RUN_URL.run.app/v1",
      "apiKey": "YOUR_LITELLM_MASTER_KEY"
    }
  ]
}

---

🧪 E2E Verification Curl Commands

To verify your newly deployed Cloud Run proxy, execute these from any terminal:

1. Non-Streaming Chat Completion Request

curl -X POST https://YOUR_CLOUD_RUN_URL.run.app/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_LITELLM_MASTER_KEY" \
  -d '{
    "model": "gemini-2.5-flash",
    "messages": [
      {"role": "user", "content": "Write a one-word confirmation."}
    ]
  }'

2. Real-time Streaming Code Assistant Simulation

curl -X POST https://YOUR_CLOUD_RUN_URL.run.app/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_LITELLM_MASTER_KEY" \
  -d '{
    "model": "gemini-3.1-pro-preview",
    "stream": true,
    "messages": [
      {"role": "user", "content": "Count from 1 to 3."}
    ]
  }'

---

📊 Cloud Environment Variables

If you ever need to update settings via the Google Cloud Console or gcloud run services update, these are the core environment variables managed by the service:

Variable Key	Purpose	Description
GCP_PROJECT_ID	Backend Targeting	Core GCP Project ID containing your Vertex AI APIs
GCP_REGION	Vertex Endpoint Regionality	Location of Vertex AI regional endpoint (e.g. us-central1, europe-west3)
LITELLM_MASTER_KEY	Service Security Gateway	Secret token used to authenticate all inbound IDE API requests
PORT	System Runtime	Injected dynamically by Cloud Run, defaults to 4000 if missing