diff --git a/sample_solutions/DocSummarization/.env.example b/sample_solutions/DocSummarization/.env.example index 296dcca5..c5519ce6 100644 --- a/sample_solutions/DocSummarization/.env.example +++ b/sample_solutions/DocSummarization/.env.example @@ -1,6 +1,38 @@ -# Docker Compose Configuration +# Inference API Configuration +# Note: model names must match one of the deployed models with Enterprise Inference. +# INFERENCE_API_ENDPOINT: URL to your inference service (without /v1 suffix). Uncomment ONE of the below. +# - For GenAI Gateway: +#INFERENCE_API_ENDPOINT=https://api.example.com +# - For APISIX Gateway (default), requires the model name in the URL path: +INFERENCE_API_ENDPOINT=https://api.example.com/Llama-3.1-8B-Instruct + +# INFERENCE_API_TOKEN: Authentication token/API key for the inference service +# - For GenAI Gateway: Your GenAI Gateway API key +# - For APISIX Gateway: Your APISIX authentication token +INFERENCE_API_TOKEN=your-pre-generated-token-here + +# Model Configuration +# IMPORTANT: Use the full model name i.e. model card +INFERENCE_MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct + +# LLM Configuration +LLM_TEMPERATURE=0.7 +LLM_MAX_TOKENS=2000 + # Local URL Endpoint (only needed for non-public domains) -# If using a local domain like api.example.com mapped to localhost: -# Set this to: api.example.com (domain without https://) -# If using a public domain, set any placeholder value like: not-needed +# If using a local domain like api.example.com mapped to localhost, set to the domain without https:// +# Otherwise, set to: not-needed LOCAL_URL_ENDPOINT=not-needed + +# Service Configuration +SERVICE_PORT=8000 +LOG_LEVEL=INFO + +# File Upload Limits +MAX_FILE_SIZE=524288000 +MAX_PDF_SIZE=52428800 +MAX_PDF_PAGES=100 + +# SSL Verification Settings +# Set to false only for dev with self-signed certs +VERIFY_SSL=true diff --git a/sample_solutions/DocSummarization/README.md b/sample_solutions/DocSummarization/README.md index 144e4148..1715e56e 100644 --- a/sample_solutions/DocSummarization/README.md +++ b/sample_solutions/DocSummarization/README.md @@ -101,7 +101,16 @@ The application consists of: Before you begin, ensure you have the following installed: - **Docker and Docker Compose** -- **Enterprise inference endpoint access** (token-based authentication) +- **Enterprise Inference endpoint access** (token-based authentication, see below for models and configs) + +#### Deploy Required Models + +See the table below for supported models, hardware, and gateway configuration. + +| Model | Xeon w/APISIX/Keycloak | Xeon w/GenAI Gateway | Gaudi w/APISIX/Keycloak | Gaudi w/GenAI Gateway | +|---|:---:|:---:|:---:|:---:| +| **meta-llama/Llama-3.1-8B-Instruct** | ❌ | ❌ | ✅ Validated on Dell XE7740 | ✅ Validated on Dell XE7740 | +| **Qwen/Qwen3-4B-Instruct-2507** | ✅ Validated on Dell XE7740 | ✅ Validated on Dell XE7740 | ❌ | ❌ | ### Required API Configuration @@ -109,30 +118,17 @@ Before you begin, ensure you have the following installed: This application supports multiple inference deployment patterns: -- **GenAI Gateway**: Provide your GenAI Gateway URL and API key +**GenAI Gateway**: Provide your GenAI Gateway URL and API key + - URL format: https://api.example.com - To generate the GenAI Gateway API key, use the [generate-vault-secrets.sh](https://github.com/opea-project/Enterprise-Inference/blob/main/core/scripts/generate-vault-secrets.sh) script - - The API key is the `litellm_master_key` value from the generated `vault.yml` file - -- **APISIX Gateway**: Provide your APISIX Gateway URL and authentication token + - The API key is the litellm_master_key value from the generated vault.yml file + +**APISIX Gateway**: Provide your APISIX Gateway URL and authentication token + - URL format: https://api.example.com/Llama-3.1-8B-Instruct + - Note: APISIX requires the model name in the URL path - To generate the APISIX authentication token, use the [generate-token.sh](https://github.com/opea-project/Enterprise-Inference/blob/main/core/scripts/generate-token.sh) script - The token is generated using Keycloak client credentials -### Local Development Configuration - -**For Local Testing Only (Optional)** - -If you're testing with a local inference endpoint using a custom domain (e.g., `api.example.com` mapped to localhost in your hosts file): - -1. Edit `backend/.env` and set: - ```bash - LOCAL_URL_ENDPOINT=api.example.com - ``` - (Use the domain name from your INFERENCE_API_ENDPOINT without `https://`) - -2. This allows Docker containers to resolve your local domain correctly. - -**Note:** For public domains or cloud-hosted endpoints, leave the default value `not-needed`. - ### Verify Docker Installation ```bash @@ -159,95 +155,28 @@ cd Enterprise-Inference/sample_solutions/DocSummarization ### Set up the Environment -This application requires **two `.env` files** for proper configuration: - -1. **Root `.env` file** (for Docker Compose variables) -2. **`backend/.env` file** (for backend application configuration) - -#### Step 1: Create Root `.env` File - -You can either copy from the example file: +This application requires an `.env` file in the root directory for proper configuration. Create it using [.env.example](./.env.example) with the commands below: ```bash cp .env.example .env ``` - -Then edit `.env` if needed, **OR** create it directly: - -```bash -# From the DocSummarization directory -cat > .env << EOF -# Docker Compose Configuration -LOCAL_URL_ENDPOINT=not-needed -EOF -``` - -**Note:** If using a local domain (e.g., `api.example.com` mapped to localhost), replace `not-needed` with your domain name (without `https://`). - -#### Step 2: Create `backend/.env` File - -You can either copy from the example file: - -```bash -cp backend/.env.example backend/.env -``` - -Then edit `backend/.env` with your actual credentials, **OR** create it directly: - -```bash -cat > backend/.env << EOF -# Inference API Configuration -# INFERENCE_API_ENDPOINT: URL to your inference service (without /v1 suffix) -# - For GenAI Gateway: https://genai-gateway.example.com -# - For APISIX Gateway: https://apisix-gateway.example.com/Llama-3.1-8B-Instruct -# Note: APISIX Gateway requires the model name in the URL path -# -# INFERENCE_API_TOKEN: Authentication token/API key for the inference service -# - For GenAI Gateway: Your GenAI Gateway API key -# - For APISIX Gateway: Your APISIX authentication token -INFERENCE_API_ENDPOINT=https://your-actual-api-endpoint.com -INFERENCE_API_TOKEN=your-actual-token-here - -# Model Configuration -# IMPORTANT: Use the full model names as they appear in your inference service -# Check available models: curl https://your-api-endpoint.com/v1/models -H "Authorization: Bearer your-token" -INFERENCE_MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct - -# LLM Configuration -LLM_TEMPERATURE=0.7 -LLM_MAX_TOKENS=2000 - -# Local URL Endpoint (for Docker) -LOCAL_URL_ENDPOINT=not-needed - -# Service Configuration -SERVICE_PORT=8000 -LOG_LEVEL=INFO - -# File Upload Limits -MAX_FILE_SIZE=524288000 -MAX_PDF_SIZE=52428800 -MAX_PDF_PAGES=100 - -# SSL Verification Settings -# Set to false only for dev with self-signed certs -VERIFY_SSL=true -EOF -``` +Then modify it as needed, with special consideration to certain environment variables mentioned below. Read through the .env file for full instructions. **Important Configuration Notes:** -- **INFERENCE_API_ENDPOINT**: Your actual inference service URL (replace `https://your-actual-api-endpoint.com`) - - For APISIX/Keycloak deployments, the model name must be included in the endpoint URL (e.g., `https://apisix-gateway.example.com/Llama-3.1-8B-Instruct`) +- **INFERENCE_API_ENDPOINT**: Your actual inference service URL (replace `https://api.example.com`) + - For APISIX/Keycloak deployments, the model name must be included in the endpoint URL (e.g., `https://api.example.com/Llama-3.1-8B-Instruct`) - **INFERENCE_API_TOKEN**: Your actual pre-generated authentication token - **INFERENCE_MODEL_NAME**: Use the exact model name from your inference service - - To check available models: `curl https://your-api-endpoint.com/v1/models -H "Authorization: Bearer your-token"` -- **LOCAL_URL_ENDPOINT**: Only needed if using local domain mapping + - To check available models: `curl https://api.example.com/v1/models -H "Authorization: Bearer your-token"` +- **LOCAL_URL_ENDPOINT**: Only needed if using local domain mapping (i.e. `api.example.com` mapped to localhost) for Docker containers to resolve correctly. + - Use the domain name from INFERENCE_API_ENDPOINT without `https://` + - For public domains or cloud-hosted endpoints, leave the default value `not-needed` - **VERIFY_SSL**: Controls SSL certificate verification (default: `true`) - Set to `false` only for development environments with self-signed certificates - Keep as `true` for production environments -**Note**: The docker-compose.yml file automatically loads environment variables from both `.env` (root) and `./backend/.env` (backend) files. +**Note**: The docker-compose.yaml file automatically loads environment variables from `.env` for the backend service. ### Running the Application @@ -329,14 +258,3 @@ docker compose down For comprehensive troubleshooting guidance, common issues, and solutions, refer to: [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) - ---- - -## Additional Info - -The following models have been validated with DocSummarization: - -| Model | Hardware | -|-------|----------| -| **meta-llama/Llama-3.1-8B-Instruct** | Gaudi | -| **Qwen/Qwen3-4B-Instruct-2507** | Xeon | diff --git a/sample_solutions/DocSummarization/backend/.env.example b/sample_solutions/DocSummarization/backend/.env.example deleted file mode 100644 index ca336a55..00000000 --- a/sample_solutions/DocSummarization/backend/.env.example +++ /dev/null @@ -1,39 +0,0 @@ -# Inference API Configuration -# INFERENCE_API_ENDPOINT: URL to your inference service (without /v1 suffix) -# - For GenAI Gateway: https://genai-gateway.example.com -# - For APISIX Gateway: https://apisix-gateway.example.com/Llama-3.1-8B-Instruct -# Note: APISIX Gateway requires the model name in the URL path -# -# INFERENCE_API_TOKEN: Authentication token/API key for the inference service -# - For GenAI Gateway: Your GenAI Gateway API key -# - For APISIX Gateway: Your APISIX authentication token -INFERENCE_API_ENDPOINT=https://api.example.com -INFERENCE_API_TOKEN=your-pre-generated-token-here - -# Model Configuration -# IMPORTANT: Use the full model names as they appear in your inference service -# Check available models: curl https://your-api-endpoint.com/v1/models -H "Authorization: Bearer your-token" -INFERENCE_MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct - -# LLM Configuration -LLM_TEMPERATURE=0.7 -LLM_MAX_TOKENS=2000 - -# Local URL Endpoint (only needed for non-public domains) -# If using a local domain like api.example.com mapped to localhost: -# Set this to: api.example.com (domain without https://) -# If using a public domain, set any placeholder value like: not-needed -LOCAL_URL_ENDPOINT=not-needed - -# Service Configuration -SERVICE_PORT=8000 -LOG_LEVEL=INFO - -# File Upload Limits -MAX_FILE_SIZE=524288000 -MAX_PDF_SIZE=52428800 -MAX_PDF_PAGES=100 - -# SSL Verification Settings -# Set to false only for dev with self-signed certs -VERIFY_SSL=true diff --git a/sample_solutions/DocSummarization/docker-compose.yml b/sample_solutions/DocSummarization/docker-compose.yml index 44ff960a..717b2671 100644 --- a/sample_solutions/DocSummarization/docker-compose.yml +++ b/sample_solutions/DocSummarization/docker-compose.yml @@ -8,7 +8,7 @@ services: ports: - "8000:8000" env_file: - - ./backend/.env + - .env volumes: - ./backend:/app extra_hosts: @@ -34,32 +34,3 @@ services: networks: app_network: driver: bridge - -# ============================================================================= -# Usage Instructions: -# ============================================================================= -# -# 1. Create .env file in root directory (for Docker Compose variables): -# cat > .env << EOF -# LOCAL_URL_ENDPOINT=not-needed -# EOF -# -# 2. Copy backend/.env.example to backend/.env and configure your inference API credentials: -# cp backend/.env.example backend/.env -# # Edit backend/.env with your INFERENCE_API_ENDPOINT and INFERENCE_API_TOKEN -# -# 3. Start the application: -# docker compose up -d --build -# -# 4. Access the application: -# - Frontend: http://localhost:5173 -# - Backend API: http://localhost:8000 -# - API Docs: http://localhost:8000/docs -# -# 5. View logs: -# docker compose logs -f -# -# 6. Stop the application: -# docker compose down -# -# =============================================================================