Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 36 additions & 4 deletions sample_solutions/DocSummarization/.env.example
Original file line number Diff line number Diff line change
@@ -1,6 +1,38 @@
# Docker Compose Configuration
# Inference API Configuration
# Note: model names must match one of the deployed models with Enterprise Inference.
# INFERENCE_API_ENDPOINT: URL to your inference service (without /v1 suffix). Uncomment ONE of the below.
# - For GenAI Gateway:
#INFERENCE_API_ENDPOINT=https://api.example.com
# - For APISIX Gateway (default), requires the model name in the URL path:
INFERENCE_API_ENDPOINT=https://api.example.com/Llama-3.1-8B-Instruct

# INFERENCE_API_TOKEN: Authentication token/API key for the inference service
# - For GenAI Gateway: Your GenAI Gateway API key
# - For APISIX Gateway: Your APISIX authentication token
INFERENCE_API_TOKEN=your-pre-generated-token-here

# Model Configuration
# IMPORTANT: Use the full model name i.e. model card
INFERENCE_MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct

# LLM Configuration
LLM_TEMPERATURE=0.7
LLM_MAX_TOKENS=2000

# Local URL Endpoint (only needed for non-public domains)
# If using a local domain like api.example.com mapped to localhost:
# Set this to: api.example.com (domain without https://)
# If using a public domain, set any placeholder value like: not-needed
# If using a local domain like api.example.com mapped to localhost, set to the domain without https://
# Otherwise, set to: not-needed
LOCAL_URL_ENDPOINT=not-needed

# Service Configuration
SERVICE_PORT=8000
LOG_LEVEL=INFO

# File Upload Limits
MAX_FILE_SIZE=524288000
MAX_PDF_SIZE=52428800
MAX_PDF_PAGES=100

# SSL Verification Settings
# Set to false only for dev with self-signed certs
VERIFY_SSL=true
134 changes: 26 additions & 108 deletions sample_solutions/DocSummarization/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,38 +101,34 @@ The application consists of:
Before you begin, ensure you have the following installed:

- **Docker and Docker Compose**
- **Enterprise inference endpoint access** (token-based authentication)
- **Enterprise Inference endpoint access** (token-based authentication, see below for models and configs)

#### Deploy Required Models

See the table below for supported models, hardware, and gateway configuration.

| Model | Xeon w/APISIX/Keycloak | Xeon w/GenAI Gateway | Gaudi w/APISIX/Keycloak | Gaudi w/GenAI Gateway |
|---|:---:|:---:|:---:|:---:|
| **meta-llama/Llama-3.1-8B-Instruct** | ❌ | ❌ | ✅ Validated on Dell XE7740 | ✅ Validated on Dell XE7740 |
| **Qwen/Qwen3-4B-Instruct-2507** | ✅ Validated on Dell XE7740 | ✅ Validated on Dell XE7740 | ❌ | ❌ |

### Required API Configuration

**For Inference Service:**

This application supports multiple inference deployment patterns:

- **GenAI Gateway**: Provide your GenAI Gateway URL and API key
**GenAI Gateway**: Provide your GenAI Gateway URL and API key
- URL format: https://api.example.com
- To generate the GenAI Gateway API key, use the [generate-vault-secrets.sh](https://github.com/opea-project/Enterprise-Inference/blob/main/core/scripts/generate-vault-secrets.sh) script
- The API key is the `litellm_master_key` value from the generated `vault.yml` file

- **APISIX Gateway**: Provide your APISIX Gateway URL and authentication token
- The API key is the litellm_master_key value from the generated vault.yml file

**APISIX Gateway**: Provide your APISIX Gateway URL and authentication token
- URL format: https://api.example.com/Llama-3.1-8B-Instruct
- Note: APISIX requires the model name in the URL path
- To generate the APISIX authentication token, use the [generate-token.sh](https://github.com/opea-project/Enterprise-Inference/blob/main/core/scripts/generate-token.sh) script
- The token is generated using Keycloak client credentials

### Local Development Configuration

**For Local Testing Only (Optional)**

If you're testing with a local inference endpoint using a custom domain (e.g., `api.example.com` mapped to localhost in your hosts file):

1. Edit `backend/.env` and set:
```bash
LOCAL_URL_ENDPOINT=api.example.com
```
(Use the domain name from your INFERENCE_API_ENDPOINT without `https://`)

2. This allows Docker containers to resolve your local domain correctly.

**Note:** For public domains or cloud-hosted endpoints, leave the default value `not-needed`.

### Verify Docker Installation

```bash
Expand All @@ -159,95 +155,28 @@ cd Enterprise-Inference/sample_solutions/DocSummarization

### Set up the Environment

This application requires **two `.env` files** for proper configuration:

1. **Root `.env` file** (for Docker Compose variables)
2. **`backend/.env` file** (for backend application configuration)

#### Step 1: Create Root `.env` File

You can either copy from the example file:
This application requires an `.env` file in the root directory for proper configuration. Create it using [.env.example](./.env.example) with the commands below:

```bash
cp .env.example .env
```

Then edit `.env` if needed, **OR** create it directly:

```bash
# From the DocSummarization directory
cat > .env << EOF
# Docker Compose Configuration
LOCAL_URL_ENDPOINT=not-needed
EOF
```

**Note:** If using a local domain (e.g., `api.example.com` mapped to localhost), replace `not-needed` with your domain name (without `https://`).

#### Step 2: Create `backend/.env` File

You can either copy from the example file:

```bash
cp backend/.env.example backend/.env
```

Then edit `backend/.env` with your actual credentials, **OR** create it directly:

```bash
cat > backend/.env << EOF
# Inference API Configuration
# INFERENCE_API_ENDPOINT: URL to your inference service (without /v1 suffix)
# - For GenAI Gateway: https://genai-gateway.example.com
# - For APISIX Gateway: https://apisix-gateway.example.com/Llama-3.1-8B-Instruct
# Note: APISIX Gateway requires the model name in the URL path
#
# INFERENCE_API_TOKEN: Authentication token/API key for the inference service
# - For GenAI Gateway: Your GenAI Gateway API key
# - For APISIX Gateway: Your APISIX authentication token
INFERENCE_API_ENDPOINT=https://your-actual-api-endpoint.com
INFERENCE_API_TOKEN=your-actual-token-here

# Model Configuration
# IMPORTANT: Use the full model names as they appear in your inference service
# Check available models: curl https://your-api-endpoint.com/v1/models -H "Authorization: Bearer your-token"
INFERENCE_MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct

# LLM Configuration
LLM_TEMPERATURE=0.7
LLM_MAX_TOKENS=2000

# Local URL Endpoint (for Docker)
LOCAL_URL_ENDPOINT=not-needed

# Service Configuration
SERVICE_PORT=8000
LOG_LEVEL=INFO

# File Upload Limits
MAX_FILE_SIZE=524288000
MAX_PDF_SIZE=52428800
MAX_PDF_PAGES=100

# SSL Verification Settings
# Set to false only for dev with self-signed certs
VERIFY_SSL=true
EOF
```
Then modify it as needed, with special consideration to certain environment variables mentioned below. Read through the .env file for full instructions.

**Important Configuration Notes:**

- **INFERENCE_API_ENDPOINT**: Your actual inference service URL (replace `https://your-actual-api-endpoint.com`)
- For APISIX/Keycloak deployments, the model name must be included in the endpoint URL (e.g., `https://apisix-gateway.example.com/Llama-3.1-8B-Instruct`)
- **INFERENCE_API_ENDPOINT**: Your actual inference service URL (replace `https://api.example.com`)
- For APISIX/Keycloak deployments, the model name must be included in the endpoint URL (e.g., `https://api.example.com/Llama-3.1-8B-Instruct`)
- **INFERENCE_API_TOKEN**: Your actual pre-generated authentication token
- **INFERENCE_MODEL_NAME**: Use the exact model name from your inference service
- To check available models: `curl https://your-api-endpoint.com/v1/models -H "Authorization: Bearer your-token"`
- **LOCAL_URL_ENDPOINT**: Only needed if using local domain mapping
- To check available models: `curl https://api.example.com/v1/models -H "Authorization: Bearer your-token"`
- **LOCAL_URL_ENDPOINT**: Only needed if using local domain mapping (i.e. `api.example.com` mapped to localhost) for Docker containers to resolve correctly.
- Use the domain name from INFERENCE_API_ENDPOINT without `https://`
- For public domains or cloud-hosted endpoints, leave the default value `not-needed`
- **VERIFY_SSL**: Controls SSL certificate verification (default: `true`)
- Set to `false` only for development environments with self-signed certificates
- Keep as `true` for production environments

**Note**: The docker-compose.yml file automatically loads environment variables from both `.env` (root) and `./backend/.env` (backend) files.
**Note**: The docker-compose.yaml file automatically loads environment variables from `.env` for the backend service.

### Running the Application

Expand Down Expand Up @@ -329,14 +258,3 @@ docker compose down
For comprehensive troubleshooting guidance, common issues, and solutions, refer to:

[TROUBLESHOOTING.md](./TROUBLESHOOTING.md)

---

## Additional Info

The following models have been validated with DocSummarization:

| Model | Hardware |
|-------|----------|
| **meta-llama/Llama-3.1-8B-Instruct** | Gaudi |
| **Qwen/Qwen3-4B-Instruct-2507** | Xeon |
39 changes: 0 additions & 39 deletions sample_solutions/DocSummarization/backend/.env.example

This file was deleted.

31 changes: 1 addition & 30 deletions sample_solutions/DocSummarization/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ services:
ports:
- "8000:8000"
env_file:
- ./backend/.env
- .env
volumes:
- ./backend:/app
extra_hosts:
Expand All @@ -34,32 +34,3 @@ services:
networks:
app_network:
driver: bridge

# =============================================================================
# Usage Instructions:
# =============================================================================
#
# 1. Create .env file in root directory (for Docker Compose variables):
# cat > .env << EOF
# LOCAL_URL_ENDPOINT=not-needed
# EOF
#
# 2. Copy backend/.env.example to backend/.env and configure your inference API credentials:
# cp backend/.env.example backend/.env
# # Edit backend/.env with your INFERENCE_API_ENDPOINT and INFERENCE_API_TOKEN
#
# 3. Start the application:
# docker compose up -d --build
#
# 4. Access the application:
# - Frontend: http://localhost:5173
# - Backend API: http://localhost:8000
# - API Docs: http://localhost:8000/docs
#
# 5. View logs:
# docker compose logs -f
#
# 6. Stop the application:
# docker compose down
#
# =============================================================================
Loading